Double-channel data transmission redundancy method and device and computer equipment
By dividing the storage area using a sliding window mechanism, the performance consumption and memory usage issues of existing data transmission deduplication algorithms are resolved, achieving efficient and reliable data packet processing and ensuring the normal operation of applications such as games.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING YOUZHUJU NETWORK TECH CO LTD
- Filing Date
- 2022-10-13
- Publication Date
- 2026-07-21
AI Technical Summary
In real-time interactive scenarios, existing data transmission deduplication algorithms such as hash tables and array methods suffer from high performance consumption, high memory usage, and sequence number wrapping issues, leading to data packet loss and affecting the normal operation of applications such as games.
A sliding window mechanism is used to divide the storage area of data packets. The range of the storage address is determined by the sequence information. The live, tolerable and expired intervals are distinguished and marked and deduplicated to avoid hash collisions and duplicate sequence numbers.
It reduces memory usage, simplifies computational logic, avoids hash collisions and packet dropping, and ensures the reliability of data transmission and the continuity of the game.
Smart Images
Figure CN115551012B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a dual-channel data transmission deduplication method, apparatus, and computer equipment. Background Technology
[0002] In real-time interactive scenarios with high latency requirements, such as mobile games, two links, such as mobile networks and Wi-Fi, are typically used to transmit data in order to ensure data transmission speed and reduce packet loss rate. However, this process results in the same data being transmitted twice. Therefore, data deduplication is required at the receiving end to prevent the duplicate processing of the same data.
[0003] In existing technologies, methods for data deduplication, such as using hash table deduplication algorithms, can cause performance consumption when calculating hash values, and the logic is relatively complex. Furthermore, hash collisions can occur. If data packets with hash collisions are discarded, it can easily lead to the discarding of data packets that actually need to be stored.
[0004] Another deduplication algorithm is similar to the hash table-based deduplication algorithm, except that the hash table is replaced by an array. However, this method needs to store all msgids, which will inevitably consume a large amount of memory.
[0005] Furthermore, none of the methods mentioned above solve the serial number wrapping problem. This is because the msgid is always represented by 4 bytes, with a value range of 0-2^32-1. Suppose the game runs for a long time, causing the msgid to increment from 0 to 2^32-1 and then start incrementing again from 0. In this case, because packets with msgids of 0, 1, 2, etc., were marked at the start of the game, these packets are discarded, preventing the game from continuing. Summary of the Invention
[0006] This disclosure provides a dual-channel data transmission deduplication method, apparatus, and computer equipment to solve some or all of the technical problems in the prior art.
[0007] Firstly, this disclosure provides a dual-channel data transmission deduplication method, including:
[0008] Obtain the sequence information of the data packets;
[0009] Based on the sequence information, determine the region range to which the storage address of the sequence information belongs;
[0010] When the range of the storage address is determined to be within the first interval, the storage address is marked and deduplicated. The first interval is formed by a sliding window.
[0011] Alternatively, when it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window; the second and third intervals are sequentially shifted according to the displacement of the sliding window; and the storage addresses corresponding to the sequence information are marked.
[0012] Alternatively, when it is determined that the storage address corresponding to any sequence information is currently located within the third interval, the data packet corresponding to that sequence information is discarded. Here, the first interval, the second interval, and the third interval together constitute the total closed-loop area.
[0013] The method provided in this embodiment obtains the sequence information of a data packet, determines the region to which the storage address of the sequence information belongs based on the sequence information, and if the region to which the storage address belongs is within the first interval, it indicates that the data packet is a valid data packet and can be marked and deduplicated to complete the marking of the received data packet. Of course, if it is a duplicate data packet, it is discarded after deduplication. The first interval can be understood as the survival interval. The second interval can be understood as the tolerance interval. When the storage address of the sequence information belongs to the second interval, it indicates that the network may have been disconnected, resulting in the loss of some data. Therefore, the error position corresponding to the sequence number of the newly received data packet has exceeded the region currently "circled" by the sliding window. At this time, the sliding window can be moved until the storage address falls within the sliding window, and the storage address of the sequence information is marked to indicate that the data packet has been received. At the same time, according to the displacement of the sliding window, the second interval and the third interval are shifted sequentially. The third interval is the data packet expiration interval. The first interval, the second interval, and the third interval constitute a "closed loop" region. If the storage address corresponding to the sequence information of any data packet does not belong to either the live range or the tolerable range, then the data packet is considered "old" and needs to be discarded. This method avoids hash collisions caused by hash algorithms during data deduplication, and the calculation logic is simple. Because the sequence information of data packets in this disclosure is only marked within the first range, it does not consume a large amount of memory. Furthermore, the deduplication process in this disclosure does not need to consider whether the sequence number information is duplicated; it only determines whether duplicate data packets exist based on the markings in the storage address. Therefore, even if the sequence number information is duplicated, there will be no conflict because the storage addresses are different, and naturally, the game will not be unable to proceed due to the discarding of data packets.
[0014] Secondly, this disclosure provides a dual-channel data transmission deduplication device, the device comprising:
[0015] The acquisition module is used to acquire the sequence information of data packets;
[0016] The determination module is used to determine the region range to which the storage address of the sequence information belongs based on the sequence information;
[0017] The processing module is used to mark and deduplicate the storage address when it is determined that the region to which the storage address belongs is within the first interval. The first interval is composed of a sliding window.
[0018] Alternatively, when it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window; the second and third intervals are sequentially shifted according to the displacement of the sliding window; and the storage addresses corresponding to the sequence information are marked.
[0019] Alternatively, when it is determined that the storage address corresponding to any sequence information is currently located within the third interval, the data packet corresponding to that sequence information is discarded. Here, the first interval, the second interval, and the third interval together constitute the total closed-loop area.
[0020] The dual-channel data transmission deduplication device provided in this embodiment acquires the sequence information of data packets and determines the region to which the storage address of the sequence information belongs based on the sequence information. If the region to which the storage address belongs is within the first interval, it indicates that the data packet is a valid data packet and can be marked and deduplicated to complete the marking of the received data packet. Of course, if it is a duplicate data packet, it is discarded after deduplication. The first interval can be understood as the survival interval. The second interval can be understood as the tolerance interval. When the storage address of the sequence information belongs to the second interval, it indicates that the network may have been disconnected, resulting in the loss of some data. Therefore, the error position corresponding to the sequence number of the newly received data packet has exceeded the region currently "circled" by the sliding window. At this time, the sliding window can be moved until the storage address falls within the sliding window, and the storage address of the sequence information is marked to indicate that the data packet has been received. At the same time, the second interval and the third interval are sequentially shifted according to the displacement of the sliding window. The third interval is the data packet expiration interval. The first interval, the second interval, and the third interval constitute a "closed loop" region. If the storage address corresponding to the sequence information of any data packet does not belong to either the live range or the tolerable range, then the data packet is considered "old" and needs to be discarded. This method avoids hash collisions caused by hash algorithms during data deduplication, and the calculation logic is simple. Because the sequence information of data packets in this disclosure is only marked within the first range, it does not consume a large amount of memory. Furthermore, the deduplication process in this disclosure does not need to consider whether the sequence number information is duplicated; it only determines whether duplicate data packets exist based on the markings in the storage address. Therefore, even if the sequence number information is duplicated, there will be no conflict because the storage addresses are different, and naturally, the game will not be unable to proceed due to the discarding of data packets.
[0021] Thirdly, a computer device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0022] Memory, used to store computer programs;
[0023] When a processor executes a program stored in memory, it implements the steps of the dual-channel data transmission deduplication method according to any embodiment of the first aspect.
[0024] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by an edge node, implements the steps of the dual-channel data transmission deduplication method, apparatus, and computer device method as described in any embodiment of the first aspect. Attached Figure Description
[0025] Figure 1 This is a schematic flowchart of a dual-channel data transmission deduplication method provided in an embodiment of the present disclosure;
[0026] Figure 2 This is a schematic diagram illustrating the process of data transmission from the client to the service server provided in this disclosure.
[0027] Figure 3 This is a schematic diagram illustrating the process of data transmission from the business server to the client provided in this disclosure;
[0028] Figure 4 This is a schematic diagram of the structural layout of the total closed-loop area provided in this disclosure;
[0029] Figure 5 This is a schematic diagram illustrating the marking of the sequence numbers corresponding to the data packets to be discarded provided in this disclosure;
[0030] Figure 6 This is a flowchart illustrating a method for moving a sliding window to make a storage address fall within the sliding window, as provided in this disclosure.
[0031] Figure 7 A flowchart illustrating another method for moving a sliding window to make a storage address fall within the sliding window, as provided in this disclosure;
[0032] Figure 8 A schematic diagram of the structural layout of another total closed-loop interval range provided in this disclosure;
[0033] Figure 9 A flowchart illustrating another method for moving a sliding window to make a storage address fall within the sliding window, as provided in this disclosure;
[0034] Figure 10 This is a schematic diagram illustrating the specific implementation process of a sliding window that allows a storage address to fall within the sliding window, as provided in this disclosure.
[0035] Figure 11 This is a schematic diagram illustrating a specific example of marking and deduplicating storage addresses provided in this disclosure;
[0036] Figure 12 This is a schematic diagram illustrating another specific example of marking and deduplicating storage addresses provided in this disclosure;
[0037] Figure 13 This is a schematic diagram illustrating a specific example where the sequence information is not marked after falling into the third interval, as provided in this disclosure.
[0038] Figure 14This is a schematic diagram of the structure of a dual-channel data transmission and de-weighting device provided in an embodiment of this disclosure;
[0039] Figure 15 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0041] To facilitate understanding of the embodiments of this disclosure, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this disclosure.
[0042] In response to the technical problems mentioned in the background art, this disclosure provides a dual-channel data transmission deduplication method, apparatus, and computer equipment method.
[0043] The specific steps for deduplication in dual-channel data transmission will be described in detail below. Please refer to the accompanying drawings provided in the embodiments of this disclosure. Figure 1 This is a schematic flowchart illustrating a dual-channel data transmission deduplication method provided in this disclosure. Before introducing the dual-channel data transmission deduplication method provided in this disclosure, we will first introduce an example of a specific application scenario to which this method can be applied. See details... Figure 2 and Figure 3 As shown.
[0044] Figure 2 The diagram illustrates the data transmission process from the client to the business server. The client sends data to the proxy server via both mobile network and WiFi channels. The proxy performs deduplication and then sends the data to the business server.
[0045] Figure 2 The diagram illustrates the transmission of four data packets, each corresponding to a sequence number: 1, 2, 3, and 4. Dual-channel transmission primarily involves two-way data transfer between the client and the proxy server. After deduplication by the proxy server, the data is transmitted to the business server.
[0046] Figure 3The diagram illustrates the data transmission process from the service server to the client. The service server transmits data packets with sequence numbers 1, 2, 3, and 4 to the proxy server. The proxy server then transmits the data to the client via both mobile network and WiFi channels. After receiving the dual-channel data packets, the client performs deduplication to ensure that only one copy of the data packets is received.
[0047] The specific deduplication method for dual-channel data transmission can be implemented using the following steps:
[0048] Step 110: Obtain the sequence information of the data packet.
[0049] Step 120: Determine the region range to which the storage address of the sequence information belongs based on the sequence information.
[0050] Specifically, considering that the sequence number (msgid) of a data packet ranges from 0 to (2^32)-1, we can assume a total closed-loop region, where the storage addresses range from 0 to (2^32)-1. When the sequence information and the storage address have the same value, it is determined that the sequence information can be stored at that storage address. Then, the region to which that storage address belongs is determined.
[0051] Specifically, the total closed-loop area can be divided into three sections: a first section, a second section, and a third section. In reality, only the first section is actual existing storage space; the second and third sections are merely theoretically configured storage spaces for ease of explanation and do not exist in actual application. That is, within the total closed-loop area, only the first section is actual existing storage space; the others are theoretically virtual storage spaces created to implement this disclosed solution.
[0052] The first interval is formed by a sliding window; that is, as the sliding window slides, the storage addresses included in the first interval also change accordingly. As the window slides, the second and third intervals also move sequentially along with the sliding window. See Figure 4 for details. Figure 4 A schematic diagram illustrating the overall closed-loop region is provided. Figure 4 The middle arrow and dashed line indicate that when the sliding window moves by a preset displacement, the second and third intervals will also move by a preset displacement in sequence.
[0053] To determine the range of the storage address of sequence information, we mainly need to see which range the storage address of the current sequence information belongs to.
[0054] This disclosure introduces an expiration mechanism. This mechanism is based on the assumption that upon receiving a data packet with msgid N, packets whose absolute difference between msgid and N is greater than or equal to W are unlikely to survive on the link. These are expired packets and must be discarded.
[0055] Therefore, the first interval, or the size of the sliding window, is W. Packets within the sliding window represent the packets we expect to receive, and these packets within this interval will be marked and deduplicated. If a packet is outside the sliding window, it needs to be further determined whether it belongs to the second interval. The second interval can be understood as a "tolerance interval," designed to prevent situations such as network outages from causing data packet transmission interruptions, resulting in packets outside the sliding window being received and falling into the second interval. This indicates a significant out-of-order / packet loss situation on the network. If this occurs, the sliding window needs to be moved, at least to a position that covers the storage address of the current packet. For example, if the movement distance is H, then packets with sequence numbers between NW and N-W+H (covering both ends) need to be discarded. This means that these packets are already outside the sliding window and should not survive on the data chain, thus adapting to the latest network conditions.
[0056] In fact, due to the movement of the sliding window, the second and third intervals also move accordingly. This means that the sequence number falls into the third interval from N-W+1 to N-W+H (covering both ends). As explained below, when the storage address corresponding to any sequence information currently belongs to the third interval, the data packet corresponding to that sequence information is discarded. The general principle remains the same: only data packets corresponding to sequence numbers falling within the sliding window will survive on the transmission link. If the distance from either endpoint of the sliding window (i.e., the absolute difference between the sequence number of the data packet to be confirmed and the sequence number of either endpoint of the sliding window) is less than or equal to W, it can be marked; otherwise, the data packet to be confirmed will be discarded. See the appendix for details. Figure 5 As shown, attached Figure 5 This diagram illustrates which data packets corresponding to which sequence numbers need to be discarded as the sliding window moves.
[0057] However, since the packets from N-W+1 to N-W+H have been discarded, the network sender will not receive an acknowledgment of receipt for the packets from N-W+1 to N-W+H. It will naturally resend these data packets according to the new sequence number, so there is no need to worry about these data packets not being received by the receiver.
[0058] The area of the second interval can be manually set according to network conditions. Within the total closed-loop area, all areas except those included in the first and second intervals can be included in the third interval. However, only the sequence numbers within the first interval (sliding window) need to be marked and deduplicated. Therefore, only the marked sequence numbers within the first interval actually occupy memory. This significantly reduces the memory usage.
[0059] Step 130: When the range of the storage address is determined to be within the first interval, the storage address is marked and deduplicated.
[0060] As described above, when the storage address of the sequence information belongs to the first interval, the storage address is marked and deduplicated within the first interval.
[0061] The marking process involves marking the storage address corresponding to the sequence information as 1. The deduplication process involves determining whether the storage address has already been marked as 1. If it has already been marked as 1, the currently received data packet is considered a duplicate and is discarded without being marked again.
[0062] or,
[0063] Step 140: When it is determined that the range of the storage address is within the second interval, move the sliding window until the storage address falls into the sliding window.
[0064] Specifically, as described above, if the storage address falls within the second interval, it belongs to the "tolerance interval," indicating that a network failure occurred before this moment, and many data packets did not reach the receiving end, resulting in severe packet loss / out-of-order delivery. The sliding window needs to be moved based on the storage address corresponding to the sequence information of the currently received data packets until the storage address falls within the sliding window. Then, data packets corresponding to the sequence information already stored before the sliding window are discarded. See the above explanation for details, which will not be elaborated upon here.
[0065] Step 150: According to the displacement of the sliding window, sequentially translate the second and third intervals.
[0066] Step 160: Mark the storage address corresponding to the sequence information.
[0067] Specifically, the principle of sequentially moving the second and third intervals according to the displacement of the sliding window has been explained above and will not be elaborated further here. The reason for marking the storage address corresponding to the current sequence information is that, undoubtedly, this is the first time this data packet has been received, so marking is naturally necessary. And precisely because it is the first time, deduplication is not required.
[0068] or,
[0069] Step 170: When it is determined that the current area range of the storage address corresponding to any sequence information is within the third interval, discard the data packet corresponding to that sequence information.
[0070] This section has already been explained in detail above, so it will not be repeated here.
[0071] The dual-channel data transmission deduplication method provided in this disclosure obtains the sequence information of data packets, determines the region to which the storage address of the sequence information belongs based on the sequence information, and if the region to which the storage address belongs is within the first interval, it indicates that the data packet is a valid data packet, and it can be marked and deduplicated to complete the marking of the received data packet. Of course, if it is a duplicate data packet, it is discarded after deduplication. The first interval can be understood as the survival interval. The second interval can be understood as the tolerance interval. When the storage address of the sequence information belongs to the second interval, it indicates that the network may have been disconnected, resulting in the loss of some data. Therefore, the error position corresponding to the sequence number of the newly received data packet has exceeded the region currently "circled" by the sliding window. At this time, the sliding window can be moved until the storage address falls within the sliding window, and the storage address of the sequence information is marked to indicate that the data packet has been received. At the same time, according to the displacement of the sliding window, the second interval and the third interval are shifted sequentially. The third interval is the data packet expiration interval. The first interval, the second interval, and the third interval constitute a "closed loop" region. If the storage address corresponding to the sequence information of any data packet does not belong to either the live range or the tolerable range, then the data packet is considered "old" and needs to be discarded. This method avoids hash collisions caused by hash algorithms during data deduplication, and the calculation logic is simple. Because the sequence information of data packets in this disclosure is only marked within the first range, it does not consume a large amount of memory. Furthermore, the deduplication process in this disclosure does not need to consider whether the sequence number information is duplicated; it only determines whether duplicate data packets exist based on the markings in the storage address. Therefore, even if the sequence number information is duplicated, there will be no conflict because the storage addresses are different, and naturally, the game will not be unable to proceed due to the discarding of data packets.
[0072] It should be explained that although this disclosure uses the same storage address and serial number, it does not cause the problems mentioned in the background section:
[0073] If the game has been going on for a long time, causing the msgid to increment from 0 to 2^32-1 and then start incrementing from 0 again, then because the packets with msgids of 0, 1, 2, etc. were marked at the beginning of the game, these packets will be discarded, causing the game to be unable to continue.
[0074] The underlying principle is the existence of an expiration mechanism in this disclosure. As explained above, this disclosure only marks and deduplicates the sequence numbers corresponding to the data packets in the window. Assuming the game has been running for a long time, the marking information of packets with msgid values of 0, 1, 2, etc., from the previous round will have long been discarded. When a data packet with 0, 1, 2 is received again in the current round and falls within the sliding window, it will be remarked and deduplicated. Other data packets are handled similarly. Therefore, even if data packets with duplicate sequence numbers exist in different polls, they will not be discarded due to duplicate marking, ensuring the effective operation of the game.
[0075] Optionally, as described above, since the total closed-loop region is divided into three spaces in this disclosure, it should include at least three critical addresses. The region between the first critical address and the second critical address is the first interval, the region between the second critical address and the third critical address is the second interval, and the region between the third critical address and the first critical address is the third interval. The storage addresses contained in the first interval, the second interval, and the third interval are adjusted sequentially with the displacement.
[0076] See details Figure 4 As shown, Figure 4 The diagram illustrates the positions of the first critical address, the second critical address, and the third critical address. These three critical addresses are actually different memory addresses mentioned above. Furthermore, the memory addresses corresponding to the first, second, and third critical addresses change as the sliding window moves.
[0077] Optionally, in another embodiment of this disclosure, when it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window. This can be achieved in the following manner, see details below. Figure 6 As shown, the method includes the following steps:
[0078] Step 610: Determine the first distance between the storage address and the first critical address, and determine the first displacement of the sliding window based on the first distance.
[0079] or,
[0080] Step 620: Determine the second distance between the storage address and the second critical address, and determine the second movement displacement of the sliding window based on the second distance.
[0081] Step 630: Move the sliding window according to the first or second displacement so that the storage address falls into the sliding window.
[0082] Specifically, assume that the first interval does not cover the first critical address but covers the second critical address, the second interval does not cover the second critical address but covers the third critical address, and the third interval does not cover the third critical address but covers the first critical address.
[0083] In the first implementation, for example, if the first critical address is 4, the storage address corresponding to the left endpoint of the first interval (sliding window) is (4+1)5, and the sequence information corresponding to the data packet received at the current moment is 20, then the sliding window can be moved by 15 (20-5) units, so that the left endpoint of the sliding window is 19. At this time, the storage address 20 can be included in the first interval.
[0084] In the second implementation, for example, if the second critical address corresponding to the right end of the first interval (sliding window) is 16, and the sequence information corresponding to the data packet received at the current moment is 20, then the sliding window can be moved by 4 units. At this time, the new second critical address is 20, which means that the current sliding window also contains the sequence number 20.
[0085] The two implementation methods described above can be understood as sequential movement methods. Whether using the left or right endpoint of the sliding window as a reference, the distance between the sequence information of the currently received data packet and either endpoint of the sliding window is determined to establish a movement displacement. This displacement is then used to move the sliding window until the storage address falls within it. The specific critical address included in each interval can be defined based on the actual situation. As the critical addresses included in different intervals vary, the algorithm for determining the movement displacement changes slightly. However, this change is merely an adjustment to the algorithm itself and will not be explained further here.
[0086] Optionally, besides the method described above, other methods can also be used to move the sliding window until the storage address falls within it. See details... Figure 7 As shown. Before introducing this method, the first interval needs to be divided into a first sub-interval and a second sub-interval using the fourth critical address, and the second interval needs to be divided into a third sub-interval and a fourth sub-interval using the fifth critical address. Furthermore, the area ranges of the first and third sub-intervals should be configured to be equal.
[0087] Then, perform the following steps:
[0088] Step 710: When the storage address falls within the third sub-interval, while keeping the storage address contained in the second sub-interval unchanged, clear all the markers in the first sub-interval and move the first sub-interval to the area range of the third sub-interval so that the storage address falls within the first sub-interval.
[0089] Step 720: Determine the third distance between the first critical address and the fourth critical address at the current moment.
[0090] Step 730: Move the second and third intervals sequentially according to the third distance.
[0091] See details Figure 8 As shown, Figure 8 This diagram illustrates another possible structural layout of the total closed-loop interval. In this layout, A is the first sub-interval, B is the second sub-interval, and A and B together constitute the first interval; C1 is the third sub-interval, C2 is the fourth sub-interval, and C1 and C2 together constitute the second interval; D is the third interval, and the area encompassed by A is equal to the area encompassed by C1.
[0092] When a data packet's sequence number falls within the range of interval C1, keep interval B unchanged and clear all markers in interval A. Figure 8 In the attached diagram pointed to by step 1, the thick black lines represent the marked positions (this will also be the case in the following text and will not be repeated). Move interval A to interval C1. Then, determine the third distance between the first critical address and the fourth critical address at this moment. Since A and C are equal, the third distance is actually the width of A. Then, according to the third distance, move the second and third intervals sequentially (see...). Figure 8 (See the attached diagram pointed to by step 2). After the move is complete, the storage address has fallen within the first sub-interval (see...). Figure 8 (See attached diagram for step 3).
[0093] In another scenario, when the storage address falls within the fourth sub-interval, the sliding window is moved until the storage address falls within the sliding window. This needs to be implemented as follows (see details). Figure 9 As shown, Figure 9 This diagram illustrates another method flow for moving a sliding window until a storage address falls within the sliding window, as disclosed in this disclosure.
[0094] Specifically, the methods and steps are as follows:
[0095] Step 910: When the storage address falls in the fourth sub-interval, determine the third distance between the storage address and the first critical address, and determine the third displacement of the sliding window based on the third distance.
[0096] Alternatively, in step 920, a fourth distance between the storage address and the second critical address is determined, and based on the fourth distance, a fourth displacement of the sliding window is determined.
[0097] Step 930: Move the sliding window according to the third or fourth movement displacement until the storage address falls into the sliding window.
[0098] Specifically, the methods and steps for determining the third and fourth distances, the third displacement based on the third distance, and the fourth displacement based on the fourth distance are the same as those for determining the first and second distances, the first displacement, and the second displacement described above, and will not be repeated here.
[0099] The method steps of moving the sliding window according to the third or fourth displacement until the storage address falls into the sliding window can also refer to the similar method steps described in the first embodiment of this disclosure, which involve moving the sliding window according to the first or second displacement to make the storage address fall into the sliding window. These steps will not be described in detail here.
[0100] However, in addition to the above-mentioned method of moving the sliding window according to the third or fourth displacement until the storage address falls into the sliding window, the following methods may also be included.
[0101] In one optional example, the area of the first sub-interval is the same as the area of the second sub-interval.
[0102] Set the range of the first sub-interval to be the same as the range of the second sub-interval. Then, when moving, you can use A as the movement step size, and move by the width of A each time.
[0103] For details, please refer to [link / reference]. Figure 10 As shown, the steps may include:
[0104] Step 1: Clear all markers in A and B;
[0105] Steps 2-4: Move windows A and B by the width of A (movement step) each time until the sequence information can fall within A.
[0106] Step 5: At this point, the sequence information falls within window A. The corresponding position of the sequence information should be set to 1 to indicate that the data packet corresponding to the sequence information has been received.
[0107] The reason for setting A and B to be equal is to ensure efficiency and reduce computational load when moving the sliding window. Specifically, setting window A and window B to be equal means that movement can be directly based on the width (movement step) of A. In contrast, if window A is smaller than window B, considering the possibility of out-of-order data packet transmission, assuming that the corresponding sequence information belongs to window B during data packet transmission (meaning the sequence information in window B has already been marked), even if the window sequence information is not yet fully marked, window A still needs to be moved to the previous position of B, and then window B moves forward. The marked data in the original position of window B needs to be copied to the area of window A. However, because the area of window B is larger than that of window A, further calculations are needed to determine how much data needs to be stored in the area of A and which data needs to remain in the area of window B. This introduces many unnecessary calculations, inevitably reducing the efficiency of the sliding window movement. Therefore, it is best to set window A and window B to be equal to ensure efficiency and reduce computational load when moving the sliding window.
[0108] Further optionally, the range of the second interval is greater than or equal to the range of the first sub-interval.
[0109] Specifically, since C consists of two parts, C1 and C2, where C1 needs to be equal to A, this ensures that when A needs to be shifted to the position in C, there must be a specific memory address that maps to the memory address in A. Therefore, the range of C is greater than or equal to the range of A.
[0110] Furthermore, Figure 11 and Figure 12 The diagram illustrates that when sequence information falls within the first or second sub-interval, there is no need to move the sliding window. Then, the corresponding storage address is set to 1, and a deduplication operation is performed.
[0111] Case 1: The serial number msgid falls within A / B, and the corresponding position is not set to 1;
[0112] This indicates that the msgid is being received for the first time. Setting the corresponding bit to 1 signifies that the packet corresponding to the msgid has been received. See details... Figure 11 As shown.
[0113] Case 2: The msgid falls within A / B, and the corresponding position is already set to 1.
[0114] This indicates that the msgid has been received for the second time. In this case, the packet corresponding to that msgid should be discarded immediately. See details below. Figure 12 As shown, Figure 12 The value in the middle indicates that the position that has been set to 1 will no longer change.
[0115] Furthermore, Figure 13 The diagram illustrates the situation when msgid falls within the third interval.
[0116] When the msgid falls within the third interval, the packet corresponding to that msgid can be considered an expired / illegal packet and can be discarded directly.
[0117] It should be noted that, for the sake of brevity, Figure 8 , Figures 10 to 13 In the diagram, the total closed-loop region is not represented as a "ring," but rather as a "rectangular" shape. However, this does not mean the entire region is rectangular; the entire region is still ring-shaped. The diagram can also be shown in... Figure 8 , Figures 10 to 13 There are clues to be found within.
[0118] The above are embodiments of the method for deduplication of dual-channel data transmission provided in this disclosure. Other embodiments of deduplication of dual-channel data transmission provided in this disclosure are described below.
[0119] Figure 14 This is a schematic diagram of a dual-channel data transmission deduplication device provided in an embodiment of the present disclosure. The device includes: an acquisition module 1401, a determination module 1402, and a processing module 1403.
[0120] The acquisition module 1401 is used to acquire the sequence information of data packets;
[0121] The determining module 1402 is used to determine the region range to which the storage address of the sequence information belongs based on the sequence information;
[0122] Processing module 1403 is used to mark and deduplicate the storage address when it is determined that the region to which the storage address belongs is within the first interval. The first interval is composed of a sliding window.
[0123] Alternatively, when it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window; the second and third intervals are sequentially shifted according to the displacement of the sliding window; and the storage addresses corresponding to the sequence information are marked.
[0124] Alternatively, when it is determined that the storage address corresponding to any sequence information is currently located within the third interval, the data packet corresponding to that sequence information is discarded. Here, the first interval, the second interval, and the third interval together constitute the total closed-loop area.
[0125] Optionally, the total closed-loop region includes at least three critical addresses. The region between the first and second critical addresses is the first interval, the region between the second and third critical addresses is the second interval, and the region between the third and first critical addresses is the third interval. The storage addresses contained in the first, second, and third intervals are adjusted sequentially with the displacement.
[0126] Optionally, the processing module 1403 is specifically used to determine the first distance between the storage address and the first critical address, and to determine the first displacement of the sliding window based on the first distance;
[0127] Alternatively, determine the second distance between the storage address and the second critical address, and determine the second movement displacement of the sliding window based on the second distance;
[0128] Move the sliding window according to the first or second displacement so that the storage address falls into the sliding window.
[0129] Optionally, the first interval is divided into a first sub-interval and a second sub-interval through a fourth critical address, and the second interval is divided into a third sub-interval and a fourth sub-interval through a fifth critical address, with the area range of the first sub-interval and the third sub-interval being equal;
[0130] The processing module 1403 is specifically used to, when the storage address falls within the third sub-interval, clear all the markers in the first sub-interval while keeping the storage address contained in the second sub-interval unchanged, and move the first sub-interval to the area range of the third sub-interval so that the storage address falls within the first sub-interval;
[0131] Determine the third distance between the first critical address and the fourth critical address at the current moment;
[0132] Based on the third distance, move the second and third intervals sequentially.
[0133] Optionally, the processing module 1403 is specifically used to determine the third distance between the storage address and the first critical address when the storage address falls in the fourth sub-interval, and determine the third moving displacement of the sliding window based on the third distance; or, determine the fourth distance between the storage address and the second critical address, and determine the fourth moving displacement of the sliding window based on the fourth distance.
[0134] Move the sliding window according to the third or fourth displacement until the storage address falls into the sliding window.
[0135] Optionally, the range of the second interval is greater than or equal to the range of the first sub-interval.
[0136] Optionally, the area of the first sub-interval is the same as the area of the second sub-interval.
[0137] The functions performed by each component in the dual-channel data transmission and weight-reduction device provided in this disclosure have been described in detail in any of the above method embodiments, and therefore will not be repeated here.
[0138] This disclosure provides a dual-channel data transmission deduplication device that acquires the sequence information of data packets and determines the region to which the storage address of the sequence information belongs based on the sequence information. If the region to which the storage address belongs is within a first interval, it indicates that the data packet is a valid data packet and can be marked and deduplicated to complete the marking of the received data packet. Of course, if it is a duplicate data packet, it is discarded after deduplication. The first interval can be understood as the survival interval. The second interval can be understood as the tolerance interval. When the storage address of the sequence information belongs to the second interval, it indicates that the network may have been disconnected, resulting in the loss of some data. Therefore, the error position corresponding to the sequence number of the newly received data packet has exceeded the region currently "defined" by the sliding window. At this time, the sliding window can be moved until the storage address falls within the sliding window, and the storage address of the sequence information is marked to indicate that the data packet has been received. At the same time, according to the displacement of the sliding window, the second interval and the third interval are shifted sequentially. The third interval is the data packet expiration interval. The first interval, the second interval, and the third interval constitute a "closed loop" region. If the storage address corresponding to the sequence information of any data packet does not belong to either the live range or the tolerable range, then the data packet is considered "old" and needs to be discarded. This method avoids hash collisions caused by hash algorithms during data deduplication, and the calculation logic is simple. Because the sequence information of data packets in this disclosure is only marked within the first range, it does not consume a large amount of memory. Furthermore, the deduplication process in this disclosure does not need to consider whether the sequence number information is duplicated; it only determines whether duplicate data packets exist based on the markings in the storage address. Therefore, even if the sequence number information is duplicated, there will be no conflict because the storage addresses are different, and naturally, the game will not be unable to proceed due to the discarding of data packets.
[0139] like Figure 15 As shown, this embodiment of the present disclosure provides a computer device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.
[0140] Memory 113 is used to store computer programs;
[0141] In one embodiment of this disclosure, when the processor 111 executes a program stored in the memory 113, it implements the dual-channel data transmission deduplication method provided in any of the foregoing method embodiments, the method comprising:
[0142] Obtain the sequence information of the data packets;
[0143] Based on the sequence information, determine the region range to which the storage address of the sequence information belongs;
[0144] When the range of the storage address is determined to be within the first interval, the storage address is marked and deduplicated. The first interval is formed by a sliding window.
[0145] Alternatively, when it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window; the second and third intervals are sequentially shifted according to the displacement of the sliding window; and the storage addresses corresponding to the sequence information are marked.
[0146] Alternatively, when it is determined that the storage address corresponding to any sequence information is currently located within the third interval, the data packet corresponding to that sequence information is discarded. Here, the first interval, the second interval, and the third interval together constitute the total closed-loop area.
[0147] Optionally, the total closed-loop region includes at least three critical addresses. The region between the first and second critical addresses is the first interval, the region between the second and third critical addresses is the second interval, and the region between the third and first critical addresses is the third interval. The storage addresses contained in the first, second, and third intervals are adjusted sequentially with the displacement.
[0148] Optionally, when it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window, specifically including:
[0149] Determine the first distance between the storage address and the first critical address, and determine the first displacement of the sliding window based on the first distance;
[0150] Alternatively, determine the second distance between the storage address and the second critical address, and determine the second movement displacement of the sliding window based on the second distance;
[0151] Move the sliding window according to the first or second displacement so that the storage address falls into the sliding window.
[0152] Optionally, the first interval is divided into a first sub-interval and a second sub-interval through a fourth critical address, and the second interval is divided into a third sub-interval and a fourth sub-interval through a fifth critical address, with the area range of the first sub-interval and the third sub-interval being equal;
[0153] Once the region containing the storage address is determined to be within the second interval, the sliding window is moved until the storage address falls within the sliding window, including:
[0154] When the storage address falls within the third sub-interval, while keeping the storage address contained in the second sub-interval unchanged, clear all the markers in the first sub-interval and move the first sub-interval to the area range of the third sub-interval so that the storage address falls within the first sub-interval;
[0155] Determine the third distance between the first critical address and the fourth critical address at the current moment;
[0156] Based on the third distance, move the second and third intervals sequentially.
[0157] Optionally, when it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window, including:
[0158] When the storage address falls within the fourth sub-interval, determine the third distance between the storage address and the first critical address, and determine the third displacement of the sliding window based on the third distance; or, determine the fourth distance between the storage address and the second critical address, and determine the fourth displacement of the sliding window based on the fourth distance.
[0159] Move the sliding window according to the third or fourth displacement until the storage address falls into the sliding window.
[0160] Optionally, the range of the second interval is greater than or equal to the range of the first sub-interval.
[0161] Optionally, the area of the first sub-interval is the same as the area of the second sub-interval.
[0162] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by an edge node, implements the steps of the dual-channel data transmission deduplication method provided in any of the foregoing method embodiments.
[0163] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0164] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A dual-channel data transmission deduplication method, characterized in that, The method includes: Obtain the sequence information of the data packets; Based on the sequence information, determine the region range to which the storage address of the sequence information belongs; When it is determined that the region to which the storage address belongs is within the first interval, the storage address is marked and deduplicated. The first interval is formed by a sliding window. When it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window; the second interval and the third interval are sequentially shifted according to the displacement of the sliding window; the storage address corresponding to the sequence information is marked. When it is determined that the current area range of the storage address corresponding to any sequence information is within the third interval, the data packet corresponding to the sequence information is discarded. The first interval, the second interval, and the third interval together constitute the total closed-loop area range. The total closed-loop region includes at least three critical addresses. The region between the first critical address and the second critical address is the first interval, the region between the second critical address and the third critical address is the second interval, and the region between the third critical address and the first critical address is the third interval. The storage addresses contained in the first interval, the second interval, and the third interval are adjusted sequentially with the displacement.
2. The method according to claim 1, characterized in that, When it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window, specifically including: Determine a first distance between the storage address and the first critical address, and determine a first displacement of the sliding window based on the first distance; Alternatively, determine a second distance between the storage address and the second critical address, and determine a second movement displacement of the sliding window based on the second distance; The sliding window is moved according to the first or the second displacement so that the storage address falls into the sliding window.
3. The method according to claim 1, characterized in that, The first interval is divided into a first sub-interval and a second sub-interval by a fourth critical address, and the second interval is divided into a third sub-interval and a fourth sub-interval by a fifth critical address, and the area range of the first sub-interval and the third sub-interval is equal. When it is determined that the region to which the storage address belongs is within the second interval, moving the sliding window until the storage address falls within the sliding window includes: When the storage address falls within the third sub-interval, while keeping the storage address contained in the second sub-interval unchanged, clear all the markers in the first sub-interval and move the first sub-interval to the area range of the third sub-interval so that the storage address falls within the first sub-interval; Determine the third distance between the first critical address and the fourth critical address at the current moment; According to the third distance, the second interval and the third interval are moved sequentially.
4. The method according to claim 3, characterized in that, When it is determined that the region to which the storage address belongs is within the second interval, moving the sliding window until the storage address falls within the sliding window includes: When the storage address falls within the fourth sub-interval, a third distance between the storage address and the first critical address is determined, and a third displacement of the sliding window is determined based on the third distance; or, a fourth distance between the storage address and the second critical address is determined, and a fourth displacement of the sliding window is determined based on the fourth distance. The sliding window is moved according to the third or fourth displacement until the storage address falls into the sliding window.
5. The method according to claim 3 or 4, characterized in that, The range of the second interval is greater than or equal to the range of the first sub-interval.
6. The method according to claim 3 or 4, characterized in that, The area of the first sub-interval is the same as that of the second sub-interval.
7. A dual-channel data transmission and weight decomposition device, characterized in that, The device includes: The acquisition module is used to acquire the sequence information of data packets; The determining module is used to determine the region range to which the storage address of the sequence information belongs based on the sequence information; The processing module is used to perform marking and deduplication processing on the storage address within the first interval when it is determined that the region to which the storage address belongs is within the first interval, wherein the first interval is composed of a sliding window; When it is determined that the region to which the storage address belongs is within the second interval, the sliding window is moved until the storage address falls within the sliding window; the second interval and the third interval are sequentially shifted according to the displacement of the sliding window; the storage address corresponding to the sequence information is marked. When it is determined that the current area range of the storage address corresponding to any sequence information is within the third interval, the data packet corresponding to the sequence information is discarded. The first interval, the second interval, and the third interval together constitute the total closed-loop area range. The total closed-loop region includes at least three critical addresses. The region between the first critical address and the second critical address is the first interval, the region between the second critical address and the third critical address is the second interval, and the region between the third critical address and the first critical address is the third interval. The storage addresses contained in the first interval, the second interval, and the third interval are adjusted sequentially with the displacement.
8. A computer device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the dual-channel data transmission deduplication method according to any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the edge node, it implements the steps of the dual-channel data transmission deduplication method as described in any one of claims 1-6.