Network data packet deduplication sequencing system

Abstract

The invention discloses a network data packet deduplication sequencing system, which comprises a data packet processing module used for receiving data packets, calculating the number of a falling sub-cache region according to the number of the data packets, obtaining the expected data packet number of a corresponding session table item, comparing the data packet number with the expected data packet number, and sending the sequence packet when the comparison result is a sequence packet, when the comparison result is an out-of-order packet, caching the out-of-order packet to a corresponding sub-cache region, and when the sub-cache region is invalid, dynamically applying to a sub-cache region resource pool management module, and creating an overtime event; the timeout processing module is used for reading the corresponding session table item according to the received timeout event, verifying the timeout event, discharging the data packet meeting the condition in the sub-cache region after the verification is passed, determining the stop position of timeout discharging according to the arrival timestamp of the subsequent data packet in the sub-cache region of the timeout event, and creating a new timeout event; and the sub-cache region resource pool management module is used for dynamically applying and releasing the sub-cache region.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and particularly to a network data packet deduplication and reordering system. Background Art

[0002] Wide Area Network (WAN) optimization is an optimization solution that provides high-performance remote data access through some optimization technologies, thereby improving the performance of applications over a wide area network. In the current best-effort service model provided by the Internet, when data packets are transmitted over a lossy network, problems such as packet loss and network congestion may occur, seriously affecting application performance and user experience. To address this, many enterprises adopt WAN optimization technologies and use means such as redundant transmission and multi-path transmission to reduce the packet loss rate and improve data transmission throughput. Redundant transmission means that the sender copies multiple copies of a data packet and sends it to the receiver. Multi-path transmission means that a data packet can reach the receiver via different paths from the sender. However, redundant transmission and multi-path transmission may cause packet duplication and out-of-order at the receiver, which is not conducive to the processing of applications. Therefore, a data packet deduplication and reordering system is needed to restore the original traffic that is in order and without duplicates.

[0003] Due to its high performance and low latency characteristics, FPGA has become an increasingly popular network traffic processing platform. A Field Programmable Gate Array (FPGA) is a semi-custom circuit developed on the basis of programmable logic devices such as PAL, GAL, and CPLD, and can implement the programmable functions of the device according to the description in register-transfer level (rtl) language. An intelligent network card based on FPGA devices often stores data such as large-scale session tables, buffer areas, and buffered data packets in off-chip DDR or on-chip BRAM outside the FPGA.

[0004] Currently, there are significant defects in the existing methods for implementing a data traffic deduplication and reordering function module on an FPGA intelligent network card:

[0005] Method 1: Use a linked list to insert packets in sequence for sorting and assembly. However, this method requires traversing the buffered data packets sequentially from the head of the linked list, which consumes a large amount of computing resources in scenarios where the deduplication and reordering buffer area increases and the degree of out-of-order increases. Moreover, the discrete linked list node addresses also reduce the memory access efficiency of storage.

[0006] Method 2: Use an array or memory area to store packets in the corresponding offset positions for sorting and assembly. However, the size of the buffer area of this method is fixed, and the buffer area corresponds to each session one by one. In the case of an increasing number of sessions and a deeper buffer area, a large amount of resources need to be reserved.

[0007] In addition, during timeout drainage (buffer holes caused by packet loss), existing methods empty the entire buffer, causing packets that have not timed out to also be drained.

[0008] Chinese Invention Patent Publication No. CN107454276B, titled "A User Terminal Device, Its Data Forwarding Method, and a Communication System", proposes a packet sorting method. This method maintains an expected value of the packet number; when the incoming packet matches the expected value, the packet is sent, and it is checked whether the packets in the buffer queue are of the expected value. If so, the drainage continues; when the incoming packet does not match the expected value, the packet is stored in the buffer queue in order, and a timer is activated; when the timer expires, all the packets in the buffer queue are sent out. After out-of-order packets arrive, this method needs to allocate an entire buffer queue, which occupies a large amount of storage space in the case of an increase in the number of sessions and a high demand for the depth of the sorting buffer. Moreover, when the buffer times out, all the packets in the buffer are emptied, which will cause the packets that have not timed out in the buffer to be drained together, without giving enough waiting time to the possible out-of-order packets that may arrive, reducing the chance of reordering the data stream.

[0009] US Patent No. US7477644B2, titled "Method and system of efficient packet reordering", proposes a packet sorting method. This method uses multiple linked list headers to manage the packets in multiple out-of-order intervals. When a packet arrives, it first checks whether it can be incorporated into any out-of-order interval. If it can be incorporated, it is connected to the linked list and sorted. If it cannot be incorporated, a new linked list header is created to start this sorting interval. Although the linked list of this method does not need to traverse the interior of the linked list, it needs to traverse the head and tail intervals of multiple linked list headers, and merging or sorting may be necessary when needed, resulting in a relatively large computational pressure. Moreover, the maximum number of linked list headers for each session determines the deduplication and sorting ability of this session, which is more suitable for out-of-order patterns with continuous intervals. The out-of-order reordering method is not general and has poor performance in the case of scattered out-of-order.

[0010] When implementing the out-of-order reordering function of network traffic, the foregoing methods have significant loopholes and deficiencies, restricting the further improvement of the performance and sorting ability of the deduplication and sorting module on the FPGA. Summary of the Invention

[0011] The purpose of the present invention is to overcome the defects of the prior art and propose a network packet deduplication and sorting system.

[0012] To achieve the above purpose, the present invention proposes a network packet deduplication and sorting system, which includes:

[0013] The data packet processing module is used to receive data packets, calculate the sub-buffer number into which the packet falls according to the packet number, obtain the expected packet number of the corresponding session table entry, compare the packet number with the expected packet number, and send the packet when the comparison result is a sequential packet. When the comparison result is an out-of-order packet, the packet is cached in the corresponding sub-buffer, and when the sub-buffer is invalid, a dynamic application is made to the sub-buffer resource pool management module, and a timeout event is created;

[0014] The timeout processing module is used to read the corresponding session table entry according to the received timeout event, verify the timeout event. After the verification passes, the data packets that meet the conditions in the sub-buffer are discharged, and according to the arrival timestamps of the subsequent data packets in the sub-buffer of the timeout event, the stop position of the timeout discharge is determined and a new timeout event is created; and

[0015] The sub-buffer resource pool management module is used to dynamically apply for and release sub-buffers.

[0016] Preferably, the calculation formula for the sub-buffer number pn_subbuffer_id into which the packet falls is:

[0017] pn_subbuffer_id = (pkt_pn / subbuffer_pkt_depth) % session_subbuffer_count

[0018] Where pkt_pn is the packet number, subbuffer_pkt_depth is the data packet cache depth of the sub-buffer, session_subbuffer_count is the total number of sub-buffers of the session, / represents integer division, and % represents modulo operation.

[0019] Preferably, the session table entry is used to store the session identifier, the expected packet number, the address array of the sub-buffer, and the number of data packet descriptors already stored in each sub-buffer, the timeout verification bit, and the valid bit.

[0020] Preferably, the comparison of the packet number with the expected packet number, and when the comparison result is a sequential packet, the packet is sent, and when the comparison result is an out-of-order packet, the packet is cached in the corresponding sub-buffer; includes:

[0021] When the packet number is equal to the expected packet number, it is a sequential packet, the packet is sent, and the consecutive valid packets starting from the packet number in the corresponding sub-buffer are output;

[0022] When the packet number is greater than the expected packet number and within the system cache capacity, it is an out-of-order packet. According to the calculated sub-buffer number into which the packet falls, if there is already a packet at the corresponding position in the sub-buffer or the sub-buffer is full, the out-of-order packet is a duplicate packet and is directly discarded; otherwise, it is cached at a specific offset position in the sub-buffer.

[0023] Preferably, the invalidation of the sub-buffer is that the valid bit of the sub-buffer in the session entry is 0.

[0024] Preferably, the timeout event includes: the session entry address, the timeout sub-buffer number, the timeout sub-buffer packet offset, and the timeout check bit.

[0025] Preferably, the verification of the timeout event includes:

[0026] If the timeout check bit of the timeout event is inconsistent with the timeout check bit of the timeout sub-buffer in the session entry, or the valid bit of the timeout sub-buffer in the session entry is 0, then the timeout event verification fails; otherwise, the verification passes.

[0027] Preferably, after the verification passes, the packets that meet the conditions in the sub-buffer are discharged; including:

[0028] Discharge all the packets before the packet number of the timeout event in the sub-buffer, the packet of the timeout event, and the consecutive valid packets after the packet number of the timeout event.

[0029] Preferably, determining the stop position of the timeout discharge according to the arrival timestamps of the subsequent packets in the sub-buffer of the timeout event and creating a new timeout event; including:

[0030] According to the arrival timestamps of the non-timeout packets in the sub-buffer of the timeout event, obtain the remaining time until the set timeout time, and use it as the timeout time of the new timeout event;

[0031] Use the offset of the non-timeout packet in the sub-buffer as the packet offset of the new timeout event;

[0032] Update the expected packet number of the session entry to: the packet number corresponding to the start position of the buffer hole before the non-timeout packet.

[0033] Preferably, the processing process of the sub-buffer resource pool management module includes:

[0034] When the packet processing module or the timeout processing module applies for a sub-buffer, allocate a sub-buffer address from the sub-buffer address pool;

[0035] When the packet processing module or the timeout processing module releases the sub-buffer, empty the sub-buffer and recycle the sub-buffer address to the sub-buffer address pool.

[0036] Compared with the prior art, the advantages of the present invention are:

[0037] 1. The deduplication and sorting buffer in the present invention consists of several sub-buffers. The session table entry stores an array of sub-buffer addresses. The space resources of the sub-buffers are dynamically applied for and released, with high space utilization efficiency, supporting the construction of a large-depth deduplication and sorting buffer for a large number of sessions.

[0038] 2. In the present invention, according to the packet number, the number of session sub-buffers, and the depth of the sub-buffers, the specific offset position of the packet descriptor in a certain sub-buffer can be quickly located, and the sorting performance is relatively high.

[0039] 3. When the packets in the buffer time out and are discharged, the position where the discharge stops is determined according to the arrival timestamp of the buffered packets, and a new timeout event is created. Only the timed-out packets are discharged, and the non-timed-out packets can continue to wait in the buffer for out-of-order rearrangement, increasing the opportunity for rearranging the data stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a composition diagram of the network packet deduplication and sorting system of the present invention;

[0041] Figure 2 It is a system block diagram of an embodiment of the present invention;

[0042] Figure 3 It is a processing flow chart of a packet request;

[0043] Figure 4 It is a processing flow chart of a timeout request;

[0044] Figure 5 It is a data structure of a session table entry;

[0045] Figure 6 It is a data structure of a sub-buffer information node;

[0046] Figure 7 It is a data structure of a sub-buffer;

[0047] Figure 8 It is a data structure of a packet descriptor information node;

[0048] Figure 9 It is a message structure for creating a timeout event, which is sent by the packet processing module or the timeout processing module to the timeout module;

[0049] Figure 10 It is a data structure of a timeout event;

[0050] Figure 11 It is a schematic diagram of the packet descriptors in sub-buffer No. 0 and sub-buffer No. 1 in an embodiment;

[0051] Figure 12 It is a schematic diagram of the packet descriptor and timestamp in sub-buffer No. 0 in an embodiment. Detailed implementation manners

[0052] The object of the present invention is to overcome the defects of the above data packet out-of-order rearrangement method, and an innovative duplicate removal and sorting method and system are proposed.

[0053] The technical solution of the present invention will be described in detail below with reference to the drawings and embodiments. In the drawings, it is agreed that the right side is the low-order bit of the data and the lower side is the low-order bit of the data.

[0054] Embodiment 1

[0055] An embodiment of the present invention proposes a network data packet duplicate removal and sorting system, which includes: a data packet processing module 103, a timeout processing module 104, and a sub-buffer resource pool management module 105. Among them,

[0056] The data packet processing module 103 is responsible for processing incoming data packets.

[0057] The timeout processing module 104 is responsible for processing triggered timeout events.

[0058] The sub-buffer resource management module 105 is responsible for the dynamic application and release of sub-buffers.

[0059] Figure 2 is the system block diagram of the embodiment of the present invention. There are also in the figure:

[0060] The traffic demultiplexing module 101 is responsible for separating the traffic that needs to be duplicate-removed and sorted from the irrelevant traffic.

[0061] The traffic multiplexing module 102 is responsible for merging the irrelevant traffic, sequential packets, and the output traffic of the duplicate removal and sorting buffer.

[0062] The timeout module 106 is responsible for triggering a timeout event after a specific time interval.

[0063] The data packet storage module 107 is responsible for storing data packets and converting them into data packet descriptors; or receiving data packet descriptors and outputting corresponding data packets.

[0064] The data packet traffic containing duplicate packets and out-of-order packets is input through the 201 interface. After being split by the 101 traffic demultiplexing DEMUX (Demultiplexer) module, the traffic that needs to be processed by the duplicate removal and sorting module is output through the 202 interface, and the irrelevant traffic is output through the 203 interface. In one implementation manner, technicians set the specific fields of the data packets that may be duplicate and out-of-order to distinguish this type of traffic from other ordinary traffic, facilitating the differential processing of different traffic.

[0065] After the data packet processing module 103 receives the traffic to be processed 202, it parses the session and number information of the data packet, reads the corresponding session table entry for further processing. In one implementation, the data packet session uniquely identifies a network connection, which can be a communication five-tuple, including the source IP address, source port, destination IP address, destination port, and transport layer protocol; the data packet number identifies the data order of the session transmission content and is the basis for deduplication and sorting, which can be a 32-bit continuously increasing natural number.

[0066] In a specific embodiment, when there are data packet descriptors to be discharged left in the data packet storage management module 107. Since the DDR read / write rate for storing data packets is slow, if a sequential packet arrives at this time, in order to prevent the discharge time of the sequential packet through the 205 interface of the data packet processing module 103 from being earlier than the discharge time of the discharge interface 204 of the data packet corresponding to the left data packet descriptor, resulting in out-of-order, this sequential packet can be sent to the data packet storage management module 107 through a special channel and discharged through the interface 204 to avoid out-of-order.

[0067] The data packet processing module 103 is the core module for processing incoming data packets. Figure 3 is the processing flow chart for data packet requests. The timeout processing module 104 is the core module for processing triggered timeout events. Figure 4 is the processing flow chart for timeout requests. The data packet processing module 103 and the timeout processing module 104 jointly participate in the update and maintenance of the session table entries.

[0068] In a specific implementation, the data structure of the session table entry is as Figure 5 shown. The session table entry includes the session identifier session_info, the expected next data packet number EPN, and several sub-buffer information nodes pktdesc_subbuffer_node. Among them, the data structure of the sub-buffer information node is as Figure 6 shown. The sub-buffer information node pktdesc_subbuffer_node contains the relative address addr of the sub-buffer, the number of stored data packet descriptors pktdesc_count, the timeout checksum timeout_cksum, and the valid bit valid. Figure 5 In, the deduplication and sorting buffer of the table entry consists of 16 sub-buffers, and the sub-buffers are numbered 0, 1, 2,..., 15 in sequence. Each sub-buffer has a 32-bit sub-buffer information node in the table entry.

[0069] In a specific embodiment, a field for the head sub-buffer number can be added to the session table entry to facilitate logical programming processing. However, since the head sub-buffer number and the expected data packet number EPN of the table entry are in a one-to-one correspondence relationship, the field for the head sub-buffer number can also be omitted.

[0070] In a specific embodiment, a session hit timestamp field can be added to the session entry table to facilitate the implementation of session keep-alive and timeout replacement functions.

[0071] In a specific implementation, the packet buffer depth of a single sub-buffer is 8192, as Figure 7 is a schematic diagram of the data structure of the sub-buffer. The packet descriptor sub-buffer pktdesc_subbuffer consists of 8192 packet descriptor cache information nodes pktdesc. As Figure 8 is a schematic diagram of the data structure of the packet descriptor information node. The packet descriptor information node pktdesc consists of a 24-bit packet descriptor real_pktdesc, a 7-bit packet timestamp timestamp, and a 1-bit valid bit valid. There is a one-to-one correspondence between a packet and its packet descriptor, which is the storage form of the packet in the buffer. The packet storage management module 107 realizes the function of outputting a packet descriptor for an input packet and outputting a packet for an input packet descriptor. The timestamp of the packet is the arrival time of the packet, in milliseconds, and plays a role in subsequent timeout event updates. The valid bit of the packet descriptor represents whether the packet descriptor information node stores a packet descriptor, 1 means stored, and 0 means empty.

[0072] After the packet processing module 103 performs logical processing based on the incoming packet information, the packet can be dropped, sent through the 205 interface, or cached through the 206 interface. It should be noted that for a packet cached through the 206 interface, the packet storage module 107 will feedback the packet descriptor corresponding to the packet to the packet processing module 103.

[0073] In a specific implementation, taking the session packet stream as an example of sequential traffic. When the first packet (packet number is 1) of a certain session arrives at the deduplication and reordering module, the packet processing module 103 reads the session entry table and finds that the session in the table is empty. So it overwrites the session in the table with the packet session, initializes the table EPN to the packet PN + 1, that is, 1 + 1 = 2, and sends the packet. Subsequently, the packet numbered 2 arrives. The session entry table management module reads the corresponding table entry and finds that the table EPN = the packet number = 2. This packet is a sequential packet. Also, because the sub-buffer corresponding to number 2 (that is, the sub-buffer numbered 0) is empty and there is no packet to be discharged in the buffer, it updates the table EPN to the packet PN + 1, that is, 2 + 1 = 3, and sends the packet. And so on. The sequential traffic (the packet numbers increase sequentially) will cause the EPN of the session entry table to increase sequentially, but this session does not occupy additional sub-buffer resources.

[0074] In a specific embodiment, taking the duplicate packet as an example. The sequence number of a certain session data packet is 200, and the out-of-order packet arrives at the deduplication and sorting module first. After the packet is cached, a packet descriptor is obtained, and the corresponding packet descriptor information node in the sub-buffer fills in the packet descriptor, the packet timestamp, and the valid position 1. Subsequently, the duplicate packet of this packet, that is, the second packet with the packet number 200, arrives at the deduplication and sorting module. By reading the specific position of this packet number in the sub-buffer, it is found that the packet descriptor information node here has been occupied (the valid bit is 1), so it is determined that this packet is a duplicate packet, and the packet loss operation is performed.

[0075] In a specific embodiment, taking the first out-of-order packet in the sub-buffer as an example. The arrival sequence numbers of a certain session data packets are 1, 3, 5, 7, 8200. When the packet with the packet number 7 is processed by the deduplication module, the EPN of this session entry is 2, and only the sub-buffer numbered 0 is valid. There are 3 packet descriptors stored in this sub-buffer. When the packet with the packet number 8200 arrives at the deduplication and sorting module, the packet processing module 103 reads the entry, calculates the corresponding sub-buffer of the packet number 8200, that is, the sub-buffer numbered 1 is invalid, so it applies to the sub-buffer resource pool management module 105 for a new sub-buffer, creates a timeout event for this packet, caches the packet through the 206 interface, and after obtaining the packet descriptor returned by the packet storage module 107, stores the descriptor in the corresponding offset position of the packet number 8200 in the sub-buffer numbered 1 (the 9th position in the sub-buffer numbered 1), and updates the information node of the sub-buffer numbered 1 in the session entry (the relative address of the sub-buffer is updated, the number of packet descriptors is written as 1, the check bit is updated, and the valid position is set to 1).

[0076] Particularly, if the arriving out-of-order packet is not the first out-of-order packet in the sub-buffer, the behaviors of applying for a new sub-buffer and creating a timeout event can be skipped in the above process. In order to prevent the system from being pressured by the establishment of a large number of timeout events, in the design of the present invention, the establishment of timeout events is based on the establishment of sub-buffers, and the timeout events are updated by appropriate methods after timeout.

[0077] It should be noted that the creation of timeout events can also be based on the establishment of the first sub-buffer in the entire buffer area, that is, there is at most one timeout event for a session, so as to further save the performance pressure of the timeout module. However, when the buffer depth is relatively deep, the pressure of scanning the entire buffer area to update timeout events is relatively large, and it is easy to cause uncontrollable timeout errors across intervals, so it is not used in the specific embodiment. It is more reasonable to use sub-buffers for timeout control and management.

[0078] Create a timeout event corresponding to a data packet. The information required includes the timeout event and the trigger time of the timeout event. The timeout event includes the session table entry address, the timeout sub-buffer number, the data packet offset in the timeout sub-buffer, and the timeout checksum. In a specific embodiment, Figure 9 For the message structure of establishing a timeout event sent by the data packet processing module 103 or the timeout processing module 104 to the timeout module 106, the message for creating the timeout event includes the timeout event timeout_event and the time timeout_ms from the trigger of the timeout event. In a specific embodiment, Figure 10 For the data structure of the timeout event, the timeout event includes the session address session_addr, the timeout sub-buffer number subbuffer_id, the data packet offset in the timeout sub-buffer fst_pkt_offset, and the timeout checksum timeout_cksum

[0079] In a specific embodiment, to reduce the hardware computation amount, the position of a data packet number in a certain sub-buffer can be obtained by taking the modulo of the number and the sub-buffer depth. Taking the example where the EPN is 2 and the head sub-buffer number is 0, the data packet numbers corresponding to the data packet descriptor information nodes in the 0th sub-buffer are 0, 1, 2,..., 8191 in sequence, and the data packet numbers corresponding to the data packet descriptor information nodes in the 1st sub-buffer are 8192, 8193,..., 16383 in sequence.

[0080] In a specific embodiment, take the example where an in-order packet arrives when the deduplication and sorting buffer stores out-of-order packets. The arrival number sequence of a certain session data packet is 8188, 8190, 8191, 8192,..., 8200, 8205, 8206, 8189. After the data packet numbered 8206 is processed, the EPN of the session table entry is 8189. There are two data packet descriptors in the 0th sub-buffer and 11 data packet descriptors in the 1st sub-buffer. The descriptor statuses of sub-buffers 0 and 1 are as Figure 11 shown. When the data packet numbered 8189 enters the deduplication and sorting module for processing, since the data packet number 8189 = the EPN of the table entry and it is an in-order packet, the data packet 8189 is sent, and then the data packets corresponding to the valid data packet descriptors are continuously sent backward starting from 8189 + 1 = 8190. Therefore, the data packet descriptors of 8190, 8191, 8192,..., 8200 are sent by the data packet processing module 103 to the data packet storage module 107, and the corresponding data packets are sent out through the 204 interface and merged into the 207 output interface. Since the data packet processing module 103 scans that the data packet descriptor information node numbered 8201 is empty, the EPN of the session table entry is updated to 8201.

[0081] It should be noted that the discharging operation of consecutive valid packet descriptors triggered by the sequence packet can span sub-buffer areas until the empty packet descriptors are reached.

[0082] It should be noted that when the number of confirmed valid packet descriptors in the sub-buffer area is 0, the sub-buffer area will be released and uniformly cleared. In this way, there is no need to clear the scattered packet descriptors during discharging, thereby reducing the frequency of clearing operations and improving the write efficiency of storage. The release operation of the sub-buffer area is that the packet processing module 103 or the timeout processing module 104 sends the relative address of the sub-buffer area to the sub-buffer area resource pool management module 105, and the sub-buffer area resource pool management module 105 realizes the clearing and recycling of the sub-buffer area.

[0083] The timeout processing module 104 can receive the packet timeout processing request sent by the timeout module 106, read the relevant session table entry, and after passing the timeout verification, perform the discharging operation of the timeout packet and update or create a new timeout event as appropriate.

[0084] Among them, the timeout event verification is to prevent the event from being invalid when the timeout event is triggered. For example, the sub-buffer area of the timeout event has been emptied and released, or the sub-buffer area of the timeout event has been circularly overwritten and utilized, etc. In a specific embodiment, the check bit of the sub-buffer area can be composed of the offset of the sub-buffer area timeout packet (or the first packet of the sub-buffer area) and the timestamp taking part of the bits to increase the uniqueness of the timeout event check bit. When the sub-buffer area of the timeout event is valid (the valid bit of the sub-buffer area information node in the session table entry is 1) and the timeout check bit offset is consistent (the timeout_cksum of the timeout event is equal to the timeout_cksum of the timeout sub-buffer area information node), the timeout event verification passes; otherwise, it fails.

[0085] In a specific embodiment, taking the packet timeout event processing as an example. After a period of out-of-order traffic, the EPN of a certain session table entry is 34. A number of packet descriptors are cached in sub-buffer area 0. The first out-of-order packet number cached in sub-buffer area 0 is 100, the timestamp when this out-of-order packet arrives is 10 ms, and a corresponding packet timeout event is created when this out-of-order packet arrives. The sub-buffer area number of the timeout event is 0, and the packet offset of the timeout event is 100. Assume that the packet timeout threshold set by the user is 50 ms. The packet descriptor caching situation of sub-buffer area 0 is as Figure 12As shown, the data packet numbers and arrival timestamps are marked. After a period of time, such as 50 ms later, the timeout event created by the out-of-order packet numbered 100 is triggered and sent to the timeout processing module 104 by the timeout module 106 for processing. At this time, the global timestamp is obviously 10 ms + 50 ms = 60 ms. The timeout processing module 104 verifies that the timeout event passes. First, it discharges all the data packets corresponding to the data packet descriptors numbered from 35 to 100 in the 0th sub-buffer area between the EPN and the timeout event data packet offset. Subsequently, it continues to discharge the consecutive valid data packet descriptors, corresponding to the numbers 101 and 102 in the 0th sub-buffer area, until an empty data packet descriptor is encountered. Since there are still data packets in this timeout sub-buffer area, it continues to scan backward and finds that the number of the timeout data packet is 105. It discharges the data packet numbered 105, and then continues to discharge the consecutive valid data packet descriptors, corresponding to the number 106, until an empty data packet descriptor is encountered. Since there are still data packets in this timeout sub-buffer area, it continues to scan backward until it finds an un-timed-out data packet, corresponding to the number 108. Therefore, a new timeout event for this sub-buffer area is created. The sub-buffer area number of the new timeout event is 0, and the timeout sub-buffer area data packet offset is 105. Based on the timestamp of the data packet numbered 108 being 30 ms, the timeout trigger interval of the new timeout event is the data packet timeout threshold - (global timestamp - data packet timestamp), that is, 50 ms - (60 ms - 30 ms) = 20 ms. That is to say, 20 ms later, the data packet numbered 108 should trigger a new timeout event.

[0086] Therefore, when processing timeout events, the arrival timestamps of the data packets in the data packet descriptor information node play an important role in determining the stop position of the timeout discharge and the process of updating and creating timeout events.

[0087] It should be noted that when scanning after the timeout data packet and encountering another timeout data packet, similar logical operations are still performed. If the timeout sub-buffer area is confirmed to be emptied, the sub-buffer area is released and no new timeout event is created for it.

[0088] It should be noted that strictly speaking, when updating the timeout event of the sub-buffer area, the data packet that arrives at this sub-buffer area first and is not timed out after the old timeout data packet should be selected, that is, the data packet closest to the set timeout time. Therefore, traversing the buffer area to find the data packet closest to the timeout time is also a possible implementation solution. However, to avoid the performance consumption of traversal, here the first un-timed-out data packet after the old timeout data packet is used for timeout update, which is an optimization in design implementation.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the embodiments, those of ordinary skill in the art should understand that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A network data packet deduplication and sorting system, characterized in that The system includes: A data packet processing module, configured to receive a data packet, calculate the sub-buffer number it falls into according to the data packet number, obtain the expected data packet number of the corresponding session table entry, compare the data packet number with the expected data packet number. When the comparison result indicates sequential packets, send them; when the comparison result indicates out-of-order packets, cache them to the corresponding sub-buffer, and dynamically apply to the sub-buffer resource pool management module when the sub-buffer is invalid, and create a timeout event; A timeout processing module, configured to read the corresponding session table entry according to the received timeout event, verify the timeout event. After the verification passes, discharge the data packets that meet the conditions in the sub-buffer, determine the stop position of the timeout discharge according to the arrival timestamps of the subsequent data packets in the sub-buffer of the timeout event, and create a new timeout event; and, A sub-buffer resource pool management module, configured to dynamically apply for and release sub-buffers.

2. The network data packet deduplication and sorting system according to claim 1, wherein The calculation formula for the sub-buffer number pn_subbuffer_id it falls into is: pn_subbuffer_id = (pkt_pn / subbuffer_pkt_depth) % session_subbuffer_count where, pkt_pn is the data packet number, subbuffer_pkt_depth is the data packet cache depth of the sub-buffer, session_subbuffer_count is the total number of sub-buffers of the session, / represents integer division, and % represents modulo operation.

3. The network data packet deduplication and sorting system according to claim 1, wherein The session table entry is used to store the session identifier, the expected data packet number, the address array of the sub-buffer, and the number of data packet descriptors already stored in each sub-buffer, the timeout verification bit, and the valid bit.

4. The network data packet deduplication and sorting system according to claim 3, wherein The comparison of the data packet number with the expected data packet number, when the comparison result indicates sequential packets, send them; when the comparison result indicates out-of-order packets, cache them to the corresponding sub-buffer; includes: When the data packet number is equal to the expected data packet number, it is a sequential packet, send the data packet, and output the consecutive valid data packets starting from the data packet number in the corresponding sub-buffer; When the data packet number is greater than the expected data packet number and within the system cache capacity, it is an out-of-order packet. According to the calculated sub-buffer number it falls into, if there is already a packet at the corresponding position in the sub-buffer or the sub-buffer is full, the out-of-order packet is a duplicate packet and is directly discarded; otherwise, cache it to a specific offset position in the sub-buffer.

5. The network data packet deduplication and sorting system according to claim 4, wherein The invalidation of the sub-buffer means: the valid bit of the sub-buffer in the session table entry is 0.

6. The network data packet deduplication and sorting system according to claim 1, wherein The timeout event includes: the session table entry address, the timeout sub-buffer number, the data packet offset in the timeout sub-buffer, and the timeout verification bit.

7. The network data packet deduplication and sorting system according to claim 6, wherein The verification of the timeout event includes: If the timeout verification bit of the timeout event is inconsistent with the timeout verification bit of the timeout sub-buffer in the session table entry, or the valid bit of the timeout sub-buffer in the session table entry is 0, then the timeout event verification fails; otherwise, the verification passes.

8. The network data packet deduplication and sorting system according to claim 6, wherein After the verification passes, discharging the data packets that meet the conditions in the sub-buffer; includes: Discharging all the data packets before the data packet number of the timeout event in the sub-buffer, the data packet of the timeout event, and the consecutive valid data packets after the data packet number of the timeout event.

9. The network data packet deduplication and sorting system according to claim 1, wherein Determine the stop position of timeout evacuation based on the arrival timestamps of subsequent data packets in the sub-buffer of the timeout event, and create a new timeout event, including: Obtain the remaining time until the set timeout time based on the arrival timestamps of the non-timeout data packets in the sub-buffer of the timeout event, and use it as the timeout time of the new timeout event; Use the offset of the non-timeout data packet in the sub-buffer as the data packet offset of the new timeout event; Update the expected data packet number in the session entry to the data packet number corresponding to the starting position of the buffer hole before the non-timeout data packet.

10. The network data packet deduplication and sorting system according to claim 1, wherein The processing process of the sub-buffer resource pool management module includes: When the data packet processing module or the timeout processing module applies for a sub-buffer, allocate a sub-buffer address from the sub-buffer address pool; When the data packet processing module or the timeout processing module releases the sub-buffer, clear the sub-buffer and recycle the sub-buffer address to the sub-buffer address pool.

Images