A TCP flow reassembly method based on behavior learning

Abstract

The application discloses a TCP flow recombination method based on behavior learning and belongs to the technical field of network flow processing, and comprises the following steps: performing session and service initialization on received TCP messages, determining the message type according to the comparison result of the message sequence number and the session expected sequence number and performing cache management; carrying out session-level disorder behavior statistics, calculating session disorder scores, and aggregating multiple session scores to obtain a service-level disorder mode; making a recombination control decision in combination with the service-level disorder mode and the disorder cache queue cache level, performing recombination output and release on the cache queue according to the decision; and finally outputting the recombined TCP data flow to an upper-layer application; the application guarantees the sustainable output of data flow, reduces the disorderly growth of cache resources, avoids long-term blocking of sessions, can distinguish and depict the disorder characteristics of different sessions, makes the processing of similar flows more consistent, and adapts to engineering deployment and long-term operation requirements.

Description

Technical Field

[0001] This invention belongs to the field of network traffic processing technology, specifically relating to a TCP stream reassembly method based on behavior learning. Background Technology

[0002] In network traffic processing scenarios, collected TCP packets need to be reassembled to restore continuous data content. Traffic processing devices are mostly deployed in the middle of network links and do not directly participate in TCP connections. In real network environments, due to factors such as link congestion, route switching, and TCP retransmission, TCP packets received by intermediate devices often suffer from out-of-order, duplicate, or lost packets. This problem is more pronounced in high-throughput or complex network environments.

[0003] Existing TCP stream reassembly technologies are mostly based on sequence number maintenance and interval management, caching and sorting out-of-order packets. However, when packets are missing for extended periods or highly out of order, sessions are prone to waiting for missing packets for long periods, causing the out-of-order buffer to grow continuously, leading to increased processing latency and decreased system throughput. Furthermore, existing technologies lack the ability to continuously statistically analyze and pattern the out-of-order behavior of TCP sessions, and there is no mechanism to proactively control the reassembly process based on behavioral information. This makes it difficult to simultaneously ensure continuous data output, resource control, and long-term stable system operation in complex network environments. Summary of the Invention

[0004] To address the problems mentioned in the background art, this invention provides a TCP stream reassembly method based on behavior learning, which solves the problems of excessive resource consumption, continuous cache growth, high processing latency, and long-term session blocking in traditional TCP stream reassembly scenarios with high out-of-order delivery.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A TCP stream reassembly method based on behavior learning includes the following steps: S1: Session, service initialization, and packet type determination; After receiving a TCP packet, query or create a TCP session based on the packet's source IP address, destination IP address, source port, and destination port; initialize the session's bidirectional sequential buffer queue, out-of-order buffer queue, and expected sequence number; query or initialize the TCP service based on the packet's destination IP address and destination port; establish an association between the TCP session and the TCP service; then, based on the comparison result between the TCP packet's own sequence number and the expected sequence number of the corresponding transmission direction of the session, determine whether the packet is sequential, out-of-order, or a duplicate / processed packet, and perform the corresponding buffer management operation; S2: Session and Service Behavior Analysis; For packets that have completed type determination, first perform TCP session-level out-of-order behavior data statistics, initialize a statistical window with a fixed number of packets, and accumulate statistical data such as the total number of packets, the number of out-of-order packets, and the total number of out-of-order offsets within the window; when the statistical window closes or the session ends normally, calculate the index scores of the out-of-order occurrence ratio, average out-of-order distance, and average number of out-of-order padding packets, and obtain the session out-of-order score through weighted summation; then aggregate multiple session out-of-order scores, and when the number of service-related sessions reaches a threshold, calculate the service-level out-of-order score, and classify and update the service-level out-of-order modes of low, medium, and severe out-of-order; S3: Reassembly control and cache handling; If the TCP service to which the current packet belongs has generated a service-level out-of-order mode, determine the cache level based on the cache quantity of the out-of-order cache queue in the corresponding transmission direction of the packet, and make a reassembly control decision based on the cache level and the service-level out-of-order mode; Based on the decision result, perform reassembly output and cache release operations on the local and remote cache queues, prioritize the processing of the remote out-of-order cache queue, and then process the local sequential and out-of-order cache queues according to the rules; if the out-of-order cache queue exceeds the maximum limit, it is directly output and released; S4: Data Output; Outputs the reassembled TCP data stream to the upper-layer application to achieve continuous data output.

[0006] Compared with the prior art, the beneficial effects of the present invention are: 1. In high-concurrency, continuously online network processing scenarios, this invention can maintain the continuity of TCP data stream output, reduce the disorderly growth of cache resources, effectively reduce the risk of excessive cache occupation and processing delay accumulation during the reassembly process, and improve the long-term stability and engineering applicability of the system.

[0007] 2. When TCP packets are out of order or partially missing, the reassembly process can still continue, avoiding long-term session blocking due to waiting for individual packets and ensuring the continuous output capability of the data stream.

[0008] 3. It can distinguish and characterize the out-of-order and missing characteristics of different TCP sessions, avoid the amplified impact of a small number of abnormal sessions on system resources and processing capabilities, and enable the system to maintain stable operation when facing sessions with different degrees of out-of-order conditions.

[0009] 4. By aggregating the overall out-of-order characteristics of multiple sessions at the service level, the system has stronger consistency and predictability when processing similar business traffic, which is conducive to the stability of resource usage and improves the controllability of the reorganization process. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the process of this application. Detailed Implementation

[0011] To facilitate understanding of the technical content of this invention by those skilled in the art, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of the invention.

[0012] A behavior-learning-based TCP stream reassembly method, such as Figure 1 As shown, it includes the following steps: S1: Session, service initialization, and message classification, specifically including: S10: Session and service management; After receiving a TCP packet, the system used to run the method of this application obtains existing TCP session information based on the packet's four-tuple (source IP address, destination IP address, source port, and destination port). If the session does not exist, it creates the corresponding TCP session information. During the creation process, it initializes source-to-destination direction 1 information and destination-to-source direction 2 information based on the first TCP packet's four-tuple. Both direction 1 and direction 2 information include sequential buffer queues and out-of-order buffer queues, as well as the expected sequence number. The system obtains existing TCP services based on the destination IP and destination port of the TCP packet. If the service does not exist, it initializes the corresponding TCP service, including TCP service behavior identification aggregation state information and behavior identification results, and associates the TCP session with the TCP service.

[0013] S11: Message order determination and cache management; Based on the TCP session information obtained in S10, the system retrieves the specific TCP session direction information according to the TCP packet four-tuple. Then, it determines the packet sequence number for that direction: if the sequence number equals the expected sequence number, the packet is added to the sequential buffer queue for that direction and marked as an ordered packet; if the sequence number is greater than the expected sequence number, the packet is added to the out-of-order buffer queue for that direction and marked as an out-of-order packet; if the sequence number is less than the expected sequence number, it is determined to be a duplicate or already processed packet, discarded, and no further steps are executed.

[0014] S2: Session and service behavior analysis, specifically including: S20: Session-level behavior statistics; First, if the currently processed TCP packet is marked as an out-of-order packet in S11, and this is the first time an out-of-order packet has appeared in this TCP session, then a fixed statistical window of 50 packets is initialized. Second, if the statistical window exists and the total number of packets within the window does not exceed 50, the following operations are performed: 1) Accumulate the total number of messages within the window; 2) If it is an out-of-order packet, accumulate the number of out-of-order packets within the window, calculate the difference between the TCP packet sequence number and the expected sequence number in the corresponding direction, and add it to the total number of out-of-order offsets within the window. 3) Based on the current TCP payload byte length of the message, accumulate the payload byte length within the window; 4) If the TCP packet currently being processed is marked as an ordered packet in S11, add the number of packets in the current out-of-order buffer queue to the total number of packets waiting to be filled in the window, and add the number of filling attempts in the window.

[0015] S21: Session-level behavior result output; If the count reaches 50, the statistics window for this TCP session is set to closed. When the statistics window is closed or the session ends normally, the system calculates and obtains corresponding index scores for the out-of-order occurrence rate, average out-of-order distance, and average number of out-of-order padding packets based on the S20 statistics median value, as follows: 1) Regarding the percentage of out-of-order packets, the system calculates the ratio of the number of out-of-order packets within a window to the total number of packets within the window. A score of 2 points is given when the ratio is no greater than 0.2; a score of 3 points is given when the ratio is greater than 0.2 but no greater than 0.5; and a score of 5 points is given when the ratio is greater than 0.5. 2) Regarding the average out-of-order distance, the system calculates the average out-of-order distance by the ratio of the total number of out-of-order offsets within the window to the number of out-of-order packets within the window, and calculates the average packet load length by the ratio of the payload byte length within the window to the total number of packets within the window. When the average out-of-order distance is less than or equal to the average packet load length, the score is 2 points; when the average out-of-order distance is greater than the average packet load length but less than or equal to twice the average packet load length, the score is 3 points; when the average out-of-order distance is greater than twice the average packet load length, the score is 5 points. 3) Regarding the average number of packets continuously interrupted by out-of-order completion, the system calculates the ratio of the total number of completion waits within the window to the number of completion attempts within the window to obtain the average number of packets continuously interrupted by out-of-order completion. When its value is less than or equal to 1, the corresponding score is 2 points; when its value is greater than 1 and less than or equal to 2, the corresponding score is 3 points; when its value is greater than 2, the corresponding score is 5 points.

[0016] After obtaining the behavioral scores for each indicator, the system calculates a weighted sum of the scores to obtain a session disorder score, with a total score of 5 points. The calculation formula is: Session disorder score = 0.5 * Disorder occurrence percentage score + 0.3 * Average disorder distance score + 0.2 * Average number of disorder completion packets score.

[0017] S22: Summary of Service-Level Out-of-Order Modes; The service-level out-of-order mode is derived by analyzing and summarizing the results of more than a certain number of associated session-level behaviors, and supports continuous updates. Once the conditions in S21 are met and the session out-of-order score is calculated, the following specific actions are taken: 1) Add the out-of-order session score to the total session score of the TCP service, and also add it to the total session statistics of the TCP service; 2) When the number of TCP sessions associated with the service is greater than 50, calculate the service-level out-of-order score = total TCP service session score / total number of TCP service sessions; when the score is less than or equal to 2, set the service-level out-of-order mode to low out-of-order; when the score is greater than 2 and less than or equal to 3, set the service-level out-of-order mode to medium out-of-order; when the score is greater than or equal to 4, set the service-level out-of-order mode to severe out-of-order.

[0018] S3: Reorganization control and cache processing, specifically including: S30: Reorganization Control and Decision-Making; If the TCP service to which the current packet belongs has a service-level out-of-order mode, and it is not triggered directly by the current packet, the reassembly control decision can be determined jointly based on the number of out-of-order buffer queues in the current packet direction and the service-level out-of-order mode.

[0019] First, determine the cache level of the out-of-order cache queue: 1) If the number of out-of-order buffer queues in the direction of the message is less than or equal to 50, the buffer level is low; 2) If the number of out-of-order buffer queues in the direction corresponding to the message is greater than 50 and less than or equal to 100, then the buffer level is medium. 3) If the number of out-of-order buffer queues in the direction of the message is greater than 100, the buffer level is high.

[0020] Then, based on the out-of-order cache queue cache level and service-level out-of-order mode in the current direction, the reorganization control decision is determined as follows: 1) If the cache level is low, the reorganization control decision will be to retain the out-of-order cache.

[0021] 2) Cache level is medium. For service-level out-of-order mode, which is low or medium out-of-order, the reorganization control decision is to retain the out-of-order cache; for service-level out-of-order mode, which is severe out-of-order, the reorganization control decision is to release the out-of-order cache and output the result.

[0022] 3) Cache level is high. In low out-of-order service mode, the reorganization control decision is to retain the out-of-order cache; in medium or severe out-of-order service mode, the reorganization control decision is to release the out-of-order cache and output the cache.

[0023] S31: Reassemble output and release buffer; This primarily handles the reassembly and release of sequential and out-of-order buffer queues in both the local and remote directions of a TCP session. First, it processes the out-of-order buffer queue from the remote end. It checks if buffered data exists; if so, it directly reassembles the current queue and releases the buffer. Then, it processes the sequential and out-of-order buffer queues from the local end, as follows: 1) Check if cached data exists in the sequential cache queue; if so, reassemble the output and release the cache. 2) If there is a reorganization control decision result in step S30, then process the out-of-order cache queue according to the result.

[0024] 3) Check if the out-of-order cache queue exceeds the maximum limit of 200. If it does, reassemble the output and release the cache.

[0025] S4: Data output; outputs the TCP data stream reassembled from S3 to the upper-layer application.

[0026] Explanation of keywords and terms used in this embodiment: TCP Session: In this system, a TCP session is uniquely identified by its source IP, destination IP, source port, and destination port. It is used to associate received TCP packets and maintain session-related state information, including relevant buffer queues and behavioral analysis metrics.

[0027] TCP Service: In this system, the TCP service is uniquely identified by the destination IP and destination port. It is used to classify multiple TCP sessions and maintain service-level state information to support session behavior aggregation, pattern analysis, and reassembly control.

[0028] Expected Sequence Number: The expected sequence number refers to the sequence number of the next consecutive TCP data byte expected to be received during the current reassembly process in a specific transmission direction of a TCP session. This sequence number is used to characterize the logical sequential progression of the data stream and is an important reference for determining whether subsequent arriving packets are in order, out of order, or have already been processed.

[0029] Out-of-order occurrence percentage: The out-of-order occurrence percentage is a statistical indicator used to describe the frequency of out-of-order packets in a TCP session within a certain statistical range. This indicator reflects the relative proportion of out-of-order TCP packets to all received packets within the statistical window, and is used to characterize whether out-of-order phenomena frequently occur in the current phase of the session. The higher the out-of-order occurrence percentage, the more significant the deviation of the arrival order from the expected order in the TCP session within the statistical period.

[0030] Average Out-of-Order Distance (AOD): AOD is a statistical metric used to describe the degree to which out-of-order packets deviate from their normal arrival order. This metric reflects the overall average level of the deviation of the sequence number of each out-of-order packet from its expected sequence number within a statistical window. AOD can distinguish between "slightly early arrival" and "significantly large" out-of-order situations, thus characterizing the severity of out-of-order behavior, rather than simply indicating whether out-of-order delivery has occurred.

[0031] Average Out-of-Order Completion Duration: The average out-of-order completion duration is a statistical metric used to describe the scale of packet accumulation in the out-of-order buffer under out-of-order conditions. This metric reflects the average number of packets accumulated or retained in the out-of-order buffer queue after an out-of-order event, until the missing sequence numbers are completed and the data flow resumes continuous operation. This metric characterizes the sustained impact of out-of-order conditions on buffer resource usage and reflects the engineering impact of out-of-order behavior during actual reassembly.

[0032] Those skilled in the art should understand that the above embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Any reasonable modifications, equivalent substitutions, or adaptive improvements made based on the technical concepts disclosed in this invention without departing from the essential spirit of the invention should be considered to fall within the scope of protection defined by the claims of this invention.

Claims

1. A TCP stream reassembly method based on behavior learning, characterized in that, Includes the following steps: S1: Perform session and service initialization operations on the received TCP packets, and then determine the packet type and manage the corresponding buffer based on the TCP packet's sequence number. Specifically: Based on the four-tuple of the TCP packet, a TCP session and TCP service are created or associated. Then, based on the comparison result of the sequence number of the TCP packet and the expected sequence number of the corresponding direction of the TCP session, the packet type is determined, and corresponding cache management is performed based on the determination result. S1 specifically includes: S10: After receiving a TCP packet, query the existing TCP session information based on the source IP address, destination IP address, source port, and destination port of the TCP packet. If there is no corresponding TCP session information, create the TCP session information. When creating the session, initialize the transmission direction from the source IP address and source port to the destination IP address and destination port, as well as the transmission direction from the destination IP address and destination port to the source IP address and source port, based on the source IP address, destination port, and destination port of the first TCP packet. The information for both transmission directions includes the sequential buffer queue, the out-of-order buffer queue, and the expected sequence number. Based on the destination IP address and destination port of the TCP packet, the existing TCP service is queried. If no corresponding TCP service is found, the TCP service is initialized. The TCP service contains TCP service behavior identification aggregated state information and behavior identification results, and establishes an association between the TCP session and the TCP service. S11: Based on the TCP session obtained in S10, determine the specific TCP session transmission direction information according to the source IP address, destination IP address, source port and destination port of the TCP packet, extract the sequence number of the TCP packet itself, compare it with the expected sequence number of the corresponding transmission direction of the TCP session, complete the packet type determination and perform the corresponding buffer management operation. The rules for determining message sequence numbers and managing caches in S11 are as follows: If the sequence number of a TCP packet is equal to the current expected sequence number of the corresponding TCP session transmission direction, the packet is added to the sequential buffer queue of the corresponding transmission direction and marked as a sequential packet. If the sequence number of a TCP packet is greater than the current expected sequence number of the corresponding TCP session transmission direction, the packet is added to the out-of-order buffer queue of the corresponding transmission direction and marked as an out-of-order packet. If the sequence number of a TCP packet is less than the current expected sequence number in the corresponding TCP session transmission direction, the packet is determined to be a duplicate or already processed packet, and the packet is discarded without further processing. S2: Perform session and service behavior analysis on TCP packets that have completed type determination. First, perform TCP session-level out-of-order behavior data statistics, then calculate the session out-of-order score, and finally summarize to obtain the TCP service-level out-of-order mode. The specific statistics of TCP session-level out-of-order behavior data in S2 are as follows: S20: If the TCP packet being processed is marked as out-of-order in S11, and this is the first time an out-of-order packet has appeared in this TCP session, then initialize a fixed statistical window containing K packets; when the statistical window is in a valid state and the total number of packets in the window does not exceed K, accumulate the total number of packets in the window, specifically, K=50; If a packet is marked as out of order in S11, and this is not the first time an out-of-order packet has appeared in this TCP session, then the number of out-of-order packets in the window is accumulated, the difference between the sequence number of the TCP packet itself and the expected sequence number in the corresponding transmission direction is calculated and accumulated into the total number of out-of-order offsets in the window, and the length of the payload bytes in the window is accumulated according to the length of the TCP payload bytes of the current TCP packet. If a message is marked as sequential in S11, the number of messages in the current out-of-order buffer queue is added to the total number of messages waiting to be filled in the window, and the number of times the message is filled in the window is also added. The specific calculation of session disorder scoring in S2 is as follows: S21: If the number of packets counted in the statistics window reaches K, then the statistics window for this TCP session is set to the closed state; When the statistics window is closed or the TCP session ends normally, based on the statistics from S20, the scores for out-of-order occurrence percentage, average out-of-order distance, and average number of out-of-order padding packets are calculated. These scores are then weighted and summed to obtain the session out-of-order score. The calculation formula is as follows: Out-of-order score = A × Out-of-order occurrence percentage score + B × Average out-of-order distance score + C × Average number of out-of-order completion packets score; Where A, B, and C are weighting coefficients, and A+B+C=1. Specifically, A=0.5, B=0.3, and C=0.

2. The TCP service-level out-of-order mode summarized in S2 is as follows: S22: When S21 meets the judgment condition and the session out-of-order score is calculated, the session out-of-order score is added to the total session score of the corresponding TCP service, and the total session count of the TCP service is also added. When the number of TCP sessions associated with the TCP service is greater than K, the service-level out-of-order score is calculated as: total session score of TCP service / total session count of TCP service. When the service-level out-of-order score is ≤ D, it is set to low out-of-order mode; when D < service-level out-of-order score ≤ E, it is set to medium out-of-order mode; when the service-level out-of-order score is ≥ G, it is set to severe out-of-order mode. The TCP service-level out-of-order mode supports continuous updates, where D=2, E=3, and G=4. S3: Based on the TCP service-level out-of-order mode and the number of buffers in the buffer queue corresponding to the transmission direction of the TCP packet, determine the reassembly control decision result, and perform reassembly output and buffer release operations on the buffer queue according to the result; S4: Output the reassembled TCP data stream to the upper-layer application.

2. The TCP stream reassembly method based on behavior learning according to claim 1, characterized in that, The scoring rules for the indicators of out-of-order occurrence rate, average out-of-order distance, and average number of out-of-order padding packets are as follows: Out-of-order packet occurrence rate = number of out-of-order packets in the window / total number of packets in the window. When this rate is ≤ X, the score is D; when X < this rate ≤ Y, the score is E; when this rate > Y, the score is F. Where X = 0.2, Y = 0.5, and F = 5. Average out-of-order distance = Total out-of-order offsets within the window / Number of out-of-order packets within the window; Average packet payload length = payload bytes within the window / total number of packets within the window; The score is D when the average out-of-order distance is less than or equal to the average packet load length; the score is E when the average packet load length is less than or equal to Z × average packet load length; and the score is F when the average out-of-order distance is greater than Z × average packet load length, where Z = 2. Average number of packets for out-of-order completion = total number of completion waits within the window / number of completion attempts within the window. When this value is ≤ M, the score is D; when M < this value ≤ N, the score is E; when this value > N, the score is F. Where M = 1 and N = 2.

3. The TCP stream reassembly method based on behavior learning according to claim 2, characterized in that, The specific details of the restructuring control decision outcome determined in S3 are as follows: S30: If the TCP service to which the current TCP packet belongs has generated a service-level out-of-order mode, then determine the out-of-order buffer queue level based on the number of out-of-order buffer queues in the corresponding transmission direction of the TCP packet, and then combine the buffer level with the service-level out-of-order mode to determine the reassembly control decision result. The cache level determination rules are as follows: when the number of out-of-order cache queue packets is ≤ P, it is a low cache level; when P < the number of out-of-order cache queue packets is ≤ Q, it is a medium cache level; and when the number of out-of-order cache queue packets is > Q, it is a high cache level, where P=50 and Q=100.

4. The TCP stream reassembly method based on behavior learning according to claim 3, characterized in that, In S3, the specific steps for performing reorganization and cache release operations on the cache queue are as follows: S31: Prioritize processing out-of-order buffer queues in the direction of transmission from destination IP address and destination port to source IP address and source port. If the queue contains buffered data, directly reassemble the current queue and release the corresponding buffer. Next, process the sequential and out-of-order buffer queues for the source IP address, source port to destination IP address, and destination port transmission direction. First, check if there is buffered data in the sequential buffer queue. If so, perform reassembly output and release the corresponding buffer. If there is a reassembly control decision result determined by S30, process the out-of-order buffer queue according to the result. At the same time, check if the number of packets in the out-of-order buffer queue exceeds the maximum limit R. If it does, directly perform reassembly output and release the corresponding buffer, where R=200.

5. The TCP stream reassembly method based on behavior learning according to claim 4, characterized in that, The reorganization control decision result combining cache level and service-level out-of-order modes is as follows: When the cache level is low, the decision is to retain the cached data in the out-of-order cache queue; When the cache level is medium, if the service-level out-of-order mode is low or medium out-of-order, the cached data in the out-of-order cache queue is retained; if the service-level out-of-order mode is severe out-of-order, the cached data in the out-of-order cache queue is released and the output operation is performed. When the cache level is high, if the service-level out-of-order mode is low, the cached data in the out-of-order cache queue is retained; if the service-level out-of-order mode is medium or severe, the cached data in the out-of-order cache queue is released and an output operation is performed.

Images