Method, product and device for processing data packets during database replay

Abstract

This invention provides a data packet processing method, product, and device for database playback. The method includes: for each received data packet, performing a sequence number verification operation based on the sequence number status structure corresponding to the data packet's transmission direction; if the data packet is diverted to the out-of-order processing flow and is an out-of-order data packet, creating out-of-order segments to store the out-of-order data packet and inserting the out-of-order segments sequentially into the corresponding out-of-order linked list; if the data packet is diverted to the sequential processing flow and is a sequential data packet, updating the sequence number status structure corresponding to the sequential data packet, appending the payload data of the sequential data packet to the stream buffer, and parsing the sequential data packet; after processing a data packet, performing a continuity check on the out-of-order linked list, and if a target out-of-order segment that can be continuously concatenated is found, then continuously reassembling the target out-of-order segment. This method can ensure the integrity and accuracy of MySQL protocol parsing.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data packet processing method, product, and device for database playback. Background Technology

[0002] Driven by the domestic IT innovation policy, the replacement of databases with domestically produced ones in key industries has become a core necessity. Currently, foreign commercial databases such as Oracle and MySQL still dominate in sectors like finance, telecommunications, and government. Enterprises urgently need to complete the domestic migration with the lowest cost and minimal business risk. Heterogeneous database replay technology captures real business loads in the production environment, accurately reproduces and compares the results in the target database environment, and can quickly locate compatibility issues and performance bottlenecks after migration. It is a core supporting technology for achieving smooth domestic replacement.

[0003] Current mainstream heterogeneous database replay solutions are mostly based on network protocol packet capture. However, in real production environments, TCP packet out-of-order delivery is a common occurrence due to factors such as network path differences, intermediate device reordering, packet retransmission, and multi-path transmission. Existing solutions either discard out-of-order packets, leading to incomplete SQL command parsing and missing business logic, or employ fixed timeout mechanisms, causing severe parsing delays and redundant processing issues.

[0004] The aforementioned defects directly lead to the loss of database session state, the breakage of transaction context, a large number of unexpected execution failures during the replay process, and serious distortion between the replay results and the actual load behavior of the source end. This fails to meet the stringent requirements of critical industries for replay consistency and severely restricts the implementation of domestic database replacement. Summary of the Invention

[0005] In view of the above problems, the present invention proposes a data packet processing method, product and device in the database playback process that overcomes or at least partially solves the above problems.

[0006] One objective of this invention is to ensure high-fidelity parsing of MySQL protocol streams, maintain database session state consistency and transaction context continuity, and improve the authenticity and credibility of heterogeneous database playback results.

[0007] A further objective of this invention is to achieve accurate identification and efficient processing of data packets across all scenarios.

[0008] Specifically, the present invention provides a data packet processing method during database playback, comprising: In response to the Transmission Control Protocol connection, independent sequence number state structures are created for the first transmission direction and the second transmission direction, respectively. The first transmission direction is the client-to-server direction of the connection, and the second transmission direction is the server-to-client direction of the connection. For each received data packet, a sequence number verification operation is performed based on the sequence number status structure corresponding to the transmission direction of the data packet. The sequence number verification operation is used to determine the processing flow and type of the data packet. When a data packet is diverted to the out-of-order processing flow and is an out-of-order data packet, an out-of-order segment is created to store the out-of-order data packet, and the out-of-order segment is inserted into the out-of-order linked list corresponding to the transmission direction in order. When a data packet is diverted to the sequential processing flow and belongs to the sequential data packet, the sequence number status structure corresponding to the transmission direction of the sequential data packet is updated, the payload data of the sequential data packet is appended to the stream buffer, and the sequential data packet is parsed and processed. After processing an out-of-order data packet or a sequential data packet, the continuity of the out-of-order linked list corresponding to the transmission direction of the out-of-order data packet or the sequential data packet is checked. If a target out-of-order segment that can be continuously spliced ​​is found, the target out-of-order segment is continuously reassembled.

[0009] Optionally, the sequence number status structure includes the next expected sequence number to be received, the highest sequence number already received, and the initial sequence number; The steps for performing sequence number verification based on the sequence number state structure corresponding to the transmission direction of the data packet include: determining the starting sequence number of the data packet; determining whether the starting sequence number is equal to the expected sequence number; if the starting sequence number is equal to the expected sequence number, determining that the data packet is a sequential data packet and routing it to the sequential processing flow; if the starting sequence number is not equal to the expected sequence number, routing the data packet to the out-of-order processing flow; in the out-of-order processing flow, calculating the ending sequence number of the data packet based on the starting sequence number and the length of the payload data; based on the starting sequence number and ending sequence number of the data packet, combined with the initial sequence number and the highest sequence number in the sequence number state structure, determining whether the data packet is a completely retransmitted data packet, and discarding the data packet if so; if the data packet is not a completely retransmitted data packet, determining that the data packet is an out-of-order data packet.

[0010] Optionally, the sequence number state structure may also include a state initialization flag; Before determining the starting sequence number of the data packet, the process also includes: determining whether the current connection has been initialized based on the status initialization flag; if not, re-execute the initialization operation. The steps for updating the sequence number status structure corresponding to the transmission direction of the sequential data packet include: updating the expected sequence number in the sequence number status structure to the sum of the starting sequence number of the sequential data packet and the length of the payload data; if the updated expected sequence number is greater than the highest sequence number in the sequence number status structure, then the highest sequence number is updated synchronously.

[0011] Optionally, the steps of creating out-of-order segment storage for out-of-order data packets and inserting the out-of-order segments sequentially into the out-of-order linked list corresponding to their respective transmission directions include: Create out-of-order segments and store the payload data, starting sequence number, and data length of out-of-order data packets in the out-of-order segments; Traverse the disordered linked list and match the insertion position according to the sequence number; Store the out-of-order segment at the insertion position.

[0012] Optionally, the step of performing continuity verification on the out-of-order linked list corresponding to the transmission direction of the out-of-order data packet or the sequential data packet includes: starting the verification from the head of the out-of-order linked list; if the starting sequence number of the first out-of-order segment in the out-of-order linked list is equal to the expected sequence number corresponding to the out-of-order data packet or the sequential data packet, then the out-of-order segment is determined to be a target out-of-order segment that can be continuously spliced. The steps for continuous reassembly of the target out-of-order segment include: copying the payload data of the target out-of-order segment to the stream buffer; updating the expected sequence number to the sum of the starting sequence number and the data length of the target out-of-order segment; if the updated expected sequence number is greater than the highest sequence number, then synchronously updating the highest sequence number; removing the target out-of-order segment from the out-of-order list and releasing the corresponding memory space; continuing to perform continuity checks until the out-of-order list is empty or the head out-of-order segment no longer meets the conditions; and parsing the newly added payload data in the stream buffer.

[0013] Optionally, the step of determining whether a data packet is a complete retransmission data packet includes: if the starting sequence number is greater than or equal to the initial sequence number and the ending sequence number is less than or equal to the highest sequence number, then the data packet is determined to be a complete retransmission data packet.

[0014] Optionally, the step of performing sequence number verification based on the sequence number state structure corresponding to the transmission direction of the data packet further includes: If the start sequence number of a data packet is less than the highest sequence number and the end sequence number of the data packet is greater than the highest sequence number, then the data packet is determined to be a partially overlapping data packet. In the case of partially overlapping data packets, extract the new payload data exceeding the highest sequence number in the partially overlapping data packets as the new data packets; Re-perform the sequence number verification operation on the newly added data packets.

[0015] Optionally, the serial number comparison method includes: a wrap-around processing method, which converts the two serial numbers to be compared into signed numeric differences for comparison; The steps of appending the payload data of sequential data packets to the stream buffer and parsing the sequential data packets include: Determine if the available capacity of the stream buffer is sufficient to hold the load data; If the available capacity is insufficient to accommodate the load data, then an expansion operation will be performed; If the available capacity is sufficient to hold the load data, append the load data to the stream buffer; Sequential data packets are parsed in the stream buffer to obtain and process the valid information within them.

[0016] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the steps of the data packet processing method in the database playback process described above.

[0017] According to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor executes the machine-executable program to implement the steps of the data packet processing method in the database playback process described above.

[0018] The data packet processing method in the database playback process of the present invention, in response to the Transmission Control Protocol (TCP) connection, creates independent sequence number state structures for a first transmission direction and a second transmission direction, where the first transmission direction is the client-to-server direction and the second transmission direction is the server-to-client direction. For each received data packet, a sequence number verification operation is performed based on the sequence number state structure corresponding to the transmission direction to which the data packet belongs. The sequence number verification operation is used to determine the processing flow corresponding to the data packet and the type of the data packet. When a data packet is diverted to the out-of-order processing flow and is an out-of-order data packet, an out-of-order segment is created to store the out-of-order data packet, and the out-of-order segment is inserted sequentially into the out-of-order linked list corresponding to the transmission direction to which it belongs. When a data packet is diverted to the sequential processing flow and is a sequential data packet, the sequence number state structure corresponding to the transmission direction to which the sequential data packet belongs is updated, the payload data of the sequential data packet is appended to the stream buffer, and the sequential data packet is parsed. After processing an out-of-order data packet or a sequential data packet, a continuity check is performed on the out-of-order data packet or the out-of-order linked list corresponding to the transmission direction to which the sequential data packet belongs. If a target out-of-order segment that can be continuously spliced ​​is found, the target out-of-order segment is continuously reassembled. This method enables independent and orderly management of bidirectional data streams in the TCP Transmission Control Protocol, lossless buffering and real-time reassembly of out-of-order data packets, effectively eliminating the impact of out-of-order data packets and retransmissions on database playback. While ensuring the integrity of playback data, it improves real-time processing, guarantees continuous parsing of database sessions and transactions, and significantly improves the fidelity and reliability of database playback.

[0019] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0020] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart illustrating a data packet processing method during database playback according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating a data packet processing method during database playback according to another embodiment of the present invention; Figure 3 This is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; and Figure 5This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] Those skilled in the art should understand that the embodiments described below are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. These partial embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present invention.

[0022] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0023] Currently, foreign databases such as Oracle and MySQL still dominate key industries in China, creating a deep technological dependence. Driven by the domestic IT innovation policy, enterprises hope to replace foreign databases with domestic ones at the lowest cost, but they still face three major difficulties: domestic databases lag behind foreign products in performance and compatibility; testing environments cannot realistically replicate production scenarios; and manual migration is inefficient and prone to errors. Therefore, heterogeneous database replay technology has become an important solution. This technology captures real production workloads for precise testing, and through replay comparison analysis, it can quickly identify performance bottlenecks and compatibility issues after migration.

[0024] In traditional heterogeneous database playback schemes based on network protocol analysis and packet capture, TCP packets may become out of order due to the complex and ever-changing network environment: 1. Differences in network paths 2. Reordering intermediate devices 3. Data packet loss and retransmission 4. Multipath transmission mechanism Traditional parsing methods either discard out-of-order packets, leading to incomplete command parsing, or simply wait, causing packet parsing delays and duplications. This makes it impossible to guarantee key indicators such as session state, SQL execution process, and even the correct parsing of data packets—core elements for measuring the effectiveness and correctness of replay.

[0025] In similar database load replays or application simulations, the common limitations are mainly reflected in the fact that most existing solutions are based on simple network traffic capture and protocol parsing, lacking the ability to deeply recreate the complex interactive states in real production environments. These solutions often assume that network transmission is ordered and stable, making it difficult to handle abnormal scenarios such as TCP out-of-order delivery, packet loss, and retransmission caused by network jitter, multipath transmission, or interference from intermediate devices.

[0026] Therefore, during the intermediate parsing steps, parsing interruptions or failures can easily occur due to disordered data packet order, making it impossible to completely and accurately reconstruct the database session and SQL command sequence. This parsing-level defect directly impacts the final replay stage: due to incomplete SQL statement parsing, lost session state information, or broken transaction contexts, numerous unexpected SQL execution failures, business logic interruptions, and even data inconsistencies may occur during replay, resulting in significant deviations and severe distortion between the replay results and the actual load behavior at the source.

[0027] However, in real-world customer business scenarios, especially in critical sectors such as finance, telecommunications, and government, users have extremely stringent requirements for the consistency and accuracy of replay results. This not only requires the replay system to correctly parse each SQL statement, but also to maintain session state consistency throughout the replay process, fully reproduce transaction boundaries and dependencies, and achieve accurate behavior matching and difference analysis during the replay result comparison phase. Therefore, designing and implementing a MySQL Packet out-of-order reassembly method for heterogeneous database replay, to ensure the authenticity and integrity of network packet capture-based load balancing, has become a fundamental requirement for implementing replay functionality from MySQL to other heterogeneous databases.

[0028] To address this core need, this invention proposes an innovative method, designing and implementing a MySQL Packet out-of-order reassembly mechanism for heterogeneous database replay. This method not only supports the detection and reassembly of out-of-order TCP packets in complex network environments but also strives to achieve high-fidelity parsing that mimics the actual interaction process at the source end. This ensures that the replay behavior more closely resembles real-world business scenarios, significantly improving the applicability, operability, and reliability of database replay technology in heterogeneous migration and simulation testing.

[0029] Specifically, the present invention provides a data packet processing method during database playback. Figure 1 This is a flowchart illustrating a data packet processing method during database playback according to an embodiment of the present invention, as shown below. Figure 1 As shown, the data packet processing method in the database playback process includes at least the following steps S101 to S105.

[0030] Step S101: In response to the Transmission Control Protocol (TCP) connection, independent sequence number state structures are created for the first and second transmission directions, respectively. The sequence number state structure typically includes the next expected sequence number to be received, the highest received sequence number, the initial sequence number, the head pointer of the out-of-order list, and a state initialization flag. An optional example is as follows: next_expected_seq: The next expected sequence number to be received (i.e., the expected sequence number). highest_seq: The highest sequence number received. initial_seq: Initial sequence number out_of_order_list: Pointer to the head of the out-of-order linked list initialized: State initialization flag.

[0031] This approach, which addresses the bidirectional communication characteristics of TCP, establishes independent state management systems for the two transmission directions, avoids crosstalk between bidirectional data stream sequence numbers, and provides a unified benchmark for subsequent data packet verification and out-of-order reordering. This is a fundamental prerequisite for ensuring the integrity of the database session state.

[0032] Step S102: For each received data packet, perform a sequence number verification operation based on the sequence number status structure corresponding to the transmission direction of the data packet. This step is the core pre-processing step for data packet processing. All received TCP data packets must first be matched to their transmission direction, and then the entire process verification is completed based on the sequence number status structure corresponding to that direction. This ensures that all judgment logic is executed based on the sequence number benchmark of the corresponding direction, avoiding cross-direction logic confusion.

[0033] In some optional embodiments, the steps of performing sequence number verification based on the sequence number state structure corresponding to the transmission direction of the data packet generally include: determining the starting sequence number of the data packet; determining whether the starting sequence number is equal to the expected sequence number; if the starting sequence number is equal to the expected sequence number, determining that the data packet is a sequential data packet and diverting it to the sequential processing flow; if the starting sequence number is not equal to the expected sequence number, diverting the data packet to the out-of-order processing flow. This can achieve fast diversion and avoid invalid calculations blocking the main process.

[0034] In the out-of-order processing flow, the end sequence number of the data packet is calculated based on the start sequence number and the length of the payload data. Based on the start and end sequence numbers, and combined with the initial and highest sequence numbers in the sequence number status structure, it is determined whether the data packet is a completely retransmitted packet. If so, the packet is discarded. If the data packet is not a completely retransmitted packet, it is determined to be an out-of-order packet. This step employs a logic of first splitting and then verifying to ensure that the processing flow of normally ordered data packets is not blocked by out-of-order or retransmitted packets, completely solving the parsing delay problem caused by traditional solutions that simply wait for out-of-order packets. Simultaneously, it accurately identifies and discards completely retransmitted packets, avoiding problems such as duplicate SQL parsing and distorted replay results caused by redundant data.

[0035] The steps for determining whether a data packet is a complete retransmission typically include: if the starting sequence number is greater than or equal to the initial sequence number and the ending sequence number is less than or equal to the highest sequence number, then the data packet is determined to be a complete retransmission data packet. When the entire sequence number range of the data packet falls within the already received sequence number range, it indicates that all payload data of the data packet has been received and processed, and it belongs to completely redundant retransmission data.

[0036] Optionally, before determining the starting sequence number of the data packet, the process may further include: determining whether the current connection has been successfully initialized based on the state initialization flag; if not, re-executing the initialization operation. This optional embodiment adds a fault tolerance mechanism to the process. For extreme scenarios such as TCP connection establishment anomalies or state loss, the state initialization status is checked before processing data packets. If initialization is incomplete, the initialization process is completed, ensuring that all data packet processing is based on a valid sequence number. This improves the fault tolerance and stability of the method, avoiding data packet processing failures and system crashes caused by connection anomalies or state loss.

[0037] Step S103: When a data packet is out-of-order and is diverted to the out-of-order processing flow, an out-of-order segment is created to store the out-of-order data packet, and the out-of-order segments are inserted sequentially into the out-of-order linked list corresponding to their respective transmission directions. This step, for non-retransmitted out-of-order data packets diverted to the out-of-order processing flow, completes caching through an independent out-of-order segment structure. It abandons the traditional approach of directly discarding out-of-order packets, achieving lossless caching of out-of-order data packets, ensuring the integrity of business data, and fundamentally avoiding the problems of incomplete SQL command parsing and playback distortion caused by discarding out-of-order packets.

[0038] In some optional embodiments, the steps of creating out-of-order segments to store out-of-order data packets and inserting the out-of-order segments sequentially into the out-of-order linked list corresponding to the transmission direction generally include: creating out-of-order segments and storing the payload data, starting sequence number, and data length of the out-of-order data packets in the out-of-order segments; traversing the out-of-order linked list and matching the insertion position sequentially according to the sequence number; and storing the out-of-order segments at the insertion positions. The sequential matching of insertion positions is generally in ascending order. This method creates a dedicated out-of-order segment structure for each out-of-order data packet through explicit caching and insertion rules, fully storing its payload data and sequence number metadata. Furthermore, matching the insertion position in ascending order of sequence number maintains the ascending sequence number characteristic of the out-of-order linked list. By caching out-of-order data packets in ascending order using an ordered linked list, subsequent continuity checks only need to check the head of the list, eliminating the need to traverse the entire list, significantly improving processing efficiency in high-concurrency scenarios. If descending order caching is chosen, checks can also be performed starting from the tail of the linked list. Those skilled in the art can design according to actual needs.

[0039] Step S104: If the data packet is a sequential data packet and has been diverted to the sequential processing flow, update the sequence number status structure corresponding to the transmission direction of the sequential data packet, append the payload data of the sequential data packet to the stream buffer, and perform parsing processing on the sequential data packet. For data packets in normal order, first complete the iterative update of the sequence number status, then write the payload data into the stream buffer, synchronously trigger application layer protocol parsing, and realize the normal processing of sequential data packets.

[0040] In some optional embodiments, the step of updating the sequence number state structure corresponding to the transmission direction of the sequential data packet generally includes: updating the expected sequence number in the sequence number state structure to the sum of the start sequence number of the sequential data packet and the payload data length; if the updated expected sequence number is greater than the highest sequence number in the sequence number state structure, then the highest sequence number is updated synchronously. In this way, after processing each sequential data packet, the expected sequence number is iterated to the end sequence number of that data packet, and the highest received sequence number is updated synchronously. This provides the latest benchmark for the verification of subsequent data packets and the reassembly of out-of-order packets, ensuring that the judgment of subsequent data packet verification and out-of-order reassembly is always based on the latest data stream state.

[0041] Optionally, the steps of appending the payload data of sequential data packets to the stream buffer and parsing the sequential data packets generally include: determining whether the available capacity of the stream buffer is sufficient to hold the payload data; if the available capacity is insufficient to hold the payload data, performing an expansion operation; if the available capacity is sufficient to hold the payload data, appending the payload data to the stream buffer; and parsing the sequential data packets in the stream buffer to obtain and process the valid information within them. Dynamic buffer management avoids both data loss due to buffer overflows and resource waste caused by excessive pre-allocation of memory, achieving efficient memory utilization.

[0042] Step S105: After either step S103 or step S104 is completed, perform continuity verification on the out-of-order linked list corresponding to the transmission direction of the out-of-order data packet or the sequential data packet. If a target out-of-order segment that can be continuously spliced ​​is found, perform continuous reassembly processing on the target out-of-order segment.

[0043] In some optional embodiments, the step of performing continuity verification on the out-of-order linked list corresponding to the transmission direction of out-of-order data packets or sequential data packets generally includes: starting the verification from the head of the out-of-order linked list; if the starting sequence number of the first out-of-order segment in the out-of-order linked list is equal to the expected sequence number corresponding to the out-of-order data packet or sequential data packet, then the out-of-order segment is determined to be a target out-of-order segment that can be continuously spliced. This embodiment clarifies the rules for continuity verification. Thanks to the ascending order characteristic of the out-of-order linked list, it is only necessary to verify the out-of-order segment at the head of the linked list to determine whether there is valid data that can be continuously spliced, without traversing the entire linked list. This greatly simplifies the logic of continuity verification, improves the execution efficiency of the reassembly operation, and ensures the processing performance in high-concurrency business scenarios.

[0044] Optionally, the steps for continuous reassembly of the target out-of-order segment generally include: copying the payload data of the target out-of-order segment to the stream buffer; updating the expected sequence number to the sum of the starting sequence number and the data length of the target out-of-order segment; if the updated expected sequence number is greater than the highest sequence number, then synchronously updating the highest sequence number; removing the target out-of-order segment from the out-of-order list and releasing the corresponding memory space; continuing to perform continuity checks until the out-of-order list is empty or the head out-of-order segment does not meet the conditions; and parsing the newly added payload data in the stream buffer. This embodiment clarifies the complete loop logic of continuous reassembly. After verifying that a target out-of-order segment that can be spliced ​​is found, its payload data is first written to the stream buffer, then the sequence number status is iteratively updated, the processed out-of-order segments are removed and released, and then the continuity check is performed cyclically until there are no more continuous out-of-order segments that can be spliced. Finally, the newly added continuous data is parsed using the MySQL protocol. Through loop processing logic, all consecutive out-of-order segments can be spliced ​​together at once to fill in gaps in the data stream and prevent MySQL protocol parsing from being interrupted due to data gaps. After reassembly, memory is released immediately to avoid invalid memory occupation and ensure system stability under long-term high load operation. Protocol parsing is triggered immediately after reassembly, realizing real-time restoration of out-of-order data and ensuring the integrity and real-time performance of database playback.

[0045] Optionally, in practical applications, when there is a large amount of traffic, the sequence number of each data packet must be unique. However, since it is finite, the sequence numbers will be exhausted at some point. When available, previously used sequence numbers can be reused as needed; this reuse of sequence numbers is called sequence number wrapping. Considering that the sequence number wrapping problem may affect the comparison between sequence numbers, this invention chooses to employ a wrapping processing method, which converts the two sequence numbers to be compared into signed numerical differences for comparison.

[0046] For example, when handling sequence number comparisons, a special macro is used to handle 32-bit wrapping: Serial number comparison macro: SEQ_LT(a,b): Whether a is less than b (considering wrapping); SEQ_LE(a,b): Whether a is less than or equal to b; SEQ_GT(a,b): Whether a is greater than b; SEQ_GE(a,b): Whether a is greater than or equal to b.

[0047] Implementation principle: Convert the 32-bit unsigned sequence number to a signed 32-bit difference for comparison: #define SEQ_LT(a,b)((int32_t)((a)-(b))<0).

[0048] This method ensures that the serial number comparison logic remains accurate and effective in high-traffic business scenarios such as finance and telecommunications, guaranteeing the stability of the system during long-term operation.

[0049] This method achieves core effects such as real-time processing, zero data loss, and high memory efficiency. It can correctly reassemble MySQL protocol streams in complex network environments, accurately identify and handle various abnormal scenarios such as retransmitted packets, overlapping packets, and sequence number wrapping, ensuring the integrity and accuracy of MySQL protocol parsing. It maintains the consistency of database session state throughout the process, fully reproduces transaction boundaries and dependencies, and completely solves the parsing failure and playback distortion problems caused by TCP out-of-order delivery in traditional heterogeneous database playback solutions. It significantly improves the authenticity and credibility of heterogeneous database playback results, meets the stringent requirements of critical industries such as finance, telecommunications, and government for load balancing, and provides core technical support for the smooth and low-risk migration of domestic databases in the context of information technology innovation.

[0050] In some alternative embodiments, the step of performing sequence number verification based on the sequence number state structure corresponding to the transmission direction of the data packet may also include: if the starting sequence number of the data packet is less than the highest sequence number and the ending sequence number of the data packet is greater than the highest sequence number, then the data packet is determined to be a partially overlapping data packet; in the case of a partially overlapping data packet, the newly added payload data exceeding the highest sequence number in the partially overlapping data packet is extracted as a new data packet; the sequence number verification operation is re-executed on the new data packet. For partially overlapping packets (the first half of the data has been received, and the second half is new data) commonly found in TCP retransmission scenarios, only the newly added payload data exceeding the received highest sequence number is extracted, the received overlapping portion is discarded, and the new data is used as a new data packet to re-execute the verification process. This avoids the problems of duplicate data and SQL parsing errors caused by processing all partially overlapping packets, and also avoids the loss of valid data caused by discarding all packets, ensuring the uniqueness and integrity of the data stream, and further improving the accuracy of MySQL protocol parsing.

[0051] Figure 2 This is a flowchart illustrating a data packet processing method during database playback according to another embodiment of the present invention, as shown below. Figure 2 As shown, the data packet processing method in the database playback process includes at least the following steps S201 to S218.

[0052] The prerequisite for execution in this embodiment is that the sequence number state structure initialization operation has been completed. When a newly established Transmission Control Protocol (TCP) connection is detected, independent sequence number state structures are created for the first transmission direction (client to server direction) and the second transmission direction (server to client direction). The sequence number state structure includes the next expected sequence number to be received, the highest received sequence number, the initial sequence number, the out-of-order list head pointer, and the state initialization flag, providing a unified and accurate sequence number reference for subsequent data packet processing.

[0053] Step S201: Receive data packets. The execution process of this step is generally as follows: The system uses a network packet capture module to acquire TCP data packets during the MySQL database business interaction process, matches the TCP connection and transmission direction to which the data packet belongs, and locks the independent sequence number state structure corresponding to that direction, providing a benchmark for subsequent verification operations. Furthermore, all sequence number comparisons in this step can employ a wrap-around processing method, such as converting the 32-bit unsigned sequence number into a 32-bit signed difference for comparison, thus adapting to extreme scenarios of TCP sequence number wrap-around.

[0054] Step S202: Determine if the starting sequence number is equal to the expected sequence number. This step typically involves extracting the starting sequence number from the current TCP packet header and comparing it with the next expected sequence number to be received in the sequence number state structure of the transmission direction to which the packet belongs. This determination quickly separates ordered and out-of-order packets, ensuring that the processing flow of normally ordered packets is not blocked by out-of-order packets. This completely solves the parsing delay problem caused by the traditional approach of simply waiting for out-of-order packets, significantly improving the efficiency of the main processing flow.

[0055] Step S203: If the determination in step S202 is correct, proceed to the sequential packet processing flow. When the starting sequence number equals the expected sequence number, it indicates that the data packet arrived in order according to the TCP transmission specification and belongs to the normal business data flow, thus entering the dedicated sequential packet processing flow. This completely isolates the normal ordered business data flow from the abnormal out-of-order data flow, ensuring the real-time processing of normal business data, preventing out-of-order data packets from interfering with the main process, and ensuring the real-time performance of database playback.

[0056] Step S204: Update the sequence number status structure corresponding to the transmission direction of the sequential data packet. The execution process of this step is generally as follows: update the next expected sequence number to be received in the sequence number status structure to the sum of the starting sequence number of the current sequential data packet and the payload data length; if the updated expected sequence number is greater than the highest received sequence number in the sequence number status structure, then synchronously update the highest received sequence number, completing the iteration of the sequence number baseline. This real-time maintenance of the latest sequence number tracking baseline provides an accurate basis for subsequent data packet verification and out-of-order packet reassembly judgment, ensuring the logical coherence of the entire data stream processing process.

[0057] Step S205: Append the payload data of the sequential data packets to the stream buffer. The execution process of this step is generally as follows: First, check the available capacity of the stream buffer. If the available capacity is insufficient to accommodate the current payload data, perform a dynamic expansion operation. After expansion, append the payload data of the sequential data packets to the stream buffer and update the total data length of the stream buffer. This dynamic buffer management mechanism avoids both buffer overflow leading to business data loss and excessive pre-allocation of memory, achieving efficient memory utilization. Simultaneously, it ensures the continuity and integrity of the MySQL protocol data stream, providing a stable data carrier for upper-layer protocol parsing.

[0058] Step S206: Perform parsing processing. This step typically involves parsing the complete MySQL protocol data packet from the stream buffer, extracting and processing valid database commands, response information, etc., to reconstruct the database session interaction process and SQL execution logic.

[0059] Step S207: If the determination in step S202 is negative, proceed to the out-of-order packet processing flow. When the starting sequence number is not equal to the expected sequence number, it indicates that the data packet did not arrive in the order specified by the TCP transmission standard and is an out-of-order data packet, thus entering a dedicated out-of-order packet processing flow. This independent branch processing of out-of-order data packets avoids the shortcomings of traditional solutions that directly discard out-of-order packets, providing a process foundation for lossless caching and subsequent reassembly of out-of-order data packets, and fundamentally preventing the problems of incomplete SQL command parsing and playback distortion caused by discarding out-of-order packets.

[0060] Step S208: Determine whether it is a completely retransmitted data packet. The execution logic of this step is generally as follows: First, calculate the end sequence number of the data packet based on the start sequence number and the length of the payload data; then, based on the start sequence number, end sequence number, and the initial sequence number and the highest received sequence number in the sequence number state structure, determine whether it is a completely retransmitted data packet. The determination rule is: the start sequence number is greater than or equal to the initial sequence number, and the end sequence number is less than or equal to the highest received sequence number.

[0061] Furthermore, if it is determined that the data packet is not a completely retransmitted data packet, it can be further verified whether it is a partially overlapping data packet. If the starting sequence number of the data packet is less than the highest received sequence number and the ending sequence number is greater than the highest received sequence number, it is determined to be a partially overlapping data packet. Only the newly added payload data exceeding the highest received sequence number is extracted as the new data packet, and the sequence number verification operation is re-performed on the new data packet. This can accurately identify completely redundant retransmitted data packets and partially overlapping data packets, eliminate invalid redundant data, and retain only valid new data, avoiding the waste of system resources and the disorder of playback results caused by repeated processing and repeated parsing, while ensuring that valid business data is not lost.

[0062] Step S209: If the determination in step S208 is yes, discard the fully retransmitted data packet. Since all payload data of the fully retransmitted data packet has been received and processed, and there is no new valid data, the data packet is directly discarded, and the operation log is recorded for subsequent auditing and troubleshooting. This can quickly eliminate invalid and redundant data, reduce invalid calculations in subsequent processes, save system memory and CPU resources, and avoid SQL duplicate execution and business logic errors caused by duplicate data, thus ensuring the accuracy of the replay results.

[0063] Step S210: If the determination in step S208 is negative, create an out-of-order segment to store the out-of-order data packets. The execution logic of this step is generally as follows: allocate independent memory space to create the out-of-order segment structure, copy the payload data of the out-of-order data packets to an independent buffer, and simultaneously record core metadata such as the starting sequence number and data length of the data packet, thus achieving independent caching of out-of-order data packets. This allows for lossless and independent caching of out-of-order data packets, without occupying the buffer of normal business flow, avoiding interference from out-of-order data with the parsing process of sequential packets, and completely preserving all valid business data, achieving zero data loss.

[0064] Step S211: Insert the out-of-order segment into the out-of-order linked list corresponding to the transmission direction in ascending order. The execution logic of this step is generally as follows: traverse the out-of-order linked list corresponding to the current transmission direction, match the insertion position in ascending order of sequence number (stop when the sequence number of an existing out-of-order segment in the linked list is less than the sequence number of the current out-of-order segment), insert the new out-of-order segment into the linked list, always maintaining the ascending order of the sequence number of the out-of-order linked list, and record the operation log at the same time.

[0065] By caching out-of-order data packets using an ordered linked list, subsequent continuity checks only need to check the head of the linked list, eliminating the need to traverse the entire list. This significantly improves processing efficiency in high-concurrency business scenarios and provides an ordered data foundation for subsequent continuous reassembly.

[0066] Step S212: After step S206 or step S211 is completed, determine whether the starting sequence number of the first out-of-order segment is equal to the expected sequence number. This step adopts a dual-trigger mechanism: whether the sequential packet parsing process is completed (step S206) or the out-of-order packet ordered caching is completed (step S211), the continuity check of the out-of-order linked list is immediately triggered; the check logic is to check from the head of the out-of-order linked list and determine whether the starting sequence number of the first out-of-order segment is equal to the latest expected sequence number. This is one of the core innovations of this invention. It abandons the logic of fixed timeout waiting to trigger reassembly in traditional solutions and adopts event-triggered real-time reassembly. There is no need to set a timeout waiting time, which not only ensures zero loss of business data, but also minimizes parsing latency, and completely solves the core contradiction that traditional solutions cannot simultaneously take into account data integrity and real-time performance.

[0067] Step S213: If the determination in step S212 is negative or step S209 is completed, wait for the next data packet. If there are no consecutively concatenated out-of-order segments at the head of the out-of-order list, or if the discarding process of the completely retransmitted data packet has been completed and there is no valid data to process, return to the process entry point and wait to receive the next TCP data packet, maintaining the closed-loop operation of the entire processing flow.

[0068] Step S214: If the determination in step S212 is yes, copy the payload data of the target out-of-order segment to the stream buffer. When a target out-of-order segment that can be continuously spliced ​​is found, copy the payload data stored in the out-of-order segment completely to the stream buffer of the normal business flow to fill in the gaps in the data stream. Incorporating the valid data in the out-of-order cache into the normal business data stream can complete the continuous sequence of the TCP data stream, avoid interruption of MySQL protocol parsing due to data gaps, and ensure the integrity of protocol parsing.

[0069] Step S215: Update the sequence number status. The specific update operation can generally be: update the next expected sequence number to be received in the sequence number status structure to the sum of the starting sequence number of the current target out-of-order segment and the data length; if the updated expected sequence number is greater than the highest received sequence number, then synchronously update the highest received sequence number.

[0070] Step S216: Remove the target out-of-order segment from the out-of-order linked list and release the corresponding memory space.

[0071] Step S217: Determine if there is another target out-of-order segment that can be continuously concatenated. If the determination in step S217 is yes, continue to execute step S214. After completing the reorganization processing of the current target out-of-order segment, check again whether the out-of-order segment at the head of the out-of-order linked list meets the continuous concatenation condition. If it does, enter the loop and continue processing the next concatenable out-of-order segment, repeatedly executing steps S214 to S217. Through the loop processing logic, all continuous concatenable out-of-order segments are concatenated at once, completely filling all gaps in the data stream, ensuring the high continuity of the MySQL protocol data stream, avoiding performance loss caused by multiple reorganization triggers, and improving reorganization efficiency.

[0072] Step S218: If the determination in step S217 is negative, the continuous reassembly ends. After step S218 is completed, step S206 continues to parse the reassembled payload data. When there are no continuous, concatenable out-of-order segments in the out-of-order list, the current continuous reassembly process ends; and for the newly added, reassembled continuous payload data in the stream buffer, the parsing process in step S206 is performed to restore the corresponding MySQL protocol content. Protocol parsing is triggered immediately after out-of-order reassembly, achieving deep linkage between TCP out-of-order reassembly and MySQL application layer protocol parsing, ensuring that the business SQL commands and session states corresponding to the out-of-order data are completely restored, and guaranteeing high fidelity throughout the entire heterogeneous database replay process.

[0073] The method of this invention maintains an independent sequence number state and out-of-order linked list for each TCP direction, enabling real-time, zero-loss data reassembly in the event of out-of-order network packets. It can accurately identify retransmitted packets and skip redundant processing. At the same time, through the ordered insertion and continuity check mechanism of the linked list, it ensures that out-of-order packets are correctly sorted and reassembled into the data stream in a timely manner, thereby guaranteeing the integrity and accuracy of MySQL protocol parsing and laying the foundation for the implementation of heterogeneous database replay technology.

[0074] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0075] It should be understood that in some embodiments, the components may be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0076] This embodiment also provides a computer program product 10, a computer-readable storage medium 20, and a computer device 30. Figure 3This is a schematic diagram of a computer program product 10 according to an embodiment of the present invention. Figure 4 This is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention. Figure 5 This is a schematic diagram of a computer device 30 according to an embodiment of the present invention. The computer program product 10 includes a computer program 11, which, when executed by the processor 32, implements the steps of the data packet processing method in the database playback process described above. A computer-readable storage medium 20 stores the computer program 11 thereon, which, when executed by the processor 32, implements the steps of the data packet processing method in the database playback process described above. The computer device 30 may include a memory 31, a processor 32, and the computer program 11 stored in the memory 31 and running on the processor 32.

[0077] The computer program 11 used to perform the operations of this invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages. The computer program 11 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, Field-Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from computer-readable program instructions.

[0078] For the purposes of this embodiment, computer program product 10 is a related product containing computer program 11. For the purposes of this embodiment, computer-readable storage medium 20 is a tangible device capable of holding and storing computer program 11, and can be any device capable of containing, storing, communicating, propagating, or transmitting program 11 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage medium 20 include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0079] Computer device 30 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 30 can be a cloud computing node. Computer device 30 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 30 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0080] Computer device 30 may include a processor 32 adapted to execute stored instructions and a memory 31 that provides temporary storage space for the operation of said instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 31 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0081] Computer device 30 may also include a network adapter / interface and an input / output (I / O) interface. The I / O interface allows external devices that can be connected to the computer device to input and output data. The network adapter / interface provides communication between the computer device and a network, typically represented as a communication network.

[0082] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A data packet processing method during database playback, comprising: In response to the Transmission Control Protocol connection, independent sequence number state structures are created for a first transmission direction and a second transmission direction, wherein the first transmission direction is the client-to-server direction of the connection and the second transmission direction is the server-to-client direction of the connection. For each received data packet, a sequence number verification operation is performed based on the sequence number status structure corresponding to the transmission direction to which the data packet belongs. The sequence number verification operation is used to determine the processing flow corresponding to the data packet and the type of the data packet. When the data packet is diverted to the out-of-order processing flow and is an out-of-order data packet, an out-of-order segment is created to store the out-of-order data packet, and the out-of-order segment is inserted into the out-of-order linked list corresponding to the transmission direction in order; When the data packet is diverted to the sequential processing flow and belongs to the sequential data packet, the sequence number status structure corresponding to the transmission direction of the sequential data packet is updated, the payload data of the sequential data packet is appended to the stream buffer, and the sequential data packet is parsed. After processing one out-of-order data packet or one sequential data packet, the continuity of the out-of-order linked list corresponding to the transmission direction of the out-of-order data packet or the sequential data packet is checked. If a target out-of-order segment that can be continuously spliced ​​is found, the target out-of-order segment is continuously reassembled.

2. The data packet processing method in the database playback process according to claim 1, wherein, The sequence number status structure includes the next expected sequence number to be received, the highest sequence number already received, and the initial sequence number; The steps of performing sequence number verification based on the sequence number state structure corresponding to the transmission direction of the data packet include: determining the starting sequence number of the data packet; determining whether the starting sequence number is equal to the expected sequence number; if the starting sequence number is equal to the expected sequence number, determining that the data packet is the ordered data packet and diverting it to the ordered processing flow; if the starting sequence number is not equal to the expected sequence number, diverting the data packet to the out-of-order processing flow; in the out-of-order processing flow, calculating the ending sequence number of the data packet based on the starting sequence number and the length of the payload data; determining whether the data packet is a completely retransmitted data packet based on the starting sequence number and the ending sequence number of the data packet, combined with the initial sequence number and the highest sequence number in the sequence number state structure, if so, discarding the data packet; if the data packet is not a completely retransmitted data packet, determining that the data packet is the out-of-order data packet.

3. The data packet processing method in the database playback process according to claim 2, wherein, The sequence number status structure also includes a status initialization flag; Before the step of determining the starting sequence number of the data packet, the method further includes: determining whether the current connection has been initialized based on the status initialization flag; if not, re-executing the initialization operation. The step of updating the sequence number status structure corresponding to the transmission direction of the sequential data packet includes: updating the expected sequence number in the sequence number status structure to the sum of the starting sequence number of the sequential data packet and the length of the payload data; if the updated expected sequence number is greater than the highest sequence number in the sequence number status structure, then the highest sequence number is updated synchronously.

4. The data packet processing method in the database playback process according to claim 2, wherein, The steps of creating out-of-order segments to store the out-of-order data packets and inserting the out-of-order segments sequentially into the out-of-order linked list corresponding to the transmission direction include: Create the out-of-order segment, and store the payload data, starting sequence number, and data length of the out-of-order data packet in the out-of-order segment; Traverse the disordered linked list and match the insertion position according to the sequence number; The out-of-order segment is stored at the insertion position.

5. The data packet processing method in the database playback process according to claim 2, wherein, The step of performing continuity verification on the out-of-order linked list corresponding to the transmission direction of the out-of-order data packet or the sequential data packet includes: starting the verification from the head of the out-of-order linked list; if the starting sequence number of the first out-of-order segment in the out-of-order linked list is equal to the expected sequence number corresponding to the out-of-order data packet or the sequential data packet, then the out-of-order segment is determined to be the target out-of-order segment that can be continuously spliced. The steps for continuously reassembling the target out-of-order segment include: copying the load data of the target out-of-order segment to the stream buffer; updating the expected sequence number to the sum of the starting sequence number and the data length of the target out-of-order segment; if the updated expected sequence number is greater than the highest sequence number, then synchronously updating the highest sequence number; removing the target out-of-order segment from the out-of-order list and releasing the corresponding memory space; continuing the continuity check until the out-of-order list is empty or the head out-of-order segment does not meet the conditions; and parsing the newly added load data in the stream buffer.

6. The data packet processing method in the database playback process according to claim 2, wherein, The step of determining whether the data packet is a complete retransmission data packet includes: if the starting sequence number is greater than or equal to the initial sequence number and the ending sequence number is less than or equal to the highest sequence number, then the data packet is determined to be the complete retransmission data packet.

7. The data packet processing method in the database playback process according to claim 2, wherein, The step of performing sequence number verification based on the sequence number status structure corresponding to the transmission direction of the data packet further includes: If the start sequence number of the data packet is less than the highest sequence number and the end sequence number of the data packet is greater than the highest sequence number, then the data packet is determined to be a partially overlapping data packet. In the case that the data packet is a partially overlapping data packet, the newly added payload data exceeding the highest sequence number in the partially overlapping data packet is extracted as a new data packet; The sequence number verification operation is performed again on the newly added data packet.

8. The data packet processing method in the database playback process according to claim 1, wherein, The serial number comparison method includes: a wrap-around processing method, wherein the wrap-around processing method is used to convert the two serial numbers to be compared into signed numeric differences for comparison; The steps of appending the payload data of the sequential data packets to the stream buffer and parsing the sequential data packets include: Determine whether the available capacity of the stream buffer is sufficient to accommodate the load data; If the available capacity is insufficient to accommodate the load data, then an expansion operation is performed; If the available capacity is sufficient to accommodate the load data, the load data is appended to the stream buffer; The sequential data packets are parsed in the stream buffer to obtain and process the valid information contained therein.

9. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the data packet processing method in the database playback process according to any one of claims 1 to 8.

10. A computer device comprising a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor, when executing the machine-executable program, implements the steps of the data packet processing method in a database playback process according to any one of claims 1 to 8.

Images