Transaction node data pre-remote synchronization system based on queuing machine

Abstract

Embodiments of the present disclosure disclose a queuing machine-based transaction node data pre-remote synchronization system. A specific implementation of the system includes a target client and a server, wherein: the target client performs the following steps: obtaining local configuration information and local historical synchronization site information; generating incremental synchronization request information, and sending the incremental synchronization request information to the server; the server performs the following steps: performing incremental connection point positioning processing on the synchronization site information to obtain log file connection point information; reading log incremental data packets; performing compression stream push processing on the log incremental data packets, and sending the log incremental data packets to the target client; the target client performs the following steps: storing the log incremental data packets to the local queuing machine; updating the synchronization site information to obtain updated synchronization site information, and writing the updated synchronization site information to a local synchronization site file for persistent storage. The implementation reduces the waste of transmission resources.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a pre-synchronization system for transaction node data based on a queuing machine. Background Technology

[0002] Statistics show that my country's financial trading systems process billions of transactions daily, with system availability requirements generally exceeding 99.999%, meaning annual downtime of no more than 5 minutes. Meanwhile, financial institutions face increasingly complex disaster scenarios. In core scenarios such as securities matching and futures trading, trading systems need to handle massive amounts of requests in real time, demanding high data consistency. To prevent service interruptions due to data center power outages, network failures, or hardware malfunctions, trading systems commonly employ a primary-backup or off-site disaster recovery architecture. When the primary node fails or undergoes planned maintenance, the system needs to switch traffic to the backup node. To ensure uninterrupted business operations and no data loss after the switch, the requirement for pre-synchronization of trading node data across different locations has been introduced. Pre-synchronization of trading node data across different locations refers to a technology that continuously replicates the primary node's transaction log data to a remote backup node during normal primary node operation, keeping the backup node's memory state (such as order book and fund freeze status) close to that of the primary node. During pre-synchronization, it is necessary to record position information including the current log file identifier and byte offset, which the primary node uses to continue pushing subsequent data. Currently, when performing pre-synchronization of transaction node data, the common approach is to continuously push QLog data blocks from the source end to the target end. When the target end receives the QLog data block, it only records the currently received file number and offset in memory.

[0003] However, when using the above method to perform pre-synchronization of transaction node data in different locations, the following technical problems often arise: The source continuously pushes QLog data blocks to the target. When the target receives a QLog data block, it only records the currently received position information (including log file identifier and file byte offset) in memory. Once the target process crashes or the network is interrupted, the position information temporarily stored in memory will be lost, causing the target to be unable to provide accurate synchronization progress after the connection is restored. This means that after the connection is restored, the source cannot know the position where the target has successfully received the data and can only start data transmission from the moment the session was established. This results in the repeated transmission of a large amount of data, wasting transmission resources. Secondly, in actual operation and maintenance, disaster recovery nodes need to be upgraded regularly to fix bugs or introduce new features. However, different versions of disaster recovery nodes often use different internal data structures or serialization protocols. Older versions of disaster recovery nodes may use binary or Protobuf format to store positions, while newer versions may have migrated to JSON or XML format, or the field definitions (e.g., the data type of the log file identifier changes from Int32 to Int64) may have changed. This causes the new version of the disaster recovery node to be unable to parse the position files left over from the old version when it starts up. This prevents the disaster recovery node from recovering valid log file identifiers and file byte offsets from the old location files, thus making it unable to inform the master node of the physical location of the last synchronization interruption. The master node resets its local location to the starting position, causing a large amount of already transmitted data to be transmitted repeatedly, wasting computing and network resources.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose a pre-synchronization system and method for transaction node data based on a queuing machine to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a pre-emptive remote synchronization system for transaction node data based on a queuing machine. The system includes: a target client and a server, wherein the target client and the server are communicatively connected, and the target client is configured to perform the following steps: acquiring local configuration information and local historical synchronization point information; generating incremental synchronization request information including transmission parameter configuration information and synchronization point information based on the local configuration information and the local historical synchronization point information; and sending the incremental synchronization request information to the server; the server is configured to perform the following incremental streaming push step: responding to receiving the incremental synchronization request sent by the target client... The request information involves performing incremental connection point location processing on the synchronization point information included in the incremental synchronization request information to obtain log file connection point information; based on the log file connection point information, reading the log incremental data packet; and performing compressed streaming push processing on the log incremental data packet according to the transmission parameter configuration information included in the incremental synchronization request information to send the log incremental data packet to the target client; the target client is further configured to perform the following steps: in response to receiving the log incremental data packet, storing the log incremental data packet in a local queuing machine based on the transmission parameter configuration information; and updating the synchronization point information to obtain updated synchronization point information.

[0008] Secondly, some embodiments of this disclosure provide a method for pre-synchronization of transaction node data in a remote location based on a queuing machine. The method includes: acquiring local configuration information and local historical synchronization point information; generating incremental synchronization request information including transmission parameter configuration information and synchronization point information based on the local configuration information and the local historical synchronization point information; and sending the incremental synchronization request information to the server. The server is configured to perform the following incremental streaming push steps: in response to receiving the incremental synchronization request information sent by the target client, performing incremental connection point location processing on the synchronization point information included in the incremental synchronization request information to obtain log file connection point information; reading log incremental data packets based on the log file connection point information; performing compressed streaming push processing on the log incremental data packets according to the transmission parameter configuration information included in the incremental synchronization request information to send the log incremental data packets to the target client; in response to receiving the log incremental data packets, storing the log incremental data packets in a local queuing machine based on the transmission parameter configuration information; and updating the synchronization point information to obtain updated synchronization point information.

[0009] The above embodiments of this disclosure have the following beneficial effects: the pre-synchronization system for transaction node data based on a queuing machine in some embodiments of this disclosure reduces the waste of transmission resources. Specifically, the waste of transmission resources is caused by the fact that the source continuously pushes QLog data blocks to the target. When the target receives the QLog data block, it only records the currently received position information (including log file identifier and file byte offset) in memory. Once the target process crashes or the network is interrupted, the position information temporarily stored in memory will be lost, resulting in the target being unable to provide accurate synchronization progress after the connection is restored. As a result, the source cannot know the position where the target has successfully received the data after the connection is restored, and can only start data transmission from the moment the session is established. This leads to the repeated transmission of a large amount of data, resulting in the waste of transmission resources. Based on this, the pre-synchronization system for transaction node data based on a queuing machine in some embodiments of this disclosure includes: a server and a target client. The target client is communicatively connected to the server, wherein the target client is configured to perform the following steps: First, obtain local configuration information and local historical synchronization position information. This allows the target client to load the log file identifier, file byte offset, and logical data sequence number from the last successfully synchronized file in its local persistent storage. Even if the target client process crashes or the system restarts, the target client can still provide a starting point for subsequent breakpoint resumption, avoiding the need to retransmit data from scratch due to lost points. Then, based on the aforementioned local configuration information and the aforementioned local historical synchronization point information, an incremental synchronization request message including transmission parameter configuration information and synchronization point information is generated and sent to the aforementioned server. This allows the target client and the server to reach an agreement on the transmission strategy, ensuring that each step of subsequent data transmission is performed under mutually agreed-upon rules. This avoids transmission anomalies and duplicate requests caused by parameter mismatches, reducing retransmission overhead. The aforementioned server is configured to perform the following incremental streaming push steps: First, in response to receiving the incremental synchronization request message sent by the aforementioned target client, it performs incremental connection point location processing on the synchronization point information included in the incremental synchronization request message to obtain the log file connection point information. Therefore, the server can open the corresponding QLog file based on the log file identifier and file byte offset in the target client's request, convert the logical data sequence number into a physical read offset within the file, so that the server can jump to the first unsent record after the breakpoint, physically avoiding the retransmission of processed data from the file header or session start position. Then, based on the aforementioned log file join point information, the incremental log data packet is read. Finally, according to the transmission parameter configuration information included in the aforementioned incremental synchronization request information, the incremental log data packet is compressed and streamed for push, so as to send the incremental log data packet to the target client.Therefore, the server can continuously read incremental log data packets and send them to the target client according to the corresponding compression configuration information and rate limiting policy. The target client is further configured to perform the following steps: First, in response to receiving the incremental log data packets, it stores the incremental log data packets in a local queuing machine based on the transmission parameter configuration information. This allows the target client to decompress the received incremental log data packets sequentially and write them to the local queuing machine, completing the data synchronization. Then, the synchronization point information is updated to obtain the updated synchronization point information, and this updated synchronization point information is written to a local synchronization point file for persistent storage. This allows the target client to write the latest synchronized log file identifier, logical data sequence number, and file byte offset to the local synchronization point file after each batch of data is written, ensuring persistent storage. Even if the target client's process crashes or the network is interrupted, and the point information in memory is lost, it can be reloaded from the local synchronization point file after restarting, thus restoring the data synchronization progress to the state before the interruption. Also, because the target client sends incremental synchronization request information with synchronization point information, the server reads incremental log data packets based on the incremental synchronization request information. Then, the incremental log data packets are sent to the target client according to the corresponding compression configuration information and rate limiting policy. The target client stores the incremental log data packets and updates the synchronization point information. This synchronization point information is persistently stored, allowing the server to locate the last interrupted position using the synchronization point information provided by the target client after any anomaly recovery, and only push unconfirmed incremental data. This avoids retransmissions caused by lost or corrupted memory points, reducing the waste of transmission resources. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0011] Figure 1 This is an architecture diagram of an exemplary system for a pre-distributed remote synchronization system of transaction node data based on a queuing machine, according to this disclosure. Figure 2 This is a flowchart of some embodiments of the pre-synchronization method for transaction node data based on a queuing machine according to the present disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] Figure 1 An exemplary system architecture 100 for a pre-location synchronization system of transaction node data based on a queuing machine, to which some embodiments of the present disclosure may be applied, is shown.

[0019] like Figure 1 As shown, system architecture 100 may include: target client 101, server 102, and network 103. Network 103 serves as the medium for providing a communication link between target client 101 and server 102. Network 103 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0020] In some embodiments, the target client 101 described above can be configured to perform the following steps: First, obtain local configuration information and local historical synchronization point information.

[0021] In some embodiments, the target client 101 can obtain local configuration information and local historical synchronization point information. The local configuration information can be the target client's startup configuration and local running configuration, including at least the server address and port, the target client's system number, node number, QLog file directory, point file path, compression identifier, candidate compression algorithm, rate limiting threshold, buffer size, and idle waiting time. The target client can be a software function module or independent application deployed on a data consumption end (e.g., a business server, backup node, or the host where the analysis engine resides). In practice, firstly, the target client can locate the local configuration file according to the path specified in the startup command. Then, it calls a parser corresponding to the local configuration file format (e.g., a DOM parser, a JSON parsing library) to read the key-value pairs or structured data in the local configuration file as local configuration information. Finally, it reads the local historical synchronization point information from the local synchronization point file on the local disk. The local configuration file can be a structured parameter file pre-stored on a persistent storage medium (e.g., a disk) on the host where the target client resides. The local configuration file records the static configuration parameters necessary for the target client's startup and operation. The aforementioned local synchronization point file is a dedicated file stored on the physical machine or container persistent storage (e.g., disk or SSD) of the client. It records checkpoints in the synchronization progress, allowing the target client to resume synchronization from the last interrupted position after a restart, network interruption, or process crash. The aforementioned local historical synchronization point information may include the log file identifier (e.g., file_id), file byte offset (e.g., file_pos), logical data sequence number (e.g., data_id), and transaction date of the most recent successful synchronization event. The log file identifier (e.g., file_id) is a unique ID used to identify the log file or data shard containing the data. The file byte offset (e.g., file_pos) represents the byte offset position within the file, pointing to the end point of this synchronization (i.e., the start point of the next synchronization). The logical data sequence number (e.g., data_id) is a globally incrementing sequence number recorded at the data source end. For example, the log file identifier could be file_id=12. The file byte offset could be file_pos=409600. The logical data sequence number could be data_id=880120. The transaction date mentioned above could be trade_date=20260522.

[0022] Second, based on local configuration information and local historical synchronization point information, an incremental synchronization request information including transmission parameter configuration information and synchronization point information is generated, and the incremental synchronization request information is sent to the server.

[0023] In some embodiments, the target client 101 can generate incremental synchronization request information, including transmission parameter configuration information and synchronization point information, based on the local configuration information and the local historical synchronization point information, and send the incremental synchronization request information to the server. In practice, the target client 101 can send the incremental synchronization request information to the server through the communication connection between the target client and the server.

[0024] In some optional implementations of certain embodiments, the target client 101 can generate incremental synchronization request information including transmission parameter configuration information and synchronization position information based on the aforementioned local configuration information and the aforementioned local historical synchronization position information through the following steps: The first step involves performing parameter query processing on the server based on the aforementioned communication connection to obtain server parameter information. This server parameter information includes various parameter details. Each parameter can be a set of key-value pairs (e.g., [max_batch_size: 1000]) parsed from the server's response message by the target client when initiating a parameter query request. In practice, the target client first generates a parameter query request message according to a predefined communication protocol (e.g., a private format of a binary protocol, or the application layer protocol of Protobuf). Then, it sends the parameter query request message through the established communication channel (Socket connection) and starts a timeout timer. After receiving the request, the server constructs a set of parameters stored as key-value pairs using parameters from its local configuration file (e.g., max_connections, supported_compress_algorithms, default_speed_limit, etc.) and its current runtime state, which serves as the response message. Finally, the target client reads the response message before the timeout. Next, the target client parses the parameter values ​​from the response message as candidate parameter information. Then, it reads the static parameters from the local configuration file. Afterward, for each parameter in the server-side parameter information, the target client performs the following steps: First, in response to determining that the static parameters include the parameter information, it determines the parameter value in the static parameters as the target value of the parameter information. Then, in response to determining that the static parameters do not include the parameter information, it checks whether the candidate parameter information includes the parameter information. Then, in response to determining that the candidate parameter information includes the parameter information, it determines the corresponding parameter value in the candidate parameter information as the parameter information. In response to determining that the candidate parameter information does not include the parameter information, it uses a preset system default value as the parameter information. Finally, it determines the obtained parameter information as the server-side parameter information. As an example, one parameter in the server-side parameter information is the maximum batch size (e.g., max_batch_size). First, it checks the local configuration file; if the local configuration file configures the maximum batch size as 500, then it directly uses 500 as the value of the maximum batch size. Next, if the local configuration file is checked and a maximum batch size field is missing (or the value is commented out, empty, or invalid), then the corresponding field in the server response message is checked. If the server returns a maximum batch size of 1000, then 1000 is used as the maximum batch size value. Then, if the maximum batch size is not configured in the local configuration file and the server does not send a maximum batch size value, the system's built-in default value (e.g., maximum batch size = 200) is used.The parameter query request message can carry the target client's own identifier (such as client ID or tenant name) so that the server can return personalized parameters related to the target client. The current runtime state can be the real-time, dynamic, and transient information of the server's internal process and environment at the moment the request is received. The current runtime state can include, but is not limited to, hardware load (e.g., current CPU utilization, memory usage, disk I / O latency, network bandwidth utilization, etc.), connection status (e.g., the current total number of socket connections, number of active threads, database connection pool utilization), request throughput (e.g., current QPS (queries per second), request queue backlog length (number of tasks waiting to be processed)), cumulative business value (total request processing volume since service startup, current cache hit rate, cumulative count of specific business logic (e.g., "total number of verification codes issued today")), system timestamp (the server's current precise time (often used for time-sensitive business verification or synchronization)), and runtime (the duration for which the service process has been started and running continuously).

[0025] The second step involves generating the target client identifier, session identifier, and transmission parameter configuration information based on the aforementioned local configuration information. In practice, firstly, the target client can determine the compression identifier and candidate compression algorithms included in the aforementioned local configuration information as the compression configuration information. Then, the rate limiting threshold included in the aforementioned local configuration information is determined as the transmission speed. Next, the compression configuration information and the transmission speed are determined as the transmission parameter configuration information. Then, a session identifier is generated using a high-precision timestamp (e.g., nanosecond level) combined with a random number. Finally, the system number and node number of the target client included in the aforementioned local configuration information are combined to obtain the target client identifier. For example, the session identifier could be 20260522143025678_12345. If the target client's system number is system_id=1001 and its node number is node_id=01, then the target client identifier could be client_id=1001-01. The session identifier allows the server to distinguish between different sessions and identify whether a new session from the same target client is occurring during an abnormal reconnection, facilitating server management of session resources. The compression flag mentioned above can be a Boolean configuration parameter used to control whether the target client allows compression operations on incremental log data packets during this synchronization session. The candidate compression algorithm mentioned above can be a string or a string array configuration parameter used to specify the name of the compression algorithm or a preferred list that the target client expects to use.

[0026] The third step is to determine the aforementioned local historical synchronization site information as synchronization site information.

[0027] The fourth step involves structurally encapsulating the aforementioned local configuration information, server-side parameter information, target client identifier, transmission parameter configuration information, session identifier, and synchronization point information to obtain incremental synchronization request information. In practice, the target client can serialize the aforementioned local configuration information, server-side parameter information, target client identifier, transmission parameter configuration information, session identifier, and synchronization point information into a request message as incremental synchronization request information, according to the communication protocol agreed upon with the server (e.g., JSON, Protobuf application layer protocol).

[0028] In some embodiments, the server 102 described above can be configured to perform the following incremental streaming push steps: First, in response to receiving the incremental synchronization request information sent by the target client, the incremental connection point location processing is performed on the synchronization point information included in the incremental synchronization request information to obtain the log file connection point information.

[0029] In some embodiments, the server 102 may, in response to receiving an incremental synchronization request from the target client, perform incremental join point location processing on the synchronization point information included in the incremental synchronization request information to obtain log file join point information. The synchronization point information includes a log file identifier, logical data sequence number, and file byte offset; the incremental synchronization request information includes the transaction date. The server may be a software service module deployed on a data source node (e.g., a business database host, a message queue producer cluster, or a log generation server).

[0030] In addressing the technical challenges mentioned above, the application scenario—massive data synchronization and disaster recovery (e.g., data center power outages, fiber optic cable severance, widespread hardware failures, ransomware attacks)—often presents the following technical issues: Due to network interruptions, system upgrades, or initial synchronization, the log file identifiers and file byte offsets stored in the local location file on the target client (disaster recovery end) become unusable (e.g., after a software upgrade, the new client version may change the storage format of the local location file (e.g., from binary to JSON, or changes in field meanings), rendering the old log file identifiers and file byte offsets unusable). This prevents the target client from recovering valid log file identifiers and file byte offsets from the old location file, thus hindering its ability to inform the server of the physical location of the last synchronization interruption. The server resets the local location to the starting position, resulting in the repeated transmission of large amounts of already transmitted data, wasting computational and network resources. This application scenario requires real-time, massive data volume, and highly reliable data synchronization. Faced with these technical challenges, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the server 102 can be configured to respond to receiving incremental synchronization request information sent by the target client by performing incremental connection point location processing on the synchronization point information included in the incremental synchronization request information to obtain log file connection point information: The first step is to obtain the root directory of the log files. In practice, the server can first load its local configuration file at startup. Then, a configuration parsing module (such as a JSON parser) is called to read the key-value pairs in the local configuration file and store them as the root directory string in the server's runtime context. Finally, the root directory string is determined as the root directory of the log files. The local configuration file can be a structured parameter file pre-stored on persistent storage media (e.g., disk) on the server's host. This local configuration file records the static configuration parameters necessary for server startup and operation. It is organized in key-value pairs (e.g., log.root.dir= / data / qlo). The local configuration file may include, but is not limited to, the server address and port, the root directory of the log files, candidate compression algorithms, rate limiting thresholds, buffer size, idle wait time, compression permission flags, and security compliance information. The compression permission flag can be a boolean configuration parameter stored in the server's local configuration file, used to control whether the server allows compression operations when transmitting incremental log data. The root directory for the log files mentioned above can be a top-level directory path, used to centrally store the QLog files corresponding to all data to be synchronized. For example, the root directory for the log files could be / var / qlog / .

[0031] The second step involves searching the root directory of the log files based on the transaction dates mentioned above to obtain the individual log files. In practice, the server can append the transaction dates to the root directory of the log files to form a complete log directory path. Then, the QLog files stored in the log directory path are identified as the individual log files. For example, the log directory path could be / var / qlog / 20260522 / .

[0032] The third step involves filtering the log files based on the aforementioned log file identifiers to obtain a candidate log file list. In practice, the server first determines the log file identifiers as the target identifiers. Then, files whose log file identifiers are not less than the target identifier are identified as candidate log files. Finally, the candidate log files are arranged in ascending order of their log file identifiers to obtain the candidate log file list. Each candidate log file in the candidate log file list includes transaction data. Each transaction data consists of a record header and a record body. The record body can be the actual business data payload stored. The record header is a fixed-length field area at the beginning of the transaction data record, describing the metadata of that record. The record header can include the data node type, data number, record length, and file offset. The data node type can be an integer or enumerated value, used to identify the business operation type corresponding to this record. For example, 0x01 represents an insert operation, 0x02 represents an update operation, 0x03 represents a delete operation, and 0x04 represents a system heartbeat or metadata record. The aforementioned data number can be a globally monotonically increasing integer, serving as the logical sequence number of this record. The server assigns a unique data number to each record when writing it. This data number is independent of the record's physical storage location in the QLog file, but maintains a strictly monotonically increasing relationship. The aforementioned record length can be an integer, representing the total number of bytes occupied by each transaction (including the record header and record body). The aforementioned file offset can be an integer, representing the byte offset of the first byte of this record (i.e., the starting position of the record header) within the current QLog file. This byte offset can be implicitly obtained from the current position of the file read pointer, or explicitly written into the record header for random access.

[0033] The fourth step is to read the record header of each candidate log file in the candidate log file list above. Each record header information may include a data number.

[0034] The fifth step is to generate a record header sequence based on the obtained record headers. In practice, the server can sort the record headers according to the size of the data numbers included in each record header information to obtain the record header sequence.

[0035] Step 6: Perform the following steps on the above sequence of record header information: Sub-step one: Determine the first record header information in the above record header information sequence as the target record header information.

[0036] Sub-step two involves determining the logical data sequence number included in the above synchronization site information as the target number.

[0037] Sub-step three: Extract the data number from the target record header information as the data number to be matched.

[0038] Sub-step four: In response to determining that the number of the data to be matched is less than the target number, the target record header information is deleted from the record header information sequence to update the record header information sequence, and the above steps are executed again based on the updated record header information sequence.

[0039] Sub-step five: In response to determining that the data number to be matched is equal to the target number, log file join point information is generated based on the data number to be matched. In practice, firstly, the server can determine the record header corresponding to the data number to be matched as the target record header. Then, the file offset included in the target record header is determined as the file byte offset. Finally, the log file identifier corresponding to the candidate log file where the target record header is located and the file byte offset are determined as the log file join point information.

[0040] Sub-step six: In response to determining that the number of the data to be matched is greater than the target number, a backtracking and refined positioning process is performed on the record header information sequence to obtain the log file join point information. In practice, firstly, the server can determine the record header preceding the target record header in the record header list as the first target record header. Then, the data number included in the first target record header is determined as the first data to be matched number. Next, the difference between the data to be matched number and the first data to be matched number is determined as the first target value. Then, the product of the first target value and a fixed length is determined as the record offset increment. Then, the sum of the file offset included in the first target record header and the record offset increment is determined as the target position. Then, the target position is determined as the file byte offset. Finally, the log file identifier corresponding to the file to be matched where the first target record header is located and the file byte offset are determined as the log file join point information.

[0041] The above technical solution, combined with the second to third and related contents, serves as an inventive point of this disclosure, solving the technical problem of "data duplication, wasting computational and network resources." Factors leading to data duplication and wasted computational and network resources are often as follows: Due to network interruptions, system upgrades, or initial synchronization, the log file identifiers and file byte offsets stored in the local location file of the target client (disaster recovery end) become unusable (for example, when the target client software version is upgraded, the new client version may change the storage format of the local location file (e.g., from binary format to JSON format, or the meaning of fields changes), causing the old log file identifiers and file byte offsets to become unusable). This prevents the target client from recovering valid log file identifiers and file byte offsets from the old location file, thus preventing it from informing the server of the physical location of the last synchronization interruption. The server resets the local location to the starting position, resulting in a large amount of already transmitted data being repeatedly transmitted, wasting computational and network resources. Solving these factors can reduce the waste of computational and network resources. To achieve this effect, firstly, the root directory of the log files is obtained. Then, based on the aforementioned transaction dates, the root directory of the log files is searched to obtain the various log files. Next, based on the aforementioned log file identifiers, the log files are filtered to obtain a candidate log file list. This reduces the search space from all historical log files (potentially containing months or even years of data) to a few consecutive files from a specific transaction date (usually involving only a few files). This avoids unnecessary scanning of invalid files by the server, reducing disk I / O overhead and search time, and lowering computational resource consumption. Then, the record header of each candidate log file in the candidate log file list is read. Each record header includes a data number. This allows for the rapid acquisition of the data number (data_id) and its corresponding physical offset (file_pos) of each operation record, solely through record header parsing, without loading the complete record body (i.e., without reading business data). This eliminates the need for repeated reading of large record bodies in subsequent location processes, reducing CPU and memory overhead per location operation. Then, based on the obtained record headers, a record header sequence is generated. Next, the following steps are performed on the record header information sequence: First, the first record header in the sequence is identified as the target record header. Then, the logical data sequence number included in the above synchronization site information is determined as the target number. Next, the data number in the target record header information is extracted as the data number to be matched. Afterwards, in response to determining that the data number to be matched is less than the target number, the target record header information is deleted from the record header information sequence to update the record header information sequence, and based on the updated record header information sequence, the above steps are performed again.Next, in response to determining that the data number to be matched is equal to the target number, log file join point information is generated based on the data number to be matched. Finally, in response to determining that the data number to be matched is greater than the target number, the sequence of record header information is refined and backtracked to obtain the log file join point information. This allows handling situations where the target number is not present in the record header due to discontinuous target numbers, cross-file issues, or version differences. It ensures that even if physically discontinuous, the correct offset closest to the target position can still be returned. Because, combining the second and third steps, the above process does not rely on the format or storage content of any external site file, but directly derives the breakpoint position from the structured metadata (data number in the record header) stored in the log file itself. Therefore, even if the client's site file is lost or unavailable due to upgrades, corruption, or format changes, the server can still relocate based on the data number provided by the client, thus avoiding resetting the site to the starting position (file_id=0). This further avoids the repeated fetching and processing of large amounts of previously consumed historical data, reducing redundant data transmission and wasting computational and network resources.

[0042] Second, based on the log file join point information, read the log incremental data packets.

[0043] In some embodiments, the server 102 can read incremental log data packets based on the log file join point information. The log file join point information may include a log file identifier, logical data sequence number, and file byte offset. Each of the QLog files can be a sequentially written log file generated by the server on its local disk for persistently recording each state change of the queue.

[0044] In some optional embodiments, the server 102 can read incremental log data packets based on the log file join point information. In practice, the server can locate the corresponding QLog file based on the log file identifier in the log file join point information, directly move the current read pointer to the byte offset indicated by the file byte offset through the file system's seek interface, and then start sequentially reading data to obtain incremental log data packets. The incremental log data packets can be incremental pipeline record encapsulation units read by the server from the current position of the located QLog file. The incremental log data packets can contain one or more consecutive records. Each consecutive record includes at least a file number, file offset, logical data sequence number, data node type, record length, record body, and verification information. The verification information can be a piece of redundant data appended to the header of each record for verifying data integrity. The core purpose of the verification information is to ensure that the target client, after receiving the data, can confirm that the record has not been damaged or tampered with during transmission, and that no silent data corruption has occurred during disk storage.

[0045] Third, based on the transmission parameter configuration information included in the incremental synchronization request information, the log incremental data packets are compressed and streamed for push processing to send the log incremental data packets to the target client.

[0046] In some embodiments, the server 102 may perform compressed streaming push processing on the incremental log data packets based on the transmission parameter configuration information included in the incremental synchronization request information, so as to send the incremental log data packets to the target client. The transmission parameter configuration information may include compression configuration information and transmission speed.

[0047] In some optional implementations of certain embodiments, the server 102 can be configured to perform compressed streaming push processing on the incremental log data packets according to the transmission parameter configuration information included in the incremental synchronization request information, in order to send the incremental log data packets to the target client: The first step is to dynamically compress the log incremental data packets according to the compression configuration information included in the above transmission parameter configuration information to obtain compressed log incremental data packets.

[0048] The second step is to send the compressed log incremental data packets to the target client based on the aforementioned transmission speed. In practice, the server can send the compressed incremental data packets to the target client at the aforementioned transmission speed.

[0049] In addressing the technical challenges of the aforementioned background technologies, the application scenario—a cross-data center or cross-regional data synchronization link, such as the real-time synchronization of incremental operation logs (QLog) of transaction queues (que) between the primary data center and the disaster recovery center in a financial transaction system—often presents the following technical problems: Sending incremental operation logs to the target client at a fixed sending rate cannot adapt to the real-time fluctuations in cross-data center link bandwidth. During network congestion, the fixed high rate exacerbates packet loss and retransmissions, leading to a decrease in effective throughput and increased synchronization latency. Conversely, when the network is idle, the fixed low rate fails to fully utilize available bandwidth, resulting in resource waste and extended data catch-up time. Furthermore, since cross-data center links typically carry both business transaction data and synchronization replication data simultaneously, during sudden surges in business traffic, synchronization data may accumulate for extended periods due to bandwidth constraints, causing severe data lag at the target end and resulting in the loss of a large amount of transaction data during disaster recovery switchover. This leads to low network resource utilization and bandwidth waste. Therefore, this application scenario requires the following characteristics: real-time, efficient, and low-latency data synchronization. Faced with the above technical problems, we decided to adopt the following solution: Optionally, the aforementioned execution entity may also perform compressed streaming push processing on the aforementioned log incremental data packets according to the transmission parameter configuration information included in the aforementioned incremental synchronization request information, in order to send the aforementioned log incremental data packets to the aforementioned target client: The first step is to compress the log incremental data packets according to the compression configuration information included in the above transmission parameter configuration information to obtain compressed log incremental data packets.

[0050] The second step involves periodically collecting network status information to obtain a network status information stack. This network status information may include, but is not limited to, the total number of TCP segments sent, the total number of retransmitted segments, round-trip time (RTT), congestion window size (cwnd, in segments), and maximum segment size (MSS, in bytes). In practice, the server can first periodically obtain statistics about the current TCP connection by calling the socket interface provided by the operating system (e.g., the getsockopt function in Linux combined with the TCP_INFO option). Then, the statistics about the current TCP connection are used as network status information. Finally, this network status information is stored in the network status information stack. The statistics about the current TCP connection include, but are not limited to, the total number of TCP segments sent, the total number of retransmitted segments, round-trip time (RTT), congestion window size (cwnd, in segments), and maximum segment size (MSS, in bytes).

[0051] The third step is to extract the first and second network state information from the above network state information stack as the first target state information and the second target state information.

[0052] The fourth step involves processing the first and second target status information to generate the packet loss rate. In practice, firstly, the server can determine the total number of cumulatively sent TCP segments included in the first target status information as the first target total number of segments. Then, it determines the total number of cumulatively sent TCP segments included in the second target status information as the second target total number of segments. Next, the difference between the first and second target total number of segments is determined as the total number of segments sent in the current period. Then, the total number of cumulatively retransmitted segments included in the first target status information is determined as the first target retransmitted segment total number. Then, the total number of cumulatively retransmitted segments included in the second target status information is determined as the second target retransmitted segment total number. Then, the difference between the first and second target retransmitted segment total number is determined as the retransmission increment in the current period. Finally, the retransmission increment and the total number of segments sent in the current period are used to determine the packet loss rate.

[0053] The fifth step involves generating available bandwidth information based on the congestion window size, maximum segment length, and round-trip time included in the first target state information. In practice, the server can first determine the target value as the product of the congestion window size and the maximum segment length. Then, the ratio of the target value to the round-trip time is determined as the available bandwidth information. This available bandwidth information reflects the maximum theoretical throughput that the current network path can achieve without exacerbating congestion.

[0054] Step 6: Perform network congestion detection processing on the aforementioned packet loss rate information and the aforementioned first target state information to obtain network congestion information. In practice, firstly, the server can, in response to determining that the aforementioned packet loss rate information and the aforementioned first target state information satisfy one of the preset conditions, determine the preset information representing network congestion as network congestion information. Then, in response to determining that neither the aforementioned packet loss rate information nor the aforementioned first target state information satisfies one of the preset conditions, determine the preset information representing network non-congestion as network congestion information. The aforementioned preset conditions include: the packet loss rate information being greater than a preset packet loss rate threshold; the congestion window size included in the first target state information being less than a preset congestion window threshold; and the round-trip time included in the first target state information being greater than a preset round-trip time threshold. The aforementioned preset packet loss rate threshold can be a pre-set value used to quantitatively determine the packet loss rate information to determine whether the current network is in a congested state. The aforementioned preset congestion window threshold can be a pre-set value used to quantitatively determine the congestion window size to determine whether the current network is in a congested state. The aforementioned preset round-trip time threshold can be a pre-defined value used to quantitatively determine whether the network is currently congested. For example, the congestion window threshold can be 80% of the congestion window size included in the network status information of the previous period. The aforementioned preset round-trip time threshold can be 1.5 times the historical average round-trip time. The aforementioned historical average round-trip time is calculated by using an exponentially weighted moving average of multiple collected round-trip time values.

[0055] Step 7: Based on the aforementioned network congestion information and available bandwidth information, adaptive rate limiting is performed on the transmission speed included in the aforementioned transmission parameter configuration information to obtain the target transmission speed. In practice, firstly, in response to determining that the aforementioned network congestion information indicates network congestion, the server determines the product of the aforementioned transmission speed and a first preset value as the target transmission speed to reduce the transmission speed. Then, in response to determining that the aforementioned network congestion information indicates network non-congestion and that the aforementioned available bandwidth information is greater than a preset value, the server determines the product of the aforementioned transmission speed and a second preset value as the target transmission speed to increase the transmission speed. Finally, in response to determining that the aforementioned network congestion information indicates network non-congestion and that the aforementioned available bandwidth information is less than or equal to a preset value, the server determines the aforementioned transmission speed as the target transmission speed. The first preset value can be a pre-set value used to reduce the transmission speed. The second preset value can be a pre-set value used to increase the transmission speed. The preset value can be a pre-set value used to determine whether the available bandwidth is higher than the transmission speed. For example, the first preset value can be 0.9. The second preset value can be 1.1. The first and second preset values ​​mentioned above can be adaptively adjusted according to network stability requirements (e.g., 0.8 / 1.15 for satellite links and 0.95 / 1.05 for local area networks). The preset values ​​can be set to 120% of the transmission speed.

[0056] Step 8: Based on the target transmission speed, the compressed log incremental data packets are sent using priority flow control to the target client. In practice, the server first checks if the operating system supports the IP_TOS (IPv4) or IPv6_TCLASS (IPv6) socket options. If supported, the DSCP flag value used in this synchronization session is determined according to the operation and maintenance policy configuration file. For example, a DSCP value of 46 can be configured, corresponding to Expedited Forwarding (EF) behavior; other values ​​such as 10 (corresponding to low-latency service) can also be configured. If no DSCP value is specified in the configuration file, 0 is used by default (best-effort service). Then, the socket option setting function (e.g., the setsockopt function) is called to set the DSCP flag value for the socket of the current TCP connection. After the setting is completed, every IP packet sent from the socket will automatically carry the DSCP flag value in the Type of Service (ToS) or Traffic Class field of the IP header. Subsequently, IP packets carrying the aforementioned compressed log incremental data packets are sent to the network device according to the target transmission speed. The network device then parses the DSCP flag value in the IP packet header and maps it to the corresponding priority forwarding queue. This ensures that, in the event of network congestion, data packets in the priority forwarding queue are prioritized for delivery to the target client, guaranteeing the minimum bandwidth and low-latency forwarding of the compressed log incremental data packets even under fluctuating business traffic conditions. The network device is pre-configured with DSCP-based queue scheduling rules. The aforementioned operation and maintenance policy configuration file can be a structured configuration file (such as JSON, YAML, XML, or key-value pair format) stored on the server's local disk (or in a remote configuration center). It is used to centrally manage operation and maintenance-level parameters such as network transmission information sets, security compliance information sets, and priority control information sets involved in the operation of the QLog synchronization service. This configuration file is the main interface for operation and maintenance personnel to strategically adjust the synchronization service. Its core function is to decouple policy decisions from program code, allowing operation and maintenance personnel to flexibly adjust synchronization behavior to adapt to different network environments and business needs without modifying or recompiling the code. The aforementioned network transmission information set can be a set of rules and constraints for controlling the behavior of QLog incremental data during cross-data center network transmission. Its core function is to balance transmission efficiency, bandwidth utilization and system resource consumption, and ensure that data can be transmitted from the server to the target client in the best way.The aforementioned security and compliance information set can be a set of rules used to control the security behavior, data protection, and compliance checks of incremental log data during the synchronization process. Its function is to ensure that when server-side transaction nodes transmit incremental log data externally, they meet internal enterprise security standards, industry regulatory requirements (e.g., financial industry data security standards), and legal compliance requirements for data export. The aforementioned priority control information set can be a set of configuration rules used to set network transmission priorities for incremental log data and ensure it receives differentiated quality of service on the shared link. Its function is to ensure that synchronized data still receives minimum bandwidth guarantees and low-latency forwarding during business traffic surges, avoiding severe data lag on the target client side due to bandwidth congestion. The aforementioned DSCP-based queue scheduling rules can allocate an independent priority forwarding queue to the data stream corresponding to a DSCP value of 46 and reserve a certain proportion of minimum bandwidth guarantees (e.g., 10% of the total link bandwidth), or configure it as a strict priority queue. For example, the aforementioned network devices can be intermediate network devices such as network switches and routers.

[0057] The above-mentioned technical solution and related content, as an inventive point of this disclosure, solve the technical problem of "low network resource utilization and bandwidth waste." Factors leading to low network resource utilization and bandwidth waste often include: sending incremental operation logs to the target client at a fixed sending rate. Because of this fixed sending rate, it cannot adapt to the real-time fluctuations in bandwidth across data center links. When network congestion occurs, the fixed high rate exacerbates packet loss and retransmission, leading to a decrease in effective throughput and an increase in synchronization delay. When the network is idle, the fixed low rate cannot fully utilize available bandwidth, resulting in resource waste and extending data catch-up time. Simultaneously, since cross-data center links typically carry both business transaction data and synchronous replication data, when business traffic surges, synchronization data may accumulate for a long time due to bandwidth constraints, causing severe data lag at the target end and loss of a large amount of transaction data during disaster recovery switching. This results in low network resource utilization and bandwidth waste. Solving these factors can improve network resource utilization and reduce bandwidth waste. To achieve this effect, firstly, based on the compression configuration information included in the above transmission parameter configuration information, the above-mentioned log incremental data packets are compressed to obtain compressed log incremental data packets. Therefore, the aforementioned incremental log data packets can be compressed to maximize the transmission efficiency of incremental data under limited and fluctuating cross-data center network bandwidth. Then, network status information is periodically collected to obtain a network status information stack. This stack includes round-trip time, congestion window size, and maximum segment length. This provides a network status information stack for determining network congestion. Next, the first and second network status information in the stack are extracted as the first and second target status information. Then, packet loss rate generation processing is performed on the first and second target status information to obtain packet loss rate information. This provides the packet loss rate information for the current time used to determine network congestion. Then, based on the congestion window size, maximum segment length, and round-trip time included in the first target status information, available bandwidth information is generated. This provides available bandwidth information reflecting the maximum theoretical throughput achievable by the current network path without exacerbating congestion. Finally, network congestion detection processing is performed on the packet loss rate information and the first target status information to obtain network congestion information. Therefore, the current network congestion situation can be determined based on the packet loss rate and network status information. Next, based on the aforementioned network congestion information and available bandwidth information, adaptive rate limiting is applied to the transmission speed included in the aforementioned transmission parameter configuration information to obtain the target transmission speed. Thus, the log incremental data packet sending rate can be dynamically adjusted according to network congestion and available bandwidth information. This allows for proactively reducing the rate to alleviate congestion and reduce packet loss during network congestion, and proactively increasing the rate to fully utilize bandwidth resources during network idle periods, thereby improving synchronization efficiency while ensuring transmission reliability.Finally, based on the aforementioned target transmission speed, the compressed log incremental data packets are sent using priority flow control to the target client. This allows for differentiated forwarding services at the network layer by setting DSCP network priority tags on the compressed log incremental data packet stream. Even during periods of sudden traffic surges, minimum bandwidth guarantees are maintained, effectively preventing prolonged delays due to bandwidth congestion and ensuring the timeliness of target data during disaster recovery. Furthermore, by monitoring the network status of the TCP connection between the server and the target client in real time during the transmission of log incremental data packets, and dynamically adjusting the data transmission rate using adaptive rate limiting based on this information, the system can proactively reduce the rate to alleviate congestion and reduce packet loss during network congestion, and proactively increase the rate to fully utilize bandwidth resources during periods of network idleness. This ensures both transmission reliability and improved synchronization efficiency. Meanwhile, by setting DSCP network priority tags for incremental log data packets through priority flow control, these packets receive differentiated forwarding services at the network layer. This ensures minimum bandwidth guarantees even during periods of sudden business traffic surges, effectively preventing prolonged delays in synchronization data due to bandwidth congestion and guaranteeing the timeliness of target data during disaster recovery failover. This improves network resource utilization and reduces bandwidth waste.

[0058] In addressing the technical challenges mentioned above, the application scenario involves a cross-regional disaster recovery architecture. The data synchronization link between the source production center (e.g., Shanghai) and the off-site disaster recovery center (e.g., Guiyang) needs to simultaneously handle two tasks: real-time transaction log (QLog) synchronization and batch reconciliation data distribution. This often presents the following technical challenges: when transmitting incremental log data, the compression method used is typically determined by a single dimension (e.g., bandwidth or CPU usage). However, due to significant day-night bandwidth differences across regional networks (effective bandwidth drops by 40% during peak daytime hours and recovers to over 90% during off-peak hours) and day-night synchronization data differences (daytime QLog records online transaction logs (e.g., payments, transfers, account openings), generated in real-time by the business system with a highly templated structure; nighttime QLog records end-of-day batch reconciliation details (e.g., settlement documents, error records, batch status changes), generated by batch tasks with diverse fields and varying lengths), and the need for server-side CPU resources to prioritize the normal processing of transaction data, the system faces significant challenges. Judging the compression method used when transmitting incremental log data from a single dimension leads to a waste of computing resources. (For example, disabling compression simply because of high CPU load may waste bandwidth; enabling a high compression ratio algorithm simply because bandwidth is sufficient may unnecessarily consume CPU.) This application scenario requires the following characteristics: effectively saving cross-regional bandwidth resources, improving disaster recovery synchronization timeliness (shortening catch-up time), reducing unnecessary CPU consumption on the server side during data synchronization, and ensuring the normal operation of transaction services. Faced with the above technical challenges, we have decided to adopt the following solution: Optionally, the server 102 can dynamically compress the log incremental data packet according to the compression configuration information included in the transmission parameter configuration information through the following steps to obtain a compressed log incremental data packet: The first step is to obtain load status information, actual throughput, round-trip time (RTT) sample sequence, and retransmission event count. In practice, the server can first read performance counters provided by the operating system (e.g., / proc / stat on Linux or GetSystemTimes on Windows) to calculate the percentage of non-idle time as CPU utilization at fixed sampling intervals. Then, this CPU utilization is used as load status information. Next, the server maintains a sliding time window (e.g., the last 10 seconds), and after sending each data block (or batch of records), the number of bytes sent is incremented by a counter within the window. Every fixed interval (e.g., 1 second), the current accumulated value is read and subtracted from the value of the previous second to obtain the instantaneous throughput as the actual throughput. Then, dedicated RTT probe messages (e.g., heartbeat messages carrying local timestamps) are periodically sent. Upon receiving the RTT probe messages, the target client immediately returns them as is (or returns the received timestamp). When the server receives the response, it calculates the difference (in milliseconds) between the current time and the timestamp in the request as the RTT sample. Next, the obtained round-trip time samples are stored in chronological order in a fixed-length circular buffer (e.g., retaining the most recent 100 samples), forming a round-trip time sample sequence. The actual throughput mentioned above can be the number of data bytes (or records) successfully transmitted per unit time. Then, notification events from the TCP protocol stack can be subscribed to via Netlink sockets. When a retransmission event triggered by a timeout retransmission (RTO) or fast retransmit is detected, the corresponding counter is incremented. Finally, the value of the counter is determined as the number of retransmission events.

[0059] The second step involves performing compression effect detection processing on the aforementioned incremental log data packets to obtain compression effect information. In practice, firstly, the server can parse the record length field of each record within the incremental log data packet, summing the record lengths of all records to obtain the actual byte length of the data packet to be sent. Then, in response to determining that the actual byte length is greater than or equal to a preset compression effect threshold, the preset information indicating suitability for compression is identified as compression effect information. Subsequently, in response to determining that the actual byte length is less than the preset compression effect threshold, the preset information indicating unsuitability for compression is identified as compression effect information. The preset compression effect threshold can be a pre-set value used to determine whether the network transmission time saved by the compression operation can typically offset the additional CPU computing power consumed by compression. The preset compression effect threshold can be statically configured based on a comparison of network bandwidth and CPU performance in the actual business scenario, or dynamically adjusted based on historical statistics.

[0060] The third step is to generate the bandwidth utilization rate based on the actual throughput mentioned above. In practice, firstly, the server can call the GetIfEntry2API function to obtain the IF_ENTRY_TABLE2 structure through the interface's LUID (Local Unique Identifier). The TransmitLinkSpeed ​​and ReceiveLinkSpeed ​​fields in the IF_ENTRY_TABLE2 structure return the negotiated rate of the current link in bits per second (bps), which is used as the theoretical maximum bandwidth limit of the network interface. Then, the ratio of the actual throughput to the theoretical maximum bandwidth limit of the network interface is used as the bandwidth utilization rate.

[0061] The fourth step involves performing congestion detection processing on the aforementioned round-trip time sample sequence and the number of retransmission events to obtain network congestion status information. In practice, firstly, the server can set a sliding window of length N in the aforementioned round-trip event sample sequence, and select the minimum value of the round-trip event samples within the sliding window as the ideal latency benchmark for the current link. Then, starting from the most recent successful request... The current round-trip time sample is extracted from the response interaction. Then, the difference between the current round-trip time sample and the ideal latency benchmark of the current link is used as the latency offset. If the latency offset is greater than a preset latency inflation threshold, the network congestion status information is determined to be congested. Then, if the latency offset is less than the preset latency inflation threshold, the network congestion status information is determined to be normal. If the number of retransmission events is greater than a preset retransmission threshold, the network congestion status information is determined to be congested. If the number of retransmission events is less than or equal to the preset retransmission threshold, the network congestion status information is determined to be normal. The preset latency inflation threshold can be a pre-configured benchmark parameter used to determine whether a network link is congested. N can be a pre-set length. The preset retransmission threshold can be a pre-configured value used to evaluate whether retransmission events occurring per unit time are abnormally dense. The round-trip time sample can be the total time elapsed for the server during a complete request-response interaction, from sending a request message to receiving the corresponding response message, typically in milliseconds (ms). The above time consumption reflects the total latency experienced by the data packet during its round trip between the server and the target client. The sliding window described above can be used to store samples of the round-trip times of the N most recent successful request-response interactions.

[0062] Fifth, based on the bandwidth utilization rate and the round-trip time sample sequence, generate bandwidth status information. In practice, firstly, the server can, in response to determining that the bandwidth utilization rate is less than or equal to a preset utilization threshold, perform the following steps: First, set a sliding window mechanism for the round-trip time sample sequence, and extract the minimum value from the sliding window (e.g., the most recent 30 round-trip time samples) as the idle baseline delay of the current link. Then, from the most recent successful request... The current round-trip time (RTT) sample is extracted from the response interaction. Next, the difference between the idle baseline delay of the current link and the current RTT sample is determined as the RTT offset. Then, in response to determining that the RTT offset is less than a preset stable delay threshold (e.g., less than 20% of the idle baseline delay of the current link), and that there have been no packet loss events in the most recent several windows (i.e., zero or very low retransmission count), the bandwidth status information is determined to be sufficient. Finally, in response to determining that the RTT offset is greater than or equal to the preset stable delay threshold, the bandwidth status information is determined to be moderate. Then, in response to determining that the bandwidth utilization is greater than a preset utilization threshold, the bandwidth status information is determined to be strained. The preset utilization threshold can be a pre-set value used to determine whether the network transmission capacity has reached saturation. The preset stable delay threshold can be a preset offset parameter (which can be an absolute time value or a relative proportion value) used to determine whether the offset of the current RTT relative to the historical minimum RTT is stable.

[0063] Step 6: Perform network link policy detection processing on the aforementioned bandwidth utilization, network congestion status information, and bandwidth status information to obtain network link policy result information. In practice, firstly, the server can read the compression permission flag and security compliance information from its local configuration file. Then, in response to determining that the read security compliance information requires compression to be prohibited or that the compression permission flag is false, the network link policy result information is determined to prohibit compression. Next, in response to determining that the read security compliance information does not require compression to be prohibited or that the compression permission flag is true, the following steps are executed: Firstly, in response to determining that the network congestion status information is congested, the network link policy result information is determined to prohibit compression. Then, in response to determining that the network congestion status information is normal, the following steps are executed: Firstly, in response to determining that the bandwidth status information is sufficient, the network link policy result information is determined to have sufficient bandwidth and does not require compression. In response to determining that the bandwidth status information is moderate or strained, the network link policy result information is determined to allow compression. The aforementioned security compliance information can be a set of predefined security rules and configuration requirements on the server side, used to control data security behavior and compliance checks during QLog synchronization. Its core function is to ensure that source transaction nodes, when transmitting incremental QLog data, meet internal enterprise security standards, industry regulatory requirements (such as financial industry data security standards), and legal compliance requirements for data export. This security compliance information includes whether plaintext transmission is mandatory (for example, certain financial or government scenarios prohibit compression of specific data to prevent compression side-channel attacks). The compression permission flag can be a configuration parameter on the server side used to control whether compression processing is allowed for incremental log data packets during transmission. If set to false, the server rejects all compression requests and directly forces plaintext transmission.

[0064] Step 7: Based on the network link policy result information, the load status information, and the compression effect information, the compression configuration information is adaptively updated to obtain the updated compression configuration information. In practice, firstly, the server can, in response to determining that the network link policy result information indicates compression is prohibited or that bandwidth is sufficient and compression is unnecessary, determine that the compression configuration information does not need compression, thereby updating the compression configuration information to obtain the updated compression configuration information. In response to determining that the network link policy result information indicates compression is allowed, the following steps are performed: Firstly, in response to determining that the load status information is greater than or equal to a preset threshold or that the compression effect information indicates that compression is not suitable, the compression configuration information is determined to not need compression, thereby updating the compression configuration information to obtain the updated compression configuration information. Then, in response to determining that the load status information is less than a preset threshold or that the compression effect information indicates that compression is suitable, the following steps are performed: Firstly, the list of locally supported preset compression algorithms is queried to determine whether the compression configuration information exists in the compression algorithm list. Then, in response to determining that the compression configuration information exists in the compression algorithm list, the compression configuration information is determined as the updated compression configuration information. Next, in response to the determination that the aforementioned compression configuration information does not exist in the aforementioned compression algorithm list and that the aforementioned target client allows compression downgrading, a standard compression algorithm is selected from the preset compression algorithm list as the updated compression configuration information. The aforementioned standard compression algorithm is an algorithm that is enabled by default on the server, has moderate computational overhead, and a stable compression ratio (e.g., the fast level of ZSTD or the low compression level of GZIP). The aforementioned preset compression algorithm list can be a data structure maintained by the server during startup or operation, used to record the set of all compression algorithms that the server can currently recognize, load, and successfully invoke. The aforementioned preset threshold can be a pre-set value used to determine whether the load status information is suitable for compression.

[0065] Step 8: Based on the updated compression configuration information, the incremental log data packets are compressed to obtain compressed incremental log data packets. In practice, the server calls the compression encoder corresponding to the updated compression configuration information to compress the incremental log data packets and includes the corresponding compression algorithm identifier in the packet header for decompression by the target client. The compression algorithm identifier is a specific field written by the server in the header of the incremental log data packet after compression, used to announce the specific compression algorithm type used in this data packet to the target client. The compression algorithm identifier is represented by a predefined enumeration value or a fixed-length numerical code. For example, a compression algorithm identifier of 0x00 indicates no compression, and a compression algorithm identifier of 0x01 indicates LZ4 compression. The compression encoder can be a software module or program component in the server that converts the incremental log data packets (plaintext format) into a compressed binary data stream according to a specific compression algorithm. Its core function is to compress the large original log records into smaller data packets, thereby reducing the amount of data transmitted across data center networks and saving bandwidth resources.

[0066] The above-described technical solution, combined with its related content, serves as an inventive point of this disclosure, addressing the technical problem of "waste of computing resources." Factors leading to this waste of computing resources often include: when transmitting incremental log data, the compression method used for data transmission is typically judged by a single dimension (e.g., bandwidth or CPU alone). However, due to significant day-night bandwidth differences across geographically distributed networks (effective bandwidth drops by 40% during peak daytime business periods and recovers to over 90% during the early morning off-peak period) and day-night synchronization data differences (daytime QLog records online transaction flows (such as payments, transfers, and account openings), generated in real-time by the business system, with a highly templated structure; nighttime: QLog records end-of-day batch reconciliation details (such as settlement documents, error records, and batch status changes), generated by batch tasks, with diverse fields and varying lengths), and the need for server-side CPU resources to prioritize the normal processing of transaction business, judging the compression method used for transmitting incremental log data by a single dimension leads to a waste of computing resources. (For example, disabling compression solely due to high CPU load may waste bandwidth; enabling high compression ratio algorithms solely due to sufficient bandwidth may unnecessarily consume CPU.) Addressing these factors can reduce wasted computing resources. To achieve this, firstly, load status information, actual throughput, round-trip time sample sequences, and retransmission event counts are obtained. This provides multi-dimensional, real-time data support for subsequent congestion detection, bandwidth assessment, and compression decisions. Then, the compression effect of the incremental log data packets is detected to obtain compression effect information. This allows for the assessment of the expected benefits of compression (such as estimated compression ratio and compression time), avoiding compression operations when the benefits are extremely low (e.g., compression ratio < 1.2) or when the compressed file size actually increases. Next, bandwidth utilization is generated based on the actual throughput. This provides the actual network bandwidth occupancy, offering a basis for determining whether there is surplus bandwidth available for transmitting uncompressed data. Finally, congestion detection is performed on the round-trip time sample sequences and retransmission event counts to obtain network congestion status information. Therefore, it is possible to determine whether the current network is congested, providing a decision signal for the compression strategy regarding whether to prioritize bandwidth saving. Then, based on the bandwidth utilization and round-trip time sample sequence, bandwidth status information is generated. Thus, a comprehensive link quality label can be generated by combining two dimensions (bandwidth utilization and round-trip time), obtaining bandwidth status information describing the available bandwidth margin and stability trend of the current link. Next, network link policy detection processing is performed on the bandwidth utilization, network congestion status information, and bandwidth status information to obtain network link policy result information. Therefore, the bandwidth utilization, network congestion status information, and bandwidth status information can be comprehensively integrated and logically fused to obtain network link policy result information.This avoids resource waste caused by enabling high compression ratio algorithms solely based on sufficient bandwidth or disabling compression solely due to high CPU load. Then, based on the network link policy results, load status information, and compression effect information, the compression configuration information is adaptively updated to obtain the updated compression configuration information. Thus, by comprehensively balancing network status (needing to save bandwidth), server status (needing to save CPU), and compression effect (worth compressing), an optimal compression configuration information is obtained, ensuring that each compression decision adapts to the current network and load status. Finally, based on the updated compression configuration information, the incremental log data packets are compressed to obtain compressed incremental log data packets. This is because compression effect detection, congestion detection, and bandwidth status analysis generate quantitative information across various dimensions. Then, through network link policy detection and adaptive compression configuration updates, multi-dimensional information is comprehensively judged to generate the compression configuration information. This ensures that the compression method used when transmitting incremental log data is always based on multi-dimensional real-time data, rather than a single-dimensional judgment or static configuration. This distinguishes between compression worthwhile and unworthwhile scenarios, avoiding CPU consumption by performing compression operations when the compression benefit is extremely low, and preventing bandwidth waste due to disabling compression during network congestion. It achieves adaptation of the compression strategy to real-time scenarios, effectively reducing the consumption of computing resources.

[0067] In some embodiments, the target client 101 described above may be further configured to perform the following steps: First, in response to receiving the log incremental data packet, the log incremental data packet is stored in the local queuing machine based on the transmission parameter configuration information.

[0068] In some embodiments, the target client may, in response to receiving the log incremental data packet, store the log incremental data packet in a local queuing machine based on the transmission parameter configuration information.

[0069] In some optional implementations of certain embodiments, the target client 101 may, in response to receiving the log incremental data packet, store the log incremental data packet in a local queuing machine based on the transmission parameter configuration information by following these steps: The first step, in response to receiving the log incremental data packet sent by the aforementioned server, is to decompress the log incremental data packet based on the aforementioned transmission parameters to obtain a decompressed data packet. This decompressed data packet may include various transaction data. In practice, the target client can perform corresponding decompression processing on the log incremental data packet according to the compression configuration information included in the aforementioned transmission parameters to obtain the decompressed data packet.

[0070] The second step involves performing integrity verification on the decompressed data packets to obtain a list of verification record information. Each verification record in this list corresponds to one transaction data item from the various transaction data sets. In practice, the target client first reads the verification information field (e.g., CRC32 checksum, Adler-32, or a custom XOR checksum) stored in the header of each transaction data record in the decompressed data packets. Simultaneously, it recalculates the verification value of the record body (i.e., the actual business load data) using the same verification algorithm used to generate the aforementioned verification information field, obtaining the target verification value. Then, the target verification value is compared with the verification information field stored in the record header to obtain comparison result information. Finally, the obtained comparison result information is used to determine the verification record information list. The verification result information can be either the target verification value and the verification value stored in the record header are the same, or the target verification value and the verification value stored in the record header are different.

[0071] The third step involves performing the following steps on each verification record in the above verification record information list: In sub-step one, in response to determining that the above verification record information meets the preset verification conditions, the transaction data corresponding to the above verification record information is stored in the local queuing machine. The preset conditions may be that the target verification value is the same as the verification value stored in the record header.

[0072] Sub-step two: In response to the determination that the above verification record information does not meet the preset verification conditions, the transaction data corresponding to the above verification record information is retransmitted to obtain the retransmitted transaction data, and the retransmitted transaction data is stored in the local queuing machine. In practice, firstly, the client can identify the transaction data corresponding to the above verification record information as the target transaction data. Then, a retransmission request including the log file identifier and file byte offset of the target transaction data is constructed. Afterwards, the retransmission request is sent to the server through the established synchronization connection, so that after receiving the retransmission request, the server can open the corresponding QLog file according to the log file identifier, locate the offset position indicated by the file byte offset, read the record, and re-push it to the target client. The local queuing machine can be a data buffer queue implemented based on a shared memory mechanism, used to efficiently transfer incremental data records in a multi-process or multi-threaded environment. The data structure of the local queuing machine includes, but is not limited to, a circular buffer, a linked queue, or a concurrent queue implemented based on a lock-free algorithm. The capacity of the queuing machine is determined by the preset shared memory size. When the queue reaches its capacity limit, write operations can choose to block and wait, discard old data, or trigger an overflow alarm. The specific strategy depends on the system configuration.

[0073] Second, the synchronization site information is updated to obtain the updated synchronization site information, and the updated synchronization site information is written to the local synchronization site file for persistent storage.

[0074] In some embodiments, the target client 101 can update the synchronization point information to obtain updated synchronization point information, and write the updated synchronization point information to a local synchronization point file for persistent storage. In practice, firstly, the target client can determine the updated synchronization point information as the log file identifier, logical data sequence number, and file byte offset of the last data entry in the log incremental data packet. Then, the updated synchronization point information is written to a local synchronization point file for persistent storage. Optionally, the target client 101 may also perform the following steps: The first step, in response to the detected communication connection anomaly, is to extract the interrupted save position information from the aforementioned local synchronization position file. This interrupted save position information refers to the synchronization position information corresponding to the last transaction data successfully written to the local synchronization position file and persisted before the communication connection was interrupted. This information includes the log file identifier, logical data sequence number, and file byte offset. This interrupted save position information serves as the starting point for the client to initiate a resume request to the server after the connection is restored.

[0075] The second step is to generate incremental synchronization request information based on the aforementioned interrupted saved position information. It should be noted that the method used to generate incremental synchronization request information based on the aforementioned intermediate saved position information is the same as the method used to generate incremental synchronization request information including transmission parameter configuration information and synchronization position information based on the aforementioned local configuration information and the aforementioned local historical synchronization position information.

[0076] Third, in response to the confirmation that the communication connection has been restored, the incremental synchronization request information is sent to the server so that the server can execute the incremental streaming push step again based on the incremental synchronization request information.

[0077] The above embodiments of this disclosure have the following beneficial effects: the pre-synchronization system for transaction node data based on a queuing machine in some embodiments of this disclosure reduces the waste of transmission resources. Specifically, the waste of transmission resources is caused by the fact that the source continuously pushes QLog data blocks to the target. When the target receives the QLog data block, it only records the currently received position information (including log file identifier and file byte offset) in memory. Once the target process crashes or the network is interrupted, the position information temporarily stored in memory will be lost, resulting in the target being unable to provide accurate synchronization progress after the connection is restored. As a result, the source cannot know the position where the target has successfully received the data after the connection is restored, and can only start data transmission from the moment the session is established. This leads to the repeated transmission of a large amount of data, resulting in the waste of transmission resources. Based on this, the pre-synchronization system for transaction node data based on a queuing machine in some embodiments of this disclosure includes: a server and a target client. The target client is communicatively connected to the server, wherein the target client is configured to perform the following steps: First, obtain local configuration information and local historical synchronization position information. This allows the target client to load the log file identifier, file byte offset, and logical data sequence number from the last successfully synchronized file in its local persistent storage. Even if the target client process crashes or the system restarts, the target client can still provide a starting point for subsequent breakpoint resumption, avoiding the need to retransmit data from scratch due to lost points. Then, based on the aforementioned local configuration information and the aforementioned local historical synchronization point information, an incremental synchronization request message including transmission parameter configuration information and synchronization point information is generated and sent to the aforementioned server. This allows the target client and the server to reach an agreement on the transmission strategy, ensuring that each step of subsequent data transmission is performed under mutually agreed-upon rules. This avoids transmission anomalies and duplicate requests caused by parameter mismatches, reducing retransmission overhead. The aforementioned server is configured to perform the following incremental streaming push steps: First, in response to receiving the incremental synchronization request message sent by the aforementioned target client, it performs incremental connection point location processing on the synchronization point information included in the incremental synchronization request message to obtain the log file connection point information. Therefore, the server can open the corresponding QLog file based on the log file identifier and file byte offset in the target client's request, convert the logical data sequence number into a physical read offset within the file, so that the server can jump to the first unsent record after the breakpoint, physically avoiding the retransmission of processed data from the file header or session start position. Then, based on the aforementioned log file join point information, the incremental log data packet is read. Finally, according to the transmission parameter configuration information included in the aforementioned incremental synchronization request information, the incremental log data packet is compressed and streamed for push, so as to send the incremental log data packet to the target client.Therefore, the server can continuously read incremental log data packets and send them to the target client according to the corresponding compression configuration information and rate limiting policy. The target client is further configured to perform the following steps: First, in response to receiving the incremental log data packets, it stores the incremental log data packets in a local queuing machine based on the transmission parameter configuration information. This allows the target client to decompress the received incremental log data packets sequentially and write them to the local queuing machine, completing the data synchronization. Then, the synchronization point information is updated to obtain the updated synchronization point information, and this updated synchronization point information is written to a local synchronization point file for persistent storage. This allows the target client to write the latest synchronized log file identifier, logical data sequence number, and file byte offset to the local synchronization point file after each batch of data is written, ensuring persistent storage. Even if the target client's process crashes or the network is interrupted, and the point information in memory is lost, it can be reloaded from the local synchronization point file after restarting, thus restoring the data synchronization progress to the state before the interruption. Also, because the target client sends incremental synchronization request information with synchronization point information, the server reads incremental log data packets based on the incremental synchronization request information. Then, the incremental log data packets are sent to the target client according to the corresponding compression configuration information and rate limiting policy. The target client stores the incremental log data packets and updates the synchronization point information. This synchronization point information is persistently stored, allowing the server to locate the last interrupted position using the synchronization point information provided by the target client after any anomaly recovery, and only push unconfirmed incremental data. This avoids retransmissions caused by lost or corrupted memory points, reducing the waste of transmission resources.

[0078] Figure 2 The flowchart 200 illustrates some embodiments of the queuing machine-based transaction node data pre-synchronization method for target clients included in the above-described queuing machine-based transaction node data pre-synchronization system according to this disclosure. This queuing machine-based transaction node data pre-synchronization method includes the following steps: Step 201: Obtain local configuration information and local historical synchronization point information.

[0079] In some embodiments, the execution entity of the pre-synchronization system for transaction node data based on a queuing machine (e.g., the target client of the pre-synchronization system for transaction node data based on a queuing machine) can obtain local configuration information and local historical synchronization point information.

[0080] Step 202: Based on local configuration information and local historical synchronization point information, generate incremental synchronization request information including transmission parameter configuration information and synchronization point information, and send the incremental synchronization request information to the server for the server to perform the traffic push step.

[0081] In some embodiments, the execution entity can generate incremental synchronization request information, including transmission parameter configuration information and synchronization point information, based on local configuration information and local historical synchronization point information, and send the incremental synchronization request information to the server for the server to perform traffic push steps. The server performs the following incremental streaming push steps: First, in response to receiving the incremental synchronization request information sent by the target client, it performs incremental connection point location processing on the synchronization point information included in the incremental synchronization request information to obtain log file connection point information. Then, based on the log file connection point information, it reads log incremental data packets. Finally, according to the transmission parameter configuration information included in the incremental synchronization request information, it performs compressed streaming push processing on the log incremental data packets to send them to the target client.

[0082] Step 203: In response to receiving the log incremental data packet, the log incremental data packet is stored in the local queuing machine based on the transmission parameter configuration information.

[0083] In some embodiments, the execution entity may, in response to receiving the log incremental data packet, store the log incremental data packet in a local queuing machine based on the transmission parameter configuration information.

[0084] Step 204: Update the synchronization site information to obtain the updated synchronization site information, and write the updated synchronization site information to a local synchronization site file for persistent storage.

[0085] In some embodiments, the execution entity may update the synchronization site information to obtain updated synchronization site information, and write the updated synchronization site information into a local synchronization site file for persistent storage.

[0086] The above embodiments of this disclosure have the following beneficial effects: the pre-synchronization system for transaction node data based on a queuing machine in some embodiments of this disclosure reduces the waste of transmission resources. Specifically, the waste of transmission resources is caused by the fact that the source continuously pushes QLog data blocks to the target. When the target receives the QLog data block, it only records the currently received position information (including log file identifier and file byte offset) in memory. Once the target process crashes or the network is interrupted, the position information temporarily stored in memory will be lost, resulting in the target being unable to provide accurate synchronization progress after the connection is restored. As a result, the source cannot know the position where the target has successfully received the data after the connection is restored, and can only start data transmission from the moment the session is established. This leads to the repeated transmission of a large amount of data, resulting in the waste of transmission resources. Based on this, the pre-synchronization system for transaction node data based on a queuing machine in some embodiments of this disclosure includes: a server and a target client. The target client is communicatively connected to the server, wherein the target client is configured to perform the following steps: First, obtain local configuration information and local historical synchronization position information. This allows the target client to load the log file identifier, file byte offset, and logical data sequence number from the last successfully synchronized log file in its local persistent storage. Even if the target client process crashes or the system restarts, the target client can still provide a starting point for subsequent breakpoint resumption, avoiding the need to retransmit data from scratch due to lost points. Then, based on the aforementioned local configuration information and local historical synchronization point information, an incremental synchronization request is generated, including transmission parameter configuration information and synchronization point information. This incremental synchronization request is then sent to the server, allowing the server to perform the following incremental streaming push steps: First, in response to receiving the incremental synchronization request from the target client, the server performs incremental join point location processing on the synchronization point information included in the incremental synchronization request to obtain the log file join point information. This allows the server to open the corresponding QLog file based on the log file identifier and file byte offset in the target client's request, converting the logical data sequence number into a physical read offset within the file. This allows the server to jump to the first unsent record after the breakpoint, physically avoiding the retransmission of processed data from the file header or session start position. Then, based on the aforementioned log file join point information, the server reads the incremental log data packets. Finally, based on the transmission parameter configuration information included in the incremental synchronization request information, the incremental log data packets are compressed and streamed for delivery to the target client. This ensures that the target client and server agree on the transmission strategy, guaranteeing that each step of subsequent data transmission follows mutually agreed-upon rules. It avoids transmission anomalies and duplicate requests caused by parameter mismatches, reducing retransmission overhead.Subsequently, in response to receiving the aforementioned log incremental data packets, the server stores these packets in the local queuing machine based on the aforementioned transmission parameter configuration information. This allows the target client to sequentially decompress the received log incremental data packets and write them to the local queuing machine, completing the current data synchronization. Then, the synchronization point information is updated to obtain the updated synchronization point information, which is written to the local synchronization point file for persistent storage. This allows the target client to write the latest synchronized log file identifier, logical data sequence number, and file byte offset to the local synchronization point file after each batch of data is written, ensuring persistent storage. Even if the target process crashes or the network is interrupted, resulting in the loss of point information in memory, the synchronization point information can be reloaded from the local synchronization point file upon restart, thus restoring the data synchronization progress to its pre-interruption state. Furthermore, because the target client sends incremental synchronization request information containing synchronization point information, the server reads the log incremental data packets based on the incremental synchronization request information. Then, the server sends the log incremental data packets to the target client according to the corresponding compression configuration information and rate limiting policy. The target client stores incremental log data packets and updates synchronization point information. This synchronization point information is persistently stored, allowing the server to locate the point of interruption after any anomaly recovery, using the synchronization point information provided by the target client, and only push unconfirmed incremental data. This avoids retransmissions caused by lost or corrupted memory points, reducing waste of transmission resources.

[0087] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of technical features, but should also cover other technical solutions formed by arbitrary combinations of technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A pre-emptive remote synchronization system for transaction node data based on a queuing machine, comprising a target client and a server, wherein: The target client communicates with the server, wherein the target client is configured to perform the following steps: Obtain local configuration information and local historical synchronization point information; Based on the local configuration information and the local historical synchronization point information, an incremental synchronization request information including transmission parameter configuration information and synchronization point information is generated, and the incremental synchronization request information is sent to the server. The server is configured to perform the following incremental streaming push steps: In response to receiving the incremental synchronization request information sent by the target client, the incremental connection point location processing is performed on the synchronization site information included in the incremental synchronization request information to obtain the log file connection point information; Based on the log file join point information, read the log incremental data packet; Based on the transmission parameter configuration information included in the incremental synchronization request information, the log incremental data packet is compressed and streamed for push processing, so as to send the log incremental data packet to the target client; The target client is further configured to perform the following steps: In response to receiving the log incremental data packet, the log incremental data packet is stored in the local queuing machine based on the transmission parameter configuration information; The synchronization site information is updated to obtain updated synchronization site information, and the updated synchronization site information is written to a local synchronization site file for persistent storage.

2. The pre-synchronization system for transaction node data based on a queuing machine according to claim 1, wherein, The target client is further configured to: Based on the communication connection, parameter query processing is performed on the server to obtain server parameter information; Based on the local configuration information, generate the target client identifier, session identifier, and transmission parameter configuration information; The local historical synchronization site information is determined as the synchronization site information; The local configuration information, the server parameter information, the target client identifier, the transmission parameter configuration information, the session identifier, and the synchronization point information are structured and encapsulated to obtain incremental synchronization request information.

3. The pre-synchronization system for transaction node data based on a queuing machine according to claim 1, wherein, The transmission parameter configuration information includes compression configuration information and transmission speed, and the server is further configured as follows: Based on the compression configuration information included in the transmission parameter configuration information, the log incremental data packet is dynamically compressed to obtain a compressed log incremental data packet. Based on the transmission speed, the compressed log incremental data packet is sent to the target client.

4. The pre-synchronization system for transaction node data based on a queuing machine according to claim 1, wherein, The target client is further configured to: In response to detecting an abnormal communication connection, the interrupted saved site information is extracted from the local synchronization site file; Based on the interruption-saved location information, an incremental synchronization request information is generated; In response to determining that the communication connection has been restored, the incremental synchronization request information is sent to the server so that the server can execute the incremental streaming push step again based on the incremental synchronization request information.

5. The pre-synchronization system for transaction node data based on a queuing machine according to claim 1, wherein, The target client is further configured to: In response to receiving the log incremental data packet sent by the server, the log incremental data packet is decompressed based on the transmission parameters to obtain a decompressed data packet, wherein the decompressed data packet includes various transaction data; The decompressed data packet is subjected to integrity verification processing to obtain a verification record information list, wherein each verification record information in the verification record information list corresponds to one transaction data in each transaction data; Perform the following steps for each verification record in the verification record information list: In response to determining that the verification record information meets the preset verification conditions, the transaction data corresponding to the verification record information is stored in the local queuing machine; In response to determining that the verification record information does not meet the preset verification conditions, the transaction data corresponding to the verification record information is retransmitted to obtain the retransmitted transaction data, and the retransmitted transaction data is stored in the local queuing machine.

6. A method for pre-synchronization of transaction node data based on a queuing machine, applied to the target client included in the pre-synchronization system for transaction node data based on a queuing machine as described in any one of claims 1-5, the method comprising: Obtain local configuration information and local historical synchronization point information; Based on the local configuration information and the local historical synchronization point information, an incremental synchronization request information including transmission parameter configuration information and synchronization point information is generated, and the incremental synchronization request information is sent to the server so that the server can perform an incremental streaming push step. The server performs the following incremental streaming push steps: in response to receiving the incremental synchronization request information sent by the target client, it performs incremental connection point location processing on the synchronization point information included in the incremental synchronization request information to obtain log file connection point information; based on the log file connection point information, it reads log incremental data packets; and according to the transmission parameter configuration information included in the incremental synchronization request information, it performs compressed streaming push processing on the log incremental data packets to send the log incremental data packets to the target client. In response to receiving the log incremental data packet, the log incremental data packet is stored in the local queuing machine based on the transmission parameter configuration information; The synchronization site information is updated to obtain updated synchronization site information, and the updated synchronization site information is written to a local synchronization site file for persistent storage.

Images