Data synchronization method, apparatus, device, storage medium, and computer program product

Abstract

This invention discloses a data synchronization method, apparatus, device, storage medium, and computer program product. The method, applied to a master node, includes: responding to a data object write request and generating a metadata sequence containing index information of each data segment; constructing a replication control flow based on this sequence to describe transmission order constraints and completion conditions; sending the metadata sequence to slave nodes; sending the corresponding data segments while satisfying the order constraints; and sending a synchronization completion flag when the transmission status meets the completion conditions. This invention combines the metadata sequence with the replication control flow, transforming the traditional content-based driving approach into a unified scheduling control based on metadata. Through pre-planning the transmission order and a judgment mechanism, it achieves structured control of the synchronization process, eliminating the need for overall processing of complete objects, significantly improving the execution efficiency and controllability of data synchronization, and reducing the uncertainty of handling complex objects.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data synchronization method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] With the development of distributed database technology, data storage and processing architectures based on multi-node collaboration have become the mainstream solution. In such database systems, data redundancy backup, fault recovery, and improved system availability are typically achieved through data synchronization mechanisms between master and slave nodes.

[0003] Existing data synchronization solutions typically rely on log replication, data snapshot transmission, or incremental synchronization mechanisms based on change records. In practice, the master node translates data object write operations into corresponding synchronization information and transmits it to the slave nodes over the network. The slave nodes then update their local data based on the received information. This type of solution achieves good synchronization results when handling data with simple structures.

[0004] However, with the increasing complexity of business data and the diversification of data object scale and structure, the scale of data content that needs to be processed during data synchronization has increased significantly. Against this backdrop, existing technologies have gradually revealed certain limitations in terms of data synchronization efficiency, resource consumption, and the stability of the synchronization process. For example, problems such as high data transmission overhead, increased synchronization latency, and increased difficulty in controlling the synchronization process in complex data scenarios may occur during synchronization, thus impacting the overall performance of the database system.

[0005] Therefore, improving the efficiency and stability of the data synchronization process while ensuring data consistency has become one of the key technical directions that urgently need to be optimized in the design of distributed database systems.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] In view of the problems in the prior art, the purpose of this invention is to provide a data synchronization method, apparatus, device, storage medium and computer program product, to overcome the difficulties of related technologies and improve the efficiency and stability of the data synchronization process.

[0008] The first aspect of this disclosure provides a data synchronization method applied to a master node, comprising: In response to a write request for a data object, a metadata sequence associated with the data object is generated. The metadata sequence contains index information of each data segment formed after the data object is divided. A replication control flow is constructed based on the metadata sequence. The replication control flow is used to describe the transmission order constraints of each data segment and the completion conditions for the data object. Send a sequence of metadata to the slave node; The corresponding data segment is obtained based on the index information, and the data segment is sent to the slave node under the condition of satisfying the transmission order constraint. If the data segment transmission status meets the completion conditions, send a synchronization completion flag associated with the data object to the slave node.

[0009] In some implementations, the data synchronization method further includes: In response to write requests for data objects, persist data fragments to a storage area independent of the log structure; The metadata sequence also includes location information used to indicate the physical location parameters of each data fragment in the storage area; Retrieve the corresponding data fragments based on the index information, including: The corresponding data fragments are read from the storage area based on the location information and index information.

[0010] In some implementations, generating a sequence of metadata associated with a data object includes: Data objects are divided into multiple data fragments based on a preset fragmentation strategy; For each data segment, generate corresponding index information, which includes the logical position of the data segment in the data object.

[0011] In some implementations, the metadata sequence also includes integrity feature values ​​for each data segment; When constructing the replication control flow, logical rules for data verification by slave nodes are defined based on integrity feature values.

[0012] In some implementations, the transmission of metadata sequences and the transmission of data fragments are performed through logically isolated transmission paths.

[0013] In some implementations, execution is performed via logically isolated transport paths, including: Send metadata sequences to slave nodes via the control channel; Data segments are sent to slave nodes through a data transmission channel that is logically isolated from the control channel, in accordance with transmission order constraints.

[0014] In some implementations, the control channel is a streaming link based on database write-ahead logs; The data transmission channel is a block transmission link based on an independent network connection.

[0015] In some implementations, the data synchronization method further includes: Maintain the synchronization state machine of data objects based on the replication control flow; Based on the feedback results of the data fragments in the data transmission channel, the synchronous state machine is driven to transition between different states.

[0016] In some implementations, driving the synchronous state machine to transition between different states includes: Upon detecting any abnormal transmission of a data segment, the synchronization state is transitioned to the abnormal state, and the corresponding data segment is retransmitted; or, In response to the synchronization state machine failing to transition to the completed state within a preset time period, a failure command is sent to the slave node to trigger the slave node to clean up the storage space of the received data fragments. For example, the slave node ensures that incomplete data fragments are not committed to the database kernel by rolling back the corresponding logical storage area.

[0017] In some implementations, the completion condition includes at least one of the following: The last data segment in the replication control flow has been successfully sent; Received a confirmation message from the slave node indicating that all data segments have been received.

[0018] In some implementations, the corresponding data fragment is obtained based on index information, including: Parse the replication control flow to identify the set of data fragment indexes to be sent; A prefetch buffer is built in memory based on a set of data fragment indexes, and multiple data fragments are prefetched into the prefetch buffer in batches.

[0019] A second aspect of this disclosure provides a data synchronization apparatus applied to a master node, comprising: The metadata generation module, in response to a write request from a data object, generates a metadata sequence associated with the data object. The metadata sequence contains index information of each data segment formed after the data object is divided. The module constructs a replication control flow based on the metadata sequence. The replication control flow is used to describe the transmission order constraints of each data segment and the completion conditions for the data object. The first sending module sends a sequence of metadata to the slave nodes; The second sending module obtains the corresponding data segment based on the index information and sends the data segment to the slave node under the condition of satisfying the transmission order constraint. The third sending module sends a synchronization completion flag associated with the data object to the slave node when the sending status of the data segment meets the completion conditions.

[0020] A third aspect of this disclosure provides an electronic device, characterized in that it comprises: Processor; and Memory, which stores the processor's executable instructions; The processor is configured to execute the data synchronization method of any of the above embodiments by executing executable instructions.

[0021] The fourth aspect of this disclosure provides a computer-readable storage medium for storing a program, characterized in that the program, when executed, implements the data synchronization method of any of the above embodiments.

[0022] The fifth aspect of this disclosure provides a computer program product having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the data synchronization method of any of the above embodiments.

[0023] The data synchronization method, apparatus, device, storage medium, and computer program product proposed in this disclosure have the following advantages: Through the above implementation method, the master node introduces a mechanism combining metadata sequences and replication control flow during data synchronization. This transforms the data synchronization process from a traditional method directly driven by data content to a method based on unified scheduling and control using metadata. By constructing and distributing metadata sequences before data transmission and using replication control flow to constrain the transmission order and completion conditions of data segments, pre-planning and process control of the data synchronization process can be achieved, thus ensuring that the transmission of data segments has a clear execution order and completion determination mechanism.

[0024] Therefore, achieving structured control over the data synchronization process without relying on the overall processing of complete data objects helps improve the execution efficiency and controllability of the data synchronization process, and reduces the processing uncertainty brought about by complex data objects during the synchronization process.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0027] Figure 1 A flowchart illustrating a data synchronization method provided in one embodiment of this disclosure; Figure 2 A schematic diagram illustrating the application scenario and interaction process of a data synchronization system provided in one embodiment of this disclosure; Figure 3 A structural block diagram of a data synchronization device provided in one embodiment of this disclosure; Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided according to one embodiment of the present disclosure. Detailed Implementation

[0028] To make the technical solution, the technical problem solved, and the technical effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the described embodiments are merely exemplary embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make various modifications and variations to the embodiments without departing from the spirit and scope of the present invention, and all such modifications and variations should be considered to fall within the scope of protection of the present invention.

[0029] Techniques, methods, and apparatus known to those skilled in the art are not discussed in detail where appropriate, but should be considered part of this specification. In all examples shown herein, any specific numerical values ​​or parameters should be interpreted as exemplary only and not as limiting the invention.

[0030] Figure 1 The flowchart illustrates a data synchronization method provided in this disclosure. The executing entity of the data synchronization method can be the master node in a distributed database system, which can be a service node responsible for handling write requests and distributing data. In this embodiment, the master node can establish communication connections with one or more slave nodes to achieve synchronization of data objects between different nodes.

[0031] like Figure 1 As shown, the data synchronization method includes, but is not limited to, the following steps: Step 110: In response to a write request for a data object, generate a metadata sequence associated with the data object, the metadata sequence containing index information of each data segment formed after the data object is divided; Step 120: Construct a replication control flow based on the metadata sequence. The replication control flow is used to describe the transmission order constraints of each data segment and the completion conditions for the data object. Step 130: Send the metadata sequence to the slave node; Step 140: Obtain the corresponding data segment based on the index information, and send the data segment to the slave node under the condition of satisfying the transmission order constraint; Step 150: If the transmission status of the data segment meets the completion condition, send a synchronization completion identifier associated with the data object to the slave node.

[0032] Using this implementation, a data segment can be understood as a sub-unit of a data object, with each data segment corresponding to a portion of the data object's content. The metadata sequence can be understood as a set of structured descriptive information built around the data object. This metadata sequence is used not only to identify the data segments after the data object has been divided, but also to describe the logical relationships between the data segments. Correspondingly, index information can be used to indicate the relative position or order of each data segment within the data object, thereby transforming the data object from a holistic representation into a structured form that can be managed segment by segment.

[0033] In this embodiment, the replication control flow can be understood as a set of control structures or rules used to guide the data synchronization process. It can be generated based on the metadata sequence and is used to transform the structural relationships described in the metadata into an executable synchronization strategy.

[0034] Specifically, replication control flow can constrain the transmission order of data segments, ensuring that data segment transmission is no longer unordered or simply sequential, but rather scheduled based on predefined constraints. Simultaneously, replication control flow is also used to define the completion conditions of the data object synchronization process, thus providing a clear basis for determining the termination of the synchronization process.

[0035] In this implementation, when the master node sends data segments, it does not transmit them directly according to the write order or a simple traversal method. Instead, it schedules the transmission of data segments based on the transmission order constraints defined in the replication control flow.

[0036] Specifically, the master node can determine the data segment to be sent based on the index information in the metadata sequence, and execute the sending operation if the corresponding order constraints are met. Thus, the data segment sending process is subject to unified scheduling and constraints of the replication control flow, giving the data synchronization process a clear execution path and control boundaries.

[0037] In this way, data transmission behavior changes from being passively triggered to being executed in a controlled manner, thereby improving the consistency and predictability of the synchronization process.

[0038] In this implementation, the master node makes a unified judgment on the transmission status of data segments based on predefined completion conditions in the replication control flow. When the completion conditions are met, the master node sends a synchronization completion flag to indicate that the data synchronization process corresponding to the data object has ended.

[0039] A synchronization completion flag can be understood as an identifier indicating that the data synchronization process for a certain data object has ended. Upon receiving this flag, a slave node can determine whether the synchronization of the data object is complete and perform subsequent processing operations accordingly, such as data commit, status update, or resource release.

[0040] Through the above implementation method, the master node introduces a mechanism combining metadata sequences and replication control flow during data synchronization. This transforms the data synchronization process from a traditional method directly driven by data content to a method based on unified scheduling and control using metadata. By constructing and distributing metadata sequences before data transmission and using replication control flow to constrain the transmission order and completion conditions of data segments, pre-planning and process control of the data synchronization process can be achieved, thus ensuring that the transmission of data segments has a clear execution order and completion determination mechanism.

[0041] Therefore, achieving structured control over the data synchronization process without relying on the overall processing of complete data objects helps improve the execution efficiency and controllability of the data synchronization process, and reduces the processing uncertainty brought about by complex data objects during the synchronization process.

[0042] In this embodiment of the disclosure, in step 110 above, the write request for the data object can be initiated by an external client, which can be an application service, a data acquisition component, or another data producer. The client can submit a write request to the master node through a network interface (such as an HTTP interface, RPC interface, or database write protocol interface). The write request can include the data object to be written and basic information related to the data object.

[0043] Upon receiving a write request, the master node can parse it using its internal data access module to extract the data object. Subsequently, the master node can submit the data object to the data processing flow to perform preprocessing operations related to data synchronization.

[0044] In some implementations, the master node does not directly include the data object as a whole in the synchronization process before synchronizing the data object, but first performs a structured splitting of the data object.

[0045] Specifically, the master node can partition data objects based on a preset sharding strategy to form multiple data fragments. A preset sharding strategy can be understood as a set of rules guiding the partitioning of data objects, which can be set according to database system design requirements or data characteristics. For example, the sharding strategy can be based on the size of the data objects, or it can be based on the internal structure of the data or data boundaries; this disclosure does not limit this.

[0046] After the data objects are partitioned, the master node can generate corresponding index information for each data segment. The index information describes the positional or logical relationship of the data segment within the original data object; for example, it can identify the relative order, logical position, or association of the data segment within the overall data object. Through the index information, a mapping relationship can be established between data segments and the original data object, allowing the data segments to be logically recombined into the original data object.

[0047] Therefore, through the above implementation method, the data object is transformed into a structured representation consisting of multiple data fragments and their index information before entering the synchronization process, thereby providing support for subsequent data synchronization scheduling.

[0048] Furthermore, the master node can construct a metadata sequence based on the index information of each data segment. The metadata sequence can be understood as an ordered set of information built around the data object, which is used to uniformly describe the structural characteristics of the data object after it has been divided and the relationships between the data segments.

[0049] In one implementation, the metadata sequence can be organized in the order corresponding to the data segments, so that each information unit in the sequence can be associated with its corresponding data segment.

[0050] It should be noted that the generation of the metadata sequence is completed during the preprocessing stage before the actual data synchronization and transmission. By generating this metadata sequence in advance, the master node can achieve unified scheduling of data segment transmission without repeatedly parsing the complete object during subsequent synchronization.

[0051] Therefore, by generating a metadata sequence during the write request processing stage, the structural information of the data object can be extracted from the data content of the data object and expressed in an independent information form, thus providing a basis for constructing the replication control flow based on the metadata sequence, and enabling the data synchronization process to have structured and schedulable characteristics.

[0052] In some implementations, while generating the metadata sequence, the master node can also perform persistent storage operations on the data fragments.

[0053] Specifically, in response to a write request for a data object, the master node, after dividing the data object into multiple data fragments, can write each data fragment to a pre-configured storage area. The storage area can be a file system, an object storage system, or other persistent media used to hold the data content.

[0054] In this embodiment, the storage area and the log structure are independent of each other. The log structure can be understood as recording data change processes or operation information, while the storage area is mainly used to hold the data object content itself. By decoupling data fragments from the log structure, large amounts of data can be avoided from being directly recorded in the log, thereby reducing the growth rate of log files and improving the I / O throughput of the database system.

[0055] After the data fragments are persistently stored, the master node can obtain the physical location parameters of each data fragment in the storage area. The physical location parameters can be used to indicate the specific storage location of the data fragment in the storage medium, and may include, for example, the storage path, storage offset, or other identification information related to the storage location.

[0056] The master node can use physical location parameters as positioning information and write them together with the index information of the corresponding data segment into the metadata sequence. That is, the metadata sequence constitutes a dual description structure containing logical index and physical location.

[0057] Therefore, the meta-information sequence can not only describe the logical relationships between data segments, but also provide a basis for locating data segments for access.

[0058] In subsequent step 140, the master node can determine the data segment to be acquired based on the index information in the metadata sequence, and further combine it with the location information to read the corresponding data segment from the storage area. In this way, the master node can directly locate and read data segments based on the metadata sequence without relying on re-parsing the complete data object. This location-reading method supports random access, meaning the master node can read and send data segments non-linearly based on network conditions or priority.

[0059] This implementation separates persistent storage of data content from log recording and establishes the ability to locate data fragments through metadata sequences, making access to data content more direct during data synchronization and thus helping to reduce redundant overhead in the data processing process.

[0060] In some implementations, when generating the metadata sequence in step 110, the master node can also generate a corresponding integrity feature value for each data segment and write the integrity feature value into the metadata sequence.

[0061] In this context, the integrity feature value can be understood as verification information used to characterize the consistency of data fragment content. In specific implementation, the master node can perform predetermined data processing operations based on the content of the data fragment to generate this integrity feature value.

[0062] For example, digest calculation, checksum generation, or other processing operations that reflect the characteristics of the data content can be performed on the data fragment. This disclosure does not limit the specific algorithm form, such as cyclic redundancy check (CRC), MD5 digest algorithm, SHA series algorithms, etc. The core is that when the data fragment changes during transmission or storage, the corresponding integrity characteristic value can reflect the change.

[0063] After generating the integrity feature values, the master node can associate them with the corresponding data segment index information and organize them together into the metadata sequence. Thus, the metadata sequence contains not only the structural information of the data segments but also the data feature information used for verification.

[0064] In the subsequent step 120, when constructing the replication control flow, the master node can further define logical rules for the slave nodes to perform data verification based on integrity feature values. Specifically, the master node can introduce verification constraints for each data segment in the replication control flow, ensuring that the data segment meets predetermined integrity conditions after transmission.

[0065] In some implementations, under this logical rule, the slave node performs the following operation: Perform predetermined processing operations on the received data segment to obtain the first feature value; The first feature value is compared with the corresponding integrity feature value in the received metadata sequence; If the comparison is consistent, the data segment is determined to have passed the integrity check and is allowed to participate in subsequent data reassembly or state advancement; If the comparison is inconsistent, the data segment is determined to be abnormal, and the corresponding processing flow is triggered, such as requesting retransmission or marking a synchronization abnormality.

[0066] In some implementations, the replication control flow can also associate the integrity check result with the completion conditions in subsequent step 150. For example, the passing of integrity checks for all data fragments can be used as one of the prerequisites for determining the completion of data object synchronization, thereby avoiding premature termination of the synchronization process in the event of data inconsistency.

[0067] Furthermore, integrity features can also be used to enhance transmission order constraints. For example, when multiple data segments can be transmitted in parallel, certain critical data segments can be required to pass integrity checks before subsequent related data segments are allowed to enter the transmission or acknowledgment phase, thereby improving the reliability of the overall synchronization process to some extent.

[0068] By incorporating the integrity feature values ​​of data segments into the metadata sequence and defining corresponding verification logic rules in the replication control flow, the data synchronization process can not only proceed in a predetermined order, but also constrain and verify the consistency of data content during transmission, thereby improving the reliability and accuracy of data synchronization.

[0069] In this embodiment of the disclosure, in step 120, the master node can further construct the replication control flow based on the metadata sequence.

[0070] In this context, replication control flow can be understood as a control description structure oriented towards the data synchronization process. It is used to express the transmission order of data segments and the rules for determining synchronization completion. Unlike structures that only record data content, replication control flow focuses more on modeling the "transmission behavior".

[0071] Specifically, the master node can traverse the index information of each data segment contained in the metadata sequence and determine the logical order relationship between the data segments based on the index information. Since the metadata sequence pre-contains the index information of all data segments, the master node can deduce the global data segment transmission sequence through lightweight computation without touching the actual data content of the data objects.

[0072] In some implementations, the index information may explicitly include the relative position of the data fragments within the data object (e.g., starting offset or sequence number), and the master node may directly determine the transmission order of each data fragment based on this information.

[0073] In other implementations, the index information may only reflect the relative relationships between data segments, and the master node may determine a transmission sequence that satisfies logical constraints through sorting, topology analysis, or other sequential derivation methods.

[0074] Building upon this, the master node can construct a control structure to describe the sequential relationship, such as generating an ordered queue, a linked structure, or a directed graph structure, where each node corresponds to a data segment, and each edge represents the sequential dependency between data segments. For example, nodes with an in-degree of zero in a directed graph can be used as the initial sending segments to achieve automated scheduling of complex dependencies.

[0075] After determining the logical order of the data segments, the master node can encode this order relationship into a transmission order constraint.

[0076] Transmission order constraints can be used to define the scheduling rules for data segments during transmission. For example, in one implementation, the master node can organize data segments into a linear sequence in a defined order and require that subsequent data segments are only allowed to be transmitted after the preceding data segments have been transmitted.

[0077] In another implementation, transmission order constraints can also support partial parallelism. That is, data segments without dependencies can be sent concurrently, while data segments with sequential dependencies need to satisfy the order constraint.

[0078] For example, for non-overlapping, non-continuous data segments, the replication control flow can define parallel transmission branches, thereby leveraging multithreading or multichannel to improve synchronous bandwidth utilization.

[0079] Therefore, the replication control flow can describe not only strict sequential transmission models, but also transmission models with a certain degree of parallelism, thus adapting to different system performance requirements.

[0080] In some implementations, during the construction of the replication control flow, the master node can also define completion conditions for data objects based on the metadata sequence. These completion conditions are used to determine whether the synchronization process of the data object has reached the end state.

[0081] Specifically, the completion conditions may include at least one of the following: First, the master node detects that the last data segment defined in the replication control flow has been successfully sent. Successful sending means that the data segment has completed the sending operation or has been marked as sent in the master node's sending queue.

[0082] Secondly, the master node receives a reception completion acknowledgment message from the slave node indicating that all data segments of the data object have been correctly received.

[0083] In some implementations, the master node can use any of the above conditions as the basis for completion determination; in other implementations, a combined determination method can also be used, that is, the synchronization process is considered complete only when multiple completion conditions are met.

[0084] In some implementations, the master node can also encapsulate transmission order constraints and completion conditions into a unified control description structure to drive the execution and state management of the data synchronization process. In this implementation, the replication control flow is presented in the form of a control description structure, which may include, but is not limited to, the following information: The data segment identifiers and their corresponding order; The transmission status identifier for each data segment (e.g., pending transmission, in transmission, completed); Dependencies between data fragments; A conditional expression used to determine if the entire process is complete.

[0085] Through the control description structure described above, the master node can uniformly schedule the sending process of data segments based on the replication control flow in subsequent steps, without having to repeatedly parse the original data objects or recalculate the order relationship.

[0086] After the replication control flow is constructed, when the master node executes step 140, it can directly schedule the sending process of each data segment step by step according to the transmission order constraints defined in the replication control flow.

[0087] Meanwhile, during step 150, the master node can monitor the sending status of data segments based on the completion conditions defined in the replication control flow, and trigger the sending of a synchronization completion flag when the completion conditions are met.

[0088] The above implementation method separates data description from transmission control during data synchronization: the metadata sequence is used to describe the data structure, while the replication control flow is used to describe the transmission behavior.

[0089] This method of constructing replication control flow based on metadata enables the master node to directly and finely control the transmission of data fragments without relying on the parsing of complete data objects. This improves the controllability and execution efficiency of the data synchronization process and provides a unified basis for subsequent sequential scheduling and completion determination.

[0090] In some implementations, in step 130, the transmission of a metadata sequence is used for the initialization process of establishing a synchronization context between the master and slave nodes. Through this process, the slave node can obtain control information describing the structure and transmission rules of the data object in advance before receiving actual data fragments.

[0091] In some implementations, the master node can trigger the transmission of the metadata sequence immediately after completing the replication control flow construction, or it can trigger it when preset transmission conditions are met, such as: the metadata sequence construction is completed; the corresponding data object writing process enters the synchronization stage; or, a preset synchronization scheduling window is reached.

[0092] By controlling the timing of transmission, the metadata sequence can be made to arrive at the slave node before the data fragment is sent.

[0093] In one implementation, the master node can encapsulate the metadata sequence into a structured control message for transmission. This control message may contain: Identification information corresponding to data objects (such as tags used to distinguish different synchronization tasks); Index information for each data segment; Description of sequence constraints related to replication control flow; Control parameters related to the completion conditions; Optional verification information (such as integrity feature values).

[0094] In this way, the metadata sequence can not only describe the data structure, but also carry control information to drive the slave node to execute synchronization logic.

[0095] In some implementations, the metadata sequence can be transmitted via a dedicated control channel. The control channel can be understood as a communication path used to transmit control information, logically distinct from the subsequent data channel used to transmit data segments. In this implementation, the control channel and data channel employ different transmission protocols or are assigned different transmission priorities to ensure the timeliness of the control information.

[0096] The aforementioned control channel can have the following characteristics: The transmission priority is high to ensure that the metadata sequence can reach the slave node first; It transmits relatively small amounts of data and is suitable for low-latency transmission. Supports sequential or reliability guarantees to avoid loss or out-of-order control information.

[0097] By transmitting metadata sequences and data fragments separately, control information can be prevented from being blocked by large-scale data transmission, thereby improving the overall response speed of synchronous scheduling.

[0098] In some implementations, after receiving the metadata sequence, the slave node can perform the following preprocessing operations: Establish the corresponding receiving context based on the data object identifier; Initialize the data segment receiving structure based on the index information and order constraints in the metadata sequence; Based on the control parameters in the metadata sequence, configure the verification rules or completion judgment conditions during the receiving process; Allocate corresponding logical storage areas for subsequent data segments.

[0099] For example, the node can pre-allocate the corresponding storage block in memory or persistent medium based on the location information in the metadata sequence to achieve zero-copy or direct data writing.

[0100] Through the above preprocessing, the slave node can complete the reception preparation in advance before the data fragment actually arrives, thereby avoiding additional parsing or structure construction during the data reception process.

[0101] After completing the transmission of the metadata sequence, the master node can continue to execute step 140 based on the replication control flow, that is, send data segments according to the predetermined transmission order constraints.

[0102] Since the slave node has already established a receiving context based on the metadata sequence, it can directly process the received data segments according to the metadata sequence without having to re-derive the data structure or transmission rules.

[0103] By sending a sequence of metadata in advance, the data synchronization process adopts a control-first, data-following execution mode. This mode allows slave nodes to know the structural information and transmission rules of data objects in advance, thus enabling direct processing when data fragments arrive, reducing intermediate parsing and scheduling overhead.

[0104] Furthermore, by transmitting metadata sequences separately from data fragments, the risk of control information being blocked during transmission can be reduced, thereby improving the efficiency and stability of the entire data synchronization process.

[0105] In some implementations, step 140 above can be understood as a process of controlled scheduling and transmission of data content under the drive of the replication control flow.

[0106] In some implementations, the master node can determine the data segment to be sent based on the index information recorded in the metadata sequence.

[0107] In practice, the master node can locate the corresponding data segment by combining index information and pre-established data segment mapping relationships.

[0108] In conjunction with the above implementation methods, the master node can further read corresponding data fragments from a storage area independent of the log structure based on location information. For example, the master node can determine the target data fragment identifier based on index information, and then read the corresponding data fragment content from the storage area through the physical location parameters associated with that identifier.

[0109] In some implementations, when the master node performs the step 140 of "obtaining the corresponding data fragment based on the index information", it can combine the replication control flow to preprocess the data fragment to be sent in order to improve the efficiency of data reading and transmission.

[0110] Specifically, the master node can first parse the replication control flow to identify the set of data segment indices to be sent in the current phase. The data segment index set can be understood as a group of data segments to be sent within the current sending window or current scheduling period, determined according to the transmission order constraints in the replication control flow. The master node identifies this set by parsing the replication control flow as the basis for performing batch prefetching.

[0111] After obtaining the set of data fragment indices, the master node can build a prefetch buffer in memory based on this set. The prefetch buffer can be a contiguous or segmented memory space used to temporarily store data fragments read from the underlying storage medium.

[0112] In practical implementation, the master node can read multiple data segments from the storage area in batches according to the order or batch strategy corresponding to the data segment index set, and load them into the prefetch buffer. For example, the master node can predict the segment demand in the next scheduling cycle based on the transmission order constraints in the replication control flow, and thus load the corresponding data segments into the prefetch buffer in advance.

[0113] In some implementations, the batch prefetching process may employ a sequential read approach or an optimization strategy based on access locality to reduce the performance overhead of random disk access.

[0114] Subsequently, when performing the data fragment sending operation, the master node can directly extract the corresponding data fragment from the prefetch buffer for sending, without triggering the storage read operation again during the sending phase.

[0115] In some implementations, the master node can also dynamically adjust the size of the prefetch buffer or the prefetch batch based on the replication control flow to adapt to different data object sizes or network bandwidth conditions. For example, in high-throughput scenarios, the master node monitors the feedback latency of the slave nodes in real time. When the feedback latency is below a threshold, the prefetch window can be appropriately expanded to improve the continuity of data transmission. In resource-constrained scenarios, the prefetch size can be reduced to decrease memory usage.

[0116] By introducing a prefetch buffer mechanism based on replication control flow, the storage access latency during data segment transmission can be effectively reduced, I / O wait time can be decreased, thereby improving the overall throughput performance and stability of data synchronization.

[0117] In a further implementation, the master node can monitor network bandwidth utilization and the I / O load status of the storage system in real time.

[0118] If the network throughput is detected to be higher than the preset threshold and memory resources are sufficient, the master node dynamically expands the window size of the prefetch buffer to increase the number of data segments read in a single batch. For example, the preset threshold can be preset based on the system's historical operating data. Conversely, the prefetch window is reduced to decrease memory usage and avoid unnecessary I / O warm-up. This dynamic adjustment mechanism enables the data synchronization process to adapt to different hardware operating environments.

[0119] In some implementations, after acquiring a data fragment, the master node may not send it immediately, but instead first determine whether the current data fragment satisfies the transmission order constraints defined in the replication control flow.

[0120] Specifically, the master node can maintain scheduling status information corresponding to the replication control flow to record the transmission progress of each data segment. For example, a transmission status flag (such as "pending transmission", "transmitted", "acknowledged") can be set for each data segment.

[0121] In some implementations, transmission order constraints can be manifested as a sequential dependency, meaning that a current data segment is only allowed to be sent if the preceding data segment has reached a preset state (such as being successfully sent or having received an acknowledgment).

[0122] In other implementations, for data segments that do not have dependencies, the master node can allow them to be sent in parallel if resource conditions are met, thereby improving overall transmission efficiency.

[0123] Through the above mechanism, the master node can flexibly schedule the transmission of data segments while ensuring data order consistency.

[0124] After satisfying the transmission order constraints, the master node can send the corresponding data fragment to the slave node.

[0125] In some implementations, sending the metadata sequence in step 130 and sending the data fragment in step 140 can be performed through logically isolated transmission paths.

[0126] Logical isolation can be understood as distinguishing, at the system architecture or communication protocol level, the path used to transmit metadata sequences from the path used to transmit data fragments, making them independent of each other in scheduling, bandwidth allocation, or processing. In this embodiment, this independence ensures that the transmission of control information is not blocked by large-scale data flows.

[0127] By using the above methods, we can avoid the problem of large data volumes consuming transmission resources when sending data segments, which would affect the timeliness of the transmission of metadata sequences.

[0128] Alternatively, logically isolated transmission paths can also allow metadata sequences and data fragments to be scheduled according to their respective priorities. For example, low-latency transmission of metadata sequences and synchronization control information can be prioritized, while the transmission of data fragments can be scheduled in a high-throughput but queuing mode.

[0129] This implementation method decouples the transmission of control information from the transmission of data content during the data synchronization process on the execution path, thereby improving the stability and scheduling efficiency of the overall synchronization process.

[0130] In some implementations, logically isolated transmission paths can be specifically achieved by separating the control channel from the data transmission channel.

[0131] Specifically, during step 130, the master node can send a sequence of metadata to the slave node via the control channel. The control channel can be used to carry control-related information, characterized by its small data volume and time-sensitivity.

[0132] During step 140, the master node can send data fragments to the slave node through a data transmission channel logically isolated from the control channel, following the transmission order constraints defined in the replication control flow. The data transmission channel can be used to carry data content, and its design can focus on data throughput capabilities.

[0133] In some implementations, the control channel and the data transmission channel can correspond to different communication connections, different protocol stacks, or different processing threads, thereby achieving resource isolation at the system level.

[0134] In this way, the metadata sequence can arrive at the slave node before or independently of the data segment, thereby enabling the slave node to establish a receiving context in advance based on the metadata.

[0135] In some implementations, the control channel and the data transmission channel can be implemented using different types of transmission links.

[0136] Specifically, the control channel can be a streaming link based on the database write-ahead log (WAL). The master node appends the metadata sequence as part of the log record and transmits it to the slave nodes using the log replication mechanism. Through the log system's sequential guarantee capability, it ensures that the control information corresponding to the metadata sequence arrives before the data fragment and is not lost.

[0137] Accordingly, the data transmission channel can be a block transmission link based on an independent network connection. For example, data fragments can be sent through independent TCP connections or other block-oriented communication methods. In this link, methods such as chunked transmission, pipelined transmission, or batch transmission can be used to improve data transmission efficiency.

[0138] Due to the differences in link type between the control channel and the data transmission channel, control information and data content can be optimized and configured according to their respective characteristics, thereby improving the overall synchronization performance.

[0139] In some implementations, during the transmission of data fragments, the master node can associate the data fragments with their corresponding identification information (such as object identifiers or data fragment indexes) so that the slave nodes can correctly identify and process the data fragments after receiving them.

[0140] Furthermore, in some implementations, the master node can also record the transmission status of data segments during the transmission process and update the status based on feedback information returned by the slave node. For example, after receiving a reception confirmation from the slave node, the corresponding data segment is marked as "acknowledged".

[0141] During the entire execution of step 140, the master node can repeatedly perform the following operations: The set of data segments that can be sent is determined based on the replication control flow; Select data segments that meet the conditions from the set of data segments; Retrieve the content of the corresponding data segment; Trigger the send operation; Update the state information in the copy control flow.

[0142] Through the aforementioned cyclic scheduling mechanism, the master node can gradually complete the transmission of all data segments.

[0143] In some implementations, if an anomaly occurs during data segment transmission (e.g., transmission failure or timeout), the master node can handle it based on the state information in the replication control flow. For example, it can mark the corresponding data segment as abnormal, suspend the transmission of subsequent data segments that depend on that segment, trigger a retransmission mechanism, or reschedule. Through the topology dependencies in the replication control flow, the master node can accurately locate the affected subsequent segments, thereby ensuring the consistency of global logic in the event of local anomalies.

[0144] Furthermore, the master node can dynamically adjust the sending rhythm based on the current system load or network conditions, such as limiting the number of concurrent sending requests or adjusting the sending order priority. For example, a sliding window protocol or token bucket algorithm can be used to control the instantaneous sending traffic.

[0145] Through the above implementation methods, the data segment transmission process is uniformly scheduled by the replication control flow, realizing an execution mode of on-demand acquisition and controlled transmission.

[0146] On the one hand, by directly locating data fragments based on index information, the repeated parsing of complete data objects is avoided; on the other hand, by controlling the transmission order constraint, the data synchronization process has a clear order and controllability.

[0147] This mechanism helps improve the efficiency of data synchronization while ensuring data consistency, and reduces the system overhead caused by disordered transmission or repeated scheduling.

[0148] In some implementations, in step 150, the synchronization completion flag can be understood as control information used to mark that the synchronization process of a data object has reached the end state. It is used to notify the slave node that the data segment corresponding to the current data object has been fully transmitted and can enter the subsequent processing stage.

[0149] In some implementations, the master node can manage the synchronization process of data objects in a unified manner based on the replication control flow constructed in step 120, and maintain the corresponding synchronization state machine accordingly.

[0150] The synchronization state machine is used to describe the state evolution of a data object during the replica synchronization process. This synchronization state machine may include, but is not limited to, the following states: initial state (to be synchronized), in-transmission state, partially completed state, completed state, and abnormal state.

[0151] In its implementation, the master node instantiates a corresponding synchronization state machine for each data object. During step 140, the master node drives the synchronization state machine to transition states based on the transmission behavior and feedback results of each data segment. Feedback results include successful transmission flags, failed transmission flags, timeout information, and reception acknowledgments returned by the slave node through the control channel. By parsing the feedback results, the master node identifies the current real-time state of the data segment and updates the synchronization state machine accordingly.

[0152] For example, when at least one data segment is successfully transmitted, the synchronization state machine can transition from the initial state to the in-transmission state; when some data segments meet the preset transmission or acknowledgment conditions, it can transition to the partially completed state; and when all data segments meet the completion conditions defined in the replication control flow, it transitions to the completed state.

[0153] In some implementations, the master node can also maintain corresponding transmission status information for each data segment, such as status flags like "sent," "sent successfully," and "acknowledged," and make an overall determination based on the status information combined with the completion conditions in the replication control flow. Completion conditions may include, but are not limited to: all data segments have been sent, or all data segments have been acknowledged by the slave nodes, or a combination of the above conditions.

[0154] For example, using a combined judgment method can further ensure the reliability of data synchronization and prevent situations where the master node misjudges that the data has been sent successfully when the slave node has not actually received it due to network packet loss.

[0155] Based on the above determination results, when the synchronization state machine enters the completion state, the master node triggers the sending of the synchronization completion flag in step 150, thereby realizing the closed-loop confirmation of the data object synchronization process.

[0156] In some implementations, the master node can also monitor the synchronization process using an anomaly detection mechanism. When an anomaly is detected in the transmission of any data segment, such as transmission failure, failure to receive acknowledgment within a preset time, or network connection failure, the master node can transition the synchronization state machine from the current state to the anomaly state.

[0157] When the synchronization state machine enters an abnormal state, the master node triggers a retransmission mechanism according to the replication control flow, or sends an alarm message to the management terminal. Specifically, the retransmission mechanism includes: the master node relocating the abnormal data segment based on the replication control flow and re-executing the data acquisition and transmission process. In some implementations, the master node can constrain the retransmission strategy, such as limiting the number of retransmissions or adjusting the retransmission priority.

[0158] Furthermore, in some implementations, the master node can also control the synchronization process based on a time dimension. For example, a preset duration threshold can be set for the synchronization state machine to limit the maximum allowed time for a data object to go from the start of synchronization to completion. When it is detected that the synchronization state machine has not reached the completion state within the preset duration, it can be determined that there is an abnormal risk in the current synchronization process.

[0159] In this scenario, the master node can send a failure command to the slave node to notify it to terminate the synchronization process of the current data object. For example, this failure command could be sent from the master node to the slave node via the control channel mentioned earlier.

[0160] On the slave node side, it can respond to failure commands and perform cleanup operations on received but not yet reassembled data segments, such as deleting incomplete data segments, releasing temporary storage space, or rolling back intermediate states, thereby preventing invalid data from occupying system resources.

[0161] In some implementations, the above-described exception handling mechanism can also be coordinated with the completion condition determination. For example, the master node sends a synchronization completion flag only when the synchronization state machine has not entered an abnormal state and the completion condition is met; otherwise, the exception handling process is executed first.

[0162] By introducing a mechanism that combines replication control flow with a synchronization state machine, the data synchronization process has clear state expression, controllable state transition paths, and robust exception handling capabilities. This ensures data consistency while improving the controllability of the synchronization process for complex data objects and the efficiency of system resource utilization.

[0163] In some implementations, the synchronization completion identifier may include the following information: The object identifier corresponding to the data object is used to uniquely identify the current synchronization task; Synchronize and complete status indication information; Optional verification digest or status digest information is used to assist slave nodes in confirming consistency. Or other control fields used to indicate the end of synchronization.

[0164] By carrying an object identifier in the synchronization completion identifier, the slave node can associate the object identifier with previously received metadata sequences and data fragments.

[0165] In some implementations, the master node can send a synchronization completion flag to the slave node via a control channel.

[0166] Since the synchronization completion identifier is a control information, its data volume is small and its timing requirements are high. Therefore, it can be consistent with the transmission path of the metadata sequence in step 130, thereby ensuring its priority and transmission reliability.

[0167] During the transmission process, the master node can send the synchronization completion identifier as a separate control message, or it can embed it into the control flow as a specific type of control instruction.

[0168] In some implementations, the slave node can perform the following operations after receiving the synchronization completion flag: Locate the corresponding data object based on the object identifier in the synchronization completion identifier; Check whether all data segments corresponding to the data object have been received; If the integrity verification conditions are met, the data fragments will be combined into a complete data object or their temporary storage state will be released. Update the local synchronization status and mark the data object as synchronized.

[0169] In this way, the slave node can use the synchronization completion flag as a trigger signal to complete the state transition from the receiving phase to the available phase.

[0170] In some implementations, if an anomaly (such as unacknowledged partial data segments) is detected before or after sending the synchronization completion flag, the master node may delay sending the synchronization completion flag or append an anomaly flag after sending it to trigger compensation processing by the slave node.

[0171] Furthermore, in some implementations, if a slave node does not receive a synchronization completion flag within a preset time, it can proactively initiate a status query request to the master node to confirm the synchronization status. If a slave node detects that data segment reception has stalled and the synchronization duration exceeds a preset protection threshold, it sends a heartbeat query request to the master node to synchronize the state machines at both ends.

[0172] By introducing a synchronization completion flag after the data segment is sent, the entire data synchronization process forms a clear end signal, thereby avoiding inferring whether synchronization is complete from the uncertain state of node dependencies.

[0173] This mechanism allows the data synchronization process to evolve from data transmission driven to state driven, which helps improve the determinism and controllability of the synchronization process, and also facilitates subsequent operations such as data activation or resource release by slave nodes at the correct time.

[0174] Figure 2 This invention illustrates a data synchronization system provided by an embodiment of the present disclosure. For example... Figure 2 As shown, in this embodiment, the distributed database system includes a master node (Node-A) and slave nodes (Node-B). The data synchronization process occurs between the client and the master and slave nodes, where the master node acts as the sender and the slave node acts as the receiver. The two establish a synchronization link through the network to achieve data object replication synchronization.

[0175] In the above data synchronization system architecture, the master node is used to receive data object write requests from clients and perform data object splitting, persistence, and synchronization scheduling; the slave node is used to receive metadata sequences and data fragments sent by the master node and complete the corresponding data writing and reassembly processes.

[0176] In some implementations, the master node may include independent data storage areas and log structures. The data storage area stores binary data files (e.g., xxx.bin) corresponding to data segments, while the log structure stores write-ahead log files (e.g., xxx.wal). Physically separating data content from log information helps reduce log size and improve system processing efficiency.

[0177] Reference Figure 2 The data synchronization process in this embodiment, as shown in the flowchart, may include the following steps: Step 210: The client initiates a write request for the data object.

[0178] The client sends a write request for a data object to the master node through the access layer. The data object can be a structured data object, such as a JSON object or other binary structured data.

[0179] Step 220: Data object sharding and persistent storage.

[0180] After receiving a write request, the master node can divide the data object into multiple data fragments based on a preset sharding strategy (such as based on data size or logical boundaries). Subsequently, the master node persists the data fragments to data folder 1 (e.g., xxx.bin) in the data storage area using a streaming write method.

[0181] Step 230: Meta-information sequence generation and log recording.

[0182] During the data fragment persistence process, the master node can extract the index information and storage location parameters corresponding to each data fragment and generate a metadata sequence associated with the data object. The metadata sequence is written to the write-ahead log file to describe the logical relationships between the data fragments and their location information in the storage medium.

[0183] Furthermore, the master node constructs a replication control flow based on the metadata sequence. The replication control flow is used to describe the transmission order constraints of each data segment and can be further used to define the completion conditions for data objects, thereby providing a unified scheduling basis for subsequent data synchronization processes.

[0184] Step 240: Based on the replication control flow, read the corresponding data segment from the data storage area according to the transmission order constraints.

[0185] Step 250: Coordinated transmission of metadata sequences and data fragments.

[0186] During the data synchronization phase, the master node can transmit metadata sequences and data fragments separately through logically isolated transmission paths, including: A sequence of metadata is sent through the control channel to enable the slave node to establish the corresponding data receiving context. The read data fragments are sent to the slave node through the data transmission channel.

[0187] like Figure 2 As shown, for example, dashed lines represent the synchronization stream of the metadata sequence, and solid lines represent the data stream of data segments.

[0188] In some implementations, the master node can also build a prefetch buffer in memory based on the replication control flow to prefetch multiple data segments in batches, thereby reducing I / O latency during the data reading process.

[0189] The slave node receives the metadata sequence through the control channel and establishes data reception rules based on the index information in the metadata sequence. When receiving data segments, the slave node can identify and process the data segments according to the index information.

[0190] In some implementations, slave nodes can perform consistency checks based on index information, such as identifying duplicate data segments and determining whether to perform an overwrite operation based on preset rules (such as timestamps or version information).

[0191] Step 260: After receiving the data, the slave node streams the data fragments to the data folder 2 (e.g., xxx.bin) in the local data storage area to complete the local persistence of the data object.

[0192] For example, the slave node pre-allocates a memory buffer based on the metadata sequence. When it receives out-of-order data segments, it uses the index information to place them at the corresponding logical offset position, thereby achieving out-of-order reception and ordered reassembly.

[0193] Finally, the master node can monitor the sending status of data segments based on the replication control flow, and when the completion conditions are met, send a synchronization completion flag to the slave node to indicate that the synchronization process of the data object has ended.

[0194] By using the above technical solution, the present invention separates the metadata sequence and data fragments and transmits them through different transmission paths, thereby decoupling the control information corresponding to the metadata sequence from the data content, thus avoiding the blockage of the control process by large-scale data transmission and improving the responsiveness of the synchronization process.

[0195] Meanwhile, by introducing replication control flow to uniformly describe the transmission order and completion conditions of data segments, the data synchronization process becomes schedulable and controllable, which helps to improve the stability of the synchronization process of complex data objects.

[0196] Finally, by performing data processing and consistency control based on index information on the slave node side, it is beneficial to improve the robustness of the data synchronization process and enhance the system's recovery capability under abnormal conditions.

[0197] Figure 3 A modular structure diagram of the data synchronization device provided in this disclosure is shown. This data synchronization device 300 can be deployed on the master node side of a distributed data processing system. Its execution entity can be a data storage node (e.g., a Data Node), and it can run under a distributed parallel processing architecture. The functional modules can communicate with each other through interface calls, remote procedure calls, or message mechanisms, thereby forming a data synchronization processing link corresponding to the data writing process.

[0198] like Figure 3 As shown, the data synchronization device 300 may include: The metadata generation module 310, in response to the write request of the data object, generates a metadata sequence associated with the data object. The metadata sequence contains index information of each data segment formed after the data object is divided. Module 320 is constructed to build a replication control flow based on the metadata sequence. The replication control flow is used to describe the transmission order constraints of each data segment and the completion conditions for the data object. The first sending module 330 sends a sequence of metadata to the slave node; The second sending module 340 obtains the corresponding data segment based on the index information and sends the data segment to the slave node under the condition of satisfying the transmission order constraint. The third sending module 350 sends a synchronization completion flag associated with the data object to the slave node when the sending status of the data segment meets the completion conditions.

[0199] In some implementations, the data synchronization device 300 may further include a storage management module (not shown) for persistently storing data fragments formed by dividing the data object in a storage area independent of the log structure when responding to a write request for the data object. The storage area may be a local file system, a distributed file system, or object storage space.

[0200] In this embodiment, when generating the metadata sequence, the metadata generation module 310 can also generate location information indicating the physical location parameters of each data segment in the storage area. Thus, when the second sending module 340 acquires a data segment, it can read the corresponding data segment from the storage area based on the location information and index information, thereby achieving decoupled access between the data content and the log structure.

[0201] In some implementations, the metadata generation module 310 may include a fragmentation processing unit (not shown) for dividing the data object into multiple data segments based on a preset fragmentation strategy. The preset fragmentation strategy may be determined based on a size threshold of the data object, data structure boundaries, or streaming input characteristics. The metadata generation module 310 may also generate corresponding index information for each data segment, wherein the index information is used to identify the logical position of the data segment in the original data object.

[0202] In some implementations, the metadata generation module 310 can also generate integrity feature values, such as checksums or hash values, for each data segment. When constructing the replication control flow, the control flow construction module 320 can define logical rules for performing data verification by slave nodes based on the integrity feature values, so as to support slave nodes in performing integrity verification and sequence reconstruction after receiving data segments.

[0203] In some implementations, the first sending module 330 and the second sending module 340 can perform sending operations through logically isolated transmission paths. Specifically, the data synchronization device 300 may further include a channel management module (not shown) for configuring a control channel and a data transmission channel, wherein the control channel is used to transmit metadata sequences and the data transmission channel is used to transmit data segments.

[0204] In further implementations, the control channel can be a streaming link based on database write-ahead logs, while the data transmission channel can be a block transmission link based on independent network connections, such as a data channel based on TCP connections or remote direct memory access. This channel isolation mechanism reduces interference from large-scale data transmissions on control signaling.

[0205] In some embodiments, the data synchronization device 300 may further include a state management module (not shown) for maintaining a synchronization state machine for the data object according to the replication control flow. The state management module may drive the synchronization state machine to transition between different states based on feedback results of data fragments in the data transmission channel.

[0206] In a further implementation, when any data segment transmission anomaly is detected, the state management module can transition the synchronization state to the abnormal state and trigger the second sending module 340 to perform the retransmission operation of the corresponding data segment; or, when it is detected that the synchronization state machine has not transitioned to the completed state within a preset time period, the third sending module 350 can send a failure command to the slave node under the control of the state management module to trigger the slave node to perform storage space cleanup on the received data segments.

[0207] In some implementations, the replication control flow constructed by the control flow construction module 320 may include completion condition definitions. Specifically, the state management module may determine synchronization completion based on at least one of the following: When it is detected that the last data segment defined in the replication control flow has been successfully sent; Alternatively, when a confirmation message indicating that all data segments have been received is received from the slave node.

[0208] When the completion conditions are met, the third sending module 350 sends a synchronization completion flag.

[0209] In some implementations, the second sending module 340 may include a prefetch processing unit (not shown) for parsing the copy control flow before acquiring data segments to identify a set of data segment indices to be sent. The prefetch processing unit may construct a prefetch buffer in memory based on the index set and prefetch multiple data segments in batches from the storage area into the prefetch buffer.

[0210] During data transmission, the second transmission module 340 can directly retrieve data segments from the prefetch buffer and transmit them, thereby reducing the latency caused by real-time storage access and improving the transmission efficiency of data synchronization.

[0211] By implementing the aforementioned data synchronization device, fine-grained control and efficient transmission of data objects can be achieved during the synchronization process. By employing a processing mechanism that separates metadata sequences from data segments, and by combining replication control flow to uniformly describe the transmission order and completion conditions of data segments, the synchronization process is transformed from a single data stream transmission into a schedulable and controllable multi-stage processing flow, thereby improving the controllability and execution efficiency of the synchronization process.

[0212] This disclosure also provides an electronic device, including a processor and a memory, wherein the memory stores executable instructions that can be executed by the processor. When the processor executes the executable instructions, it implements the steps of the data synchronization methods of the foregoing embodiments.

[0213] As described above, the electronic device of the present invention introduces a control mechanism centered on metadata sequences during processor execution, splitting the data object synchronization process into an execution mode that coordinates metadata-driven control flow and data fragment transmission flow. Specifically, by generating a metadata sequence containing data fragment index information and constructing a replication control flow based on this metadata sequence, the processor can uniformly schedule and determine the transmission order of data fragments and the synchronization completion conditions. Simultaneously, by sequentially acquiring and sending data fragments according to the index information and triggering the transmission of a synchronization completion flag when the completion conditions are met, staged control and closed-loop management of the data synchronization process are achieved.

[0214] By employing the above methods, electronic devices no longer rely on the overall processing of complete data objects when performing data synchronization tasks. Instead, they perform collaborative processing based on fine-grained data fragments and their corresponding control information. This improves the controllability and execution efficiency of the data synchronization process, reduces resource consumption while ensuring synchronization consistency, and enhances adaptability to complex data object synchronization scenarios.

[0215] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0216] Figure 4 This is a schematic diagram of the electronic device of the present invention. See below for reference. Figure 4To describe an electronic device 400 according to this embodiment of the present invention. Figure 4 The electronic device 400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0217] like Figure 4 As shown, the electronic device 400 is presented in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different platform components (including storage unit 420 and processing unit 410), a display unit 440, etc.

[0218] The storage unit stores program code, which can be executed by the processing unit 410 to perform the steps described in the data synchronization method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 410 can perform actions such as... Figure 1 The steps are shown.

[0219] Storage unit 420 may include a readable medium in the form of an active storage unit, such as random access memory (RAM) 421 and / or cache memory 422, and may further include read-only memory (ROM) 423.

[0220] Storage unit 420 may also include a program / utility business tool 424 having a set (at least one) program module 425, such program module 425 including but not limited to: operating system, one or more application programs, other program modules and program data, each of these examples or some combination of these may include an implementation of a network environment.

[0221] Bus 430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0222] Electronic device 400 can also communicate with one or more external devices 401 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 400, and / or with any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. Network adapter 460 can communicate with other modules of electronic device 400 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0223] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or electronic device, etc.) to execute the methods according to the embodiments of this disclosure.

[0224] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a computer program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this disclosure may also be implemented as a computer program product including program code that, when the computer program product is run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0225] The program product for implementing the above-described method according to embodiments of the present disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0226] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0227] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0228] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0229] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on an IoT terminal, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0230] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0231] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0232] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or electronic device, etc.) to execute the methods according to the embodiments of this disclosure.

[0233] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A data synchronization method, characterized in that, Applied to the master node, including: In response to a write request for a data object, a metadata sequence associated with the data object is generated, the metadata sequence containing index information of each data segment formed after the data object is divided; A replication control flow is constructed based on the metadata sequence. The replication control flow is used to describe the transmission order constraints of each data segment and the completion conditions for the data object. Send the metadata sequence to the slave node; Based on the index information, the corresponding data segment is obtained, and the data segment is sent to the slave node under the condition that the transmission order constraint is satisfied; If the transmission status of the data segment meets the completion condition, a synchronization completion identifier associated with the data object is sent to the slave node.

2. The data synchronization method according to claim 1, characterized in that, The data synchronization method further includes: In response to the write request of the data object, the data fragment is persistently stored in a storage area independent of the log structure; The metadata sequence also includes location information for indicating the physical location parameters of each data fragment in the storage area; The step of obtaining the corresponding data fragment based on the index information includes: Based on the location information and the index information, the corresponding data fragment is read from the storage area.

3. The data synchronization method according to claim 1, characterized in that, The generation of the metadata sequence associated with the data object includes: The data object is divided into multiple data fragments based on a preset fragmentation strategy; For each data segment, corresponding index information is generated, and the index information includes the logical position of the data segment in the data object.

4. The data synchronization method according to claim 1, characterized in that, The metadata sequence also includes the integrity feature value of each data segment; When constructing the replication control flow, logical rules for data verification by the slave nodes are defined based on the integrity feature value.

5. The data synchronization method according to claim 1, characterized in that, The sending of the metadata sequence and the sending of the data fragment are performed through logically isolated transmission paths.

6. The data synchronization method according to claim 5, characterized in that, The execution via a logically isolated transmission path includes: The metadata sequence is sent to the slave node via the control channel; The data fragments are sent to the slave node through a data transmission channel that is logically isolated from the control channel, in accordance with the transmission order constraints.

7. The data synchronization method according to claim 6, characterized in that, The control channel is a streaming link based on database write-ahead logs; The data transmission channel is a block transmission link based on an independent network connection.

8. The data synchronization method according to claim 6, characterized in that, The data synchronization method further includes: Maintain the synchronization state machine of the data object according to the replication control flow; Based on the feedback results of the data fragment in the data transmission channel, the synchronous state machine is driven to transition between different states.

9. The data synchronization method according to claim 8, characterized in that, The process of driving the synchronous state machine to transition between different states includes: Upon detecting any abnormal transmission of a data segment, the synchronization state is transitioned to the abnormal state, and the corresponding data segment is retransmitted; or, If the synchronization state machine fails to transition to the completed state within a preset time period, a failure command is sent to the slave node to trigger the slave node to clean up the storage space of the received data segments.

10. The data synchronization method according to claim 1, characterized in that, The completion conditions include at least one of the following: It was detected that the last data segment in the replication control flow had been successfully transmitted; The system receives a confirmation message from the slave node indicating that all data segments have been received.

11. The data synchronization method according to claim 1, characterized in that, The step of obtaining the corresponding data fragment based on the index information includes: The replication control flow is parsed to identify the set of data fragment indices to be sent; A prefetch buffer is constructed in memory based on the data fragment index set, and multiple data fragments are prefetched into the prefetch buffer in batches.

12. A data synchronization device, characterized in that, Applied to the master node, including: The metadata generation module, in response to a write request for a data object, generates a metadata sequence associated with the data object, the metadata sequence containing index information of each data segment formed after the data object is divided; The construction module constructs a replication control flow based on the metadata sequence. The replication control flow is used to describe the transmission order constraints of each data segment and the completion conditions for the data object. The first sending module sends the metadata sequence to the slave node; The second sending module obtains the corresponding data segment based on the index information, and sends the data segment to the slave node under the condition of satisfying the transmission order constraint; The third sending module, when the sending status of the data segment meets the completion condition, sends a synchronization completion identifier associated with the data object to the slave node.

13. An electronic device, characterized in that, include: processor; as well as A memory in which executable instructions of the processor are stored; The processor is configured to execute the data synchronization method of any one of claims 1 to 11 by executing the executable instructions.

14. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, it implements the data synchronization method according to any one of claims 1 to 11.

15. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data synchronization method according to any one of claims 1 to 11.

Images