Data synchronization method and device, equipment and storage medium
By splitting the large amount of data to be synchronized into multiple data sets in the data synchronization processing link and building corresponding data synchronization tasks, the problem of low data synchronization efficiency in different network environments is solved, and more efficient data is achieved transmission.
Patent Information
- Application Number
- CN202411546126.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-11
AI Technical Summary
When data synchronization is performed between two components in different network environments, data synchronization with larger amounts of data is less efficient and consumes more network resources.
By determining the data synchronization processing link, obtaining the data to be synchronized and splitting it into multiple data sets when the amount of data exceeds the preset threshold, building a data synchronization task for each data set based on the data synchronization processing link, and through these The task passes the dataset from the source node to the destination node.
By splitting the data to be synchronized in large data volumes into multiple data sets and processing them in parallel in different network environments, the problem of network transmission of large data volumes is overcome and the efficiency of data synchronization is improved.
Smart Images
Figure CN119420766A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing, and in particular to a data synchronization method, device, equipment and storage medium. Background Art
[0002] Data synchronization refers to the process of synchronizing data from one node to another, for example, synchronizing data from component A to component B. Currently, when synchronizing data between two components in different network environments, if the amount of data to be synchronized is large, more network resources will be consumed during the network transmission of the synchronized data, resulting in low data synchronization efficiency.
[0003] Therefore, when synchronizing data between components in different network environments, how to improve the efficiency of data synchronization is a technical problem that urgently needs to be solved. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a data synchronization method, apparatus, device and storage medium.
[0005] In a first aspect, an embodiment of the present disclosure provides a data synchronization method, the method comprising:
[0006] Determine a data synchronization processing link configured for the data to be synchronized; wherein the data synchronization processing link is configured with at least a source node and a destination node of the data to be synchronized, and the source node and the destination node are respectively deployed in different network environments;
[0007] Acquire the data to be synchronized from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, split the data to be synchronized into multiple data sets; wherein the data volume of the data sets does not exceed the preset data volume threshold;
[0008] Based on the data synchronization processing link, corresponding data synchronization tasks are respectively constructed for the multiple data sets, and the corresponding data sets are transferred from the source node to the destination node by executing the data synchronization tasks.
[0009] In an optional implementation manner, the data synchronization processing link is further configured with at least one data processing node, and the at least one data processing node is respectively configured with a corresponding data processing mode, and the corresponding data set is transferred from the source node to the destination node by executing the data synchronization task, including:
[0010] Sending the corresponding data set to the first data processing node in the data synchronization processing link by executing the data synchronization task;
[0011] After calling the first data processing node to process the data in the data set based on the corresponding data processing method, a processed data set is obtained, and the processed data set is transmitted to the destination node.
[0012] In an optional implementation manner, before transmitting the corresponding data set from the source node to the destination node by executing the data synchronization task, the method further includes:
[0013] The corresponding data set is split into multiple data units by executing the data synchronization task; wherein the multiple data units are used to process the data in the data set in batches, and the multiple data units are processed in parallel between different data processing nodes in the data synchronization processing link.
[0014] In an optional implementation manner, the sending the corresponding data set to the first data processing node in the data synchronization processing link by executing the data synchronization task includes:
[0015] Sending the first data unit in the corresponding data set to the first data processing node in the data synchronization processing link by executing the data synchronization task;
[0016] Correspondingly, the calling of the first data processing node to process the data in the data set based on the corresponding data processing method to obtain a processed data set includes:
[0017] The first data processing node is called to process each data in the first data unit in turn based on the corresponding data processing method using a streaming data processing method to obtain a first processed data unit, and a processed data set is obtained based on the first processed data unit.
[0018] In an optional implementation manner, the determining a data synchronization processing link configured for the data to be synchronized includes:
[0019] Receiving a source node, a destination node, at least one data processing node, and a connection relationship between the nodes determined for the data to be synchronized on the directed acyclic graph orchestration interface; wherein the at least one data processing node is respectively configured with a corresponding data processing mode;
[0020] A data synchronization processing link for the data to be synchronized is generated based on the source node, the destination node, the at least one data processing node, and the connection relationship between the nodes.
[0021] In an optional implementation manner, the transferring of the corresponding data set from the source node to the destination node by executing the data synchronization task includes:
[0022] By executing the data synchronization task based on memory, the corresponding data set is transferred from the source node to the destination node.
[0023] In an optional implementation manner, the source node and the destination node of the data to be synchronized are functional components deployed in different network environments respectively.
[0024] In a second aspect, the present disclosure provides a data synchronization device, the device comprising:
[0025] A determination module, used to determine a data synchronization processing link configured for the data to be synchronized; wherein the data synchronization processing link is configured with at least a source node and a destination node of the data to be synchronized, and the source node and the destination node are respectively deployed in different network environments;
[0026] an acquisition module, configured to acquire the data to be synchronized from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, split the data to be synchronized into multiple data sets; wherein the data volume of the data sets does not exceed the preset data volume threshold;
[0027] A transfer module is used to construct corresponding data synchronization tasks for the multiple data sets respectively based on the data synchronization processing link, and transfer the corresponding data sets from the source node to the destination node by executing the data synchronization tasks.
[0028] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a data synchronization method as provided in an embodiment of the present disclosure.
[0029] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the data synchronization method provided by the embodiment of the present disclosure.
[0030] In a fifth aspect, the present disclosure provides a computer program product, wherein the computer program product comprises a computer program / instructions, and the computer program / instructions implement the above method when executed by a processor.
[0031] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0032] In the data synchronization method provided by the embodiments of the present disclosure, first, a data synchronization processing link configured for the data to be synchronized is determined, wherein at least a source node and a destination node of the data to be synchronized are configured in the data synchronization processing link, and the source node and the destination node are respectively deployed in different network environments, the data to be synchronized is obtained from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, the data to be synchronized is split into multiple data sets, wherein the data volume of the data set does not exceed the preset data volume threshold, corresponding data synchronization tasks are respectively constructed for the multiple data sets based on the data synchronization processing link, and the corresponding data sets are transferred from the source node to the destination node by executing the data synchronization tasks.
[0033] In the embodiment of the present disclosure, the data to be synchronized that exceeds a preset data volume threshold is split into multiple data sets, and data synchronization tasks corresponding to each data set are constructed based on a data synchronization processing link. The corresponding data set is transmitted from the source node to the destination node as a transmission unit by executing the data synchronization task, thereby overcoming the network transmission problem of large amounts of data to be synchronized, thereby improving the data synchronization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0035] Figure 1 A flowchart of a data synchronization method provided by an embodiment of the present disclosure;
[0036] Figure 2 A schematic diagram of a data synchronization processing link provided by an embodiment of the present disclosure;
[0037] Figure 3 A process diagram of a data synchronization method provided by an embodiment of the present disclosure;
[0038] Figure 4 A schematic diagram of the structure of a data synchronization device provided by an embodiment of the present disclosure;
[0039] Figure 5 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0040] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0041] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0042] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0043] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0044] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0045] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0046] Currently, data synchronization between two components deployed in different network environments needs to be completed through network requests. However, if the amount of data to be synchronized is large, more network resources will be consumed during the network transmission process, resulting in low data synchronization efficiency.
[0047] Therefore, when synchronizing data between different components deployed in different network environments, how to improve the efficiency of data synchronization is a technical problem that needs to be solved urgently.
[0048] To this end, an embodiment of the present disclosure provides a data synchronization method, which determines a data synchronization processing link configured for the data to be synchronized, wherein the data synchronization processing link is configured with at least a source node and a destination node of the data to be synchronized, and the source node and the destination node are respectively deployed in different network environments. The data to be synchronized is obtained from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, the data to be synchronized is split into multiple data sets, wherein the data volume of the data set does not exceed the preset data volume threshold, and corresponding data synchronization tasks are respectively constructed for the multiple data sets based on the data synchronization processing link, and the corresponding data sets are transferred from the source node to the destination node by executing the data synchronization tasks.
[0049] The present invention divides the data to be synchronized that exceeds a preset data volume threshold into multiple data sets, constructs data synchronization tasks corresponding to each data set based on a data synchronization processing link, and transmits the data set as a transmission unit from a source node to a destination node by executing the data synchronization task, thereby overcoming the network transmission problem of large amounts of data to be synchronized and improving data synchronization efficiency.
[0050] Based on this, an embodiment of the present disclosure provides a data synchronization method, which can be applied to any device, for example, the device includes data to be synchronized. The method is introduced below in conjunction with specific embodiments.
[0051] Figure 1 The present invention provides a flow chart of a data synchronization method provided in an embodiment of the present invention. The method can be executed by a data synchronization device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:
[0052] S101: Determine a data synchronization processing link configured for data to be synchronized.
[0053] The data synchronization processing link is at least configured with a source node and a destination node of the data to be synchronized, and the source node and the destination node are respectively deployed in different network environments.
[0054] The data to be synchronized in the embodiment of the present disclosure may be any data that needs to be synchronized. In order to perform data synchronization on the data to be synchronized, the embodiment of the present disclosure also configures a data synchronization processing link, which is used to define a processing path for data synchronization. The data synchronization processing link includes information such as nodes that need to be passed through during the data synchronization process and transmission paths.
[0055] In practical applications, the configuration method of the data synchronization processing link can be configured through a directed acyclic graph orchestration interface. Specifically, on the directed acyclic graph orchestration interface, a selection operation of a source node and a destination node for the data to be synchronized is received, and a connection relationship between the nodes is established. Then, based on the source node and the destination node, and the connection relationship between the nodes, a data synchronization processing link for the data to be synchronized is generated.
[0056] A directed acyclic graph (DAG) is a directed graph without loops, that is, there is no path from a vertex to its starting point through a series of edges. The directed acyclic graph consists of vertices and directed edges. The vertices in the directed acyclic graph are used to identify nodes in data synchronization, and the directed edges are used to identify the execution order relationship between nodes.
[0057] The source node configured in the data synchronization processing link is the transmission starting position of the data to be synchronized, that is, the node where the data to be synchronized is located. The data to be synchronized starts to perform data synchronization operations from the source node. For example, the source node can be a functional component deployed on device A that needs to perform data synchronization; the destination node configured in the data synchronization processing link is the synchronization target node of the data to be synchronized, that is, the node where the data to be synchronized finally arrives. For example, the destination node can be a functional component deployed on device B, that is, the data to be synchronized is synchronized from a functional node on device A to a functional node on device B.
[0058] Among them, the source node and the destination node are in different network environments, which can be understood as the source node and the destination node are in different network environments. Specifically, the different network environments can refer to different network areas. For example, the source node is in the network environment of area A, and the destination node is in the network environment of area B. The different network environments can also refer to different computer rooms connected to the source node and the destination node. For example, the source node is connected to computer room A, and the destination node is connected to computer room B.
[0059] Specifically, in an application scenario, the source node and the destination node of the data to be synchronized are functional components deployed in different network environments. The data synchronization method provided by the embodiment of the present disclosure can realize data synchronization from the source functional component to the destination functional component.
[0060] S102: Acquire the data to be synchronized from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, split the data to be synchronized into multiple data sets.
[0061] The data volume of the data set does not exceed the preset data volume threshold.
[0062] In actual applications, if the amount of data to be synchronized is large or does not meet the data processing capacity of the memory, in order to achieve synchronization of the data to be synchronized, the embodiment of the present disclosure will determine whether the amount of data to be synchronized exceeds the preset data amount threshold. The preset data amount threshold is a pre-set value used to measure whether the amount of data to be synchronized is large enough to directly synchronize data.
[0063] If it is determined that the amount of data to be synchronized exceeds a preset data amount threshold, in order to transmit the data to be synchronized from the source node to the destination node, the embodiment of the present disclosure needs to split the data to be synchronized to obtain data sets, and use the data sets as transmission objects to reduce the amount of the data to be synchronized and improve data synchronization efficiency.
[0064] Among them, for the data splitting method, the splitting can be performed according to certain rules so that the data volume of the split data set does not exceed the preset data volume threshold. The embodiment of the present disclosure does not limit the splitting rules. For example, assuming that the preset data volume threshold is 100MB and the data volume of the data to be synchronized is 500MB, the data to be synchronized can be split into 5 data sets, each with a data volume of 100MB, so as to improve the data synchronization efficiency while reducing the data volume.
[0065] In an optional implementation, when the data to be synchronized is obtained from the source node, the data volume of the data to be synchronized is calculated, and it is determined whether the data volume exceeds a preset data volume threshold. If the data volume exceeds the preset data volume threshold, the data to be synchronized is split to obtain multiple data sets corresponding to the data to be synchronized.
[0066] S103: construct corresponding data synchronization tasks for the multiple data sets respectively based on the data synchronization processing link, and transmit the corresponding data sets from the source node to the destination node by executing the data synchronization tasks.
[0067] After the data to be synchronized is split into multiple data sets, corresponding data synchronization tasks are constructed for the multiple data sets, and the data synchronization tasks are used to transfer the corresponding data sets from the source node to the destination node. The data synchronization tasks may include operations such as data transmission.
[0068] Based on the data synchronization task corresponding to the data set, the corresponding data synchronization task is executed for the data set, and the corresponding data set is transmitted to the destination node.
[0069] When transferring a data set from a source node to a destination node based on a data synchronization task, in order to ensure data security and avoid data leakage, the present disclosure may implement the data synchronization task based on memory.
[0070] In an optional implementation, if the data synchronization task is created by the functional component where the source node is located, the data synchronization task can be executed based on the memory of the functional component where the source node is located, and the corresponding data set can be transferred from the source node to the destination node.
[0071] In another optional implementation, if the data synchronization task is created by the functional component where the destination node is located, the data synchronization task can be executed based on the memory of the functional component where the destination node is located, and the corresponding data set can be pulled from the source node to the destination node.
[0072] In practical applications, before a data set is transferred from a source node to a destination node, data mapping, data processing and other operations may be performed on the data set to convert the data into data required by the destination node.
[0073] In an optional implementation, a data processing node can be configured in the data synchronization processing link, and the data processing node is configured with a corresponding data processing method. Specifically, a selection operation of a source node, a destination node, and at least one data processing node determined for the data to be synchronized on a directed acyclic graph arrangement interface is received, and a connection operation between each node on the directed acyclic graph arrangement interface is received, and the data processing node is configured with a corresponding data processing method. Then, based on the source node, the destination node, the at least one data processing node, and the connection relationship between each node, a data synchronization processing link for the data to be synchronized is generated.
[0074] After obtaining the data synchronization processing link, a data synchronization task is constructed based on the data synchronization processing link and the data set, and the data set corresponding to the data synchronization task is sent to the first data processing node in the data synchronization processing link by executing the data synchronization task.
[0075] Then, the first data processing node is called to process the data in the data set based on the corresponding data processing method to obtain a processed data set, and the processed data set is transmitted to the destination node.
[0076] That is, the data processing node in the data synchronization processing link is used to perform corresponding data processing on the data set and transmit the processed data set to the destination node.
[0077] Among them, one or more data processing nodes can be configured in the data synchronization processing link, which can be determined based on the data processing task for the data set, and the data processing nodes can be connected in series or in parallel, which is determined based on the needs of data synchronization. Among them, one data processing node corresponds to one data processing task.
[0078] like Figure 2As shown, it is a schematic diagram of a data synchronization processing link provided by an embodiment of the present disclosure. By determining the source node, the destination node, the data processing node 201, and the data processing node 202 for the data to be synchronized on the directed acyclic graph arrangement interface, and then, based on the source node, the destination node, the data processing node, and the connection relationship between each node, the data synchronization processing link of the data to be synchronized is generated. The data synchronization processing link includes a source node, a target node, and a data processing node 201 and a data processing node 202, wherein the data processing nodes 201 and 202 are connected in parallel, and the data to be synchronized can be transmitted from the source node to the destination node after the data processing corresponding to the data processing node 201 and the data processing node 202, and the data results are merged.
[0079] In addition, the data synchronization processing link can also be modified on the directed acyclic graph arrangement interface, as described above. Figure 2 Also shown are add controls and delete controls.
[0080] The efficiency of data synchronization is improved by dividing the data to be synchronized into multiple data sets, constructing corresponding data synchronization tasks for the multiple data sets, executing the data synchronization tasks, and transmitting the data sets as transmission units to the destination node.
[0081] The data synchronization method provided by the embodiment of the present disclosure specifically determines a data synchronization processing link configured for the data to be synchronized, wherein the data synchronization processing link is configured with at least a source node and a destination node of the data to be synchronized, and the source node and the destination node are respectively deployed in different network environments, the data to be synchronized is obtained from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, the data to be synchronized is split into multiple data sets, wherein the data volume of the data set does not exceed the preset data volume threshold, corresponding data synchronization tasks are respectively constructed for the multiple data sets based on the data synchronization processing link, and the corresponding data sets are transferred from the source node to the destination node by executing the data synchronization tasks.
[0082] In the embodiment of the present disclosure, the data to be synchronized that exceeds a preset data volume threshold is split into multiple data sets, and data synchronization tasks corresponding to each data set are constructed based on a data synchronization processing link. The corresponding data set is transmitted from the source node to the destination node as a transmission unit by executing the data synchronization task, thereby overcoming the network transmission problem of large amounts of data to be synchronized, thereby improving the data synchronization efficiency.
[0083] To further improve data synchronization efficiency, the disclosed embodiment can obtain the data to be synchronized from the source node after determining the data synchronization processing link configured for the data to be synchronized, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, split the data to be synchronized into multiple data sets, and can also split the data sets into multiple data units, wherein the data synchronization processing link is configured with at least the source node of the data to be synchronized, the destination node, and at least one data processing node.
[0084] In an optional implementation, corresponding data synchronization tasks are constructed for the multiple data sets based on the data synchronization processing link, and the corresponding data sets are split into multiple data units by executing the data synchronization tasks. The data units are used to process the data in the data sets in batches, that is, the data units after the data sets are split are processed for the data units. The data units are processed in parallel between different data processing nodes in the data synchronization processing link, that is, multiple data units corresponding to a data set can be processed simultaneously between different data processing nodes in the data synchronization processing link.
[0085] For example, there are data processing nodes A and B in the data synchronization processing link. A data set is split into data unit 1 and data unit 2. Data unit 1 is processed at node A. After data unit 1 is processed at node A, it is transmitted to node B for further processing. Data unit 2 is transmitted to node A for processing. Data unit 1 and data unit 2 are processed in parallel.
[0086] After obtaining a plurality of data units corresponding to the data set, in order to transfer the data set from the source node to the destination node, the embodiment of the present disclosure may transfer the data set to the destination node based on the data unit.
[0087] In an optional implementation, corresponding data synchronization tasks are constructed for the multiple data sets respectively based on the data synchronization processing link, the corresponding data sets are split into multiple data units by executing the data synchronization tasks, and the first data unit in the data set is sent to the first data processing node in the data synchronization processing link.
[0088] The first data unit is a data unit in any data set, the first data unit corresponds to a first data processing node, and the first data processing node corresponds to the above data set.
[0089] Based on the introduction of the above content, the first data processing node in the embodiment of the present disclosure can perform data processing on each data unit in the data set based on the data processing method to obtain a processed data unit, and then obtain a processed data set.
[0090] For the processing method of the first data processing node for each data unit in the data set, in order to ensure that each data in the data unit can be processed, the embodiment of the present disclosure can use a streaming data processing method to process each data in the first data unit in sequence. Among them, the streaming data processing method can refer to a continuous data processing method. For example, after the first data of a data unit is processed by data processing node 1, the first data enters the next data processing node 2 for further processing, and data processing node 1 continues to process the second data in the data unit, and the processing of each data in the data unit is completed in the above manner.
[0091] In an optional implementation, the first data processing node is called to process each data in the first data unit in turn using a streaming data processing method based on a corresponding data processing method to obtain a first processed data unit, and a processed data set is obtained based on the first processed data unit.
[0092] For obtaining a processed data set based on the first processed data unit, specifically, it may include merging each processed data unit corresponding to the data set to obtain a processed data set corresponding to the data set, that is, merging the processed data units corresponding to the same data set to obtain a processed data set corresponding to the data set.
[0093] In an optional implementation, after obtaining a processed data set based on a processed data unit, the data set corresponding to the data to be synchronized can be transferred from a source node to a destination node, which may include obtaining a processed data set based on the processed data unit, and calling a first data processing node to transfer the processed data set to the destination node based on a corresponding data processing method.
[0094] By splitting the data set into multiple data units corresponding to the data set, different data processing nodes in the synchronous data processing link can process multiple data units in parallel, thereby improving data synchronization efficiency. At the same time, the data processing nodes use streaming data processing to process each data in the data unit in turn, ensuring that each data can be processed.
[0095] In order to facilitate understanding of the content of the above embodiment, the present disclosure also provides a process diagram of a data synchronization method, referring to Figure 3 .
[0096] As shown in the figure, the data to be synchronized is obtained from the source node, and it is determined whether the data volume of the data to be synchronized exceeds the preset data volume threshold. After determining that the data volume of the data to be synchronized exceeds the preset data volume threshold, the data to be synchronized is split into multiple data sets, such as data set 1 and data set 2.
[0097] Then, corresponding data synchronization tasks are constructed for the above multiple data sets based on the data synchronization processing link, and the data synchronization tasks are executed, and the data set corresponding to the data synchronization task is split into multiple data units, such as splitting data set 1 into data unit 1 and data unit 2. The multiple data units are used to process the data in the corresponding data set in batches, and the multiple data units are processed in parallel between different data processing nodes in the data synchronization processing link.
[0098] Taking data unit 1 and data unit 2 as an example, by executing the data synchronization task, data unit 1 in data set 1 is sent to data processing node 1 in the data synchronization processing link. Data processing node 1 processes each data in data unit 1 in turn based on the corresponding data processing method, thereby forming a streaming processing of the data units to ensure that each data can be processed.
[0099] When the last data in data unit 1 is processed and transmitted to processing node 2, data unit 2 in data set 1 is sent to data processing node 1. Data processing node 1 processes each data in data unit 2 in turn based on the corresponding data processing method. Data processing node 2 processes each data in data unit 1 in turn based on the corresponding data processing method, thereby forming multiple data units of data sets processed in parallel by different data processing nodes, thereby improving data synchronization efficiency. After data unit 1 and data unit 2 have been processed by data processing node 1, data processing node 1, and data processing node 3, processed data unit 1 of data unit 1 and processed data unit 2 of data unit 2 are obtained, and based on the above processed data units, processed data unit 1 and processed data unit 2 are merged and processed to obtain processed data set 1 corresponding to data set 1, and then the processed data set 1 is transmitted to the destination node.
[0100] In order to implement the above embodiments, the present disclosure also proposes a data synchronization device. Figure 4 Schematic diagram of the structure of a data synchronization device provided by an embodiment of the present disclosure, the device can be implemented by software and / or hardware, and can generally be integrated in an electronic device. Figure 4 As shown, the device comprises:
[0101] The determination module 401 is used to determine a data synchronization processing link configured for the data to be synchronized; wherein the data synchronization processing link is configured with at least a source node and a destination node of the data to be synchronized, and the source node and the destination node are respectively deployed in different network environments;
[0102] The acquisition module 402 is used to acquire the data to be synchronized from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, split the data to be synchronized into multiple data sets; wherein the data volume of the data sets does not exceed the preset data volume threshold;
[0103] The transfer module 403 is used to construct corresponding data synchronization tasks for the multiple data sets respectively based on the data synchronization processing link, and transfer the corresponding data sets from the source node to the destination node by executing the data synchronization tasks.
[0104] In an optional implementation manner, the data synchronization processing link is further configured with at least one data processing node, and the at least one data processing node is respectively configured with a corresponding data processing mode, and the transmission module includes:
[0105] A sending submodule, configured to send the corresponding data set to the first data processing node in the data synchronization processing link by executing the data synchronization task;
[0106] The data processing submodule is used to call the first data processing node to process the data in the data set based on the corresponding data processing method to obtain a processed data set, and transmit the processed data set to the destination node.
[0107] In an optional implementation, the device further includes:
[0108] A splitting module is used to split the corresponding data set into multiple data units by executing the data synchronization task; wherein the multiple data units are used to process the data in the data set in batches, and the multiple data units are processed in parallel between different data processing nodes in the data synchronization processing link.
[0109] In an optional implementation manner, the sending module is specifically used to:
[0110] Sending the first data unit in the corresponding data set to the first data processing node in the data synchronization processing link by executing the data synchronization task;
[0111] Accordingly, the data processing module is specifically used for:
[0112] The first data processing node is called to process each data in the first data unit in turn based on the corresponding data processing method using a streaming data processing method to obtain a first processed data unit, and a processed data set is obtained based on the first processed data unit.
[0113] In an optional implementation manner, the determining module includes:
[0114] A receiving submodule, used to receive a source node, a destination node, at least one data processing node, and a connection relationship between the nodes determined for the data to be synchronized on the directed acyclic graph orchestration interface; wherein the at least one data processing node is respectively configured with a corresponding data processing mode;
[0115] A generating submodule is used to generate a data synchronization processing link for the data to be synchronized based on the source node, the destination node, the at least one data processing node, and the connection relationship between the nodes.
[0116] In an optional implementation manner, the transmission module is specifically used to:
[0117] By executing the data synchronization task based on memory, the corresponding data set is transferred from the source node to the destination node.
[0118] In an optional implementation manner, the source node and the destination node of the data to be synchronized are functional components deployed in different network environments respectively.
[0119] In the data synchronization device provided by the embodiment of the present disclosure, first, a data synchronization processing link configured for the data to be synchronized is determined, wherein the data synchronization processing link is configured with at least a source node and a destination node of the data to be synchronized, and the source node and the destination node are respectively deployed in different network environments, the data to be synchronized is obtained from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, the data to be synchronized is split into multiple data sets, wherein the data volume of the data set does not exceed the preset data volume threshold, corresponding data synchronization tasks are respectively constructed for the multiple data sets based on the data synchronization processing link, and the corresponding data sets are transferred from the source node to the destination node by executing the data synchronization tasks.
[0120] In the embodiment of the present disclosure, the data to be synchronized that exceeds a preset data volume threshold is split into multiple data sets, and data synchronization tasks corresponding to each data set are constructed based on a data synchronization processing link. The corresponding data set is transmitted from the source node to the destination node as a transmission unit by executing the data synchronization task, thereby overcoming the network transmission problem of large amounts of data to be synchronized, thereby improving the data synchronization efficiency.
[0121] The data synchronization device provided in the embodiments of the present disclosure can execute the data synchronization method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.
[0122] In order to implement the above embodiments, the present disclosure further proposes a computer program product, including a computer program / instruction, which implements the data synchronization method in the above embodiments when executed by a processor.
[0123] In addition to the above-mentioned method and apparatus, the embodiments of the present disclosure further provide a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device implements the data synchronization method described in the embodiments of the present disclosure.
[0124] In addition, the present disclosure also provides a data synchronization device, see Figure 5 As shown, this may include:
[0125] Processor 501, memory 502, input device 503 and output device 504. The number of processors 501 in the data synchronization device can be one or more. Figure 5 In some embodiments of the present disclosure, the processor 501, the memory 502, the input device 503 and the output device 504 may be connected via a bus or other means, wherein: Figure 5 The example of connecting through bus is taken in the following.
[0126] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing of the data synchronization device by running the software programs and modules stored in the memory 502. The memory 502 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory 502 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The input device 503 can be used to receive input digital or character information, and generate signal input related to user settings and function control of the data synchronization device.
[0127] Specifically in this embodiment, the processor 501 will load the executable files corresponding to the processes of one or more applications into the memory 502 according to the following instructions, and the processor 501 will run the applications stored in the memory 502, thereby realizing the various functions of the above-mentioned data synchronization device.
[0128] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0129] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data synchronization method, characterized in that: include: Determine a data synchronization processing link configured for the data to be synchronized; wherein the data synchronization processing link is configured with at least a source node and a destination node of the data to be synchronized, and the source node and the destination node are respectively deployed in different network environments; Acquire the data to be synchronized from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, split the data to be synchronized into multiple data sets; wherein the data volume of the data sets does not exceed the preset data volume threshold; Based on the data synchronization processing link, corresponding data synchronization tasks are respectively constructed for the multiple data sets, and the corresponding data sets are transferred from the source node to the destination node by executing the data synchronization tasks.
2. The method according to claim 1, characterized in that The data synchronization processing link is further configured with at least one data processing node, and the at least one data processing node is respectively configured with a corresponding data processing mode. The corresponding data set is transferred from the source node to the destination node by executing the data synchronization task, including: Sending the corresponding data set to the first data processing node in the data synchronization processing link by executing the data synchronization task; After calling the first data processing node to process the data in the data set based on the corresponding data processing method, a processed data set is obtained, and the processed data set is transmitted to the destination node.
3. The method according to claim 2, characterized in that Before transferring the corresponding data set from the source node to the destination node by executing the data synchronization task, the method further includes: The corresponding data set is split into multiple data units by executing the data synchronization task; wherein the multiple data units are used to process the data in the data set in batches, and the multiple data units are processed in parallel between different data processing nodes in the data synchronization processing link.
4. The method according to claim 3, characterized in that The sending the corresponding data set to the first data processing node in the data synchronization processing link by executing the data synchronization task includes: Sending the first data unit in the corresponding data set to the first data processing node in the data synchronization processing link by executing the data synchronization task; Correspondingly, the calling of the first data processing node to process the data in the data set based on the corresponding data processing method to obtain a processed data set includes: The first data processing node is called to process each data in the first data unit in turn based on the corresponding data processing method using a streaming data processing method to obtain a first processed data unit, and a processed data set is obtained based on the first processed data unit.
5. The method according to claim 2, characterized in that: The step of determining a data synchronization processing link configured for the data to be synchronized includes: Receiving a source node, a destination node, at least one data processing node, and a connection relationship between the nodes determined for the data to be synchronized on the directed acyclic graph orchestration interface; wherein the at least one data processing node is respectively configured with a corresponding data processing mode; Based on the source node, the destination node, the at least one data processing node, and the connection relationship between the nodes, a data synchronization processing link for the data to be synchronized is generated.
6. The method according to claim 1, characterized in that The transferring of the corresponding data set from the source node to the destination node by executing the data synchronization task includes: By executing the data synchronization task based on memory, the corresponding data set is transferred from the source node to the destination node.
7. The method according to claim 1, characterized in that The source node and the destination node of the data to be synchronized are functional components deployed in different network environments respectively.
8. A data synchronization device, characterized in that: The device comprises: A determination module, used to determine a data synchronization processing link configured for the data to be synchronized; wherein the data synchronization processing link is configured with at least a source node and a destination node of the data to be synchronized, and the source node and the destination node are respectively deployed in different network environments; an acquisition module, configured to acquire the data to be synchronized from the source node, and after determining that the data volume of the data to be synchronized exceeds a preset data volume threshold, split the data to be synchronized into multiple data sets; wherein the data volume of the data sets does not exceed the preset data volume threshold; A transfer module is used to construct corresponding data synchronization tasks for the multiple data sets respectively based on the data synchronization processing link, and transfer the corresponding data sets from the source node to the destination node by executing the data synchronization tasks.
9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the data synchronization method described in any one of claims 1-10 above.
10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the data synchronization method described in any one of claims 1 to 7.
11. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.