A data synchronization method and device, electronic equipment and storage medium

Abstract

The application discloses a data synchronization method and device, electronic equipment and storage medium. The data synchronization method comprises the following steps: obtaining a data synchronization rule applied in a merging rule of a source end database; obtaining an extraction rule of a cluster node corresponding to the source end database; determining to-be-synchronized data in the cluster node according to the data synchronization rule and the extraction rule; determining a target end database according to the data synchronization rule, and synchronizing the to-be-synchronized data to the target end database. According to the application, the data synchronization rule is stored as the merging rule, the data synchronization rule is convenient to view and manage, the to-be-synchronized data is obtained by simultaneously analyzing and processing each storage node, the to-be-synchronized data is synchronized to the target end database according to the data synchronization rule, the resource consumption of data synchronization is reduced, and the use experience of a user is improved.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and in particular to a data synchronization method, apparatus, electronic device, and storage medium. Background Technology

[0002] During database synchronization, synchronization needs to be managed, which involves establishing data synchronization rules, determining the target location for data synchronization, and setting various parameters during synchronization. When synchronizing from the source database to the target database, multiple different data synchronization rules can be established according to different needs.

[0003] For databases from the same source, if data synchronization rules are not merged, the same database content needs to be parsed multiple times, resulting in a certain degree of resource waste. Database cluster nodes can include compute nodes and storage nodes. Data is distributed across multiple storage nodes according to a certain strategy. Each storage node has the same data structure, but its data content is relatively independent. During data synchronization, each storage node needs to be parsed separately, leading to resource waste. Furthermore, when storage nodes change, rules need to be re-established. Therefore, how to reduce resource consumption while ensuring normal data synchronization is a pressing issue. Summary of the Invention

[0004] This invention provides a data synchronization method, apparatus, electronic device, and storage medium to achieve unified management of data synchronization rules in the same database and reduce the resource consumption of data synchronization.

[0005] According to one aspect of the present invention, a data synchronization method is provided, wherein the method includes:

[0006] Obtain the data synchronization rules applied in the merge rules of the source database;

[0007] Obtain the extraction rules for the cluster nodes of the corresponding source database;

[0008] The data to be synchronized is determined on the cluster nodes according to the data synchronization rules and extraction rules.

[0009] The target database is determined according to the data synchronization rules, and the corresponding data to be synchronized is synchronized to the target database.

[0010] According to another aspect of the present invention, a data synchronization device is provided, wherein the device comprises:

[0011] The synchronization rule acquisition module is used to acquire the data synchronization rules applied in the merging rules of the source database;

[0012] The extraction rule acquisition module is used to acquire the extraction rules of the cluster nodes of the corresponding source database.

[0013] The data synchronization determination module is used to determine the data to be synchronized on the cluster nodes according to the data synchronization rules and extraction rules.

[0014] The data synchronization module is used to determine the target database according to the data synchronization rules and synchronize the corresponding data to be synchronized to the target database.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor;

[0017] and memory that is communicatively connected to at least one processor;

[0018] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to perform the data synchronization method of any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the data synchronization method of any embodiment of the present invention.

[0020] The technical solution of this invention obtains the data synchronization rules applied in the merging rules of the source database, obtains the extraction rules of the corresponding cluster nodes of the source database, determines the data to be synchronized on the cluster nodes according to the data synchronization rules and extraction rules, determines the target database according to the data synchronization rules, and synchronizes the corresponding data to be synchronized to the target database. This realizes the merging and management of data synchronization rules belonging to the same database, and the unified processing of various storage nodes. It also performs simultaneous parsing and processing on various storage nodes to obtain the data to be synchronized, reducing the resource consumption of data synchronization and improving the user experience.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1This is a flowchart of a data synchronization method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a data synchronization method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a flowchart of a data synchronization method provided according to Embodiment 3 of the present invention;

[0026] Figure 4 This is a mapping relationship structure diagram provided in Embodiment 3 of the present invention;

[0027] Figure 5 This is a flowchart of a rule management method provided according to Embodiment 3 of the present invention;

[0028] Figure 6 This is a flowchart of a data synchronization method provided according to Embodiment 3 of the present invention;

[0029] Figure 7 This is a schematic diagram of the structure of a data synchronization device according to Embodiment 4 of the present invention;

[0030] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the data synchronization method of this invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a data synchronization method according to Embodiment 1 of the present invention. This embodiment is applicable to the situation of synchronizing data from a source database to a target database. The method can be executed by a data synchronization device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S110. Obtain the data synchronization rules applied in the merge rules of the source database.

[0036] In this context, the source database can refer to the database that needs to be synchronized. The merge rule can be a rule generated by merging all data synchronization rules in the source database. In one embodiment, the merge rule may include at least one data synchronization rule. Data synchronization rules can be rules established according to data synchronization requirements and used for data synchronization. In actual operation, data synchronization rules can be created according to user needs, and can include an identifier for the data to be synchronized and an identifier for the target database, so that data synchronization can be performed based on these identifiers.

[0037] In this embodiment of the invention, the merging rules of the source database can be obtained, and the data synchronization rules applied in the merging rules can be determined. In actual operation, there may be a correspondence between the merging rules and the data synchronization rules, and the data synchronization rules in the merging rules can be obtained based on this correspondence. For example, the correspondence may include a mapping relationship, which can identify whether a data synchronization rule is included in a certain merging rule. In one embodiment, each data synchronization rule can be identified by a universally unique identifier (UUID).

[0038] After obtaining the data synchronization rules from the merge rules, the applicable data synchronization rules can be determined. In actual operation, the rule status of each data synchronization rule can be obtained to determine whether the rule status of each data synchronization rule is started or stopped. The data synchronization rules with the start status are then used as the applied data synchronization rules.

[0039] S120. Obtain the extraction rules for the cluster nodes of the corresponding source database.

[0040] In this context, a cluster node refers to a storage node (Data Node, DN) responsible for storing data within a cluster. Extraction rules can be established for each cluster node to facilitate the extraction of data stored on that node. These extraction rules are defined based on data extraction requirements, and each cluster node has its own set of extraction rules.

[0041] In this embodiment, the cluster nodes of the source database cluster can be determined, and the extraction rules corresponding to each cluster node can be obtained. In one embodiment, there may be instances where cluster nodes are added or removed from the cluster. When obtaining the extraction rules for each cluster node, it can be determined whether each cluster node has a corresponding extraction rule and whether each extraction rule has a corresponding cluster node. When any cluster node does not have a corresponding extraction rule, an extraction rule can be created for that cluster node; when any extraction rule does not have a corresponding cluster node, the extraction rule can be deleted to update the extraction rules of the cluster node, ensuring that each cluster node has a corresponding extraction rule.

[0042] S130. Determine the data to be synchronized on the cluster nodes according to the data synchronization rules and extraction rules.

[0043] In this embodiment of the invention, after determining the application's data synchronization rules and obtaining the extraction rules of the cluster nodes, data can be extracted according to the application's data synchronization rules and the cluster nodes' extraction rules to determine the data to be synchronized. In actual operation, the extraction rules can be invoked according to the application's data synchronization rules to extract the data corresponding to each data synchronization rule from the cluster nodes as the data to be synchronized.

[0044] In one embodiment, since there can be multiple data synchronization rules, and there may be duplicate data to be synchronized extracted from each data synchronization rule, the data extracted according to each data synchronization rule can be used as initial synchronization data. Identical initial synchronization data can be merged according to their identifiers to generate the data to be synchronized. In one embodiment, the identifier of the initial synchronization data may include, but is not limited to, the name and field information of the initial synchronization data.

[0045] S140. Determine the target database according to the data synchronization rules, and synchronize the corresponding data to be synchronized to the target database.

[0046] The target database can refer to the database that is being synchronized.

[0047] In the embodiments of the invention, the corresponding data to be synchronized can be matched according to each data synchronization rule to determine the target database, and the matched data to be synchronized can be synchronized to the corresponding target database. In actual operation, the identifier of the data to be synchronized and the identifier of the target database can be extracted from the data synchronization rule to determine the data to be synchronized and the target database corresponding to the data synchronization rule, and the data to be synchronized can be synchronized according to each data synchronization rule.

[0048] In one embodiment, the data to be synchronized can be extracted and parsed from the Write Ahead Log (WAL), the data to be synchronized can be matched with the data synchronization rules according to each data synchronization rule, and the data to be synchronized can be synchronized in the target database according to the data synchronization rules.

[0049] This invention, in its embodiments, obtains the data synchronization rules applied in the merging rules of the source database, acquires the extraction rules of the corresponding cluster nodes of the source database, determines the data to be synchronized on the cluster nodes according to the data synchronization rules and extraction rules, determines the target database according to the data synchronization rules, and synchronizes the corresponding data to be synchronized to the target database. This achieves merged management of data synchronization rules belonging to the same database, allowing each data synchronization rule to be executed simultaneously. It also unifies the processing of various storage nodes, performing simultaneous parsing and processing on each storage node to obtain the data to be synchronized, reducing the resource consumption of data synchronization and improving the user experience.

[0050] In one embodiment, before obtaining the data synchronization rules applied in the merge rules of the source database, the method further includes:

[0051] Create at least two data synchronization rules according to the data synchronization requirements; the data synchronization rules include: the identifier of the data to be synchronized and the identifier of the target database;

[0052] Merge at least two data synchronization rules into a merge rule and add a mapping relationship between the merge rule and the data synchronization rule to the mapping relationship set.

[0053] The data to be synchronized identifier can be information used to identify the data to be synchronized. In one embodiment, the data to be synchronized identifier may include the location information of the data to be synchronized, and the data to be synchronized can be extracted according to the identifier. The target database identifier can be information used to identify the target synchronization location of the data to be synchronized, and the target synchronization location of the data to be synchronized can be determined according to the target database identifier. The mapping relationship set can be a collection storing the mapping relationships between merging rules and data synchronization rules, and the mapping relationship between merging rules and data synchronization rules can be determined in the mapping relationship set.

[0054] In this embodiment of the invention, data synchronization rules can be created according to data synchronization requirements, and the number of data synchronization rules is not limited. In actual operation, the identifier of the data to be synchronized and the identifier of the target database can be determined according to the data synchronization requirements, and data synchronization rules can be created based on these identifiers. After the data synchronization rules are created, at least two data synchronization rules can be merged and stored as a merged rule, and the mapping relationship between each data synchronization rule and the merged rule can be determined and stored in a mapping relationship set.

[0055] In one embodiment, each data synchronization rule can be identified by a UUID. The merging rule can store all data synchronization rules belonging to the source database. The system can query the mapping relationship set according to the UUID to see if a corresponding merging rule exists. If it exists, a mapping relationship between the data synchronization rule and the merging rule is established and the mapping relationship is stored in the mapping relationship set. If it does not exist, a merging rule is established.

[0056] Example 2

[0057] Figure 2 This is a flowchart of a data synchronization method according to Embodiment 2 of the present invention. This embodiment is a further description of a data synchronization method based on the above embodiments. Figure 2 As shown, the method includes:

[0058] S210. Find at least one data synchronization rule according to the mapping relationship of the merging rules.

[0059] In this embodiment of the invention, a mapping relationship for the merging rules can be determined, and the data synchronization rules associated with the merging rules can be found according to the mapping relationship. In actual operation, all data synchronization rules associated with the merging rules can be extracted through the mapping relationship, and the number of data synchronization rules is not limited.

[0060] S220. Use the data synchronization rules with the rule status of "started" in each data synchronization rule as the data synchronization rules for the application.

[0061] The rule status can be an identifier indicating whether a data synchronization rule is applied. The rule status can include "started" and "stopped." When the rule status of a data synchronization rule is "started," the rule can be run to perform data synchronization; when the rule status is "stopped," the rule can be stopped. In one embodiment, the rule status can be set according to user needs.

[0062] In the embodiments of the invention, after obtaining the data synchronization rules, the rule status of each data synchronization rule can be extracted, and the data synchronization rule with the rule status of "started" can be used as the data synchronization rule of the application.

[0063] S230, Extract the cluster node corresponding to the source database.

[0064] In the embodiments of the invention, the data of the source database can be stored in multiple cluster nodes, and the cluster nodes of the cluster where the source database is located can be obtained to determine the cluster nodes storing the source database information.

[0065] S240. Determine the extraction rules that match each cluster node.

[0066] In the embodiments of the invention, extraction rules can be extracted to determine the extraction rules that match each cluster node. There can be a one-to-one correspondence between cluster nodes and extraction rules. In one embodiment, a mapping relationship between each cluster node and extraction rules can be pre-established, and the extraction rules matching each cluster node can be extracted through this mapping relationship.

[0067] In one embodiment, determining the extraction rules that match each cluster node further includes:

[0068] Determine whether each cluster node has a corresponding unique extraction rule;

[0069] If it does not exist, add or delete extraction rules to ensure that each cluster node has a unique extraction rule.

[0070] In this embodiment of the invention, after obtaining the extraction rules, cluster nodes can be matched with the extraction rules to determine whether each cluster node has a corresponding unique extraction rule. When a cluster node does not have a corresponding extraction rule, an extraction rule can be added or deleted. For example, when any cluster node does not have a corresponding unique extraction rule, an extraction rule corresponding to that cluster node can be added; when any extraction rule does not have a corresponding cluster node, that extraction rule can be deleted, so that each cluster node has a corresponding unique extraction rule.

[0071] S250. Extract initial synchronization data from cluster nodes according to data synchronization rules, and merge initial synchronization data with the same data name.

[0072] The initial synchronization data can be the synchronization data extracted according to the synchronization rules of each data.

[0073] In this embodiment of the invention, initial synchronization data can be extracted from the cluster nodes according to each data synchronization rule. Since there may be duplicates of the initial synchronization data extracted from each data synchronization rule, the data name of the initial synchronization data can be obtained, and initial synchronization data with the same data name can be merged.

[0074] S260. Use the initial synchronization data as the data to be synchronized.

[0075] S270. Extract the target synchronization data from the corresponding data to be synchronized according to the data synchronization rules.

[0076] In this embodiment of the invention, the corresponding data to be synchronized can be matched according to each data synchronization rule as the target synchronization data. In actual operation, the identifier of the data to be synchronized corresponding to the data synchronization rule can be extracted, the target synchronization data corresponding to each data synchronization rule can be determined, and the target synchronization data can be extracted from the data to be synchronized.

[0077] S280. Synchronize the target synchronization data to the target database according to the target database identifier of the data synchronization rules.

[0078] In the embodiments of the invention, the target location for each target data synchronization can be determined by the target database identifier, and the target synchronization data can be synchronized to the target database according to the target location.

[0079] In this embodiment of the invention, at least one data synchronization rule is found according to the mapping relationship of the merging rules. The data synchronization rules with the rule status of "started" are used as the applied data synchronization rules. The cluster nodes corresponding to the source database are extracted, and the extraction rules matching each cluster node are determined. Initial synchronization data is extracted from the cluster nodes according to the data synchronization rules, and initial synchronization data with the same data name are merged. The initial synchronization data is used as the data to be synchronized. The target synchronization data is extracted from the corresponding data to be synchronized according to the data synchronization rules. The target synchronization data is synchronized to the target database according to the target database identifier of the data synchronization rules. This achieves unified processing of each storage node, performs simultaneous parsing and processing on each storage node to obtain the data to be synchronized, reduces the resource consumption of data synchronization, and improves the user experience.

[0080] Example 3

[0081] Figure 3 This is a flowchart of a data synchronization method according to Embodiment 3 of the present invention. This embodiment is based on the above embodiment, taking the database as a PostgreSQL database and two data synchronization rules as examples, namely rule 1 and rule 2. The rule status of rule 1 is started and the start status of rule 2 is stopped, to further illustrate a data synchronization method.

[0082] S310. Create data synchronization rules 1 and 2 for the source database.

[0083] In one embodiment, rule 1 and rule 2 can be identified by UUID. For example, rule 1 is UUID1 and rule 2 is UUID2.

[0084] S320. Query whether there is a corresponding merging rule uuid_combine based on the source database identifier in rule 1.

[0085] S330. If no corresponding merge rule is found, then use the uuid1 of the current rule 1 as the merge rule uuid_combine, and establish the source database mapping relationship combine.map.

[0086] For example, the mapping relationship of the source database is uuid_combine; the mapping relationship between the merge rule and the data synchronization rules (rule 1 and rule 2) is origin.map. Among them, the mapping relationship between the merge rule and rule 1 is uuid_combine:uuid1.

[0087] S340. If a corresponding merge rule is found, add a new data synchronization rule and a mapping relationship between the merge rule and origin.map.

[0088] For example, the added mapping relationship can be uuid_combine:uuid1,uuid2.

[0089] S350. Obtain the cluster nodes (dn1, dn2, dn3) corresponding to the extraction database, and establish the extraction rules for each cluster node.

[0090] S360: Query the extraction rule mapping relationship child.map. If the cluster node does not find the corresponding extraction rule, create a new mapping relationship to child.map.

[0091] For example, the child.map mapping relationship can include:

[0092] dn1:uuid_child1;

[0093] dn2:uuid_child2

[0094] dn3:uuid_child3

[0095] S370. Check if the cluster node and extraction rule are consistent. If they are inconsistent, add or delete the corresponding extraction rule and update child_map.

[0096] S380. Obtain the rule status of Rule 1 and Rule 2. Among the data synchronization rules, the data synchronization rule with the rule status of "started" is used as the data synchronization rule of the application. Obtain the data to be synchronized.

[0097] In practice, the rule states of Rule 1 and Rule 2 can be pre-set and saved. The rule states of Rule 1 and Rule 2 can be extracted, and the initiated data synchronization rule can be used as the application's data synchronization rule to obtain the data to be synchronized.

[0098] For example, you can enable rule 1 by saving its status to `origin_rule.status` (i.e., `uuid1:enable`) and stop rule 2 by saving its status to `origin_rule.status` (i.e., `uuid2:disable`). If all rules are stopped, then the extraction of rules is terminated.

[0099] In one embodiment, data to be synchronized that have the same data name can be merged.

[0100] S390. Determine the target database according to the data synchronization rules, and synchronize the corresponding data to be synchronized to the target database.

[0101] In practice, the WAL logs can be analyzed, and the analyzed data to be synchronized can be matched according to the original rules (rule 1 and rule 2). If a match is found, the result is written to the target database corresponding to that rule.

[0102] In one embodiment, Figure 4 This is a mapping relationship structure diagram provided according to Embodiment 3 of the present invention. For example... Figure 4 As shown, the source database mapping relationship (combine.map) can include the source data: uuid_combine. The mapping relationship between the merge rule and the data synchronization rule (rule 1 and rule 2) (origin.map) includes: uuid_combine: uuid1, uuid2. The extraction rule mapping relationship (child.map) can include dn1: uuid_child1; dn2: uuid_child2; dn3: uuid_child3. The rule status (origin_rule.status) can be uuid1:enable and uuid2:disable.

[0103] In one embodiment, Figure 5 This is a flowchart of a rule management method provided according to Embodiment 3 of the present invention. Figure 5 As shown, the method includes:

[0104] Create data synchronization rules 1 (identified by uuid1) and 2 (identified by uuid2) for the source database. Based on the source database identifiers in rules 1 and 2, query whether a corresponding merge rule (uuid_combine) exists. If rules 1 and 2 do not exist in the merge rules, add the mapping relationships between rules 1 and the merge rule, and between rules 2 and the merge rule, to the mapping file combine.map. If rules 1 and 2 exist in the merge rules, query the extraction rules corresponding to the cluster nodes to determine if each cluster node has a corresponding extraction rule; if not, add or delete extraction rules to ensure that each cluster node has a corresponding extraction rule. When each cluster node has a corresponding extraction rule, perform WA log extraction.

[0105] In one embodiment, Figure 6 This is a flowchart of a data synchronization method provided according to Embodiment 3 of the present invention. This embodiment uses starting rule 1 and stopping rule 2 as an example to extract data from the write-ahead log. Figure 6 As shown, the method includes:

[0106] Start Rule 1 and stop Rule 2 according to the data synchronization requirements, and save the rule status of Rule 1 and Rule 2 to origin_rule.status. For example, the saved rule status can be uuid1:enable and uuid2:disable. Determine whether all data synchronization rules (Rule 1 and Rule 2) have stopped. If stopped, terminate the extraction rules; if stopped, determine the extraction rules corresponding to each cluster node, obtain the initial synchronization data, merge the initial synchronization data according to the data name as the data to be synchronized, and refresh the extraction rules corresponding to each cluster node. Parse the data to be synchronized in the Wal log. After parsing the record, determine the target synchronization data corresponding to Rule 1 and the data to be synchronized corresponding to Rule 2, and send the data to be synchronized to the target database according to the data to be synchronized corresponding to Rule 1 and Rule 2 respectively.

[0107] Example 4

[0108] Figure 7 This is a schematic diagram of a data synchronization device according to Embodiment 4 of the present invention. Figure 7 As shown, the device includes: a synchronization rule acquisition module 41, an extraction rule acquisition module 42, a synchronization data determination module 43, and a synchronization data synchronization module 44.

[0109] Among them, the synchronization rule acquisition module 41 is used to acquire the data synchronization rules applied in the merging rules of the source database.

[0110] The extraction rule acquisition module 42 is used to acquire the extraction rules of the cluster nodes of the corresponding source database.

[0111] The synchronization data determination module 43 is used to determine the data to be synchronized on the cluster nodes according to the data synchronization rules and extraction rules.

[0112] The data synchronization module 44 is used to determine the target database according to the data synchronization rules and synchronize the corresponding data to be synchronized to the target database.

[0113] In this embodiment of the invention, a synchronization rule acquisition module obtains the data synchronization rules applied in the merging rules of the source database, an extraction rule acquisition module obtains the extraction rules of the corresponding cluster nodes of the source database, a synchronization data determination module determines the data to be synchronized on the cluster nodes according to the data synchronization rules and the extraction rules, and a synchronization data synchronization module determines the target database according to the data synchronization rules and synchronizes the corresponding data to be synchronized to the target database. This achieves merged management of data synchronization rules belonging to the same database, and each data synchronization rule can be performed simultaneously. It also unifies the processing of various storage nodes, performing simultaneous parsing and processing on each storage node to obtain the data to be synchronized, reducing the resource consumption of data synchronization and improving the user experience.

[0114] In one embodiment, the data synchronization device further includes:

[0115] The rule creation module is used to create at least two data synchronization rules according to data synchronization requirements; the data synchronization rules include: the identifier of the data to be synchronized and the identifier of the target database;

[0116] The mapping relationship determination module is used to merge and store at least two data synchronization rules into a merge rule, and add the mapping relationship between the merge rule and the data synchronization rule to the mapping relationship set.

[0117] In one embodiment, the synchronization rule acquisition module 41 includes:

[0118] The synchronization rule lookup unit is used to find at least one data synchronization rule according to the mapping relationship of the merging rules;

[0119] The synchronization rule determination unit is used to select the data synchronization rules with the rule status of "started" from the data synchronization rules as the data synchronization rules for the application.

[0120] In one embodiment, the extraction rule acquisition module 42 includes:

[0121] The cluster node extraction unit is used to extract the cluster nodes corresponding to the source database.

[0122] The extraction rule determination unit is used to determine the extraction rules that match each cluster node.

[0123] In one embodiment, the extraction rule determination unit includes:

[0124] The extraction rule matching unit is used to determine whether each cluster node has a corresponding unique extraction rule.

[0125] The extraction rule update unit is used to add or delete extraction rules if they do not exist, so that each cluster node has a unique extraction rule.

[0126] In one embodiment, the synchronization data determination module 43 includes:

[0127] The initial data merging unit is used to extract initial synchronization data from cluster nodes according to data synchronization rules and merge initial synchronization data with the same data name.

[0128] The synchronization data determination unit is used to take the initial synchronization data as the data to be synchronized.

[0129] In one embodiment, the data synchronization module 44 includes:

[0130] The data extraction unit is used to extract the target synchronization data from the corresponding data to be synchronized according to the data synchronization rules.

[0131] The data synchronization unit is used to synchronize target synchronization data to the target database according to the target database identifier in the data synchronization rules.

[0132] The data synchronization device provided in the embodiments of the present invention can execute the data synchronization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0133] Example 5

[0134] Figure 8 This is a schematic diagram of the structure of an electronic device 10 implementing the data synchronization method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0135] like Figure 8As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0136] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0137] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data synchronization methods.

[0138] In some embodiments, the data synchronization method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data synchronization method X described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data synchronization method by any other suitable means (e.g., by means of firmware).

[0139] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0143] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0144] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0145] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0146] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data synchronization method, characterized in that, include: Obtain the data synchronization rules applied in the merge rules of the source database; wherein, the merge rules are generated by merging all data synchronization rules in the source database; Obtain the extraction rules for the cluster nodes corresponding to the source database; The data to be synchronized is determined on the cluster node according to the data synchronization rules and the extraction rules. The target database is determined according to the data synchronization rules, and the corresponding data to be synchronized is synchronized to the target database. Prior to the data synchronization rules applied in the merging rules for obtaining the source database, the following are also included: Create at least two data synchronization rules according to the data synchronization requirements; wherein, the data synchronization rules include: the identifier of the data to be synchronized and the identifier of the target database; At least two of the data synchronization rules are merged and stored as the merged rule, and a mapping relationship between the merged rule and the data synchronization rule is added to the mapping relationship set; The data synchronization rules applied in the merging rules of the source database include: At least one of the data synchronization rules is found according to the mapping relationship of the merging rules; The data synchronization rules whose rule status is "started" among the data synchronization rules are used as the data synchronization rules of the application. The extraction rules for obtaining the cluster nodes corresponding to the source database include: Extract the cluster node corresponding to the source database; Determine the extraction rules that match each of the cluster nodes; The step of determining the extraction rules that match each of the cluster nodes further includes: Determine whether each cluster node has a corresponding unique extraction rule; If it does not exist, the extraction rule is added or deleted so that each cluster node has a unique extraction rule; wherein, when any cluster node does not have a corresponding extraction rule, an extraction rule is created for that cluster node; when any extraction rule does not have a corresponding cluster node, the extraction rule is deleted.

2. The method according to claim 1, characterized in that, The step of determining the data to be synchronized on the cluster nodes according to the data synchronization rules and the extraction rules includes: According to the data synchronization rules, initial synchronization data is extracted from the cluster nodes and the initial synchronization data with the same data name are merged. The initial synchronization data is used as the data to be synchronized.

3. The method according to claim 1, characterized in that, The step of determining the target database according to the data synchronization rules and synchronizing the corresponding data to be synchronized to the target database includes: Extract the target synchronization data from the corresponding data to be synchronized according to the data synchronization rules; The target synchronization data is synchronized to the target database according to the target database identifier of the data synchronization rules.

4. A data synchronization device, characterized in that, include: The synchronization rule acquisition module is used to acquire the data synchronization rules applied in the merging rules of the source database; wherein, the merging rules are generated by merging all data synchronization rules in the source database. The extraction rule acquisition module is used to acquire the extraction rules of the cluster nodes corresponding to the source database. The data synchronization determination module is used to determine the data to be synchronized on the cluster node according to the data synchronization rules and the extraction rules. The data synchronization module is used to determine the target database according to the data synchronization rules and synchronize the corresponding data to be synchronized to the target database. The data synchronization device further includes: The rule creation module is used to create at least two data synchronization rules according to data synchronization requirements; wherein, the data synchronization rules include: the identifier of the data to be synchronized and the identifier of the target database; The rule creation module is used to merge and store at least two data synchronization rules into the merged rule, and add the mapping relationship between the merged rule and the data synchronization rule to the mapping relationship set; The synchronization rule acquisition module includes: A synchronization rule lookup unit is used to look up at least one of the data synchronization rules according to the mapping relationship of the merging rules; A synchronization rule determination unit is used to determine the data synchronization rule whose rule status is "started" among the data synchronization rules as the data synchronization rule of the application. The extraction rule acquisition module includes: Cluster node extraction unit, used to extract the cluster nodes corresponding to the source database; An extraction rule determination unit is used to determine the extraction rules that match each of the cluster nodes; The extraction rule determination unit includes: An extraction rule matching unit is used to determine whether each cluster node has a corresponding unique extraction rule. An extraction rule update unit is used to add or delete extraction rules if they do not exist, so that each cluster node has a unique extraction rule; wherein, when any cluster node does not have a corresponding extraction rule, an extraction rule is created for that cluster node; when any extraction rule does not have a corresponding cluster node, the extraction rule is deleted.

5. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the data synchronization method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data synchronization method of any one of claims 1-3.

Images