A data synchronization method, apparatus, device, and storage medium

Abstract

The application discloses a data synchronization method and device, equipment and a storage medium, comprising: generating an external synchronization task according to a synchronization link and a synchronization list by a master cluster, and sending the external synchronization task to a backup cluster through the synchronization link; when the backup cluster receives the external synchronization task, calling remote procedure call service RPC information of the master cluster, and generating an internal synchronization task according to the RPC information; executing the internal synchronization task by the backup cluster, so as to synchronize the metadata to be synchronized to a backup cluster distributed key-value storage database FoundationDB, and synchronize the data to be synchronized to the backup cluster object storage. In the data transmission process, no additional node resources are introduced, and only the master cluster and the backup cluster need to synchronize data based on the generated task, so that efficient and stable data synchronization of the database on the computing separation cloud is realized.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data synchronization method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of 5G technology, numerous applications generate massive amounts of data. To ensure data security, backing up and storing data across multiple nodes has become a primary strategy for mitigating risks. Cloud databases with compute-storage separation store metadata and data on platforms other than compute nodes. For example, metadata is stored in the distributed key-value store FoundationDB, while data is stored in object storage. This makes the use of compute and storage nodes more flexible, significantly improving resource utilization and performance.

[0003] However, due to the isolation of data storage and computing nodes, data synchronization needs to cross multiple dimensions, which usually requires a lot of transmission resources. Therefore, ensuring efficient and stable transmission of data and metadata during full synchronization has become a top priority. Summary of the Invention

[0004] This invention provides a data synchronization method, apparatus, device, and storage medium to achieve efficient and stable data synchronization.

[0005] According to a first aspect of this invention, a data synchronization method is provided, applied to a cloud database comprising a primary cluster and a backup cluster, the method comprising:

[0006] The master cluster creates a synchronization link and a synchronization list, wherein the synchronization list contains metadata to be synchronized in the master cluster's distributed key-value storage database FoundationDB and data to be synchronized in the master cluster's object storage.

[0007] The primary cluster generates external synchronization tasks based on the synchronization link and the synchronization list, and sends the external synchronization tasks to the backup cluster through the synchronization link.

[0008] When the backup cluster receives the external synchronization task, it retrieves the Remote Procedure Call (RPC) service information of the primary cluster and generates an internal synchronization task based on the RPC information.

[0009] The internal synchronization task is executed through the backup cluster to synchronize the metadata to be synchronized to the backup cluster FoundationDB and the data to be synchronized to the backup cluster object storage.

[0010] According to another aspect of the present invention, a data synchronization apparatus is provided, applied to a cloud database comprising a primary cluster and a backup cluster, the apparatus comprising:

[0011] The synchronization link and synchronization list creation module is used to create a synchronization link and a synchronization list through the master cluster. The synchronization list includes metadata to be synchronized in the distributed key-value storage database FoundationDB of the master cluster and data to be synchronized in the object storage of the master cluster.

[0012] An external synchronization task generation module is used to generate external synchronization tasks through the primary cluster based on the synchronization link and the synchronization list, and send the external synchronization tasks to the backup cluster through the synchronization link.

[0013] An internal synchronization task generation module is used to retrieve the Remote Procedure Call (RPC) service information of the primary cluster when the backup cluster receives the external synchronization task, and generate an internal synchronization task based on the RPC information.

[0014] An internal synchronization task execution module is used to execute the internal synchronization task through the backup cluster to synchronize the metadata to be synchronized to the backup cluster FoundationDB and to synchronize the data to be synchronized to the backup cluster object storage.

[0015] According to another aspect of the present invention, a terminal device is provided, the terminal device comprising: one or more processors;

[0016] Storage device for storing one or more programs.

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a storage medium for computer-executable instructions is provided, on which a computer program is stored, which, when executed by a processor, implements the method as described in any of the embodiments of the present invention.

[0019] The technical solution of this invention involves the primary cluster creating an external synchronization task and sending it to the backup cluster. The backup cluster then creates and executes an internal synchronization task based on the received external synchronization task to synchronize the data and metadata in the storage space associated with the primary cluster. No additional node resources are required during data transmission; only data synchronization between the primary and backup clusters is needed based on the generated task. This achieves efficient and stable data synchronization of the storage-compute separation cloud database.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a data synchronization method provided according to Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart of a data synchronization method provided according to Embodiment 2 of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a data synchronization device according to Embodiment 3 of the present invention;

[0025] Figure 4 This invention provides a structural block diagram of a terminal device. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or terminal device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or terminal devices.

[0028] Example 1

[0029] Figure 1 This is a flowchart illustrating a data synchronization method provided in an embodiment of the present invention. This embodiment is applicable to data synchronization of cloud databases. The method can be executed by a data synchronization device, which can be implemented in hardware and / or software, and can be integrated into a terminal device. Figure 1 As shown, the method includes:

[0030] Step S101: Create a synchronization link and a synchronization list through the master cluster.

[0031] Specifically, in this embodiment, the primary cluster is mainly used for computation, while the primary cluster's distributed key-value store database FoundationDB and primary cluster object storage are used to store metadata and data, respectively. Furthermore, the primary cluster and its associated FoundationDB and object storage are independent. This embodiment primarily synchronizes data from the primary cluster's FoundationDB and object storage to the backup cluster's FoundationDB and object storage, respectively. For metadata synchronization, a Remote Procedure Call (RPC) service is deployed on both the primary and backup clusters. This eliminates direct interaction between the FoundationDB instances in the primary and backup clusters; metadata requests from the backup cluster to the primary cluster are proxied by the RPC service in the backup cluster, avoiding cross-domain access issues for FoundationDB. Additionally, data is stored in the object storage S3 service. For data synchronization, the backup cluster can directly read data from the primary cluster's object storage, thereby reducing intermediate nodes and lowering the load on the primary cluster nodes.

[0032] Optionally, a synchronization link and a synchronization list are created through the primary cluster, including: obtaining the attribute information of the backup cluster through the primary cluster and creating a synchronization link based on the attribute information, wherein the attribute information includes the management node IP, username, and password of the backup cluster; receiving the user's synchronization request through the primary cluster and creating a synchronization list based on the synchronization request, wherein the synchronization list contains multiple lists to be synchronized, and the metadata and data to be synchronized are located in the lists to be synchronized.

[0033] Specifically, this implementation method creates a synchronization link and a synchronization list through the primary cluster. Before synchronizing data, the primary cluster first obtains the attribute information of the backup cluster. This data information may include the IP address of the backup cluster's management node, the backup cluster's username, and the backup cluster's password. The management node is the coordinator node, and its IP address is 172.16.0.17. Since there are multiple nodes in the backup cluster, this implementation method can use the management node's username (did) and password (111) as the backup cluster's username and password. Alternatively, the username and password of a specific node in the backup cluster can be selected as the backup cluster's username and password; this implementation method does not impose any limitations on this. After obtaining the attribute information of the backup cluster, a synchronization link can be created based on the attribute information. For example, the SQL example for creating a synchronization link is as follows: `sync add link 'link1' connect to '172.16.0.17' user 'did' identifiedby '111'`, where the name of the created synchronization link is `link1`. Of course, this implementation method is only an example and does not impose any specific limitations on the content of the SQL for creating a synchronization link.

[0034] In addition, the primary cluster also receives synchronization requests from users. These requests contain a list of items that the user needs to synchronize. Therefore, a synchronization list can be created based on these requests. This synchronization list contains at least one item to be synchronized, which in turn contains metadata and data to be synchronized. For example, the SQL for creating a synchronization list is as follows: `sync add table vc1.db1.t1.vc2.db2.t2 with list 'list1'`. Here, `vc1.db1.t1` represents the primary cluster virtual cluster name, primary cluster database name, and primary cluster table name; `vc2.db2.t2` represents the backup cluster virtual cluster name, backup cluster database name, and backup cluster table name; and `list1` represents the synchronization list name. Of course, this implementation is only an example and does not impose any specific limitations on the content of the SQL for creating the synchronization list.

[0035] Step S102: The primary cluster generates an external synchronization task based on the synchronization link and the synchronization list, and sends the external synchronization task to the backup cluster through the synchronization link.

[0036] Optionally, the main cluster generates an external synchronization task based on the synchronization link and the synchronization list, including: receiving the target execution node identifier in the backup cluster specified by the user through the main cluster; obtaining the link name of the created synchronization link; and creating an external synchronization task based on the link name, the target execution node identifier, and the synchronization list.

[0037] Specifically, in this embodiment, after the primary cluster creates the synchronization link and synchronization list, it generates external synchronization tasks based on these links and lists. For example, the SQL example for creating an external synchronization task is as follows: `sync tables with link link 'link1' list 'list1' [warehouse 'wh1']`. Here, the name `link1` is extracted from the created synchronization link, and the name `list1` is extracted from the created synchronization list. `warehouse 'wh1'` represents the identifier of the target execution node in the backup cluster that actually performs the synchronization operation. This target execution node can be specified by the user or pre-configured. This embodiment is merely an example and does not specify a particular number of target execution nodes. Furthermore, in this embodiment, "external synchronization task" refers to an external object to the backup cluster, i.e., a task generated by the primary cluster. Tasks generated by the backup cluster itself are called internal tasks. Therefore, in this embodiment, "internal" and "external" primarily refer to the naming of the task's generating entity.

[0038] It should be noted that when the primary cluster generates an external synchronization task, the external synchronization task mainly includes information such as the synchronization link for communication between the primary cluster and the backup cluster, the target execution node in the backup cluster that executes the task, and the specific data source that needs to be synchronized. Therefore, the primary cluster will send the above-mentioned external synchronization task to the backup cluster through the created synchronization link, so that the backup cluster can perform the corresponding data synchronization operation according to the instructions of the external synchronization task.

[0039] Step S103: When the standby cluster receives an external synchronization task, it retrieves the Remote Procedure Call (RPC) service information of the primary cluster and generates an internal synchronization task based on the RPC information.

[0040] Optionally, an internal synchronization task is generated based on RPC information, including: generating an internal synchronization task through the management node of the standby cluster based on RPC information, wherein the RPC information includes the server instance name, RPC service node IP, RPC service port and cluster conflict-range versioning (CRV) transaction version; and establishing a subordinate relationship between the internal synchronization task and the external synchronization task.

[0041] Specifically, the standby cluster receives external synchronization tasks sent by the primary cluster through its management node. Upon receiving the task, the management node sends an RPC information retrieval request to the primary cluster. The primary cluster then sends pre-configured RPC information to the standby cluster based on this request. This RPC information includes the server instance name, RPC service node IP, RPC service port, and cluster CRV transaction version. After receiving the RPC information, the standby cluster's management node generates an internal synchronization task. For example, the SQL for creating an internal synchronization task is as follows: `Sync add task'10' with link_info='inst:ip1,ip2,ip3:port' crv='123'`, where '10' represents the task identifier, 'inst' represents the server instance name, 'ip1,ip2,ip3' represent the RPC service node IPs, 'port' represents the RPC service port, and '123' represents the cluster CRV transaction version. This is merely an example and does not impose specific limitations on the SQL content for creating the internal synchronization task.

[0042] It should be noted that in this embodiment, after the management node of the standby cluster creates the internal synchronization task, it establishes a subordinate relationship between the internal and external synchronization tasks. That is, when the internal synchronization task of the standby cluster is completed, the external synchronization task to which it belongs is also completed. The external synchronization task contains the relevant tasks that the primary cluster requires the standby cluster to execute, while the internal synchronization task contains information about the RPC service proxy that the standby cluster requires the target execution node to call when performing data synchronization. After the standby cluster management node distributes the tasks to the standby cluster compute nodes, the compute nodes determine whether there are metadata changes based on the cluster CRV version information. If there are metadata changes, they synchronize table information and request metadata from the RPC service. Once a certain amount of data is reached, each compute node generates a metadata acquisition and writing task to write metadata to the standby cluster FoundationDB until all metadata that needs to be synchronized is synchronized. The certain amount reached can be specified by the user by adjusting parameters.

[0043] Step S104: Execute an internal synchronization task through the standby cluster to synchronize the metadata to be synchronized to the standby cluster FoundationDB and the data to be synchronized to the standby cluster object storage.

[0044] Optionally, the internal synchronization task is executed through the standby cluster, including: the standby cluster's management node generates a metadata synchronization command based on the internal and external synchronization tasks, and sends the metadata synchronization command to the target execution node, so that the target execution node connects to the primary cluster FoundationDB via RPC information and synchronizes the metadata to be synchronized in the primary cluster FoundationDB to the standby cluster FoundationDB; when the standby cluster's management node receives the metadata synchronization completion message from the target execution node, it generates a data synchronization command based on the internal and external synchronization tasks, and sends the data synchronization command to the target execution node, so that the target execution node synchronizes the data to be synchronized in the primary cluster object storage to the standby cluster object storage.

[0045] Specifically, after generating an internal synchronization task, the management node of the standby cluster queries its subordinate external synchronization tasks to obtain the target execution node. Then, based on the internal and external synchronization tasks, it generates a metadata synchronization command and sends the generated command to the target execution node. For example, the SQL example for generating the metadata synchronization command is as follows:

[0046] The command `Sync table metadata 'vc00001:test:t1:vc00001:test:t1:1:1:1' with task '10'` specifies the metadata sequence. In `vc00001:test:t1:vc00001:test:t1:1:1:1`, the first group `vc00001:test:t1` represents the source primary cluster, source database, and source table, while the second group `vc00001:test:t1` represents the target backup cluster, target database, and target table. This demonstrates that although the names are the same, their meanings differ depending on their position. The `metaversion:truncver:config` directive represents the metadata version number, truncated version number, and configuration file—equivalent to the configuration information related to metadata synchronization. The `with task '10'` option indicates that the metadata synchronization command includes information about the RPC service proxy contained in the internal synchronization task. Therefore, after receiving the metadata synchronization command, the target execution node will connect to the primary cluster FoundationDB via RPC information and synchronize the metadata to be synchronized in the primary cluster FoundationDB to the backup cluster FoundationDB. Thus, during metadata synchronization, the RPC service deployed in the primary and backup clusters eliminates the need for direct interaction between the primary and backup FoundationDBs. The metadata requests from the backup cluster to the primary cluster are proxied by the RPC service in the backup cluster, thereby avoiding the problem of cross-domain access to FoundationDB.

[0047] In this implementation, after the target execution node completes metadata synchronization, it generates a metadata synchronization completion message and sends it back to the management node. Upon receiving the message, the management node generates a data synchronization command based on internal and external synchronization tasks and sends it to the target execution node. Therefore, in this implementation, synchronization occurs sequentially: metadata synchronization is performed first, followed by data synchronization. Metadata primarily describes information about the data, such as attributes, structure, source, and purpose. For example, the SQL example for generating the data synchronization command is as follows:

[0048] The command `Sync table data 'vc00001:test:t1:vc00001:test:t1:1:1:1:config' withtask '10'` executes a data synchronization command. The specific meanings of each parameter are largely the same as in the SQL example of the metadata synchronization command described above. The last parameter, `config`, contains the data storage connection information of the main cluster. The two are distinguished by "data" and "metadata". Upon receiving the data synchronization command, the target execution node can directly read the data to be synchronized from the main cluster object storage, reducing intermediate nodes and lowering the load on the main cluster nodes. Therefore, data synchronization does not require an RPC server proxy. Thus, in this implementation, cross-domain access is avoided during metadata synchronization to ensure metadata stability, while data synchronization reduces intermediate nodes by reading directly from the source storage, lowering the load on the source cluster. Furthermore, after the backup cluster management node distributes the task to the backup cluster compute nodes, the compute nodes determine whether there are data changes through metadata information. If there are data changes and a certain amount of data is reached, data acquisition and writing tasks are generated on each compute node to write data to the backup cluster object storage until all data that needs to be synchronized is synchronized. The certain amount reached at this time can also be specified by the user by adjusting parameters.

[0049] Optionally, after the backup cluster executes the internal synchronization task, the process further includes: when the backup cluster's management node receives the data synchronization completion message from the target execution node, it determines that the internal synchronization task has been completed; when the backup cluster's management node determines that the internal synchronization task has been completed, it generates an external synchronization task completion message and sends the external synchronization task completion message to the primary cluster; the primary cluster deletes the synchronization link and synchronization list based on the external synchronization task completion message.

[0050] Specifically, after the target execution node in the standby cluster completes data synchronization, it generates a data synchronization completion message and sends it to the management node. Therefore, when the management node of the standby cluster receives the data completion message, it determines that the internal synchronization task has been completed. Since the internal synchronization task is subordinate to the management node corresponding to the external synchronization task, when the completion of the internal synchronization task is confirmed, an external synchronization task completion message is generated and sent to the primary cluster via the synchronization link. To avoid unnecessary resource consumption, the primary cluster deletes the previously created synchronization link and synchronization list after receiving the external synchronization task completion message.

[0051] The technical solution of this invention involves the primary cluster creating an external synchronization task and sending it to the backup cluster. The backup cluster then creates and executes an internal synchronization task based on the received external synchronization task to synchronize the data and metadata in the storage space associated with the primary cluster. No additional node resources are required during data transmission; only data synchronization between the primary and backup clusters is needed based on the generated task. This achieves efficient and stable data synchronization of the storage-compute separation cloud database.

[0052] Example 2

[0053] Figure 2 This is a flowchart of a data synchronization method provided by an embodiment of the present invention. Based on the above embodiment, after executing the internal synchronization task through the backup cluster, this embodiment further includes: detecting the metadata in the backup cluster's FoundationDB and the data in the backup cluster's object storage through the backup cluster; when inconsistencies are detected between the metadata in the backup cluster's FoundationDB and the primary cluster's FoundationDB, or between the data in the backup cluster's object storage and the primary cluster's object storage, a synchronization anomaly alarm is generated. Figure 2 As shown, the method includes:

[0054] Step S201: Create a synchronization link and a synchronization list through the master cluster.

[0055] Optionally, a synchronization link and a synchronization list are created through the primary cluster, including: obtaining the attribute information of the backup cluster through the primary cluster and creating a synchronization link based on the attribute information, wherein the attribute information includes the management node IP, username, and password of the backup cluster; receiving the user's synchronization request through the primary cluster and creating a synchronization list based on the synchronization request, wherein the synchronization list contains multiple lists to be synchronized, and the metadata and data to be synchronized are located in the lists to be synchronized.

[0056] Step S202: The primary cluster generates an external synchronization task based on the synchronization link and the synchronization list, and sends the external synchronization task to the backup cluster through the synchronization link.

[0057] Optionally, the main cluster generates an external synchronization task based on the synchronization link and the synchronization list, including: receiving the target execution node identifier in the backup cluster specified by the user through the main cluster; obtaining the link name of the created synchronization link; and creating an external synchronization task based on the link name, the target execution node identifier, and the synchronization list.

[0058] Step S203: When the standby cluster receives an external synchronization task, it retrieves the Remote Procedure Call (RPC) service information of the primary cluster and generates an internal synchronization task based on the RPC information.

[0059] Optionally, an internal synchronization task is generated based on RPC information, including: generating an internal synchronization task through the management node of the backup cluster based on RPC information, wherein the RPC information includes the server instance name, RPC service node IP, RPC service port and cluster CRV transaction version; and establishing a subordinate relationship between the internal synchronization task and the external synchronization task.

[0060] Step S204: Execute an internal synchronization task through the standby cluster to synchronize the metadata to be synchronized to the standby cluster FoundationDB and the data to be synchronized to the standby cluster object storage.

[0061] Optionally, the internal synchronization task is executed through the standby cluster, including: the standby cluster's management node generates a metadata synchronization command based on the internal and external synchronization tasks, and sends the metadata synchronization command to the target execution node, so that the target execution node connects to the primary cluster FoundationDB via RPC information and synchronizes the metadata to be synchronized in the primary cluster FoundationDB to the standby cluster FoundationDB; when the standby cluster's management node receives the metadata synchronization completion message from the target execution node, it generates a data synchronization command based on the internal and external synchronization tasks, and sends the data synchronization command to the target execution node, so that the target execution node synchronizes the data to be synchronized in the primary cluster object storage to the standby cluster object storage.

[0062] Optionally, after the backup cluster executes the internal synchronization task, the process further includes: when the backup cluster's management node receives the data synchronization completion message from the target execution node, it determines that the internal synchronization task has been completed; when the backup cluster's management node determines that the internal synchronization task has been completed, it generates an external synchronization task completion message and sends the external synchronization task completion message to the primary cluster; the primary cluster deletes the synchronization link and synchronization list based on the external synchronization task completion message.

[0063] Step S205: Detect the metadata in the backup cluster FoundationDB and the data in the backup cluster object storage through the backup cluster.

[0064] Specifically, after the backup cluster completes the synchronization of metadata and data in this embodiment, it checks the metadata in the backup cluster's FoundationDB and the data in the backup cluster's object storage. During the check, the backup cluster requests viewing permissions for the primary cluster's FoundationDB and object storage. The primary cluster verifies the backup cluster's identity; if the verification is successful, the primary cluster grants the backup cluster viewing permissions for the primary cluster's FoundationDB and object storage. Once the backup cluster obtains viewing permissions, it compares the synchronization status with the actual situation in the primary cluster's FoundationDB and object storage. Specifically, it checks whether the metadata contained in the backup cluster's FoundationDB and the primary cluster's FoundationDB are the same, and whether the data contained in the backup cluster's object storage and the primary cluster's object storage are the same.

[0065] To improve detection efficiency during comparisons, data comparison or block comparison can be used. For example, for metadata comparison, first check if the amount of metadata in the backup cluster's FoundationDB and the primary cluster's FoundationDB is the same. If it is determined that the amount of metadata in the backup cluster's FoundationDB is less than the amount of metadata in the primary cluster's FoundationDB, then it is directly determined that there are missing items in the backup cluster's metadata synchronization. If it is determined that the amount of metadata is the same, the metadata in the backup cluster's FoundationDB and the primary cluster's FoundationDB can be partitioned according to type or time period, and the data in the corresponding partitions can be compared to detect whether there are metadata synchronization errors.

[0066] Step S206: When it is detected that the metadata in the backup cluster FoundationDB is inconsistent with that in the primary cluster FoundationDB, or that the data in the backup cluster object storage is inconsistent with that in the primary cluster object storage, a synchronization anomaly alarm message is generated.

[0067] Specifically, if any of the following events occur, as determined by the above detection, the metadata in the backup cluster's FoundationDB is inconsistent with that in the primary cluster's FoundationDB, or the data in the backup cluster's object storage is inconsistent with that in the primary cluster's object storage: then the backup cluster synchronization has failed. In this case, a synchronization anomaly alarm message will be generated. This alarm message can be in the form of an image or an audio message. This embodiment does not limit the specific form of the alarm message. The generated synchronization anomaly alarm message will be played to alert maintenance personnel that a synchronization error has occurred. This allows maintenance personnel to promptly maintain the software modules or hardware devices based on the alarm message, thereby improving the efficiency and accuracy of data synchronization.

[0068] The technical solution of this invention involves the primary cluster creating an external synchronization task and sending it to the backup cluster. The backup cluster then creates and executes an internal synchronization task based on the received external synchronization task to synchronize the data and metadata in the storage space associated with the primary cluster. No additional node resources are required during data transmission; only data synchronization between the primary and backup clusters is needed based on the generated task. This achieves efficient and stable data synchronization of the storage-compute separation cloud database.

[0069] Example 3

[0070] Figure 3 This is a schematic diagram of an automatic driving mode switching device provided in an embodiment of the present invention. Figure 3 As shown, the device includes: a synchronization link and a synchronization list creation module 310, an external synchronization task generation module 320, an internal synchronization task generation module 330, and an internal synchronization task execution module 340.

[0071] The synchronization link and synchronization list creation module 310 is used to create a synchronization link and a synchronization list through the main cluster. The synchronization list contains metadata to be synchronized in the distributed key-value storage database FoundationDB of the main cluster and data to be synchronized in the object storage of the main cluster.

[0072] The external synchronization task generation module 320 is used to generate external synchronization tasks through the main cluster based on the synchronization link and the synchronization list, and send the external synchronization tasks to the backup cluster through the synchronization link.

[0073] The internal synchronization task generation module 330 is used to retrieve the remote procedure call service (RPC) information of the primary cluster when the standby cluster receives an external synchronization task, and generate an internal synchronization task based on the RPC information.

[0074] The internal synchronization task execution module 340 is used to execute internal synchronization tasks through the standby cluster to synchronize the metadata to be synchronized to the standby cluster FoundationDB and the data to be synchronized to the standby cluster object storage.

[0075] Optionally, a synchronization link and synchronization list creation module is used to obtain the attribute information of the backup cluster through the master cluster and create a synchronization link based on the attribute information. The attribute information includes the management node IP of the backup cluster, the username of the backup cluster, and the password of the backup cluster.

[0076] The system receives synchronization requests from users through the main cluster and creates a synchronization list based on these requests. The synchronization list contains multiple lists to be synchronized, with metadata and data to be synchronized located in the lists to be synchronized.

[0077] Optionally, an external synchronization task generation module is used to receive the target execution node identifier in the backup cluster specified by the user through the main cluster;

[0078] Obtain the link name of the created synchronization link, and create an external synchronization task based on the link name, the target execution node identifier, and the synchronization list.

[0079] Optionally, an internal synchronization task generation module is used to generate internal synchronization tasks based on RPC information through the management node of the backup cluster. The RPC information includes the server instance name, RPC service node IP, RPC service port, and cluster CRV transaction version.

[0080] Establish a hierarchical relationship between internal synchronization tasks and external synchronization tasks.

[0081] Optionally, the internal synchronization task execution module is used to generate metadata synchronization commands based on internal and external synchronization tasks through the management node of the standby cluster, and send the metadata synchronization commands to the target execution node so that the target execution node can connect to the primary cluster FoundationDB through RPC information and synchronize the metadata to be synchronized in the primary cluster FoundationDB to the standby cluster FoundationDB.

[0082] When the management node of the standby cluster receives the metadata synchronization completion message from the target execution node, it generates a data synchronization command based on the internal and external synchronization tasks, and sends the data synchronization command to the target execution node so that the target execution node can synchronize the data to be synchronized in the primary cluster object storage to the standby cluster object storage.

[0083] Optionally, the device also includes a synchronization link and a synchronization list deletion module, used to determine that the internal synchronization task has been completed when the management node of the standby cluster receives a data synchronization completion message from the target execution node;

[0084] When the management node of the standby cluster determines that the internal synchronization task has been completed, it generates an external synchronization task completion message and sends the external synchronization task completion message to the primary cluster.

[0085] The main cluster deletes the synchronization link and synchronization list based on the completion message of the external synchronization task.

[0086] Optionally, the device also includes an alarm module for detecting metadata in the backup cluster FoundationDB and data in the backup cluster object storage via the backup cluster;

[0087] When inconsistencies are detected between the metadata in the backup cluster's FoundationDB and the primary cluster's FoundationDB, or between the data in the backup cluster's object storage and the primary cluster's object storage, a synchronization error alarm is generated.

[0088] The data synchronization device provided in this embodiment of the invention can execute a data synchronization method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0089] Example 4

[0090] Figure 4 A schematic diagram of a terminal device 10 that can be used to implement embodiments of the present invention is shown. The terminal device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The terminal device can also represent various forms of mobile devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0091] The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0092] like Figure 4As shown, the terminal device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the terminal device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0093] Multiple components in terminal device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows terminal device 10 to exchange information / data with other terminal devices through computer networks such as the Internet and / or various telecommunications networks.

[0094] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data synchronization methods.

[0095] In some embodiments, the data synchronization method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on terminal device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data synchronization method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data synchronization method by any other suitable means (e.g., by means of firmware).

[0096] Various embodiments of the apparatuses and techniques described above herein can be implemented in digital electronic circuit devices, integrated circuit devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), device-on-a-chip (SoCs), complex programmable logic terminal devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable device including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage device, at least one input device, and at least one output device, and transmitting data and instructions to the storage device, the at least one input device, and the at least one output device.

[0097] Computer programs used to implement the data synchronization method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other business-uninterrupted data migration device, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0098] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution apparatus, device, or terminal device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage terminal devices, magnetic storage terminal devices, or any suitable combination thereof.

[0099] To provide interaction with a user, the apparatus and techniques described herein can be implemented on a terminal device having: a display device (e.g., a touchscreen) for displaying information to the user; and buttons through which the user can provide input to the terminal device. Other types of apparatus can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including voice input, speech input, or haptic input).

[0100] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0101] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data synchronization method, characterized in that, Applied to cloud databases comprising a primary cluster and a backup cluster, the method includes: The master cluster creates a synchronization link and a synchronization list, wherein the synchronization list contains metadata to be synchronized in the master cluster's distributed key-value storage database FoundationDB and data to be synchronized in the master cluster's object storage. The primary cluster generates external synchronization tasks based on the synchronization link and the synchronization list, and sends the external synchronization tasks to the backup cluster through the synchronization link. When the backup cluster receives the external synchronization task, it retrieves the Remote Procedure Call (RPC) service information of the primary cluster and generates an internal synchronization task based on the RPC information. The internal synchronization task is executed through the backup cluster to synchronize the metadata to be synchronized to the backup cluster FoundationDB and the data to be synchronized to the backup cluster object storage. The step of generating an internal synchronization task based on the RPC information includes: generating the internal synchronization task based on the RPC information through the management node of the backup cluster, and establishing a subordinate relationship between the internal synchronization task and the external synchronization task; The step of executing the internal synchronization task through the backup cluster includes: generating a metadata synchronization command through the management node of the backup cluster based on the internal synchronization task and the external synchronization task, and sending the metadata synchronization command to the target execution node, so that the target execution node connects to the primary cluster FoundationDB through the RPC information and synchronizes the metadata to be synchronized in the primary cluster FoundationDB to the backup cluster FoundationDB. When the management node of the backup cluster receives the metadata synchronization completion message from the target execution node, it generates a data synchronization command based on the internal synchronization task and the external synchronization task, and sends the data synchronization command to the target execution node so that the target execution node synchronizes the data to be synchronized in the primary cluster object storage to the backup cluster object storage.

2. The method according to claim 1, characterized in that, The creation of synchronization links and synchronization lists through the main cluster includes: The primary cluster obtains the attribute information of the backup cluster and creates the synchronization link based on the attribute information, wherein the attribute information includes the management node IP of the backup cluster, the username of the backup cluster, and the password of the backup cluster. The main cluster receives user synchronization requests and creates a synchronization list based on the synchronization requests. The synchronization list contains multiple lists to be synchronized, and the metadata and data to be synchronized are located in the lists to be synchronized.

3. The method according to claim 1, characterized in that, The process of generating external synchronization tasks through the main cluster based on the synchronization link and the synchronization list includes: The primary cluster receives the target execution node identifier in the backup cluster specified by the user. Obtain the link name of the created synchronization link, and create the external synchronization task based on the link name, the target execution node identifier, and the synchronization list.

4. The method according to claim 1, characterized in that, The RPC information includes the server instance name, RPC service node IP, RPC service port, and cluster conflict range version control (CRV) transaction version.

5. The method according to claim 1, characterized in that, After executing the internal synchronization task through the backup cluster, the process further includes: When the management node of the backup cluster receives the data synchronization completion message from the target execution node, it determines that the internal synchronization task has been completed. When the management node of the backup cluster determines that the internal synchronization task has been completed, it generates an external synchronization task completion message and sends the external synchronization task completion message to the primary cluster. The main cluster deletes the synchronization link and the synchronization list based on the external synchronization task completion message.

6. The method according to claim 1, characterized in that, After executing the internal synchronization task through the backup cluster, the process further includes: The backup cluster is used to detect the metadata in the backup cluster's FoundationDB and the data in the backup cluster's object storage. When it is detected that the metadata in the backup cluster FoundationDB is inconsistent with that in the primary cluster FoundationDB, or that the data in the backup cluster object storage is inconsistent with that in the primary cluster object storage, a synchronization error alarm message is generated.

7. A data synchronization device, characterized in that, The device, applicable to cloud databases comprising a primary cluster and a backup cluster, includes: The synchronization link and synchronization list creation module is used to create a synchronization link and a synchronization list through the master cluster. The synchronization list includes metadata to be synchronized in the distributed key-value storage database FoundationDB of the master cluster and data to be synchronized in the object storage of the master cluster. An external synchronization task generation module is used to generate external synchronization tasks through the primary cluster based on the synchronization link and the synchronization list, and send the external synchronization tasks to the backup cluster through the synchronization link. An internal synchronization task generation module is used to retrieve the Remote Procedure Call (RPC) service information of the primary cluster when the backup cluster receives the external synchronization task, and generate an internal synchronization task based on the RPC information. An internal synchronization task execution module is used to execute the internal synchronization task through the backup cluster to synchronize the metadata to be synchronized to the backup cluster FoundationDB and synchronize the data to be synchronized to the backup cluster object storage. An internal synchronization task generation module is used to generate the internal synchronization task based on the RPC information through the management node of the backup cluster, and to establish the subordinate relationship between the internal synchronization task and the external synchronization task. The internal synchronization task generation module is further configured to generate a metadata synchronization command through the management node of the backup cluster based on the internal synchronization task and the external synchronization task, and send the metadata synchronization command to the target execution node, so that the target execution node connects to the primary cluster FoundationDB through the RPC information and synchronizes the metadata to be synchronized in the primary cluster FoundationDB to the backup cluster FoundationDB. When the management node of the backup cluster receives the metadata synchronization completion message from the target execution node, it generates a data synchronization command based on the internal synchronization task and the external synchronization task, and sends the data synchronization command to the target execution node so that the target execution node synchronizes the data to be synchronized in the primary cluster object storage to the backup cluster object storage.

8. A terminal device, characterized in that, The terminal device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A storage medium for computer-executable instructions, wherein a computer program is stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images