Log synchronization method, device, system, equipment and storage medium

Through message center analysis and broadcast data source identification, the synchronization inconsistency caused by data migration of shard nodes in distributed databases is solved, and data consistency and efficient synchronization between shard nodes are achieved.

CN114817190BActive Publication Date: 2024-11-12ALIBABA CLOUD COMPUTING CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210474789.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-29
Publication Date
2024-11-12
Estimated Expiration
2042-04-29

AI Technical Summary

Technical Problem

In distributed databases, data migration between sharded nodes leads to inconsistent data synchronization, existing technology solutions are complex or lack of universality, and synchronization efficiency is limited.

Method used

By introducing a message center, the data source identifier in the log to be synchronized by the shard nodes is parsed, and broadcast messages are sent to each shard node, carrying address information and identification, so that the shard node decides whether to send the log to be synchronized, ensuring that the data sources are consistent and synchronous efficiency.

Benefits of technology

Enhanced coordination of distributed database data synchronization, ensures that the tasks sent by all shard nodes are consistent, and avoids the problem of limited synchronization efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114817190B_ABST
    Figure CN114817190B_ABST
Patent Text Reader

Abstract

The embodiments of this specification provide a method, apparatus, system, device, and storage medium for log synchronization, the method comprising: a shard node sends a log to be synchronized to a message center, the log to be synchronized contains an identifier representing the source of data, the source of data includes data created locally by the shard node or migrated to the shard node by other shard nodes; the message center parses the identifier from the log to be synchronized, and sends a broadcast message to each of the shard nodes, the broadcast message carries the address information of the shard node and the identifier; the shard node receives the broadcast message based on the address information, and determines whether to send the log to be synchronized based on the identifier. The purpose is to solve the problem of inconsistent information during distributed database log synchronization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of database synchronization, and in particular to a method, apparatus, system, device, and storage medium for log synchronization. Background Art

[0002] Distributed databases generally have multiple independent physical nodes. During the distributed database data synchronization process, the data write logs are generally managed separately by each shard node of the source database. However, due to load balancing or some manual operations, data migration may occur between shard nodes. When data migration occurs between shard nodes, the data synchronized from the shard node to the target database may eventually be inconsistent. To address this problem, there are currently two mainstream approaches to real-time synchronization of incremental data in distributed databases. The first is to configure tasks for each shard node of the distributed database on the source side and disable data migration between shard nodes on the source side, so as to ensure the final consistency of information synchronized to the target database. The disadvantage of this approach is that each shard node of the source database needs to be configured. When there are many shard nodes in the source database, the operation becomes very complicated, and data migration cannot be performed between shard nodes of the source database. The second approach is for the distributed database to provide a unified log stream, such as the change-stream solution of MongoDB. The principle is to pull incremental logs from different shard nodes, merge and sort them, and then output the log stream. This approach can ensure that there is data migration between shard nodes in the distributed database on the source side, while ensuring that the final information synchronized to the target database is consistent. However, the disadvantage of this approach is that it is not universal, and the synchronization efficiency is limited by the output efficiency of the source log after merging and sorting. Summary of the invention

[0003] To overcome the problems existing in the related art, the embodiments of this specification provide a log synchronization method, apparatus, system, device and storage medium.

[0004] According to a first aspect of an embodiment of this specification, a system for log synchronization is provided, the system comprising:

[0005] Distributed database and message center;

[0006] The distributed database includes a source database and a target database. The source database includes a plurality of shard nodes. The shard nodes are used to send logs to be synchronized to the message center. The logs to be synchronized include identifiers representing data sources. The data sources include data created locally by the shard node or migrated to the shard node by other shard nodes.

[0007] The message center is used to parse the identifier from the log to be synchronized, and send a broadcast message to each of the shard nodes, wherein the broadcast message carries the address information of the shard node and the identifier;

[0008] The sharding node is further configured to receive the broadcast message based on the address information, and determine whether to send the log to be synchronized based on the identifier.

[0009] According to a second aspect of an embodiment of this specification, a log synchronization method is provided, and the method includes:

[0010] The shard node sends a log to be synchronized, wherein the log to be synchronized contains an identifier representing a source of data, wherein the source of data includes data created locally by the shard node or migrated to the shard node by other shard nodes;

[0011] The shard node receives a broadcast message based on the address information of the shard node, wherein the broadcast message carries the address information of the shard node that sends the log to be synchronized and the identifier, wherein the identifier is obtained by parsing the log to be synchronized;

[0012] The shard node receives the broadcast message based on the address information, and determines whether to send the log to be synchronized based on the identifier.

[0013] According to a third aspect of an embodiment of this specification, a log synchronization method is provided, which is applied to a message center, and the method includes:

[0014] Receive a log to be synchronized sent by the shard node, wherein the log to be synchronized includes an identifier representing a source of data, wherein the source of the data includes data created locally by the shard node or migrated to the shard node by other shard nodes;

[0015] The identifier is parsed based on the log to be synchronized, and a broadcast message is sent to each of the shard nodes, wherein the broadcast message includes the address information of the shard node and the identifier; so that the shard node receives the broadcast message based on the address information, and determines whether to send the log to be synchronized based on the identifier.

[0016] According to a fourth aspect of the embodiments of this specification, a log synchronization device is provided, the device comprising: a receiving module, configured to receive a log to be synchronized sent by the sharding node, the log to be synchronized comprising an identifier representing a data source, the data source comprising data locally created by the sharding node or migrated to the sharding node by other sharding nodes;

[0017] A parsing module is used to parse the received log to be synchronized to obtain the identifier; a broadcast module is used to send a broadcast message to each of the shard nodes, wherein the broadcast message includes the address information of the shard node and the identifier, so that the shard node receives the broadcast message based on the address information, and based on the identifier, the shard node determines whether to send the log to be synchronized.

[0018] According to a fifth aspect of the embodiments of this specification, an electronic device is provided, the electronic device comprising:

[0019] processor;

[0020] a memory for storing processor-executable instructions;

[0021] The processor is configured to perform the operations described in any one of the methods described in the first aspect above.

[0022] According to a sixth aspect of an embodiment of the present application, a readable storage medium is provided, including:

[0023] When the instruction is executed by the processor, the operation described in any one of the methods described in the first aspect above is implemented.

[0024] The technical solutions provided by the embodiments of the present specification may have the following beneficial effects:

[0025] In an embodiment of the present specification, connection links are established between the shard nodes of the source database and the target database respectively, and the connection links of the shard nodes are monitored through a message center. A broadcast message is constructed based on the data source of the log tasks to be sent for each link monitored, and is published to each shard node in a broadcast manner, so that the shard node can know whether the data source of the log to be sent is migrated to the node from other shard nodes or created by the node itself, and then decide whether to send the log to the destination database, thereby enhancing the coordination of distributed database data synchronization. Moreover, since each shard node refers to the information of the tasks to be sent by other shard nodes before sending the tasks to be sent, the tasks sent by all shard nodes are consistent.

[0026] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings herein are incorporated in the specification and constitute a part of the embodiments of the specification, illustrate embodiments consistent with the embodiments of the specification, and together with the description, are used to explain the principles of the embodiments of the specification.

[0028] Figure 1AThe embodiment of this specification is a flowchart of a log synchronization method according to an exemplary embodiment.

[0029] Figure 1B The embodiment of this specification is a flowchart of a log synchronization method according to an exemplary embodiment.

[0030] Figure 2A It is a prior art.

[0031] Figure 2B It is a distributed database structure diagram shown in an embodiment of this specification according to an exemplary embodiment.

[0032] Figure 3 It is a structural diagram of a new message constructed by a message center according to an exemplary embodiment of this specification.

[0033] Figure 4 It is a structural diagram of another message center shown in an embodiment of this specification according to an exemplary embodiment.

[0034] Figure 5 The embodiment of this specification shows a block diagram of a log synchronization device according to an exemplary embodiment.

[0035] Figure 6 It is a block diagram of a log synchronization system shown in an embodiment of this specification according to an exemplary embodiment.

[0036] Figure 7 The embodiment of this specification is a flowchart of a log synchronization scheme applied in a cloud database scenario according to an exemplary embodiment.

[0037] Figure 8 It is a block diagram of an electronic device shown in an embodiment of this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0038] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of this specification as detailed in the appended claims.

[0039] The terms used in the embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of this specification. The singular forms of "a", "said" and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0041] Distributed databases generally have multiple independent physical nodes. During the distributed database data synchronization process, the logs to which data is written are generally managed separately by each shard node of the source database. However, due to load balancing or some manual operations, data movement and migration may occur between shard nodes. When data movement and migration occur between shard nodes, the information data synchronized from the shard node to the target database may eventually be inconsistent.

[0042] One of the means to solve this problem is to ensure that the logs to be synchronized that are migrated between the shard nodes of the source database are not synchronized to the target database. If the connection between the shard nodes of the source database is directly cut off, the data information in one shard node cannot be shared by other shard nodes, hindering the flow of information; if the logs to be synchronized sent by the shard nodes are sorted before synchronization and the logs to be synchronized obtained by migration are excluded, the synchronization efficiency will be limited.

[0043] To address this problem, an embodiment of this specification proposes a method for distributed database log synchronization, which uses a message center to parse the logs to be synchronized of the source shard nodes, and broadcasts the relevant information obtained from the analysis to all shard nodes, so that each shard node can decide whether to send the logs to be synchronized of the current time sequence based on the broadcast message.

[0044] Next, the embodiments of this specification are described in detail.

[0045] Figure 2AThe schematic diagram is a common structural diagram of a distributed database, which includes a target database 201, a source database 202 and various shard nodes 203; the shard nodes belong to the source database of the distributed database.

[0046] Combination Figure 2A and Figure 1A , Figure 1A The embodiment of this specification is a flowchart of a log synchronization method according to an exemplary embodiment, including the following steps:

[0047] In step S101, the shard node sends a log to be synchronized, which contains an identifier representing a data source, and the data source includes data created locally by the shard node or migrated to the shard node by other shard nodes;

[0048] In step S102, the shard node receives a broadcast message based on the address information of the shard node, where the broadcast message carries the address information of the shard node that sends the log to be synchronized and an identifier representing the source of the data, where the identifier is obtained by parsing the log to be synchronized;

[0049] In step S103, the shard node receives the broadcast message based on the address information, and determines whether to send the log to be synchronized based on the identifier representing the data source.

[0050] In step S103, based on the identifier, determining whether to send the log to be synchronized may also include: when the data source is locally created by the sharding node, sending the log to be synchronized to the target database, or, when the data source is other sharding nodes migrated to the sharding node, not sending the log to be synchronized. The broadcast message may also include the timing information of the log to be synchronized of the sharding node. After the sharding node determines that the log to be synchronized migrated to the sharding node from other sharding nodes not to be sent according to the identifier, the sharding node jumps to the log to be synchronized created by the local node according to the timing information and sends it to the target database. The specific steps are: after the sharding node receives the broadcast message, it first finds the corresponding broadcast message according to its own address information, and reads the identifier of the data source in the broadcast message. If the data source corresponding to the identifier is a local operation, then the sharding node sends the log to be synchronized normally. If the data source corresponding to the identifier is other When the shard node migrates to this shard node, the sending of the log to be synchronized is stopped, and the timing information of the log to be synchronized of all broadcast messages in the broadcast message is obtained. If all the timing information is greater than the timing information of the log to be synchronized of the shard node, the operation of skipping the log to be synchronized is executed. If there is a timing information less than the timing information of the log to be synchronized of the shard node in these broadcast messages, then the shard node corresponding to the broadcast message with the timing information less than the timing information of the log to be synchronized of the shard node is waited for to complete the processing of the log to be synchronized, and then the shard node executes the skip operation on its own log to be synchronized.

[0051] In one embodiment, the shard node sends the logs to be synchronized created by the local node to the target database based on the identifier, or does not send the logs to be synchronized migrated to the shard node from other shard nodes. Before all the logs to be synchronized are sent, these logs to be synchronized can be sorted based on time sequence in the shard node, and each subsequent sending or skipping is based on time sequence.

[0052] In another embodiment, if Figure 1B As shown, Figure 1B This is a log synchronization method shown in an embodiment of this specification according to an exemplary embodiment, including the following steps:

[0053] In step S104, a to-be-synchronized log sent by a shard node is received, wherein the to-be-synchronized log includes an identifier representing a data source, and the data source includes data created locally by the shard node or migrated to the shard node by other shard nodes;

[0054] In step S105, an identifier is parsed based on the log to be synchronized, and a broadcast message is sent to each shard node, the broadcast message including the address information and identifier of the shard node; so that the shard node receives the broadcast message based on the address information, and determines whether to send the log to be synchronized based on the identifier.

[0055] In step S105

[0056] Based on the identifier, determining whether to send the log to be synchronized may also include: when the data source is created locally by the shard node, sending the log to be synchronized to the target database, or, when the data source is other shard nodes migrated to the shard node, not sending the log to be synchronized. The broadcast message may also include the timing information of the log to be synchronized of the shard node, so that after the shard node determines the log to be synchronized of other shard nodes migrated to the shard node not to be sent according to the identifier, it jumps to the log to be synchronized created by the local node according to the timing information and sends it to the target database.

[0057] Figure 1B The various steps of the method shown may be implemented using a logical entity that is independent of the distributed nodes and the target database. The logical entity may be physically independent of the devices where the distributed nodes and the target database are located. Of course, the implementation method of loading the logical entity onto the devices where the distributed nodes or the target database are located is not excluded.

[0058] In the embodiments of this specification, the conventional distributed database network architecture is improved, such as Figure 2B As shown, the distributed database architecture of the embodiment of this specification adds a message center 204 to the traditional distributed database structure to achieve Figure 1BThe logical entity of each step may be implemented through the message center 204 .

[0059] In one embodiment, the message center 204 can be configured as a data class with two members. The two members are the address information of the shard node and the data source identifier of the log to be synchronized sent by the shard node. The identifier needs to be obtained by the message center after parsing the log to be synchronized. After receiving the message corresponding to the above members, the message center 204 will construct a new message and broadcast it to each shard node, so that the shard node receives the broadcast message based on the address information. The broadcast message contains at least the address information of the shard node and the data source identifier of the log to be synchronized, and determines whether to send the log to be synchronized based on the identifier. As an example, when constructing a new message, the timing information of the log to be synchronized of the shard node can be used as part of the new message. The data structure of the broadcast message constructed by the newly constructed message center is as follows: Figure 3 As shown, in this example, the message broadcast by the message center includes: the address information of the shard node, the data source identifier and the timing information of the log to be synchronized. The shard node finds the message that matches its own address information in the message broadcast by the message center according to the address information, and sends the log to be synchronized created by the local node to the target database according to the data source identifier in the message that matches itself, or does not send the log to be synchronized migrated from other shard nodes to this shard node. When the shard node of the source database of the distributed database does not send the log to be synchronized that other shard nodes migrate to the shard node, it will jump to the log to be synchronized created by the local node according to the above timing information and send it to the target database of the distributed database. The specific steps are as follows: after the shard node receives the message broadcast by the message center, it first finds the corresponding broadcast message according to its own address information, reads the identifier of the data source in the broadcast message, if the data source corresponding to the identifier is a local operation, then the shard node sends the log to be synchronized normally, if the data source corresponding to the identifier is other shard nodes migrated to the shard node, then stop sending the log to be synchronized, and obtain the timing information of the log to be synchronized of all broadcast messages in the broadcast message, if all the timing information is greater than the timing information of the log to be synchronized of the shard node, then execute the operation of skipping the log to be synchronized, if there is a timing information less than the log to be synchronized of the shard node in these broadcast messages, then wait for the shard node corresponding to the broadcast message less than the timing information of the log to be synchronized of the shard node to complete the processing of the log to be synchronized, and then the shard node executes the skip operation on its own log to be synchronized.

[0060] In another embodiment, the message center may also have a specific data structure, such as Figure 4As shown, it includes a destination database 401, a source database 402, a shard node 403 in the source database, a message center 404, and a log to be synchronized 405, wherein the data structure of the new message constructed by the message center 404 is composed of multiple data structure fields 406. In this embodiment, the data structure uses three fields, jobld, isfrommigrate and checkpoint, to represent the task name, the message source identifier and the timing information of the log to be synchronized, respectively, and the above-mentioned task name is the address information of the shard node 403. In this example, the working principle of the message center 404 is: receiving the to-be-synchronized log 405 of the shard node 403 of the source database 402, and then parsing the information required for the fields that constitute the data structure from the to-be-synchronized log 405. The information of all shard nodes constitutes a two-dimensional data group, and the message center broadcasts the two-dimensional data group to the shard nodes 403. The to-be-synchronized log 405 of each shard node 403 can correspond to a row of data in the above two-dimensional data group. The corresponding rule is: the address information of the shard node 403 where the to-be-synchronized log 405 is located corresponds to the data of the jobld field of a row of the above two-dimensional data group. After the correspondence, read the identifier of the data source in isfrommigrate. If the data source corresponding to the identifier is a local operation, the shard node sends the log to be synchronized normally. If the data source corresponding to the identifier is migrated from other shard nodes to the shard node, then stop sending the log to be synchronized, and obtain the timing information of the log to be synchronized of all checkpoint fields of the two-dimensional data group in the broadcast message. If all the timing information is greater than the timing information of the log to be synchronized of the shard node, then skip the operation of the log to be synchronized. If there is timing information less than the log to be synchronized of the shard node in these broadcast messages, then wait for the shard node corresponding to the broadcast message that is less than the timing information of the log to be synchronized of the shard node to complete the processing of the log to be synchronized, and then the shard node skips its own log to be synchronized.

[0061] It should be understood that this embodiment only lists several ways to implement the message center and cannot be regarded as a limitation of the technical solutions of the embodiments of this specification. Those skilled in the art can obtain other configuration methods of the message center based on the functions of the message center in the embodiments of this specification without any creative work.

[0062] In another embodiment, the present specification provides a log synchronization device, such as Figure 5 As shown, Figure 5 is a block diagram of a device according to an exemplary embodiment of the present specification, the device comprising:

[0063] The receiving module 501 is used to receive the log to be synchronized sent by the sharding node, where the log to be synchronized contains an identifier representing the source of data, where the source of data includes data created locally by the sharding node or migrated to the sharding node by other sharding nodes;

[0064] The parsing module 502 is used to parse the received log to be synchronized to obtain the identifier;

[0065] The broadcast module 503 is used to send a broadcast message to each shard node, where the broadcast message includes the address information of the shard node and the above-mentioned identifier, so that the shard node receives the above-mentioned broadcast message based on the above-mentioned address information, and based on the above-mentioned identifier, the shard node determines whether to send the log to be synchronized.

[0066] In this embodiment, the broadcast message sent by the broadcast module 503 enables the sharding node to determine whether to send the log to be synchronized based on the identifier, and may also include: when the data source is created locally by the sharding node, the log to be synchronized is sent to the target database, or, when the data source is other sharding nodes migrated to the sharding node, the log to be synchronized is not sent. The broadcast message sent by the broadcast module 503 also carries the timing information of the log to be synchronized of the sharding node. This information enables the sharding node to jump to the log to be synchronized created by the local node according to the timing information and send it to the target database after determining that the log to be synchronized migrated to the sharding node from other sharding nodes not to be sent according to the identifier. The specific content of this operation is: after the sharding node receives the message broadcast by the message center, it first finds the corresponding broadcast message according to its own address information, and reads the identifier of the data source in the broadcast message. If the identifier is a local operation, then the sharding node sends the log to be synchronized normally, such as If the identifier is the migration of other shard nodes, then stop sending the log to be synchronized, and obtain the timing information of the log to be synchronized of all broadcast messages in the broadcast message. If all the timing information is greater than the timing information of the log to be synchronized of the shard node, then skip the operation of the log to be synchronized. If there is a timing information less than the timing information of the log to be synchronized of the shard node in these broadcast messages, then wait for the shard node corresponding to the broadcast message with a timing information less than the timing information of the log to be synchronized of the shard node to complete the processing of the log to be synchronized, and then the shard node will skip its own log to be synchronized.

[0067] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0068] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment scheme of this specification. Those of ordinary skill in the art can understand and implement it without paying creative labor.

[0069] In another embodiment, the present specification also provides a system for log synchronization, such as Figure 6 As shown, the system includes:

[0070] A distributed database 601 and a message center 602; wherein the distributed database 601 includes a target database 603 and a source database 604, and the source database 604 includes a plurality of shard nodes 605;

[0071] The shard node 605 is used to send a log to be synchronized to the message center 602. The log to be synchronized includes an identifier representing a source of data. The source of data includes data created locally by a shard node 605 and / or migrated to the shard node by other shard nodes.

[0072] The message center 602 is used to parse the identifier representing the source of the data from the above-mentioned log to be synchronized, and send a broadcast message to each shard node 605. The broadcast message carries the address information of the shard node 605 and the identifier representing the source of the data, so that the shard node 605 receives the broadcast message based on the address information, and determines whether to send the log to be synchronized based on the identifier representing the source of the data.

[0073] In one embodiment, the sharding node determines whether to send the log to be synchronized based on the identifier, which may also include: when the data source is created locally on the sharding node, the log to be synchronized is sent to the target database, or, when the data source is other sharding nodes migrated to the sharding node, the log to be synchronized is not sent. The broadcast message sent by the message center 602 to each sharding node 605 also carries the timing information of the log to be synchronized of the sharding node 605. The sharding node is also used to jump to the log to be synchronized created by the local node according to the timing information and send it to the target database after determining that the log to be synchronized of other sharding nodes migrated to the sharding node according to the identifier is not to be sent; the broadcast message sent by the message center 602 causes the sharding node 605 not to send the log to be synchronized of other sharding nodes migrated to a certain sharding node, and also causes the sharding node 605 to execute the operation of jumping to the log to be synchronized created by the local node according to the above timing information and sending it to the destination database 603. The specific content of the operation is: after receiving the message broadcast by the message center, the sharding node first determines the log to be synchronized created by the local node according to its own The corresponding broadcast message is found based on the address information, and the identifier of the data source in the broadcast message is read. If the identifier is a local operation, the shard node sends the log to be synchronized normally. If the identifier is the migration of other shard nodes, the sending of the log to be synchronized is stopped, and the timing information of the log to be synchronized of all broadcast messages in the broadcast message is obtained. If all the timing information is greater than the timing information of the log to be synchronized of the shard node, the operation of skipping the log to be synchronized is executed. If there is timing information in these broadcast messages that is less than the timing information of the log to be synchronized of the shard node, then the shard node corresponding to the broadcast message that is less than the timing information of the log to be synchronized of the shard node is waited for to complete the processing of the log to be synchronized, and then the shard node executes the skip operation on its own log to be synchronized.

[0074] One application scenario of the log synchronization solution provided in the embodiments of this specification is to be applied in a cloud database. For example, when a user is shopping on Taobao, the original address is located in place A, then the Taobao source database in place A will store the user's account information. When the user leaves place A and goes to place B, the Taobao source database originally in place A will migrate the user's account information to place B. In this scenario, it includes a cloud target database, a cloud server, and a client source end shard node. Among them, the message center can be deployed on a cloud server, such as Figure 7 The following description will be given by taking a cloud server as an example to implement the solution of the embodiments of this specification.

[0075] like Figure 7 As shown, the distributed server architecture includes a cloud target database 701, a client source database shard node 702 and a cloud server 703.

[0076] In step S701, the client source database shard node 702 sends the log to be synchronized to the cloud server 703;

[0077] In step S702, the cloud server 703 parses the address information of the client source database shard node 702, the data source identifier of the log to be synchronized, and the time sequence information of the log to be synchronized according to the log to be synchronized, wherein the data source identifier includes a shard node locally created and / or migrated to the shard node by other shard nodes; and then broadcasts the above message to the client source database shard node 702;

[0078] In step S703, the client source database shard node 702 receives the corresponding broadcast message based on its own address information, and determines the data source identifier in the broadcast message;

[0079] If the data source is identified as locally created, S704 is executed to send the current log to be synchronized;

[0080] If the data source is identified as being sent by other nodes, S705 is executed to jump to the sending of the log to be synchronized created by the local node.

[0081] Based on the above identifier, the log to be synchronized created by the local node is sent to the cloud target database, or the log to be synchronized migrated to the shard node by other shard nodes is not sent; wherein, after the shard node does not send the log to be synchronized migrated to a certain shard node by other shard nodes, the shard node also executes the operation of jumping to the log to be synchronized created by the local node according to the above timing information and sending it to the cloud target database 701. The specific content of the operation is: after the shard node receives the message broadcast by the message center, it first finds the corresponding broadcast message according to its own address information, reads the identifier of the data source in the broadcast message, and if the identifier is a local operation, then the shard node The shard node sends the log to be synchronized normally. If the mark indicates that other shard nodes are migrating, the sending of the log to be synchronized is stopped, and the timing information of the log to be synchronized of all broadcast messages in the broadcast message is obtained. If all the timing information is greater than the timing information of the log to be synchronized of the shard node, the operation of skipping the log to be synchronized is executed. If there is a timing information less than the timing information of the log to be synchronized of the shard node in these broadcast messages, then the shard node corresponding to the broadcast message with the timing information less than the timing information of the log to be synchronized of the shard node is waited for to complete the processing of the log to be synchronized, and then the shard node executes the skipping operation on its own log to be synchronized.

[0082] Accordingly, this specification also provides a computer device, such as Figure 8 As shown, including:

[0083] A processor 81, a memory 82 and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any of the above method embodiments when executing the program.

[0084] The present specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to instruct related hardware to complete the method described in any of the above method embodiments.

[0085] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0086] Those skilled in the art will readily appreciate other implementations of the embodiments of the present specification after considering the specification and practicing the inventions claimed herein. The embodiments of the present specification are intended to cover any variations, uses, or adaptations of the embodiments of the present specification, which follow the general principles of the embodiments of the present specification and include common knowledge or customary techniques in the art that are not claimed in the embodiments of the present specification. The description and embodiments are to be considered merely as exemplary, and the true scope and spirit of the embodiments of the present specification are indicated by the claims below.

[0087] It should be understood that the embodiments of the present specification are not limited to the precise structures described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present specification is limited only by the appended claims.

[0088] The above description is only a preferred embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of this specification should be included in the scope of protection of the embodiments of this specification.

Claims

1. A log synchronization system, the system comprising: Distributed database and message center; The distributed database includes a source database and a target database. The source database includes a plurality of shard nodes. The shard nodes are used to send logs to be synchronized to the message center. The logs to be synchronized include identifiers representing data sources. The data sources include data created locally by the shard node or migrated to the shard node by other shard nodes. The message center is used to parse the identifier from the log to be synchronized, and send a broadcast message to each of the shard nodes, wherein the broadcast message carries the address information of the shard node and the identifier; The sharding node is also used to receive the broadcast message based on the address information, and determine whether to send the log to be synchronized based on the identifier; wherein, determining whether to send the log to be synchronized based on the identifier includes: when the data source corresponding to the identifier is created locally by the sharding node, sending the log to be synchronized to the target database, or, when the data source corresponding to the identifier is migrated from other sharding nodes to the sharding node, not sending the log to be synchronized.

2. The system according to claim 1, The broadcast message sent by the message center to each of the shard nodes also carries the timing information of the log to be synchronized. The shard node is also used to jump to the log to be synchronized created locally by the shard node according to the timing information and send it to the target database after determining that other shard nodes that do not send the log have migrated to the log to be synchronized of the shard node according to the identifier.

3. A log synchronization method, applied to the log synchronization system according to any one of claims 1-2, wherein the distributed database includes a source database and a target database, the source database includes a plurality of shard nodes, and the method includes: The shard node sends a log to be synchronized, wherein the log to be synchronized contains an identifier representing a source of data, wherein the source of data includes data created locally by the shard node or migrated to the shard node by other shard nodes; The shard node receives a broadcast message based on the address information of the shard node, wherein the broadcast message carries the address information of the shard node that sends the log to be synchronized and the identifier, wherein the identifier is obtained by parsing the log to be synchronized; The shard node determines whether to send the log to be synchronized based on the identifier.

4. According to the method of claim 3, determining whether to send the log to be synchronized based on the identifier comprises: In the case where the data source is created locally on the shard node, the log to be synchronized is sent to the target database, or, When the data source is other shard nodes migrated to the shard node, the log to be synchronized is not sent.

5. The method according to claim 3, The broadcast message also carries the timing information of the log to be synchronized of the shard node. After the shard node determines that other shard nodes that are not sent have migrated to the log to be synchronized of the shard node based on the identifier, the shard node jumps to the log to be synchronized created locally by the shard node according to the timing information and sends it to the target database.

6. A log synchronization method, applied to the log synchronization system according to any one of claims 1-2, wherein the distributed database includes a source database and a target database, the source database includes a plurality of shard nodes, and the method includes: Receive a log to be synchronized sent by the shard node, wherein the log to be synchronized includes an identifier representing a source of data, wherein the source of the data includes data created locally by the shard node or migrated to the shard node by other shard nodes; Parsing the identifier based on the log to be synchronized, and sending a broadcast message to each of the shard nodes, wherein the broadcast message includes the address information of the shard node and the identifier; So that the shard node receives the broadcast message based on the address information, and determines whether to send the log to be synchronized based on the identifier.

7. According to the method of claim 6, the broadcasting message to enable the sharding node to determine whether to send the log to be synchronized based on the identifier comprises: In the case where the data source is created locally on the shard node, the log to be synchronized is sent to the target database, or, When the data source is other shard nodes migrated to the shard node, the log to be synchronized is not sent.

8. The method according to claim 6, The broadcast message also carries the timing information of the log to be synchronized of the shard node, so that after the shard node determines that other shard nodes that are not sent have migrated to the log to be synchronized of the shard node according to the identifier, the shard node jumps to the log to be synchronized created locally by the shard node according to the timing information and sends it to the target database.

9. A log synchronization device, applied to the log synchronization system according to any one of claims 1-2, wherein the distributed database includes a source database and a target database, the source database includes a plurality of shard nodes, and the device includes: A receiving module, configured to receive a log to be synchronized sent by the sharding node, wherein the log to be synchronized includes an identifier representing a source of data, wherein the source of the data includes data created locally by the sharding node or migrated to the sharding node by other sharding nodes; A parsing module, used for parsing the received log to be synchronized to obtain the identifier; A broadcast module, configured to send a broadcast message to each of the shard nodes, wherein the broadcast message includes the address information of the shard node and the identifier, so that the shard node receives the broadcast message based on the address information, and determines whether to send the log to be synchronized based on the identifier; The broadcast module is specifically used to: when the data source corresponding to the identifier is created locally by the shard node, send the log to be synchronized to the target database; or, when the data source corresponding to the identifier is migrated from other shard nodes to the shard node, not send the log to be synchronized.

10. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the method as claimed in any one of claims 3 to 8.

11. A computer-readable storage medium having computer instructions stored thereon, comprising: When the instruction is executed by the processor, the method described in any one of claims 3 to 8 is implemented.