Data synchronization method, device, computer equipment and storage medium

Abstract

The present application relates to a data synchronization method, device, computer equipment, storage medium and computer program product, and relates to the field of database technology. It can be used in the field of financial technology or other related fields. The method includes: parsing the data operation log of the source database to be expanded, pushing at least one synchronization operation message corresponding to the data table change operation to the message queue corresponding to the target message topic; generating consumer program configuration information according to the data table identifier corresponding to each data table in the source database; generating consumer programs that subscribe to different message topics according to the consumer program configuration information; executing each consumer program; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information. The use of this method can improve the efficiency of database expansion.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method, apparatus, computer equipment, storage medium, and computer program product for database data synchronization. Background Art

[0002] With the continuous development of Internet technology, the scale and concurrent access of enterprise-level databases have increased dramatically, and distributed databases have been rapidly and widely used. Distributed databases are multiple interconnected databases, usually located on multiple servers, that communicate with each other to achieve common goals.

[0003] To meet business needs and handle large amounts of data, distributed databases need to be expanded. Existing distributed system database expansion solutions require system downtime. After all data is imported into the expanded database, a background program is launched to filter and clean the data in both the old and new databases. When the data volume is large, database expansion can be very time-consuming.

[0004] Therefore, in the related art, there is a problem of low database expansion efficiency in database expansion. Summary of the Invention

[0005] Based on this, it is necessary to provide a data synchronization method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve data synchronization efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a data synchronization method. The method comprises:

[0007] Parse the data operation log of the source database to be expanded, and for at least one data table change operation in the source database, push a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic; the target message topic is the message topic corresponding to the data table operated on by the data table change operation;

[0008] Generate consumer program configuration information according to the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information includes a mapping relationship between the data tables in the source database and the message consumption configuration information; the message consumption configuration information includes a message subject and routing information;

[0009] Generate consumer programs that subscribe to different message topics according to the consumer program configuration information;

[0010] Execute each of the consumer programs; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information; the target database is the expanded database corresponding to the source database.

[0011] In one embodiment, generating the consumer program configuration information according to the data table identifier corresponding to each data table in the source database includes:

[0012] Obtaining data table identifiers corresponding to each data table in the source database;

[0013] Establishing a first mapping relationship between each of the data table identifiers and each of the message topics, and establishing a second mapping relationship between each of the data table identifiers and each of the routing information;

[0014] The consumer program configuration information is generated according to the first mapping relationship and the second mapping relationship.

[0015] In one embodiment, the establishing of a first mapping relationship between each of the data table identifiers and each of the message topics includes:

[0016] Get the message subject format template;

[0017] Supplement the data table identifier corresponding to each data table to the message subject format template to obtain the message subject corresponding to each data table;

[0018] The first mapping relationship is obtained according to the message topics corresponding to each of the data tables.

[0019] In one embodiment, executing the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information includes:

[0020] According to the corresponding routing information, determining a sub-database to be operated on in the target database as a target sub-database corresponding to the target synchronization operation message;

[0021] In the target sub-database corresponding to the target synchronization operation message, a data table change operation matching the target synchronization operation message is executed.

[0022] In one embodiment, the routing information includes a routing field and a routing algorithm; and determining a sub-database in the target database for performing an operation based on the corresponding routing information includes:

[0023] Use the corresponding routing algorithm to calculate the corresponding routing field and determine the sub-database identifier;

[0024] The sub-library in the target database corresponding to the sub-library identifier is used as the sub-library for the operation to be executed.

[0025] In one embodiment, parsing the data operation log of the source database to be expanded, and for at least one data table change operation of the source database, pushing a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic, includes:

[0026] Parsing the data operation log to obtain a change operation record corresponding to at least one data table change operation on the source database;

[0027] The change operation record is encapsulated into a message message in a preset format to obtain a synchronization operation message corresponding to the at least one data table change operation of the source database.

[0028] In one embodiment, the message queue corresponding to the target message topic includes at least one partition message queue; and pushing the synchronization operation message corresponding to the at least one data table change operation to the message queue corresponding to the target message topic includes:

[0029] Determine the partition identifier corresponding to the synchronization operation message under the target message topic;

[0030] Push the synchronization operation message to the at least one partition message queue, the partition message queue corresponding to the partition identifier.

[0031] In a second aspect, the present application further provides a data synchronization device. The device comprises:

[0032] a parsing module configured to parse the data operation log of the source database to be expanded, and, for at least one data table change operation of the source database, push a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic; the target message topic being a message topic corresponding to the data table operated on by the data table change operation;

[0033] An information generation module, configured to generate consumer program configuration information according to data table identifiers corresponding to the data tables in the source database; the consumer program configuration information includes a mapping relationship between the data tables in the source database and the message consumption configuration information; the message consumption configuration information includes a message subject and routing information;

[0034] A program generation module, configured to generate consumer programs that subscribe to different message topics according to the consumer program configuration information;

[0035] An execution module is used to execute each of the consumer programs; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information; the target database is the expanded database corresponding to the source database.

[0036] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0037] Parse the data operation log of the source database to be expanded, and for at least one data table change operation in the source database, push a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic; the target message topic is the message topic corresponding to the data table operated on by the data table change operation;

[0038] Generate consumer program configuration information according to the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information includes a mapping relationship between the data tables in the source database and the message consumption configuration information; the message consumption configuration information includes a message subject and routing information;

[0039] Generate consumer programs that subscribe to different message topics according to the consumer program configuration information;

[0040] Execute each of the consumer programs; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information; the target database is the expanded database corresponding to the source database.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0042] Parse the data operation log of the source database to be expanded, and for at least one data table change operation in the source database, push a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic; the target message topic is the message topic corresponding to the data table operated on by the data table change operation;

[0043] Generate consumer program configuration information according to the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information includes a mapping relationship between the data tables in the source database and the message consumption configuration information; the message consumption configuration information includes a message subject and routing information;

[0044] Generate consumer programs that subscribe to different message topics according to the consumer program configuration information;

[0045] Execute each of the consumer programs; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information; the target database is the expanded database corresponding to the source database.

[0046] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0047] Parse the data operation log of the source database to be expanded, and for at least one data table change operation in the source database, push a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic; the target message topic is the message topic corresponding to the data table operated on by the data table change operation;

[0048] Generate consumer program configuration information according to the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information includes a mapping relationship between the data tables in the source database and the message consumption configuration information; the message consumption configuration information includes a message subject and routing information;

[0049] Generate consumer programs that subscribe to different message topics according to the consumer program configuration information;

[0050] Execute each of the consumer programs; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information; the target database is the expanded database corresponding to the source database.

[0051] The above-mentioned data synchronization method, device, computer equipment, storage medium and computer program product parse the data operation log of the source database to be expanded, and for at least one data table change operation of the source database, push at least one synchronization operation message corresponding to the data table change operation to the message queue corresponding to the target message topic; the target message topic is the message topic corresponding to the data table operated by the data table change operation; then, according to the data table identifier corresponding to each data table in the source database, consumer program configuration information is generated; the consumer program configuration information includes the mapping relationship between each data table in the source database and the message consumption configuration information; the message consumption configuration information includes the message topic and routing information; then, according to the consumer program configuration information, a consumer program subscribing to different message topics is generated; then, each consumer program is executed; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information; the target database is the expanded database corresponding to the source database.

[0052] In this way, since there is a mapping relationship between each data table and the message consumption configuration information, the mapping relationship can be used to flexibly generate consumer programs that subscribe to different message topics. Therefore, the consumer program can perform data table change operations that match the pulled target synchronization operation messages in the expanded database corresponding to the source database according to the corresponding routing information. This realizes the flexible synchronization of data in the source data to the expanded database according to the corresponding message consumption configuration information, and realizes data synchronization during non-system downtime. There is no need to import data when the system is down, which shortens the waiting time for data synchronization and effectively improves the efficiency of database expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 1 is a flow chart of a data synchronization method according to an embodiment;

[0054] Figure 2 A flowchart illustrating steps for generating consumer program configuration information in one embodiment;

[0055] Figure 3 A schematic diagram of a Kafka producer submitting a synchronous operation message in one embodiment;

[0056] Figure 4 is a flow chart of a data synchronization method according to another embodiment;

[0057] Figure 5 A schematic diagram of a process of DRP data parsing and replication and Kafka data synchronization in one embodiment;

[0058] Figure 6is a structural block diagram of a data synchronization device in one embodiment;

[0059] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0061] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0062] In one embodiment, Figure 1 As shown, a data synchronization method is provided. This embodiment uses the method applied to a server as an example. The server can be implemented as an independent server or a server cluster composed of multiple servers. The method includes the following steps:

[0063] Step S110 , parsing the data operation log of the source database to be expanded, and for at least one data table change operation of the source database, pushing at least one synchronization operation message corresponding to the data table change operation to the message queue corresponding to the target message topic.

[0064] Each synchronization operation message specifies its target message topic.

[0065] The target message topic is the message topic corresponding to the data table operated by the data table change operation.

[0066] The source database may be a distributed system database, for example, but not limited to, a MySQL database.

[0067] The data operation log is the bin.log log.

[0068] The data table modification operation includes at least one of data operations such as INSERT, UPDATE, and DELETE.

[0069] The data table operated by the data table change operation is the data table in the source database. When the source database is a MySQL database, the data table is a MySQL table.

[0070] The message queue can be a Kafka message queue. Kafka is a high-throughput distributed publish-subscribe messaging system. Kafka message queues are primarily targeted at scenarios with high cross-region latency. They support both offline and real-time data processing, ensuring the stability and security of cross-region data transmission.

[0071] Each data table in the source database corresponds to a message topic, that is, the data table and the message topic have a one-to-one correspondence.

[0072] The synchronization operation message is a message message used to determine a data table change operation on a source database, so as to instruct a synchronization operation to be performed on an expanded database corresponding to the source database.

[0073] Among them, the synchronization operation message in this application can also be simply referred to as a message.

[0074] In a specific implementation, the server can parse the data operation log of the source database to be expanded. The data operation log records the operation log of the MySQL database during the DML (data manipulation language, including but not limited to INSERT, UPDATE, and DELETE) process. In this way, the server can determine the change operation record corresponding to at least one data table change operation of the source database, and generate a message message corresponding to the data table change operation based on the change operation record corresponding to the data table change operation as the synchronization operation message corresponding to the data table change operation. Since the data table change operation is performed on each data table of the source database, and each data table has a corresponding message topic, the synchronization operation message corresponding to the data table change operation can be matched with the message topic corresponding to the corresponding data table. The corresponding message topic will be used as the target message topic specified by the synchronization operation message. In this way, the server can push the synchronization operation message corresponding to the at least one data table change operation to the message queue corresponding to the target message topic. Among them, the target message topic is the message topic corresponding to the data table operated by the data table change operation.

[0075] Step S120 : generating consumer program configuration information according to the data table identifier corresponding to each data table in the source database.

[0076] The consumer program configuration information includes the mapping relationship between each data table in the source database and the message consumption configuration information.

[0077] Among them, the message consumption configuration information includes message subject and routing information.

[0078] In the specific implementation, each data table in the source database can be configured with corresponding routing information, and the routing information and message subject can constitute the message consumption configuration information. In order to establish a mapping relationship between each data table and the message consumption configuration information, the server can obtain the data table identifier corresponding to each data table. The data table identifier is used to represent the data table. In this way, the server can establish a mapping relationship between each data table and the message consumption configuration information based on the data table identifier corresponding to each data table. The above mapping relationship serves as the consumer program configuration information.

[0079] Step S130: Generate consumer programs that subscribe to different message topics based on the consumer program configuration information.

[0080] In a specific implementation, the server can obtain a consumer format template and generate a corresponding consumer program for each data table based on the consumer program configuration information, that is, based on the mapping relationship between each data table and the message consumption configuration information. Specifically, the message consumption configuration information includes message topics and routing information. The server can replace the message topics and routing information in the consumer format template based on the message topics and routing information corresponding to the data table to generate a corresponding consumer program for each data table. Because each data table corresponds to a different message topic, consumer programs can be generated that subscribe to different message topics.

[0081] Step S140: executing each consumer program.

[0082] The consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic.

[0083] The consumer program is further configured to execute a data table change operation matching the target synchronization operation message in a sub-database of the target database according to the corresponding routing information.

[0084] The target database is the expanded database corresponding to the source database.

[0085] In the specific implementation, the server can execute each consumer program, and the consumer program can pull the synchronization operation message from the message queue corresponding to the subscribed message topic as the target synchronization operation message corresponding to the consumer program. In this way, the consumer program can execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information, and the target database is the expanded database corresponding to the source database.

[0086] The op_type operation type identifier in the target synchronization operation message informs the consumer program of the corresponding data table modification operation in the source database. The data dictionary for op_type includes INSERT, UPDATE, and DELETE. For example, the operation type identifier for an INSERT operation is "op_type":"INSERT". This allows the consumer program to execute the data table modification operation matching the target synchronization operation message in the target database's shard.

[0087] In practical applications, a server can have at least two operating logics. One operating logic is used to execute step S110. Specifically, the server can execute step S110 via a data replication platform (e.g., a DRP platform), which is the Kafka producer. The other operating logic is used to execute steps S120 to S140, and the consumer program is the Kafka consumer.

[0088] In the above data synchronization method, by parsing the data operation log of the source database to be expanded, for at least one data table change operation of the source database, at least one synchronization operation message corresponding to the data table change operation is pushed to the message queue corresponding to the target message topic; the target message topic is the message topic corresponding to the data table operated by the data table change operation; then, according to the data table identifier corresponding to each data table in the source database, the consumer program configuration information is generated; the consumer program configuration information includes the mapping relationship between each data table in the source database and the message consumption configuration information; the message consumption configuration information includes the message topic and routing information; then, according to the consumer program configuration information, a consumer program subscribing to different message topics is generated; then, each consumer program is executed; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information; the target database is the expanded database corresponding to the source database.

[0089] In this way, since there is a mapping relationship between each data table and the message consumption configuration information, the mapping relationship can be used to flexibly generate consumer programs that subscribe to different message topics. Therefore, the consumer program can perform data table change operations that match the pulled target synchronization operation messages in the expanded database corresponding to the source database according to the corresponding routing information. This realizes the flexible synchronization of data in the source data to the expanded database according to the corresponding message consumption configuration information, and realizes data synchronization during non-system downtime. There is no need to import data when the system is down, which shortens the waiting time for data synchronization and effectively improves the efficiency of database expansion.

[0090] In one embodiment, Figure 2As shown, step S130 generates consumer program configuration information according to the data table identifier corresponding to each data table in the source database, including:

[0091] Step S210: Obtain the data table identifier corresponding to each data table in the source database.

[0092] Step S220: establishing a first mapping relationship between each data table identifier and each message subject, and establishing a second mapping relationship between each data table identifier and each routing information.

[0093] Step S230: Generate consumer program configuration information according to the first mapping relationship and the second mapping relationship.

[0094] In actual applications, the data table identifier may be the table name corresponding to the data table.

[0095] In a specific implementation, when the server generates consumer program configuration information based on the data table identifiers corresponding to each data table in the source database, the server can obtain the data table identifiers corresponding to each data table in the source database. Then, the server can establish a mapping relationship between each data table identifier and each message topic as a first mapping relationship, and can also establish a mapping relationship between each data table identifier and each routing information as a second mapping relationship. In this way, since the message consumption configuration information includes the message topic and routing information, based on the first mapping relationship and the second mapping relationship, the server can obtain the mapping relationship between each data table in the source database and the message consumption configuration information to generate the consumer program configuration information.

[0096] In the process of obtaining the data table identifier corresponding to each data table, the server may query the source database for the data table identifier corresponding to each data table in response to an SQL (Structured Query Language) instruction.

[0097] The technical solution of this embodiment is to obtain the data table identifier corresponding to each data table in the source database; establish a first mapping relationship between each data table identifier and each message topic, and establish a second mapping relationship between each data table identifier and each routing information; in this way, since the message consumption configuration information includes the message topic and routing information, according to the first mapping relationship and the second mapping relationship, the mapping relationship between each data table in the source database and the message consumption configuration information can be accurately obtained to generate consumer program configuration information.

[0098] In one embodiment, a first mapping relationship is established between each data table identifier and each message topic, including: obtaining a message topic format template; adding the data table identifier corresponding to each data table to the message topic format template to obtain the message topic corresponding to each data table; and obtaining the first mapping relationship based on the message topic corresponding to each data table.

[0099] In a specific implementation, when establishing the first mapping relationship between each data table identifier and each message topic, the server may obtain a message topic format template. This message topic format template is a naming convention for message topics. The message topic format template may be as follows: "DRP_MYSQL_" + MySQL database server SET alias + "_" + db name + "_" + table name; where table name represents the data table identifier. Then, by appending the data table identifier corresponding to each data table to the message topic format template, the message topic corresponding to each data table can be obtained.

[0100] For example, when the SET alias of the MySQL database server is "DB0001", the data table identified as "TABLE1" has a corresponding message subject of "DRP_MYSQL_DB0001_DBNAME_TABLE1"; the data table identified as "TABLE2" has a corresponding message subject of "DRP_MYSQL_DB0001_DBNAME_TABLE2".

[0101] In this way, according to the message topics corresponding to the data tables, the server can obtain the first mapping relationship between the data table identifiers and the message topics.

[0102] The technical solution of this embodiment obtains a message subject format template; the data table identifier corresponding to each data table is added to the message subject format template to obtain the message subject corresponding to each data table, so that the data table operated by the data table change operation corresponds to a different message subject, so that the synchronization operation message corresponding to the data table change operation can specify the message subject corresponding to the data table operated by the data table change operation; then, according to the message subject corresponding to each data table, the first mapping relationship between each data table identifier and each message subject can be accurately obtained.

[0103] In one embodiment, according to the corresponding routing information, a data table change operation that matches the target synchronization operation message is performed in a sub-library of the target database, including: according to the corresponding routing information, determining the sub-library to be operated on in the target database as the target sub-library corresponding to the target synchronization operation message; in the target sub-library corresponding to the target synchronization operation message, performing a data table change operation that matches the target synchronization operation message.

[0104] In the specific implementation, when the consumer program executes the data table change operation that matches the target synchronization operation message in the sub-database of the target database according to the corresponding routing information, the consumer program corresponding to each data table can determine the sub-database to be executed in the target database according to the corresponding routing information, as the target sub-database corresponding to the target synchronization operation message, and then, in the target sub-database corresponding to the target synchronization operation message, execute the data table change operation that matches the pulled target synchronization operation message.

[0105] The technical solution of this embodiment can accurately determine the sub-library for the operation to be performed in the target database based on the corresponding routing information, so as to serve as the target sub-library corresponding to the target synchronization operation message; then, in the target sub-library corresponding to the target synchronization operation message, the data table change operation matching the target synchronization operation message is executed. In this way, the data in the source database can be accurately synchronized to the sub-library to be stored in the target database.

[0106] In one embodiment, the routing information includes a routing field and a routing algorithm; based on the corresponding routing information, the sub-library for the operation to be executed is determined in the target database, including: using the corresponding routing algorithm, calculating the corresponding routing field, and determining the sub-library identifier; and using the sub-library in the target database corresponding to the sub-library identifier as the sub-library for the operation to be executed.

[0107] In a specific implementation, routing information includes routing fields and routing algorithms. When the consumer program determines the sub-database for the operation to be executed in the target database based on the corresponding routing information, the consumer program can use the corresponding routing algorithm to calculate the corresponding routing fields, determine the sub-database identifier, and then use the sub-database in the target database that corresponds to the sub-database identifier as the sub-database for the operation to be executed.

[0108] In some embodiments, different data tables in the same database support various routing algorithms, including but not limited to common routing algorithms such as consistent hash routing algorithms and specified routing algorithms, and can also be custom routing algorithms, that is, data tables can support various custom routings. In this way, the routing algorithms corresponding to each data table can be flexibly configured.

[0109] Thus, the consumer program configuration information includes the mapping relationship between each data table in the source database and the message consumption configuration information. The message consumption configuration information includes the message subject and routing information. The routing information includes the routing field and routing algorithm. To facilitate understanding by those skilled in the art, Table 1 provides a consumer program configuration information:

[0110] Table 1 Consumer program configuration information

[0111]

[0112] For example, if there are 8 shards in the target database (SET1 database, SET2 database, SET3 database, SET4 database, SET5 database, SET6 database, SET7 database and SET8 database), the routing field of TABLE1 is routerKey1, and the shard identifier calculated by the corresponding routing algorithm, the consistent hash routing algorithm, is 5, then the SET5 database in the target database will be used as the shard to be operated on. The consumer program will execute the data table change operation matching the target synchronization operation message in the SET5 database, and other shards do not need to be processed.

[0113] The technical solution of this embodiment adopts the corresponding routing algorithm to calculate the corresponding routing field and determine the sub-library identifier; the sub-library corresponding to the sub-library identifier in the target database is used as the sub-library for the operation to be executed; in this way, each data table corresponds to a different routing algorithm, and the corresponding routing field is calculated according to the routing algorithm, so that the sub-library for the operation to be executed that matches the target synchronization operation message generated based on the data table change operation can be determined, so that the data table change operation matching the target synchronization operation message is executed in the sub-library for the operation to be executed, thereby realizing the flexible allocation of data in the source data to the sub-libraries in the expanded database according to the corresponding routing algorithm, routing field and message subject.

[0114] In one embodiment, the data operation log of the source database to be expanded is parsed, and for at least one data table change operation of the source database, a synchronization operation message corresponding to at least one data table change operation is pushed to a message queue corresponding to a target message topic, including: parsing the data operation log to obtain a change operation record corresponding to at least one data table change operation of the source database; encapsulating the change operation record into a message message in a preset format to obtain a synchronization operation message corresponding to at least one data table change operation of the source database.

[0115] In a specific implementation, when the server parses the data operation log of the source database to be expanded and pushes at least one synchronization operation message corresponding to at least one data table change operation of the source database to the message queue corresponding to the target message topic, the server can parse the data operation log to obtain the change operation record corresponding to at least one data table change operation of the source database, and then encapsulate the change operation record into a message message in a preset format to obtain the synchronization operation message corresponding to at least one data table change operation of the source database.

[0116] The technical solution of this embodiment can obtain a change operation record corresponding to at least one data table change operation on the source database by parsing the data operation log; then, the change operation record is encapsulated into a message message in a preset format to obtain a synchronization operation message corresponding to at least one data table change operation on the source database, so that the consumer program can accurately determine the data table change operation performed in the source database through the synchronization operation message.

[0117] In one embodiment, the message queue corresponding to the target message topic includes at least one partition message queue; pushing at least one synchronization operation message corresponding to a data table change operation to the message queue corresponding to the target message topic includes: determining the partition identifier corresponding to the synchronization operation message under the target message topic; pushing the synchronization operation message to at least one partition message queue, the partition message queue corresponding to the partition identifier.

[0118] In specific implementations, a Kafka cluster contains one or more Kafka servers. An independent Kafka server is called a broker. When pushing a synchronization operation message corresponding to at least one data table change operation to the message queue corresponding to the target message topic, the broker is used to receive the synchronization operation messages sent by the producer and route these synchronization operation messages to the message queue in the broker, so that Kafka can dynamically add the synchronization operation messages sent by the producer to the disk and give each synchronization operation message an offset. Therefore, for Kafka, a broker is an instance of an application. In order to balance the load, the message topic is divided into multiple partitions, and each broker stores one or more partitions. Each partition can be used as a partition message queue. Therefore, the message queue corresponding to the target message topic includes at least one partition message queue.

[0119] In this way, when the server pushes the synchronization operation message corresponding to at least one data table change operation to the message queue corresponding to the target message topic, the server can determine the partition identifier corresponding to the synchronization operation message under the target message topic, send the synchronization operation message to the broker, and route the synchronization operation message to at least one partition message queue through the broker, the partition message queue corresponding to the partition identifier.

[0120] In actual applications, the data replication platform acts as a Kafka producer, determines the partition identifier corresponding to the synchronization operation message under the target message topic, sends the synchronization operation message to the broker, and the broker sends the synchronization operation message to at least one partition message queue. In the process of the partition message queue corresponding to the partition identifier, in order to facilitate understanding by those skilled in the art, Figure 3A diagram of the principle of Kafka producer submitting synchronous operation messages is provided. Figure 3 As shown, when a synchronous operation message is sent to a broker using the send() method (producer.send(record)), it must go through a series of processes, including the serializer and partitioner, before it can actually be sent to the broker. After serialization, the synchronous operation message needs to determine the partition to which it is sent. If the ProducerRecord of the synchronous operation message specifies the partition field, the partitioner is not required, as the partition represents the partition identifier corresponding to the partition to which it is sent. If the ProducerRecord of the synchronous operation message does not specify the partition field, the partitioner is required. The partitioner calculates the partition value based on the key field to obtain the partition identifier. The partitioner's function is to assign a partition to the synchronous operation message. After determining the partition identifier for the synchronous operation message within the target message topic, the synchronous operation message is sent to the broker. If the synchronous operation message fails to be written, the broker uses the partitioner to assign the synchronous operation message to a partition for retry.

[0121] The technical solution of this embodiment is that the message queue corresponding to the target message topic includes at least one partition message queue; by determining the partition identifier corresponding to the synchronization operation message under the target message topic; pushing the synchronization operation message to at least one partition message queue, the partition message queue corresponding to the partition identifier; in this way, by allocating different synchronization operation messages to different partition message queues, load balancing is facilitated.

[0122] In another embodiment, Figure 4 As shown, a data synchronization method is provided, which is described by taking the method applied to a server as an example, and includes the following steps:

[0123] Step S410 , parsing the data operation log of the source database to be expanded, and obtaining a change operation record corresponding to at least one data table change operation on the source database.

[0124] Step S420: encapsulate the change operation record into a message message in a preset format to obtain a synchronization operation message corresponding to at least one data table change operation on the source database.

[0125] Step S430: Push at least one synchronization operation message corresponding to the data table modification operation to the message queue corresponding to the target message topic.

[0126] Step S440: Obtain the data table identifier corresponding to each data table in the source database.

[0127] Step S450: establishing a first mapping relationship between each data table identifier and each message subject, and establishing a second mapping relationship between each data table identifier and each routing information.

[0128] Step S460: Generate consumer program configuration information according to the first mapping relationship and the second mapping relationship.

[0129] Step S470: Generate consumer programs that subscribe to different message topics based on the consumer program configuration information.

[0130] Step S480: Execute each consumer program.

[0131] It should be noted that the specific limitations of the above steps can refer to the specific limitations of a data synchronization method above.

[0132] In some embodiments, in order to facilitate understanding by those skilled in the art, when the DRP platform and the Kafka message queue are used for data synchronization, Figure 5 This article provides a flow chart of DRP (Data Replication Platform) data parsing and replication and Kafka data synchronization. Figure 5 As shown, the parsing service and the storage service run on the server in the above embodiment. The parsing service is equivalent to a Kafka producer; the storage service is used to generate consumer programs and is equivalent to a Kafka consumer. The message engine cluster is a Kafka cluster. The monitoring service corresponding to the parsing service is used to monitor data anomalies during the operation of the parsing service, and the monitoring service corresponding to the storage service is used to monitor data anomalies during the operation of the storage service.

[0133] In actual applications, the parsing service and the storage service can also run on different servers respectively, which is not specifically limited here.

[0134] In some embodiments, a Kafka cluster typically includes multiple brokers. ZooKeeper is used to manage and coordinate Kafka brokers. Each Kafka broker coordinates with other Kafka brokers through ZooKeeper. When a new broker is added to the Kafka system or a broker fails, the ZooKeeper service notifies producers and consumers. Producers and consumers then begin coordinating with other brokers.

[0135] Specifically, Zookeeper allows producers to push synchronization operation messages to every proxy. When a proxy fails, Zookeeper notifies producers not to send synchronization operation messages to the failed proxy. Consumers also stop receiving synchronization operation messages from the failed proxy and instead receive them from other healthy proxies. If, during expansion, it is discovered that proxy resources are insufficient to rapidly synchronize large amounts of data, additional proxies can be added. For example, if there were originally only four proxies, adding four more will result in a total of eight. Zookeeper notifies producers of the new available proxies and can simultaneously push synchronization operation messages to them. Consumers can then receive synchronization operation messages from one of eight proxies, rather than from one of four. This new proxy distributes the load.

[0136] To balance the load, topics are divided into multiple partitions, and each broker stores one or more partitions. Multiple producers and consumers can produce and obtain synchronous operation messages at the same time. Kafka's Topic (message topic) can be logically considered as a queue (message queue). Each consumption must specify its Topic, which can be simply understood as specifying which queue to put this synchronous operation message into. In order to linearly increase Kafka's throughput, the Topic is physically divided into one or more Partitions. Each Partition physically corresponds to a folder, which stores all synchronous operation messages and index files of this Partition.

[0137] To ensure data correctness, all synchronization operation messages in the latest log segment can be iterated to verify the validity of each message item. A message item is valid only if its size plus the offset is less than the file size and the CRC32 value of the payload in the message is equal to the CRC value stored in the message. If data corruption is detected, the file is truncated according to the last valid offset. The message sender can rewind back to any position and re-consume. When the consumer fails, the smallest offset can be selected to re-read the consumption message.

[0138] To ensure the reliability of message delivery, you can choose the corresponding message delivery mode according to different needs in actual applications. Generally, there are three types:

[0139] At most once, the message may be lost, but it will never be transmitted repeatedly.

[0140] At least one message will never be lost, but it may be transmitted repeatedly

[0141] Exactly once Each message will be transmitted once and only once, which is often what users want.

[0142] As described above, the technical solution of this embodiment adopts DRP data parsing replication and Kafka data synchronization technology to realize the distributed system's support for flexible routing algorithm sharding expansion, which can solve the problem of long expansion time of Mysql database distributed system with large data volume, and the need for overall rollback of the system in the event of unexpected situations, resulting in the inability to meet emergency rollback data backup requirements.

[0143] Kafka provides message persistence with a time complexity of O(1), guaranteeing constant-time access even for terabytes of data. DRP and Kafka can synchronize and clean data during non-downtime, eliminating the need for data import and cleanup during downtime to minimize data synchronization wait times. Furthermore, the Kafka client supports most current mainstream languages, including C, C++, Erlang, Java, .net, Perl, PHP, Python, Ruby, Go, and Javascript. You can use any of these languages ​​to communicate with the Kafka server.

[0144] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0145] Based on the same inventive concept, an embodiment of the present application further provides a data synchronization device for implementing the aforementioned data synchronization method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the data synchronization device provided below can be found in the above-mentioned limitations of the data synchronization method and will not be further elaborated here.

[0146] In one embodiment, Figure 6 As shown, a data synchronization device is provided, including: a parsing module 610, an information generating module 620, a program generating module 630 and an execution module 640, wherein:

[0147] Parsing module 610 is used to parse the data operation log of the source database to be expanded, and for at least one data table change operation of the source database, push the synchronization operation message corresponding to the at least one data table change operation to the message queue corresponding to the target message topic; the target message topic is the message topic corresponding to the data table operated by the data table change operation.

[0148] The information generation module 620 is used to generate consumer program configuration information based on the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information includes the mapping relationship between the data tables in the source database and the message consumption configuration information; the message consumption configuration information includes the message subject and routing information.

[0149] The program generation module 630 is used to generate consumer programs that subscribe to different message topics according to the consumer program configuration information.

[0150] Execution module 640 is used to execute each of the consumer programs; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to execute the data table change operation matching the target synchronization operation message in the sub-database of the target database according to the corresponding routing information; the target database is the expanded database corresponding to the source database.

[0151] In one embodiment, the information generation module 620 is specifically used to obtain the data table identifier corresponding to each data table in the source database; establish a first mapping relationship between each data table identifier and each message topic, and establish a second mapping relationship between each data table identifier and each routing information; and generate the consumer program configuration information based on the first mapping relationship and the second mapping relationship.

[0152] In one embodiment, the information generation module 620 is specifically used to obtain a message subject format template; add the data table identifier corresponding to each data table to the message subject format template to obtain the message subject corresponding to each data table; and obtain the first mapping relationship based on the message subject corresponding to each data table.

[0153] In one embodiment, the execution module 640 is specifically used to determine the sub-database on which the operation to be performed is performed in the target database according to the corresponding routing information, as the target sub-database corresponding to the target synchronization operation message; and in the target sub-database corresponding to the target synchronization operation message, perform a data table change operation that matches the target synchronization operation message.

[0154] In one embodiment, the routing information includes a routing field and a routing algorithm; the execution module 640 is specifically used to adopt the corresponding routing algorithm, calculate the corresponding routing field, and determine the sub-library identifier; and use the sub-library corresponding to the sub-library identifier in the target database as the sub-library for the operation to be executed.

[0155] In one embodiment, the parsing module 610 is specifically used to parse the data operation log to obtain a change operation record corresponding to at least one data table change operation of the source database; encapsulate the change operation record into a message message in a preset format to obtain a synchronization operation message corresponding to at least one data table change operation of the source database.

[0156] In one embodiment, the message queue corresponding to the target message topic includes at least one partition message queue; the parsing module 610 is specifically used to determine the partition identifier corresponding to the synchronization operation message under the target message topic; and push the synchronization operation message to the at least one partition message queue, the partition message queue corresponding to the partition identifier.

[0157] Each module in the above-mentioned data synchronization device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0158] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store consumer program configuration information data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a data synchronization method.

[0159] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0160] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0161] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0162] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0164] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0165] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Parse the data operation log of the source database to be expanded, and for at least one data table change operation in the source database, push a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic; the target message topic is the message topic corresponding to the data table operated on by the data table change operation; Generate consumer program configuration information according to the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information includes a mapping relationship between the data tables in the source database and the message consumption configuration information; the message consumption configuration information includes a message subject and routing information; Generate consumer programs that subscribe to different message topics according to the consumer program configuration information; Executing each of the consumer programs; the consumer program is used to pull the target synchronization operation message from the message queue corresponding to the subscribed message topic, and is also used to perform a data table change operation matching the target synchronization operation message in a sub-database of the target database according to the corresponding routing information; the target database is the expanded database corresponding to the source database; Generating consumer program configuration information according to data table identifiers corresponding to each data table in the source database includes: Obtaining data table identifiers corresponding to each data table in the source database; Establishing a first mapping relationship between each of the data table identifiers and each of the message topics, and establishing a second mapping relationship between each of the data table identifiers and each of the routing information; generating the consumer program configuration information according to the first mapping relationship and the second mapping relationship; The establishing of a first mapping relationship between each of the data table identifiers and each of the message topics includes: Get the message subject format template; Supplement the data table identifier corresponding to each data table to the message subject format template to obtain the message subject corresponding to each data table; The first mapping relationship is obtained according to the message topics corresponding to each of the data tables.

2. The method according to claim 1, characterized in that The step of executing, in a sub-database of a target database according to the corresponding routing information, a data table change operation matching the target synchronization operation message includes: According to the corresponding routing information, determining a sub-database to be operated on in the target database as a target sub-database corresponding to the target synchronization operation message; In the target sub-database corresponding to the target synchronization operation message, a data table change operation matching the target synchronization operation message is executed.

3. The method according to claim 2, characterized in that The routing information includes a routing field and a routing algorithm; and determining a sub-database for executing an operation in the target database according to the corresponding routing information includes: Use the corresponding routing algorithm to calculate the corresponding routing field and determine the sub-database identifier; The sub-library in the target database corresponding to the sub-library identifier is used as the sub-library for the operation to be executed.

4. The method according to claim 1, wherein The step of parsing the data operation log of the source database to be expanded, and pushing a synchronization operation message corresponding to at least one data table change operation of the source database to a message queue corresponding to a target message topic, includes: Parsing the data operation log to obtain a change operation record corresponding to at least one data table change operation on the source database; The change operation record is encapsulated into a message message in a preset format to obtain a synchronization operation message corresponding to the at least one data table change operation of the source database.

5. The method according to claim 1, wherein The message queue corresponding to the target message topic includes at least one partition message queue; and pushing the synchronization operation message corresponding to the at least one data table change operation to the message queue corresponding to the target message topic includes: Determine the partition identifier corresponding to the synchronization operation message under the target message topic; Push the synchronization operation message to the at least one partition message queue, the partition message queue corresponding to the partition identifier.

6. A data synchronization device, characterized in that: The device comprises: a parsing module configured to parse the data operation log of the source database to be expanded, and, for at least one data table change operation of the source database, push a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic; the target message topic being a message topic corresponding to the data table operated on by the data table change operation; An information generation module, configured to generate consumer program configuration information according to data table identifiers corresponding to the data tables in the source database; the consumer program configuration information includes a mapping relationship between the data tables in the source database and the message consumption configuration information; the message consumption configuration information includes a message subject and routing information; A program generation module, configured to generate consumer programs that subscribe to different message topics according to the consumer program configuration information; An execution module, configured to execute each of the consumer programs; the consumer program is configured to pull a target synchronization operation message from a message queue corresponding to the subscribed message topic, and further configured to execute a data table change operation matching the target synchronization operation message in a sub-database of a target database according to the corresponding routing information; the target database is an expanded database corresponding to the source database; The information generation module is specifically configured to obtain a data table identifier corresponding to each data table in the source database; establish a first mapping relationship between each data table identifier and each message topic, and establish a second mapping relationship between each data table identifier and each routing information; and generate the consumer program configuration information based on the first mapping relationship and the second mapping relationship; The information generation module is also specifically used to obtain a message subject format template; add the data table identifier corresponding to each data table to the message subject format template to obtain the message subject corresponding to each data table; and obtain the first mapping relationship based on the message subject corresponding to each data table.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images