Data synchronization method, system and device, computer equipment, computer software and storage medium

Abstract

The invention discloses a data synchronization method, system and device, computer equipment, computer software and a storage medium, and the method comprises the steps: obtaining a first operation message from a cache queue, the first operation message being used for recording an operation event for a first database; determining a target thread according to a message identifier of the first operation message and thread information of a thread pool corresponding to a second database; analyzing the first operation message through the target thread to obtain the operation event, and executing the operation event on a second database through the target thread. According to the embodiment of the invention, when batch processing is carried out on the operation messages in the cache queue through the thread pool, the target thread for processing the first operation message is determined according to the message identifier of the operation message and the thread information of the thread pool, and the corresponding operation messages are sequentially processed through the target thread. And the data consistency of data synchronization is ensured on the basis of realizing rapid data synchronization.

Description

Technical Field

[0001] The present application relates to the technical field of data synchronization, and in particular to a data synchronization method, system, device, computer equipment, computer software and storage medium (computer-readable storage medium). Background Art

[0002] With the rapid development of the Internet, the emergence of massive data and changes in data structure have brought huge challenges to all walks of life in data processing issues; data synchronization is an extremely common and important data processing task, which involves the migration of data from one location to another. Data migration was initially done manually, which not only consumed manpower but also had a slow data synchronization speed. Summary of the invention

[0003] In response to the above technical problems, the embodiments of the present application provide a data synchronization method, system, apparatus, computer equipment, computer software and storage medium.

[0004] In a first aspect, an embodiment of the present application provides a data synchronization method, including:

[0005] Obtaining a first operation message from the cache queue, where the first operation message is used to record an operation event on the first database;

[0006] Determine a target thread according to the message identifier of the first operation message and the thread information of the thread pool corresponding to the second database;

[0007] The first operation message is parsed through the target thread to obtain the operation event, and the operation event is executed on the second database through the target thread.

[0008] In a second aspect, an embodiment of the present application provides a data synchronization system, wherein the data synchronization system includes a master server and a slave server, wherein:

[0009] The main server is used to determine a first operation message according to a database log of the first database and send the first operation message to the message middleware, where the first operation message is used to record an operation event on the first database;

[0010] A message middleware, configured to receive the first operation message and save the message to a first partition corresponding to the first database;

[0011] A message middleware, configured to receive a message request from a slave server, obtain the first operation message from the first partition, and send the message to a cache queue of the slave server;

[0012] A slave server, configured to obtain the first operation message from a cache queue;

[0013] The slave server is configured to determine a target thread according to a message identifier of the first operation message and thread information of a thread pool corresponding to the second database;

[0014] The slave server is configured to parse the first operation message through the target thread to obtain the operation event, and execute the operation event on the second database through the target thread.

[0015] In a third aspect, an embodiment of the present application provides a data synchronization device, including:

[0016] An acquisition module, used for acquiring a first operation message from a cache queue, where the first operation message is used for recording an operation event on a first database;

[0017] A determination module, configured to determine a target thread according to a message identifier of the first operation message and thread information of a thread pool corresponding to the second database;

[0018] A synchronization module is used to parse the first operation message through the target thread to obtain the operation event, and execute the operation event on the second database through the target thread.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer device, wherein the computer device is at least one of a master server and a slave server, and the computer device includes a memory storing multiple instructions; a processor loads instructions from the memory to execute the steps of any data synchronization method provided in the embodiment of the present application.

[0020] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps of any data synchronization method provided in an embodiment of the present application.

[0021] In a sixth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of any one of the data synchronization methods provided in the embodiments of the present application.

[0022] The scheme of the embodiment of the present application is adopted to obtain operation messages from the cache queue, and the operation messages record operation events on the first database. When the operation messages are batch processed, in order to ensure the processing order of each operation message, the target thread for processing each operation message is determined according to the message identifier of the operation message and the thread information of the thread pool corresponding to the second database, the operation message is parsed by the target thread to obtain the operation event, and the operation event corresponding to the operation message is sequentially executed on the second database through the target thread, so that the second database and the first database execute the same operation event, and data synchronization is performed between the second database and the first database. In the embodiment of the present application, when the operation messages in the cache queue are batch processed by the thread pool, the target thread for processing the first operation message is determined according to the message identifier of the operation message and the thread information of the thread pool, and the corresponding operation message is sequentially processed by the target thread, so as to ensure the data consistency of data synchronization on the basis of achieving fast data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is one of the scenario diagrams of the data synchronization method provided in the embodiments of the present application;

[0025] Figure 2 This is the second scenario diagram of the data synchronization method provided in the embodiment of the present application;

[0026] Figure 3 is a schematic diagram of an application environment of the data synchronization method provided in an embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of a scenario based on log master-slave replication in the data synchronization method provided in an embodiment of the present application;

[0028] Figure 5 This is a schematic diagram of a scenario for introducing the principle of data synchronization in the data synchronization method provided in an embodiment of the present application;

[0029] Figure 6 It is a flowchart of an embodiment of a data synchronization method provided in an embodiment of the present application;

[0030] Figure 7 It is a schematic diagram of a scenario of log parsing in a data synchronization method provided in an embodiment of the present application;

[0031] Figure 8It is a schematic diagram of a scenario of target partition message subscription in a message middleware in a data synchronization method provided in an embodiment of the present application;

[0032] Fig. 9 It is a scenario diagram of the software architecture in the data synchronization method provided in the embodiment of the present application;

[0033] Fig.10 is a schematic diagram of a scenario of data synchronization application in the data synchronization method provided in an embodiment of the present application;

[0034] Fig.11 It is a schematic diagram of the architecture of the data synchronization system provided in the embodiment of the present application;

[0035] Fig.12 is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0036] Fig.13 It is a schematic diagram of the internal structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0038] The data synchronization method in one embodiment of the present application can be run on a local terminal device or a server. When the game interaction method is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0039] The quality of technical solutions in data synchronization methods is usually evaluated based on at least one indicator of data consistency, data latency, data scale, data source type, synchronization object, and architecture, among which:

[0040] (1) Data consistency: refers to the simultaneous change or sameness of data, that is, the data copies stored in multiple nodes maintain the same state and value to ensure the accuracy and reliability of the data. In a distributed system, due to factors such as data replication, sharding, and concurrent execution of distributed transactions, data inconsistency may occur. At this time, it is necessary to ensure data consistency. Data consistency is the most basic guarantee indicator for data synchronization;

[0041] (2) Data latency: The time difference between data from the source database to the target database. For example, scenarios such as bank transfers have very high requirements for data consistency and timeliness.

[0042] (3) Data scale: refers to the magnitude or size of data, usually used to describe the size or capacity of a data set. Different business data magnitudes have different performance requirements, and the complexity of data synchronization system architecture design also varies. Data synchronization requires a trade-off between data synchronization solutions based on data scale.

[0043] (4) Data source type: Data source types mainly include relational databases, non-relational databases, cloud computing service platforms, distributed computing frameworks, big data processing frameworks, and spreadsheet software. Data source types may be diverse. Homogeneous storage is ideal, and generally synchronization to heterogeneous data sources is required.

[0044] (5) Synchronization object: The synchronization object refers to the object for data synchronization. For example, the synchronization object can be the data itself or the data structure. Whether the metadata message such as the data correspondence table structure changes can be automatically synchronized is also a consideration indicator of operation and maintenance costs.

[0045] (6) Architecture: refers to the architecture used for the data synchronization system. The data synchronization system architecture directly determines the system complexity, scalability, development and implementation costs, operation and maintenance costs, machine resource costs, etc.

[0046] In order to better understand a data synchronization method, system, device, equipment and storage medium provided in the embodiments of the present application, the embodiments of the present application first introduce the background technology of existing data synchronization, and refer to Figure 1 , Figure 1 This is a schematic diagram of a scenario of a data synchronization method provided in an embodiment of the present application, including:

[0047] The data synchronization method in the prior art involves: a business main system, system A, message middleware and system B, wherein the business main system includes an active database, and the business data in the business main system is changed, that is, the data in the source database is changed, and it is determined whether it is a strong transaction; if it is a strong transaction, the write API provided by system A is called to perform data synchronization operation; if it is not a strong transaction, data synchronization is performed through the message middleware, wherein,

[0048] (1) Arrange at the main business entrance of the main business system, and write the main business table when there is a business data change;

[0049] (2) The main table is written successfully, and the synchronization logic is performed to determine whether it is a strong transaction:

[0050] If it is a strong transaction (also called synchronous write), determine the strong transaction type. If it is a local transaction, manage it through Spring@Transactional to achieve synchronization. If it is a distributed transaction, call the write API provided by system A to write data to target table A. If the target table A is written successfully, coordinate the main table in the source database to commit, otherwise rollback;

[0051] If it is not a strong transaction (also called asynchronous write): the main business entrance of the main business system produces messages, sends them to the message middleware, notifies system B to consume them, and writes them to the target table B. If the write is successful, it ends directly; if it fails, the message middleware retry mechanism is used to compensate to ensure eventual consistency.

[0052] The data synchronization method in the prior art has the following disadvantages:

[0053] (1) Whether writing synchronously or asynchronously, it is difficult to ensure consistency;

[0054] Synchronous write distributed transactions, due to the high complexity of distributed systems, including network instability, service failures, etc.

[0055] Asynchronous writing may still fail after being retried through the message middleware MQ, for example, due to database failure, network anomalies, business anomalies, etc. Even if the retry write succeeds, data delay may cause the data written later to be overwritten and restored, resulting in dirty data.

[0056] (2) The main business system calls the API of system A for synchronization, which has the following disadvantages:

[0057] The API response time of the main business system increases. If more and more subsystems need to be synchronized, it will cause an uncontrollable situation and seriously affect the business experience.

[0058] The hard-coded double-write method is intrusive to the business code and does not take advantage of the maintainability of the code.

[0059] For further information, see Figure 2 , Figure 2 : is another scenario schematic diagram of the data synchronization method provided in an embodiment of the present application, including:

[0060] A synchronization system is created between the business master system and slave system B. The synchronization system includes data reading and writing components, which trigger data synchronization tasks regularly and determine the data synchronization strategy of data synchronization tasks. The data reading and writing components perform batch pulling and batch writing according to the data synchronization strategy.

[0061] (1) The synchronization system triggers a scheduled task. The synchronization system reads the data synchronization strategy based on the scheduled task. The data synchronization strategy is divided into full synchronization coverage and incremental synchronization. Different strategies can be selected for different tables.

[0062] (2) The synchronization system pulls data from the main business system. If it is a full synchronization, the offset starts from 0 and the data is pulled in pages. If it is an incremental synchronization, the offset position of the last synchronization is read and the data is pulled in pages.

[0063] (3) The synchronization system writes the pulled data into system B. If it is a full synchronization, the target table must be cleared for the first overwrite write, and the data pulled by paging in the previous step is written into the target table. If it is an incremental synchronization, the data is written into the target table.

[0064] Data synchronization through the synchronization system also has the following problems:

[0065] (1) It is necessary to query data on the source database of the main business system, which puts pressure on the source database, especially when the frequency of scheduled tasks is high. In serious cases, it may directly cause the source database to crash;

[0066] (2) The data synchronization delay problem cannot be solved. If the scheduled task frequency is set too low, the synchronization delay will be very large. Increasing the scheduled task frequency will lead to an increase in the source database.

[0067] (3) The applicable scenarios are very limited:

[0068] There are performance bottlenecks in both data pulling and writing, especially full coverage synchronization, which is only suitable for scenarios with small data volumes and cannot meet the needs of scenarios with large data volumes.

[0069] It is only applicable to simple data models where there is no association between tables. If the tables are associated, pulling and synchronous writing requires a dependency, which is very difficult to implement.

[0070] Based on the problems existing in the data synchronization solutions in the prior art, a data synchronization method, system, device, equipment and storage medium are proposed in the embodiments of the present application. In order to better understand the data synchronization method, system, device, equipment and storage medium provided in the embodiments of the present application, the application environment applicable to the embodiments of the present application is described below.

[0071] See also Figure 3 , Figure 3Schematic diagram of the application environment of the data synchronization method provided in the embodiment of the present application. As an implementation method, the data synchronization method provided in the embodiment of the present application can be applied to the same computer device. The computer device can be as follows: Figure 3 The server 110 shown in the figure can be connected to the terminal device 120 through a network. The network is used to provide a medium for a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the computer device can also be a smart phone, a laptop computer, etc.

[0072] It should be understood that Figure 3 The server 110, network and terminal device 120 are merely illustrative. Depending on the implementation requirements, there may be any number of servers, networks and terminal devices. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the terminal device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 at the same time.

[0073] The data synchronization method provided in the embodiments of the present application can also be applied to different computer devices. For example, the data synchronization method is applied to computer device A corresponding to the source database, and the data synchronization method is also applied to computer device B corresponding to the target database. The computer device to which the data synchronization method provided in the embodiments of the present application is applied does not constitute a limitation on the technical solution of the present application.

[0074] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is a computer device as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown in the accompanying drawings.

[0075] For ease of understanding, the embodiments of the present application provide an introduction to the principle of data synchronization based on database logs.

[0076] Reference Figure 4 , Figure 4 is a schematic diagram of a scenario based on log master-slave replication in a data synchronization method provided in an embodiment of the present application, wherein:

[0077] The master server Master corresponding to the source database records the changes in the binary log Binary Log, that is, the binary log records the binary log events Binary Log Events; the slave server Slave corresponding to the target database copies the master server Master's Binary Log Events to its own relay log Relay Log; the slave server Slave redoes the events in the relay log to perform data synchronization operations.

[0078] In the embodiment of the present application, the master-slave replication Binlog synchronization principle is used. The synchronization not only supports DML (table record update SQL), but also supports DDL (same database protocol and same table structure), which solves the pain point that manual intervention in operation and maintenance is required when table metadata changes.

[0079] Among them, Binary Log and Binlog are the same concept, which refers to the log used to record data change operations in the database system. Binlog is a term in the MySQL database, which is used to record all data change operations performed on the database, including insert, update, and delete operations. Binary Log is a concept in some other database systems (such as PostgreSQL).

[0080] Reference Figure 5 , Figure 5 This is a schematic diagram of a scenario for introducing the principle of data synchronization in the data synchronization method provided in the embodiment of the present application. In the embodiment of the present application, Canal is used to capture and parse the binary log of the MySQL database in real time. Canal provides a reliable method to capture database change events and transmit these events to an external system for real-time data synchronization and processing. Specifically, it includes:

[0081] Canal simulates the interaction protocol of MySQL Slave. Canal disguises itself as MySQL Slave and sends a dump protocol to MySQLMaster. MySQL Master receives the dump request and starts pushing Binary Log to Slave. Canal parses the Binary Log object for data synchronization.

[0082] Based on the above application environment and data synchronization related background introduction, an embodiment of the data synchronization method of the present application is proposed.

[0083] Reference Figure 6 , Figure 6201 to 203, including:

[0084] Step 201: Obtain a first operation message from a cache queue, where the first operation message is used to record an operation event on a first database.

[0085] The data synchronization method in the embodiment of the present application is applied to the computer device corresponding to the target database. In order to distinguish the description, in the embodiment of the present application, the computer device corresponding to the source database is called the master server, and the computer device corresponding to the target database is called the slave server.

[0086] The data synchronization method is to synchronize the updated target data in the source database to the target database. The existing data synchronization technical solution first determines the target data to be synchronized in the source database, and directly synchronizes the target data to the target database. For such a data synchronization solution, if the data update frequency of the source database is high, it is necessary to access the source database in real time to obtain the target data for synchronization. Such a data synchronization method puts a lot of pressure on the source database; if the data synchronization frequency is reduced, the target database will lag behind. Based on this, the technical solution of this application does not directly obtain the target data from the source database, but synchronizes the data according to the database log of the source database.

[0087] The source database is the database used as the source of data during data replication or data synchronization. Usually, the source database is a database with a copy of the original data, which needs to be copied to the target database or other system for further processing or analysis. The source database can be various types of databases, including but not limited to relational databases, non-relational databases, and column storage databases. Relational databases such as MySQL, Oracle, SQL Server; non-relational databases such as MongoDB, Cassandra, and column storage databases such as HBase.

[0088] The first database in the embodiment of the present application can be used as a source database in one application scenario and as a target database in another application scenario. For ease of understanding, the embodiment of the present application is described by taking the first database as the source database as an example.

[0089] In an embodiment of the present application, an operation message in a cache queue is obtained from a server, and the operation message is used to record operation events on a source database, and the operation events include but are not limited to: write events, transaction events, and table structure events, wherein write events are insert, update, and delete operations on a database, including write operations on a table, and transaction commit and rollback events; transaction events record the start, commit, and rollback operations of a transaction, and transaction events combine multiple write events together to ensure the consistency of database changes; table structure events record changes to a table structure, such as creating a table, changing a table structure, or deleting a table.

[0090] In the embodiment of the present application, the source of the operation message in the cache queue from the server is not limited, for example,

[0091] Source 1: The master server corresponding to the source database directly sends the parsed operation message to the cache queue of the slave server;

[0092] Source 2: The primary server corresponding to the source database sends the parsed operation message to the message middleware, and the slave server obtains the operation message from the message middleware and saves it to its own cache queue;

[0093] Source 3: The master server corresponding to the source database sends the database log to the node corresponding to the log parser. The log parsing node sends the parsed operation message to the cache queue of the slave server.

[0094] Source 4: The primary server corresponding to the source database sends the database log to the log parsing node. The log parsing node sends the parsed operation message to the message middleware, accesses the message middleware from the server to obtain the operation message and saves it to its own cache queue.

[0095] In an embodiment of the present application, the slave server obtains batches of operation messages from the cache queue, and the operation messages are derived from the database log of the source database. This eliminates the need for the slave server to frequently access the master server, thereby reducing the pressure on the master server. At the same time, the slave server can batch process the operation messages in the cache queue to improve data synchronization efficiency.

[0096] Step 202: Determine a target thread according to the message identifier of the first operation message and the thread information of the thread pool corresponding to the second database.

[0097] Get the message identifier of the operation message from the server. The message identifier refers to an identification message that uniquely identifies the operation message. The message identifier can be obtained by processing at least one of the source database type, source database identifier, source data table identifier and message primary key. For example, the format of the message identifier is database type_database name_table name_primary key.

[0098] For example, the message identifier corresponding to the operation message of generating order 1 is: MySQL_db1_order_0_1; the message identifier corresponding to the operation message of generating order 2 is: MySQL_db1_order_0_2; the message identifier corresponding to the operation message of canceling order 2 is: MySQL_db1_order_0_3, and so on; from the server, based on the message identifier of each operation message, it can be determined that the source database in the embodiment of the present application is the relational database MySQL, and operation messages such as generating an order or canceling an order are operations on the data in the data table db1 of the source database. The time sequence of the three operation messages is: the operation message of generating order 1 is earlier than the operation message of generating order 2, and the operation message of generating order 2 is earlier than the operation message of canceling order 2.

[0099] The message identifier of the operation message is processed from the server to obtain the source database, source data table, time information, etc. corresponding to the operation message; in the embodiment of the present application, the message identifier of the operation message is processed from the server, wherein the processing method of the message identifier is determined according to the identification content of the message identifier itself, and / or the content that needs to be obtained from the message identifier; for example, the format of the message identifier is database type_library name_table name_primary key; if the content that needs to be obtained from the server is the source database, the processing method of the message identifier is to extract the library name of the source database; if the content that needs to be obtained from the server is the time sequence of the operation messages, the message identifier is hashed from the server, and the time sequence of the operation messages is determined according to the hash value after the hash calculation.

[0100] In the embodiment of the present application, a slave server can correspond to multiple target databases to be synchronized. The slave server sets a corresponding thread pool for each target database to be synchronized, and distributes operation messages through its thread pool. The thread pool pre-creates a fixed number of threads and uses a queue to receive and schedule tasks, thereby avoiding the overhead of frequent creation and destruction of threads, realizing thread management and reuse, and improving the performance and efficiency of multi-threaded applications.

[0101] Obtain thread information of the target database's thread pool from the server. The thread information includes the number of threads in the thread pool, thread status, etc.

[0102] The slave server determines the time information of each operation message according to the message identifier of the operation message; the slave server determines the thread state of each thread in the thread pool according to the thread information of the thread pool; the slave server determines the target thread according to the time information of each operation message and the thread state of each thread, specifically:

[0103] Mode 1: The slave server performs hash processing on the message identifier of the first operation message to obtain a hash value of the first operation message; performs modulo processing on the hash value and the number of threads in the thread pool to determine the target line.

[0104] For example, the message identifier of the operation message is hashed from the server and converted into a numerical value, wherein the message identifier is hashed to obtain a hash value, and the size of the hash value corresponds one-to-one to the time information of the operation message. The operation message with a smaller hash value has the earliest time information and needs to be processed first; for example, there are 10 operation messages, and the message identifiers of the operation messages are hashed to obtain 10 numerical values; combined with the thread information of the thread pool, if the thread pool has 5 threads, then these 10 numerical values ​​are corresponded to the 5 threads, 1 and 6 are assigned to the first thread, 2 and 7 are assigned to the second thread, and so on.

[0105] Method 2: The slave server determines the time priority of the first operation message according to the message identifier of the first operation message; determines the target thread according to the time priority of the first operation message and the thread status of each thread in the corresponding thread pool of the second database.

[0106] The slave server determines the time priority based on the message identifier of the operation message and the time information of the operation message; the slave server determines the target thread based on the time priority of the operation message and the thread status of each thread in the corresponding thread pool of the database to ensure the sequential execution of data synchronization.

[0107] Step 203: parse the first operation message through the target thread to obtain the operation event, and execute the operation event on the second database through the target thread.

[0108] In the embodiment of the present application, data synchronization from the server includes:

[0109] (1) parsing the first operation message through the target thread to obtain the operation event; the operation event includes: at least one of an insert event, an update event, and a delete event;

[0110] (2) executing the operation event on the second database through the target thread;

[0111] (3) If the operation event is executed, verify the execution result of the second database to obtain a data synchronization result.

[0112] That is, in the embodiment of the present application, the slave server parses the operation message through the target thread to obtain an operation event; the operation event includes: at least one of an insert event, an update event and a delete event; the operation event is executed on the target database through the target thread; after the operation event is executed, the slave server verifies the execution result of the target database. If the data of the source database and the target database are consistent, the data synchronization is completed, otherwise.

[0113] The slave server obtains the operation message to be processed from its own cache queue, and assigns the operation message to the target thread corresponding to its message identifier. In the embodiment of the present application, the timing characteristics of the operation message are obtained by processing the message identifier of the operation message, so that the thread pool controls multiple threads to concurrently process each operation message according to the timing characteristics of the operation message, thereby ensuring data synchronization efficiency, avoiding the disorder of the order of each operation message, and reducing the generation of dirty data.

[0114] The slave server processes the operation message through the target thread to obtain the operation event. The target thread executes the operation event to obtain the target data and synchronize it to the target database. That is, the computer device performs SQL parsing on the message to obtain the operation event. The slave server performs data business filtering, data processing and conversion according to the operation event to obtain the target data. Finally, SQL playback is executed to synchronize the target data to the target database.

[0115] In the embodiment of the present application, operation messages are obtained from the cache queue, and the operation message records are used to record operation events on the source database. When the operation messages are batch processed, in order to ensure the processing order of each operation message, the target thread for processing each operation message is determined according to the message identifier of the operation message and the thread information of the thread pool corresponding to the target database, the operation message is parsed by the target thread to obtain the operation event, and the operation event corresponding to the operation message is sequentially executed on the target database through the target thread, so that the target database and the source database execute the same operation event. In the embodiment of the present application, when the operation messages in the cache queue are batch processed by the thread pool, the target thread for processing the operation message is determined according to the message identifier of the operation message and the thread information of the thread pool, and the corresponding operation messages are sequentially processed by the target thread, thereby ensuring the data consistency of data synchronization on the basis of achieving fast data synchronization.

[0116] In a distributed application scenario of data synchronization, or a scenario where the target database has a different data structure from the source database, it is still difficult to ensure that the scheduled tasks of multiple target databases and / or multiple source databases are correctly executed (no repeated synchronization, stable operation); therefore, in an embodiment of the present application, the source database sends an operation message to the message middleware, and the message middleware sends the operation message to the slave server corresponding to the target database for data synchronization, specifically, including:

[0117] (1) receiving a first operation message pushed by the message middleware;

[0118] (2) Determine a cache queue according to the partition identifier of the first partition in which the first operation message is stored in the message middleware, and save the first operation message to the cache queue.

[0119] On the basis of the previous embodiment of the present application, an operation message pushed by the message middleware is received from the server, the cache queue of the operation message is determined from the server according to the partition identifier of the first partition where the operation message is stored in the message middleware, and the operation message is saved to the cache queue from the server. In the embodiment of the present application, the operation message is stored in a specific cache queue, and the operation message is obtained from the cache queue from the server for data synchronization, which can ensure the timing of data synchronization in distributed scenarios or special scenarios such as data heterogeneity, making the application scenarios of data synchronization more extensive.

[0120] In the embodiment of the present application, after obtaining the operation message from the server, the slave server determines the cache queue according to the partition identifier of the first partition storing the first operation message in the message middleware and saves the first operation message to the cache queue. In this way, when data is synchronized from the server, if the data structure of the source database is different from the data structure of the target database, the sequential synchronization of the target database is ensured by ensuring that the operation messages of multiple source databases are executed in sequence.

[0121] In an embodiment of the present application, according to the partition identifier of the first partition where the first operation message is stored in the message middleware, determining a cache queue and saving the first operation message to the cache queue further includes:

[0122] Obtain the partition identifier of the storage partition corresponding to the operation message and / or the message identifier of the operation message from the server, and route the operation message to the cache queue according to the partition identifier and / or the message identifier of the operation message. In the embodiment of the present application, the target partition in the message middleware may be at least one, and the operation message is routed from the server to the cache queue. Each message in the message cache queue is processed through multiple threads to effectively improve the consumption throughput of each partition.

[0123] Further, an embodiment of the present application is proposed on the basis of the above embodiment of the present application. In the embodiment of the present application, the target thread parses the first operation message to obtain the operation event, and after the target thread executes the operation event on the second database to complete data synchronization, the second database can also be used as a new source database. Specifically:

[0124] (1) determining a second operation message according to a database log of the second database;

[0125] (2) Pushing the second operation message to the second partition corresponding to the second operation message in the message middleware.

[0126] In the embodiment of the present application, the second database serves as a new source database, and the slave server switches to the master server. In the embodiment of the present application, the master server obtains the database log of the new source database, and the database log includes the following contents: the transaction start and end record the timestamp of the transaction start and commit or rollback; the data modification operation records the detailed messages of the operations such as inserting, updating, deleting and the like on the database data, including the modified data object, the value before and after the modification, etc.; the transaction identifier records the transaction identifier of each operation, which is used to track and restore the execution of the transaction; the log sequence number assigns a unique serial number to each log record to ensure the orderliness and integrity of the log; the content of the database log in the embodiment of the present application does not constitute a limitation on the technical solution of the present application.

[0127] The main server parses the database log to obtain the operation information. In the embodiment of the present application, the main server pushes the second operation message to the second partition corresponding to the second operation message in the message middleware, specifically including:

[0128] (1) performing hash processing on the message identifier of the second operation message to obtain a partition identifier of the second operation message;

[0129] (2) determining, in the message middleware, a second partition corresponding to the partition identifier;

[0130] (3) Push the second operation message to the second partition.

[0131] Specifically, the main server obtains the operation message in the database log, and the message identifier of the operation message, wherein the message identifier is obtained by processing at least one of the source database type, the source database identifier, the source data table identifier and the message primary key; the main server hashes the message identifier of the message to obtain the partition identifier; the main server pushes the operation message to the second partition corresponding to the partition identifier in the message middleware, so as to perform concurrent processing of messages in different partitions.

[0132] Among them, hash processing refers to a processing algorithm for converting a message identifier into a partition identifier. The hash processing algorithm constitutes a specific limitation, that is, in the embodiment of the present application, a mapping relationship between a message identifier and a partition is pre-set, and the message identifier is converted into a partition identifier according to the mapping relationship, wherein the partition identifier refers to an identification message that uniquely identifies a partition, and the partition can be quickly accessed according to the partition identifier.

[0133] Furthermore, in the embodiment of the present application, since there are many messages in the database log, taking into account factors such as the speed of converting the message identifier to the partition identifier and memory usage, a method of converting the message identifier to the partition identifier is provided in the embodiment of the present application: hashing the message identifier of the operation message to obtain the partition identifier; in the embodiment of the present application, the message identifier is directly hashed to obtain the partition identifier, which speeds up the data processing steps and reduces memory usage.

[0134] In the data synchronization method of the embodiment of the present application, data synchronization is achieved by scanning the changes in the source database in real time or periodically, and transmitting the changed data to the target database or other system; the source database monitors and captures any modification operations on the data in the source database so as to apply these changes to the target database or other system.

[0135] In this embodiment, after the master server obtains the operation message in the database log, it synchronizes the data through the message middleware instead of directly performing the data synchronization operation, which can avoid the increase of the pressure on the source database caused by real-time access to the source database.

[0136] It is understandable that the relevant technology discloses sending a message to a message middleware, and the corresponding slave server of the target database subscribes to the message middleware for message consumption. If the writing is successful, it ends directly; if it fails, the message middleware retry mechanism is used for compensation, and the writing may still fail due to the retry, such as database failure, network abnormality, business abnormality, etc.; even if the writing is successful after the retry, because the writing timing is uncertain, if the delay is long, the data written later may be overwritten and restored, resulting in dirty data. The embodiment of the present application is an improvement on the problem of dirty data existing in data synchronization when using message middleware.

[0137] After obtaining the operation message in the embodiment of the present application, the message identifier of the operation message is obtained, and the operation message is sent to the partition corresponding to the message identifier of the operation message. Specifically, it includes:

[0138] (1) performing hash processing on the message identifier of the second operation message to obtain a partition identifier of the second operation message;

[0139] (2) determining, in the message middleware, a second partition corresponding to the partition identifier;

[0140] (3) Pushing the second operation message to the second partition.

[0141] Among them, the message identifier refers to an identification message that uniquely identifies the operation message. The message identifier can be obtained by processing at least one of the source database type, source database identifier, source data table identifier and message primary key. For example, the format of the message identifier is database type_database name_table name_primary key. For example, the message identifier is: MySQL_db1_order_0_1.

[0142] In the embodiment of the present application, the main server divides different operation messages into different partitions in the message middleware, which can realize the simultaneous operation of multiple source databases. That is, if the source database is increased, it is only necessary to add a new partition in the message middleware to handle the newly added source database, thereby realizing multi-source data synchronization operation.

[0143] In other words, the business system and data synchronization architecture corresponding to the source database in the embodiment of the present application are completely decoupled, there will be no code intrusion into the original business system, and it is scalable; the business data processing logic can be flexibly customized, providing standard interfaces, and the business can be implemented independently, and data synchronization and business data updates do not interfere with each other.

[0144] In the embodiment of the present application, the main server pushes the operation message to the partition corresponding to the message identifier of the operation message in the message middleware, so that different partitions are parallel, and the thread pool is used for concurrent consumption within the partition, which effectively improves the consumption throughput of a single partition. Specifically:

[0145] Obtain the operation message in the target partition in the message middleware from the server, where the number of target partitions is not limited and there can be at least one target partition. Route the operation message from the server to the cache queue, and process each message in the message cache queue through multiple threads to effectively improve the consumption throughput of each partition.

[0146] In the embodiment of the present application, the slave server obtains operation messages from the target partition of the message middleware, obtains messages according to the partition granularity, and the partitions are parallelized, which effectively improves the consumption throughput of a single partition.

[0147] In the embodiment of the present application, operation messages are obtained by parsing the database log of the source database, and data synchronization is performed according to the operation messages, thereby reducing the pressure on the source database; the message is pushed to the corresponding partition of the message middleware according to the message identifier, and the target partition to be synchronized in the message middleware is determined. By consuming multiple partitions in the message middleware in parallel, each partition corresponds to multiple threads, and messages in the partition are consumed concurrently, the consumption throughput is improved. At the same time, by allocating messages to the target threads corresponding to their message identifiers, the message processing order is ensured, and the data consistency of data synchronization is guaranteed.

[0148] Furthermore, in some embodiments of the present application, due to different data sources and different data update situations, using the same data synchronization method may cause resource waste. In the embodiments of the present application, incremental data synchronization is performed from the server. Specifically,

[0149] (1) sending incremental synchronization events to the first database;

[0150] (2) determining a first operation message to be synchronized according to a log position in the incremental synchronization event;

[0151] (3) Sending a request for obtaining the first operation message to the message middleware.

[0152] That is, in the embodiment of the present application, the log position corresponding to the incremental synchronization event is extracted from the server, and the first operation message to be synchronized is determined from the server based on the log position in the incremental synchronization event; the server sends a request to obtain the first operation message to the message middleware, so that the message middleware feeds back the corresponding operation information to achieve accurate synchronization of data.

[0153] The slave server receives the incremental synchronization event, determines the log position corresponding to the incremental synchronization event, generates a retrieval request containing the log position and sends it to the source database, so that the master server corresponding to the source database pushes the database log according to the log position. The master server receives the database log and parses the database log through the log parser to obtain the operation message and the updated log position, and sends it to the message middleware, so that the slave server obtains the corresponding operation message from the message middleware.

[0154] For ease of understanding, refer to Figure 7 , Figure 7 Schematic diagram of log parsing scenario in the data synchronization method provided in the embodiment of the present application. In the embodiment of the present application, the computer device creates a connection Connection with the source database MySQL, the source database MySQL pushes the database log Binary Log through the connection Connection, obtains the message in the database log through the event parser Event Parser and the log parser Binlog Parser, and establishes an event connection EventSink according to the message to send the message to the message middleware, specifically, including:

[0155] 1. Get the last parsed log position Log Position;

[0156] 2. Send the log request BINLOG_DUMP with Log Position to the source database MySQL;

[0157] 3. The source database MySQL starts pushing Binary Log;

[0158] 4. Parse Binary Log through Event Parser and Binlog Parser to obtain operation messages and updated log locations;

[0159] 5. Establish an event connection Event Sink and push it to the message middleware;

[0160] 6. Record the parsed log location and update the log location to return to step 1 to apply for the database log again.

[0161] In the embodiment of the present application, the database log is obtained and further parsed to obtain the log position of the message and update to ensure the integrity of data synchronization.

[0162] In an embodiment of the present application, step 202 determines the target thread according to the message identifier of the first operation message and the thread information of the thread pool corresponding to the second database, and also includes:

[0163] That is, in the embodiment of the present application, the message identifier is converted into a hash value by a hash function (such as a hash function). Then, the hash value is mapped to a fixed range using a modulo operation. The fixed range in the embodiment of the present application is determined according to the number of threads in the thread pool. For example, if the thread information of the thread pool is 10, the range size of the hash modulo is 10. If we use a hash function to calculate the hash value of a message identifier as 27, the hash value is mapped to a position between 0 and 9 by performing a modulo operation on the hash value. For 27, modulo 10 gives a remainder of 7, so the message identifier will be mapped to position 7 of the hash table.

[0164] The advantage of hash modulus in the embodiment of the present application is that it is simple and fast. In the embodiment of the present application, the method of obtaining sequential messages through hash modulus speeds up the data processing steps and reduces memory usage. It should be noted that in some cases, hash conflicts may occur, that is, different message identifiers obtain the same mapping through hash modulus, and a prompt is required at this time.

[0165] For ease of understanding, the embodiment of the present application provides a scenario in which messages of a target partition in a message middleware are pulled and consumed from a server until data synchronization is completed; Figure 8 , Figure 8 A schematic diagram of a scenario of target partition message subscription in a message middleware in a data synchronization method provided in an embodiment of the present application.

[0166] Specifically, Figure 8In the example, the message is pulled from the server from the message middleware (msg is an abbreviation, usually used to represent "message", which means message. In the field of computers and communications, "msg" is usually used to represent data packets in transmission or messages transmitted in communication). In the embodiment of the present application, messages are pulled from the target partition Partition0, the target partition Partition1 and the target partition Partition2, and then the messages are routed to the cache queue Blocking Queue corresponding to the partition identifier according to the partition identifier P0, P1 and P2 of the target partition, and then the messages are assigned to the target thread according to the order of the messages through the thread pool corresponding to the target partition. For example,

[0167] The thread pool corresponding to the partition identifier P0 includes thread 1, thread 2, and thread 3. The thread pool corresponding to P0 controls message 1 to be processed by thread 1, message 2 to be processed by thread 2, and message 3 to be processed by thread 3.

[0168] The thread pool corresponding to the partition identifier P1 includes thread 1, thread 2, and thread 3. The thread pool corresponding to P1 controls message 4 to be processed by thread 1, message 5 to be processed by thread 2, and message 6 to be processed by thread 3.

[0169] The thread pool corresponding to the partition identifier P2 includes thread 1, thread 2 and thread 3. The thread pool corresponding to P2 controls message 7 to be processed by thread 1, message 8 to be processed by thread 2, and message 9 to be processed by thread 3.

[0170] In the embodiment of the present application, sequential consumption is achieved while taking high performance into consideration. Thanks to the refined message pulling and displacement submission granularity, partitions are parallelized, and concurrent consumption is performed within a partition through a thread pool, which effectively improves the consumption throughput of a single partition.

[0171] Further, refer to Figure 8 In an embodiment of the present application, the data synchronization method further includes:

[0172] (1) If the number of operation messages in the cache queue reaches a preset threshold, stop obtaining operation messages from the target partition;

[0173] (2) When the message consumption in the cache queue is completed, the operation message is obtained from the target partition again.

[0174] The queue length of the cache queue is obtained from the server in real time, and message consumption is controlled according to the queue length of the cache queue. For example, a single partition pulls 100 messages at a time, and the local message cache queue is set to 1000. It can be filled after 10 initial pulls. Considering that it is necessary to limit the pull at this time, for example, call org.apache.kafka.clients.consumer.KafkaConsumer#pause to pause consumption of the partition; until the consumption of messages in the cache queue is completed, resume consuming messages from the target partition. In the embodiment of the present application, a reasonable queue length is set for the cache queue to avoid the cache queue being too large, which may cause memory overflow.

[0175] For ease of understanding, the software structure in the data synchronization method in an embodiment of the present application is described. Fig. 9 , Fig. 9 It is a scenario diagram of the software architecture in the data synchronization method provided in the embodiment of the present application. The data synchronization software in the embodiment of the present application can support database isomorphism and heterogeneity, that is, the data table structure of the source database in the embodiment of the present application is the same as the data table structure of the target database; or, the data table structure of the source database is different from the data table structure of the target database.

[0176] Specifically, in an embodiment of the present application, a computer device obtains configuration files of a source database and a target database, wherein the database configuration files include but are not limited to: database connection information, database character set and sorting rules, character set and sorting rules used by a specified database, database storage path and space setting information, and cache setting information.

[0177] The computer device determines to report the monitoring data according to the configuration information, collects logs to obtain log files, and performs data synchronization operations through the table structure synchronization module. In the embodiment of the present application, the table structure synchronization module supports the full synchronization module to pull data for full synchronization, and also supports the Canal service for incremental synchronization. Among them, the incremental data synchronization requires determining whether it is DDL (Data Definition Language) data or DML (Data Manipulation Language) data, and then performing data synchronization.

[0178] Among them, DDL data is used to define and manage the structure and metadata of the database. DDL is used to create, modify and delete database objects, such as tables, views, indexes, constraints, etc. DML data is used to operate on data in the database, including querying (retrieval), inserting, updating and deleting data.

[0179] In the embodiment of the present application, data synchronization utilizes the master-slave replication Binlog synchronization principle, which not only supports DML (table record update SQL), but also supports DDL (same database protocol and same table structure), solving the pain point that manual intervention in operation and maintenance is required when table metadata changes.

[0180] For ease of understanding, refer to Fig.10 , Fig.10 is a schematic diagram of a scenario of data synchronization application in a data synchronization method provided in an embodiment of the present application. The technical solution of the embodiment of the present application is to combine the above embodiments for a specific scenario, wherein:

[0181] (1) The tables involved in this application example, the source database is MySQL, the business system now has an order table: order, which is synchronized to the target system, the target database is TiDB, and has system asset identification and order amount ratio settlement logic;

[0182] (2) Enable Binlog for the source order table;

[0183] (3) Configure the Canal source data source MySQL, the source database name is db1, the source table name is sub-table order_{0,64}; the target data source TiDB, the target database name is db2, the target table name is order (summary table), and the starting log position is 0;

[0184] (4) Binlog subscription parsing, where the connection is used to obtain the location of the last successful parsing (if it is the first time to start, the initial specified location or the Binlog location of the current database is obtained); the connection is established and the Binlog_Dump command is sent, where

[0185] Binlog_Dump is a command in MySQL that is used to read and obtain data change events from the binary log (Binlog) on ​​the MySQL master server. This command is usually used by the MySQL slave server (MySQL Slave) to replicate data with the master server (MySQL Master).

[0186] When a slave server is configured for master-slave replication with a master server, the slave server sends a request to the master server using the Binlog_Dump command to obtain binary log events for update operations. The master server sends these events to the slave server, thereby replicating the data change operations on the master server.

[0187] The Binlog_Dump command requires the following information to be specified: Binlog File Name: The name of the binary log file to be read. Binlog Position: The position in the binary log file from which to start reading events. Slave Server ID: A unique identifier for the slave server, used to communicate and identify with the master server.

[0188] The Binlog_Dump command is sent to the master server, which will start sending binary log events to the slave server. The slave server can read and process these events using the corresponding mechanism to replicate and synchronize data. It is worth noting that the Binlog_Dump command requires appropriate permissions and configuration to work properly. The slave server must have the appropriate replication permissions, and the master server must be configured to allow the slave server to connect and perform replication operations.

[0189] The source database MySQL starts pushing Binary Logs.

[0190] The received Binary Log is parsed through the EventParser protocol and supplemented with some specific information (such as field name, field type, primary key information, and unsigned type processing); it is passed to the EventSink module for data storage, which is a blocking operation until the storage is successful. After the storage is successful, the Binary Log position is recorded regularly.

[0191] (5) The data update Binlog log of order_0 with order number 1 is pushed to Kafka. The message identification key format is database type_database name_table name_primary key, that is, MySQL_db1_order_0_1;

[0192] Kafka production and consumption design: the number of partitions is 30, the number of synchronous system consumption machines is 10, and each machine consumes 3 partitions;

[0193] (6) Consumption realization

[0194] Asynchronous pre-fetching: Figure 8 , the message identification key is MySQL_db1_order_0_1, and it is routed to partition P1 after hash modulo. The synchronous system asynchronously pulls the operation message to the local cache queue (BlockingQueue) for consumption. A single partition pulls 100 messages at a time, and the local cache queue is set to 1000. It can be full after 10 initial pulls. Considering that it is necessary to limit the pull at this time (the local cache queue is too large, which may cause memory overflow), org.apache.kafka.clients.consumer.KafkaConsumer#pause is called to suspend consumption of the partition;

[0195] Consumer thread design: Each partition initializes a thread pool, in which the number of core threads is set to 3 (configurable and tunable), and then the message identifier keyMySQL_db1_order_0_1 is hashed and modulo (modulus is 3) and assigned to the thread with a specific thread name (p0_order_thread_1) for processing.

[0196] Consumption logic: Fig. 9 As shown:

[0197] (a) Parse Binlog to obtain the original SQL: update order_0 set amout = 100.00 where orderNo = 1;

[0198] (b) By querying the order number, the system asset identification cache is changed to true, indicating that the asset is owned. The order amount ratio is further queried to be 50%, and the SQL obtained is: update order set amout = 50.00 where orderNo = 1;

[0199] (c) Execute the SQL in the previous step (b) to update the TiDB order summary table, and the consumption is completed;

[0200] In the embodiment of the present application, manual submission of offset is also managed according to the partition granularity (so that the interaction between each partition is not affected, and there is no need to wait for the main thread to finish all consumption). After each thread in the partition thread pool finishes consumption, it will determine whether there are any messages in the BlockingQueue (using CountDownLatch, the count is the length of the BlockingQueue). If consumption is completed, the partition offset is committed and the partition consumption is resumed, that is, org.apache.kafka.clients.consumer.Kafka Consumer#resume is called to start the next round of consumption of the partition until data synchronization is completed.

[0201] In the embodiment of the present application, messages are obtained by parsing database logs, and data synchronization is performed based on the messages, thereby reducing the pressure on the source database; messages are pushed to the corresponding partition of the message middleware according to the message identifier, and the target partition to be synchronized in the message middleware is determined. By consuming multiple partitions in the message middleware in parallel, each partition corresponds to multiple threads, and messages in the partition are consumed concurrently, the consumption throughput is improved. At the same time, by allocating messages to the target threads corresponding to their message identifiers, the message processing order is guaranteed, and the data consistency of data synchronization is guaranteed.

[0202] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0203] Based on the same inventive concept, the present application embodiment also provides a data synchronization system for implementing the above-mentioned data synchronization method. Fig.11 , Fig.11 : is a schematic diagram of the architecture of the data synchronization system provided in the embodiment of the present application; the data synchronization system in the embodiment of the present application includes a master server and a slave server, the master server is a computer device corresponding to the source database, and the slave server is a computer device corresponding to the target database. The data synchronization system in the embodiment of the present application implements a data synchronization method, wherein:

[0204] The main server is used to determine a first operation message according to a database log of the first database and send the first operation message to the message middleware, where the first operation message is used to record an operation event on the first database;

[0205] A message middleware, configured to receive the first operation message and save the message to a first partition corresponding to the first database;

[0206] A message middleware, configured to receive a message request from a slave server, obtain the first operation message from the first partition, and send the message to a cache queue of the slave server;

[0207] A slave server, configured to obtain the first operation message from a cache queue;

[0208] The slave server is configured to determine a target thread according to a message identifier of the first operation message and thread information of a thread pool corresponding to the second database;

[0209] The slave server is configured to parse the first operation message through the target thread to obtain the operation event, and execute the operation event on the second database through the target thread.

[0210] refer to Fig.11In the embodiment of the present application, the main server stores the source database, and the main server communicates with a binlog parsing architecture, which includes a parsing module and a message middleware (or a kafka cluster);

[0211] The parsing module includes: event parser eventParser, event storage eventSink, Canal server, event storage system eventStore, metadata manager metaManager and distributed coordination service ZooKeeper, that is,

[0212] The event parser eventParser is used to parse the received event data and extract useful messages.

[0213] Event storage eventSink is used to receive and store event data and is usually used together with an event parser.

[0214] Canal server is used to capture and transmit database change events and is often used for data synchronization and streaming with the database.

[0215] The event storage system eventStore is used to persist event data. It is usually a storage system that supports high throughput and reliability, such as a distributed database.

[0216] The metadata manager metaManager is used to manage and maintain metadata (such as partitions, event status, etc.) in the event processing system.

[0217] The distributed coordination service ZooKeeper is used for configuration management, naming services, distributed locks, etc. in distributed systems.

[0218] The message middleware includes: partitions and message brokers in the message middleware, where:

[0219] Partition partition0, partition partition1, partition partition2, and partition partition3: Partitions are used to disperse data for storage and processing. Different partitions can be operated in parallel.

[0220] broker1, broker2: message brokers in the message middleware, used to transmit messages between different systems.

[0221] refer to Fig.11 In the embodiment of the present application, the slave server saves the target database, and the slave server is connected to the data synchronization device, and the data synchronization device includes: a consumption model and an SQL processing engine, wherein:

[0222] The consumption model includes: routing algorithm module, sequential queue (also called cache queue), consumption controller and multi-threaded orchestrator for message consumption.

[0223] The SQL processing engine includes: an SQL parsing module, a data filtering module, a data processing module and an SQL reply module. The SQL processing engine is used to perform data synchronization operations.

[0224] In the embodiment of the present application, data synchronization is performed through a data synchronization system. The main server obtains messages by parsing database logs, and data synchronization is performed according to the messages, thereby reducing the pressure on the source database. The slave server pushes the message to the corresponding partition of the message middleware according to the message identifier, determines the target partition to be synchronized in the message middleware, and consumes multiple partitions in the message middleware in parallel. Each partition corresponds to multiple threads, and messages in the partition are consumed concurrently to improve consumption throughput. At the same time, by allocating messages to target threads corresponding to their message identifiers, the message processing order is guaranteed, and the data consistency of data synchronization is guaranteed.

[0225] At the same time, in the embodiment of the present application, the business system corresponding to the source database and the code design of data synchronization are decoupled through the message middleware, and the distributed design solution can save messages of multiple database logs at the same time, realizing horizontal expansion.

[0226] Based on the same inventive concept, the embodiment of the present application also provides a data synchronization device for implementing the data synchronization method involved above, and a data synchronization device for implementing the data synchronization method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the embodiments of one or more data synchronization devices and data synchronization devices provided below can refer to the above limitations on the data synchronization method and data synchronization method, and the specific limitations will not be repeated here.

[0227] This embodiment also provides a data synchronization device, which can be integrated into a terminal device. Fig.12 As shown, Fig.12 : is a schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application, and the data synchronization device may include:

[0228] The acquisition module 301 is used to acquire a first operation message from a cache queue, where the first operation message is used to record an operation event on a first database;

[0229] A determination module 302 is used to determine a target thread according to a message identifier of the first operation message and thread information of a thread pool corresponding to the second database;

[0230] The synchronization module 303 is used to parse the first operation message through the target thread to obtain the operation event, and execute the operation event on the second database through the target thread.

[0231] The above-mentioned data synchronization device adopts the scheme of the embodiment of the present application to obtain operation messages from the cache queue. The operation messages record the operation events on the first database. When the operation messages are batch processed, in order to ensure the processing order of each operation message, the target thread for processing each operation message is determined according to the message identifier of the operation message and the thread information of the thread pool corresponding to the second database, the operation message is parsed by the target thread to obtain the operation event, and the operation event corresponding to the operation message is sequentially executed on the second database through the target thread, so that the second database and the first database execute the same operation event, and data synchronization is performed between the second database and the first database. In the embodiment of the present application, when the operation messages in the cache queue are batch processed by the thread pool, the target thread for processing the first operation message is determined according to the message identifier of the operation message and the thread information of the thread pool, and the corresponding operation message is sequentially processed by the target thread, thereby ensuring the data consistency of data synchronization on the basis of achieving fast data synchronization.

[0232] In some embodiments, the determination module 302 in the data synchronization device includes:

[0233] A hash processing unit, configured to perform hash processing on the message identifier of the first operation message to obtain a hash value of the first operation message;

[0234] The first determining unit is used to perform modulo processing on the hash value and the number of threads in the thread pool to determine the target thread.

[0235] In some embodiments, the determination module 302 in the data synchronization device includes:

[0236] a time determination module, configured to determine a time priority of the first operation message according to a message identifier of the first operation message;

[0237] The second determining unit is used to determine the target thread according to the time priority of the first operation message and the thread status of each thread in the thread pool corresponding to the second database.

[0238] In some embodiments, the data synchronization device further includes:

[0239] A message receiving module, used for receiving a first operation message pushed by the message middleware;

[0240] The message cache module is used to determine a cache queue according to a partition identifier of a first partition in which the first operation message is stored in the message middleware, and save the first operation message to the cache queue.

[0241] In some embodiments, the data synchronization device further includes:

[0242] A sending module, used for sending the incremental synchronization event to the first database;

[0243] A determination module, configured to determine a first operation message to be synchronized according to a log position in the incremental synchronization event;

[0244] A push module is used to send a request for obtaining the first operation message to the message middleware.

[0245] In some embodiments, the data synchronization device further includes:

[0246] a message determination module, configured to determine a second operation message according to a database log of the second database;

[0247] A message pushing module is used to push the second operation message to the second partition corresponding to the second operation message in the message middleware.

[0248] In some embodiments, the message push module in the data synchronization device further includes:

[0249] A hash processing unit, used to perform hash processing on the message identifier of the second operation message to obtain a partition identifier of the second operation message;

[0250] A partition determination unit, configured to determine, in the message middleware, a second partition corresponding to the partition identifier;

[0251] A message pushing unit is used to push the second operation message to the second partition.

[0252] In some embodiments, the synchronization module 303 in the data synchronization device includes:

[0253] An event determination unit, configured to parse the first operation message through the target thread to obtain the operation event; the operation event includes: at least one of an insert event, an update event, and a delete event;

[0254] An event execution unit, configured to execute the operation event on the second database through the target thread;

[0255] The synchronization verification unit is used to verify the execution result of the second database if the operation event is executed successfully, so as to obtain the data synchronization result.

[0256] Based on the same inventive concept, the embodiment of the present application further provides a computer device, which may be a server or a terminal device, and includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above data synchronization method when executing the computer program. Thereby, various functions are realized, for example:

[0257] Obtaining a first operation message from the cache queue, where the first operation message is used to record an operation event on the first database;

[0258] Determine a target thread according to the message identifier of the first operation message and the thread information of the thread pool corresponding to the second database;

[0259] The first operation message is parsed through the target thread to obtain the operation event, and the operation event is executed on the second database through the target thread.

[0260] The computer device in the embodiment of the present application obtains messages by parsing database logs, synchronizes data according to the messages, and reduces the pressure on the source database; pushes the messages to the corresponding partitions of the message middleware according to the message identifier, determines the target partition to be synchronized in the message middleware, and consumes multiple partitions in the message middleware in parallel, each partition corresponds to multiple threads, and the messages in the partitions are consumed concurrently, thereby improving the consumption throughput. At the same time, by allocating messages to the target threads corresponding to their message identifiers, the message processing order is guaranteed, and the data consistency of data synchronization is guaranteed.

[0261] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0262] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows: Fig.13 shown. Fig.13It is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application, and the computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Wherein, the processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data synchronization method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0263] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0264] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0265] Since the computer program in the computer-readable storage medium can execute any one of the data synchronization methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the data synchronization methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0266] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.

[0267] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0268] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0269] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0270] In the above-mentioned data synchronization device, computer-readable storage medium, computer equipment, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described data synchronization device, computer-readable storage medium, computer program product, computer equipment, and its corresponding units can refer to the description of the data synchronization method in the above embodiment, and will not be repeated here.

[0271] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0272] The above is a detailed introduction to a data synchronization method, system, device, computer equipment, computer-readable storage medium and computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data synchronization method, characterized in that: The method comprises: Obtaining a first operation message from the cache queue, where the first operation message is used to record an operation event on the first database; Determine a target thread according to the message identifier of the first operation message and the thread information of the thread pool corresponding to the second database; The first operation message is parsed through the target thread to obtain the operation event, and the operation event is executed on the second database through the target thread.

2. The data synchronization method according to claim 1, characterized in that: The determining the target thread according to the message identifier of the first operation message and the thread information of the thread pool corresponding to the second database includes: Performing hash processing on the message identifier of the first operation message to obtain a hash value of the first operation message; The hash value and the number of threads in the thread pool are modulo processed to determine the target thread.

3. The data synchronization method according to claim 1, characterized in that: The determining the target thread according to the message identifier of the first operation message and the thread information of the thread pool corresponding to the second database includes: Determining a time priority of the first operation message according to a message identifier of the first operation message; The target thread is determined according to the time priority of the first operation message and the thread status of each thread in the thread pool corresponding to the second database.

4. The data synchronization method according to claim 1, characterized in that: Before obtaining the first operation message from the cache queue, the method further includes: Receive a first operation message pushed by the message middleware; A cache queue is determined according to a partition identifier of a first partition in which the first operation message is stored in the message middleware, and the first operation message is saved in the cache queue.

5. The data synchronization method according to claim 4, characterized in that: Before the receiving message middleware pushes the first operation message, the method further includes: Sending incremental synchronization events to the first database; Determining a first operation message according to a log position in the incremental synchronization event; Sending a request for obtaining the first operation message to the message middleware.

6. The data synchronization method according to claim 1, characterized in that: After parsing the first operation message through the target thread to obtain the operation event, and executing the operation event on the second database through the target thread, the method further includes: determining a second operation message according to the database log of the second database; The second operation message is pushed to a second partition corresponding to the second operation message in the message middleware.

7. The data synchronization method according to claim 6, characterized in that: The pushing the second operation message to the second partition corresponding to the second operation message in the message middleware includes: Performing hash processing on the message identifier of the second operation message to obtain a partition identifier of the second operation message; Determine, in the message middleware, a second partition corresponding to the partition identifier; The second operation message is pushed to the second partition.

8. The data synchronization method according to any one of claims 1 to 7, characterized in that: The step of parsing the first operation message through the target thread to obtain the operation event, and executing the operation event on the second database through the target thread includes: Parsing the first operation message through the target thread to obtain the operation event; the operation event includes: at least one of an insert event, an update event and a delete event; Executing the operation event on the second database through the target thread; If the operation event is executed successfully, the execution result of the second database is verified to obtain a data synchronization result.

9. A data synchronization system, characterized in that: The data synchronization system includes a master server, a message middleware and a slave server, wherein: The main server is used to determine a first operation message according to a database log of the first database and send the first operation message to the message middleware, where the first operation message is used to record an operation event on the first database; A message middleware, configured to receive the first operation message and save the message to a first partition corresponding to the first database; A message middleware, configured to receive a message request from a slave server, obtain the first operation message from the first partition, and send the message to a cache queue of the slave server; A slave server, configured to obtain the first operation message from a cache queue; The slave server is configured to determine a target thread according to a message identifier of the first operation message and thread information of a thread pool corresponding to the second database; The slave server is configured to parse the first operation message through the target thread to obtain the operation event, and execute the operation event on the second database through the target thread.

10. A data synchronization device, characterized in that: The device comprises: An acquisition module, used to acquire a first operation message from a cache queue, where the first operation message is used to record an operation event on a first database; A determination module, configured to determine a target thread according to a message identifier of the first operation message and thread information of a thread pool corresponding to the second database; A synchronization module is used to parse the first operation message through the target thread to obtain the operation event, and execute the operation event on the second database through the target thread.

11. A computer device, characterized in that: The computer device is at least one of a master server and a slave server, and includes a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the data synchronization method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the data synchronization method according to any one of claims 1 to 8.

13. A computer software, characterized in that The computer software comprises a computer program, and the computer program is used by a processor to execute the steps of the data synchronization method according to any one of claims 1 to 8.

Images