Data synchronization method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data synchronization method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a current transaction of a source database, wherein the current transaction carries a current event stream format file; wherein the current event stream format file is determined based on conversion of a first logic log file of a source database; each transaction carries a corresponding event stream format file; converting the current event stream format file into a second logic log file of the target database; and in response to a data synchronization request of the target database, pushing the second logic log file to the target database based on a physical replication protocol of the target database. By adopting the method, high data synchronization efficiency can be realized, and resource occupation during data synchronization is low.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data synchronization method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Businesses have many choices for using databases, such as traditional databases such as MYSQL, PG (PostgreSQL), and new databases such as Mongodb and Tidb. There are also various distributed database products or solutions. Business needs vary, and sometimes you need to use multiple databases at the same time to meet different needs, and sometimes you need to migrate the database or do disaster recovery backup.

[0003] The common data synchronization solution is to use message middleware such as Kafka and RocketMQ as core components. Additional tools are needed to convert data between the source data and the target database. The data synchronization can only be completed with the cooperation of multiple components. The traditional method introduces middleware, develops corresponding synchronization tools, and deploys complex middleware services and synchronization tools, which leads to the problem of high computer resource usage. Summary of the invention

[0004] Based on this, it is necessary to provide a data synchronization method, apparatus, computer device, computer-readable storage medium and computer program product that can reduce computer resource usage in response to the above technical problems.

[0005] In a first aspect, the present application provides a data synchronization method, the method comprising:

[0006] Acquire a current transaction of a source database, wherein the current transaction carries a current event stream format file; wherein the current event stream format file is determined based on the conversion of a first logical log file of the source database; each transaction carries a corresponding event stream format file;

[0007] Convert the current event stream format file into a second logical log file of the target database;

[0008] In response to a data synchronization request of the target database, the second logical log file is pushed to the target database based on a physical replication protocol of the target database.

[0009] In one embodiment, before obtaining the current event stream format file of the source database, the method further includes:

[0010] Listen to the event port and obtain the event stream format file of the source database;

[0011] Writing the event stream format file into an event stream queue;

[0012] The step of obtaining the current event stream format file of the source database includes:

[0013] The current event stream format file of the database is obtained from the event stream queue.

[0014] In one embodiment, obtaining the current event stream format file of the database from the event stream queue includes:

[0015] Obtaining a corresponding event stream format file from the event stream queue through each of at least two threads;

[0016] The converting the current event stream format file into a second logical log file of the target database comprises:

[0017] Each thread of the at least two threads converts the event stream format file obtained by itself into a corresponding second logical log file.

[0018] In one embodiment, the method further comprises:

[0019] Outputting each of the second logical log files to a log buffer in the order of the event stream queue;

[0020] The step of pushing the second logical log file to the target database based on the physical replication protocol of the target database in response to the data synchronization request of the target database includes:

[0021] Listening to the port of the target database, and receiving a data synchronization request from the target database;

[0022] The second logical log file is obtained from the log cache according to the database synchronization request, and the second logical log file is pushed to the target database based on the target database physical replication protocol.

[0023] In one embodiment, the data synchronization request carries location information, and the location information is used to identify location information of a target byte in the second logical log file; obtaining the second logical log file from the log cache according to the database synchronization request, and pushing the second logical log file to the target database based on the target database physical replication protocol, includes:

[0024] Obtaining the to-be-synchronized bytes in the second logical log file from the log cache according to the position information; wherein the to-be-synchronized bytes represent the byte information after the position where the target byte is located;

[0025] The bytes to be synchronized are pushed to the target database based on the target database physical replication protocol.

[0026] In one embodiment, obtaining the current transaction of the source database includes:

[0027] Get the event stream format file carried by the jth transaction, where the event stream format file carried by the jth transaction represents the last successfully converted event stream format file in the previous conversion task of the conversion task where the current event stream format file is located; j is a positive integer greater than or equal to 1;

[0028] The (j+1)th event stream format file is used as the current transaction of the source database.

[0029] In a second aspect, the present application further provides a data synchronization device, the device comprising:

[0030] An acquisition module is used to acquire a current transaction of a source database, wherein the current transaction carries a current event stream format file; wherein the current event stream format file is determined based on the conversion of a first logical log file of the source database; and each transaction carries a corresponding event stream format file;

[0031] A conversion module, used for converting the current event stream format file into a second logical log file of a target database;

[0032] The push module is used to respond to the data synchronization request of the target database and push the second logical log file to the target database based on the physical replication protocol of the target database.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0035] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0036] The above-mentioned data synchronization method, device, computer equipment, computer-readable storage medium and computer program product obtain the current transaction of the source database, and the current transaction carries the current event stream format file; wherein the current event stream format file is determined based on the conversion of the first logical log file of the source database; on the one hand, the current event stream format file of the source database is converted into the second logical log file of the target database, and only the current event stream format file needs to be converted, which reduces the complexity of the deployment and operation and maintenance of synchronization tools and middleware, thereby reducing the occupation of computer resources; on the other hand, in response to the data synchronization request of the target database, the second logical log file is pushed to the target database based on the physical replication protocol of the target database, and the second logical log file can be directly copied through the physical replication protocol, which is faster and more efficient than the way of executing a large number of SQL statements by logical replication. Thus, high data synchronization efficiency and low resource occupation during data synchronization are achieved, that is, high efficiency and low consumption are achieved in the process of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] FIG1 is an application environment diagram of a data synchronization method in one embodiment;

[0039] FIG2 is an application environment diagram of a data synchronization method according to another embodiment;

[0040] FIG3 is a schematic diagram of a flow chart of a data synchronization method in one embodiment;

[0041] FIG4 is a schematic diagram of a process of storing an event stream queue and obtaining a current event stream format file from the event stream queue in one embodiment;

[0042] FIG5 is a schematic diagram of a process of converting event stream format files by multiple threads in one embodiment;

[0043] FIG6 is a schematic diagram of a process of reading and pushing a second logical log file in one embodiment;

[0044] 7 is a schematic diagram of a process of obtaining a second logical log file according to site information and pushing the second logical log file in one embodiment;

[0045] FIG8 is a schematic diagram of a process of converting a (j+1)th event stream format file from a jth event stream format file as a starting point in one embodiment;

[0046] FIG9 is a schematic diagram of module division of a conversion server in one embodiment;

[0047] FIG10 is a structural block diagram of a data synchronization device in one embodiment;

[0048] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The data synchronization method provided by the embodiment of the present application can be applied to the application environment shown in FIG. 1. The source database 106 communicates with the server 102 via a network. The source database 106 stores the data to be synchronized. The source database 106 can be integrated on the server 102, or placed on a cloud or other network server. The source database 106 can be a source database or multiple source databases. The target database 104 communicates with the server 102 via a network. The target database 104 stores and synchronizes the completed data. The target database 104 can be integrated on the server 102, or placed on a cloud or other network server. The server 102 obtains the current event stream format file of the source database 106; wherein the current event stream format file is determined based on the conversion of the first logical log file of the source database; converts the current event stream format file into the second logical log file of the target database; and in response to the data synchronization request of the target database, pushes the second logical log file to the target database 104 based on the physical replication protocol of the target database. The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server 102 may also be a conversion server.

[0051] The data synchronization method provided in the embodiment of the present application can be applied in the application environment shown in FIG. 2. The source database 106 is a source database, and the source database 106 communicates with the server 102 via a network. The source database 106 stores the data to be synchronized. The source database 106 can be integrated on the server 102, or placed on a cloud or other network server. The target database 104 communicates with the server 102 via a network. The target database 104 stores and synchronizes the completed data.

[0052] In an exemplary embodiment, as shown in FIG3 , a data synchronization method is provided, which is described by taking the method applied to the server 102 in FIG1 or FIG2 as an example, and includes the following steps S302 to S306 . Among them:

[0053] Step S302, obtaining the current transaction of the source database, where the current transaction carries a current event stream format file.

[0054] Each transaction carries a corresponding event stream format file. A transaction is also called an event. The event stream format (event format) is a universal format, and the data carried by each transaction is called an event stream format file. The event stream format file is determined based on the conversion of the first logical log file of the source database.

[0055] The server may be a convertserver.

[0056] Optionally, before obtaining the current transaction of the source database (source DB), the conversion server obtains the first logical log files of one or more source databases; converts the first logical log files into event stream format files according to the order in which the first logical log files are obtained, and encapsulates each event stream format file as an event or transaction, and transmits it to the conversion server through the event stream event stream. The conversion server selects one or more of the event stream format files as the current event stream format file.

[0057] Optionally, the conversion server uses each or each batch of event stream format files in one or more source databases as the current event stream format file in sequence.

[0058] Step S304: convert the current event stream format file into a second logical log file of the target database.

[0059] The target database can be a relational database or a non-relational database. Relational databases include MYSQL, Oracle Database, SQLite, etc. Non-relational databases include Redis, MongoDB, etc.

[0060] Optionally, the conversion server converts the current event stream format file carried by the current transaction into a logical log file acceptable to the target database, which may be referred to as a second logical log file. If the current event stream format file carried by the current transaction is one event stream format file, the conversion will be performed directly on the current event stream format file; if the current event stream format file carried by the current transaction is multiple event stream format files, the conversion task may be performed serially or in parallel on multiple event stream format files.

[0061] Optionally, if the target database takes MYSQL as an example, the second logical log file is illustrated by taking a logical log file in binlog format as an example, the source database converts the first logical log file into a common event format as a current event stream format file, encapsulates the current event stream format file as a current transaction or event, and transmits the current event stream format file to the conversion server through the event stream eventstream. The conversion server obtains the current event stream format file in event format, and converts the current event stream format file into the second logical log file of the target database, such as a logical log file in binlog format.

[0062] Optionally, after the conversion server converts the current event stream format file into the second logical log file of the target database in a parallel or serial manner, the second logical log file is stored in the log cache in the server to respond to the data synchronization request of the target database. The fields and field definitions carried by the event stream format file are shown in Table 1. Among them, the event stream format file supports DML (Data Manipulation Language) and DDL (Data Definition Language), which are distinguished by the field opType. The event stream format file is transmitted after being serialized in flatbuffer format; among them, FlatBuffers is an efficient cross-platform serialization library that allows fast access to structured data without parsing. FlatBuffers describes the data structure by defining a Schema file and uses this Schema file to generate serialization code. The serialized data is stored in a ByteBuffer, which contains the metadata and actual data values ​​of the data. FlatBuffers uses strict alignment rules and byte order to ensure cross-platform compatibility.

[0063] Table 1 Fields carried by event stream format files and their definitions

[0064]

[0065] Step S306: In response to the data synchronization request of the target database, the second logical log file is pushed to the target database based on the physical replication protocol of the target database.

[0066] The physical replication protocol is a protocol that sends the physical files (such as data files, log files, etc.) of the source database or their changes (such as changes in the binary log) to the target database in a byte-to-byte manner.

[0067] Optionally, the target database (or other tool or platform that needs to synchronize data) initiates a synchronization request, such as pulling a logical log file in binlog format from the server through the target database physical replication protocol. The conversion server pushes the second logical log file to the target database based on the target database physical replication protocol in response to the synchronization request.

[0068] Optionally, taking MYSQL as an example of the target database, the conversion server implements the MYSQL physical replication protocol, disguises itself as a MYSQL server, and supports responding to data synchronization requests. The target database MYSQL (or other tools or platforms that need to pull up BINLOG) sends a data synchronization request to the conversion server to pull the logical log files in binlog format stored in the server through the MYSQL physical replication protocol. The conversion server responds to the data synchronization request of MYSQL, establishes a transmission channel based on the MYSQL physical replication protocol, and pushes the logical log files in binlog format to MYSQL in the form of a logical log file stream binlogstream in binlog format. If the target database is another database, the second logical log file is pushed to the other database according to the physical replication protocol of the other database.

[0069] It should be noted that the conversion task of the conversion server for converting the current event stream format file into the second logical log file of the target database and the push task of pushing the second logical log file to the target database are two asynchronous tasks. One is to execute the conversion task for each current event stream format file; the other is to execute the push task for the converted second logical log file. The execution of the second task is based on the completion of the first task, but the second logical log file that has been converted is not pushed immediately after the current event stream format file is converted in real time. For example, if the current event stream format file obtained by the conversion server is numbered 6 (starting from 1), it means that the task of converting the event stream format files with numbers 1, 2, 3, 4, and 5 into the second logical log file has been completed. On the one hand, the conversion server converts the event stream format file with number 6 into the second logical log file; on the other hand, it can also respond to the synchronization request of the target database and execute the push task of the second logical log file corresponding to any one or more event stream format files with numbers 1-5.

[0070] In the above data synchronization method, the current event stream format file of the source database is obtained; wherein the current event stream format file is determined based on the conversion of the first logical log file of the source database; on the one hand, the current event stream format file of the source database is converted into the second logical log file of the target database, and only the current event stream format file needs to be converted, which reduces the complexity of the deployment and operation and maintenance of synchronization tools and middleware, thereby reducing the occupation of computer resources; on the other hand, in response to the data synchronization request of the target database, the second logical log file is pushed to the target database based on the physical replication protocol of the target database, and the second logical log file can be directly copied through the physical replication protocol, which is faster and more efficient than the method of executing a large number of SQL statements by logical replication. Thus, high data synchronization efficiency and low resource occupation during data synchronization are achieved, that is, high efficiency and low consumption are achieved in the process of data synchronization.

[0071] In an exemplary embodiment, the event stream queue is stored and the current event stream format file is obtained from the event stream queue, as shown in FIG4 , before the current event stream format file of the source database is obtained, steps S402 to S406 are included. Among them:

[0072] Step S402: monitor the event port and obtain the event stream format file of the source database.

[0073] Optionally, the conversion server monitors the event port through an acquisition module to obtain the event stream format file of the source database. The acquisition module monitors the event port event port through an event handle event handle to receive the event stream format file from the source database or update the event stream format file.

[0074] Step S404: write the event stream format file into the event stream queue.

[0075] Among them, the event stream queue is a queue that processes real-time data based on the first-in-first-out (FIFO) principle.

[0076] Optionally, the acquisition module in the conversion server writes the event stream format file or the updated event stream format file into the event stream queue event queue through the event handle event handle.

[0077] The current event stream format file of the source database is obtained, including step S406, obtaining the current event stream format file of the database from the event stream queue.

[0078] Optionally, the conversion server itself or the conversion server obtains the event stream format file of the database from the event stream queue as the current event stream format file through the conversion model.

[0079] In this embodiment, the event stream format file of the source database is obtained by listening to the port, and the event stream format file is written and read in real time by means of an event stream queue, which can improve the throughput and real-time performance of processing the event stream format file, and can also maintain a high degree of consistency with the sequence number of the event stream format file in the source database, thereby maintaining the original order of the event stream format file.

[0080] Continuing from the previous embodiment, multiple threads perform conversion of event stream format files, as shown in FIG5 , including steps S502 to S504, wherein: the current event stream format file of the database is obtained from the event stream queue, including step S502, wherein each thread in at least two threads obtains the corresponding event stream format file from the event stream queue.

[0081] Optionally, the conversion server obtains corresponding transactions from the event queue in a loop through each of at least two thread workers in the conversion module binlog converter. The transaction carries an event stream format file.

[0082] Converting the current event stream format file into a second logical log file of the target database includes step S504: each of the at least two threads converts the event stream format file obtained by each thread into a corresponding second logical log file.

[0083] Optionally, the conversion server converts the acquired event stream format file into a corresponding second logical log file through each of at least two threads in the conversion module binlog converter, that is, converts the event stream format file into a logical log file in the binlog format that conforms to the target database.

[0084] In this embodiment, the event stream format file conversion task is executed in parallel by multiple threads, so that the conversion efficiency can be improved.

[0085] In an exemplary embodiment, the second logical log file is obtained from the log cache and pushed to the target database, as shown in FIG6 , and the method includes steps S602 to S406 . Among them:

[0086] Step S602: Output each second logic log file to the log buffer in the order of the event stream queue.

[0087] Optionally, the conversion results are queued and output to the log cache in the order of the event stream queue, that is, each second logical log file is output to the binlog cache, and the location information of each second logical log file output most recently and the sequence number of the event stream format file corresponding to the second logical log file are recorded.

[0088] Step S604: monitor the port of the target database and receive a data synchronization request from the target database.

[0089] Optionally, the conversion server listens to the port of the target database through the push module to obtain the data synchronization request of the target database. The push module listens to the event port through the replication handle replication handle to receive the data synchronization request from the target database. If the target database is MYSQL, the replication handle replicationhandle listens to the event port mysql port to receive the binlog replication request initiated by the target database mysql (or other tools or platforms that need to pull up BINLOG).

[0090] Step S606: Obtain the second logical log file from the log cache according to the database synchronization request, and push the second logical log file to the target database based on the target database physical replication protocol.

[0091] Optionally, the push module in the conversion server obtains the second logical log file from the log cache binlog cache through the replication handle replication handle according to the database synchronization request, that is, the replication request, and establishes a transmission channel according to the replication request, such as the replication channel pushes the second logical log file to the target database in the form of a binlog stream log stream.

[0092] In this embodiment, the target database is widely adapted by supporting the physical replication protocol of the target database.

[0093] In an exemplary embodiment, as shown in FIG7 , the data synchronization request carries location information, and the location information is used to identify the location information of the target byte in the second logical log file; the second logical log file is obtained from the log cache according to the database synchronization request, and the second logical log file is pushed to the target database based on the target database physical replication protocol, including steps S702 to S704. Wherein:

[0094] Step S702: Obtain the bytes to be synchronized in the second logical log file from the log cache according to the location information.

[0095] The location information is used to identify the location information of the target byte in the second logical log file; the target byte is used to mark the pause point in the last task of pushing the second logical log file to the target database. The second logical log file includes a number of byte information, such as byte0, byte1...byten. After the last push task, byte0 to byte100 in the second logical log file are successfully stored in the target database, and the push task is terminated at this time. Byte100 is the target byte. The byte to be synchronized represents the byte information after the position of the target byte, that is, the byte information after byte100.

[0096] In actual applications, in MySQL, the position information is used to specify the position from which to start reading or writing binlog events. The conversion server obtains the bytes to be synchronized in the second logical log file from the log cache binlog cache according to the position information, such as the byte information after byte100.

[0097] Step S704: Push the bytes to be synchronized to the target database based on the physical replication protocol of the target database.

[0098] Optionally, the conversion server pushes the bytes to be synchronized, such as the bytes after byte 100, to the target database based on the physical replication protocol of the target database. After receiving the push, the target database stores each byte of information and records the location information of the second logical log file that was successfully stored. When executing the next synchronization task, the location information is carried in the data synchronization request and sent to the conversion server. The conversion server restarts the push of the second logical log file starting from the location information.

[0099] In this embodiment, the breakpoint of the last push task is recorded through the location information, so that the breakpoint resume of the front-end and back-end conversion can be realized, and the resume of transmission at any location is realized to achieve data integrity.

[0100] In an exemplary embodiment, as shown in FIG8 , obtaining the current transaction of the source database includes steps S802 to S804 . Among them:

[0101] Step S802, obtaining the event stream format file carried by the j-th transaction.

[0102] The event stream format file carried by the jth transaction represents the last successfully converted event stream format file in the previous conversion task of the conversion task where the current event stream format file is located; j is a positive integer greater than or equal to 1.

[0103] Optionally, after the source database starts the transmission connection, it first needs to obtain the sequence number of the last successfully converted event stream format file from the conversion server, such as j, and start the transmission of a new current transaction from the sequence number.

[0104] Step S804: Use the (j+1)th event stream format file as the current transaction of the source database.

[0105] Optionally, the conversion server uses the (j+1)th event stream format file as the current transaction of the source database. The conversion server converts the current event stream format file, that is, the (j+1)th event stream format file, into the second logical log file of the target database and stores it in the log cache binlog cache, so that when responding to the synchronization request of the target database, the second logical log file stored in the log cache binlog cache is pushed.

[0106] In this embodiment, the sequence number of the last successfully converted event stream format file is obtained, the sequence number of the last executed conversion task is recorded, and the transmission of a new transaction is started from the sequence number, thereby ensuring data integrity.

[0107] In an exemplary embodiment, as shown in FIG9 , the conversion server convertserver includes an event port event port, an event handle event handle, an event stream queue event queue, a conversion module binlogconverter, a log cache binlog cache, a replication handle replication handle, and a target database listening port mysql port.

[0108] Before obtaining the current transaction of the source database (source DB), the conversion server obtains the first logical log file of one or more source databases; converts the first logical log file into an event stream format file according to the acquisition order of the first logical log file, and encapsulates each event stream format file into an event or transaction, and transmits it to the conversion server through the event stream eventstream. The acquisition module in the conversion server listens to the event port event port through the event handle event handle, receives the event stream format file or the updated event stream format file from the source database, and writes the event stream format file or the updated event stream format file into the event stream queue event queue. The conversion server cyclically obtains the corresponding transaction from the event stream queue eventqueue through each of the at least two thread workers in the conversion module binlog converter. The transaction carries the event stream format file. The conversion server converts the event stream format file obtained by each of the at least two threads in the conversion module binlogconverter into the corresponding second logical log file, that is, converts the event stream format file into a logical log file in the binlog format that conforms to the target database. The conversion results are queued and output to the log cache in the order of the event stream queue event queue, that is, each second logical log file is output to the binlog cache. The location information of each output second logical log file and the sequence number of the event stream format file corresponding to the second logical log file are recorded, such as j.

[0109] After the source database starts the transmission connection again, it first needs to obtain the sequence number of the last successfully converted event stream format file from the conversion server, such as j, and start the transmission of the new current transaction from this sequence number. The conversion server uses the (j+1)th event stream format file as the current transaction of the source database. The conversion server converts the current event stream format file, that is, the (j+1)th event stream format file, into the second logical log file of the target database and stores it in the log cache binlogcache, so that when responding to the synchronization request of the target database later, the second logical log file stored in the log cache binlog cache is pushed.

[0110] The conversion server listens to the port of the target database through the push module to obtain the data synchronization request of the target database. Among them, the push module listens to the event port through the replication handle replication handle to receive the data synchronization request from the target database. If the target database is mysql, the replication handle replication handle listens to the event port mysql port to receive the binlog replication request initiated by the target database mysql (or other tools or platforms that need to pull up BINLOG). The push module in the conversion server obtains the second logical log file from the log cache binlog cache through the replication handle replication handle according to the database synchronization request, that is, the replication request, and establishes a transmission channel according to the replication request, such as the replication channel pushing the second logical log file to the target database in the form of binlog stream log stream. Among them, the second logical log file includes several byte information, such as byte0, byte1...byten. After the last push task, the byte0 to byte100 in the second logical log file are successfully stored in the target database, and the push task is terminated at this time. Byte100 is the target byte. The byte to be synchronized represents the byte information after the position of the target byte, that is, the byte information after byte100.

[0111] In MySQL, the conversion server obtains the bytes to be synchronized in the second logical log file from the log cache binlog cache according to the location information, such as the byte information after byte100. The conversion server pushes the bytes to be synchronized, such as the byte information after byte100, to the target database based on the physical replication protocol of the target database. After receiving the push, the target database stores each byte information and records the location information of the second logical log file that was successfully stored. When executing the next synchronization task, the location information is carried through the data synchronization request and sent to the conversion server. The conversion server restarts the push of the second logical log file starting from the location information.

[0112] In this embodiment, on the one hand, the current event stream format file of the source database is converted into the second logical log file of the target database. Only the current event stream format file needs to be converted, which reduces the complexity of the deployment and operation and maintenance of synchronization tools and middleware, thereby reducing the occupation of computer resources; on the other hand, in response to the data synchronization request of the target database, the second logical log file is pushed to the target database based on the physical replication protocol of the target database. The second logical log file can be directly copied through the physical replication protocol. Compared with the method of executing a large number of SQL statements by logical replication, it is faster and has high replication efficiency. Thus, high data synchronization efficiency and low resource occupation during data synchronization are achieved, that is, high efficiency and low consumption are achieved in the process of data synchronization.

[0113] The source database only needs to generate a general event stream format file output from the first logical log file to connect to the conversion service; during the generation, the source database first logical log file can generate an event stream format file and the conversion service as a service, so that it can be integrated into the deployment environment of the source database as a product's ability to synchronize data externally. It can easily support the needs of multiple source databases to one target database. And any MYSQL target database can be directly connected to the conversion service to perform replication; various clients or custom tools can pull data through the binlog replication protocol, such as tools on the warehousing platform, custom synchronization tools, etc.

[0114] By acquiring the event stream format file of the source database through the listening port, and writing and reading the event stream format file in real time through the event stream queue, the throughput and real-time performance of processing the event stream format file can be improved, and the sequence number of the event stream format file in the source database can be highly consistent, maintaining the original order of the event stream format file.

[0115] On the one hand, the breakpoint of the last push task is recorded through the site information, so that the breakpoint resumption of the front and back ends of the conversion can be realized, and the resumption of transmission at any site can be achieved; on the other hand, the sequence number of the last conversion task is recorded by obtaining the sequence number of the last successfully converted event stream format file, and the transmission of a new transaction is started from this sequence number, both of which ensure data integrity.

[0116] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0117] Based on the same inventive concept, the embodiment of the present application also provides a data synchronization device for implementing the data synchronization method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data synchronization device embodiments provided below can refer to the limitations on the data synchronization method above, and will not be repeated here.

[0118] In an exemplary embodiment, as shown in FIG. 10 , a data synchronization device is provided, including: an acquisition module 1001, a conversion module 1002 and a push module 1003, wherein:

[0119] The acquisition module 1001 is used to acquire the current transaction of the source database, which carries a current event stream format file; wherein the current event stream format file is determined based on the conversion of the first logical log file of the source database; each transaction carries a corresponding event stream format file.

[0120] The conversion module 1002 is used to convert the current event stream format file into a second logical log file of the target database.

[0121] The push module 1003 is used to push the second logical log file to the target database based on the physical replication protocol of the target database in response to the data synchronization request of the target database.

[0122] In an exemplary embodiment, a data synchronization device further includes: a monitoring storage module, which is used to monitor an event port and obtain an event stream format file of a source database; write the event stream format file into an event stream queue;

[0123] The acquisition module 1001 is also used to acquire the current event stream format file of the source database from the event stream queue.

[0124] In an exemplary embodiment, the acquisition module 1001 is further configured to acquire a corresponding event stream format file from the event stream queue through each of at least two threads.

[0125] The conversion module 1002 is also used for each of the at least two threads to convert the event stream format file obtained by each thread into a corresponding second logical log file.

[0126] In an exemplary embodiment, a data synchronization device further includes:

[0127] The cache module is used to output each second logic log file to the log cache in the order of the event stream queue.

[0128] The push module 1003 is also used to monitor the port of the target database, receive a data synchronization request from the target database, obtain the second logical log file from the log cache according to the database synchronization request, and push the second logical log file to the target database based on the target database physical replication protocol.

[0129] In an exemplary embodiment, the data synchronization request carries location information, and the location information is used to identify the location information of the target byte in the second logical log file; the push module 1003 is also used to obtain the bytes to be synchronized in the second logical log file from the log cache according to the location information; wherein the bytes to be synchronized represent the byte information after the location of the target byte; and the bytes to be synchronized are pushed to the target database based on the physical replication protocol of the target database.

[0130] In an exemplary embodiment, the acquisition module 1001 is further used to: acquire the event stream format file carried by the j-th transaction, the event stream format file carried by the j-th transaction represents the last successfully converted event stream format file in the previous conversion task of the conversion task where the current event stream format file is located; j is a positive integer greater than or equal to 1; and use the (j+1)-th event stream format file as the current transaction of the source database.

[0131] Each module in the above data synchronization device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0132] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store logical log data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0133] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0135] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0136] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0137] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Acquire a current transaction of a source database, wherein the current transaction carries a current event stream format file; wherein the current event stream format file is determined based on the conversion of a first logical log file of the source database; each transaction carries a corresponding event stream format file; Convert the current event stream format file into a second logical log file of the target database; In response to a data synchronization request of the target database, the second logical log file is pushed to the target database based on a physical replication protocol of the target database.

2. The method according to claim 1, characterized in that Before obtaining the current event stream format file of the source database, the method further includes: Listen to the event port and obtain the event stream format file of the source database; Writing the event stream format file into an event stream queue; The step of obtaining the current event stream format file of the source database includes: The current event stream format file of the source database is obtained from the event stream queue.

3. The method according to claim 2, characterized in that Obtaining the current event stream format file of the database from the event stream queue, including: Obtaining a corresponding event stream format file from the event stream queue through each of at least two threads; The converting the current event stream format file into a second logical log file of the target database comprises: Each thread of the at least two threads converts the event stream format file obtained by itself into a corresponding second logical log file.

4. The method according to any one of claims 2 or 3, characterized in that: The method further comprises: Outputting each of the second logical log files to a log buffer in the order of the event stream queue; The step of responding to the data synchronization request of the target database and pushing the second logical log file to the target database based on the physical replication protocol of the target database includes: Listening to the port of the target database and receiving a data synchronization request from the target database; The second logical log file is obtained from the log cache according to the database synchronization request, and the second logical log file is pushed to the target database based on the target database physical replication protocol.

5. The method according to claim 4, characterized in that The data synchronization request carries location information, where the location information is used to identify location information of a target byte in the second logical log file; acquiring the second logical log file from the log cache according to the database synchronization request, and pushing the second logical log file to the target database based on the target database physical replication protocol, includes: Obtaining the to-be-synchronized bytes in the second logical log file from the log cache according to the position information; wherein the to-be-synchronized bytes represent the byte information after the position where the target byte is located; The bytes to be synchronized are pushed to the target database based on the target database physical replication protocol.

6. The method according to claim 1, characterized in that Get the current transaction of the source database, including: Get the event stream format file carried by the jth transaction, where the event stream format file carried by the jth transaction represents the last successfully converted event stream format file in the previous conversion task of the conversion task where the current event stream format file is located; j is a positive integer greater than or equal to 1; The (j+1)th event stream format file is used as the current transaction of the source database.

7. A data synchronization device, characterized in that: The device comprises: An acquisition module is used to acquire a current transaction of a source database, wherein the current transaction carries a current event stream format file; wherein the current event stream format file is determined based on the conversion of a first logical log file of the source database; and each transaction carries a corresponding event stream format file; A conversion module, used for converting the current event stream format file into a second logical log file of a target database; The push module is used to respond to the data synchronization request of the target database and push the second logical log file to the target database based on the physical replication protocol of the target database.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images