Data synchronization method and device

Abstract

The invention discloses a data synchronization method and device. A specific embodiment of the method comprises the steps of determining whether a target node of a database comprises to-be-synchronized pre-written log data or not; in response to the determination, the pre-written log data is synchronized to the target storage system through a new synchronization function after an original synchronization function of the target node is replaced, and the original synchronization function is used for persisting the pre-written log data to the target node. According to the method, the original synchronization function is replaced by the new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system, the synchronization process does not need to use other components and is completely fused with the original high-availability architecture of the database, and the synchronization efficiency is improved. On the basis of reducing the complexity of the system structure, the overall high availability and real-time data release capability of the system are ensured.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, specifically to the field of database and cloud service technology, and in particular to a data synchronization method, device, computer-readable medium, electronic device, and computer program product. Background Art

[0002] The MySQL system can store data update operations in the form of binlog (binary log) (e.g., mysql-bin.000001, mysql-bin.000002, mysql-bin.000003, the suffix values ​​of the log files increase in the order of log generation time). By parsing and replaying the binlog data in sequence, the data source (i.e., the MySQL instance that generates the binlog data) can be synchronized (replicated). Existing third-party real-time data publishing tools based on MySQL binlog are generally implemented by simulating the master-slave replication protocol, which requires the introduction of additional components on the basis of the original system structure. Summary of the invention

[0003] The embodiments of the present application provide a data synchronization method, device, computer-readable medium, electronic device, and computer program product.

[0004] In a first aspect, an embodiment of the present application provides a data synchronization method, comprising: determining whether a target node of a database includes pre-written log data to be synchronized; in response to the determination, synchronizing the pre-written log data to a target storage system by replacing an original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0005] In some examples, the target node is a slave node of a database, and the above method also includes: synchronizing the original pre-write log data in the master node of the database to the relay log of the target node; replaying the original pre-write log data in the relay log to obtain the pre-write log data generated during the replay process, and storing the pre-write log data in the system cache of the operating system where the database is located.

[0006] In some examples, the above-mentioned method of synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function includes: in response to determining that the target node calls the original synchronization function, calling the new synchronization function through the function hook corresponding to the original synchronization function, and synchronizing the pre-written log data to the target storage system, wherein during the startup process of the application of the target node, the dynamic link library including the new synchronization function is loaded.

[0007] In some examples, the above method also includes: asynchronously persisting the pre-written log data to the target node.

[0008] In some examples, the method further includes: in response to the target node being restarted, synchronizing data on the target node with reference to log data in the target storage system.

[0009] In some examples, the above-mentioned data synchronization of the target node with reference to the log data in the target storage system includes: determining the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located, wherein the suffix value of the file name of the log file is positively correlated with the file generation time; and synchronizing the log files in the target storage system whose file name suffix values ​​are between the target suffix value and the maximum value to the target node.

[0010] In some examples, the above-mentioned synchronization of pre-written log data to the target storage system includes: determining whether the log file to be synchronized where the pre-written log data is located is synchronized to the target storage system for the first time; in response to determining that it is, generating a synchronized log file in the target storage system according to the suffix value of the file name of the log file to be synchronized; and synchronizing the pre-written log data to the synchronized log file using an object storage method.

[0011] In some examples, the target storage system includes a distributed storage system and / or a message queue in a third-party storage system.

[0012] In a second aspect, an embodiment of the present application provides a data synchronization device, comprising: a determination unit, configured to determine whether a target node of a database includes pre-written log data to be synchronized; a synchronization unit, configured to synchronize the pre-written log data to a target storage system in response to the determination including by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0013] In some examples, the target node is a slave node of a database, and the above-mentioned device also includes: a relay unit, configured to synchronize the original pre-write log data in the master node of the database to the relay log of the target node; a replay unit, configured to replay the original pre-write log data in the relay log, obtain the pre-write log data generated during the replay process, and store the pre-write log data in the system cache of the operating system where the database is located.

[0014] In some examples, the synchronization unit is further configured to: in response to determining that the target node calls the original synchronization function, call the new synchronization function through the function hook corresponding to the original synchronization function, and synchronize the pre-written log data to the target storage system, wherein during the startup process of the application of the target node, the dynamic link library including the new synchronization function is loaded.

[0015] In some examples, the apparatus further includes: an asynchronous persistence unit configured to asynchronously persist the pre-written log data to the target node.

[0016] In some examples, the apparatus further includes: a restart recovery unit configured to synchronize data on the target node with reference to log data in the target storage system in response to the target node being restarted.

[0017] In some examples, the restart recovery unit is further configured to: determine the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located, wherein the suffix value of the file name of the log file is positively correlated with the file generation time; synchronize the log files in the target storage system whose file name suffix values ​​are between the target suffix value and the maximum value to the target node.

[0018] In some examples, the synchronization unit is further configured to: determine whether the log file to be synchronized where the pre-written log data is located is being synchronized to the target storage system for the first time; in response to determining yes, generate a synchronized log file in the target storage system based on the suffix value of the file name of the log file to be synchronized; and synchronize the pre-written log data to the synchronized log file using an object storage method.

[0019] In some examples, the target storage system includes a distributed storage system and / or a message queue in a third-party storage system.

[0020] In a third aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0021] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0022] In a fifth aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0023] The data synchronization method and device provided by the embodiments of the present application determine whether the target node of the database includes pre-written log data to be synchronized; in response to the determination, the pre-written log data is synchronized to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node, thereby replacing the original synchronization function with the new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system, and the synchronization process does not need to rely on other components, and is completely integrated with the native high-availability architecture of the database, thereby reducing the complexity of the system structure and ensuring the overall high availability and real-time data publishing capabilities of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0025] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

[0026] Figure 2 is a flow chart of an embodiment of a data synchronization method according to the present application;

[0027] Figure 3 This is a schematic diagram of the master-slave replication mechanism in the native master-slave mode of the database;

[0028] Figure 4 This is a schematic diagram of the master-slave synchronization process of the database based on the group submission mechanism;

[0029] Figure 5 is a schematic diagram showing the performance in a one-master-three-slave mode;

[0030] Figure 6 It is a data synchronization schematic diagram of a third-party real-time data publishing tool;

[0031] Figure 7 It is a structural diagram of the system architecture of existing third-party real-time data publishing tools;

[0032] Figure 8 This is a schematic diagram of the high availability structure based on Canal;

[0033] Fig. 9 This is a schematic diagram of the architecture in stand-alone mode;

[0034] Fig.10 is a schematic diagram of an application scenario of the data synchronization method according to this embodiment;

[0035] Fig.11 This is a schematic diagram of the architecture in master-slave mode;

[0036] Fig.12 This is a schematic diagram of the data storage format under distributed object storage;

[0037] Fig.13 This is a schematic diagram of the replacement principle under the Linux system;

[0038] Fig.14 It is a schematic diagram of the structure of the dynamic link library;

[0039] Fig.15 is a flow chart of another embodiment of the data synchronization method according to the present application;

[0040] Fig.16 A structural diagram of an embodiment of a data synchronization device according to the present application;

[0041] Fig.17 A schematic diagram of the structure of a computer system suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0042] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0045] Figure 1 An exemplary architecture 100 to which the data synchronization method and apparatus of the present application can be applied is shown.

[0046] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The communication connection between the terminal devices 101, 102, 103 constitutes a topological network, and the network 104 is used to provide a medium for the communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0047] The terminal devices 101, 102, 103 can interact with the server 105 through the network 104 to receive or send data, etc. The terminal devices 101, 102, 103 can be hardware devices or software that support network connection for data interaction and data processing. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to smart phones, car computers, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0048] The server 105 may be a server that provides various services, for example, based on the data update operation issued by the user to the database through the terminal devices 101, 102, 103, the pre-written log data is obtained, and the pre-written log data is synchronized to the background processing server of the target storage system by replacing the original synchronization function of the target node with a new synchronization function. As an example, the server 105 may be a cloud server.

[0049] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0050] It should also be noted that the data synchronization method provided in the embodiments of the present application is generally executed by a server, but the possibility of being executed by a terminal device, or being executed by a server and a terminal device in cooperation with each other is not excluded. Accordingly, the various parts (such as various units) included in the data synchronization device can be all set in the server, or all set in the terminal device, or can be set in the server and the terminal device respectively.

[0051] It should be understood that Figure 1The number of terminal devices, networks, and servers in the system is only illustrative. Any number of terminal devices, networks, and servers may be provided as required. When the electronic device on which the data synchronization method is running does not need to perform data transmission with other electronic devices, the system architecture may only include the electronic device (e.g., server or terminal device) on which the data synchronization method is running.

[0052] Continue to refer Figure 2 , shows a process 200 of an embodiment of a data synchronization method, comprising the following steps:

[0053] Step 201 : Determine whether the target node of the database includes pre-written log data to be synchronized.

[0054] In this embodiment, the execution subject of the data synchronization method (for example Figure 1 A terminal device or server in the database can determine whether the target node of the database includes pre-written log data to be synchronized.

[0055] The database may be a database involved in a synchronization process of pre-writing log data, such as MySQL, MariaDB, Percona Server for MySQL, PostgreSQL, and the like.

[0056] Pre-write log data (WAL, Write-Ahead Logging) is a log mechanism widely used in database systems. Its core concept is to record data update operations in log files before actually updating data files. For databases, data update content is written to both data files (data files will have a large number of scattered writes) and log files (log files are append-only files with good write performance). When submitting a write transaction, only the log file needs to be persisted (this is "pre-write"), and then the data file is asynchronously persisted when the opportunity arises. When submitting a write transaction, the reason why it is not necessary to perform persistence operations on the data file is because the data file can be repaired by replaying the log file. In this embodiment, the reason why only the pre-write log data needs to be synchronized and the data file does not need to be synchronized is that the data update operation can be replayed based on the pre-write log data to obtain the data corresponding to the data file. For example, the pre-write log data is binlog (binary log, binary log) data.

[0057] The database generally includes multiple nodes, for example, a master node (master database) and multiple slave nodes (slave databases). The target node can be any node in the database.

[0058] Taking the MySQL system as an example, continue to refer to Figure 3, which shows a schematic diagram of the master-slave replication mechanism in the native master-slave mode of the database system.

[0059] MySQL system can store data update operations in the form of binlog (such as: mysql-bin.000001, mysql-bin.000002, mysql-bin.000003, the suffix of the log file increases in the order of log generation time). By parsing and replaying the binlog data in sequence, the data source (i.e., the MySQL instance that generates the binlog) can be replicated. In a MySQL data synchronization solution, the system used for binlog dump is called a "Publish / Subscribe" system, or a CDC (Change data capture) system. First, the binlog generated by the MySQL instance (i.e., the data producer producer) can be collected and temporarily stored in the above dump system; then, these log files can be pushed to different data consumers (consumers) or subscribed by different consumers. Through the above dump system, the MySQL instance can be synchronized with multiple heterogeneous data consumers (such as: relational databases, such as PostgreSQL, Oracle; document databases, such as ElasticSearch; KV databases, such as TiKV).

[0060] The MySQL master node (Master) converts data updates into binlog format and transmits binlog data to the slave node through the data replication protocol between the master and slave nodes (i.e., binlogdump command). In the slave node, the IO (input and output) thread is responsible for network communication with the master node and dumps the binlog data content to the local disk file in the form of "Relay log". The SQL (Structured Query Language) thread in the slave node synchronizes data with the master node by reading and parsing the relay log and replaying the log content.

[0061] In native master-slave replication mode, slave nodes can be set to read-only mode. The read-only mode of the slave node and the data synchronization process from the master node to the slave node, which is responsible for the SQL thread, are two independent processes. The read-only setting of the slave node mainly affects the external data write operation to the slave node, while the data synchronization process from the master node to the slave node is automatically completed through the replication mechanism of the MySQL system and is not affected by the read-only status of the slave node.

[0062] According to the different data reliability, the synchronization process is divided into asynchronous replication, semi-synchronous replication and synchronous replication. Among them, in the asynchronous replication mode, after the master node sends binlog data to the slave node, it can complete the submission of the write transaction without waiting for the relay log of the slave node to land; in the semi-synchronous replication mode, after the relay log in the slave node lands, it sends an ack (acknowledgement) packet to the master node, and the master node can complete the submission of the write transaction only after receiving the ack. In this way, on the one hand, the consistency of the write transaction between the master and the slave can be guaranteed, and on the other hand, the slave node does not need to confirm the submission after it is completely consistent with the master node (this mechanism is called "synchronous replication"), which is a solution that takes into account both performance and consistency.

[0063] Continue to refer Figure 4 , which shows a schematic diagram of the master-slave synchronization process of the database based on the group submission mechanism.

[0064] The binlog group commit mechanism of MySQL database can be regarded as a way to optimize write performance, that is, to commit a group of transactions to be committed in batches instead of executing the commit action for each transaction separately. The specific implementation method is that within a certain time interval, the transactions that enter the committable state will store their updated contents in the binlog cache (binary log file cache); then the transaction with the smallest transaction number in this group of transactions is selected as the commit leader, which is responsible for driving the commit action of all transactions in the binlog cache. On the one hand, the data in the binlog cache will be appended to the end of the local binlog file of the master node and persisted to the disk through the "flush" (i.e., fdatasync) operation; on the other hand, this part of the binlog will be sent to the slave node (slave) through the MySQL master-slave replication protocol. When the binlogdump reply (ack packet) from the slave node is received, the commit of this group of transactions can be confirmed and all waiting transactions in this commit group can be awakened. The reason why fdatasync is used as the confirmation operation for log persistence is that in order to improve file writing performance, the operating system does not directly persist data to disk in each write operation, but first writes data to the system cache (called pagecache in Linux system), and then persists the binlog in the system cache to disk in batches through the fdatasync operation.

[0065] The above group submission mechanism is actually an implementation process similar to "two-stage submission". Process ② (the master node persists the binlog data to the local file) and process ③ (the master node sends the binlogdump data packet to the slave node after receiving the binlogdump request from the slave node) are asynchronous and concurrent. When both process ② and process ④ (the slave node sends the confirmation packet corresponding to the data packet in process ③ to the master node) are completed, process ⑤ is executed to wake up all group members in the group submission and return the submission success to the client. Therefore, the overall time consumption of a group submission is the time consumption of the longest execution time in process ② and process ③—>④, that is:

[0066] Ttotal=max(T2,T3+T4)

[0067] Among them, Ttotal represents the total time consumption of a group commit transaction, and T2, T3, and T4 represent the time consumption of processes ②, ③, and ④ respectively. In general, the overall time consumption of processes ③ and ④ is significantly greater than the time consumption of process ②. In this way, the above formula can be approximated as:

[0068] Ttotal≈T3+T4

[0069] When there are multiple slave nodes, a binlogdump thread is created between the master node and each slave node, and the T3+T4 time depends on the slave node with the slowest ack reply speed among all the slave nodes. Generally speaking, the more slave nodes there are, the worse the master-slave replication performance.

[0070] Continue to refer Figure 5 , shows a schematic diagram of the performance model under the one-master-three-slave (one master node and three slave nodes) mode.

[0071] There is a parameter called rpl_semi_sync_master_wait_for_slave_count in the MySQL system, which is used to specify how many ack responses from slave nodes are received at least before a commit can be confirmed. When there are 3 slave nodes and the expected number of ack responses indicated by the above parameter is at least 2, it means that a commit can be confirmed when 2 acks are received.

[0072] Continue to refer Figure 6 , showing the data synchronization principle diagram of a third-party real-time data publishing tool.

[0073] Third-party real-time data publishing tools based on MySQL binlog are generally implemented by simulating the master-slave replication protocol. The more mainstream open source implementations include Canal, Maxwell, etc. Taking Canal as an example, the Canal server simulates itself as a slave node of the MySQL system, pulls binlog data in real time through the binlogdump command, and then pushes it to other databases or message queues (such as Kafka, RocketMQ, etc.).

[0074] Continue to refer Figure 7 , which shows a schematic diagram of the system architecture of a third-party real-time data publishing tool. Current binlog real-time publishing technologies require the addition of many new components to the original system. Taking the synchronization of binlog data from MySQL to a distributed storage system (such as HDFS) as an example, three components, Canal, Kafka, and Flume, need to be introduced between MySQL and HDFS. Of course, you can also develop a Canal client adapter (Canal client) yourself to directly connect to HDFS; but at least two components, Canal server and Canal client, need to be introduced.

[0075] The MySQL database itself can ensure its own high availability through the master-slave replication mechanism. However, after the introduction of new components, the new components themselves need to have a high availability mechanism. Figure 8 , shows a schematic diagram of the high availability structure based on Canal. Taking Canal as an example, Canal high availability is guaranteed by Zookeeper (a distributed application coordination service, so an additional component is introduced):

[0076] In step 1, Canal server A and Canal server B simultaneously initiate a startup request to the Zookeeper cluster; in step 2, the Zookeeper cluster selects one of the two Canal servers as the startup node (for example, Canal server A) and notifies it to start; in step 3, when Canal server A is unavailable, the Zookeeper cluster finds that Canal server A is offline and notifies Canal server B to start; in step 4, at this time, when a Canal client tries to connect, it first queries the Zookeeper cluster to obtain the host information of the current Canal server (for example, Canal server B); in step 5, the Canal client initiates a connection to the Canal server returned by Zookeeper.

[0077] It can be seen that the above-mentioned MySQL binlog real-time publishing technology not only requires the introduction of additional components, but also requires the implementation of additional high-availability mechanisms for the additional components. This not only increases the complexity of the system and reduces the overall availability of the system (the more components, the lower the overall availability); in addition, the deployment of the new components and their high-availability mechanisms also requires the introduction of additional hardware, thereby increasing the overall cost of system deployment.

[0078] In this embodiment, the target node can be the master node in the database, and the master node is no longer provided with a slave node. This mode can be called a stand-alone mode, or a single master node mode, or a single master library mode. When the master node in the database serves as the data publishing end, the above-mentioned execution subject needs to determine whether the target node of the database includes the pre-written log data to be synchronized. Specifically, the target storage system to be synchronized with the data simulates the master-slave replication protocol, sends a binlogdump command to the master node, determines the synchronized data in the target storage system by parsing the binlogdump command, and then determines whether the master node includes the pre-written log data to be synchronized other than the synchronized data based on the synchronized data.

[0079] Step 202, in response to the determination, includes synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with the new synchronization function.

[0080] In this embodiment, in response to the determination, the execution subject synchronizes the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0081] The original synchronization function, such as the above-mentioned fsync or fdatasync function, is used to persist the pre-written log data to the local target node. In this embodiment, the original synchronization function can be modified to obtain a new synchronization function after replacement. In the new synchronization function, the operation of persisting the pre-written log data to the local disk of the target node is no longer performed, but the pre-written log data that has not been written to the disk is written to the target storage system. Among them, the target storage system can be any storage system with data storage function.

[0082] The target storage system can be a distributed storage system and / or a message queue in a third-party storage system. A distributed storage system is a storage method that stores data in multiple physical nodes in a dispersed manner and achieves unified management and access through network connections. It has the advantages of high availability, high performance, and high scalability. Examples include HDFS (Hadoop Distributed File System), Ceph, and XDFS. A message queue is a cross-process communication mechanism used to asynchronously transfer data between different applications or different components of the same application. It allows producers and consumers to communicate in a decoupled manner, that is, producers do not need to know the specific implementation of consumers, nor do they need to wait for consumers to process data before sending the next piece of data. Examples include RabbitMQ, Kafka, and ActiveMQ.

[0083] As an example, the source code of the database system may be modified in an intrusive manner to modify the original synchronization function into a new synchronization function. As another example, the original synchronization function in the database system may be replaced with a new synchronization function in a non-intrusive manner.

[0084] Continue to refer Fig. 9 , shows a schematic diagram of the architecture for data synchronization from the master node to the target storage system in stand-alone mode.

[0085] This embodiment can achieve the same performance and reliability as the native master-slave replication mechanism of the database system. Fig. 9 In the example, assume that there is a MySQL instance M (primary node) and a backup X of instance M; as long as all binlogs (that is, data increment Δ) generated after the data point backed up by X are backed up to the target storage system, the instance M can be restored to the latest state using backup X and incremental log Δ. In addition, by periodically backing up M (generating backups X1, X2, X3, etc.), binlogs can be continuously advanced and eliminated (that is, rotate operation: eliminating and deleting binlogs generated before backup Xn). In general, the target storage system provides high data reliability through a multi-copy (at least 3 copies) mechanism; in the case of 3 copies, Fig. 9 The reliability of the stand-alone mode is equivalent to the native one-master-three-slave mode of the database system. Similar to the rpl_semi_sync_master_wait_for_slave_count mechanism of MySQL master-slave replication, a distributed storage with three replicas only needs two of the replicas to reply ack when confirming a write operation. Therefore, the performance of writing pre-written daily log data to distributed storage is also similar to the one-master-three-slave MySQL replication protocol.

[0086] Continue to see Fig.10 , Fig.10 FIG. 1000 is a schematic diagram of an application scenario of the data synchronization method according to this embodiment. Fig.10 In the application scenario, user 1001 issues a data update operation to the master node 1003 of the database server through the terminal device 1002. The master node 1003 uses the group submission mechanism to determine the pre-written log data to be synchronized. Then, the pre-written log data is synchronized to the target storage system 1004 by replacing the original synchronization function of the target node with the new synchronization function. The original synchronization function is used to persist the pre-written log data to the target node 1003.

[0087] The method provided by the above-mentioned embodiment of the present application determines whether the target node of the database includes pre-written log data to be synchronized; in response to the determination, includes synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node, thereby replacing the original synchronization function with the new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system, and the synchronization process does not need to rely on other components, and is completely integrated with the native high-availability architecture of the database. On the basis of reducing the complexity of the system structure, the overall high availability and real-time data publishing capabilities of the system are guaranteed.

[0088] In some optional implementations of this embodiment, the target node is a slave node of the database. For example, in a one-master-one-slave mode of the database, the target node may be the only slave node; for another example, in a one-master-multiple-slave mode of the database, the target node may be any slave node among multiple slave nodes.

[0089] In this implementation, the above execution subject may also perform the following operations:

[0090] First, synchronize the original pre-written log data in the database's master node to the relay log of the target node.

[0091] It should be noted that when the target node is a slave node, the synchronization process between the master node and each slave node, as well as the data processing process within the master node still use the native information processing process of the database system, and only the synchronization scheme shown in the above embodiment 200 is used in the data synchronization process between the target node and the target storage system.

[0092] As an example, the synchronization process between the master node and the corresponding slave nodes, as well as the data processing process within the master node, can still use the above-mentioned group submission mechanism to process information.

[0093] Continue to refer to the above Figure 3 and 4 , for the original pre-written log data in the binlog Cache of the database's primary node, it is synchronized to the relay log Replay log of the target node through the binlog dump thread.

[0094] Second, the original pre-written log data in the relay log is replayed to obtain the pre-written log data generated during the replay process, and the pre-written log data is stored in the system cache of the operating system where the database is located.

[0095] In this implementation, the original pre-written log data in the relay log Replay log is replayed to obtain the pre-written log file, and the pre-written log file is stored in the system cache page cache of the operating system where the database is located.

[0096] Continue to refer Fig.11 , shows a schematic diagram of the architecture in the master-slave mode. The synchronization mode in this implementation requires the master node to synchronize data with the slave node, and then the slave node synchronizes data with the target storage system, so it can be called the master-slave mode, or cascade mode.

[0097] Compared with the stand-alone mode, the advantages of the master-slave mode include: the performance of the master node is the same as the native master-slave mode of the database system, which is higher than the stand-alone mode; the master node only needs to synchronize with one slave node, and the binlog parsing and uploading of the target storage system are undertaken by the slave node. The slave node is generally not connected to the query and has a lower load, thus saving the master node overhead and allowing the master database to undertake more query loads; HA (high availability) performance is better, and the slave database can quickly take over when the master database fails, reducing business jitter (while the stand-alone mode can only be restored by backup, and the recovery time is longer).

[0098] The disadvantage of the master-slave mode lies mainly in real-time performance. In stand-alone mode, it can be ensured that when the master node confirms data submission, the corresponding pre-written log data has been published to the target storage system; however, since the master-slave mode uses the slave node to upload pre-written log data in a "cascade" manner, the log data on the target storage system will lag behind the master node to a certain extent.

[0099] For the master-slave mode, due to the existence of the master-slave node switching operation, the publisher of the binlog may change frequently, unlike the "stand-alone mode" where there is always only one data publisher. Therefore, the master-slave mode needs to use metadata to store the correspondence between the data publisher and the start and end points of the data GTID (Global Transaction Identifier). Among them, the data publisher is represented by Server-UUID (Universally Unique Identifier).

[0100] Continue to refer Fig.12 , showing a schematic diagram of the data storage format under distributed object storage.

[0101] Each MySQL publisher corresponds to a Publisher-id (globally unique identifiers including UUID can be used); data objects are divided into two types: metadata and binlog data. The metadata is named Publisher-id / meta / XXXXXX (XXXXXX is an auto-increment value starting from 1, fixed to a 6-digit decimal number, with the high digits filled with 0); the binlog data object is named Publisher-id / XXXXXX / YYY (XXXXXX is an auto-increment value starting from 1, fixed to a 6-digit decimal number, with the high digits filled with 0, indicating the suffix of the MySQL binlog file; YYY is an auto-increment value starting from 1, fixed to a 3-digit decimal number, with the high digits filled with 0. When multiple binlog objects store a binlog file, this value indicates the position of the object in the binlog file).

[0102] Each time the publisher is switched (i.e., the publisher's Server-uuid changes), one metadata object is added; the metadata object with the largest number (i.e., XXXXXX value) indicates the current status.

[0103] The content of the metadata object includes the following three fields:

[0104] Previous-GTIDs: The GTID range generated by the previous publisher;

[0105] Previous-Binlogs: The binlog file suffix range generated by the previous publisher (that is, the XXXXXX value in Publisher-id / XXXXXX / YYY);

[0106] Current-Master: Server-uuid of the current publisher.

[0107] by Fig.12 Taking the data shown as an example, metadata object No. 1 indicates that the first publisher is ServerA; metadata No. 2 indicates that ServerA published data with GTIDs from 1 to 140, generated two binlog files (corresponding to the three data objects Publisher-id / 000001 / 001, Publisher-id / 000001 / 002, and Publisher-id / 000002 / 001), and the current publisher is ServerB; metadata No. 3 indicates that ServerB did not publish any updates, and the current publisher is ServerC; metadata No. 4 indicates that ServerC published data with GTIDs from 141 to 630, generated a binlog file numbered 3, and the current publisher is ServerD. Fig.12 In this state, the next binlog object to be uploaded to the distributed storage is named Publisher-id / 000004 / 001.

[0108] In some implementations of this embodiment, the execution subject may execute step 202 in the following manner: in response to determining that the target node calls the original synchronization function, a new synchronization function is called through a function hook corresponding to the original synchronization function to synchronize the pre-written log data to the target storage system.

[0109] During the startup of the application program of the target node, a dynamic link library including the new synchronization function is loaded.

[0110] In this implementation, dynamic library injection and function hook technology are used to ensure that no intrusion into the source code of the database system is required. Fig.13 , shows a schematic diagram of the replacement principle under the Linux system.

[0111] In Linux systems, functions declared with the "weak symbol" attribute in applications can be overridden by "strong symbols" with the same name; in addition, during the application startup phase, a dynamic link library can be preloaded into the program through the "LD_PRELOAD" mechanism. The C runtime library (glibc) in Linux systems generally declares library functions (such as malloc / free) in the form of weak symbols; if a strong symbol with the same name is implemented in a dynamic library, and this dynamic link library is loaded through "LD_PRELOAD" when the application starts, the strong symbol function implemented in the dynamic library can be used to replace the weak symbol function of the glibc native runtime library. For example, using the jemalloc library to replace the Linux native memory allocator (that is, replace functions such as malloc / free), it can be started as follows:

[0112] [Linux]LD_PRELOAD= / your_library_dir / jemalloc.so mysqld--defaults-file=my.cnf&

[0113] In this implementation, the above technology is used to implement a dynamic link library, and it is loaded when MySQL is started through the LD_PRELOAD method, so as to cover the Linux native file read and write library functions (such as: fsync, fdatasync, write, pwrite, etc.), realize the function replacement in a non-invasive way, and improve the flexibility of the implementation process.

[0114] Continue to refer Fig. 9 In stand-alone mode, when there is pre-written log data to be synchronized in the master node, in response to determining that the master node calls the original synchronization function, a new synchronization function is called through the function hook corresponding to the original synchronization function to synchronize the pre-written log data to the target storage system (e.g., a distributed object storage system).

[0115] Continue to refer Fig.11 In the master-slave mode, the data synchronization process between the master node and the slave node adopts the above-mentioned Figure 3 , 4 When it is determined that there is pre-written log data to be synchronized from the slave node, in response to the determination that the slave node calls the original synchronization function, the new synchronization function is called through the function hook corresponding to the original synchronization function to synchronize the pre-written log data to the target storage system.

[0116] In some implementations of this embodiment, the execution subject may further perform the following operation: asynchronously persisting the pre-written log data to the target node.

[0117] In this implementation, it is not required to execute the persistence process of the pre-written log data in the target node in real time.

[0118] As an example, in response to reaching a preset persistence condition, the pre-written log data is asynchronously persisted to the target node. The preset persistence condition is, for example, that the amount of pre-written log data in the system cache reaches a preset data amount threshold, and the execution time from the last persistence process exceeds a preset time threshold. The preset data amount threshold and the preset time threshold can be specifically set according to actual conditions and are not limited here.

[0119] In this implementation, the pre-written log data is asynchronously persisted to the target node, which reduces the data processing pressure of the target node while ensuring the data storage result.

[0120] In some implementations of this embodiment, the execution subject may further perform the following operation: in response to the target node being restarted, synchronizing data on the target node with reference to log data in the target storage system.

[0121] Synchronizing the data of the target node with reference to the log data in the target storage system is a recover operation. The reason why the recover operation is needed is that the local pre-written log data of the target node is not persisted in real time, but asynchronously persisted. Therefore, when the target node is restarted, there may be pre-written log data in the local target node that has not been asynchronously persisted. Since the data synchronization process to the target storage system is real-time, the data in the target storage system is comprehensive and reliable. At this time, it is necessary to synchronize the data of the target node with reference to the log data in the target storage system so that the log data in the target node and the target storage system are consistent.

[0122] As an example, the execution subject may compare the persistent log data in the target node with the persistent log data in the target storage system, and synchronize the log data that exists in the target storage system but not in the target node to the target node.

[0123] In this implementation, when the target node is restarted, data synchronization is performed on the target node with reference to the log data in the target storage system, thereby ensuring data consistency between the target node and the target storage system, as well as accuracy and reliability of data in the target node.

[0124] In some optional implementations of this embodiment, the execution subject may perform the data recovery process in the following manner:

[0125] First, determine the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located. The suffix value of the file name of the log file is positively correlated with the file generation time.

[0126] Second, the log files in the target storage system whose file name suffix values ​​are between the target suffix value and the maximum value are synchronized to the target node.

[0127] As an example, determine the maximum value of the suffix value of the file name of the log file in the target storage system, determine whether the data site of the local log file in the target node is located in the log file corresponding to the maximum value, and in response to not existing, reduce the maximum value by one, determine whether the data site of the local log file in the target node is located in the log file corresponding to the maximum value after reduction by one, and iterate the above process until the suffix value corresponding to the log file in the target storage system including the data site of the local log file in the target node is determined, and determine it as the target suffix value.

[0128] In this implementation, a specific data recovery method is provided, which improves the information processing efficiency and accuracy of the data recovery process.

[0129] In some implementations of this embodiment, the execution subject may perform the process of synchronizing the pre-written log data to the target storage system in the following manner:

[0130] First, it is determined whether the to-be-synchronized log file containing the pre-written log data is being synchronized to the target storage system for the first time.

[0131] Then, in response to the determination being yes, a synchronized log file in the target storage system is generated according to the suffix value of the file name of the log file to be synchronized.

[0132] In response to determining that the log file to be synchronized containing the pre-written log data is being synchronized to the target storage system for the first time, it means that there is no log file in the target storage system with the same suffix value as the file name of the log file to be synchronized; at this time, it is necessary to generate a log file in the target storage system with the same suffix value as the file name of the log file to be synchronized, that is, the synchronized log file.

[0133] Finally, object storage is used to synchronize the pre-written log data to the synchronized log file.

[0134] In this implementation, each storage object is stored in the synchronized log file in sequence, and the suffix value of the storage object increases in sequence.

[0135] In this implementation, in response to determining that the to-be-synchronized log file containing the pre-written log data is not being synchronized to the target storage system for the first time, it means that there is a log file in the target storage system with the same suffix value as the file name of the to-be-synchronized log file, namely, the synchronized log file. Then, the object storage method can be used to synchronize the pre-written log data directly to the synchronized log file, thereby improving the data synchronization efficiency of the synchronization process. Fig.12 shown.

[0136] Continue to refer Fig.14 , shows a schematic diagram of the structure of a dynamic link library. The dynamic link library of this embodiment includes three modules:

[0137] File operation library function hook: used to override the C runtime library (glibc) library functions related to file operations in the target node (such as open / creat, fsync / fdatasync, etc.), thereby replacing the target node's native operations on log files.

[0138] Binlog thread: used to perform initialization operations and the above data recovery operations during the startup phase, and asynchronously persist the pre-written log data to the target node during operation.

[0139] Third-party storage system client: used for interaction between the target node and the third-party storage system. Different client programs (generally provided by the third-party storage system supplier) can be used to adapt to different third-party storage systems (such as Amazon S3, Alibaba Cloud OSS, etc.).

[0140] For the creation (open / creat) hook in the function hook, when a new log file is created, the background binlog thread is notified to flush the previous log file (the file name suffix value is smaller than the suffix value of the current file) to the local binlog file.

[0141] For the above initialization operation, the binlog thread needs to first read the metadata object from the object storage during the startup phase. If the metadata object is not found, the suffix value of the log file is initialized to 0; otherwise, the log file with the largest file name suffix value is read from the object storage, the largest suffix value is determined, and the data position of the current log file is determined.

[0142] Continue to refer Fig.15 , shows a schematic process 1500 of another embodiment of the data synchronization method according to the present application, including the following steps:

[0143] Step 1501 , determining whether the target node of the database includes pre-written log data to be synchronized.

[0144] Step 1502, in response to determining that the target node calls the original synchronization function, a new synchronization function is called through a function hook corresponding to the original synchronization function to synchronize the pre-written log data to the distributed storage system and / or message queue in the third-party storage system.

[0145] During the startup of the application program of the target node, a dynamic link library including the new synchronization function is loaded.

[0146] Step 1503: Asynchronously persist the pre-written log data to the target node.

[0147] Step 1504, in response to the target node being restarted, determine the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located.

[0148] Among them, the suffix value of the log file name is positively correlated with the file generation time.

[0149] Step 1505: synchronize the log files in the target storage system whose file names have suffix values ​​between the target suffix value and the maximum value to the target node.

[0150] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiment, the process 1500 of the data synchronization method in this embodiment specifically illustrates the replacement process of the synchronization function, the asynchronous persistence process of the pre-written log file in the target node, and the data recovery process when the target node is restarted. The original synchronization function is replaced with a new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system. The synchronization process does not need to rely on other components and is completely integrated with the native high-availability architecture of the database. On the basis of reducing the complexity of the system structure, it ensures the overall high availability and real-time data publishing capabilities of the system.

[0151] Continue to refer Fig.16 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a data synchronization device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0152] like Fig.16As shown, a data synchronization device includes: a determination unit 1601, configured to determine whether a target node of a database includes pre-written log data to be synchronized; a synchronization unit 1602, configured to synchronize the pre-written log data to a target storage system in response to the determination including by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0153] In some implementations of this embodiment, the target node is a slave node of the database, and the above-mentioned device also includes: a relay unit (not shown in the figure), configured to synchronize the original pre-written log data in the master node of the database to the relay log of the target node; a replay unit (not shown in the figure), configured to replay the original pre-written log data in the relay log, obtain the pre-written log data generated during the replay process, and store the pre-written log data in the system cache of the operating system where the database is located.

[0154] In some implementations of this embodiment, the synchronization unit 1602 is further configured to: in response to determining that the target node calls the original synchronization function, call the new synchronization function through the function hook corresponding to the original synchronization function, and synchronize the pre-written log data to the target storage system, wherein during the startup process of the application of the target node, the dynamic link library including the new synchronization function is loaded.

[0155] In some implementations of this embodiment, the above-mentioned device also includes: an asynchronous persistence unit (not shown in the figure), which is configured to asynchronously persist the pre-written log data to the target node.

[0156] In some implementations of this embodiment, the above-mentioned device also includes: a restart recovery unit (not shown in the figure), which is configured to synchronize data of the target node with reference to log data in the target storage system in response to the target node being restarted.

[0157] In some implementations of the present embodiment, the restart recovery unit is further configured to: determine the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located, wherein the suffix value of the file name of the log file is positively correlated with the file generation time; synchronize the log files in the target storage system whose file name suffix values ​​are between the target suffix value and the maximum value to the target node.

[0158] In some implementations of this embodiment, the synchronization unit 1602 is further configured to: determine whether the log file to be synchronized where the pre-written log data is located is synchronized to the target storage system for the first time; in response to determining that it is, generate a synchronized log file in the target storage system according to the suffix value of the file name of the log file to be synchronized; and synchronize the pre-written log data to the synchronized log file using an object storage method.

[0159] In some implementations of this embodiment, the target storage system includes a distributed storage system and / or a message queue in a third-party storage system.

[0160] In this embodiment, the determination unit in the data synchronization device determines whether the target node of the database includes pre-written log data to be synchronized; the synchronization unit responds to the determination by synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node, thereby replacing the original synchronization function with the new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system, and the synchronization process does not need to rely on other components, but is completely integrated with the native high-availability architecture of the database, thereby reducing the complexity of the system structure and ensuring the overall high availability and real-time data publishing capabilities of the system.

[0161] Reference below Fig.17 , which shows a device suitable for implementing the embodiments of the present application (eg Figure 1 Schematic diagram of the structure of the computer system 1700 of the devices 101, 102, 103, 105 shown. Fig.17 The device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0162] like Fig.17 As shown, the computer system 1700 includes a processor (e.g., CPU, central processing unit) 1701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1702 or a program loaded from a storage part 1708 into a random access memory (RAM) 1703. Various programs and data required for the operation of the system 1700 are also stored in the RAM 1703. The processor 1701, the ROM 1702, and the RAM 1703 are connected to each other via a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0163] The following components are connected to the I / O interface 1705: an input section 1706 including a keyboard, a mouse, etc.; an output section 1707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1708 including a hard disk, etc.; and a communication section 1709 including a network interface card such as a LAN card, a modem, etc. The communication section 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the I / O interface 1705 as needed. A removable medium 1711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1710 as needed, so that a computer program read therefrom is installed into the storage section 1708 as needed.

[0164] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1709, and / or installed from a removable medium 1711. When the computer program is executed by the processor 1701, the above-mentioned functions defined in the method of the present application are executed.

[0165] It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0166] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the client computer, partially on the client computer, as a separate software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the client computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0167] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0168] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor comprising a determination unit and a synchronization unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases, for example, the synchronization unit may also be described as "a unit in response to determining to include, synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node".

[0169] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the computer device: determines whether the target node of the database includes pre-written log data to be synchronized; in response to the determination, includes synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0170] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A data synchronization method, comprising: Determine whether the target node of the database includes pre-written log data to be synchronized; Responsive to the determination includes synchronizing the pre-written log data to a target storage system by replacing an original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

2. The method according to claim 1, wherein: The target node is a slave node of the database, and The method further comprises: Synchronize the original pre-written log data in the master node of the database to the relay log of the target node; The original pre-written log data in the relay log is replayed to obtain the pre-written log data generated during the replay process, and the pre-written log data is stored in the system cache of the operating system where the database is located.

3. The method according to claim 1 or 2, wherein: The step of synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function includes: In response to determining that the target node calls the original synchronization function, the new synchronization function is called through the function hook corresponding to the original synchronization function, and the pre-written log data is synchronized to the target storage system, wherein during the startup process of the application of the target node, the dynamic link library including the new synchronization function is loaded.

4. The method according to claim 1, wherein: Also includes: Asynchronously persist the pre-written log data to the target node.

5. The method according to claim 4, wherein: Also includes: In response to the target node being restarted, data synchronization is performed on the target node with reference to the log data in the target storage system.

6. The method according to claim 5, wherein: The step of synchronizing data on the target node with reference to the log data in the target storage system includes: Determine a maximum value of the suffix values ​​of the file names of the log files in the target storage system and a target suffix value of the file names of the log files where the data sites in the target node are located, wherein the suffix value of the file names of the log files is positively correlated with the file generation time; The log files in the target storage system whose file names have suffix values ​​between the target suffix value and the maximum value are synchronized to the target node.

7. The method according to claim 1, wherein: The step of synchronizing the pre-written log data to the target storage system includes: Determine whether the to-be-synchronized log file containing the pre-written log data is synchronized to the target storage system for the first time; In response to the determination being yes, generating a synchronized log file in the target storage system according to the suffix value of the file name of the log file to be synchronized; The pre-written log data is synchronized to the synchronized log file by using an object storage method.

8. The method according to claim 1, wherein: The target storage system includes a distributed storage system and / or a message queue in a third-party storage system.

9. A data synchronization device, comprising: a determination unit configured to determine whether a target node of the database includes pre-written log data to be synchronized; The synchronization unit is configured to synchronize the pre-written log data to the target storage system in response to the determination, including by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

10. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

12. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images