Data synchronization method and device

Abstract

The present application discloses a data synchronization method and device. A specific implementation of the method includes: determining whether the target node of the database includes pre-written log data to be synchronized; in response to the determination, synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node. The present application replaces the original synchronization function with a new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system. The synchronization process does not need to rely on other components and is completely integrated with the native high-availability architecture of the database. On the basis of reducing the complexity of the system structure, it ensures the overall high availability and real-time data publishing capabilities of the system.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, specifically to the field of database and cloud service technology, and in particular to a data synchronization method, device, computer-readable medium, electronic device, and computer program product. Background Art

[0002] The MySQL system can store data update operations in the form of a binlog (binary log) (e.g., mysql-bin.000001, mysql-bin.000002, mysql-bin.000003, with the log file suffixes increasing in order of log generation time). By sequentially parsing and replaying the binlog data, synchronization (replication) of the data source (i.e., the MySQL instance that generates the binlog data) can be achieved. Existing third-party real-time data publishing tools based on MySQL binlog generally implement this by emulating the master-slave replication protocol, requiring the introduction of additional components on top of the existing system structure. Summary of the Invention

[0003] The embodiments of the present application provide a data synchronization method, device, computer-readable medium, electronic device, and computer program product.

[0004] In a first aspect, an embodiment of the present application provides a data synchronization method, including: determining whether a target node of a database includes pre-written log data to be synchronized; in response to the determination that it includes, synchronizing the pre-written log data to a target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0005] In some examples, the target node is a slave node of a database, and the above method also includes: synchronizing the original pre-write log data in the master node of the database to the relay log of the target node; replaying the original pre-write log data in the relay log to obtain the pre-write log data generated during the replay process, and storing the pre-write log data in the system cache of the operating system where the database is located.

[0006] In some examples, the above-mentioned method of synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function includes: in response to determining that the target node calls the original synchronization function, calling the new synchronization function through the function hook corresponding to the original synchronization function, and synchronizing the pre-written log data to the target storage system, wherein, during the startup process of the application of the target node, the dynamic link library including the new synchronization function is loaded.

[0007] In some examples, the above method also includes: asynchronously persisting the pre-written log data to the target node.

[0008] In some examples, the above method further includes: in response to the target node being restarted, synchronizing data on the target node with reference to log data in the target storage system.

[0009] In some examples, the above-mentioned data synchronization of the target node with reference to the log data in the target storage system includes: determining the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located, wherein the suffix value of the file name of the log file is positively correlated with the file generation time; and synchronizing the log files in the target storage system whose file name suffix values ​​are between the target suffix value and the maximum value to the target node.

[0010] In some examples, the above-mentioned synchronization of pre-written log data to the target storage system includes: determining whether the log file to be synchronized where the pre-written log data is located is synchronized to the target storage system for the first time; in response to the determination that it is, generating a synchronized log file in the target storage system based on the suffix value of the file name of the log file to be synchronized; and synchronizing the pre-written log data to the synchronized log file using an object storage method.

[0011] In some examples, the target storage system includes a distributed storage system and / or a message queue in a third-party storage system.

[0012] In the second aspect, an embodiment of the present application provides a data synchronization device, including: a determination unit, configured to determine whether the target node of the database includes pre-written log data to be synchronized; a synchronization unit, configured to synchronize the pre-written log data to the target storage system in response to the determination that it includes, by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0013] In some examples, the target node is a slave node of a database, and the above-mentioned device also includes: a relay unit, configured to synchronize the original pre-write log data in the master node of the database to the relay log of the target node; a replay unit, configured to replay the original pre-write log data in the relay log, obtain the pre-write log data generated during the replay process, and store the pre-write log data in the system cache of the operating system where the database is located.

[0014] In some examples, the synchronization unit is further configured to: in response to determining that the target node calls the original synchronization function, call the new synchronization function through the function hook corresponding to the original synchronization function, and synchronize the pre-written log data to the target storage system, wherein, during the startup process of the application of the target node, the dynamic link library including the new synchronization function is loaded.

[0015] In some examples, the apparatus further includes: an asynchronous persistence unit configured to asynchronously persist the pre-written log data to the target node.

[0016] In some examples, the apparatus further includes: a restart recovery unit configured to synchronize data on the target node with reference to log data in the target storage system in response to the target node being restarted.

[0017] In some examples, the restart recovery unit is further configured to: determine the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located, wherein the suffix value of the file name of the log file is positively correlated with the file generation time; synchronize the log files in the target storage system whose file name suffix values ​​are between the target suffix value and the maximum value to the target node.

[0018] In some examples, the synchronization unit is further configured to: determine whether the log file to be synchronized containing the pre-written log data is synchronized to the target storage system for the first time; in response to determining yes, generate a synchronized log file in the target storage system based on the suffix value of the file name of the log file to be synchronized; and synchronize the pre-written log data to the synchronized log file using an object storage method.

[0019] In some examples, the target storage system includes a distributed storage system and / or a message queue in a third-party storage system.

[0020] In a third aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0021] In a fourth aspect, an embodiment of the present application provides an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0022] In a fifth aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0023] The data synchronization method and device provided by the embodiments of the present application determine whether the target node of the database includes pre-written log data to be synchronized; in response to the determination, the pre-written log data is synchronized to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node, thereby replacing the original synchronization function with the new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system. The synchronization process does not need to rely on other components and is completely integrated with the native high-availability architecture of the database. On the basis of reducing the complexity of the system structure, the overall high availability and real-time data publishing capabilities of the system are guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0025] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

[0026] Figure 2 is a flow chart of an embodiment of a data synchronization method according to the present application;

[0027] Figure 3 This is a diagram of the master-slave replication mechanism in the native master-slave mode of the database;

[0028] Figure 4 This is a schematic diagram of the master-slave synchronization process of the database based on the group commit mechanism;

[0029] Figure 5 is a schematic diagram showing the performance in one master and three slaves mode;

[0030] Figure 6 It is a data synchronization schematic diagram of a third-party real-time data publishing tool;

[0031] Figure 7 This is a structural diagram of the system architecture of existing third-party real-time data publishing tools;

[0032] Figure 8 This is a schematic diagram of the high availability structure based on Canal;

[0033] Figure 9 This is a schematic diagram of the architecture in stand-alone mode;

[0034] Figure 10 is a schematic diagram of an application scenario of the data synchronization method according to this embodiment;

[0035] Figure 11 This is a schematic diagram of the architecture in master-slave mode;

[0036] Figure 12 This is a diagram of the data storage format under distributed object storage;

[0037] Figure 13 This is a schematic diagram of the replacement principle under the Linux system;

[0038] Figure 14 It is a structural diagram of the dynamic link library;

[0039] Figure 15 is a flow chart of another embodiment of the data synchronization method according to the present application;

[0040] Figure 16 A structural diagram of an embodiment of a data synchronization device according to the present application;

[0041] Figure 17 A schematic diagram of the structure of a computer system suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0042] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0043] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions disclosed herein all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to safeguard the security of user personal information, network security, and national security.

[0045] Figure 1 An exemplary architecture 100 to which the data synchronization method and apparatus of the present application can be applied is shown.

[0046] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 constitute a topological network, and network 104 is used to provide a medium for communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0047] The terminal devices 101, 102, 103 can interact with the server 105 via the network 104 to receive or send data, etc. The terminal devices 101, 102, 103 can be hardware devices or software that support network connection for data interaction and data processing. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to smart phones, car computers, tablet computers, e-book readers, laptop computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules for providing distributed services, for example, or they can be implemented as a single software or software module. No specific limitation is made here.

[0048] Server 105 can be a server that provides various services. For example, based on data update operations issued by users to the database via terminal devices 101, 102, and 103, pre-write log data is obtained, and the pre-write log data is synchronized to the background processing server of the target storage system by replacing the original synchronization function of the target node with a new synchronization function. As an example, server 105 can be a cloud server.

[0049] It should be noted that the server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., software or software modules for providing distributed services), or as a single software or software module. No specific limitations are given here.

[0050] It should also be noted that the data synchronization method provided in the embodiments of the present application is generally executed by a server, but the possibility of execution by a terminal device, or by a server and a terminal device cooperating with each other, is not excluded. Accordingly, the various parts (e.g., various units) included in the data synchronization device can be set entirely in the server, entirely in the terminal device, or separately in the server and the terminal device.

[0051] It should be understood that Figure 1The number of terminal devices, networks, and servers in the system architecture is merely illustrative. Any number of terminal devices, networks, and servers may be provided as needed. When the electronic device on which the data synchronization method is running does not need to transmit data to other electronic devices, the system architecture may only include the electronic device (e.g., server or terminal device) on which the data synchronization method is running.

[0052] Continue to refer Figure 2 , shows a process 200 of an embodiment of a data synchronization method, comprising the following steps:

[0053] Step 201: Determine whether the target node of the database includes pre-written log data to be synchronized.

[0054] In this embodiment, the execution subject of the data synchronization method (for example Figure 1 The terminal device or server in the database can determine whether the target node of the database includes pre-written log data to be synchronized.

[0055] The database may be a database involved in a synchronization process of pre-writing log data, such as MySQL, MariaDB, Percona Server for MySQL, and PostgreSQL.

[0056] Pre-write log data (WAL, Write-Ahead Logging) is a log mechanism widely used in database systems. Its core concept is to record data update operations in log files before actually updating data files. For databases, data update content is written to both data files (data files will have a large number of scattered writes) and log files (log files are append-only files with good write performance). When submitting a write transaction, it is only necessary to persist the log file (this is "pre-write"), and then choose an opportunity to asynchronously persist the data file. When submitting a write transaction, the reason why there is no need to perform persistence operations on the data file is because the data file can be repaired by replaying the log file. In this embodiment, the reason why only the pre-write log data needs to be synchronized and the data file does not need to be synchronized is also because the data update operation can be replayed based on the pre-write log data to obtain the data corresponding to the data file. For example, the pre-write log data is binlog (binary log) data.

[0057] A database generally includes multiple nodes, for example, a master node (master database) and multiple slave nodes (slave databases). The target node can be any node in the database.

[0058] Taking the MySQL system as an example, continue to refer to Figure 3, which shows a schematic diagram of the master-slave replication mechanism in the native master-slave mode of the database system.

[0059] The MySQL system can store data update operations in the form of a binlog (e.g., mysql-bin.000001, mysql-bin.000002, mysql-bin.000003, with log file suffixes increasing in chronological order of log generation). By sequentially parsing and replaying the binlog data, replication of the data source (i.e., the MySQL instance that generates the binlog) can be achieved. In a MySQL data synchronization solution, the system used for binlog dumping is called a "Publish / Subscribe" system, or a CDC (Change Data Capture) system. First, the binlogs generated by the MySQL instance (i.e., the data producer) can be collected and temporarily stored in this dump system. These log files can then be pushed to different data consumers or subscribed to by different consumers. This dump system allows MySQL instances to synchronize data with a variety of heterogeneous data consumers (e.g., relational databases such as PostgreSQL and Oracle; document databases such as ElasticSearch; and key-value databases such as TiKV).

[0060] The MySQL master node converts data updates into a binlog format and transmits this binlog data to slave nodes via the master / slave data replication protocol (i.e., the binlogdump command). In the slave nodes, the I / O (input / output) thread is responsible for network communication with the master node and dumps the binlog data to local disk files in the form of a "relay log." The SQL (Structured Query Language) thread in the slave node synchronizes data with the master node by reading and parsing the relay log and replaying the log content.

[0061] In native master-slave replication mode, slave nodes can be set to read-only mode. This is a separate process from the SQL thread's data synchronization process from the master to the slave. The read-only setting on a slave primarily affects external data writes to the slave, while data synchronization from the master to the slave is automatically performed by the MySQL replication mechanism and is unaffected by the slave's read-only status.

[0062] Depending on the data reliability, the synchronization process is categorized into asynchronous replication, semi-synchronous replication, and synchronous replication. In asynchronous replication, after the master node sends binlog data to the slave node, the write transaction can be committed without waiting for the slave node's relay log to arrive. In semi-synchronous replication, after the slave node's relay log arrives, it sends an ACK (acknowledgement) packet to the master node. The master node can only commit the write transaction after receiving the ACK. This ensures the consistency of write transactions between the master and slave nodes while eliminating the need for the slave node to fully agree with the master node before committing (this mechanism is called "synchronous replication"). This solution offers a balanced approach to performance and consistency.

[0063] Continue to refer Figure 4 , which shows a schematic diagram of the master-slave synchronization process of the database based on the group commit mechanism.

[0064] The MySQL database's binlog group commit mechanism can be considered a write performance optimization method. It batches together a group of pending transactions, rather than committing each transaction individually. Specifically, at a certain interval, transactions that enter a committable state store their updates in the binlog cache. The transaction with the lowest transaction number in the group is then selected as the commit leader, responsible for driving the commit of all transactions in the binlog cache. The data in the binlog cache is appended to the end of the master node's local binlog file and flushed to disk via a "fdatasync" operation. This data is then sent to slave nodes via MySQL's master-slave replication protocol. Receiving the slave's binlog dump reply (ack packet) confirms the commit of the group of transactions and wakes up all pending transactions in the group. The reason why fdatasync is used as the confirmation operation for log persistence is that to improve file write performance, the operating system does not directly persist data to disk in every write operation. Instead, it first writes the data to the system cache (called pagecache in Linux systems), and then uses the fdatasync operation to persist the binlogs in the system cache to disk in batches.

[0065] The above-mentioned group submission mechanism is actually an implementation process similar to "two-phase submission". Process ② (the master node persists the binlog data to a local file) and process ③ (the master node sends the binlogdump data packet to the slave node after receiving the binlogdump request from the slave node) are asynchronous and concurrent. When both process ② and process ④ (the slave node sends a confirmation packet corresponding to the data packet in process ③ to the master node) are completed, process ⑤ is executed to wake up all group members in the group submission and return a successful submission to the client. Therefore, the overall time taken for a group submission is the time taken for the longest execution process among process ② and process ③->④, that is:

[0066] Ttotal=max(T2,T3+T4)

[0067] Where Ttotal represents the total time taken to commit a group transaction, and T2, T3, and T4 represent the time taken for processes ②, ③, and ④, respectively. Generally, the total time taken for processes ③ and ④ is significantly greater than that for process ②. Therefore, the above formula can be approximated as:

[0068] Ttotal≈T3+T4

[0069] When there are multiple slave nodes, a binlogdump thread is created between the master and each slave node. The T3+T4 time is determined by the slave node with the slowest ACK response speed. Generally speaking, the more slave nodes there are, the worse the master-slave replication performance.

[0070] Continue to refer Figure 5 , shows a schematic diagram of the performance model under the one-master-three-slave (one master node, three slave nodes) mode.

[0071] MySQL has a parameter called rpl_semi_sync_master_wait_for_slave_count, which specifies the minimum number of ACK responses from slave nodes before a commit is confirmed. If there are three slave nodes and the expected number of ACK responses indicated by the parameter is at least two, a commit is confirmed after receiving two ACKs.

[0072] Continue to refer Figure 6 , which shows the data synchronization principle diagram of a third-party real-time data publishing tool.

[0073] Third-party real-time data publishing tools based on MySQL binlog typically implement this by simulating the master-slave replication protocol. Popular open-source implementations include Canal and Maxwell. For example, Canal's server simulates itself as a slave node in a MySQL system, using the binlogdump command to pull binlog data in real time and then push it to other databases or message queues (such as Kafka and RocketMQ).

[0074] Continue to refer Figure 7 , which shows a schematic diagram of the system architecture of a third-party real-time data publishing tool. Current binlog real-time publishing technologies require the addition of several new components to existing systems. For example, synchronizing binlog data from MySQL to a distributed storage system (such as HDFS) requires the introduction of three components: Canal, Kafka, and Flume. Alternatively, you can develop a custom Canal client adapter (a "Canal client") to connect directly to HDFS; however, this requires at least the introduction of the Canal server and Canal client components.

[0075] The MySQL database itself can ensure its own high availability through the master-slave replication mechanism. However, after the introduction of new components, the new components themselves need to have high availability mechanisms. Figure 8 , shows a schematic diagram of the high availability structure based on Canal. Taking Canal as an example, Canal high availability is guaranteed by Zookeeper (a distributed application coordination service, so this will introduce an additional component):

[0076] In step 1, Canal server A and Canal server B simultaneously initiate startup requests to the Zookeeper cluster; in step 2, the Zookeeper cluster selects one of the two Canal servers as the startup node (for example, A) and notifies it to start; in step 3, when Canal server A is unavailable, the Zookeeper cluster finds that A is offline and notifies Canal server B to start; in step 4, at this time, when a Canal client tries to connect, it first queries the Zookeeper cluster to obtain the host information of the current Canal server (for example, Canal server B); in step 5, the Canal client initiates a connection to the Canal server returned by Zookeeper.

[0077] As can be seen, the aforementioned MySQL binlog real-time publishing technology not only requires the introduction of additional components, but also requires the implementation of additional high-availability mechanisms for these additional components. This not only increases system complexity and reduces overall system availability (the more components, the lower the overall availability). Furthermore, the deployment of these additional components and their high-availability mechanisms requires additional hardware, increasing the overall cost of system deployment.

[0078] In this embodiment, the target node can be the master node in the database, and the master node is no longer provided with a slave node. This mode can be called a stand-alone mode, or a single master node mode, or a single master library mode. When the master node in the database serves as the data publishing end, the above-mentioned execution subject needs to determine whether the target node of the database includes pre-written log data to be synchronized. Specifically, the target storage system to be synchronized simulates the master-slave replication protocol, sends a binlogdump command to the master node, determines the synchronized data in the target storage system by parsing the binlogdump command, and then determines whether the master node includes pre-written log data to be synchronized other than the synchronized data based on the synchronized data.

[0079] Step 202 , in response to the determination, includes synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with the new synchronization function.

[0080] In this embodiment, in response to the determination, the execution entity synchronizes the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0081] The original synchronization function, such as the fsync or fdatasync function described above, is used to persist pre-written log data locally on the target node. In this embodiment, the original synchronization function can be modified to obtain a new synchronization function. In this new synchronization function, the pre-written log data is no longer persisted to the local disk of the target node. Instead, the pre-written log data that has not been written to disk is written to the target storage system. The target storage system can be any storage system with data storage capabilities.

[0082] The target storage system can be a distributed storage system and / or message queue in a third-party storage system. A distributed storage system is a storage method that stores data in a dispersed manner across multiple physical nodes and implements unified management and access through network connections. It has the advantages of high availability, high performance, and high scalability. Examples include HDFS (Hadoop Distributed File System), Ceph, and XDFS. A message queue is a cross-process communication mechanism used to asynchronously transfer data between different applications or different components of the same application. It allows producers and consumers to communicate in a decoupled manner, that is, producers do not need to know the specific implementation of consumers, nor do they need to wait for consumers to process data before sending the next piece of data. Examples include RabbitMQ, Kafka, and ActiveMQ.

[0083] As an example, the source code of the database system can be modified in an intrusive manner to modify the original synchronization function into a new synchronization function. As another example, the original synchronization function in the database system can be replaced with a new synchronization function in a non-intrusive manner.

[0084] Continue to refer Figure 9 , shows a schematic diagram of the architecture for data synchronization from the master node to the target storage system in stand-alone mode.

[0085] This embodiment can achieve the same performance and reliability as the native master-slave replication mechanism of the database system. Figure 9 In the example, assume there is a MySQL instance M (primary node) and a backup X of instance M; as long as all binlogs generated after the data point backed up by X (that is, data increments Δ) are backed up to the target storage system, the instance M can be restored to its latest state using backup X and incremental logs Δ. Moreover, by periodically backing up M (generating backups X1, X2, X3, etc.), binlogs can be continuously advanced and eliminated (that is, rotate operation: eliminating and deleting binlogs generated before backup Xn). In general, the target storage system provides high data reliability through a multi-copy mechanism (at least 3 copies); in the case of 3 copies, Figure 9 The reliability of the standalone mode is equivalent to the native one-master-three-slave model of the database system. Similar to the rpl_semi_sync_master_wait_for_slave_count mechanism in MySQL master-slave replication, a three-replica distributed storage only requires two replicas to reply with an ACK to confirm a write operation. Therefore, the performance of writing pre-written log data to distributed storage is similar to that of the one-master-three-slave MySQL replication protocol.

[0086] Continue to see Figure 10 , Figure 10 FIG1 is a schematic diagram 1000 of an application scenario of the data synchronization method according to this embodiment. Figure 10 In an application scenario, user 1001 issues a data update operation to the database server's master node 1003 via terminal device 1002. Master node 1003 uses a group commit mechanism to determine the pre-written log data to be synchronized. Then, by replacing the original synchronization function on the target node with a new synchronization function, the pre-written log data is synchronized to the target storage system 1004. The original synchronization function is used to persist the pre-written log data on target node 1003.

[0087] The method provided by the above-mentioned embodiment of the present application determines whether the target node of the database includes pre-written log data to be synchronized; in response to the determination, includes synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node, thereby replacing the original synchronization function with the new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system, and the synchronization process does not need to rely on other components, and is completely integrated with the native high-availability architecture of the database. On the basis of reducing the complexity of the system structure, the overall high availability and real-time data publishing capabilities of the system are guaranteed.

[0088] In some optional implementations of this embodiment, the target node is a slave node of the database. For example, in a one-master-one-slave mode of the database, the target node may be the only slave node; in another example, in a one-master-multiple-slave mode of the database, the target node may be any slave node among multiple slave nodes.

[0089] In this implementation, the above execution entity may also perform the following operations:

[0090] First, synchronize the original pre-written log data in the database's master node to the relay log of the target node.

[0091] It should be noted that when the target node is a slave node, the synchronization process between the master node and each slave node, as well as the data processing process within the master node still use the native information processing process of the database system, and only the synchronization scheme shown in the above embodiment 200 is used in the data synchronization process between the target node and the target storage system.

[0092] As an example, the synchronization process between the master node and the corresponding slave nodes, as well as the data processing process within the master node, can still use the above-mentioned group submission mechanism to process information.

[0093] Continue to refer to the above Figure 3 and 4 , for the original pre-written log data in the binlog cache of the database's master node, it is synchronized to the relay log Replay log of the target node through the binlog dump thread.

[0094] Second, the original pre-written log data in the relay log is replayed to obtain the pre-written log data generated during the replay process, and the pre-written log data is stored in the system cache of the operating system where the database is located.

[0095] In this implementation, the original pre-written log data in the relay log Replay log is replayed to obtain the pre-written log file, and the pre-written log file is stored in the system cache page cache of the operating system where the database is located.

[0096] Continue to refer Figure 11 , shows a schematic diagram of the architecture in master-slave mode. The synchronization mode in this implementation requires the master node to synchronize data with the slave node, and then the slave node synchronizes data with the target storage system, so it can be called master-slave mode, or cascade mode.

[0097] Compared to the stand-alone model, the advantages of the master-slave model include: the performance of the master node is the same as the native master-slave model of the database system, but higher than that of the stand-alone model; the master node only needs to synchronize with one slave node, and the slave node is responsible for parsing and uploading binlogs to the target storage system. Slave nodes generally do not receive queries and have a lower load, thus saving master node overhead and allowing the master database to handle more query loads; HA (high availability) performance is better, and the slave database can quickly take over if the master database fails, reducing business jitter (while the stand-alone model can only be restored using backups, which takes a long time).

[0098] The primary disadvantage of the master-slave model lies in real-time performance. In standalone mode, pre-written log data is guaranteed to be published to the target storage system by the time the master node confirms data submission. However, because the master-slave model uses slave nodes to upload pre-written log data in a cascade-like manner, the log data on the target storage system may lag behind the master node to a certain extent.

[0099] In master-slave mode, due to master-slave node switching, the publisher of the binlog may change frequently, unlike in standalone mode, where there is always a single publisher. Therefore, master-slave mode requires metadata to store the correspondence between the publisher and the start and end points of the data's GTID (Global Transaction Identifier). The publisher is represented by a Server-UUID (Universally Unique Identifier).

[0100] Continue to refer Figure 12 , which shows a schematic diagram of the data storage format under distributed object storage.

[0101] Each MySQL publisher corresponds to a publisher-id (a globally unique identifier, including UUID, can be used). Data objects are divided into two types: metadata and binlog data. Metadata is named publisher-id / meta / XXXXXX (XXXXXX is an auto-increment value starting at 1, fixed to a 6-digit decimal number, with the high-order bits padded with 0s). Binlog data objects are named publisher-id / XXXXXX / YYY (XXXXXX is an auto-increment value starting at 1, fixed to a 6-digit decimal number, with the high-order bits padded with 0s, indicating the suffix of the MySQL binlog file. YYY is an auto-increment value starting at 1, fixed to a 3-digit decimal number, with the high-order bits padded with 0s. When multiple binlog objects store a binlog file, this value indicates the object's position in the binlog file).

[0102] Each time the publisher is switched (i.e., the publisher's Server-uuid changes), one metadata object is added; the metadata object with the largest number (i.e., XXXXXX value) represents the current state.

[0103] The metadata object contains the following three fields:

[0104] Previous-GTIDs: The range of GTIDs generated by the previous publisher;

[0105] Previous-Binlogs: The range of binlog file suffixes generated by the previous publisher (that is, the XXXXXX value in Publisher-id / XXXXXX / YYY);

[0106] Current-Master: Server-uuid of the current publisher.

[0107] by Figure 12 Taking the data shown as an example, metadata object No. 1 indicates that the first publisher is ServerA; metadata object No. 2 indicates that ServerA published data with GTIDs from 1 to 140, generated two binlog files (corresponding to the three data objects Publisher-id / 000001 / 001, Publisher-id / 000001 / 002, and Publisher-id / 000002 / 001), and the current publisher is ServerB; metadata object No. 3 indicates that ServerB did not publish any updates, and the current publisher switched to ServerC; metadata object No. 4 indicates that ServerC published data with GTIDs from 141 to 630, generated a binlog file numbered 3, and the current publisher switched to ServerD. Figure 12 In this state, the next binlog object to be uploaded to the distributed storage is named Publisher-id / 000004 / 001.

[0108] In some implementations of this embodiment, the execution entity may execute step 202 as follows: in response to determining that the target node calls the original synchronization function, the new synchronization function is called through the function hook corresponding to the original synchronization function to synchronize the pre-written log data to the target storage system.

[0109] During the startup of the application program of the target node, the dynamic link library including the new synchronization function is loaded.

[0110] In this implementation, dynamic library injection and function hooking technology are used to ensure that no intrusion into the source code of the database system is required. Figure 13 , which shows a schematic diagram of the replacement principle under the Linux system.

[0111] In the Linux system, functions declared with the "weak symbol" attribute in the application can be overridden by the "strong symbol" of the same name; in addition, during the application startup phase, a dynamic link library can be pre-loaded into the program through the "LD_PRELOAD" mechanism. The C runtime library (glibc) in the Linux system generally declares library functions (such as malloc / free) in the form of weak symbols; if a strong symbol with the same name is implemented in a dynamic library, and this dynamic link library is loaded through "LD_PRELOAD" when the application starts, the weak symbol function of the glibc native runtime library can be replaced with the strong symbol function implemented in the dynamic library. For example, using the jemalloc library to replace the Linux native memory allocator (that is, replacing functions such as malloc / free), it can be started as follows:

[0112] [Linux]LD_PRELOAD= / your_library_dir / jemalloc.so mysqld--defaults-file=my.cnf&

[0113] In this implementation, the above technology is used to implement a dynamic link library, which is loaded when MySQL is started through the LD_PRELOAD method. This can overwrite the Linux native file read and write library functions (such as fsync, fdatasync, write, pwrite, etc.), realize the function replacement in a non-invasive way, and improve the flexibility of the implementation process.

[0114] Continue to refer Figure 9 In stand-alone mode, when there is pre-written log data to be synchronized in the master node, in response to determining that the master node calls the original synchronization function, the new synchronization function is called through the function hook corresponding to the original synchronization function to synchronize the pre-written log data to the target storage system (such as a distributed object storage system).

[0115] Continue to refer Figure 11 In the master-slave mode, the data synchronization process between the master node and the slave node adopts the above-mentioned Figure 3 、 4 When it is determined that there is pre-written log data to be synchronized from the slave node, the slave node calls the original synchronization function in response to the determination, and calls the new synchronization function through the function hook corresponding to the original synchronization function to synchronize the pre-written log data to the target storage system.

[0116] In some implementations of this embodiment, the execution subject may further perform the following operation: asynchronously persisting the pre-written log data to the target node.

[0117] In this implementation, it is not required to execute the persistence process of the pre-written log data on the target node in real time.

[0118] As an example, in response to a preset persistence condition being met, the pre-written log data is asynchronously persisted to the target node. The preset persistence condition may be, for example, that the amount of pre-written log data in the system cache reaches a preset data volume threshold, or that the time since the last persistence process was executed exceeds a preset time threshold. The preset data volume threshold and the preset time threshold can be set based on actual circumstances and are not limited here.

[0119] In this implementation, the pre-written log data is asynchronously persisted to the target node, which reduces the data processing pressure of the target node while ensuring the data storage results.

[0120] In some implementations of this embodiment, the execution subject may further perform the following operation: in response to the target node being restarted, synchronizing data on the target node with reference to log data in the target storage system.

[0121] Synchronizing the target node's data with reference to the log data in the target storage system is a recover operation. The reason for the recover operation is that the pre-written log data in the target node is not persisted in real time, but asynchronously persisted. Therefore, when the target node restarts, there may be pre-written log data in the target node that has not yet been asynchronously persisted. Since the data synchronization process to the target storage system is real-time, the data in the target storage system is comprehensive and reliable. At this time, it is necessary to synchronize the target node's data with reference to the log data in the target storage system so that the log data in the target node and the target storage system are consistent.

[0122] As an example, the execution subject may compare the persistent log data in the target node with the persistent log data in the target storage system, and synchronize the log data that exists in the target storage system but not in the target node to the target node.

[0123] In this implementation, when the target node is restarted, data synchronization is performed on the target node with reference to the log data in the target storage system, thereby ensuring data consistency between the target node and the target storage system, as well as accuracy and reliability of the data in the target node.

[0124] In some optional implementations of this embodiment, the execution entity may perform the data recovery process in the following manner:

[0125] First, determine the maximum value of the suffix value of the log file name in the target storage system and the target suffix value of the log file file name where the data site in the target node is located. The suffix value of the log file file name is positively correlated with the file generation time.

[0126] Second, log files in the target storage system whose file name suffix values ​​are between the target suffix value and the maximum value are synchronized to the target node.

[0127] As an example, determine the maximum value among the suffix values ​​of the file names of the log files in the target storage system, determine whether the data site of the local log file in the target node is located in the log file corresponding to the maximum value, and in response to not existing, subtract one from the maximum value, and determine whether the data site of the local log file in the target node is located in the log file corresponding to the subtracted maximum value. Iteratively execute the above process until the suffix value corresponding to the log file in the target storage system including the data site of the local log file in the target node is determined and determined as the target suffix value.

[0128] This implementation provides a specific data recovery method, which improves the information processing efficiency and accuracy of the data recovery process.

[0129] In some implementations of this embodiment, the execution subject may perform the process of synchronizing the pre-written log data to the target storage system in the following manner:

[0130] First, it is determined whether the to-be-synchronized log file containing the pre-written log data is being synchronized to the target storage system for the first time.

[0131] Then, in response to the determination being yes, a synchronized log file in the target storage system is generated according to the suffix value of the file name of the log file to be synchronized.

[0132] In response to determining that the log file to be synchronized containing the pre-written log data is being synchronized to the target storage system for the first time, it means that there is no log file in the target storage system with the same suffix value as the file name of the log file to be synchronized; at this time, it is necessary to generate a log file in the target storage system with the same suffix value as the file name of the log file to be synchronized, that is, the synchronized log file.

[0133] Finally, object storage is used to synchronize the pre-written log data to the synchronized log file.

[0134] In this implementation, each storage object is sequentially stored in the synchronized log file, and the suffix value of the storage object increases sequentially.

[0135] In this implementation, in response to determining that the to-be-synchronized log file containing the pre-written log data is not the first to be synchronized to the target storage system, it means that there is a log file in the target storage system with the same suffix value as the file name of the to-be-synchronized log file, that is, the synchronized log file. Then, the object storage method can be used to synchronize the pre-written log data directly to the synchronized log file, thereby improving the data synchronization efficiency of the synchronization process. Figure 12 shown.

[0136] Continue to refer Figure 14 , shows a schematic diagram of the structure of the dynamic link library. The dynamic link library of this embodiment includes three modules:

[0137] File operation library function hook: used to override the C runtime library (glibc) library functions related to file operations in the target node (such as open / creat, fsync / fdatasync, etc.), thereby replacing the target node's native operations on log files.

[0138] Binlog thread: Used to perform initialization operations and the above-mentioned data recovery operations during the startup phase, and asynchronously persist pre-written log data to the target node during operation.

[0139] Third-party storage system client: used for interaction between the target node and the third-party storage system. Different client programs (generally provided by the third-party storage system vendor) can be used to adapt to different third-party storage systems (such as Amazon S3, Alibaba Cloud OSS, etc.).

[0140] For the creation (open / creat) hook in the function hook, when a new log file is created, the background binlog thread is notified to flush the previous log file (the file name suffix value is smaller than the suffix value of the current file) to the local binlog file.

[0141] For the above initialization operation, the binlog thread needs to first read the metadata object from the object storage during the startup phase. If the metadata object is not found, the suffix value of the log file is initialized to 0. Otherwise, the log file with the largest file name suffix value is read from the object storage, the largest suffix value is determined, and the data position of the current log file is determined.

[0142] Continue to refer Figure 15 , shows a schematic process 1500 of another embodiment of the data synchronization method according to the present application, including the following steps:

[0143] Step 1501: Determine whether the target node of the database includes pre-written log data to be synchronized.

[0144] Step 1502 , in response to determining that the target node calls the original synchronization function, a new synchronization function is called through a function hook corresponding to the original synchronization function to synchronize the pre-written log data to the distributed storage system and / or message queue in the third-party storage system.

[0145] During the startup of the application program of the target node, the dynamic link library including the new synchronization function is loaded.

[0146] Step 1503: Asynchronously persist the pre-written log data to the target node.

[0147] Step 1504 : In response to the target node being restarted, determine the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located.

[0148] The suffix value of the log file name is positively correlated with the file generation time.

[0149] Step 1505: Synchronize the log files in the target storage system whose file name suffix values ​​are between the target suffix value and the maximum value to the target node.

[0150] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiment, the process 1500 of the data synchronization method in this embodiment specifically illustrates the replacement process of the synchronization function, the asynchronous persistence process of the pre-written log file on the target node, and the data recovery process when the target node is restarted. The original synchronization function is replaced with a new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system. The synchronization process does not need to rely on other components and is completely integrated with the native high-availability architecture of the database. On the basis of reducing the complexity of the system structure, it ensures the overall high availability and real-time data publishing capabilities of the system.

[0151] Continue to refer Figure 16 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a data synchronization device, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0152] like Figure 16As shown, a data synchronization device includes: a determination unit 1601, configured to determine whether the target node of the database includes pre-written log data to be synchronized; a synchronization unit 1602, configured to synchronize the pre-written log data to the target storage system in response to the determination that it includes, by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0153] In some implementations of this embodiment, the target node is a slave node of the database, and the above-mentioned device also includes: a relay unit (not shown in the figure), configured to synchronize the original pre-written log data in the master node of the database to the relay log of the target node; a replay unit (not shown in the figure), configured to replay the original pre-written log data in the relay log, obtain the pre-written log data generated during the replay process, and store the pre-written log data in the system cache of the operating system where the database is located.

[0154] In some implementations of this embodiment, the above-mentioned synchronization unit 1602 is further configured to: in response to determining that the target node calls the original synchronization function, call the new synchronization function through the function hook corresponding to the original synchronization function, and synchronize the pre-written log data to the target storage system, wherein, during the startup process of the application of the target node, the dynamic link library including the new synchronization function is loaded.

[0155] In some implementations of this embodiment, the apparatus further includes: an asynchronous persistence unit (not shown in the figure), configured to asynchronously persist the pre-written log data to the target node.

[0156] In some implementations of this embodiment, the above-mentioned device also includes: a restart recovery unit (not shown in the figure), which is configured to synchronize data on the target node with reference to log data in the target storage system in response to the target node being restarted.

[0157] In some implementations of this embodiment, the restart recovery unit is further configured to: determine the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located, wherein the suffix value of the file name of the log file is positively correlated with the file generation time; synchronize the log files in the target storage system whose file name suffix values ​​are between the target suffix value and the maximum value to the target node.

[0158] In some implementations of this embodiment, the synchronization unit 1602 is further configured to: determine whether the log file to be synchronized containing the pre-written log data is synchronized to the target storage system for the first time; in response to determining yes, generate a synchronized log file in the target storage system based on the suffix value of the file name of the log file to be synchronized; and synchronize the pre-written log data to the synchronized log file using an object storage method.

[0159] In some implementations of this embodiment, the target storage system includes a distributed storage system and / or a message queue in a third-party storage system.

[0160] In this embodiment, the determination unit in the data synchronization device determines whether the target node of the database includes pre-written log data to be synchronized; the synchronization unit responds to the determination by synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node, thereby replacing the original synchronization function with the new synchronization function, so that the local persistence process of the target node for the pre-written log data is replaced by the synchronization process of the pre-written log data to the target storage system, and the synchronization process does not need to rely on other components, and is completely integrated with the native high-availability architecture of the database. On the basis of reducing the complexity of the system structure, the overall high availability and real-time data publishing capabilities of the system are guaranteed.

[0161] Reference below Figure 17 , which shows a device suitable for implementing the embodiments of the present application (eg Figure 1 Schematic diagram of the structure of the computer system 1700 of the devices 101, 102, 103, 105 shown. Figure 17 The device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0162] like Figure 17 As shown, computer system 1700 includes a processor (e.g., CPU, central processing unit) 1701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1702 or a program loaded from a storage unit 1708 into a random access memory (RAM) 1703. Various programs and data required for the operation of system 1700 are also stored in RAM 1703. Processor 1701, ROM 1702, and RAM 1703 are connected to each other via a bus 1704. An input / output (I / O) interface 1705 is also connected to bus 1704.

[0163] The following components are connected to the I / O interface 1705: an input section 1706 including a keyboard, a mouse, and the like; an output section 1707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1708 including a hard disk; and a communication section 1709 including a network interface card such as a LAN card or a modem. The communication section 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the I / O interface 1705 as needed. Removable media 1711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1710 as needed, so that computer programs read therefrom can be installed into the storage section 1708 as needed.

[0164] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1709 and / or installed from a removable medium 1711. When the computer program is executed by the processor 1701, the above-mentioned functions defined in the method of the present application are performed.

[0165] It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0166] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the client computer, partially on the client computer, as a stand-alone software package, partially on the client computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the client computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0168] The units involved in the embodiments described in the present application can be implemented by software or by hardware. The described units can also be set in a processor, for example, can be described as: a processor, including a determination unit and a synchronization unit. Among them, the names of these units do not constitute a limitation of the unit itself under certain circumstances. For example, the synchronization unit can also be described as "a unit that synchronizes pre-written log data to a target storage system in response to determination, including by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node."

[0169] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the computer device: determines whether the target node of the database includes pre-written log data to be synchronized; in response to the determination, synchronizes the pre-written log data to the target storage system by replacing the original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node.

[0170] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A data synchronization method, comprising: Determine whether the target node of the database includes pre-written log data to be synchronized; In response to the determination, the method includes synchronizing the pre-written log data to a target storage system by replacing an original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node; The method further comprises: Asynchronously persisting the pre-written log data to the target node; In response to the target node being restarted, determining a maximum value of suffix values ​​of file names of log files in the target storage system and a target suffix value of file names of log files where the data site in the target node is located, wherein the suffix value of the file name of the log file is positively correlated with the time when the file was generated; Synchronize the log files in the target storage system whose file names have suffix values ​​between the target suffix value and the maximum value to the target node.

2. The method according to claim 1, wherein The target node is a slave node of the database, and The method further comprises: Synchronize the original pre-written log data in the master node of the database to the relay log of the target node; The original pre-written log data in the relay log is replayed to obtain the pre-written log data generated during the replay process, and the pre-written log data is stored in a system cache of the operating system where the database is located.

3. The method according to claim 1 or 2, wherein: The step of synchronizing the pre-written log data to the target storage system by replacing the original synchronization function of the target node with the new synchronization function includes: In response to determining that the target node calls the original synchronization function, the new synchronization function is called through the function hook corresponding to the original synchronization function to synchronize the pre-written log data to the target storage system, wherein during the startup process of the application of the target node, the dynamic link library including the new synchronization function is loaded.

4. The method according to claim 1, wherein Synchronizing the pre-written log data to the target storage system includes: Determining whether the to-be-synchronized log file containing the pre-written log data is being synchronized to the target storage system for the first time; In response to the determination being yes, generating a synchronized log file in the target storage system according to the suffix value of the file name of the log file to be synchronized; The pre-written log data is synchronized to the synchronized log file using an object storage method.

5. The method according to claim 1, wherein The target storage system includes a distributed storage system and / or a message queue in a third-party storage system.

6. A data synchronization device, comprising: a determining unit configured to determine whether a target node of the database includes pre-written log data to be synchronized; a synchronization unit configured to, in response to the determination, synchronize the pre-written log data to a target storage system by replacing an original synchronization function of the target node with a new synchronization function, wherein the original synchronization function is used to persist the pre-written log data to the target node; an asynchronous persistence unit, configured to asynchronously persist the pre-written log data to the target node; The restart recovery unit is configured to: in response to the target node being restarted, determine the maximum value of the suffix value of the file name of the log file in the target storage system and the target suffix value of the file name of the log file where the data site in the target node is located, wherein the suffix value of the file name of the log file is positively correlated with the file generation time; synchronize the log files in the target storage system whose file name suffix value is between the target suffix value and the maximum value to the target node.

7. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

9. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images