Data synchronization method and device

Abstract

The invention discloses a data synchronization method and device. A specific embodiment of the method comprises the following steps: taking a target storage system as a virtual master node, and establishing a connection between a slave node and the virtual master node; determining synchronized pre-written log data in the slave node; and performing data synchronization between the virtual master node and the slave node according to the synchronized pre-written log data. According to the method, the native master-slave replication mechanism of the database is simulated, a data synchronization process from the target storage system as the virtual master node to the slave node can be realized without introducing a new component between the target storage system and the slave node, the method is perfectly fused with the native master-slave replication mechanism of the database, an additional high-availability mechanism does not need to be introduced, and the data synchronization efficiency is improved. The overall availability of the system is higher; the slave nodes can be set to be in a read-only mode, so that the data consistency between the slave nodes and the target storage system is ensured.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, specifically to the field of database and cloud service technology, and in particular to a data synchronization method, device, computer-readable medium, electronic device, and computer program product. Background Art

[0002] There are two main solutions for incremental synchronization (replication) of MySQL database: one is MySQL's native master-slave replication; the other is to introduce third-party publish-subscribe middleware (such as Canal, Maxwell, etc.).

[0003] Since the storage space on a single MySQL node is limited, it is necessary to regularly delete expired binlog (binary log) files through a recycling mechanism (i.e. "binlogrotate"). In this way, once the historical binlog files are recycled, it is impossible to delay replay based on the historical binlog files or replay from a specific historical position.

[0004] By introducing the publish-subscribe middleware, binlog is dumped to a third-party storage system. As long as the storage capacity of the third-party storage system can be continuously expanded horizontally, it can even cache all binlog files generated in the history of a MySQL instance. However, when the adapter on the subscriber side connects to the slave node of the MySQL database, it can only establish a normal client-server connection. In this way, the slave node cannot be set to read-only mode, and there is a risk of inconsistent data between the master and slave nodes. Summary of the invention

[0005] The embodiments of the present application provide a data synchronization method, device, computer-readable medium, electronic device, and computer program product.

[0006] In a first aspect, an embodiment of the present application provides a data synchronization method, comprising: taking the target storage system as a virtual master node, establishing a connection between a slave node and the virtual master node; determining the synchronized pre-written log data in the slave node; and synchronizing data between the virtual master node and the slave node based on the synchronized pre-written log data.

[0007] In some examples, the target storage system is used as a virtual master node to establish a connection between the slave node and the virtual master node, including: in response to the slave node calling the original connection function, calling the new connection function through the function hook corresponding to the original connection function, in response to the preset network parameters being consistent with the master node address of the virtual master node, establishing a virtual connection between the slave node and the virtual master node, wherein the original connection function is used to establish a connection between the slave node and the actual master node, and during the startup process of the application of the slave node, the dynamic link library including the new connection function is loaded.

[0008] In some examples, the above-mentioned establishment of a connection between the slave node and the virtual master node also includes: creating a client adapted to the virtual master node in the slave node according to the node type of the virtual master node; and establishing a real connection between the slave node and the virtual master node through the client.

[0009] In some examples, the above-mentioned determination of the synchronized pre-written log data in the slave node includes: in response to the slave node calling the original sending function, calling the new sending function through the function hook corresponding to the original sending function, and sending a dump request to the virtual master node, wherein the original sending function is used to send a dump request to the actual master node corresponding to the slave node, and during the startup process of the application of the slave node, the dynamic link library including the new sending function is loaded; parsing the dump request to determine the synchronized pre-written log data of the slave node relative to the virtual master node.

[0010] In some examples, the above-mentioned data synchronization between the virtual master node and the slave node based on the synchronized pre-written log data includes: in response to the slave node calling the original polling function, calling the new polling function through the function hook corresponding to the original polling function, and determining whether the virtual master node includes unsynchronized pre-written log data other than the synchronized pre-written log data, wherein the original polling function is used to determine whether the receiving cache of the network connection between the slave node and the actual master node has received the unsynchronized pre-written log data sent by the actual master node, and during the startup process of the application of the slave node, the dynamic link library including the new polling function is loaded; in response to determining that the unsynchronized pre-written log data is included and the slave node calls the original receiving function, calling the new receiving function through the function hook corresponding to the original receiving function, and synchronizing the unsynchronized pre-written log data to the slave node, wherein the original receiving function is used to receive data from the actual master node, and during the startup process of the application of the slave node, the dynamic link library including the new receiving function is loaded.

[0011] In some examples, the above-mentioned determination of whether the virtual master node includes unsynchronized pre-written log data other than synchronized pre-written log data includes: in response to determining that the virtual master node supports polling operations, determining whether the virtual master node includes unsynchronized pre-written log data through a polling interface of the virtual master node; in response to determining that the virtual master node does not support polling operations, determining whether the virtual master node includes unsynchronized pre-written log data based on the log file indicated by the latest metadata generated in the virtual master node.

[0012] In some examples, synchronizing the unsynchronized pre-written log data to the slave node includes: storing the unsynchronized pre-written log data in a receiving cache; and dumping the unsynchronized pre-written log data in the receiving cache as a relay log of the slave node.

[0013] In some examples, using the target storage system as a virtual master node includes: using a distributed storage system and / or a message queue in a third-party storage system as a virtual master node.

[0014] In the second aspect, an embodiment of the present application provides a data synchronization device, including: a connection unit, configured to use the target storage system as a virtual master node to establish a connection between the slave node and the virtual master node; a determination unit, configured to determine the synchronized pre-written log data in the slave node; and a synchronization unit, configured to synchronize data between the virtual master node and the slave node based on the synchronized pre-written log data.

[0015] In some examples, the above-mentioned connection unit is further configured to: in response to the slave node calling the original connection function, call the new connection function through the function hook corresponding to the original connection function, and in response to the preset network parameters being consistent with the master node address of the virtual master node, establish a virtual connection between the slave node and the virtual master node, wherein the original connection function is used to establish a connection between the slave node and the actual master node, and during the startup process of the application of the slave node, the dynamic link library including the new connection function is loaded.

[0016] In some examples, the connection unit is further configured to: create a client adapted to the virtual master node in the slave node according to the node type of the virtual master node; and establish a real connection between the slave node and the virtual master node through the client.

[0017] In some examples, the above-mentioned determination unit is further configured to: in response to the slave node calling the original sending function, call the new sending function through the function hook corresponding to the original sending function, and send a dump request to the virtual master node, wherein the original sending function is used to send a dump request to the actual master node corresponding to the slave node, and during the startup process of the application of the slave node, the dynamic link library including the new sending function is loaded; parse the dump request to determine the synchronized pre-written log data of the slave node relative to the virtual master node.

[0018] In some examples, the synchronization unit is further configured to: in response to a slave node calling an original polling function, call a new polling function through a function hook corresponding to the original polling function, and determine whether the virtual master node includes unsynchronized pre-written log data other than the synchronized pre-written log data, wherein the original polling function is used to determine whether the receiving cache of the network connection between the slave node and the actual master node has received the unsynchronized pre-written log data sent by the actual master node, and during the startup of the application of the slave node, a dynamic link library including the new polling function is loaded; in response to determining that unsynchronized pre-written log data is included and the slave node calls the original receiving function, call a new receiving function through a function hook corresponding to the original receiving function, and synchronize the unsynchronized pre-written log data to the slave node, wherein the original receiving function is used to receive data from the actual master node, and during the startup of the application of the slave node, a dynamic link library including the new receiving function is loaded.

[0019] In some examples, the synchronization unit is further configured to: in response to determining that the virtual master node supports polling operations, determine whether the virtual master node includes unsynchronized pre-written log data through a polling interface of the virtual master node; in response to determining that the virtual master node does not support polling operations, determine whether the virtual master node includes unsynchronized pre-written log data based on the log file indicated by the latest metadata generated in the virtual master node.

[0020] In some examples, the synchronization unit is further configured to: store the unsynchronized pre-written log data in the receiving buffer; and dump the unsynchronized pre-written log data in the receiving buffer as a relay log of the slave node.

[0021] In some examples, the connection unit is further configured to use a distributed storage system and / or a message queue in a third-party storage system as a virtual master node.

[0022] In a third aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0023] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0024] In a fifth aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0025] The data synchronization method and device provided in the embodiment of the present application establish a connection between a slave node and the virtual master node by taking the target storage system as a virtual master node; determine the synchronized pre-written log data in the slave node; and synchronize the data between the virtual master node and the slave node according to the synchronized pre-written log data, thereby simulating the native master-slave replication mechanism of the database, and realizing the data synchronization process from the target storage system as the virtual master node to the slave node, so that the slave node can be set to read-only mode, and on the basis of ensuring the data storage effect of the target storage system, the data consistency between the slave node and the target storage system is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0027] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

[0028] Figure 2 is a flow chart of an embodiment of a data synchronization method according to the present application;

[0029] Figure 3 This is a schematic diagram of the master-slave replication mechanism in the native master-slave mode of the database;

[0030] Figure 4 This is a schematic diagram of the master-slave synchronization process of the database based on the group submission mechanism;

[0031] Figure 5 It is a data synchronization schematic diagram of a third-party real-time data publishing tool;

[0032] Figure 6 is a schematic diagram of an application scenario of the data synchronization method according to this embodiment;

[0033] Figure 7 It is a schematic diagram of the passing process between the target storage system and the slave node;

[0034] Figure 8 This is a schematic diagram of the replacement principle under the Linux system;

[0035] Fig. 9 It is a functional module diagram of the data synchronization system;

[0036] Fig.10 It is a flow chart of the interaction between the target storage system and the slave nodes;

[0037] Fig.11 This is a flowchart of the connect hook function;

[0038] Fig.12 This is a flowchart of the send hook function;

[0039] Fig.13 This is a flowchart of the poll hook function;

[0040] Fig.14 This is a flowchart of the recv hook function

[0041] Fig.15 is a flow chart of another embodiment of the data synchronization method according to the present application;

[0042] Fig.16 is a structural diagram of an embodiment of a data synchronization device according to the present application;

[0043] Fig.17 A schematic diagram of the structure of a computer system suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0047] Figure 1 An exemplary architecture 100 to which the data synchronization method and apparatus of the present application can be applied is shown.

[0048] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The communication connection between the terminal devices 101, 102, 103 constitutes a topological network, and the network 104 is used to provide a medium for the communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0049] The terminal devices 101, 102, 103 can interact with the server 105 through the network 104 to receive or send data, etc. The terminal devices 101, 102, 103 can be hardware devices or software that support network connection for data interaction and data processing. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to smart phones, car computers, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0050] Server 105 may be a server that provides various services, for example, based on the data update operation issued by the user to the database through terminal devices 101, 102, 103, the master node of the database synchronizes the pre-written log data to a target storage system with a larger data capacity for backup, and uses the target storage system as a virtual master node to implement a background processing server for the data synchronization process between the virtual master node and the slave nodes corresponding to the virtual master node. As an example, server 105 may be a cloud server.

[0051] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0052] It should also be noted that the data synchronization method provided in the embodiments of the present application is generally executed by a server, but the possibility of being executed by a terminal device, or being executed by a server and a terminal device in cooperation with each other is not excluded. Accordingly, the various parts (such as various units) included in the data synchronization device can be all set in the server, or all set in the terminal device, or can be set in the server and the terminal device respectively.

[0053] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system is only illustrative. Any number of terminal devices, networks, and servers may be provided as required. When the electronic device on which the data synchronization method is running does not need to perform data transmission with other electronic devices, the system architecture may only include the electronic device (e.g., server or terminal device) on which the data synchronization method is running.

[0054] Continue to refer Figure 2 , shows a process 200 of an embodiment of a data synchronization method, comprising the following steps:

[0055] Step 201: Use the target storage system as a virtual master node and establish a connection between the slave node and the virtual master node.

[0056] In this embodiment, the execution subject of the data synchronization method (for example Figure 1 The terminal device or server in the storage system can use the target storage system as a virtual master node to establish a connection between the slave node and the virtual master node.

[0057] Taking the MySQL database system as an example, there are two main solutions for incremental synchronization (replication): one is MySQL's native master-slave replication; the other is to introduce third-party publish-subscribe middleware (such as Canal, Maxwell, etc.). Generally speaking, the subscriber end of MySQL (i.e., the slave database) is called "slave (slave node)" or "replica (subscriber end)".

[0058] Taking the MySQL database system as an example, continue to refer to Figure 3 , which shows a schematic diagram of the master-slave replication mechanism in the native master-slave mode of the database system.

[0059] The MySQL database system can store data update operations in the form of binlog (e.g., mysql-bin.000001, mysql-bin.000002, mysql-bin.000003, the suffix of the log file increases in the order of log generation time). By parsing and replaying the binlog data in sequence, the data source (i.e., the MySQL instance that generates the binlog) can be replicated. In a MySQL data synchronization solution, the system used for binlog dumping is called a "Publish / Subscribe" system, or a CDC (Change data capture) system. First, the binlog generated by the MySQL instance (i.e., the data producer producer) can be collected and temporarily stored in the above dump system; then, these log files can be pushed to different data consumers (consumers) or subscribed to by different consumers. Through the above dump system, the MySQL instance can be synchronized with multiple heterogeneous data consumers (e.g., relational databases such as PostgreSQL and Oracle; document databases such as ElasticSearch; KV databases such as TiKV).

[0060] The MySQL master node (Master) converts data updates into binlog format and transmits binlog data to the slave node through the data replication protocol between the master and slave nodes (i.e., binlogdump command). In the slave node, the IO (input and output) thread is responsible for network communication with the master node and dumps the binlog data content to the local disk file in the form of "Relay log". The SQL (Structured Query Language) thread in the slave node synchronizes data with the master node by reading and parsing the relay log and replaying the log content.

[0061] In native master-slave replication mode, slave nodes can be set to read-only mode. The read-only mode of the slave node and the data synchronization process from the master node to the slave node, which is responsible for the SQL thread, are two independent processes. The read-only setting of the slave node mainly affects the external data write operation to the slave node, while the data synchronization process from the master node to the slave node is automatically completed through the replication mechanism of the MySQL system and is not affected by the read-only status of the slave node.

[0062] According to the different data reliability, the synchronization process is divided into asynchronous replication, semi-synchronous replication and synchronous replication. Among them, in the asynchronous replication mode, after the master node sends binlog data to the slave node, it can complete the submission of the write transaction without waiting for the relay log of the slave node to land; in the semi-synchronous replication mode, after the relay log in the slave node lands, it sends an ack (acknowledgement) packet to the master node, and the master node can complete the submission of the write transaction only after receiving the ack. In this way, on the one hand, the consistency of the write transaction between the master and the slave can be guaranteed, and on the other hand, the slave node does not need to confirm the consistency with the master node before confirming the submission (this mechanism is called "synchronous replication"), which is a solution that takes into account both performance and consistency.

[0063] Continue to refer Figure 4 , which shows a schematic diagram of the master-slave synchronization process of the database based on the group submission mechanism.

[0064] The binlog group commit mechanism of MySQL database can be regarded as a way to optimize write performance, that is, to commit a group of transactions to be committed in batches instead of executing the commit action for each transaction separately. The specific implementation method is that within a certain time interval, the transactions that enter the committable state will store their updated contents in the binlog cache (binary log file cache); then the transaction with the smallest transaction number in this group of transactions is selected as the commit leader, which is responsible for driving the commit action of all transactions in the binlog cache. On the one hand, the data in the binlog cache will be appended to the end of the local binlog file of the master node and persisted to the disk through the "flush" (i.e., fdatasync) operation; on the other hand, this part of the binlog will be sent to the slave node through the MySQL master-slave replication protocol. When the binlogdump reply (ack packet) from the slave node is received, the commit of this group of transactions can be confirmed and all waiting transactions in this commit group can be awakened. The reason why fdatasync is used as the confirmation operation for log persistence is that in order to improve file writing performance, the database system does not directly persist data to disk in each write operation. Instead, it writes data to the system cache (called page cache in Linux system) first, and then persists the binlog in the system cache to disk in batches through the fdatasync operation.

[0065] The above group submission mechanism is actually an implementation process similar to "two-stage submission". Process ② (the master node persists the binlog data to the local file) and process ③ (the master node sends the binlogdump data packet to the slave node after receiving the binlogdump request from the slave node) are asynchronous and concurrent. When both process ② and process ④ (the slave node sends the confirmation packet corresponding to the data packet in process ③ to the master node) are completed, process ⑤ is executed to wake up all group members in the group submission and return the submission success to the client. Therefore, the overall time consumption of a group submission is the time consumption of the longest execution time in process ② and process ③—>④, that is:

[0066] Ttotal=max(T2,T3+T4)

[0067] Among them, Ttotal represents the total time consumption of a group commit transaction, and T2, T3, and T4 represent the time consumption of processes ②, ③, and ④ respectively. In general, the overall time consumption of processes ③ and ④ is significantly greater than the time consumption of process ②. In this way, the above formula can be approximated as:

[0068] Ttotal≈T3+T4

[0069] When there are multiple slave nodes, a binlogdump thread is created between the master node and each slave node, and the T3+T4 time depends on the slave node with the slowest ack reply speed among all the slave nodes. Generally speaking, the more slave nodes there are, the worse the master-slave replication performance.

[0070] Continue to refer Figure 5 , showing the data synchronization principle diagram of a third-party real-time data publishing tool.

[0071] Third-party publish-subscribe middleware generally simulates the master-slave replication protocol of the MySQL database system (i.e., creates a virtual replica) and dumps the collected binlog data to a message queue or distributed storage. Taking Canal as an example, the subscriber needs to run a client program to pull the binlog data dumped by Canal from the message queue, parse it, and then replay it to the MySQL slave instance.

[0072] Canal server simulates the behavior of MySQL slave, pulls binlog data from the master node, and dumps it to Kafka (a mainstream open source message queue). On the subscription side, you need to run a client program that is responsible for subscribing to binlog from Kafka, parsing the binlog content, and replaying the update operations to the target database. To replay to the target database, the developer needs to provide an adapter program that is compatible with the target database in the client; that is, a program that can connect to the target database and perform corresponding update operations in the database. Canal is developed based on MySQL and comes with a client program that is compatible with the MySQL database system.

[0073] For the pre-written log data in the target storage system, if the third-party publish-subscribe middleware is introduced to synchronize it to the slave node, it is not only necessary to introduce new components (Canal client and adapter program) into the system, increasing the operation and maintenance burden; it also makes the slave node unable to run in "read-only" mode, and there is a risk of master-slave inconsistency.

[0074] In this embodiment, the target storage system may be a system with data storage function, and the target storage system is used as a virtual master node in the database, the purpose of which is to simulate the native master-slave replication protocol of the database system, so that the slave node can still be set to read-only mode, ensuring the data consistency between the target storage system and the slave node. The database may be a database involved in the synchronization process of pre-writing log data, such as MySQL, MariaDB, PerconaServer for MySQL, PostgreSQL and other databases.

[0075] Pre-write log data (WAL, Write-Ahead Logging) is a log mechanism widely used in database systems. Its core concept is to record data update operations in log files before actually updating data files. For databases, data update content is written to both data files (data files will have a large number of scattered writes) and log files (log files are a type of append only file with good write performance). When submitting a write transaction, only the log file needs to be persisted (this is "pre-write"), and then the data file is asynchronously persisted when the opportunity arises. When submitting a write transaction, the reason why it is not necessary to perform persistence operations on the data file is because the data file can be repaired by replaying the log file. In the present embodiment, the reason why only the pre-write log data needs to be synchronized and the data file does not need to be synchronized is because the data update operation can be replayed based on the pre-write log data to obtain the data corresponding to the data file. For example, the pre-write log data is binlog data.

[0076] The database generally includes multiple nodes, for example, a master node (master library) and multiple slave nodes (slave libraries). The target storage system in this embodiment can be used as a slave node of the master node, and obtain pre-written log data from the master node for data synchronization. As for the target storage system, it can also be used as a virtual master node to establish a virtual connection with the slave node corresponding to the target storage system. Among them, the slave node corresponding to the target storage system is generally different from the slave node corresponding to the master node.

[0077] In this embodiment, during the startup process, the slave node may use the target storage system as a virtual master node to simulate the handshake connection process with the actual master node and establish a connection between the two.

[0078] The target storage system can be any storage system with data storage function. The target storage system can be a distributed storage system and / or a message queue in a third-party storage system. A distributed storage system is a storage method that stores data in multiple physical nodes in a dispersed manner and realizes unified management and access through network connection. It has the advantages of high availability, high performance, high scalability, etc. Examples include HDFS (Hadoop Distributed File System), Ceph, XDFS, etc. A message queue is a cross-process communication mechanism used to asynchronously transfer data between different applications or different components of the same application. It allows producers and consumers to communicate in a decoupled manner, that is, producers do not need to know the specific implementation of consumers, nor do they need to wait for consumers to process data before sending the next piece of data. Examples include RabbitMQ, Kafka, ActiveMQ, etc.

[0079] In this implementation, the above-mentioned execution entity may use the distributed storage system and / or the message queue in the third-party storage system as a virtual master node.

[0080] Step 202: Determine the synchronized pre-written log data in the slave node.

[0081] In this embodiment, the execution subject determines the synchronized pre-written log data in the slave node.

[0082] As an example, the execution subject can be based on the above Figure 3 , 4 The data processing process shown determines the synchronized pre-written log data in the slave node. Specifically, the slave node sends a binlog dump request to the target storage system through a virtual connection, and the request includes the data location of the synchronized pre-written log data in the slave node. The simulated sending function injected into the database determines the synchronized pre-written log data in the slave node in the database by parsing the binlog dump request.

[0083] Step 203: synchronize data between the virtual master node and the slave node according to the synchronized pre-written log data.

[0084] In this implementation, the execution subject can synchronize data between the virtual master node and the slave node based on the synchronized pre-written log data.

[0085] As an example, the execution subject can be based on the above Figure 3 , 4The processes ③④ shown perform data synchronization between the virtual master node and the slave node. Specifically, after determining the synchronized pre-written log data, the simulated sending function injected into the database determines the unsynchronized pre-written log data other than the synchronized pre-written log data, and in the virtual receiving process, downloads the unsynchronized pre-written log data, encapsulates it into a master-slave replication protocol package, returns it to the caller (IO thread) of the receiving function, and determines that the synchronization operation is completed after receiving the ack response from the slave node.

[0086] Continue to see Figure 6 , Figure 6 FIG. 6 is a schematic diagram 600 of an application scenario of the data synchronization method according to this embodiment. Figure 6 In the application scenario, user 601 issues a data update operation to the master node 603 of the database server through the terminal device 602. The master node 603 uses the group submission mechanism to determine the pre-written log data to be synchronized, and synchronizes it to the target storage system 604 with a larger data storage capacity for backup. For the slave node (subscriber) 606 of the target storage system 604, the target storage system is used as a virtual master node, and a connection is established between the slave node and the virtual master node; the synchronized pre-written log data in the slave node is determined; and data synchronization between the virtual master node and the slave node is performed according to the synchronized pre-written log data.

[0087] The method provided by the above-mentioned embodiment of the present application establishes a connection between a slave node and a virtual master node by taking the target storage system as a virtual master node; determines the synchronized pre-written log data in the slave node; and synchronizes the data between the virtual master node and the slave node according to the synchronized pre-written log data, thereby simulating the native master-slave replication mechanism of the database. Without introducing new components between the target storage system and the slave node, the data synchronization process from the target storage system as a virtual master node to the slave node can be realized, which is perfectly integrated with the native master-slave replication mechanism of the database. Without introducing additional high-availability mechanisms, the overall availability of the system is higher. The slave node can be set to read-only mode to ensure data consistency between the slave node and the target storage system.

[0088] Continue to refer Figure 7 , which shows a schematic diagram of the interaction process between the target storage system and the slave node.

[0089] In some implementations of this embodiment, in the process of the slave node (i.e., the subscriber end), a virtual master node is simulated by modifying the system library functions related to network IO, and the data downloaded from the target storage system is encapsulated into a data packet of the MySQLbinlogdump command and passed to the binlog IO thread of the slave node. Among them, the system library functions are, for example, connect function, send function, poll function, recv function, etc.

[0090] The modification of system library functions uses dynamic library injection and function hook technology, which can ensure that there is no need to invade the source code of the MySQL database system. Similarly, the open source memory allocators jemalloc and tcmalloc also use similar technologies to implement the hook of malloc / free and other functions.

[0091] Continue to refer Figure 8 , showing a schematic diagram of the replacement principle under the Linux system. In the Linux system, functions declared with the "weak symbol" attribute in the application can be overridden by the "strong symbol" of the same name; in addition, during the application startup phase, a dynamic link library to_inject.so can be preloaded into the program through the "LD_PRELOAD" mechanism. The C runtime library (glibc) in the Linux system generally declares library functions in the form of weak symbols; if a strong symbol with the same name is implemented in a dynamic link library, and this dynamic library is loaded through "LD_PRELOAD" when the application starts, the native weak symbol function of glibc can be replaced with the strong symbol function implemented in the dynamic library.

[0092] Using the above technology, a dynamic link library is implemented and loaded when MySQL is started through the LD_PRELOAD method, which can cover the native Linux network IO related library functions (such as connect, poll, send, recv, etc.). Assuming that the dynamic link library is named "my_subscriber.so", the following command can be used to start MySQL to implement dynamic library injection and system function replacement:

[0093] [Linux]LD_PRELOAD= / your_library_dir / my_subscriber.so mysqld--defaults-file=my.cnf&

[0094] Continue to refer Fig. 9, showing a schematic diagram of the functional modules of the data synchronization system.

[0095] The data synchronization system includes the following functional modules:

[0096] Network IO related library function hook: used to override C runtime library (glibc) library functions related to file operations (such as connect, poll, send, recv, etc.), thereby simulating various behaviors related to the MySQL master node, such as connection, polling, sending, and receiving.

[0097] Virtual socket: The socket does not actually send or receive data on the network, but is only used to hook various operations on the master-slave replication connection.

[0098] Virtual master node information: used to maintain the virtual master node address, publishing end information, client context of the publishing system, binlog data cache, etc.

[0099] Client of the target storage system: used for data exchange between the target storage system and the slave nodes. Different client programs (generally provided by third-party vendors) can be used to adapt to different target storage systems (such as Amazon S3, Alibaba Cloud OSS, Kafka, etc.).

[0100] The main process of the IO thread of the native MySQL database system is: 1. Call the connect function from the node to establish a connection with the master node, and execute the connection handshake steps (that is, repeatedly send and receive command packets for connection handshake) according to the connection protocol of the MySQL system; 2. Call the send function to send a binlogdump request to the master node; 3. Call the poll function and recv function in a loop. If binlog data transmitted by the master node is found, the binlog data will be received and dumped to the relay log.

[0101] Continue to refer Fig.10 , shows the interaction flow chart between the target storage system and the slave node. In this implementation, a special network address can be set to identify the virtual master library (such as Fig.10 The address shown is: 1.1.1.1:1111). When starting the slave node through the "start slave" command of the MySQL system, the connect library function is called with the master node address of the master node as a parameter. In the connect hook function, as long as the address parameter is compared with the address of the virtual master node, it can be determined whether the current socket needs to be hooked.

[0102] On the slave node, the function hooks corresponding to each library function are called in the binlog IO thread.

[0103] like Fig.10 As shown in the figure, after using the hook method, the operation flow of the IO thread will be adjusted as follows (the connection handshake step is ignored here, because the virtual master node does not actually establish a connection with the slave node, and the response packet of the command packet can be simulated):

[0104] 1. Call the connect function to create a virtual master library connection:

[0105] Step 1: Compare the preset network parameters with the master node address of the virtual master node. If they are the same, it means connecting to the virtual master node;

[0106] Step 2: Connect to the publisher (third-party storage system, such as a distributed object storage system or a message queue);

[0107] Step 3: If the publisher connects successfully, the current socket is marked as the socket corresponding to the virtual master node.

[0108] 2. Call the send function to simulate sending a binlogdump request:

[0109] Step 4: The master node parses and caches the binlogdump request (including the binlog starting position and the GTID of the log that has been landed on the slave node).

[0110] 3. Call the poll function to wait for the incremental binlog data packet:

[0111] Step 5: Detect whether there is new binlog data on the publisher (the unsynchronized pre-written log data corresponding to the slave node).

[0112] 4. Call the recv function to receive new binlog data packets:

[0113] Step ⑥: Download the incremental binlog data from the publisher (if it is the first download, it needs to be merged with the contents of the binlogdump data package to retain only the data that has not yet been landed locally) and cache it.

[0114] Step 7: Copy data from the binlog buffer to the recv receive buffer.

[0115] It should be noted that in the above process, the poll function and the recv function are called cyclically until the slave node is stopped.

[0116] Continue to refer Fig.11 , showing a flow chart of the connect hook function (the function hook corresponding to the connect function).

[0117] Corresponding to the above steps ①-③, in some optional implementation methods of this embodiment, the above execution subject can execute the above step 201 in the following manner: in response to the slave node calling the original connection function, calling the new connection function through the function hook corresponding to the original connection function, and in response to the preset network parameters being consistent with the master node address of the virtual master node, establishing a virtual connection between the slave node and the virtual master node.

[0118] After the virtual connection is established, a mapping relationship between the socket in the virtual connection and the virtual master node is established.

[0119] The original connection function is the connect function in the system library function, which is used to establish a connection between the slave node and the actual master node. During the startup process of the application program of the slave node, the dynamic link library including the new connection function is loaded.

[0120] In this implementation, the process of replacing the original connection function with the new connection function adopts dynamic library injection and function hook technology, so that the connection function can be replaced without invading the source code of the database system, thereby improving the convenience and flexibility of the function replacement process.

[0121] In some optional implementations of this embodiment, the above-mentioned execution entity can also execute the virtual connection establishment process in the following manner: first, according to the node type of the virtual master node, a client adapted to the virtual master node is created in the slave node; then, through the client, a real connection between the slave node and the virtual master node is established.

[0122] In this implementation, different clients need to be established for different types of virtual master nodes. Taking the node type of the virtual master node as an example, a client adapted to the distributed storage system needs to be created in the slave node to establish a connection between the slave node and the distributed storage system through the client.

[0123] Taking the node type of the virtual master node as a message queue as an example, it is necessary to create a client adapted to the message queue in the slave node to establish a connection between the slave node and the message queue through the client.

[0124] It should be noted that two connections are established here: one is the physical connection between the client and the target storage system, and the other is the virtual master-slave synchronization connection between the virtual master node and the slave node. Two network sockets will be generated on the slave node respectively. During the data reception process, the slave node first reads the data through the physical connection between the client and the target storage system, and then converts the data format into the database native master-slave synchronization data packet format and returns it to the caller of the receiving function (IO thread). The IO thread does not know the existence of the virtual master node, and it thinks that the virtual connection is the actual master-slave replication connection.

[0125] In this implementation, a client adapted to the virtual master node is created according to the node type of the virtual master node, so as to establish a reliable and stable connection between the virtual master node and the slave node.

[0126] In this implementation, in response to the preset network parameters being consistent with the master node address of the virtual master node, the original connection function of the slave node is used for processing. During the creation of the virtual connection, if a connection failure occurs, an error code related to the virtual connection needs to be generated according to the error type to prompt relevant technicians to perform exception processing.

[0127] Continue to refer Fig.12 , shows a flow chart of the send hook function (the function hook corresponding to the send function).

[0128] Corresponding to the above step ④, in some optional implementations of this embodiment, the above execution subject may perform the above step 201 in the following manner:

[0129] First, in response to the slave node calling the original sending function, the new sending function is called through the function hook corresponding to the original sending function, and a dump request is sent to the virtual master node.

[0130] The original sending function is the send function in the system library function, which is used to send a dump request (binlog dump request) to the actual master node corresponding to the slave node. During the startup of the application of the slave node, the dynamic link library including the new sending function is loaded.

[0131] In this implementation, it is determined whether the currently connected socket is a socket that has a mapping relationship with the virtual master node; if so, in response to the slave node calling the original sending function, the new sending function is called through the function hook corresponding to the original sending function, and a dump request is sent to the virtual master node; if not, information processing is performed through the original connection function corresponding to the slave node; in response to determining that what is sent from the slave node to the virtual master node is not a dump request, it is necessary to simulate the connection handshake protocol so that the slave node and the virtual master node are in a virtual connection state.

[0132] Second, parse the dump request to determine the synchronized pre-written log data of the slave node relative to the virtual master node.

[0133] Specifically, the slave node may send a binlog dump request to the virtual master node through a virtual connection, and the virtual sending function parses the binlog dump request to determine the synchronized pre-written log data of the slave node relative to the virtual master node.

[0134] In this implementation, the process of replacing the original sending function with the new sending function adopts dynamic library injection and function hook technology, so that the sending function can be replaced without invading the source code of the database system, thereby improving the convenience and flexibility of the function replacement process.

[0135] Corresponding to the above steps ⑤-⑦, in some optional implementations of this embodiment, the above execution subject may perform the above step 203 in the following manner:

[0136] First, in response to the slave node calling the original polling function, the new polling function is called through the function hook corresponding to the original polling function to determine whether the virtual master node includes unsynchronized pre-written log data other than the synchronized pre-written log data.

[0137] Continued reference Fig.13 , shows a flow chart of the poll hook function (the function hook corresponding to the polling function).

[0138] Among them, the original polling function is the poll function in the system library function, which is used to determine whether the receiving cache of the network connection between the slave node and the actual master node has received the unsynchronized pre-written log data sent by the actual master node. During the startup process of the application of the slave node, the dynamic link library including the new polling function is loaded.

[0139] Second, in response to determining that unsynchronized pre-written log data is included and the original receiving function is called from the node, a new receiving function is called through a function hook corresponding to the original receiving function, and the unsynchronized pre-written log data is returned to the caller of the receiving function, that is, the binlog IO thread.

[0140] Continued reference Fig.14 , shows a flow chart of the recv hook function (the function hook corresponding to the receiving function).

[0141] The original receiving function is the recv function in the system library function, which is used to receive data from the actual master node. During the startup process of the application program of the slave node, the dynamic link library including the new receiving function is loaded.

[0142] In this implementation method, the process of replacing the original polling function with a new polling function and the process of replacing the original receiving function with a new receiving function adopt dynamic library injection and function hook technology, so that the replacement of polling function and receiving is realized without invading the source code of the database system, thereby improving the convenience and flexibility of the function replacement process.

[0143] Continue to refer Fig.13In some optional implementations of this embodiment, the execution subject may determine whether the virtual master node includes unsynchronized pre-written log data other than the synchronized pre-written log data in the following manner:

[0144] In response to determining that the virtual master node supports the polling operation, determining, through a polling interface of the virtual master node, whether the virtual master node includes unsynchronized pre-write log data.

[0145] In this manner, it is determined directly through a polling interface whether the virtual master node includes unsynchronized pre-written log data.

[0146] In response to determining that the virtual master node does not support the polling operation, it is determined whether the virtual master node includes unsynchronized pre-written log data according to the log file indicated by the metadata most recently generated in the virtual master node.

[0147] Metadata is used to store the correspondence between the data publisher and the start and end points of the data GTID (Global Transaction Identifier). The data publisher is represented by Server-UUID (Universally Unique Identifier).

[0148] In this implementation, different methods may be used in different situations to determine whether the virtual master node includes unsynchronized pre-written log data, thereby improving the flexibility of the data determination process.

[0149] Continue to refer Fig.14 In some optional implementations of the present embodiment, the above-mentioned execution subject can synchronize the unsynchronized pre-written log data to the slave node in the following manner: first, store the unsynchronized pre-written log data in the receiving cache; then, dump the unsynchronized pre-written log data in the receiving cache as the relay log of the slave node.

[0150] Specifically, determine whether the currently connected socket is a socket with a mapping relationship with the virtual master node; in response to determining yes, determine whether the virtual master node is in the handshake connection stage with the slave node; in response to determining yes, determine whether the slave node receives the binary log file of the virtual master node for the first time; in response to determining yes, download the binary log file from the virtual master node, and determine the data position indicated by the binlog dump request issued by the slave node to the virtual master node from the binary log file, and then remove the GTID included in the binlog dump from the binary log file, put the retained binary log file (unsynchronized pre-written log data) into the receiving cache of the virtual connection, and then dump it into a relay log. In the slave node, the binlog log data on the slave node will be generated based on the replay process of the relay log.

[0151] In response to determining that the currently connected socket is not a socket having a mapping relationship with the virtual master node, information processing is performed through the original receiving function corresponding to the slave node. In response to determining that the virtual master node and the slave node are in a handshake connection phase, a handshake connection protocol is simulated to complete the connection handshake process between the virtual master node and the slave node.

[0152] In response to determining that the receive buffer of the virtual connection is empty, data is directly downloaded from the target storage system and returned to the caller of the function hook of the original receive function (binlog IO thread); in response to determining that the receive buffer of the virtual connection is not empty, the data in the receive buffer of the virtual connection is returned to the caller of the function hook of the original receive function (binlog IO thread).

[0153] Continue to refer Fig.15 , shows a schematic process 1500 of another embodiment of the data synchronization method according to the present application, including the following steps:

[0154] Step 1501, in response to the slave node calling the original connection function, calling the new connection function through the function hook corresponding to the original connection function, and in response to the preset network parameters being consistent with the master node address of the virtual master node, establishing a virtual connection between the slave node and the virtual master node.

[0155] The original connection function is used to establish a connection between the slave node and the actual master node. During the startup of the application program of the slave node, a dynamic link library including the new connection function is loaded.

[0156] Step 1502, in response to the slave node calling the original sending function, calling the new sending function through the function hook corresponding to the original sending function, and sending a dump request to the virtual master node.

[0157] The original sending function is used to send a dump request to the actual master node corresponding to the slave node. During the startup of the application program of the slave node, the dynamic link library including the new sending function is loaded.

[0158] Step 1503: parse the dump request to determine the synchronized pre-written log data of the slave node relative to the virtual master node.

[0159] Step 1504, in response to the slave node calling the original polling function, calling the new polling function through the function hook corresponding to the original polling function, and determining whether the virtual master node includes unsynchronized pre-written log data other than the synchronized pre-written log data.

[0160] The original polling function is used to determine whether the receiving cache of the network connection between the slave node and the actual master node has received the unsynchronized pre-written log data sent by the actual master node. During the startup of the slave node application, the dynamic link library including the new polling function is loaded.

[0161] Step 1505 , in response to determining that the unsynchronized pre-written log data is included and the slave node calls the original receiving function, the new receiving function is called through the function hook corresponding to the original receiving function to synchronize the unsynchronized pre-written log data to the slave node.

[0162] The original receiving function is used to receive data from the actual master node, and during the startup process of the application program of the slave node, the dynamic link library including the new receiving function is loaded.

[0163] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiment, the process 1500 of the data synchronization method in this embodiment specifically illustrates the replacement process of the connection function, the sending function, the polling function and the receiving function, thereby simulating the native master-slave replication mechanism of the database, and realizing the data synchronization process of the target storage system as a virtual master node to the slave node, so that the slave node can be set to read-only mode, and on the basis of ensuring the data storage effect of the target storage system, the data consistency between the slave node and the target storage system is guaranteed.

[0164] Continue to refer Fig.16 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a data synchronization device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0165] like Fig.16As shown, a data synchronization device includes: a connection unit 1601, configured to use the target storage system as a virtual master node to establish a connection between the slave node and the virtual master node; a determination unit 1602, configured to determine the synchronized pre-written log data in the slave node; and a synchronization unit 1603, configured to perform data synchronization between the virtual master node and the slave node based on the synchronized pre-written log data.

[0166] In some examples, the connection unit 1601 is further configured to: in response to the slave node calling the original connection function, call the new connection function through the function hook corresponding to the original connection function, and in response to the preset network parameters being consistent with the master node address of the virtual master node, establish a virtual connection between the slave node and the virtual master node, wherein the original connection function is used to establish a connection between the slave node and the actual master node, and during the startup process of the application of the slave node, the dynamic link library including the new connection function is loaded.

[0167] In some examples, the connection unit 1601 is further configured to: create a client adapted to the virtual master node in the slave node according to the node type of the virtual master node; and establish a real connection between the slave node and the virtual master node through the client.

[0168] In some examples, the above-mentioned determination unit 1602 is further configured to: in response to the slave node calling the original sending function, call the new sending function through the function hook corresponding to the original sending function, and send a dump request to the virtual master node, wherein the original sending function is used to send a dump request to the actual master node corresponding to the slave node, and during the startup process of the application of the slave node, the dynamic link library including the new sending function is loaded; parse the dump request and determine the synchronized pre-written log data of the slave node relative to the virtual master node.

[0169] In some examples, the synchronization unit 1603 is further configured to: in response to the slave node calling the original polling function, call the new polling function through the function hook corresponding to the original polling function, and determine whether the virtual master node includes unsynchronized pre-written log data other than the synchronized pre-written log data, wherein the original polling function is used to determine whether the receiving cache of the network connection between the slave node and the actual master node has received the unsynchronized pre-written log data sent by the actual master node, and during the startup of the application of the slave node, a dynamic link library including the new polling function is loaded; in response to determining that the unsynchronized pre-written log data is included and the slave node calls the original receiving function, call the new receiving function through the function hook corresponding to the original receiving function, and synchronize the unsynchronized pre-written log data to the slave node, wherein the original receiving function is used to receive data from the actual master node, and during the startup of the application of the slave node, a dynamic link library including the new receiving function is loaded.

[0170] In some examples, the synchronization unit 1603 is further configured to: in response to determining that the virtual master node supports polling operations, determine whether the virtual master node includes unsynchronized pre-written log data through the polling interface of the virtual master node; in response to determining that the virtual master node does not support polling operations, determine whether the virtual master node includes unsynchronized pre-written log data based on the log file indicated by the latest metadata generated in the virtual master node.

[0171] In some examples, the synchronization unit 1603 is further configured to: store the unsynchronized pre-written log data in the receiving buffer; and dump the unsynchronized pre-written log data in the receiving buffer as a relay log of the slave node.

[0172] In some examples, the connection unit 1601 is further configured to use a distributed storage system and / or a message queue in a third-party storage system as a virtual master node.

[0173] In this embodiment, the connection unit in the data synchronization device takes the target storage system as a virtual master node and establishes a connection between the slave node and the virtual master node; the determination unit determines the synchronized pre-written log data in the slave node; the synchronization unit synchronizes the data between the virtual master node and the slave node according to the synchronized pre-written log data, thereby simulating the native master-slave replication mechanism of the database. Without introducing new components between the target storage system and the slave node, the data synchronization process of the target storage system as the virtual master node to the slave node can be realized, which is perfectly integrated with the native master-slave replication mechanism of the database. Without introducing additional high-availability mechanisms, the overall system availability is higher; the slave node can be set to read-only mode to ensure data consistency between the slave node and the target storage system.

[0174] Reference below Fig.17 , which shows a device suitable for implementing the embodiments of the present application (eg Figure 1 Schematic diagram of the structure of the computer system 1700 of the devices 101, 102, 103, 105 shown. Fig.17 The device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0175] like Fig.17As shown, the computer system 1700 includes a processor (e.g., CPU, central processing unit) 1701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1702 or a program loaded from a storage part 1708 into a random access memory (RAM) 1703. Various programs and data required for the operation of the system 1700 are also stored in the RAM 1703. The processor 1701, the ROM 1702, and the RAM 1703 are connected to each other via a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0176] The following components are connected to the I / O interface 1705: an input section 1706 including a keyboard, a mouse, etc.; an output section 1707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1708 including a hard disk, etc.; and a communication section 1709 including a network interface card such as a LAN card, a modem, etc. The communication section 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the I / O interface 1705 as needed. A removable medium 1711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1710 as needed, so that a computer program read therefrom is installed into the storage section 1708 as needed.

[0177] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1709, and / or installed from a removable medium 1711. When the computer program is executed by the processor 1701, the above-mentioned functions defined in the method of the present application are executed.

[0178] It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0179] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the client computer, partially on the client computer, as a separate software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the client computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0180] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0181] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The units described may also be set in a processor, for example, may be described as: a processor comprising a connection unit, a determination unit, and a synchronization unit. The names of these units do not constitute a limitation on the units themselves in certain circumstances, for example, the connection unit may also be described as "a unit that uses the target storage system as a virtual master node and establishes a connection between a slave node and a virtual master node".

[0182] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the computer device: uses the target storage system as a virtual master node, establishes a connection between the slave node and the virtual master node; determines the synchronized pre-written log data in the slave node; and synchronizes the data between the virtual master node and the slave node according to the synchronized pre-written log data.

[0183] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A data synchronization method, comprising: Using the target storage system as a virtual master node, establishing a connection between the slave node and the virtual master node; Determining the synchronized pre-written log data in the slave node; Data synchronization is performed between the virtual master node and the slave node according to the synchronized pre-written log data.

2. The method according to claim 1, wherein: The step of using the target storage system as a virtual master node and establishing a connection between the slave node and the virtual master node includes: In response to the slave node calling the original connection function, a new connection function is called through the function hook corresponding to the original connection function, and in response to the preset network parameters being consistent with the master node address of the virtual master node, a virtual connection between the slave node and the virtual master node is established, wherein the original connection function is used to establish a connection between the slave node and the actual master node, and during the startup process of the application of the slave node, a dynamic link library including the new connection function is loaded.

3. The method according to claim 1 or 2, wherein: The establishing of a connection between the slave node and the virtual master node further includes: According to the node type of the virtual master node, creating a client adapted to the virtual master node in the slave node; A real connection between the slave node and the virtual master node is established through the client.

4. The method according to claim 1, wherein: The determining the synchronized pre-written log data in the slave node includes: In response to the slave node calling the original sending function, calling the new sending function through the function hook corresponding to the original sending function, sending a dump request to the virtual master node, wherein the original sending function is used to send the dump request to the actual master node corresponding to the slave node, and during the startup process of the application program of the slave node, a dynamic link library including the new sending function is loaded; The dump request is parsed to determine the synchronized pre-written log data of the slave node relative to the virtual master node.

5. The method according to claim 1, wherein: The step of performing data synchronization between the virtual master node and the slave node according to the synchronized pre-written log data includes: In response to the slave node calling the original polling function, calling the new polling function through the function hook corresponding to the original polling function, determining whether the virtual master node includes unsynchronized pre-written log data other than the synchronized pre-written log data, wherein the original polling function is used to determine whether the receiving cache of the network connection between the slave node and the actual master node has received the unsynchronized pre-written log data sent by the actual master node, and during the startup process of the application of the slave node, the dynamic link library including the new polling function is loaded; In response to determining that the unsynchronized pre-written log data is included and the slave node calls the original receive function, a new receive function is called through a function hook corresponding to the original receive function to synchronize the unsynchronized pre-written log data to the slave node, wherein the original receive function is used to receive data from the actual master node, and during the startup process of the application of the slave node, a dynamic link library including the new receive function is loaded.

6. The method according to claim 5, wherein: The determining whether the virtual master node includes unsynchronized pre-written log data other than the synchronized pre-written log data comprises: In response to determining that the virtual master node supports a polling operation, determining, through a polling interface of the virtual master node, whether the virtual master node includes the unsynchronized pre-written log data; In response to determining that the virtual master node does not support the polling operation, determining whether the virtual master node includes the unsynchronized pre-written log data according to a log file indicated by the metadata most recently generated in the virtual master node.

7. The method according to claim 5, wherein: The step of synchronizing the unsynchronized pre-written log data to the slave node includes: Storing the unsynchronized pre-written log data in the receiving buffer; The unsynchronized pre-written log data in the receiving buffer is dumped as a relay log of the slave node.

8. The method according to claim 1, wherein: The step of using the target storage system as a virtual master node includes: The distributed storage system and / or the message queue in the third-party storage system is used as the virtual master node.

9. A data synchronization device, comprising: A connection unit is configured to use the target storage system as a virtual master node and establish a connection between the slave node and the virtual master node; a determining unit configured to determine the synchronized pre-written log data in the slave node; The synchronization unit is configured to perform data synchronization between the virtual master node and the slave node according to the synchronized pre-written log data.

10. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

12. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images