Data synchronization method and device

Abstract

An embodiment of the present application provides a data synchronization method and device. The data synchronization method includes: reading the current incremental file of the first database cluster through a first thread, the current incremental file is used to record the data write command executed by the first database cluster; parsing the current incremental file to obtain the data write command recorded by the current incremental file; sending a message containing the data write command to the designated topic of the message middleware, so that the second thread subscribed to the designated topic obtains the message from the designated topic; sending the message to the second database cluster through the second thread, so that the second database cluster executes and writes the data write command contained in the message to the incremental file of the second database cluster. The technical solution of the embodiment of the present application can realize fast data synchronization between database clusters.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular to a data synchronization method and device. Background Art

[0002] With the rapid development of informatization, the reliability of a single computer room cannot always be guaranteed. It is common for optical cables in computer rooms to be dug up and for power outages to occur. Application services and data often need to be deployed in multiple computer rooms, such as two computer rooms in the same city or three computer rooms in the same city and different locations, to prevent a computer room from failing and still provide reliable services to the outside world.

[0003] Cross-database synchronization, such as that of the relational database management system MySQL, can now be effectively achieved through the transmission of binary logs (binlogs). However, cross-database synchronization of key-value pairs (key-value databases) such as Redis still relies on manual operations, which has many drawbacks, such as low synchronization efficiency and long synchronization time. Summary of the Invention

[0004] The embodiments of the present application provide a data synchronization method and apparatus, thereby enabling rapid data synchronization between database clusters, at least to a certain extent.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0006] According to one aspect of an embodiment of the present application, a data synchronization method is provided, including: reading a current incremental file of a first database cluster through a first thread, the current incremental file being used to record a data write command executed by the first database cluster; parsing the current incremental file to obtain the data write command recorded in the current incremental file; sending a message containing the data write command to a designated topic of a message middleware, so that a second thread subscribed to the designated topic obtains the message from the designated topic; sending the message to a second database cluster through the second thread, so that the second database cluster executes the data write command contained in the message, and writes the executed data write command into the incremental file of the second database cluster.

[0007] According to one aspect of an embodiment of the present application, a data synchronization device is provided, including: a reading unit, configured to read a current incremental file of a first database cluster through a first thread, the current incremental file being used to record a data write command executed by the first database cluster; a first parsing unit, configured to parse the current incremental file to obtain the data write command recorded in the current incremental file; a first sending unit, configured to send a message containing the data write command to a designated topic of a message middleware, so that a second thread subscribed to the designated topic obtains the message from the designated topic; a second sending unit, configured to send the message to a second database cluster through the second thread, so that the second database cluster executes the data write command contained in the message, and writes the executed data write command into the incremental file of the second database cluster.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the device further includes: a checking unit, configured to check whether the first database cluster performs a rewrite operation on the current incremental file; and a first notification unit, configured to notify the first thread to stop reading the current incremental file if it is determined that the first database cluster performs a rewrite operation on the current incremental file.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the device further includes: an acquisition unit, configured to obtain the size of the incremental file after the first database cluster last performed a rewrite operation; a determination unit, configured to determine that the first database cluster performs a rewrite operation on the current incremental file if the difference between the size of the incremental file after the last rewrite operation and the size of the current incremental file exceeds a first specified threshold, and the size of the current incremental file reaches a second specified threshold.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the device also includes: a second notification unit, configured to notify the first thread to start reading a new incremental file in the first database cluster after the rewrite operation is completed, and the new incremental file is an incremental file used to replace the current incremental file after the rewrite operation is completed; a second parsing unit, configured to parse the new incremental file to obtain the data write command recorded in the new incremental file.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the device further includes: a first deletion unit, configured to delete the message after the rewrite operation is completed if there is a message containing the data write command in the designated topic of the message middleware; a third sending unit, configured to send a new incremental file of the first database cluster to the second database cluster, and the new incremental file is an incremental file used to replace the current incremental file after the rewrite operation is completed.

[0012] In some embodiments of the present application, based on the aforementioned scheme, before the third sending unit is configured to send the new incremental file of the first database cluster to the second database cluster, the device also includes: a second deletion unit, configured to delete the incremental file of the second database cluster and the snapshot file of the second database cluster, the snapshot file of the second database cluster is used to record the data corresponding to the second database cluster at a specified time point.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the device further includes: a judgment unit, configured to judge whether there is a message containing the data write command in the designated topic of the message middleware when it is necessary to switch from the first database cluster to the second database cluster; a first switching unit, configured to switch from the first database cluster to the second database cluster if there is no message containing the data write command in the designated topic of the message middleware; and a second switching unit, configured to switch from the first database cluster to the second database cluster after the second thread sends all the messages to the second database cluster if there is a message containing the data write command in the designated topic of the message middleware.

[0014] In some embodiments of the present application, based on the aforementioned solution, the first sending unit is configured to: send messages containing the data write command to the designated topic of the message middleware in sequence according to the time sequence of obtaining the data write command.

[0015] In some embodiments of the present application, based on the aforementioned solution, the second sending unit is configured to: send the messages to the second database cluster in sequence through the second thread in a first-in-first-out manner.

[0016] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data synchronization method as described in the above embodiment is implemented.

[0017] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the data synchronization method described in the above embodiments.

[0018] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data synchronization methods provided in the various optional embodiments described above.

[0019] In the technical solutions provided in some embodiments of the present application, the current incremental file of the first database cluster is read by a first thread. After reading the current incremental file, the current incremental file is parsed to obtain the data write command recorded in the current incremental file, and then a message containing the data write command is sent to the designated topic of the message middleware. The message is then sent to the second database cluster through a second thread, and the second database cluster executes and writes the data write command contained in the message, thereby completing the data synchronization process from the first database cluster to the second database cluster. The technical solution of the embodiment of the present application performs data synchronization by means of a message middleware. Compared with the method of manually copying incremental files, a fast and secure data synchronization technology is realized. When the first database cluster fails, the data transmission work of the message middleware will not stop, and the second database cluster can still obtain the data write command from the message middleware, so that data loss will not occur, thereby ensuring the security of the data.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0022] Figure 1A A schematic diagram showing an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied;

[0023] Figure 1BA schematic diagram of a blockchain system to which the technical solution of one embodiment of the present application can be applied is shown;

[0024] Figure 1C A schematic diagram of a blockchain to which an embodiment of the present application can be applied is shown;

[0025] Figure 1D A schematic diagram showing the generation of a new block in a blockchain to which an embodiment of the present application can be applied is shown;

[0026] Figure 2 A flow chart of a data synchronization method according to an embodiment of the present application is shown;

[0027] Figure 3 A flow chart of a data synchronization method according to an embodiment of the present application is shown;

[0028] Figure 4 A flow chart of a data synchronization method according to an embodiment of the present application is shown;

[0029] Figure 5 A flow chart of a data synchronization method according to an embodiment of the present application is shown;

[0030] Figure 6 A block diagram of a data synchronization device according to an embodiment of the present application is shown;

[0031] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0033] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0034] It should be noted that the terms used in the specification and claims of this application and the above-mentioned drawings are only used to describe the embodiments and are not intended to limit the scope of this application. It should be understood that the terms "comprise", "include", "have", etc., when used in this document, specify the presence of the stated features, wholes, steps, operations, elements, components and / or groups thereof, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0035] It will be further understood that, although the terms "first," "second," "third," etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the present invention, a first element may be referred to as a second element. Similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0038] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] Blockchain: It is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. It is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains information used to verify the validity of its information (anti-counterfeiting) and generate the next block.

[0041] In one embodiment of the present application, data synchronization can meet many application scenarios, including scenarios where data backup is performed to prevent data loss. It also includes scenarios where different read and write performances need to be provided to users for the same data. For example, in the database configured for data synchronization storage, there are both databases with high write performance to facilitate users to quickly enter data, and databases with high read performance to facilitate users to quickly find data. Or it also includes scenarios where data is synchronized between a widely promoted database and a newly developed database with higher performance. Specifically, as shown in Figure 1, a data storage system 100A, in an embodiment of the present application, the system is configured with a first database cluster 101A, a second database cluster 102A, and a message middleware 103A. An embodiment of the present application is described below based on Figure 1.

[0042] The first database cluster 101A may include multiple first nodes, which are interconnected to form the first database cluster 101A. The second database cluster 102A may include multiple second nodes, which are interconnected to form the second database cluster 102A. Both the first nodes and the second nodes may be key-value databases, such as Redis databases.

[0043] The Redis database is an in-memory database that stores data in memory and directly reads and processes data from memory. However, in-memory data is susceptible to power outages. If the machine fails or restarts, all data in memory will be lost. Therefore, ensuring the security of in-memory data is a challenge for in-memory databases. The Redis database provides two methods to ensure the security of in-memory data: snapshot files (Redis Data Base, RDB) and append-only files (AOF). The snapshot file method writes in-memory data to a snapshot file according to a specific strategy, while the append-only file method appends each write command to in-memory data as a log to an incremental file.

[0044] The message middleware 103A can be a distributed publish-subscribe system Kafka but is not limited to the distributed publish-subscribe system Kafka. The distributed publish-subscribe messaging system Kafka is a high-throughput distributed publish-subscribe messaging system. A topic is a component of the distributed publish-subscribe messaging system Kafka. It is a logical concept for storing messages, that is, a message collection. Each message sent to the distributed publish-subscribe messaging system Kafka has a topic. Messages corresponding to different topics are stored separately. Each topic can have multiple producers sending messages to it, and multiple consumers consuming the messages therein. Each topic can correspond to multiple partitions, and messages are stored in the partitions. In this embodiment, by configuring the parameter information of the topic, the same partition of the specified topic can be used to store messages containing data write commands.

[0045] The data storage system 100A of the embodiment of the present application is designed as a multi-threaded operating environment. If data synchronization is required between the first database cluster 101A and the second database cluster 102A of the data storage system, the first thread can read and parse the current incremental file of the first database cluster 101A. The current incremental file is used to record the data write commands executed by the first database cluster 101A. After the data write command is parsed, a message containing the data write command can be sent to the same partition of a designated topic of the message middleware.

[0046] Furthermore, the second thread subscribed to the specified topic can obtain messages from the specified topic, and then send the obtained messages to the second database cluster 102A, so that the second database cluster 102A executes and writes the data write command contained in the message, thereby completing the data synchronization processing from the first database cluster 101A to the second database cluster 102A.

[0047] In one embodiment of the present application, the first database cluster 101A or the second database cluster 102A may be a blockchain system, see Figure 1BThe blockchain system 100B shown is a system for data sharing between nodes. Each node 101B can receive input information during normal operation and maintain the shared data within the blockchain system 100B based on the received input information. To ensure information interoperability within the blockchain system 100B, information connections can exist between each node 101B in the blockchain system 100B, and information can be transmitted between nodes 101B through these information connections. For example, when any node 101B in the blockchain system 100B receives input information, the other nodes 101B in the blockchain system 100B obtain the input information according to the consensus algorithm and store the input information as data in the shared data, ensuring that the data stored on all nodes 101B in the blockchain system 100B is consistent.

[0048] Each node 101B in the blockchain system 100B has a corresponding node identifier, and each node 101B in the blockchain system 100B can store the node identifiers of other nodes 101B in the blockchain system 100B, so that the generated blocks can be broadcast to other nodes 101B in the blockchain system 100B based on the node identifiers of other nodes 101B. Each node 101B can maintain a node identifier list as shown in Table 1 below, and store the node name and node identifier in the node identifier list. The node identifier can be an IP (Internet Protocol, a protocol for interconnecting networks) address or any other information that can be used to identify the node. Table 1 only uses the IP address as an example for explanation.

[0049] Table 1

[0050] Node Name Node ID Node 1 117.114.151.174 Node 2 117.116.189.145 … … Node N 119.123.789.258

[0051] Figure 1C A schematic diagram of a blockchain to which an embodiment of the present application can be applied is shown.

[0052] Each node in the blockchain system 100B stores the same blockchain. The blockchain consists of multiple blocks, see Figure 1C The blockchain consists of multiple blocks. The genesis block includes a block header and a block body. The block header stores the input information feature value, version number, timestamp and difficulty value, and the block body stores the input information; the next block of the genesis block uses the genesis block as the parent block, and the next block also includes a block header and a block body. The block header stores the input information feature value of the current block, the block header feature value, version number, timestamp and difficulty value of the parent block, and so on, so that the block data stored in each block in the blockchain is associated with the block data stored in the parent block, ensuring the security of the input information in the block.

[0053] Figure 1D A schematic diagram of generating a new block in a blockchain to which an embodiment of the present application can be applied is shown.

[0054] When generating each block in the blockchain, see Figure 1D When the node where the blockchain is located receives the input information, it verifies the input information. After the verification is completed, the input information is stored in the memory pool and the hash tree used to record the input information is updated. After that, the update timestamp is updated to the time when the input information is received, and different random numbers are tried. The eigenvalue calculation is performed multiple times so that the calculated eigenvalue can satisfy the following formula:

[0055] SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x))<TARGET

[0056] Among them, SHA256 is the eigenvalue algorithm used to calculate the eigenvalue; version (version number) is the version information of the relevant block protocol in the blockchain; prev_hash is the block header eigenvalue of the parent block of the current block; merkle_root is the eigenvalue of the input information; ntime is the update time of the update timestamp; nCits is the current difficulty, which is a fixed value for a period of time and is determined again after exceeding the fixed time period; x is a random number; TARGET is the eigenvalue threshold, which can be determined based on nCits.

[0057] In this way, when a random number that satisfies the above formula is calculated, the information can be stored accordingly, and a block header and block body can be generated to obtain the current block. Subsequently, the node in the blockchain system 100B sends the newly generated block to other nodes 101B in the blockchain system 100B based on the node identifiers of other nodes 101B in the blockchain system 100B. The other nodes 101B verify the newly generated block and, after verification, add the newly generated block to their stored blockchain.

[0058] See also Figure 2 , is a flow chart of a data synchronization method according to an embodiment of the present application. The data synchronization method according to the embodiment of the present application can be applied to the data storage system 100A shown in FIG1 . Figure 2 As shown, the method includes:

[0059] Step S210: Read the current incremental file of the first database cluster through the first thread, where the current incremental file is used to record the data write command executed by the first database cluster;

[0060] Step S220: Parse the current incremental file to obtain a data write command recorded in the current incremental file;

[0061] Step S230: Send a message containing the data write command to a designated topic of the message middleware, so that a second thread subscribed to the designated topic obtains the message from the designated topic;

[0062] Step S240: Send the message to the second database cluster through the second thread, so that the second database cluster executes the data write command contained in the message and writes the executed data write command into the incremental file of the second database cluster.

[0063] These steps are described in detail below.

[0064] In step S210 , a current incremental file of a first database cluster is read through a first thread, where the current incremental file is used to record data write commands executed by the first database cluster.

[0065] The data synchronization method of this embodiment can be applied to the data storage system shown in Figure 1. The data storage system is designed as a multi-threaded operating environment. Multiple first nodes are interconnected to form a first database cluster. The first node can be a Redis database.

[0066] The Redis database is an open-source, in-memory database written in ANSI C. It supports networking, in-memory persistence, and is a log-based, key-value database. Redis stores data in memory and directly reads and processes data there. Because memory reads and writes data several orders of magnitude faster than disk, Redis significantly improves data processing efficiency by storing data in memory.

[0067] However, in-memory data is susceptible to power outages. If a machine fails or reboots, all in-memory data is lost. Therefore, the Redis database provides two modes: snapshot files (Redis Data Base, RDB) and append-only files (AOF) to ensure the security of in-memory data. The snapshot file mode writes in-memory data to a snapshot file according to a specific strategy, while the incremental file mode appends each write command to in-memory data as a log to an incremental file.

[0068] It should be noted that the incremental file mode is not enabled by default in the Redis database. In order to implement the data synchronization method of the technical solution of this embodiment, it is necessary to change the no in the configuration parameter "appendonlyno" in the configuration file of the Redis database to yes, that is, to "appendonly yes", thereby enabling the incremental file mode in the Redis database.

[0069] After the incremental file mode is enabled, the current incremental file of the first database cluster can be read through the first thread. The current incremental file records the data write command executed by the first database cluster.

[0070] It's also worth noting that there are three incremental file operation modes: continuous synchronization (always), every-second synchronization (everysec), and no active synchronization (no). The continuous synchronization strategy continuously and immediately synchronizes the data of Redis database operations to the incremental files on disk. This provides the highest data reliability, but has a significant impact on Redis database performance. The every-second synchronization strategy synchronizes the data of Redis database operations to the incremental files on disk every second. This method is more efficient and has higher data reliability, but the Redis database may lose nearly one second of data if it crashes. The no active synchronization strategy never actively synchronizes the data of Redis database operations to the incremental files on disk, resulting in poor reliability.

[0071] In this embodiment, the operation mode of the incremental file can be set to synchronize every 1 second (everysec) through a configuration parameter, that is, the configuration parameter "appendfsync everysec" is set.

[0072] In step S220, the current incremental file is parsed to obtain the data write command recorded in the current incremental file.

[0073] Since the current incremental file records the data write command executed by the first database cluster, after reading the current incremental file, the data write command can be obtained by parsing the current incremental file.

[0074] For example, if the format of the read incremental file is as follows, the first thread can parse the current incremental file line by line, and finally obtain the data write commands recorded in the current incremental file as select 0, setname jack, and set sex male.

[0075] *2# The current data write command has two parameters

[0076] $6#The first parameter length is 6 bytes, that is, select

[0077] select#The first parameter value of the current data write command

[0078] $1#The second parameter of the current data write command, the length is 1, that is, 0

[0079] 0#The second parameter value of the current data write command

[0080] *3# The current data write command has three parameters

[0081] $3#The first parameter length is 3 bytes, that is, set

[0082] set#The first parameter value of the current data write command

[0083] $4#The second parameter is 4 bytes long, namely name

[0084] name#The second parameter value of the current data write command

[0085] $4#The third parameter is 4 bytes long, i.e. jack

[0086] jack#The third parameter value of the current data write command

[0087] *3# The current data write command has three parameters

[0088] $3#The first parameter length is 3 bytes, that is, set

[0089] set#The first parameter value of the current data write command

[0090] $3#The first parameter length is 3 bytes, that is, sex

[0091] sex#The second parameter value of the current data write command

[0092] $4#The third parameter is 4 bytes long, that is, male

[0093] male#The third parameter value of the current data write command

[0094] In step S230, a message containing the data write command is sent to a designated topic of the message middleware, so that a second thread subscribed to the designated topic obtains the message from the designated topic.

[0095] In this embodiment, the message middleware can be a distributed publish-subscribe system Kafka but is not limited to the distributed publish-subscribe system Kafka. The distributed publish-subscribe messaging system Kafka is an open source stream processing platform developed by the Apache Software Foundation and written in Scala and Java. The distributed publish-subscribe messaging system Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on the website. These data are usually solved by processing logs and log aggregation due to throughput requirements. The purpose of the distributed publish-subscribe messaging system Kafka is to unify online and offline message processing through the parallel loading mechanism of the distributed system infrastructure, and to provide real-time messages through clusters.

[0096] A topic is a component of the Kafka distributed publish-subscribe messaging system. It is a logical concept for storing messages, essentially a collection of messages. Every message sent to the distributed publish-subscribe messaging system has a topic. Messages corresponding to different topics are stored separately. Each topic can have multiple producers sending messages to it and multiple consumers consuming its messages. Each topic can also have multiple partitions for storing its messages.

[0097] In this embodiment, by configuring the parameter information of the topic, the same partition of the specified topic can be used to store the message containing the data write command, so that the second thread subscribed to the specified topic can obtain the message from the specified topic.

[0098] In the data storage system of the embodiment of the present application, a message middleware mechanism is adopted, so different objects activate each other's events and complete corresponding operations by passing messages. The advantage is that the message middleware can communicate between different platforms, shield the characteristics between various platforms and protocols, achieve mutual collaboration, and transmit or store and forward messages at any time.

[0099] In one embodiment of the present application, since the data write commands recorded in the incremental file have a time sequence, when the first thread parses the incremental file line by line, the data write commands obtained also have a time sequence. Therefore, the first thread can send messages containing data write commands to the designated topic of the message middleware in sequence according to the time sequence of obtaining the data write commands, thereby ensuring the sequence of the messages.

[0100] In step S240 , the message is sent to the second database cluster through the second thread, so that the second database cluster executes the data write command contained in the message and writes the executed data write command into the incremental file of the second database cluster.

[0101] The second database cluster may include multiple second nodes, which are interconnected to form the second database cluster. The second nodes may be a key-value database, such as a Redis database.

[0102] Specifically, after obtaining the message from the designated topic of the message middleware, the second thread can further send the message to the second database cluster, so that the second database cluster executes and writes the data write command contained in the message, thereby completing the data synchronization processing from the first database cluster to the second database cluster.

[0103] In one embodiment of the present application, when the second thread sends messages to the second database cluster, it can send them in a first-in, first-out manner. When sending in a first-in, first-out manner, the message that enters the specified topic of the message middleware first will be obtained by the second thread first and sent to the second database cluster first. The message that enters the specified topic of the message middleware later will be obtained by the second thread later and sent to the second database cluster later.

[0104] The advantage of this embodiment is that even if the data write command of the first database cluster is eventually sent to the second database cluster, since the data write commands of the current incremental file record of the first database cluster are sent in sequence, it is beneficial for the second database cluster to execute the data write commands in the order of the data write commands, thereby completing data synchronization.

[0105] Based on the technical solution of the above embodiment, data synchronization is performed through the message middleware. Compared with the method of manually copying incremental files, a fast and reliable data synchronization technology is implemented. When the first database cluster fails, the data transmission work of the message middleware will not stop, and the second database cluster can still obtain the data write command from the message middleware, so that data loss will not occur, and the reliability of the data is guaranteed.

[0106] In one embodiment of the present application, since each data write command executed by the first database cluster is stored in an incremental file, each data write command is saved in the incremental file. Although this can well ensure the security of the data, the incremental file continuously accumulates data write commands. As time goes by, the incremental file will become larger and larger, occupying a large amount of storage space. Therefore, it is necessary for the first database cluster to perform a rewrite operation to compress the incremental file volume. The incremental file with a smaller file volume can be loaded faster by the first database cluster. In this embodiment, the rewrite operation of the first database cluster can also be checked, such as Figure 3 As shown, the process may specifically include steps S310 to S320, which are described in detail as follows:

[0107] Step S310: Check whether the first database cluster performs a rewrite operation on the current incremental file.

[0108] Specifically, the system may check whether the first database cluster performs a rewrite operation on the current incremental file through other threads other than the first thread and the second thread.

[0109] To facilitate understanding, the following describes the rewrite operation. When the first database cluster rewrites the current incremental file, it actually creates a child process through the parent process. The parent process executes the data write command and appends the data write command to the current incremental file, while the child process rewrites the current incremental file. While the child process is rewriting, the parent process continues to execute the data write command and appends the data write command to the incremental file buffer and the incremental file rewrite buffer. The contents of the incremental file buffer are synchronized with the current incremental file. The child process starts the rewrite operation and creates a new incremental file. The new incremental file includes the data write command merged according to the command merging rules before the first database cluster performs the rewrite operation. After the child process creates the new incremental file, it sends a signal to the parent process. After receiving the signal, the parent process calls a signal processing function. The function appends the data write command of the incremental file rewrite buffer to the new incremental file. After all the contents in the incremental file rewrite buffer are written to the new incremental file, the status of the current incremental file, the new incremental file and the first database cluster are completely consistent. Then the parent process renames the new incremental file, overwriting the current incremental file, completing the replacement of the current incremental file with the new incremental file, and the rewrite operation is completed.

[0110] In some embodiments, the condition for the parent process to create a child process to perform a rewrite operation may be: if the difference between the size of the incremental file after the last rewrite operation was performed on the first database cluster and the size of the current incremental file exceeds a first specified threshold, and the size of the current incremental file reaches a second specified threshold, then the parent process can create a child process to perform the rewrite operation on the current incremental file.

[0111] In this embodiment, the triggering condition for the rewrite operation requires not only that the difference between the size of the current incremental file and the size of the incremental file after the last rewrite operation is performed exceeds a first specified threshold, but also that the size of the current incremental file reaches a second specified threshold. The first specified threshold and the second specified threshold can be configured via configuration parameters in the configuration file of the first database cluster. For example, the first specified threshold is obtained by configuring the configuration parameter "auto-aof-rewrite-percentage", and the second specified threshold is obtained by configuring the configuration parameter "auto-aof-rewrite-min-size".

[0112] Step S320: If it is determined that the first database cluster performs a rewrite operation on the current incremental file, notify the first thread to stop reading the current incremental file.

[0113] If it is determined that the first database cluster performs a rewrite operation on the current incremental file, that is, the state of the current incremental file is no longer the same as that of the first database cluster, then data synchronization between the first database cluster and the second database cluster based on the current incremental file is inaccurate, and therefore the first thread does not need to read the current incremental file. Therefore, the first thread can be notified to stop reading the current incremental file.

[0114] In one embodiment of the present application, if the first database cluster performs a rewrite operation on the current incremental file, then the data synchronization between the first database cluster and the second database cluster can be performed after the rewrite operation is completed. Figure 4 As shown, the data synchronization method after the rewrite operation is completed may specifically include steps S410 to S420, which are described in detail as follows:

[0115] Step S410: After the rewrite operation is completed, notify the first thread to start reading a new incremental file in the first database cluster, where the new incremental file is an incremental file used to replace the current incremental file after the rewrite operation is completed.

[0116] It is easy to understand that after the rewrite operation of the current incremental file is completed, there will inevitably be a new incremental file in the first database cluster. The new incremental file is the incremental file used to replace the current incremental file. The new incremental file is used to record the data write command executed by the first database cluster. At the same time, the subsequent data write commands executed by the first database cluster will also be appended to the new incremental file. It can be seen that the new incremental file has the same status as the first database cluster. Therefore, in order to perform subsequent data synchronization between the first database cluster and the second database cluster, the first thread can be notified to start reading the new incremental file in the first database cluster.

[0117] Step S420: parse the new incremental file to obtain the data write command recorded in the new incremental file.

[0118] After reading the new incremental file, the first thread may parse the new incremental file to obtain the data write command recorded in the new incremental file.

[0119] In another embodiment of the present application, Figure 5 As shown, after the rewrite operation is completed, the data synchronization between the first database cluster and the second database cluster may include:

[0120] Step S510: After the rewrite operation is completed, if a message containing the data write command exists in the designated topic of the message middleware, the message is deleted;

[0121] Step S520: Send a new incremental file of the first database cluster to the second database cluster, where the new incremental file is an incremental file used to replace the current incremental file after the rewrite operation is completed.

[0122] In this embodiment, data synchronization between the first database cluster and the second database cluster is no longer performed by reading the new incremental file of the first database cluster through the first thread, but directly sending the new incremental file to the second database cluster. After receiving the new incremental file, the second database cluster can execute the data write command recorded in the new incremental file to complete the data writing, and append the data write command to the incremental file of the second database cluster, thereby completing the data synchronization between the first database cluster and the second database cluster.

[0123] It is worth mentioning that the new incremental file obtained after the rewrite operation is consistent with the status of the first database cluster. Therefore, the new incremental file can be directly sent to the second database cluster for data synchronization. At the same time, it should also be considered whether there is a message containing a data write command in the designated topic of the message middleware. This is because before the first database performs the rewrite operation to obtain a new incremental file, some data write commands in the current incremental file may have been sent to the designated topic in the message middleware. Compared with the new incremental file, the current incremental file is an "old file" and the data write command recorded in the current incremental file is an "old data write command". After sending the new incremental file to the second database cluster, it means that the latest status of the first database cluster has been synchronized to the second database cluster, so there is no need to continue sending the "old data write command" to the second database cluster. Therefore, if there is a message containing a data write command in the designated topic of the message middleware, the message can be deleted to avoid sending the "old data write command" in the message to the second database cluster.

[0124] In one embodiment of the present application, before sending the new incremental file of the first database cluster to the second database cluster, the incremental file of the second database cluster and the snapshot file of the second database cluster can also be deleted, so that the second database cluster only needs to execute and write the data write command of the new incremental file record of the first database cluster to complete data synchronization.

[0125] The incremental file of the second database cluster is a file that records the data write commands executed by the second database cluster, and the snapshot file of the second database cluster is a file that records the data corresponding to the second database cluster at a specified time point.

[0126] In one embodiment of the present application, if a failure occurs in the first database cluster and a switch to the second database cluster is necessary, the state of the first and second database clusters must be consistent to avoid data loss. This consistency means that messages sent from the first database cluster to a designated topic in the message middleware have also been sent to the second database cluster via the designated topic in the message middleware.

[0127] Therefore, in this embodiment, when it is necessary to switch from the first database cluster to the second database cluster, you can first determine whether there is a message containing a data write command in the designated topic of the message middleware. If there is no message containing a data write command in the designated topic of the message middleware, you can directly switch from the first database cluster to the second database cluster; on the contrary, if there is a message containing a data write command in the designated topic of the message middleware, you can switch from the first database cluster to the second database cluster after the second thread sends all the messages to the second database cluster.

[0128] The following describes an embodiment of the device of the present application, which can be used to execute the image processing method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the image processing method in the above embodiment of the present application.

[0129] Figure 6 A block diagram of an image processing apparatus according to an embodiment of the present application is shown. Figure 6 As shown, a data synchronization device 600 according to an embodiment of the present application includes: a reading unit 602 , a first parsing unit 604 , a first sending unit 606 and a second sending unit 608 .

[0130] Among them, the reading unit 602 is configured to read the current incremental file of the first database cluster through the first thread, and the current incremental file is used to record the data write command executed by the first database cluster; the first parsing unit 604 is configured to parse the current incremental file to obtain the data write command recorded in the current incremental file; the first sending unit 606 is configured to send a message containing the data write command to the designated topic of the message middleware, so that the second thread subscribed to the designated topic obtains the message from the designated topic; the second sending unit 608 is configured to send the message to the second database cluster through the second thread, so that the second database cluster executes the data write command contained in the message, and writes the executed data write command into the incremental file of the second database cluster.

[0131] In some embodiments of the present application, the device further includes: a checking unit configured to check whether the first database cluster performs a rewrite operation on the current incremental file; and a first notification unit configured to notify the first thread to stop reading the current incremental file if it is determined that the first database cluster performs a rewrite operation on the current incremental file.

[0132] In some embodiments of the present application, the device further includes: an acquisition unit configured to obtain the size of the incremental file after the first database cluster last performed a rewrite operation; a determination unit configured to determine that the first database cluster performs a rewrite operation on the current incremental file if the difference between the size of the incremental file after the last rewrite operation and the size of the current incremental file exceeds a first specified threshold, and the size of the current incremental file reaches a second specified threshold.

[0133] In some embodiments of the present application, the device also includes: a second notification unit, configured to notify the first thread to start reading a new incremental file in the first database cluster after the rewrite operation is completed, and the new incremental file is an incremental file used to replace the current incremental file after the rewrite operation is completed; a second parsing unit, configured to parse the new incremental file to obtain the data write command recorded in the new incremental file.

[0134] In some embodiments of the present application, the device also includes: a first deletion unit, configured to delete the message if there is a message containing the data write command in the designated topic of the message middleware after the rewrite operation is completed; a third sending unit, configured to send a new incremental file of the first database cluster to the second database cluster, and the new incremental file is an incremental file used to replace the current incremental file after the rewrite operation is completed.

[0135] In some embodiments of the present application, before the third sending unit is configured to send the new incremental file of the first database cluster to the second database cluster, the device further includes: a second deleting unit, configured to delete the incremental file of the second database cluster and the snapshot file of the second database cluster, the snapshot file of the second database cluster being used to record the data corresponding to the second database cluster at a specified time point.

[0136] In some embodiments of the present application, the device also includes: a judgment unit, configured to judge whether there is a message containing the data write command in the designated topic of the message middleware when it is necessary to switch from the first database cluster to the second database cluster; a first switching unit, configured to switch from the first database cluster to the second database cluster if there is no message containing the data write command in the designated topic of the message middleware; and a second switching unit, configured to switch from the first database cluster to the second database cluster after the second thread sends all the messages to the second database cluster if there is a message containing the data write command in the designated topic of the message middleware.

[0137] In some embodiments of the present application, the first sending unit 606 is configured to: send messages containing the data write command to the designated topic of the message middleware in sequence according to the time sequence of obtaining the data write command.

[0138] In some embodiments of the present application, the second sending unit 608 is configured to send the messages to the second database cluster in sequence through the second thread in a first-in-first-out manner.

[0139] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.

[0140] It should be noted that Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 into the random access memory (RAM) 703, such as the method described in the above embodiment. Various programs and data required for system operation are also stored in the RAM 703. The CPU 701, ROM 702 and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0142] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that a computer program read therefrom can be installed into the storage section 708 as needed.

[0143] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the various functions defined in the system of the present application are executed.

[0144] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0147] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device implements the method described in the above embodiments.

[0148] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0149] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0150] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0151] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: After the node of the first database cluster starts the incremental file mode, the current incremental file of the first database cluster is read through the first thread, where the current incremental file is used to record the data write command executed by the first database cluster; Parsing the current incremental file to obtain a data write command recorded in the current incremental file; Sending a message containing the data write command to a designated topic of the message middleware, so that a second thread subscribed to the designated topic obtains the message from the designated topic; Sending the message to a second database cluster through the second thread, so that the second database cluster executes the data write command contained in the message and writes the executed data write command into an incremental file of the second database cluster; the second database cluster and the first database cluster belong to different computer rooms; During data synchronization, if the first database cluster creates a child process through a parent process to perform a rewrite operation on the current incremental file, the first thread is notified to stop reading the current incremental file; wherein the parent process is used to execute the data write command, and the child process is used to perform the rewrite operation on the current incremental file and create a new incremental file, wherein the new incremental file includes the merged data write command in the current incremental file before the rewrite operation is performed and the data write command executed by the parent process during the rewrite operation; When it is necessary to switch from the first database cluster to the second database cluster, determining whether there is a message containing the data write command in the designated topic of the message middleware; If there is no message containing the data write command in the designated topic of the message middleware, switching from the first database cluster to the second database cluster; If there is a message containing the data write command in the designated topic of the message middleware, after the second thread sends all the messages to the second database cluster, the process switches from the first database cluster to the second database cluster.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining the size of the incremental file after the last rewrite operation was performed on the first database cluster; If the difference between the size of the incremental file after the last rewrite operation and the size of the current incremental file exceeds a first specified threshold, and the size of the current incremental file reaches a second specified threshold, it is determined that the first database cluster performs a rewrite operation on the current incremental file.

3. The method according to claim 1, characterized in that The method further comprises: After the rewrite operation is completed, notifying the first thread to start reading a new incremental file in the first database cluster, where the new incremental file is an incremental file that replaces the current incremental file after the rewrite operation is completed; The new incremental file is parsed to obtain a data write command recorded in the new incremental file.

4. The method according to claim 1, wherein The method further comprises: After the rewrite operation is completed, if a message containing the data write command exists in the designated topic of the message middleware, the message is deleted; Sending a new incremental file of the first database cluster to the second database cluster, wherein the new incremental file is an incremental file used to replace the current incremental file after the rewrite operation is completed.

5. The method according to claim 4, characterized in that Before sending the new incremental file of the first database cluster to the second database cluster, the method further includes: Delete the incremental files of the second database cluster and the snapshot files of the second database cluster, where the snapshot files of the second database cluster are used to record data corresponding to the second database cluster at a specified time point.

6. The method according to claim 1, characterized in that Sending a message containing the data write command to a designated topic of the message middleware includes: Messages containing the data write commands are sent to the designated topics of the message middleware in sequence according to the time sequence of obtaining the data write commands.

7. The method according to claim 1, characterized in that Sending the message to the second database cluster through the second thread includes: The messages are sent to the second database cluster in sequence through the second thread in a first-in-first-out manner.

8. A data synchronization device, characterized in that: The device comprises: a reading unit configured to read, through a first thread, a current incremental file of the first database cluster after an incremental file mode is enabled on a node of the first database cluster, the current incremental file being used to record a data write command executed by the first database cluster; A first parsing unit is configured to parse the current incremental file to obtain a data writing command recorded in the current incremental file; A first sending unit is configured to send a message containing the data write command to a designated topic of the message middleware, so that a second thread subscribed to the designated topic obtains the message from the designated topic; a second sending unit configured to send the message to a second database cluster through the second thread, so that the second database cluster executes the data write command contained in the message and writes the executed data write command into an incremental file of the second database cluster; the second database cluster and the first database cluster belong to different computer rooms; a checking unit configured to, during data synchronization, notify the first thread to stop reading the current incremental file if the first database cluster creates a child process through a parent process to perform a rewrite operation on the current incremental file; wherein the parent process is used to execute the data write command, and the child process is used to perform the rewrite operation on the current incremental file and create a new incremental file, wherein the new incremental file includes the merged data write command in the current incremental file before the rewrite operation is performed and the data write command executed by the parent process during the rewrite operation; a judging unit configured to, when switching from the first database cluster to the second database cluster is required, judge whether there is a message containing the data write command in the designated topic of the message middleware; a first switching unit, configured to switch from the first database cluster to the second database cluster if no message containing the data write command exists in the designated topic of the message middleware; The second switching unit is configured to switch from the first database cluster to the second database cluster after the second thread sends all the messages to the second database cluster if there is a message containing the data write command in the designated topic of the message middleware.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

11. A computer program product, characterized in that The computer program product includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads and executes the computer program from the computer-readable storage medium, so that the computer device performs the method according to any one of claims 1 to 7.

Images