Real-time data synchronization method and device, computer equipment and storage medium

Abstract

The invention relates to a real-time data synchronization method and device, computer equipment and a storage medium. The method comprises the following steps: determining a virtual node according to a target protocol, and linking the virtual node to a main node of a main machine room cluster; acquiring cluster data in real time from a main node of the main machine room cluster according to the virtual node, and writing the cluster data into a message queue; and controlling the target program to obtain cluster data from the message queue in real time, and synchronizing the cluster data to the standby machine room cluster. By the adoption of the method, Redis cluster isolation of the remote main machine room and the remote standby machine room can be achieved, real-time synchronization of cluster data between the remote main machine room and the remote standby machine room can be achieved across Redis clusters, the strong dependence of the standby machine room on the main machine room is reduced, conflict data caused by the split-brain problem is avoided, and then the reliability of real-time synchronization data is improved.

Description

Technical Field

[0001] The present application relates to the field of operation and maintenance technology, and in particular to a real-time data synchronization method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the continuous iteration and development of business systems, especially when the importance and scale of enterprises reach a certain level, the requirements of business systems for data are getting higher and higher. In order to deal with computer room failures (such as power outages, natural disasters, etc.), enterprises usually need to build off-site disaster recovery. The core goal of off-site disaster recovery construction is to ensure that in the event of a failure in the main computer room, the backup computer room can take over the business and maintain data consistency and availability. That is, off-site disaster recovery construction is to back up and redundantly deploy data and services in computer rooms at different locations to ensure that when one computer room fails, another computer room can continue to provide services. For business systems, Redis (open source memory data storage system) cluster plays a very important role as a high-performance cache system. Redis cluster is used to store hot data (such as user information, session data, etc.). If the Redis cluster data is lost or cannot be restored in time, it will lead to a decline in user experience and even affect the normal operation of the system. Therefore, when building off-site disaster recovery, it is a very important technical issue to ensure that the Redis cluster cache data can be synchronized in real time between different computer rooms.

[0003] In the traditional Redis cluster master-slave replication mode, a master node in the Redis cluster is responsible for processing all write requests, and one or more slave nodes corresponding to the master node are used to obtain data from the master node and synchronize. The master node transmits the write operation log (AOF) or data snapshot (RDB) to one or more slave nodes corresponding to it to achieve real-time data synchronization.

[0004] However, in the traditional Redis cluster master-slave replication mode, the master node and slave nodes of the Redis cluster are deployed across computer rooms, that is, the master node of the Redis cluster is deployed in the main computer room, and one or more slave nodes corresponding to the master node are deployed in the backup computer room, which makes the backup computer room highly dependent on the main computer room, resulting in the following risks: In the Redis sentinel mode, sentinel nodes are deployed in both the main computer room and the backup computer room. The sentinel nodes are responsible for monitoring the health of the master node and slave nodes of the Redis cluster and helping the Redis cluster to elect. When the dedicated line between the main computer room and the backup computer room fails, the sentinel nodes in the main computer room and the backup computer room cannot communicate with each other, and thus cannot reach a consensus, resulting in the sentinel nodes in the main computer room and the backup computer room each believing that the cluster node they are in is the master node of the Redis cluster, resulting in a "split brain" problem. During the "split brain" period, the nodes in the main computer room and the backup computer room, that is, the original master node and the original slave node in the Redis cluster, will start to process write operations. As a result, in the process of synchronizing data, write operations cannot be synchronized, resulting in the generation of conflicting data, which in turn affects the reliability of real-time synchronized data. Summary of the invention

[0005] Based on this, it is necessary to provide a real-time synchronization data method, device, computer equipment, computer-readable storage medium and computer program product that can reduce the strong dependence of the backup computer room on the main computer room, avoid the generation of conflicting data caused by the "brain split" problem, and thereby improve the reliability of real-time synchronization data in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a real-time data synchronization method, comprising:

[0007] Determine the virtual node according to the target protocol and link the virtual node to the master node of the main computer room cluster;

[0008] According to the virtual node, cluster data is obtained from the master node of the main computer room cluster in real time, and the cluster data is written into the message queue;

[0009] The control target program obtains cluster data from the message queue in real time and synchronizes the cluster data to the standby computer room cluster.

[0010] In one embodiment, determining a virtual node according to a target protocol and linking the virtual node to a master node of a computer room cluster includes:

[0011] Determine virtual nodes based on pre-built applications and target protocols;

[0012] Connect the virtual node to the master node of the computer room cluster, and control the virtual node to interact with the master node of the computer room cluster based on the target protocol.

[0013] In one embodiment, before obtaining cluster data from the master node of the main computer room cluster in real time according to the virtual node and writing the cluster data into the message queue, it also includes:

[0014] Use the virtual node as the data publisher of the message queue, and deploy the data publisher and message queue in the main computer room;

[0015] The cluster data is synchronized from the master node of the main computer room cluster according to the virtual node, and the cluster data is written into the message queue in real time.

[0016] In one embodiment, controlling the target program to obtain cluster data from the message queue in real time includes:

[0017] Use the target program as the data subscriber of the message queue and deploy the data subscriber in the backup computer room;

[0018] Control data subscribers obtain cluster data from the message queue in real time through a dedicated link.

[0019] In one embodiment, the method further comprises:

[0020] Get the synchronization offset from the master node of the main computer room cluster based on the virtual node;

[0021] Write the synchronization offset to the state storage program.

[0022] In one embodiment, the method further comprises:

[0023] Obtain the synchronization offset from the state storage program in real time to obtain the real-time synchronization offset;

[0024] Get the length of the replication buffer of the master node and compare the real-time synchronization offset with the length of the replication buffer;

[0025] When the real-time synchronization offset is greater than the length of the replication buffer, full replication is enabled between the virtual node and the master node of the computer room cluster.

[0026] In a second aspect, the present application also provides a real-time data synchronization device, comprising:

[0027] A link module, used to determine the virtual node according to the target protocol and link the virtual node to the master node of the main computer room cluster;

[0028] The acquisition module is used to obtain cluster data from the master node of the main computer room cluster in real time according to the virtual node, and write the cluster data into the message queue;

[0029] The synchronization module is used to control the target program to obtain cluster data from the message queue in real time and synchronize the cluster data to the backup computer room cluster.

[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Determine the virtual node according to the target protocol and link the virtual node to the master node of the main computer room cluster;

[0032] According to the virtual node, cluster data is obtained from the master node of the main computer room cluster in real time, and the cluster data is written into the message queue;

[0033] The control target program obtains cluster data from the message queue in real time and synchronizes the cluster data to the standby computer room cluster.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0035] Determine the virtual node according to the target protocol and link the virtual node to the master node of the main computer room cluster;

[0036] According to the virtual node, cluster data is obtained from the master node of the main computer room cluster in real time, and the cluster data is written into the message queue;

[0037] The control target program obtains cluster data from the message queue in real time and synchronizes the cluster data to the standby computer room cluster.

[0038] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0039] Determine the virtual node according to the target protocol and link the virtual node to the master node of the main computer room cluster;

[0040] According to the virtual node, cluster data is obtained from the master node of the main computer room cluster in real time, and the cluster data is written into the message queue;

[0041] The control target program obtains cluster data from the message queue in real time and synchronizes the cluster data to the standby computer room cluster.

[0042] The above-mentioned real-time data synchronization method, device, computer equipment, storage medium and computer program product determine the virtual node according to the target protocol, and link the virtual node to the master node of the main computer room cluster; obtain cluster data from the master node of the main computer room cluster in real time according to the virtual node, and write the cluster data into the message queue; control the target program to obtain cluster data from the message queue in real time, and synchronize the cluster data to the backup computer room cluster. This method can realize the isolation of Redis clusters in remote main computer rooms and backup computer rooms, and can realize real-time synchronization of cluster data (Redis cluster data) between remote main computer rooms and backup computer rooms across Redis clusters, reducing the strong dependence of the backup computer room on the main computer room, avoiding the generation of conflicting data caused by the "brain split" problem, and thus improving the reliability of real-time synchronization data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A diagram of an application environment of a real-time data synchronization method in an embodiment;

[0045] Figure 2 This is an architectural diagram for implementing data synchronization in the traditional way;

[0046] Figure 3 This is a flow chart of the real-time data synchronization method in this application;

[0047] Figure 4 An architectural diagram for implementing real-time synchronization of data in one embodiment;

[0048] Figure 5 A schematic diagram of a process for determining whether to enable full replication in one embodiment;

[0049] Figure 6 A structural block diagram of a real-time data synchronization device in an embodiment;

[0050] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The real-time data synchronization method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 sends a real-time synchronization data request to the server 104, and the server 104 receives the real-time synchronization data request, determines the virtual node according to the target protocol, and links the virtual node to the master node of the main computer room cluster; obtains cluster data from the master node of the main computer room cluster in real time according to the virtual node, and writes the cluster data to the message queue; controls the target program to obtain cluster data from the message queue in real time, and synchronizes the cluster data to the standby computer room cluster. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0053] The architecture diagram for implementing data synchronization in the traditional way is as follows Figure 2 As shown in the figure, the master node of the Redis (open source memory data storage system) cluster is deployed in the main computer room, and one or more slave nodes corresponding to the master node are deployed in the standby computer room, making the standby computer room highly dependent on the main computer room. When a failure occurs in the main computer room, in order to restore the service of the business system as soon as possible and ensure data consistency, it is necessary to upgrade the slave node of the Redis cluster in the standby computer room to the master node of the Redis cluster, so that the standby computer room can take over the write request of the main computer room, thereby realizing the rapid recovery of the business system service. In the traditional master-slave replication mode of the Redis cluster, since the standby computer room cannot realize the automatic master-slave switching of the Redis cluster, manual intervention is required to upgrade the slave node of the Redis cluster in the standby computer room to the master node of the Redis cluster, and modify the application side configuration to point to the master node of the new Redis cluster. This process will cause the disaster recovery time to be too long, affecting the stability of the system and the user experience. At the same time, in the Redis sentinel mode, when the dedicated line between the main computer room and the standby computer room fails, a "brain split" problem will occur. During the "split brain" period, both the main computer room and the backup computer room (the master node and the slave node in the Redis cluster) will start processing write operations, and these write operations will modify the same key-value pair. When the "split brain" is restored, the data between the nodes in the main computer room and the nodes in the backup computer room are conflicting, that is, the original master node and the original slave node in the Redis cluster both believe that they are the correct master node and have sole control over a certain key, resulting in the generation of conflicting data, and the conflicting data cannot be automatically merged.

[0054] A real-time data synchronization method provided by an embodiment of the present application realizes the isolation of Redis clusters in a remote main computer room and a backup computer room, and can realize real-time synchronization of cluster data (Redis cluster data) between a remote main computer room and a backup computer room across Redis clusters, so that even if a failure occurs in the main computer room, the Redis cluster in the backup computer room can still continue to provide services, ensuring that the Redis cluster in the backup computer room can automatically take over the business system services; at the same time, the Redis clusters in the remote main computer room and the backup computer room are isolated, that is, the Redis cluster in the main computer room and the Redis cluster in the backup computer room are completely independent, reducing the strong dependence of the backup computer room on the main computer room, and the two clusters manage their own data respectively without affecting each other, avoiding the occurrence of the "brain split" problem, thereby improving the reliability and accuracy of real-time synchronization of data.

[0055] In an exemplary embodiment, a real-time data synchronization method is provided. Figure 3 and Figure 4 , Figure 3 The flowchart of the real-time data synchronization method is shown in FIG. Figure 4 This is the architecture diagram for implementing real-time synchronization of data in this application. This method is applied to Figure 1 The server in the example is used to illustrate, including the following steps 302 to 306. Among them:

[0056] Step 302, determine a virtual node according to the target protocol, and link the virtual node to the master node of the main computer room cluster.

[0057] The target protocol refers to the master-slave synchronization protocol (Redis Serialization Protocol, RESP) of the Redis cluster. The RESP protocol is responsible for transmitting data between the master node and the slave node of the Redis cluster to ensure data synchronization, that is, the master node of the Redis cluster transmits data changes (write operations) to the slave node through the RESP protocol to ensure data consistency between the master node and the slave node. A virtual node refers to a pseudo slave node (pseudo SLAVE) of a Redis cluster. The pseudo SLAVE is not a formal slave node of a Redis cluster, but an application that "disguises" itself as a slave node to obtain data from the master node of the Redis cluster. The main computer room cluster (Redis cluster of the main computer room) refers to the Redis cluster deployed in the main computer room (or main data center). The master node of the main computer room cluster is responsible for processing write operations and data persistence.

[0058] Step 304: Obtain cluster data from the master node of the main computer room cluster in real time according to the virtual node, and write the cluster data into the message queue.

[0059] Cluster data refers to the full data or incremental data of the Redis cluster. A message queue is a communication mechanism used to transmit data in a distributed system.

[0060] Optionally, the RESP protocol of the Redis cluster allows slave nodes (whether pseudo-slave nodes or real slave nodes) to request data from the master node. Therefore, after the pseudo-SLAVE (virtual node) establishes a link with the master node of the main computer room cluster through the RESP protocol, the master node will continuously send its cluster data to the pseudo-SLAVE. Optionally, the pseudo-SLAVE encapsulates the cluster data received from the master node of the main computer room cluster in a message format (which can be a lightweight data exchange format or a binary format), and the server writes the encapsulated cluster data into the message queue.

[0061] Step 306, controlling the target program to obtain cluster data from the message queue in real time, and synchronizing the cluster data to the standby computer room cluster.

[0062] The target program refers to the consumer program, which is a program that obtains and processes messages from the message queue. The standby computer room cluster refers to the Redis cluster deployed in the standby computer room (or standby data center). The standby computer room cluster is responsible for synchronizing the cluster data of the main computer room cluster. The main computer room cluster and the standby computer room cluster are two independent Redis clusters, which can achieve the isolation of the Redis clusters of the remote main computer room and the standby computer room. Optionally, the main computer room cluster undertakes normal data processing and storage work, and the standby computer room cluster can quickly take over data and services when a failure occurs in the main computer room to provide disaster recovery support.

[0063] Exemplarily, after the control target program obtains the cluster data of the master node of the main computer room cluster from the message queue, the cluster data is replayed (ie, the same operation is performed) to the standby computer room cluster (the Redis cluster of the standby computer room).

[0064] In the above-mentioned real-time data synchronization method, the virtual node is determined according to the target protocol, and the virtual node is linked to the master node of the main computer room cluster; the cluster data is obtained from the master node of the main computer room cluster in real time according to the virtual node, and the cluster data is written into the message queue; the target program is controlled to obtain the cluster data from the message queue in real time, and the cluster data is synchronized to the standby computer room cluster. This method can realize the isolation of the Redis clusters of the remote main computer room and the standby computer room, and can realize the real-time synchronization of cluster data (Redis cluster data) between the remote main computer room and the standby computer room across the Redis cluster, so that even if the main computer room fails, the Redis cluster of the standby computer room can still continue to provide services, ensuring that the Redis cluster of the standby computer room can automatically take over the business system services; at the same time, the Redis clusters of the remote main computer room and the standby computer room are isolated, that is, the Redis cluster of the main computer room and the Redis cluster of the standby computer room are completely independent, reducing the strong dependence of the standby computer room on the main computer room, and the two clusters manage their own data respectively without affecting each other, avoiding the occurrence of the "brain split" problem, thereby improving the reliability and accuracy of real-time synchronization data.

[0065] In an exemplary embodiment, a virtual node is determined according to a target protocol, and the virtual node is linked to a master node of a computer room cluster, including: determining a virtual node according to a pre-built application and a target protocol; connecting the virtual node to the master node of the computer room cluster, and controlling the virtual node to perform data interaction with the master node of the computer room cluster based on the target protocol.

[0066] Exemplarily, an application is developed in advance to ensure that the application complies with the RESP protocol to implement the role of a pseudo SLAVE (virtual node) of a cluster. Among them, the role of the pseudo SLAVE (virtual node) is not a real slave node, but a simulated role, which can be called a pseudo slave node. This pseudo SLAVE can obtain data from the master node of the real Redis cluster by simulating the behavior of the slave node of the Redis cluster. Specifically, the server controls the pseudo SLAVE to maintain communication with the master node of the main computer room cluster based on the RESP protocol, and receives cluster data from the master node of the main computer room cluster, so that the pseudo SLAVE simulates the behavior of the slave node of the real Redis cluster. Among them, the cluster data may include the full data or incremental data of the Redis cluster.

[0067] In this embodiment, the virtual node is determined based on the pre-built application and the target protocol; the virtual node is controlled to maintain communication with the master node of the main computer room cluster based on the RESP protocol, and receives cluster data from the master node of the main computer room cluster, and can capture and synchronize all data changes on the master node of the main computer room cluster in real time.

[0068] In an exemplary embodiment, cluster data is obtained from the master node of the main computer room cluster in real time according to the virtual node, and the cluster data is written into the message queue, including: using the virtual node as the data publisher of the message queue, and deploying the data publisher and the message queue in the main computer room; synchronizing cluster data from the master node of the main computer room cluster according to the virtual node, and writing the cluster data into the message queue in real time.

[0069] Optionally, the technical architecture of producers and consumers is introduced. The technical architecture of producers and consumers is a message transmission mode. The message queue is a message queue intermediate file. The data publisher of the message queue refers to the producer in the technical architecture of producers and consumers. The producer is the sending end of the message queue and sends data to the message queue. The consumer is the receiving end of the message queue and obtains data from the message queue. That is, data can be transmitted through a message queue middleware. The message queue middleware can act as an intermediary. The producer puts the data into the message queue middleware, and the consumer obtains the data from the message queue middleware. Among them, the message queue middleware can be RocketMQ or KAFAKA message queue middleware.

[0070] Exemplarily, a pseudo SLAVE (virtual node) is used as a producer in the producer and consumer technical architecture, and the producer is deployed in the main computer room, and the message queue middleware is deployed at the same time. The message queue middleware can also be deployed in the main computer room. When the pseudo SLAVE obtains the cluster data synchronized by the main node of the main computer room cluster from the main node of the main computer room cluster, the cluster data is written into the message queue middleware in real time.

[0071] In the previous exemplary embodiment, controlling the target program to obtain cluster data from the message queue in real time includes: using the target program as a data subscriber of the message queue and deploying the data subscriber in the backup computer room; controlling the data subscriber to obtain cluster data from the message queue in real time through a dedicated link.

[0072] The target program is the consumer program, and the data subscriber of the message queue refers to the consumer in the technical architecture of the producer and consumer. The consumer program is used to obtain data from the message queue in real time. The dedicated link refers to the dedicated line across computer rooms.

[0073] For example, the consumer program (target program) is used as the consumer of the message queue in the producer and consumer technical architecture, and the consumer is deployed in the standby computer room. The consumer program is linked to the message queue middleware through a dedicated line across computer rooms, and obtains cluster data of the master node of the main computer room cluster from the message queue in real time.

[0074] After the consumer program obtains the cluster data of the master node of the main computer room cluster from the message queue in real time, the cluster data is synchronized to the backup computer room cluster. Specifically, the consumer program can perform a data write operation to write the cluster data to the backup computer room cluster to ensure data synchronization between the backup computer room cluster and the main computer room cluster.

[0075] In this embodiment, the producer and consumer technical architecture is introduced, and the message queue is used to decouple the data publisher (virtual node) and the data subscriber (target program), reduce the tight coupling between modules (such as data publishers and data subscribers), and improve flexibility and maintainability. The data publisher sends the data to the message queue, and the data subscriber processes the data asynchronously, which can improve the concurrent processing capability and reduce the delay of request response. The virtual node synchronizes the cluster data from the master node of the main computer room cluster and writes the cluster data to the message queue in real time. The message queue usually provides message persistence and retry mechanism to ensure that the data is not lost and ensure the reliable transmission of cluster data.

[0076] In an exemplary embodiment, the method further includes: obtaining a synchronization offset from a master node of a computer room cluster according to a virtual node; and writing the synchronization offset into a state storage program.

[0077] In the master-slave replication process of the Redis cluster, the master node will record the write operations it receives in order. The synchronization offset (SYNC OFFSET offset) is used to record the position or progress of the write operations that the master node has synchronized to the slave node, that is, the synchronization offset is used to indicate the progress of the data currently received by the slave node or pseudo-SLAVE during the synchronization process (master-slave replication process). The state storage program refers to ZOOKEEER (distributed coordination service). Optionally, the state storage program is used to record the progress of synchronization between the master node and pseudo-SLAVE of the main computer room cluster.

[0078] Exemplarily, the pseudo-SLAVE synchronizes cluster data from the master node of the main computer room cluster in real time. When the master node data of the main computer room cluster is updated, the pseudo-SLAVE receives the updated data and records the current synchronization progress (synchronization offset) in the state storage program. Whenever the synchronization progress (synchronization offset) is updated, the updated synchronization offset is stored in the state storage program.

[0079] In the previous exemplary embodiment, Figure 5 As shown, the method further includes determining whether to enable full replication based on the real-time synchronization offset and the length of the replication buffer. The determination of whether to enable full replication includes steps 502 to 506.

[0080] Step 502: Acquire the synchronization offset in real time from the state storage program to obtain the real-time synchronization offset.

[0081] Step 504, obtain the length of the copy buffer of the master node, and compare the real-time synchronization offset with the length of the copy buffer.

[0082] Step 506: When the real-time synchronization offset is greater than the length of the replication buffer, full replication is started between the virtual node and the master node of the computer room cluster.

[0083] The replication buffer refers to a buffer maintained by the master node of the Redis cluster for each slave node, which is used to store data that has not been replicated by the slave node. When the master node has new write operations, these operations will be written to the replication buffer first, and then synchronized to the slave node through the network. The replication buffer is a circular queue, which means that when the replication buffer is full, the new data will overwrite the oldest data. If the slave node does not process this data in time, there will be a risk of data loss.

[0084] The length of the replication buffer refers to the maximum amount of data that can be stored in this replication buffer area, which can represent the maximum number of unsynchronized data operations that can be stored in the replication buffer area of ​​the master node. Full replication means that the slave node resynchronizes all the data of the master node from the beginning under certain specific circumstances.

[0085] When the replication buffer of the master node overflows, if the pseudo-SLAVE fails to obtain certain data in time, the data in the replication buffer will be lost, thereby reducing the reliability of the synchronized data. Therefore, in this embodiment, the synchronization offset is obtained in real time from the state storage program to obtain the real-time synchronization offset, and the length of the replication buffer of the master node of the main computer room cluster is obtained. The real-time synchronization offset is compared with the length of the replication buffer to determine whether the real-time synchronization offset is greater than the length of the replication buffer. When the real-time synchronization offset is greater than the length of the replication buffer, full replication is enabled between the virtual node and the master node of the main computer room cluster. That is, the pseudo-SLAVE requests the master node of the main computer room cluster to resynchronize all data (i.e., pull the entire RDB data snapshot from the master node of the main computer room cluster), and replay the data in the standby computer room cluster to ensure that the data in the standby computer room is consistent with that in the main computer room.

[0086] In this embodiment, by judging whether the real-time synchronization offset is greater than the length of the copy buffer, it is determined whether to start full replication, thereby ensuring that the data in the backup computer room is consistent with that in the main computer room and avoiding data loss.

[0087] In another embodiment, a real-time data synchronization method is provided, the method comprising:

[0088] Determine the virtual node according to the pre-built application and the target protocol; connect the virtual node to the master node of the computer room cluster, and control the virtual node to exchange data with the master node of the computer room cluster based on the target protocol.

[0089] Use the virtual node as the data publisher of the message queue, and deploy the data publisher and message queue in the main computer room; synchronize cluster data from the master node of the main computer room cluster according to the virtual node, and write the cluster data into the message queue in real time.

[0090] The target program is used as the data subscriber of the message queue, and the data subscriber is deployed in the backup computer room; the data subscriber is controlled to obtain cluster data from the message queue in real time through a dedicated link.

[0091] Synchronize cluster data to the standby data center cluster.

[0092] Obtain the synchronization offset from the master node of the main computer room cluster according to the virtual node; write the synchronization offset into the state storage program.

[0093] The synchronization offset is obtained from the state storage program in real time to obtain the real-time synchronization offset; the length of the replication buffer of the master node is obtained, and the real-time synchronization offset is compared with the length of the replication buffer; when the real-time synchronization offset is greater than the length of the replication buffer, full replication is started between the virtual node and the master node of the main computer room cluster.

[0094] A real-time data synchronization method provided by an embodiment of the present application realizes the isolation of Redis clusters in a remote main computer room and a backup computer room, and can realize real-time synchronization of cluster data (Redis cluster data) between a remote main computer room and a backup computer room across Redis clusters, so that even if a failure occurs in the main computer room, the Redis cluster in the backup computer room can still continue to provide services, ensuring that the Redis cluster in the backup computer room can automatically take over the business system services; at the same time, the Redis clusters in the remote main computer room and the backup computer room are isolated, that is, the Redis cluster in the main computer room and the Redis cluster in the backup computer room are completely independent, reducing the strong dependence of the backup computer room on the main computer room, and the two clusters manage their own data respectively without affecting each other, avoiding the occurrence of the "brain split" problem, thereby improving the reliability and accuracy of real-time synchronization of data.

[0095] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0096] Based on the same inventive concept, the embodiment of the present application also provides a real-time data synchronization device for implementing the real-time data synchronization method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more real-time data synchronization device embodiments provided below can refer to the limitations of the real-time data synchronization method above, and will not be repeated here.

[0097] In an exemplary embodiment, Figure 6 As shown, a real-time data synchronization device is provided, including: a link module 602, an acquisition module 604 and a synchronization module 606, wherein:

[0098] A linking module 602 is used to determine a virtual node according to a target protocol and link the virtual node to a master node of a computer room cluster;

[0099] The acquisition module 604 is used to acquire cluster data from the master node of the computer room cluster in real time according to the virtual node, and write the cluster data into the message queue;

[0100] The synchronization module 606 is used to control the target program to obtain cluster data from the message queue in real time and synchronize the cluster data to the standby computer room cluster.

[0101] In an exemplary embodiment, the link module 602 is also used to determine the virtual node based on the pre-built application and the target protocol; connect the virtual node to the master node of the computer room cluster, and control the virtual node to interact with the master node of the computer room cluster based on the target protocol.

[0102] In an exemplary embodiment, the acquisition module 604 is also used to use the virtual node as the data publisher of the message queue, and deploy the data publisher and the message queue in the main computer room; synchronize the cluster data from the master node of the main computer room cluster according to the virtual node, and write the cluster data into the message queue in real time.

[0103] In an exemplary embodiment, the synchronization module 606 is also used to use the target program as a data subscriber of the message queue and deploy the data subscriber in the backup computer room; and control the data subscriber to obtain cluster data from the message queue in real time through a dedicated link.

[0104] In an exemplary embodiment, the real-time data synchronization device further includes:

[0105] The writing module is used to obtain the synchronization offset from the master node of the main computer room cluster according to the virtual node; and write the synchronization offset into the state storage program.

[0106] In an exemplary embodiment, the real-time data synchronization device further includes:

[0107] The full replication module is used to obtain the synchronization offset from the state storage program in real time to obtain the real-time synchronization offset; obtain the length of the replication buffer of the master node, and compare the real-time synchronization offset with the length of the replication buffer; when the real-time synchronization offset is greater than the length of the replication buffer, start full replication between the virtual node and the master node of the main computer room cluster.

[0108] Each module in the above-mentioned real-time data synchronization device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0109] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store cluster data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a real-time data synchronization method is implemented.

[0110] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0111] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0112] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0113] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0115] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0116] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A real-time data synchronization method, characterized in that: The method comprises: Determine a virtual node according to the target protocol, and link the virtual node to the master node of the main computer room cluster; Acquire cluster data from the master node of the computer room cluster in real time according to the virtual node, and write the cluster data into a message queue; The target program is controlled to obtain the cluster data from the message queue in real time, and synchronize the cluster data to the standby computer room cluster.

2. The method according to claim 1, characterized in that The step of determining a virtual node according to a target protocol and linking the virtual node to a master node of a computer room cluster includes: Determining the virtual node according to the pre-built application and the target protocol; The virtual node is connected to the master node of the computer room cluster, and the virtual node is controlled to perform data interaction with the master node of the computer room cluster based on the target protocol.

3. The method according to claim 1, characterized in that: The step of acquiring cluster data from the master node of the computer room cluster in real time according to the virtual node and writing the cluster data into a message queue includes: Using the virtual node as the data publisher of the message queue, and deploying the data publisher and the message queue in the main computer room; The cluster data is synchronized from the master node of the main computer room cluster according to the virtual node, and the cluster data is written into the message queue in real time.

4. The method according to claim 1, characterized in that The control target program acquires the cluster data from the message queue in real time, including: Using the target program as a data subscriber of the message queue, and deploying the data subscriber in a standby computer room; The data subscriber is controlled to obtain the cluster data from the message queue in real time through a dedicated link.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining a synchronization offset from a master node of the computer room cluster according to the virtual node; The synchronization offset is written into the state storage program.

6. The method according to claim 5, characterized in that The method further comprises: Acquire the synchronization offset in real time from the state storage program to obtain a real-time synchronization offset; Obtaining the length of the replication buffer of the master node, and comparing the real-time synchronization offset with the length of the replication buffer; When the real-time synchronization offset is greater than the length of the replication buffer, full replication is started between the virtual node and the master node of the computer room cluster.

7. A real-time data synchronization device, characterized in that: The device comprises: A linking module, used to determine a virtual node according to a target protocol and link the virtual node to a master node of a main computer room cluster; An acquisition module, used for acquiring cluster data from the master node of the computer room cluster in real time according to the virtual node, and writing the cluster data into a message queue; The synchronization module is used to control the target program to obtain the cluster data from the message queue in real time and synchronize the cluster data to the standby computer room cluster.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images