Data synchronization device and method

Abstract

The invention relates to the technical field of data synchronization, and provides a data synchronization device and method. The device comprises a receiving module used for receiving target data sent by a message queue; the target data comprises a target data change stream and a target operation sequence, and the target data change stream comprises change data synchronized from the main cluster; the verification module is used for verifying the target data change stream by using the target operation sequence, and performing exception correction on the target data change stream based on a verification result to obtain a corrected data change stream; the processing module is used for converting the correction data change stream into an executable distributed cache command; and the synchronization module is used for distributing the distributed cache command to the disaster recovery cluster. According to the invention, while cross-cluster and cross-machine-room data synchronization can be realized, the risk of data loss in the synchronization process is avoided, and smooth proceeding of the synchronization process is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of data synchronization, and in particular to a data synchronization device and method. Background Art

[0002] Distributed cache is a cache system designed to cope with large-scale data processing and high concurrent access requirements. It is deployed in a distributed environment and provides fast data reading and query services by storing data in the memory of multiple nodes. This cache system can be horizontally expanded to adapt to large data volumes and high concurrent access scenarios, and is one of the key components in big data processing frameworks and applications.

[0003] Distributed cache is connected to various big data platforms, databases, or persistent storage systems through a series of interfaces, and is provided to applications for access in the form of a universal interface, thus bridging the gap between applications and storage systems and providing applications with a unified client API and global namespace, enabling easy access to any data connected to the distributed cache.

[0004] In order to solve the problem of distributed cache data availability and ensure that application access is not affected when any node goes down, data can usually be synchronized to multiple different nodes, and automatically directed to other available nodes when a node fails.

[0005] However, this method only realizes data synchronization within the cluster and cannot avoid cluster-level node failures. Although the existing data synchronization methods can realize cross-cluster or cross-computer room data synchronization to avoid cluster-level or computer room-level node failures, there is a risk of data loss during the data synchronization process, which leads to the interruption of the synchronization process. Summary of the invention

[0006] The embodiments of the present application provide a data synchronization device and method to solve the technical problem that although the existing data synchronization method can achieve cross-cluster or cross-computer room data synchronization to avoid cluster-level or computer room-level node failures, there is a risk of data loss during the data synchronization process, thereby causing the synchronization process to be interrupted.

[0007] In a first aspect, an embodiment of the present application provides a data synchronization device, including: A receiving module, used to: receive target data sent by a message queue, and send the target data to a verification module; the target data includes a target data change stream and a target operation sequence, and the target data change stream includes change data synchronized from a main cluster; A verification module, used to verify the target data change stream using the target operation sequence, perform abnormal correction on the target data change stream based on the verification result to obtain a corrected data change stream, and send the corrected data change stream to the processing module; A processing module, configured to: convert the modified data change stream into an executable distributed cache command, and send the distributed cache command to a synchronization module; The synchronization module is used to distribute the distributed cache command to the disaster recovery cluster.

[0008] In one embodiment, it further includes: Configuration module, used to store and manage the configuration information of other modules for other modules to call when they are running; The configuration information includes connection configuration information required to establish a synchronous connection with a cluster node, verification configuration information required to verify a data change stream, processing configuration information for processing the data change stream, and synchronization configuration information for the entire data synchronization process.

[0009] In one embodiment, converting the modified data change stream into an executable distributed cache command includes: Parsing the modified data change stream to obtain modified key value data; Filtering the modified key value data to obtain filtered key value data; For the filtered key-value data whose values ​​have changed, hash grouping is performed based on the keys corresponding to the values ​​to obtain multiple groups of target key-value data; The multiple groups of target key-value data are converted into executable distributed cache commands.

[0010] In one embodiment, the parsing of the modified data change stream to obtain the modified key value data includes: Extracting valid instructions from a plurality of operation instructions in the modified data change stream; The operation data corresponding to the effective instruction is converted into key value data to obtain modified key value data.

[0011] In one embodiment, converting the multiple sets of target key-value data into executable distributed cache commands includes: Based on the mapping parameters, obtaining the target value of the target key-value data corresponding to the valid instruction from the multiple sets of target key-value data; The target value is converted into an executable distributed cache command.

[0012] In one embodiment, the method of verifying the target data change stream by using the target operation sequence and performing abnormal correction on the target data change stream based on the verification result to obtain the corrected data change stream includes: Compare the target operation sequence with the operation sequence in the target node to obtain a first comparison result; the target node includes a master node or a slave node of the master cluster; If the first comparison result is consistent, then compare the operation identifier in the target operation sequence with the operation identifier in the target data change stream to obtain a second comparison result, and compare the timestamp in the target operation sequence with the timestamp in the target data change stream to obtain a third comparison result; If the first comparison result is inconsistent, initiating a request to the receiving module to call the operation sequence in the target node, and calling the operation sequence in the target node pulled by the receiving module; Modifying the target operation sequence based on the operation sequence in the target node to obtain a modified operation sequence; Compare the operation identifier in the correction operation sequence with the operation identifier in the target data change stream to obtain a second comparison result, and compare the timestamp in the correction operation sequence with the timestamp in the target data change stream to obtain a third comparison result; Based on the second comparison result and the third comparison result, the target data change flow is abnormally corrected to obtain a corrected data change flow.

[0013] In one embodiment, based on the second comparison result and the third comparison result, performing abnormal correction on the target data change stream to obtain a corrected data change stream includes: If at least one of the second comparison result and the third comparison result is inconsistent, a request for calling the data change stream in the target node is initiated to the receiving module, and the data change stream in the target node pulled by the receiving module is called; The target data change flow is modified based on the data change flow in the target node to obtain a modified data change flow.

[0014] In one embodiment, the target data sent by the receiving message queue includes: The message queue is monitored in real time, and the target data sent by the message queue is received when the data in the message queue changes.

[0015] In one embodiment, the synchronization module is further used to: write the changed data in the main cluster into the message queue, and when the changed data is successfully written in at least two of the master node of the main cluster, the slave node of the main cluster and the message queue, send a write success message to the client in the node of the main cluster where the writing is successful.

[0016] In a second aspect, an embodiment of the present application provides a data synchronization method, including: Receive target data sent by the message queue; the target data includes a target data change stream and a target operation sequence, and the target data change stream includes change data synchronized from the main cluster; Verifying the target data change stream using the target operation sequence, and performing abnormal correction on the target data change stream based on the verification result to obtain a corrected data change stream; Converting the modified data change stream into an executable distributed cache command; Distribute the distributed cache command to the disaster recovery cluster.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing a computer program, wherein the processor implements the steps of the data synchronization method described in the second aspect when executing the program.

[0018] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the data synchronization method described in the second aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, comprising a computer program, which, when executed by a processor, implements the steps of the data synchronization method described in the second aspect.

[0020] The data synchronization device provided by the present application includes: a receiving module, which is used to receive the target data sent by the message queue, and send the target data to the verification module, the target data includes a target data change stream and a target operation sequence, the verification module is used to verify the target data change stream using the target operation sequence, and based on the verification result, the target data change stream is abnormally corrected to obtain a corrected data change stream, and the corrected data change stream is sent to the processing module, the processing module is used to convert the corrected data change stream into an executable distributed cache command, and send the distributed cache command to the synchronization module, and the synchronization module is used to distribute the distributed cache command to the disaster recovery cluster. Since the target data change stream includes the change data synchronized from the main cluster, on the one hand, the change data of the main cluster can be synchronized to the disaster recovery cluster, thereby realizing cross-cluster data synchronization, and the main cluster and the disaster recovery cluster can be set in different computer rooms to realize cross-computer room data synchronization. On the other hand, the message queue performs redundant backup of the change data in the main cluster. Therefore, in the process of synchronizing the change data of the main cluster to the disaster recovery cluster, if the data in either the main cluster or the message queue is lost, the data can be actively pulled from the other place for recovery, thereby avoiding the risk of data loss during the synchronization process and ensuring the smooth progress of the synchronization process. In summary, this application can achieve cross-cluster and cross-computer room data synchronization while avoiding the risk of data loss during the synchronization process and ensuring the smooth progress of the synchronization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 is a structural diagram of a data synchronization device provided in an embodiment of the present application; Figure 2 It is a flowchart of a data synchronization method provided in an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] Figure 1 Schematic diagram of the structure of the data synchronization device provided by the embodiment of the present application. Figure 1 , an embodiment of the present application provides a data synchronization device, which may include: The receiving module is used to: receive the target data sent by the message queue and send the target data to the verification module; the target data includes a target data change stream and a target operation sequence, and the target data change stream includes the change data synchronized from the main cluster; A verification module is used to verify the target data change stream using the target operation sequence, perform abnormal correction on the target data change stream based on the verification result, obtain a corrected data change stream, and send the corrected data change stream to the processing module; The processing module is used to: convert the modified data change stream into an executable distributed cache command, and send the distributed cache command to the synchronization module; The synchronization module is used to distribute the distributed cache commands to the disaster recovery cluster.

[0025] The data synchronization device may be arranged in a management node of a main cluster, and the message queue may be arranged in the data synchronization device.

[0026] Among them, when the receiving module receives the target data sent by the message queue, it can parse, verify, cache and other operations on the target data to ensure the integrity and correctness of the target data during the transmission process.

[0027] The target data change stream mainly records various operation instructions in the message queue cache and the data values ​​before and after the operation instructions are executed. The characteristics of the target data change stream are strongly related to the type of cache in the message queue.

[0028] The target operation sequence mainly records the unidirectionally increasing operation identifier and timestamp corresponding to the target data change flow. The clients deployed on the master node and slave node of the main cluster will send the target operation sequence to the message queue in real time.

[0029] The processing module can ensure that the modified data change stream is converted into instructions that can be executed in the disaster recovery cluster by converting the modified data change stream into executable distributed cache commands, thereby enabling data synchronization in the disaster recovery cluster to proceed smoothly.

[0030] The synchronization module can manage the status of the command distribution process, monitor and record the status of data synchronization in real time, and suspend command distribution when an abnormal synchronization status is detected.

[0031] In addition, the synchronization module can distribute the distributed cache command to one or more disaster recovery clusters, which are not limited here. By synchronizing the data changes of the main cluster to multiple disaster recovery clusters, the system scalability is improved, so that the system can adapt to dynamically changing workloads and scales, thereby responding to different business needs, and there is no additional performance consumption for the main cluster.

[0032] The data synchronization device provided in this embodiment includes: a receiving module, which is used to receive the target data sent by the message queue, and send the target data to the verification module, the target data includes a target data change stream and a target operation sequence, the verification module is used to verify the target data change stream using the target operation sequence, perform abnormal correction on the target data change stream based on the verification result, obtain the corrected data change stream, and send the corrected data change stream to the processing module, the processing module is used to convert the corrected data change stream into an executable distributed cache command, and send the distributed cache command to the synchronization module, and the synchronization module is used to distribute the distributed cache command to the disaster recovery cluster. Since the target data change stream includes the change data synchronized from the main cluster, on the one hand, the change data of the main cluster can be synchronized to the disaster recovery cluster, thereby realizing cross-cluster data synchronization, and the main cluster and the disaster recovery cluster can be set in different computer rooms to realize cross-computer room data synchronization. On the other hand, the message queue performs redundant backup of the change data in the main cluster. Therefore, in the process of synchronizing the change data of the main cluster to the disaster recovery cluster, if the data in either the main cluster or the message queue is lost, the data can be actively pulled from the other place for recovery, thereby avoiding the risk of data loss during the synchronization process and ensuring the smooth progress of the synchronization process. In summary, this embodiment can achieve cross-cluster and cross-computer room data synchronization while avoiding the risk of data loss during the synchronization process, thereby ensuring that the synchronization process proceeds smoothly.

[0033] Reference Figure 1 In one embodiment, the data synchronization device may further include: The configuration module is used to store and manage the configuration information of other modules so that other modules can call it when they are running. Specifically, the configuration information can include the following information: 1. The connection configuration information required to establish a synchronous connection with the cluster nodes, including the node's network address, communication port, user password, synchronization table list and other parameters. Through the connection configuration information, the system can flexibly configure the communication settings with the cluster nodes to ensure the correct establishment of the synchronous connection.

[0034] 2. Verification configuration information required to verify the data change stream, including the verification algorithm used, checksum, and manually configured verification process, such as personnel notification information in case of abnormalities. By verifying the configuration information, the system can ensure that the data change stream in the synchronization process meets the consistency requirements after verification, and prevent data damage or tampering during transmission.

[0035] 3. Processing configuration information for processing data change streams, including parsing rules, filtering rules, grouping rules, and conversion rules. By processing configuration information, the system can properly parse, filter, group, and convert data change streams as required, ensuring that data can be correctly processed and applied to the disaster recovery cluster during synchronization.

[0036] 4. Synchronization configuration information for the entire data synchronization process, including the synchronization frequency, synchronization time interval, and whether delayed synchronization is required to avoid misoperation of the main cluster. For example, when the main node performs misoperation such as deletion, the deletion command has not yet been executed in the disaster recovery cluster because delayed synchronization is configured. Therefore, the data in the disaster recovery cluster can be used to quickly restore the main node data to improve the security and reliability of the cluster; the synchronization node list defines the node range for data synchronization; the synchronization strategy specifies the method and priority of data synchronization. Through synchronization configuration information, the system can flexibly configure synchronization behavior according to actual needs to meet different business scenarios and performance requirements.

[0037] This embodiment uses the configuration module to enable the system to perform data synchronization in a more flexible and customizable manner to meet disaster recovery environments and requirements of different quantities and regions.

[0038] Reference Figure 1 In one embodiment, converting the modified data change stream into an executable distributed cache command may include: 1. Parse the modified data change stream to obtain the modified key value data; 2. Filter the modified key value data to obtain filtered key value data; 3. For the filtered key-value data whose values ​​have changed, hash group them according to the keys corresponding to the values ​​to obtain multiple groups of target key-value data; 4. Convert multiple sets of target key-value data into executable distributed cache commands.

[0039] In step 1, a general data parser can be introduced to extract valid instructions from multiple operation instructions in the modified data change stream, and convert the operation data corresponding to the valid instructions into key-value data to obtain modified key-value data, as shown in the following table: Table 1 Key value conversion table

[0040] Taking the valid instruction set as an example, Next2 represents the second operation data after the instruction, and Next4 represents the fourth operation data after the instruction. For the valid instruction set, the two operation data after it are converted into keys, and the fourth operation data after it is converted into values, to obtain the key value data corresponding to the valid instruction set, that is, the modified key value data corresponding to the valid instruction set. The same is true for other valid instructions, which will not be repeated here.

[0041] By converting data of different formats in the modified data change stream into a recognizable key-value structure, it is ensured that the data can be correctly understood and processed by subsequent processes. In addition, since the universal data parser can support multiple data formats, the system is more versatile and can adapt to the data parsing needs of different application scenarios.

[0042] In step 2, the modified key value data can be filtered according to the configured filtering rules, and only the required data can be processed to ensure that only the data that meets the filtering rules is passed to the subsequent processing steps. For example: Filter based on the key of the modified key-value data, such as filtering out the modified key-value data whose key contains User or Offer; Filter according to the value type of the modified key value data, such as filtering out the modified key value data whose value data type is String or Number; Filter by time range, for example, filter out the modified key value data between 00:00 and 07:00.

[0043] Since the filtering rules can be configured according to the user's selection, the data filtering function provides users with more flexible data selection.

[0044] In step 3, the hash grouping can be performed according to the parallel parameters in the configured grouping rules, and the target key value data of the same group needs to be arranged in chronological order according to the change sequence to ensure the data timing consistency.

[0045] It should be noted that flexible grouping strategies can also be formulated based on business logic, data characteristics and other factors, which are not limited here.

[0046] Through hash grouping, the process can process multiple sets of target key-value data in parallel, improve the concurrency and efficiency of data synchronization, and effectively reduce synchronization delay.

[0047] In step 4, based on the mapping parameters, target values ​​of target key-value data corresponding to valid instructions can be obtained from multiple sets of target key-value data, and the target values ​​can be converted into executable distributed cache commands.

[0048] Among them, the mapping parameters can be configured as follows: Table 2 Mapping parameter table

[0049] Taking the valid instruction set as an example, according to the above table, its corresponding instruction content set KeyNameKeyValue can be mapped. The instruction content includes the target value of the target key value data corresponding to the valid instruction set, that is, KeyName and KeyValue. It is converted into a data format compatible with the disaster recovery cluster according to the required data type to obtain an executable distributed cache command. The same applies to other valid instructions, which will not be repeated here.

[0050] The following table illustrates the data format conversion corresponding to different data types: Table 3 Data format conversion table

[0051] For example, convert KeyValue to String type to get "KeyValue"; convert KeyValue1, KeyValue2, and KeyValue3 to List type to get "KeyValue1", "KeyValue2", "KeyValue3". The format conversion of other data types is similar and will not be repeated here.

[0052] Through instruction mapping and data type conversion, it can be ensured that the target data change stream synchronized from the message queue is eventually converted into commands that can be executed in the disaster recovery cluster.

[0053] This embodiment uses the four steps of data analysis, data filtering, parallel grouping and data conversion to accurately analyze, flexibly filter, reasonably group and convert the modified data change stream on demand based on the corresponding rules configured, and finally obtains the distributed cache command executable by the disaster recovery cluster, so that data synchronization can be carried out smoothly. At the same time, through hash grouping, unrelated filtered key-value data are divided into different groups, so that subsequent conversions can be carried out in parallel in unrelated groups, thereby maximizing data synchronization efficiency.

[0054] Reference Figure 1 In one embodiment, verifying the target data change stream using the target operation sequence, and performing abnormal correction on the target data change stream based on the verification result to obtain the corrected data change stream may include: 1. Compare the target operation sequence with the operation sequence in the target node to obtain a first comparison result; The target nodes include the master node or slave node of the main cluster; 2. If the first comparison result is consistent, compare the operation identifier in the target operation sequence with the operation identifier in the target data change stream to obtain a second comparison result, and compare the timestamp in the target operation sequence with the timestamp in the target data change stream to obtain a third comparison result; 3. If the first comparison result is inconsistent, a request to call the operation sequence in the target node is initiated to the receiving module, and the operation sequence in the target node pulled by the receiving module is called; 4. Modify the target operation sequence based on the operation sequence in the target node to obtain a modified operation sequence; 5. Compare the operation identifier in the correction operation sequence with the operation identifier in the target data change stream to obtain a second comparison result, and compare the timestamp in the correction operation sequence with the timestamp in the target data change stream to obtain a third comparison result; 6. If at least one of the second comparison result and the third comparison result is inconsistent, a request for calling the data change stream in the target node is initiated to the receiving module, and the data change stream in the target node pulled by the receiving module is called; 7. Based on the data change flow in the target node, the target data change flow is modified to obtain a modified data change flow.

[0055] In step 1, the target operation sequence is compared with the operation sequence in the master node or the operation sequence in the slave node, depending on which node the change data of the master cluster is successfully written to: If the changed data is successfully written to the master node and the message queue, the target operation sequence is compared with the operation sequence in the master node; If the changed data is successfully written to the slave node and the message queue, the target operation sequence is compared with the operation sequence in the slave node; If the changed data is successfully written to the master node, the slave node, and the message queue, the target operation sequence is compared with the operation sequence in the master node or the slave node.

[0056] From step 3 to step 4: If the target operation sequence is inconsistent with the operation sequence in the master node, a request to call the operation sequence in the master node is initiated to the receiving module, and the operation sequence in the master node pulled by the receiving module is called, and the target operation sequence is corrected based on the operation sequence in the master node to obtain a corrected operation sequence; If the target operation sequence is inconsistent with the operation sequence in the slave node, a request to call the operation sequence in the slave node is initiated to the receiving module, and the operation sequence in the slave node pulled by the receiving module is called, and the target operation sequence is corrected based on the operation sequence in the slave node to obtain a corrected operation sequence; In step 6 to step 7: If the changed data is successfully written into the master node and the message queue, a request to call the data change stream in the master node is initiated to the receiving module, and the data change stream in the master node pulled by the receiving module is called, and the target data change stream is corrected based on the data change stream in the master node to obtain the corrected data change stream; If the changed data is successfully written into the slave node and the message queue, a request to call the data change stream in the slave node is initiated to the receiving module, and the data change stream in the slave node pulled by the receiving module is called, and the target data change stream is corrected based on the data change stream in the slave node to obtain the corrected data change stream; If the changed data is successfully written into the master node, slave node and message queue, a request to call the data change stream in the master node or slave node is initiated to the receiving module, and the data change stream in the master node or slave node pulled by the receiving module is called. Based on the data change stream in the master node or slave node, the target data change stream is corrected to obtain the corrected data change stream.

[0057] In step 6, if the second comparison result and the third comparison result are both consistent, the target data change stream is directly sent to the processing module for subsequent processing.

[0058] Among them, step 1 is timing detection. Since the target data change stream in the message queue includes the change data synchronized from the main cluster, the timing detection can determine whether the operation sequence and time of the target data change stream are correct by detecting whether the target operation sequence is consistent with the operation sequence in the node where the change data is successfully written by the main cluster, thereby preventing the problem of disordered operations caused by network delays and other reasons.

[0059] Steps 2 and 5 are consistency checks. After determining that the operation sequence corresponding to the target data change stream is correct, it is also necessary to detect whether the target data change stream actually changes according to the operation sequence. Therefore, consistency detection can determine whether the target data change stream changes in the correct order and time by detecting whether the operation identifier and timestamp in the target data change stream are consistent with the operation identifier and timestamp in the confirmed correct operation sequence, thereby preventing data disorder caused by network delays and other reasons, and ensuring that the data can be synchronized to the disaster recovery cluster in the correct time order.

[0060] Furthermore, when performing exception correction based on data pulling, whether it is to correct the exception of the target operation sequence or to correct the exception of the target data change flow, the new version can be given priority, merged modification and other methods can be adopted to ensure that data consistency can be maintained in the case of concurrent modification and data corruption.

[0061] At the same time, data pulling can be actively performed by the receiving module regularly to obtain the latest data updates, or it can be actively performed by the receiving module after receiving a scheduling request to ensure that the pulled data can be seamlessly integrated into the local system after processing. Data pulling can be full or incremental, depending on the current synchronization strategy, specifically: Full pull: When data synchronization is initiated for the first time or synchronization is abnormally interrupted, a full data snapshot of the main cluster is pulled based on the pull time point, and the pull time point is recorded. The changed data after the pull time point is written to the message queue by the synchronization task in the synchronization module.

[0062] Incremental pull: Pull the changed data after the latest data flow operation identifier in the message queue from the primary cluster or the backup cluster, and clear the message queue before pulling.

[0063] Furthermore, during the process of timing detection, consistency detection and abnormal correction, once a timing error, consistency error, correction error, etc. is found, the system will trigger an abnormal alarm and immediately notify relevant personnel to conduct timely intervention.

[0064] It should be noted that, in actual applications, there is no strict timing relationship between step 2 and step 3; that is, they can be executed simultaneously, or any one step can be executed first, depending on actual needs and is not limited here.

[0065] Through timing detection and consistency detection, this embodiment can ensure that the target data change flow remains consistent in the transmission and application stages, prevent data damage or tampering, ensure the reliability of the synchronization link, and avoid abnormalities in the downstream data processing link due to disordered operations or data. Through active pulling, it can actively request the latest data from other nodes to avoid waiting for passive synchronization and improve real-time performance. Through incremental pulling, only the changed data can be synchronized, reducing the amount of data transmitted, improving synchronization efficiency, and reducing latency.

[0066] Reference Figure 1 In one embodiment, receiving target data sent by a message queue may include: Monitor the message queue in real time and receive the target data sent by the message queue when the data in the message queue changes.

[0067] The receiving module can start the synchronous monitoring service on the specified network port to monitor the data changes in the message queue. When the data changes are monitored, the synchronization request from the main cluster in the message queue is synchronously perceived, and the target data sent by the message queue is received at this time.

[0068] This embodiment can actively perceive data changes in the message queue in real time by monitoring, so that data synchronization can be quickly started at the moment of data change, avoiding synchronization delays caused by passively knowing data changes.

[0069] Reference Figure 1In one embodiment, the synchronization module is also used to: write the changed data in the main cluster to the message queue, and when the changed data is successfully written in at least two of the master node of the main cluster, the slave node of the main cluster, and the message queue, send a write success message to the client in the node of the main cluster where the writing is successful.

[0070] The synchronization task in the synchronization module writes the changed data in the main cluster to the message queue, and when the changed data is successfully written in at least two of the main cluster's master node, the main cluster's slave node, and the message queue, a write success message is sent to the client in the node in the main cluster where the data is successfully written.

[0071] Among them, the data written to the master node is saved in the master copy, and the data written to the slave node is saved in the slave copy. The clients deployed in the master node and the slave node will write the identifier and execution time of the write operation into the message queue in real time, and the verification module will read this content for verification.

[0072] It should be noted that if the master node and slave node of the main cluster successfully write the changed data, but the message queue fails to write the changed data, the synchronization task will continue to write the changed data to the message queue until the writing is successful.

[0073] In addition, the synchronization module is also used to monitor the application of slave node data and alarm when an error occurs.

[0074] This embodiment introduces the Quorum mechanism for data redundancy backup to ensure that synchronized data is in multiple locations. Once data on a node is lost, data can be actively pulled from other nodes for recovery, thereby improving synchronization fault tolerance and forming a balance between consistency and availability with asynchronous updates.

[0075] Figure 2 is a flowchart of a data synchronization method provided in an embodiment of the present application; Figure 2 , this implementation provides a data synchronization method, which may include: 201. Receive target data sent by the message queue; The target data includes a target data change stream and a target operation sequence. The target data change stream includes the change data synchronized from the main cluster. 202. Verify the target data change stream using the target operation sequence, and perform abnormal correction on the target data change stream based on the verification result to obtain a corrected data change stream; 203. Convert the modified data change stream into an executable distributed cache command; 204. Distribute the distributed cache command to the disaster recovery cluster.

[0076] The data synchronization method provided in this embodiment, because the target data change stream includes the change data synchronized from the main cluster, on the one hand, can synchronize the change data of the main cluster to the disaster recovery cluster, thereby realizing data synchronization across clusters, and can realize data synchronization across computer rooms by setting the main cluster and the disaster recovery cluster in different computer rooms, on the other hand, the message queue performs redundant backup of the change data in the main cluster, so in the process of synchronizing the change data of the main cluster to the disaster recovery cluster, if the data in either the main cluster or the message queue is lost, the data can be actively pulled from the other place for recovery, thereby avoiding the risk of data loss during the synchronization process and ensuring the smooth progress of the synchronization process. In summary, this embodiment can realize data synchronization across clusters and computer rooms while avoiding the risk of data loss during the synchronization process and ensuring the smooth progress of the synchronization process.

[0077] FIG3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call a computer program in the memory 330 to execute the steps of the data synchronization method, for example including: Receive target data sent by the message queue; the target data includes a target data change stream and a target operation sequence, and the target data change stream includes change data synchronized from the main cluster; Verifying the target data change stream using the target operation sequence, and performing abnormal correction on the target data change stream based on the verification result to obtain a corrected data change stream; Converting the modified data change stream into an executable distributed cache command; Distribute the distributed cache command to the disaster recovery cluster.

[0078] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0079] On the other hand, an embodiment of the present application further provides a computer program product, the computer program product comprising a computer program, the computer program may be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can perform the steps of the data synchronization method provided in the above embodiments, for example, including: Receive target data sent by the message queue; the target data includes a target data change stream and a target operation sequence, and the target data change stream includes change data synchronized from the main cluster; Verifying the target data change stream using the target operation sequence, and performing abnormal correction on the target data change stream based on the verification result to obtain a corrected data change stream; Converting the modified data change stream into an executable distributed cache command; Distribute the distributed cache command to the disaster recovery cluster.

[0080] On the other hand, an embodiment of the present application further provides a non-transitory computer-readable storage medium on which a computer program is stored, wherein the computer program is used to enable a processor to execute the steps of the data synchronization method provided in the above embodiments, for example, including: Receive target data sent by the message queue; the target data includes a target data change stream and a target operation sequence, and the target data change stream includes change data synchronized from the main cluster; Verifying the target data change stream using the target operation sequence, and performing abnormal correction on the target data change stream based on the verification result to obtain a corrected data change stream; Converting the modified data change stream into an executable distributed cache command; Distribute the distributed cache command to the disaster recovery cluster.

[0081] The non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0082] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data synchronization device, characterized in that: include: A receiving module, used to: receive target data sent by the message queue, and send the target data to the verification module; The target data includes a target data change stream and a target operation sequence, wherein the target data change stream includes change data synchronized from the main cluster; A verification module, used to verify the target data change stream using the target operation sequence, perform abnormal correction on the target data change stream based on the verification result to obtain a corrected data change stream, and send the corrected data change stream to the processing module; A processing module, configured to: convert the modified data change stream into an executable distributed cache command, and send the distributed cache command to a synchronization module; The synchronization module is used to distribute the distributed cache command to the disaster recovery cluster.

2. The data synchronization device according to claim 1, characterized in that: Also includes: Configuration module, used to store and manage the configuration information of other modules for other modules to call when they are running; The configuration information includes connection configuration information required to establish a synchronous connection with a cluster node, verification configuration information required to verify a data change stream, processing configuration information for processing the data change stream, and synchronization configuration information for the entire data synchronization process.

3. The data synchronization device according to claim 1, characterized in that: The converting the modified data change stream into an executable distributed cache command includes: Parsing the modified data change stream to obtain modified key value data; Filtering the modified key value data to obtain filtered key value data; For the filtered key-value data whose values ​​have changed, hash grouping is performed based on the keys corresponding to the values ​​to obtain multiple groups of target key-value data; The multiple groups of target key-value data are converted into executable distributed cache commands.

4. The data synchronization device according to claim 3, characterized in that: The step of parsing the modified data change stream to obtain modified key value data includes: Extracting valid instructions from a plurality of operation instructions in the modified data change stream; The operation data corresponding to the effective instruction is converted into key value data to obtain modified key value data.

5. The data synchronization device according to claim 4, characterized in that: The converting the multiple groups of target key-value data into executable distributed cache commands includes: Based on the mapping parameters, obtaining the target value of the target key-value data corresponding to the valid instruction from the multiple sets of target key-value data; The target value is converted into an executable distributed cache command.

6. The data synchronization device according to claim 1, characterized in that: The method of verifying the target data change stream by using the target operation sequence and performing abnormal correction on the target data change stream based on the verification result to obtain the corrected data change stream includes: Compare the target operation sequence with the operation sequence in the target node to obtain a first comparison result; the target node includes a master node or a slave node of the master cluster; If the first comparison result is consistent, then compare the operation identifier in the target operation sequence with the operation identifier in the target data change stream to obtain a second comparison result, and compare the timestamp in the target operation sequence with the timestamp in the target data change stream to obtain a third comparison result; If the first comparison result is inconsistent, initiating a request to the receiving module to call the operation sequence in the target node, and calling the operation sequence in the target node pulled by the receiving module; Modifying the target operation sequence based on the operation sequence in the target node to obtain a modified operation sequence; Compare the operation identifier in the correction operation sequence with the operation identifier in the target data change stream to obtain a second comparison result, and compare the timestamp in the correction operation sequence with the timestamp in the target data change stream to obtain a third comparison result; Based on the second comparison result and the third comparison result, the target data change flow is abnormally corrected to obtain a corrected data change flow.

7. The data synchronization device according to claim 6, characterized in that: The method of performing abnormal correction on the target data change stream based on the second comparison result and the third comparison result to obtain a corrected data change stream includes: If at least one of the second comparison result and the third comparison result is inconsistent, a request for calling the data change stream in the target node is initiated to the receiving module, and the data change stream in the target node pulled by the receiving module is called; The target data change flow is modified based on the data change flow in the target node to obtain a modified data change flow.

8. The data synchronization device according to claim 1, characterized in that: The target data sent by the receiving message queue includes: The message queue is monitored in real time, and the target data sent by the message queue is received when the data in the message queue changes.

9. The data synchronization device according to claim 1, characterized in that: The synchronization module is further used to: write the changed data in the main cluster into the message queue, and when at least two of the master node of the main cluster, the slave node of the main cluster and the message queue successfully write the changed data, send a write success message to the client in the node of the main cluster where the writing is successful.

10. A data synchronization method, characterized in that: include: Receive the target data sent by the message queue; The target data includes a target data change stream and a target operation sequence, wherein the target data change stream includes change data synchronized from the main cluster; Verifying the target data change stream using the target operation sequence, and performing abnormal correction on the target data change stream based on the verification result to obtain a corrected data change stream; Converting the modified data change stream into an executable distributed cache command; Distribute the distributed cache command to the disaster recovery cluster.

11. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the data synchronization method according to claim 10 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data synchronization method according to claim 10 are implemented.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data synchronization method according to claim 10 are implemented.

Images