System and method for guaranteeing consistency of replica data of main and auxiliary nodes

Abstract

The invention relates to the technical field of data consistency, and particularly discloses a main and auxiliary node replica data consistency guarantee system and method.The system comprises a node state mapping module, a node emergency disposal module and an operation switch module; the node state mapping module is used for mapping the states of the main and auxiliary nodes, the network environment condition and the duplicate data summary state; and judging whether the main and auxiliary nodes have an abnormal state or do not have a synchronization condition based on the mapping data, if the main and auxiliary nodes have the abnormal state or do not have the synchronization condition, sending a data synchronization closing operation request to the operation switch module, and sending an emergency processing instruction to the node emergency processing module, the node emergency disposal module carries out emergency disposal on the abnormal main and auxiliary nodes based on the emergency disposal instruction; and if the main and auxiliary nodes have no abnormal state and have operation response conditions, a data synchronization operation starting request is sent to the operation switch module. And monitoring, processing and mapping are carried out on reasons causing inconsistency of copy data between the main node and the auxiliary node.

Description

Technical Field

[0001] The present invention relates to the technical field of data consistency, and in particular to a system and method for ensuring data consistency of primary and secondary node replicas. Background Art

[0002] In a distributed storage environment, the primary node can perform read and write operations, while the secondary node can only perform read operations. A primary node can have multiple secondary nodes, while a secondary node can only have one primary node. When the primary node fails, the secondary node can be upgraded to the primary node. When the data of the primary node changes, the changed copy data will be synchronized to the secondary node to ensure that the copy data of the primary node and the secondary node are consistent. However, due to performance bottlenecks, network delays, node failures and other reasons, the copy data between the primary and secondary nodes may be inconsistent. The existing technology focuses too much on the realization of strong data consistency. If a full synchronization mechanism is introduced to unilaterally solve the consistency problem, it will affect the system availability and overall performance.

[0003] Moreover, there is currently a lack of processes for handling and providing feedback on the causes of inconsistency in replica data between primary and secondary nodes, as well as a lack of consistent data access methods under abnormal consistency states, which cannot ensure system availability and overall performance. Summary of the invention

[0004] In response to the above problems, an object of the present invention is to provide a system for ensuring the consistency of replica data between primary and secondary nodes. The system monitors, handles and maps the causes of inconsistency in replica data between primary and secondary nodes by setting up a node status mapping module, a node emergency handling module, a consistency check module, a free data storage module, a data audit module, a time machine module and an operation switch module, and provides a consistent data access method under consistency abnormal states, which can support the balanced realization of strong consistency and high performance of replica data between primary and secondary nodes for distributed storage.

[0005] The second object of the present invention is to provide a method for ensuring data consistency between primary and secondary node replicas.

[0006] The first technical solution adopted by the present invention is: a data consistency guarantee system for primary and secondary node replicas, including a node status mapping module, a node emergency handling module and an operation switch module;

[0007] The node status mapping module is used to map the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data; and based on the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data, determine whether the primary and secondary nodes are in an abnormal state or do not have the synchronization conditions. If the primary and secondary nodes are in an abnormal state or do not have the synchronization conditions, send a request to close the data synchronization operation to the operation switch module, and send an emergency disposal instruction to the node emergency disposal module; if the primary and secondary nodes are not in an abnormal state and have the operation response conditions, send a request to start the data synchronization operation to the operation switch module;

[0008] The node emergency handling module is used to receive the emergency handling instruction, and perform emergency handling on abnormal primary and secondary nodes based on the emergency handling instruction;

[0009] The operation switch module is used to receive the data synchronization operation start request or the data synchronization operation close request; and to start the data synchronization operation on the primary and secondary nodes based on the data synchronization operation start request, and to close the data synchronization operation on the primary and secondary nodes based on the data synchronization operation close request.

[0010] Preferably, the node emergency handling module is further used to send a remapping request to the node status mapping module after executing the emergency handling;

[0011] The node status mapping module is also used to receive the remapping request, and remap the status of the primary and secondary nodes, network environment conditions, and the summary status of the replica data based on the remapping request.

[0012] Preferably, the primary and secondary node replica data consistency assurance system further comprises: a consistency check module, a data audit module and a time machine module;

[0013] The node status mapping module is also used to receive a data acquisition operation request sent by a user, and based on the data acquisition operation request, call the consistency check module to perform a consistency check on the replica data of the primary and secondary nodes; if the replica data of the primary and secondary nodes are consistent and meet the conditions for responding to the data acquisition operation request, then send an operation request to start acquiring replica data from the primary and secondary nodes to the operation switch module; when the replica data of the primary and secondary nodes are inconsistent or do not meet the conditions for responding to the data acquisition operation request, then send an operation request to close acquiring replica data from the primary and secondary nodes to the operation switch module, and send a request to obtain replica data to the time machine module;

[0014] The data audit module is used to audit the replica data between the current primary and secondary nodes according to the preset audit frequency, and to copy the replica data between the primary and secondary nodes after consistency check and audit to the time machine module;

[0015] The time machine module is used to store the latest consistent replica data of the primary and secondary nodes; and receive the request for obtaining the replica data, and send the latest consistent replica data of the primary and secondary nodes to the user based on the request for obtaining the replica data;

[0016] The operation switch module is also used to receive the operation request to start obtaining replica data from the primary and secondary nodes or the operation request to close the operation request to obtain replica data from the primary and secondary nodes; to obtain replica data from the primary and secondary nodes based on the operation request to start obtaining replica data from the primary and secondary nodes; and to stop obtaining replica data from the primary and secondary nodes based on the operation request to close the operation request to obtain replica data from the primary and secondary nodes.

[0017] Preferably, the dimensions audited by the data audit module include: data standardization, data integrity, data accuracy, data consistency, data accessibility and data timeliness.

[0018] Preferably, the consistency check module is further used to: perform consistency check on the copy data downloaded by the primary node and the copy data downloaded by the secondary node according to a preset frequency.

[0019] Preferably, the consistency check includes a same-copy consistency check and an associated-copy consistency check; wherein the same-copy consistency refers to the current copy consistency, and the associated-copy consistency refers to the consistency of other data copies that are associated with the current copy data.

[0020] Preferably, the consistency checking module is further used to perform the following operations:

[0021] When the replica data of the primary and secondary nodes are inconsistent, the inconsistent replica data will be removed from the primary and secondary nodes to ensure the consistency of the replica data between the primary and secondary nodes.

[0022] Preferably, the primary and secondary node replica data consistency assurance system further comprises: a free data storage module;

[0023] The free data storage module is used to store the inconsistent copy data as free data, and call the data audit module to audit the free data; and send the free data that needs to be rewritten to the primary node, and then copy it to the secondary node via the primary node.

[0024] The second technical solution adopted by the present invention is: a method for ensuring data consistency of primary and secondary node replicas, comprising the following steps:

[0025] Map the status of the primary and secondary nodes, network environment conditions, and the summary status of the replica data;

[0026] Based on the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data, it is determined whether the primary and secondary nodes are in an abnormal state or do not meet the synchronization conditions. If the primary and secondary nodes are in an abnormal state or do not meet the synchronization conditions, the operation of synchronizing the data of the primary and secondary nodes is closed, and emergency measures are taken for the abnormal primary and secondary nodes; if the primary and secondary nodes are not in an abnormal state and meet the operation response conditions, the operation of synchronizing the data of the primary and secondary nodes is started.

[0027] Preferably, the method for ensuring data consistency of primary and secondary node replicas further comprises the following steps:

[0028] Receive a data acquisition operation request sent by the user, and perform a consistency check on the copy data of the primary and secondary nodes based on the data acquisition operation request. If the copy data of the primary and secondary nodes are consistent and have the conditions to respond to the data acquisition operation request, obtain the copy data from the primary and secondary nodes; when the copy data of the primary and secondary nodes are inconsistent or do not have the conditions to respond to the data acquisition operation request, obtain the latest consistent copy data of the primary and secondary nodes pre-stored in the time machine module.

[0029] Beneficial effects of the above technical solution:

[0030] (1) The present invention provides a data consistency assurance system for replicas of primary and secondary nodes, which monitors, handles and maps the causes of inconsistency in replica data between primary and secondary nodes by setting a node status mapping module, a node emergency handling module, a consistency check module, a free data storage module, a data audit module, a time machine module and an operation switch module, and provides a consistent data access method under a consistency abnormality state, which can support the balanced realization of strong consistency and high performance of replica data between primary and secondary nodes for distributed storage.

[0031] (2) The present invention proposes a high-performance consistency assurance system for master and slave node replicas for distributed storage. Through a node status mapping module, a node emergency handling module, a time machine module, etc., it realizes the mapping of the status of the master and slave nodes, network environment conditions, and the summary status of the replica data. It can promptly handle and provide feedback on the causes of inconsistency in the replica data between the master and slave nodes; it provides cross-access between the time machine data and the current data in a specific state, supports the response to the replica data acquisition operation request during the current data synchronization operation or the node emergency handling process, and meets high performance without losing consistency.

[0032] (3) The present invention also designs a free data storage module to store inconsistent free data between the current primary and secondary nodes, including: 1) data stored in the primary node but not stored in the secondary node; 2) data not stored in the primary node but stored in the secondary node; 3) data stored in both the primary and secondary nodes but with inconsistent data item contents.

[0033] (4) The present invention also designs a data audit module to regularly audit the copy data and rewrite the free data to avoid the loss of key and important data and ensure the consistency and integrity of the copy data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the structure of a system for ensuring data consistency of primary and secondary node replicas provided by an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of the working paths of a node status mapping module, a node emergency handling module and an operation switch module provided by an embodiment of the present invention;

[0036] Figure 3 A schematic diagram of the working paths of a consistency check module, a free data storage module and a data audit module provided by an embodiment of the present invention;

[0037] Figure 4 A schematic diagram of a time machine module and a current data cross-access path provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following detailed description of the embodiments of the present invention is further described in detail in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention, that is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.

[0039] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more than two; the terms “first”, “second”, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0040] Embodiment 1

[0041] like Figure 1 As shown, an embodiment of the present invention provides a primary and secondary node replica data consistency assurance system, including a node status mapping module, a node emergency handling module, a consistency check module, a free data storage module, a data audit module, a time machine module and an operation switch module;

[0042] The node status mapping module is used to map the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data; based on the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data, it is judged whether the primary and secondary nodes are in an abnormal state or do not have the synchronization conditions; if the primary and secondary nodes are in an abnormal state or do not have the synchronization conditions, a request to close the data synchronization operation is sent to the operation switch module, and an emergency handling instruction is sent to the node emergency handling module; if the primary and secondary nodes are not in an abnormal state and have the operation response conditions, a request to start the data synchronization operation is sent to the operation switch module;

[0043] The node status mapping module is also used to receive data acquisition operation requests sent by users, and based on the data acquisition operation requests, call the consistency check module to perform consistency check on the copy data of the primary and secondary nodes. If the copy data of the primary and secondary nodes are consistent and meet the conditions for responding to the data acquisition operation request, an operation request to start obtaining copy data from the primary and secondary nodes is sent to the operation switch module; when the copy data of the primary and secondary nodes are inconsistent or do not meet the conditions for responding to the data acquisition operation request, an operation request to close obtaining copy data from the primary and secondary nodes is sent to the operation switch module, and a request to obtain copy data is sent to the time machine module.

[0044] The node emergency handling module is used to receive the emergency handling instruction, and perform emergency handling on abnormal primary and secondary nodes based on the emergency handling instruction; after the emergency handling is completed, a remapping request is sent to the node status mapping module.

[0045] The consistency check module is used to perform consistency check on the copy data of the primary and secondary nodes according to a preset frequency. If the copy data between the current primary and secondary nodes are inconsistent, the inconsistent copy data will be removed from the primary node and the secondary node. After the inconsistent copy data is removed, the copy data between the primary and secondary nodes are consistent. The data audit module is called to audit the copy data between the current primary and secondary nodes. After the audit is correct, the copy data between the current primary and secondary nodes is copied to the time machine module.

[0046] The data audit module is used to audit the replica data between the current primary and secondary nodes according to a preset audit frequency, and to copy the replica data between the primary and secondary nodes after consistency check and audit to the time machine module.

[0047] The time machine module is used to store the latest consistent replica data of the primary and secondary nodes; and receive the request for obtaining the replica data, and send the latest consistent replica data of the primary and secondary nodes to the user based on the request for obtaining the replica data.

[0048] The free data storage module is used to store inconsistent copy data as free data, and call the data audit module to audit the free data; and send the free data that needs to be rewritten to the primary node through the free data storage module, and then copy it to the secondary node via the primary node.

[0049] The operation switch module is used to receive the data synchronization operation start request or the data synchronization operation stop request; based on the data synchronization operation start request, start the operation of data synchronization between the primary and secondary nodes; based on the data synchronization operation stop request, stop the operation of data synchronization between the primary and secondary nodes;

[0050] The operation switch module is also used to receive the operation request to start obtaining replica data from the primary and secondary nodes or the operation request to close the operation request to obtain replica data from the primary and secondary nodes; based on the operation request to start obtaining replica data from the primary and secondary nodes, obtain replica data from the primary and secondary nodes; based on the operation request to close the operation request to obtain replica data from the primary and secondary nodes, stop obtaining replica data from the primary and secondary nodes.

[0051] (1) Node status mapping module;

[0052] like Figure 2 As shown, the node status mapping module is used to map the status of the primary and secondary nodes, network environment conditions, and the summary status of the replica data; wherein the status of the primary and secondary nodes includes IP address, node health identifier (True-indicates that the node is healthy and can support operations such as reading and writing / False-indicates that the node cannot support operations such as reading and writing), node activity (Active-active / Out-unable to detect node status / Fault-fault status), node pressure value (involving disk pressure, process pressure, memory pressure), version, and available resource capacity;

[0053] Network environment conditions refer to supporting users to manage the network where nodes are running by mapping the current network environment conditions, thereby ensuring efficient network performance to support node operation, including hardware, software, protocols, connection status, etc., and supporting accurate positioning of network faults;

[0054] The summary status of the replica data includes data source, data description (covering type, format, creation time, validity period, etc.), field summary, data quality level, etc.

[0055] The status of the primary and secondary nodes, network environment conditions, and replica data summary status are mapped in the following ways, including:

[0056] ① Discover the network and network devices; ② Generate a summary of the network infrastructure; ③ Generate the primary and secondary node topology structure; ④ Display the node status based on the node topology structure; ⑤ Display the summary status of the replica data stored on the node.

[0057] 1) Data synchronization operation;

[0058] The node status mapping module is used to determine whether the primary and secondary nodes are in an abnormal state or do not meet the synchronization conditions based on the status of the primary and secondary nodes, network environment conditions, and the summary status of the replica data, including:

[0059] Based on the status of the primary and secondary nodes, it is determined whether the primary and secondary nodes are in an abnormal state. The abnormal state of the primary and secondary nodes includes but is not limited to: the node state cannot be detected (out) and the fault state (fault); when the primary and secondary nodes are in an abnormal state, the corresponding primary and secondary nodes are abnormal primary and secondary nodes;

[0060] And based on the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data, it is determined whether the primary and secondary nodes do not meet the synchronization conditions. For example: when the status of the primary and secondary nodes is abnormal, or the network environment conditions are abnormal, or the summary status of the replica data is abnormal, then it is determined that the primary and secondary nodes do not meet the synchronization conditions; and when the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data are all normal, then it is determined that the primary and secondary nodes meet the synchronization conditions.

[0061] If the primary and secondary nodes are in an abnormal state or do not meet the synchronization conditions, a request to close the data synchronization operation is sent to the operation switch module, and an emergency handling instruction is sent to the node emergency handling module; if the primary and secondary nodes are not in an abnormal state and meet the operation response conditions, a request to start the data synchronization operation is sent to the operation switch module.

[0062] 2) Data acquisition operation;

[0063] The node status mapping module is also used to receive data acquisition operation requests sent by users, call the consistency check module based on the data acquisition operation request to perform consistency check on the copy data of the primary and secondary nodes, and judge whether the primary and secondary nodes have the conditions to respond to the data acquisition operation request based on the status of the primary and secondary nodes, network environment conditions, and copy data summary status; if the copy data of the primary and secondary nodes are consistent and have the conditions to respond to the data acquisition operation request, send an operation request to start obtaining copy data from the primary and secondary nodes to the operation switch module; when the copy data of the primary and secondary nodes are inconsistent or do not have the conditions to respond to the data acquisition operation request, send an operation request to close the acquisition of copy data from the primary and secondary nodes to the operation switch module, and send a request to obtain copy data to the time machine module.

[0064] Judging whether the primary and secondary nodes are qualified to respond to the data acquisition operation request based on the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data includes:

[0065] When any of the following conditions exist in the primary and secondary nodes, it is determined that the primary and secondary nodes do not have the conditions to respond to the data acquisition operation request: ① data synchronization operation or node emergency disposal operation is in progress, ② there is an inconsistency problem that needs to be handled, ③ the status of the primary and secondary nodes is abnormal, ④ the network environment conditions are abnormal, ⑤ the replica data summary status is abnormal; when the status of the primary and secondary nodes is abnormal, the network environment conditions are abnormal, or the replica data summary status is abnormal, an emergency disposal instruction is sent to the node emergency disposal module;

[0066] When the primary and secondary nodes are not in data synchronization operation or node emergency disposal operation status, and there are no abnormalities in the status of the primary and secondary nodes, network environment conditions, and replica data summary status, it is determined that the primary and secondary nodes are qualified to respond to data acquisition operation requests.

[0067] Both the request to start data synchronization operation and the request to start obtaining replica data from primary and secondary nodes are operation start requests, and both the request to close data synchronization operation and the request to close obtaining replica data from primary and secondary nodes are operation close requests.

[0068] The node status mapping module is also used to receive a remapping request sent by the node emergency handling module, remap the status of the primary and secondary nodes, network environment conditions, and the summary status of the replica data based on the remapping request, and again judge whether the primary and secondary nodes are in an abnormal state or do not meet the synchronization conditions based on the remapped status of the primary and secondary nodes, network environment conditions, and the summary status of the replica data; when the primary and secondary nodes are in no abnormal state and meet the synchronization conditions, send a request to start the data synchronization operation to the operation switch module; when the primary and secondary nodes are still in an abnormal state or do not meet the synchronization conditions, send an emergency handling instruction to the node emergency handling module again, and perform fault repair and remapping in a cycle until the primary and secondary nodes are in no abnormal state and meet the synchronization conditions;

[0069] And based on the status of the remapped primary and secondary nodes, network environment conditions, and copy data summary status, determine whether the primary and secondary nodes are qualified to respond to data acquisition operation requests; when the primary and secondary nodes are qualified to respond to data acquisition operation requests, and the copy data of the primary and secondary nodes are consistent, send an operation request to start obtaining copy data from the primary and secondary nodes to the operation switch module; when the primary and secondary nodes do not have the conditions to respond to the data acquisition operation request, send an emergency handling instruction to the node emergency handling module again, and perform fault repair and remapping in a cycle until the primary and secondary nodes are qualified to respond to data acquisition operation requests.

[0070] (2) Node emergency response module;

[0071] like Figure 2As shown, the node emergency handling module is used to receive the emergency handling instructions, and perform node emergency handling on abnormal primary and secondary nodes based on the emergency handling instructions. For example: when the node status cannot be detected or the faulty status of the primary node exceeds the set response time, the node status mapping module determines that the primary node is offline, and calls the node emergency handling module to re-elect the primary node. The re-election of the primary node can adopt a voting mechanism, or adopt a pre-set priority to upgrade one of the secondary nodes to the primary node, and then optimize the configuration based on the re-elected primary node (such as node load balancing, data rebalancing, etc.).

[0072] After the emergency handling is completed, a remapping request is sent to the node status mapping module.

[0073] (3) consistency check module;

[0074] like Figure 3 As shown, the consistency check module is used to perform consistency check on the copy data downloaded by the primary node and the copy data downloaded by the secondary node according to a preset frequency; the preset frequency can be configured by itself, and generally a consistency check is required after a data synchronization operation, and a consistency check is also required before processing a data acquisition request from a client.

[0075] If any of the following situations occurs, the replica data between the current primary and secondary nodes is determined to be inconsistent:

[0076] ① Data stored in the primary node is not stored in the secondary node; ② Data not stored (or deleted) in the primary node is still stored in the secondary node; ③ Data stored in both the primary and secondary nodes, but the contents of the stored data items are inconsistent.

[0077] If the replica data between the current primary and secondary nodes are inconsistent, the inconsistent replica data will be removed from the primary and secondary nodes. After the inconsistent replica data is removed, the replica data between the primary and secondary nodes are consistent. The data audit module is called according to the preset audit frequency to audit the replica data between the current primary and secondary nodes. After the audit is correct, the replica data between the current primary and secondary nodes is copied to the time machine module.

[0078] Furthermore, in one embodiment, it also includes: storing inconsistent replica data as free data in a free data storage module, calling a data audit module to audit the free data stored in the free data storage module; determining after the audit whether the free data needs to be rewritten to the replica, and for the free data that needs to be rewritten after the audit, calling the free data storage module to send the free data that needs to be rewritten to the primary node, and then copying it to the secondary node via the primary node to achieve consistency of the replica data between the primary and secondary nodes.

[0079] If the replica data between the current primary and secondary nodes are consistent, the data audit module is directly called according to the preset audit frequency to audit the replica data of the current primary and secondary nodes. After the audit is correct, the replica data between the current primary and secondary nodes is copied to the time machine module.

[0080] The consistency check includes the same copy consistency check and the associated copy consistency check; the same copy consistency refers to the consistency of the current copy; the associated copy consistency refers to the consistency of other data copies that are associated with the current copy data. The associated copy consistency check only involves the first-degree associated copy traced back from the current copy, and no mandatory check is performed on the consistency of the second-degree associated copy traced back based on the first-degree associated copy.

[0081] (4) Free data storage module;

[0082] The free data storage module is used to store free data (inconsistent data) between the current primary and secondary nodes. The free data includes: ① data stored in the primary node but not stored in the secondary node; ② data not stored in the primary node but stored in the secondary node; ③ data stored in both the primary and secondary nodes but with inconsistent data item contents.

[0083] The free data between the current primary and secondary nodes is stored, including: removing inconsistent data from the primary node and the secondary node, and storing them in the free data storage module.

[0084] The structure of free data storage is, for example: {primary key; data subject category; data value category; data value field; business description; association relationship; business rules; statistical caliber; corresponding data standard; data quality rules; data source; security information; label information; maintenance person in charge; creation time; pointing to storage address};

[0085] The specific descriptions of each data item are as follows:

[0086] Primary key: unique ID code of data.

[0087] Data subject category: customized by application, distinguishing between business data and management data, such as contract data, product data, location data, event data, resource item data, etc.

[0088] Data value categories: divided into basic data and service data; basic data refers to data assets that are directly or indirectly collected, precipitated, processed, or introduced through a third party based on technical tools and means, in accordance with laws, regulations and relevant provisions; service data refers to the refined information that is directly involved in business scenarios with measurable value after the basic data is processed, driven by data analysis.

[0089] Data value areas: data application areas, such as business management, customer management, risk management, and operations management.

[0090] Business description: Description of the business meaning of the data.

[0091] Association relationship: the association relationship with other data.

[0092] Business rules: business management rules for data, business quality rules, etc.

[0093] Statistical caliber: logical caliber of data processing.

[0094] Corresponding data standards: data standard information for data mapping.

[0095] Data quality rules: data quality requirements and verification rules.

[0096] Data source: data source with authority and reliable quality, distinguishing between public data and non-public data.

[0097] Security information: describes whether the data is sensitive information, data classification, whether it is desensitized, desensitization type, and the scope of authority of users at all levels.

[0098] Label information: data-specific annotation information.

[0099] Maintenance Manager: The technician who maintains the data.

[0100] Creation Time: The time when the data was created.

[0101] Points to storage location: the original storage location that the data points to (i.e., points to the original node).

[0102] (5) Data audit module;

[0103] The data audit module is used to regularly audit the replica data of the primary and secondary nodes based on the preset audit frequency to avoid the loss of important data; it is also used to audit the free data and determine whether the free data needs to be rewritten to the replica;

[0104] Determine whether the free data needs to be rewritten to the replica, including:

[0105] Based on the characteristics of the free data, such as its source, description, field, quality level, and association with the existing replica data, the necessity of rewriting the free data into the replica is evaluated. If the evaluation result is greater than the preset rewrite allowable value, the free data is rewritten into the replica.

[0106] For example, free data that is allowed to be rewritten into a replica should meet the following four characteristics: ① reliable data source, ② complete description, ③ data field content meets reference standards, and ④ high data quality level; on the basis of meeting the above four conditions, it can also meet the following conditions: ⑤ high correlation with existing replica data; the weight of the feature that should be met is greater than the weight of the feature that can be met.

[0107] Data audit dimensions include data standardization, data integrity, data accuracy, data consistency, data accessibility, and data timeliness. Among them, data standardization is mainly audited through dimensions such as data standards, data models, metadata, business domain reference models, and security specifications.

[0108] Data integrity is mainly audited through dimensions such as data element integrity and data record integrity;

[0109] Data accuracy is mainly audited through dimensions such as data content correctness, data format compliance, data duplication rate, and dirty data occurrence rate;

[0110] Data consistency is mainly audited through dimensions such as same data consistency and related data consistency;

[0111] Data accessibility is mainly audited through dimensions such as accessibility, integrity, and availability;

[0112] Data timeliness is mainly audited through dimensions such as accuracy based on time periods, timeliness based on time points, and sequentiality.

[0113] (6) Time machine module;

[0114] The time machine module is used to call the consistency check module to perform a consistency check on the copy data of the primary and secondary nodes after the latest data synchronization occurs. After the copy data of the primary and secondary nodes pass the consistency check, it is preferably used to call the data audit module to perform a data audit on the copy data of the primary and secondary nodes. After the audit is correct, the copy data of the primary and secondary nodes are stored; that is, the copy data that passes the consistency check and data audit is stored in the time machine module.

[0115] The consistent copy data stored in the time machine module can support the data synchronization operation of the primary and secondary nodes or the data acquisition operation request response during the node emergency disposal process. For example, when the node status mapping module receives the user's data acquisition operation request, the node status mapping module reads that the current primary and secondary nodes are performing data synchronization operations or node emergency disposal operations, and do not have the conditions to respond to the data acquisition operation request. It sends an operation request to close the acquisition of copy data from the primary and secondary nodes to the operation switch module, and sends a request to obtain copy data to the time machine module. Based on the request to obtain copy data, the time machine module sends the latest version of the stored consistent copy data to the requesting end (such as the client), meeting high performance without losing consistency.

[0116] (7) Operation switch module;

[0117] like Figure 4As shown, the operation switch module is used to receive a request to start a data synchronization operation or a request to close a data synchronization operation; based on the request to close the data synchronization operation, the operation of data synchronization between the primary and secondary nodes is closed, and based on the request to start the data synchronization operation, the operation of data synchronization between the primary and secondary nodes is opened;

[0118] For example, when the status of the primary and secondary nodes is normal and the synchronization conditions are met, the operation of synchronizing the data of the primary and secondary nodes is started; when the status of the primary and secondary nodes is abnormal or the synchronization conditions are not met, the operation of synchronizing the data of the primary and secondary nodes is closed; the abnormal status of the primary and secondary nodes includes but is not limited to: out (unable to detect node status); fault (fault status).

[0119] The operation switch module is also used to receive an operation request sent by the node status mapping module to start obtaining replica data from the primary and secondary nodes or to close the operation request to obtain replica data from the primary and secondary nodes; based on the operation request to start obtaining replica data from the primary and secondary nodes, start obtaining replica data from the primary and secondary nodes; based on the operation request to close obtaining replica data from the primary and secondary nodes, stop obtaining replica data from the primary and secondary nodes.

[0120] Embodiment 2

[0121] An embodiment of the present invention provides a method for ensuring data consistency of primary and secondary node replicas, comprising the following steps:

[0122] Map the status of the primary and secondary nodes, network environment conditions, and the summary status of the replica data;

[0123] Based on the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data, it is determined whether the primary and secondary nodes are in an abnormal state or do not meet the synchronization conditions. If the primary and secondary nodes are in an abnormal state or do not meet the synchronization conditions, the operation of synchronizing the data of the primary and secondary nodes is closed, and emergency measures are taken for the abnormal primary and secondary nodes; if the primary and secondary nodes are not in an abnormal state and meet the operation response conditions, the operation of synchronizing the data of the primary and secondary nodes is started.

[0124] Furthermore, in one embodiment, the method for ensuring data consistency of the primary and secondary node replicas further includes the following steps:

[0125] Receive a data acquisition operation request sent by the user, and perform a consistency check on the copy data of the primary and secondary nodes based on the data acquisition operation request. If the copy data of the primary and secondary nodes are consistent and have the conditions to respond to the data acquisition operation request, obtain the copy data from the primary and secondary nodes; when the copy data of the primary and secondary nodes are inconsistent or do not have the conditions to respond to the data acquisition operation request, obtain the latest consistent copy data of the primary and secondary nodes pre-stored in the time machine module.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0130] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0131] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A system for ensuring data consistency between primary and secondary node replicas, characterized in that: It includes a node status mapping module, a node emergency handling module and an operation switch module; The node status mapping module is used to map the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data; and based on the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data, determine whether the primary and secondary nodes are in an abnormal state or do not have the synchronization conditions. If the primary and secondary nodes are in an abnormal state or do not have the synchronization conditions, send a request to close the data synchronization operation to the operation switch module, and send an emergency disposal instruction to the node emergency disposal module; if the primary and secondary nodes are not in an abnormal state and have the operation response conditions, send a request to start the data synchronization operation to the operation switch module; The node emergency handling module is used to receive the emergency handling instruction, and perform emergency handling on abnormal primary and secondary nodes based on the emergency handling instruction; The operation switch module is used to receive the request to start the data synchronization operation or the request to close the data synchronization operation; And based on the data synchronization operation start request, the data synchronization operation on the primary and secondary nodes is started, and based on the data synchronization operation stop request, the data synchronization operation on the primary and secondary nodes is stopped.

2. The system for ensuring data consistency between primary and secondary node replicas according to claim 1, characterized in that: The node emergency handling module is further used to send a remapping request to the node status mapping module after executing the emergency handling; The node status mapping module is also used to receive the remapping request, and remap the status of the primary and secondary nodes, network environment conditions, and the summary status of the replica data based on the remapping request.

3. The system for ensuring data consistency between primary and secondary node replicas according to claim 1, characterized in that: Also includes: Consistency check module, data audit module and time machine module; The node status mapping module is also used to receive a data acquisition operation request sent by a user, and based on the data acquisition operation request, call the consistency check module to perform a consistency check on the replica data of the primary and secondary nodes. If the replica data of the primary and secondary nodes are consistent and meet the conditions for responding to the data acquisition operation request, an operation request to start acquiring replica data from the primary and secondary nodes is sent to the operation switch module; When the replica data of the primary and secondary nodes are inconsistent or do not meet the conditions for responding to the data acquisition operation request, a request to close the operation of obtaining replica data from the primary and secondary nodes is sent to the operation switch module, and a request to obtain replica data is sent to the time machine module; The data audit module is used to audit the replica data between the current primary and secondary nodes according to the preset audit frequency, and to copy the replica data between the primary and secondary nodes after consistency check and audit to the time machine module; The time machine module is used to store the latest consistent replica data of the primary and secondary nodes; and receive the request for obtaining the replica data, and send the latest consistent replica data of the primary and secondary nodes to the user based on the request for obtaining the replica data; The operation switch module is further used to receive the operation request to start obtaining the replica data from the primary and secondary nodes or to close the operation request to obtain the replica data from the primary and secondary nodes; Based on the operation request to start obtaining the replica data from the primary and secondary nodes, obtaining the replica data from the primary and secondary nodes; And based on the operation request to close the acquisition of replica data from the primary and secondary nodes, stop acquiring the replica data from the primary and secondary nodes.

4. The system for ensuring data consistency between primary and secondary node replicas according to claim 3, characterized in that: The dimensions audited by the data audit module include: data standardization, data integrity, data accuracy, data consistency, data accessibility and data timeliness.

5. The system for ensuring data consistency between primary and secondary node replicas according to claim 3, characterized in that: The consistency check module is also used to: perform consistency check on the copy data downloaded by the primary node and the copy data downloaded by the secondary node according to a preset frequency.

6. The system for ensuring data consistency between primary and secondary node replicas according to claim 3, characterized in that: The consistency check includes a same-copy consistency check and an associated-copy consistency check; wherein the same-copy consistency refers to the current copy consistency, and the associated-copy consistency refers to the consistency of other data copies that are associated with the current copy data.

7. The system for ensuring data consistency between primary and secondary node replicas according to claim 3, characterized in that: The consistency checking module is also used to perform the following operations: When the replica data of the primary and secondary nodes are inconsistent, the inconsistent replica data will be removed from the primary and secondary nodes to ensure the consistency of the replica data between the primary and secondary nodes.

8. The system for ensuring data consistency between primary and secondary node replicas according to claim 7, characterized in that: Also included is a free data storage module; The free data storage module is used to store the inconsistent copy data as free data, and call the data audit module to audit the free data; And the free data that needs to be rewritten is sent to the primary node, and then copied to the secondary node through the primary node.

9. A method for ensuring data consistency between primary and secondary node replicas, characterized in that: The following steps are involved: Map the status of the primary and secondary nodes, network environment conditions, and the summary status of the replica data; Based on the status of the primary and secondary nodes, the network environment conditions, and the summary status of the replica data, it is determined whether the primary and secondary nodes are in an abnormal state or do not meet the synchronization conditions. If the primary and secondary nodes are in an abnormal state or do not meet the synchronization conditions, the operation of synchronizing the data of the primary and secondary nodes is closed, and emergency measures are taken for the abnormal primary and secondary nodes; if the primary and secondary nodes are not in an abnormal state and meet the operation response conditions, the operation of synchronizing the data of the primary and secondary nodes is started.

10. The method for ensuring data consistency of primary and secondary node replicas according to claim 9, characterized in that: The following steps are also included: Receive a data acquisition operation request sent by the user, and perform a consistency check on the copy data of the primary and secondary nodes based on the data acquisition operation request. If the copy data of the primary and secondary nodes are consistent and have the conditions to respond to the data acquisition operation request, obtain the copy data from the primary and secondary nodes; when the copy data of the primary and secondary nodes are inconsistent or do not have the conditions to respond to the data acquisition operation request, obtain the latest consistent copy data of the primary and secondary nodes pre-stored in the time machine module.

Images