Data auditing method and device based on distributed cache, equipment and medium

Abstract

The invention provides a data auditing method and device based on distributed cache, equipment and a medium, and belongs to the technical field of data processing. The method comprises the steps that a first key is obtained from a first node of a cluster, the first key is used for identifying first to-be-audited data mapped to a first data fragment, the first data fragment is mapped to the first node, the cluster is used for distributed caching, and the first key corresponds to a first existence state, the first existence state represents that the first key exists in a second node of the cluster, second to-be-audited data corresponding to the first key is mapped to a second data fragment, and the second data fragment is mapped to the second node, and auditing processing is performed on the first to-be-audited data and the second to-be-audited data to obtain a first auditing processing result. The distributed cache of the to-be-audited data can be realized, so that the horizontal scalability of the data auditing processing process is effectively improved, and the data auditing processing efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data audit method, device, equipment and medium based on distributed cache. Background Art

[0002] With the vigorous development of Internet business and the continuous promotion of digital business, the amount of business data has shown an explosive growth trend. Data auditing refers to the process of comprehensive review, verification and evaluation of data, mainly used to ensure the accuracy, completeness and consistency of data, and to verify whether the data meets the predetermined standards and rules.

[0003] In the related art, when performing data auditing, a database stored procedure centralized computing architecture is usually adopted, which has a slow processing speed and poor scalability. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the purpose of the present invention is to propose a data audit method, device, equipment and medium based on distributed cache, which can realize distributed caching of the data to be audited, thereby effectively improving the horizontal scalability of the data audit processing process and improving the data audit processing efficiency.

[0006] To achieve the above-mentioned purpose, the data audit method based on distributed cache proposed in the first aspect of the present disclosure includes:

[0007] Obtaining a first key from a first node of a cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to the first node, and the cluster is used for the distributed cache;

[0008] Determine that the first key corresponds to a first existence state, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, second to-be-audited data corresponding to the first key has been mapped to a second data shard, and the second data shard has been mapped to the second node; and

[0009] Auditing the first data to be audited and the second data to be audited to obtain a first auditing result.

[0010] To achieve the above-mentioned purpose, a data audit device based on distributed cache is proposed in a second aspect of the present disclosure, comprising:

[0011] An acquisition module, configured to acquire a first key from a first node of a cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to the first node, and the cluster is used for the distributed cache;

[0012] a determination module, configured to determine that the first key corresponds to a first existence state, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, second to-be-audited data corresponding to the first key has been mapped to a second data shard, and the second data shard has been mapped to the second node; and

[0013] A processing module is used to perform audit processing on the first data to be audited and the second data to be audited to obtain a first audit processing result.

[0014] The computer device proposed in the third aspect embodiment of the present disclosure includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the data audit method based on distributed cache proposed in the first aspect embodiment of the present disclosure is implemented.

[0015] The fourth aspect embodiment of the present disclosure proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the data audit method based on distributed cache as proposed in the first aspect embodiment of the present disclosure.

[0016] The fifth aspect embodiment of the present disclosure proposes a computer program product. When the instructions in the computer program product are executed by a processor, the data audit method based on distributed cache proposed in the first aspect embodiment of the present disclosure is executed.

[0017] The data audit method, apparatus, device and medium based on distributed cache provided by the present disclosure obtain a first key from the first node of the cluster, wherein the first key is used to identify the first data to be audited, the first data to be audited has been mapped to the first data shard, and the first data shard has been mapped to the first node, the cluster is used for distributed cache, and it is determined that the first key corresponds to a first existence state, wherein the first existence state indicates that the first key exists in the second node of the cluster, the first node and the second node are different, the second data to be audited corresponding to the first key has been mapped to the second data shard, and the second data shard has been mapped to the second node, and the first data to be audited and the second data to be audited are audited to obtain a first audit processing result, thereby realizing distributed caching of the data to be audited, thereby effectively improving the horizontal scalability of the data audit processing process and improving the data audit processing efficiency.

[0018] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 It is a flowchart of a data audit method based on distributed cache proposed in one embodiment of the present disclosure;

[0021] Figure 2 is a flow chart of a data audit method based on distributed cache proposed in another embodiment of the present disclosure;

[0022] Figure 3 It is a schematic diagram of the structure of a data auditing solution proposed in the present disclosure;

[0023] Figure 4 is a flow chart of a data audit method based on distributed cache proposed in another embodiment of the present disclosure;

[0024] Figure 5 is a data distribution diagram proposed according to the present disclosure;

[0025] Figure 6 is a schematic diagram of data distribution change proposed according to the present disclosure;

[0026] Figure 7 is another schematic diagram of data distribution changes proposed according to the present disclosure;

[0027] Figure 8 is a schematic diagram of a distributed storage solution proposed according to the present disclosure;

[0028] Fig. 9 It is a schematic diagram of the data audit process proposed in this disclosure;

[0029] Fig.10 It is a structural diagram of a data audit device based on distributed cache proposed in one embodiment of the present disclosure;

[0030] Fig.11 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present disclosure, and are not to be construed as limitations of the present disclosure. On the contrary, the embodiments of the present disclosure include all changes, modifications, and equivalents that fall within the spirit and connotation of the appended claims.

[0032] Figure 1 It is a flowchart of a data audit method based on distributed cache proposed in an embodiment of the present disclosure.

[0033] Among them, it should be noted that the executor of the data audit method based on distributed cache in this embodiment is a data audit device based on distributed cache, which can be implemented by software and / or hardware. The device can be configured in a computer device, and the computer device can include but is not limited to a terminal, a server, etc. For example, the terminal can be a mobile phone, a handheld computer, etc.

[0034] like Figure 1 As shown, the data audit method based on distributed cache includes:

[0035] S101: Obtain a first key from a first node in a cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to a first node, and the cluster is used for distributed caching.

[0036] A cluster refers to multiple computing devices (which can be servers, computers, or other devices) connected together to work together and provide shared computing resources. These devices are usually connected through a network and are logically regarded as a single entity. By assigning computing tasks to different devices in the cluster, goals such as high-performance computing, load balancing, and high availability can be achieved. Clusters can be applied to large-scale data processing, scientific computing, Web services, and cloud computing.

[0037] A node refers to a single computing device or server in a cluster. It can be a physical computer, virtual machine, container, or other computing resource. A node is the basic unit of a cluster, and it has its own processing power, storage resources, and network connections. A node can perform various computing tasks and work with other nodes to complete cluster-level tasks. The first node refers to the node in the cluster that is used to obtain the first key.

[0038] It is understandable that there may be a large amount of data to be audited in the application scenario. In order to facilitate the identification of different data to be audited, corresponding identification identifiers can be configured for different data to be audited in the embodiment of the present disclosure to generate corresponding key-value pairs. The first key and the first data to be audited refer to a key-value pair.

[0039] The data shard may refer to a data object container or an atomic unit of data migration. The first data shard refers to the data shard to which the first data to be audited is mapped.

[0040] In the disclosed embodiment, when the first key is obtained from the first node of the cluster, the obtained first key can be used as a retrieval basis to quickly and accurately determine the first data to be audited from the multiple data to be audited.

[0041] S102: Determine that the first key corresponds to a first existence state, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, the second data to be audited corresponding to the first key has been mapped to the second data shard, and the second data shard has been mapped to the second node.

[0042] The first existence state refers to a state in which the first key exists in the second node of the cluster. The second node may refer to any node among the multiple nodes of the cluster except the first node.

[0043] It is understandable that in order to ensure high availability and fault tolerance of data, the same data can be stored in different nodes of the cluster, so that when the number of requests in the system increases, the requests can be distributed to multiple nodes for processing, thereby reducing the pressure on a single node and maintaining the stability and performance of the system. When storage demand increases, new nodes can be added to expand storage capacity without reconstructing or shutting down the entire system, thus achieving horizontal expansion of the system.

[0044] The second data shard refers to the data shard to which the second data to be audited is mapped.

[0045] That is to say, in the embodiment of the present disclosure, after obtaining the first key from the first node of the cluster, if the second data to be audited corresponding to the first key is mapped in the second node of the cluster, it can be determined that the first key corresponds to the first existence state, thereby triggering subsequent audit processing of the first data to be audited and the second data to be audited, so as to improve the reliability and consistency of data in the distributed system, reduce the impact of data errors or inconsistencies, and improve the reliability and stability of the entire system.

[0046] S103: Auditing the first data to be audited and the second data to be audited to obtain a first auditing result.

[0047] The first audit processing result refers to the processing result obtained after auditing the first data to be audited and the second data to be audited, and can be used to indicate whether the first data to be audited and the second data to be audited meet the expected requirements.

[0048] For example, in the implementation of the present disclosure, when the audit processing is performed on the first set of data to be audited and the second set of data to be audited, it can be based on any of the following methods:

[0049] (1) Data comparison method: Compare two sets of data one by one to check whether the data is completely consistent. You can compare each field of the data or specific key fields, such as unique identifiers, date and time, etc. If there are differences, you can further analyze and deal with the reasons for the differences.

[0050] (2) Checksum verification method: Generate checksums or hash values ​​for each of the two sets of data, and then compare whether these checksums or hash values ​​match. Commonly used algorithms include MD5, SHA-1, SHA-256, etc. If the checksums do not match, it means that the data is different or has been tampered with.

[0051] (3) Difference report method: Compare two sets of data by generating a difference report. Use a comparison tool or script to generate a data difference report that shows the differences between the two sets of data. This can help quickly locate and analyze data inconsistencies.

[0052] (4) Data consistency check method: Compare the consistency marks of two sets of data, such as version numbers, status marks, etc. Ensure the consistency of the two sets of data in key attributes to verify the correctness of the data.

[0053] (5) Integrity check method: Compare the integrity information of the two sets of data, such as the number of records, data range, etc. Ensure that the two sets of data are consistent in quantity and range to ensure that the data is not lost or accidentally deleted.

[0054] (6) Data sampling method: Randomly select a portion of data from two sets of data for comparison. By comparing the sampled data, the consistency between the two sets of data can be obtained, and thus the consistency of the overall data can be inferred.

[0055] Of course, any other possible method may be used to audit the first data to be audited and the second data to be audited, and there is no limitation to this.

[0056] In this embodiment, a first key is obtained from a first node of a cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to the first node, and the cluster is used for distributed caching, and a first existence state corresponding to the first key is determined, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, the second data to be audited corresponding to the first key has been mapped to a second data shard, and the second data shard has been mapped to the second node, and the first data to be audited and the second data to be audited are audited to obtain a first audit processing result, thereby enabling distributed caching of the data to be audited to effectively improve the horizontal scalability of the data audit processing process and improve the data audit processing efficiency.

[0057] Figure 2 It is a flow chart of a data audit method based on distributed cache proposed in another embodiment of the present disclosure.

[0058] like Figure 2 As shown, the data audit method based on distributed cache includes:

[0059] S201: Obtain a first key from a first node in a cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to a first node, and the cluster is used for distributed caching.

[0060] S202: Determine that the first key corresponds to a first existence state, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, the second data to be audited corresponding to the first key has been mapped to a second data shard, and the second data shard has been mapped to the second node.

[0061] The description of S201 and S202 can be specifically referred to the above embodiment, which will not be repeated here.

[0062] S203: Obtain an audit rule group, wherein the audit rule group includes: at least one audit rule.

[0063] The audit rule group refers to a set consisting of at least one audit rule. The audit rule refers to a pre-configured rule for indicating the audit process of the first data to be audited and the second data to be audited.

[0064] In the embodiment of the present disclosure, when obtaining the audit rule group, it is possible to pre-establish a communication link between the execution subject of the embodiment of the present disclosure and the big data server, and then obtain the audit rule group from the big data server. Alternatively, it is possible to receive audit rule indication information and parse the audit rule indication information to obtain the audit rule group, and there is no limitation on this.

[0065] Optionally, in some embodiments, when obtaining the audit rule group, the audit rule group can be obtained from the local cache, wherein the audit rule group is regularly read from the database to the local cache, thereby effectively improving the efficiency of obtaining the audit rule group and ensuring the real-time and reliability of the obtained audit rule group.

[0066] S204: Performing audit processing on the first data to be audited and the second data to be audited based on at least one audit rule to obtain a first audit processing result.

[0067] Optionally, in some embodiments, the audit rule is any of the following:

[0068] An exact matching rule, wherein the exact matching rule indicates that a first field value of the first data to be audited is the same as a second field value of the second data to be audited;

[0069] A related matching rule, wherein the related matching rule indicates that a first field value of the first data to be audited is the same as a first specified value, and a second field value of the second data to be audited is the same as a second specified value, and the first specified value and the second specified value are different;

[0070] A tolerance matching rule, wherein the tolerance matching rule indicates that the absolute value of the difference between the first field value of the first data to be audited and the second field value of the second data to be audited is less than or equal to a third specified value.

[0071] In this way, the audit rules can be enriched to a great extent to adapt to personalized audit needs, which can effectively improve the adaptability of the audit process.

[0072] The first field value and the second field value may be values ​​corresponding to a field in the first data to be audited and the second data to be audited, respectively, for example, a unique identifier, or a date.

[0073] The first designated value and the second designated value refer to numerical values ​​pre-designated for the first field value and the second field value. For example, when the first field value and the second field value are both used to indicate the number of objects, the first designated value may be 100 and the second designated value may be 200.

[0074] The third designated value refers to a value designated in advance for the absolute value of the difference between the first field value of the first data to be audited and the second field value of the second data to be audited.

[0075] For example, Figure 3 As shown, Figure 3 The following is a schematic diagram of a data audit solution proposed in the present disclosure, wherein the business process of data audit calculation includes the following steps:

[0076] 1) Obtain the business data of Party A and Party B after storage;

[0077] 2) Apply data audit rule group;

[0078] 3) Perform data audit and obtain discrepancy results.

[0079] Here, the amount of business data to be audited is in the billions. If the centralized computing architecture of the old solution is still used, it is usually necessary to move a batch of data from all data nodes to the computing nodes first. After the audit calculation action is completed, the audited data must be updated back to the data nodes. In this way, the performance is low. Therefore, the embodiment of the present disclosure can be based on Ignite's distributed computing feature-Entry Processor. The Entry Processor allows operations to be performed directly on the data node. It uses mobile computing to replace mobile data. The operation is serialized and sent to the node where the cached data is located, which is more efficient than serializing cached data. The steps for the Entry Processor to implement data audit and comparison calculations are as follows:

[0080] 1) Use the key of cache data (CacheA) of Party A as the cursor and traverse the key of each row of data;

[0081] 2) Check whether the key exists in the cache data of Party B (CacheB);

[0082] 3) If it does not exist, change the value of the corresponding key of CacheC to 2: This means that the business data exists on Party A but not on Party B;

[0083] 4) If it exists, then take out the value data of both A and B, and generate the audit unit Boolean expression set according to the fields used by the audit rules (including exact match, correlation match and tolerance match);

[0084] 5) If the result of any expression in the set is false, the audit result is recorded as inconsistent. Otherwise, the value of the corresponding key of CacheC is changed to 0: it means that the business data exists on both parties A and B, and the audit is consistent;

[0085] 6) Traverse all audit rules of the audit rule group. If any audit rule is not met, change the value of the corresponding key of CacheC to 3: indicating that the business data exists on both A and B, but the audit is inconsistent.

[0086] 7) Repeat this process until the cache data of party A is exhausted.

[0087] Among them, CacheA and CacheB are the business data of Party A and Party B to be audited, and CacheC is the audit result data. By using Entry Processor, all business data can complete the audit comparison calculation in its data node, improving the overall data audit performance. At the same time, Entry Processor is based on a distributed computing architecture, which realizes the distributed submission and distributed execution of data audit tasks, quickly realizes cluster parallelization, and automatically obtains the characteristics of task load balancing and automatic fault transfer.

[0088] That is to say, in the embodiment of the present disclosure, after determining that the first key corresponds to the first existence state, an audit rule group can be obtained, wherein the audit rule group includes: at least one audit rule, and based on the at least one audit rule, the first data to be audited and the second data to be audited are audited to obtain a first audit processing result, thereby effectively improving the reliability of the audit processing process of the first data to be audited and the second data to be audited based on the at least one audit rule.

[0089] Optionally, in some embodiments, when the first data to be audited and the second data to be audited are audited based on at least one audit rule to obtain a first audit processing result, the first data to be audited and the second data to be audited may be audited based on each audit rule to obtain a reference audit result corresponding to the corresponding audit rule. If at least one reference audit result indicates that the audit results of the first data to be audited and the second data to be audited are different, the second mark is used as the first audit processing result, wherein the second mark is used to indicate that the first data to be audited exists in the first node and the second data to be audited exists in the first node. The audit data exists in the second node, and the audit results are different. If all the reference audit results indicate that the audit results between the first data to be audited and the second data to be audited are the same, the third mark is used as the first audit processing result, wherein the third mark is used to indicate that the first data to be audited exists in the first node, and the second data to be audited exists in the second node, and the audit results are the same. Therefore, in the process of determining the first audit processing result, the reference audit results obtained after the corresponding processing of each audit rule can be comprehensively considered, which can effectively improve the indication effect of the obtained first audit processing result.

[0090] The reference audit result refers to the result obtained by auditing the first data to be audited and the second data to be audited based on a single audit rule.

[0091] In this embodiment, by obtaining an audit rule group, wherein the audit rule group includes: at least one audit rule, based on the at least one audit rule, the first data to be audited and the second data to be audited are audited to obtain a first audit processing result, thereby, the reliability of the audit processing process for the first data to be audited and the second data to be audited can be effectively improved based on the at least one audit rule. By obtaining the audit rule group from the local cache, wherein the audit rule group is regularly read from the database to the local cache, thereby, the efficiency of obtaining the audit rule group can be effectively improved, and the real-time and reliability of the obtained audit rule group can be guaranteed. By performing audit processing on the first data to be audited and the second data to be audited based on each audit rule to obtain a reference audit result corresponding to the corresponding audit rule, if at least one reference audit result indicates that the audit results between the first data to be audited and the second data to be audited are different, then the second mark is used as the first audit processing result, wherein the second mark is used to indicate that the first data to be audited exists in the first node, and the second data to be audited exists in the second node, and the audit results are different; if all reference audit results indicate that the audit results between the first data to be audited and the second data to be audited are the same, then the third mark is used as the first audit processing result, wherein the third mark is used to indicate that the first data to be audited exists in the first node, and the second data to be audited exists in the second node, and the audit results are the same. Therefore, in the process of determining the first audit processing result, the reference audit result obtained after the corresponding processing based on each audit rule can be comprehensively considered, which can effectively improve the indication effect of the obtained first audit processing result.

[0092] Optionally, the present disclosure also proposes a data auditing method based on distributed cache, which can determine a first hash value according to a first key and the first data to be audited, and determine a first data shard corresponding to the first data to be audited according to the first hash value and the total number of multiple candidate data shards, wherein the first data shard belongs to multiple candidate data shards. Therefore, the reliability of mapping the first data to be audited to the first data shard can be effectively improved.

[0093] The first hash value refers to a value obtained by performing hash mapping on the first key and the first data to be audited based on a preset hash algorithm.

[0094] The candidate data shard refers to any data shard mapped to a node. That is, a single node may be mapped to multiple candidate data shards at the same time.

[0095] Optionally, the present disclosure also proposes a data auditing method based on distributed cache, which can determine a second hash value according to a first key and a second data to be audited, and determine a second data shard corresponding to the second data to be audited according to the second hash value and the total number of multiple candidate data shards, wherein the second data shard belongs to multiple candidate data shards. This can effectively improve the logical rationality of mapping the second data to be audited to the second data shard.

[0096] The second hash value refers to a value obtained by performing hash mapping on the first key and the second data to be audited based on a preset hash algorithm.

[0097] That is to say, in the embodiments of the present disclosure, when using distributed cache to store data, the concept of data sharding can be introduced. Data sharding is both a data object container and an atomic unit of data migration. At the same time, the number of data shards of each data cache is guaranteed to remain unchanged during its life cycle. Therefore, a certain piece of data can be mapped to a data shard by first taking the modulus of the hash value of the data object key according to the number of data shards.

[0098] Figure 4 It is a flow chart of a data audit method based on distributed cache proposed in another embodiment of the present disclosure.

[0099] like Figure 4 As shown, the data audit method based on distributed cache includes:

[0100] S401: Selecting a first node corresponding to a first data shard from multiple candidate nodes in a cluster based on a Rendezvous hash algorithm.

[0101] The candidate node may refer to a node pre-configured in the cluster that may serve as the first node.

[0102] For example, in the embodiment of the present disclosure, when selecting the first node corresponding to the first data shard from multiple candidate nodes in the cluster based on the Rendezvous hash algorithm, the weight score of each node can be calculated for a certain data shard Pi, and the calculation algorithm is the hash function Wi,j=h(Pi, Sj). If the weight score of the mth node is the highest, then the data shard is assigned to the node. For this data shard, all remaining nodes will be sorted from large to small according to the calculated weight scores. It is assumed here that the data grid cluster is deployed with 3 nodes and the number of data shards is set to 4. Then the data distribution after applying the Rendezvous hash algorithm is as follows: Figure 5 As shown, Figure 5 This is a data distribution diagram proposed according to the present disclosure, wherein an elliptical box represents a node with the highest weight score, and the node is used to store a specified data shard.

[0103] When a new node (node ​​3) is added to the cluster, the node weight of each data shard will also change. The specific changes are as follows: Figure 6 As shown, Figure 6 This is a schematic diagram of data distribution changes proposed in the present disclosure. The nodes in the dotted boxes are nodes whose recalculated weights are lower than the weights of the newly added nodes (node ​​3) after the cluster adds new nodes. The newly added node 3 affects the data distribution of shard 3, so it needs to be migrated from node 2 to node 3.

[0104] In the embodiment of the present disclosure, when a node in the cluster is shut down, such as node 1, Figure 7 As shown, Figure 7 This is another data distribution change diagram proposed in the present disclosure. After applying the Rendezvous hash algorithm, only shard 1 stored in node 1 needs to be migrated from its backup node to node 2. Other data shards will not be affected. Compared with the consistent hashing algorithm, the Rendezvous hashing algorithm is simpler and more efficient in data distribution, without the need to generate node tokens; it also performs better in terms of monotonicity, and meets the need to expand the cluster due to a surge in business data volume. There is no need to re-hash and relocate all data, and the amount of data involved in migration is smaller.

[0105] In the disclosed embodiment, when ensuring that data can be more evenly distributed in the entire cluster, the hash function can be selected in combination with its hash uniformity. At the same time, the number of data objects is ensured to be much larger than the number of data shards, and the number of data shards is also much larger than the number of service nodes. Combined with the randomization implementation behind the Rendezvous hash algorithm, each node has an equal chance of being assigned to a certain data object.

[0106] In the embodiment of the present disclosure, when implementing cross-node parallel data import, the distributed ExecutorService capability provided by the Ignite computing grid can be used to allow the task of parsing data flowing into the library to achieve distributed submission and distributed execution, thereby realizing cross-node parallel data import and automatically obtaining the ability to load balance the loading task and automatically transfer the failure.

[0107] For example, three cache data can be imported, namely CacheA, CacheB, and CacheC. Among them, CacheA is used to store the business data of Party A for data comparison, and its key is the original keyword of Party A's business data table, and its value is a business data object; CacheB is used to store the business data of Party B for data comparison, and its key is the original keyword of Party B's business data table, and its value is a business data object; the key of CacheC is the union of the original keywords of Party A's business data table and the original keywords of Party B's business data table, and the default value is 1: indicating that the business data exists on Party B, but not on Party A

[0108] S402: Select a second node corresponding to the second data shard from multiple candidate nodes in the cluster based on the Rendezvous hash algorithm.

[0109] That is to say, in the embodiments of the present disclosure, a first node corresponding to the first data shard can be selected from multiple candidate nodes in the cluster based on the Rendezvous hash algorithm, and a second node corresponding to the second data shard can be selected from multiple candidate nodes in the cluster based on the Rendezvous hash algorithm. Thus, the adaptability between data shards and nodes can be effectively improved based on the Rendezvous hash algorithm.

[0110] S403: Map the first data to be audited to the first data shard, and map the first data shard to the first node.

[0111] S404: Map the second data to be audited to the second data shard, and map the second data shard to the second node.

[0112] That is to say, in the embodiment of the present disclosure, the first data to be audited can be mapped to the first data shard, and the first data shard can be mapped to the first node, and the second data to be audited can be mapped to the second data shard, and the second data shard can be mapped to the second node. Thus, the first data to be audited and the second data to be audited can be accommodated and migrated based on the first data shard and the second data shard, so as to effectively improve the stability of the first data to be audited and the second data to be audited during the data audit process.

[0113] For example, Figure 8 As shown, Figure 8 This is a schematic diagram of a distributed storage solution proposed in the present disclosure. When a distributed data grid is used as the main working data storage in the embodiments of the present disclosure, at least the following advantages are included:

[0114] 1. Distributed: Ignite data grid supports data sharding storage and multi-copy backup to achieve data security.

[0115] 2. Monotonicity: When a node in a distributed data grid fails, data redistribution only affects the failed node; or when a new node is added to the distributed data grid, data will only be migrated to the new node, and there will be no data migration between existing nodes.

[0116] 3. Balance: The hash result should be evenly distributed to each node, thus solving the load balancing problem from an algorithmic perspective.

[0117] Among them, distributed,

[0118] S405: Obtain a first key from a first node in the cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to a first node, and the cluster is used for distributed caching.

[0119] S406: Determine that the first key corresponds to a first existence state, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, the second data to be audited corresponding to the first key has been mapped to a second data shard, and the second data shard has been mapped to the second node.

[0120] S407: Audit the first data to be audited and the second data to be audited to obtain a first audit result.

[0121] The description of S406 and S407 can be specifically referred to the above embodiment, which will not be repeated here.

[0122] S408: Determine that the first key corresponds to a second existence state, wherein the second existence state indicates that the first key does not exist in the second node.

[0123] The second existence state may mean that none of the cluster nodes except the first node contains the to-be-audited data corresponding to the first key.

[0124] It is understandable that, among the data stored in multiple nodes of a cluster, there may be a situation where a single copy of data is only stored in one node, and the data is not stored in other nodes, which refers to the second existence state mentioned above.

[0125] S409: Generate a second audit processing result, wherein the second audit processing result includes a first mark, and the first mark is used to indicate that the first data to be audited exists in the first node, and the second data to be audited does not exist in the second node.

[0126] The second audit processing result refers to the audit processing result generated when it is determined that the first key corresponds to the second existence state.

[0127] That is to say, in the embodiment of the present disclosure, after obtaining the first key from the first node of the cluster, it can be determined that the first key corresponds to a second existence state, wherein the second existence state indicates that the first key does not exist in the second node, and a second audit processing result is generated, wherein the second audit processing result includes a first mark, and the first mark is used to indicate that the first data to be audited exists in the first node, and the second data to be audited does not exist in the second node. Therefore, based on the first mark, it is possible to accurately indicate the situation that the first key only exists in the first node, thereby effectively improving the indication accuracy of the second audit processing result.

[0128] In this embodiment, by selecting a first node corresponding to the first data shard from multiple candidate nodes in the cluster based on the Rendezvous hash algorithm, and selecting a second node corresponding to the second data shard from multiple candidate nodes in the cluster based on the Rendezvous hash algorithm, the adaptability between data shards and nodes can be effectively improved based on the Rendezvous hash algorithm. By mapping the first data to be audited to the first data shard, and mapping the first data shard to the first node, mapping the second data to be audited to the second data shard, and mapping the second data shard to the second node, the first data to be audited and the second data to be audited can be accommodated and migrated based on the first data shard and the second data shard, so as to effectively improve the stability of the first data to be audited and the second data to be audited during the data audit process. By determining that the first key corresponds to a second existence state, wherein the second existence state indicates that the first key does not exist in the second node, a second audit processing result is generated, wherein the second audit processing result includes a first mark, and the first mark is used to indicate that the first data to be audited exists in the first node, and the second data to be audited does not exist in the second node. Therefore, based on the first mark, it is possible to accurately indicate the situation that the first key only exists in the first node, thereby effectively improving the indication accuracy of the second audit processing result.

[0129] For example, Fig. 9 As shown, Fig. 9 The following is a schematic diagram of the data audit process proposed in this disclosure, which includes the following steps:

[0130] Step 1: Obtain the data to be audited;

[0131] Step 2: Import audit data into distributed cache;

[0132] Step 3: Obtain data audit rules;

[0133] Step 4: Perform data audit.

[0134] Wherein, when obtaining the data to be audited, step 1 may be to obtain the data to be audited transmitted from the business system from various external data sources. The description of steps 2 to 4 may refer to the above embodiment.

[0135] Fig.10 It is a structural diagram of a data audit device based on distributed cache proposed in one embodiment of the present disclosure.

[0136] like Fig.10 As shown, the data auditing device 100 based on distributed cache includes:

[0137] An acquisition module 1001 is used to acquire a first key from a first node of a cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to a first node, and the cluster is used for distributed caching;

[0138] A determination module 1002 is used to determine that the first key corresponds to a first existence state, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, the second to-be-audited data corresponding to the first key has been mapped to a second data shard, and the second data shard has been mapped to the second node; and

[0139] The processing module 1003 is used to perform audit processing on the first data to be audited and the second data to be audited to obtain a first audit processing result.

[0140] It should be noted that the aforementioned explanation of the data audit method based on distributed cache is also applicable to the data audit device based on distributed cache in this embodiment, and will not be repeated here.

[0141] In this embodiment, a first key is obtained from a first node of a cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to the first node, and the cluster is used for distributed caching, and a first existence state corresponding to the first key is determined, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, the second data to be audited corresponding to the first key has been mapped to a second data shard, and the second data shard has been mapped to the second node, and the first data to be audited and the second data to be audited are audited to obtain a first audit processing result, thereby enabling distributed caching of the data to be audited to effectively improve the horizontal scalability of the data audit processing process and improve the data audit processing efficiency.

[0142] Fig.11 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Fig.11The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0143] like Fig.11 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0144] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0145] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0146] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig.11 Not shown, often called a "hard drive").

[0147] although Fig.11Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (hereinafter referred to as: CD-ROM), a digital versatile disc read only memory (hereinafter referred to as: DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0148] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0149] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a human body to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. In addition, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the data audit method based on distributed cache mentioned in the above embodiment.

[0151] In order to implement the above embodiments, the present disclosure further proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data audit method based on distributed cache proposed in the above embodiments of the present disclosure is implemented.

[0152] In order to implement the above embodiments, the present disclosure further proposes a computer program product. When the instruction processor in the computer program product is executed, the data audit method based on distributed cache proposed in the above embodiments of the present disclosure is executed.

[0153] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0154] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0155] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0156] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0157] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0158] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0159] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0160] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0161] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0162] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A data audit method based on distributed cache, characterized in that: The method comprises: Obtaining a first key from a first node of a cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to the first node, and the cluster is used for the distributed cache; Determine that the first key corresponds to a first existence state, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, second to-be-audited data corresponding to the first key has been mapped to a second data shard, and the second data shard has been mapped to the second node; and Auditing the first data to be audited and the second data to be audited to obtain a first auditing result.

2. The method according to claim 1, characterized in that The method further comprises: Determining that the first key corresponds to a second existence state, wherein the second existence state indicates that the first key does not exist in the second node; A second audit processing result is generated, wherein the second audit processing result includes a first mark, and the first mark is used to indicate that the first data to be audited exists in the first node, and the second data to be audited does not exist in the second node.

3. The method according to claim 1, characterized in that The auditing the first data to be audited and the second data to be audited to obtain a first auditing result includes: Acquire an audit rule group, wherein the audit rule group includes: at least one audit rule; The first data to be audited and the second data to be audited are audited based on the at least one audit rule to obtain the first audit processing result.

4. The method according to claim 3, characterized in that The obtaining of the audit rule group includes: An audit rule group is obtained from a local cache, wherein the audit rule group is periodically read from a database to the local cache.

5. The method according to claim 3, characterized in that The audit rule is any of the following: An exact matching rule, wherein the exact matching rule indicates that a first field value of the first data to be audited is the same as a second field value of the second data to be audited; A related matching rule, wherein the related matching rule indicates that a first field value of the first data to be audited is the same as a first specified value, and a second field value of the second data to be audited is the same as a second specified value, and the first specified value and the second specified value are different; A tolerance matching rule, wherein the tolerance matching rule indicates that the absolute value of the difference between the first field value of the first data to be audited and the second field value of the second data to be audited is less than or equal to a third specified value.

6. The method according to claim 3, characterized in that The performing audit processing on the first data to be audited and the second data to be audited based on the at least one audit rule to obtain the first audit processing result includes: Performing audit processing on the first data to be audited and the second data to be audited based on each of the audit rules to obtain a reference audit result corresponding to the corresponding audit rule; If at least one of the reference audit results indicates that the audit results between the first data to be audited and the second data to be audited are different, a second mark is used as the first audit processing result, wherein the second mark is used to indicate that the first data to be audited exists in the first node, and the second data to be audited exists in the second node, and the audit results are different; If all of the reference audit results indicate that the audit results between the first data to be audited and the second data to be audited are the same, then the third mark is used as the first audit processing result, wherein the third mark is used to indicate that the first data to be audited exists in the first node, and the second data to be audited exists in the second node, and the audit results are the same.

7. The method according to any one of claims 1 to 6, characterized in that: Before obtaining the first key from the first node of the cluster, the method further includes: Mapping the first data to be audited to the first data shard, and mapping the first data shard to the first node; Map the second data to be audited to the second data shard, and map the second data shard to the second node.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Determine a first hash value according to the first key and the first data to be audited; The first data shard corresponding to the first data to be audited is determined according to the first hash value and the total number of multiple candidate data shards, wherein the first data shard belongs to the multiple candidate data shards.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Determine a second hash value according to the first key and the second data to be audited; The second data shard corresponding to the second data to be audited is determined according to the second hash value and the total number of multiple candidate data shards, wherein the second data shard belongs to the multiple candidate data shards.

10. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Selecting the first node corresponding to the first data shard from a plurality of candidate nodes in the cluster based on a Rendezvous hash algorithm; The second node corresponding to the second data shard is selected from a plurality of candidate nodes in the cluster based on the Rendezvous hash algorithm.

11. A data auditing device based on distributed cache, characterized in that: The device comprises: An acquisition module, configured to acquire a first key from a first node of a cluster, wherein the first key is used to identify first data to be audited, the first data to be audited has been mapped to a first data shard, and the first data shard has been mapped to the first node, and the cluster is used for the distributed cache; a determination module, configured to determine that the first key corresponds to a first existence state, wherein the first existence state indicates that the first key exists in a second node of the cluster, the first node and the second node are different, second to-be-audited data corresponding to the first key has been mapped to a second data shard, and the second data shard has been mapped to the second node; and A processing module is used to perform audit processing on the first data to be audited and the second data to be audited to obtain a first audit processing result.

12. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images