Data verification method and device, distributed system and storage medium

Abstract

The invention relates to the technical field of big data, and provides a data verification method and device, a distributed system and a storage medium. The method comprises the steps of generating a data acquisition strategy in response to a data verification request received from a first equipment cluster, acquiring source table data from a source database based on the data acquisition strategy, acquiring target table data from a target database based on the source table data, and verifying the target table data according to the data verification request, and generating a verification result based on comparison between the source table data and the target table data. According to the method, the verification efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, specifically to the field of data processing technology for big data, and in particular to a data verification method, device, distributed system and storage medium. Background Art

[0002] In order to meet users' needs for data search and analysis, in business scenarios such as e-commerce, data in one database is usually stored in another database for data search and analysis. However, due to many reasons such as complex link relationships between databases and long storage links, data inconsistencies occur in multiple databases. To this end, a data verification scheme is needed to verify the data in multiple databases. However, in related data verification schemes, in order to ensure the accuracy of data verification, the data is usually directly compared in full. However, the verification efficiency of this verification method is low, resulting in excessive load on the system. In addition, if data is randomly sampled for comparison, when the data is unevenly sampled, the verification accuracy will be low. Summary of the invention

[0003] The present application provides a data verification method, device, distributed system and storage medium to solve the technical problem of how to improve verification efficiency while ensuring verification accuracy.

[0004] A first aspect of an embodiment of the present application provides a data verification method, which is applied to a distributed system, wherein the distributed system includes a first device cluster and a second device cluster, and the method includes: the first device cluster receives node indicator information reported by all working nodes of the second device cluster; the first device cluster selects a target node from all working nodes of the second device cluster according to the node indicator information; the first device cluster sends a data verification request to the target node; the target node generates a data acquisition strategy according to the data verification request, and the data verification request includes library identifiers of a source database and a target database; the target node acquires source table data from the source database based on the data acquisition strategy, and acquires target table data from the target database based on the source table data; the target node generates a verification result based on a comparison between the source table data and the target table data.

[0005] A second aspect of an embodiment of the present application provides a data verification method, which is applied to a working node of a second device cluster in a distributed system, wherein the distributed system also includes a first device cluster. The method includes: generating a data acquisition strategy in response to a data verification request received from the first device cluster, wherein the data verification request includes library identifiers of a source database and a target database; acquiring source table data from the source database based on the data acquisition strategy, and acquiring target table data from the target database based on the source table data; and generating a verification result based on a comparison between the source table data and the target table data.

[0006] According to an embodiment of the present application, generating a verification result based on the comparison between the source table data and the target table data includes: if the data format of the source table data is the same as the data format of the target table data, comparing the source table data with the target table data; if there is difference data in the target table data that is different from the source table data, determining the current comparison round; determining the current comparison round between the source table data and the target table data; determining the verification result according to the current comparison round and the difference data.

[0007] According to an embodiment of the present application, determining the verification result based on the current comparison round and the difference data includes: if the current comparison round is less than a preset number threshold, obtaining first comparison data from the target database based on the data primary key of the difference data, and obtaining second comparison data from the source database based on the data foreign key of the first comparison data; if the first comparison data is the same as the second comparison data, determining the verification result as a first preset result, and the first preset result indicates that the source table data is consistent with the target table data.

[0008] According to an embodiment of the present application, determining the verification result based on the current comparison round and the difference data also includes: if the current comparison round is greater than or equal to the preset number threshold, determining the verification result as a second preset result, and the second preset result indicates that there is inconsistent data between the target table data and the source table data.

[0009] According to an embodiment of the present application, the method further includes: if the verification result is the second preset result, generating warning information of the verification result according to the difference data; and sending the warning information to a designated user.

[0010] According to an embodiment of the present application, the data acquisition strategy generated in response to the data verification request received from the first device cluster includes: reading data from the source database according to the request type of the data verification request, and determining the amount of data read; determining a verification method according to the data amount and a preset quantity threshold; if the verification method is a preset method, generating the data acquisition strategy according to the configuration table information in the source database and the sampling conditions of the configuration table information.

[0011] A third aspect of an embodiment of the present application provides a data verification device, which runs on a working node of a second device cluster in a distributed system, wherein the distributed system also includes a first device cluster, and the device includes: a generation unit, which is used to generate a data acquisition strategy in response to a data verification request received from the first device cluster, wherein the data verification request includes library identifiers of a source database and a target database; an acquisition unit, which is used to acquire source table data from the source database based on the data acquisition strategy, and acquire target table data from the target database based on the source table data; the generation unit is also used to generate a verification result based on a comparison between the source table data and the target table data.

[0012] The fourth aspect of an embodiment of the present application provides a distributed system, which includes: a first device cluster and a second device cluster, the second device cluster includes a working node; the working node includes: a memory, storing computer-readable instructions; and a processor, executing the computer-readable instructions stored in the memory to implement the data verification method.

[0013] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, in which computer-readable instructions are stored. The computer-readable instructions are executed by a processor in a working node to implement the data verification method.

[0014] In the embodiment of the present application, the first device cluster screens out the target node according to the node indicator information reported by all working nodes of the second device cluster. Since the target node is the node with the lowest load in the second device cluster, the first device cluster sends a data verification request to the target node, and then the target node processes the data verification request, which can save the waiting time of the data verification request, thereby improving the efficiency of generating the verification result. In addition, the embodiment of the present application can obtain reasonable source table data for data comparison through the generated data acquisition strategy. Since there is no need to compare the full amount of data in the source database, the generation efficiency of the verification result can be ensured, thereby avoiding the impact of excessive load on the system. At the same time, the acquisition of source table data through the data acquisition strategy can avoid the problem of uneven data sampling, thereby improving the accuracy of verification. In addition, through the source table data, the target table data can be accurately obtained for data comparison, thereby ensuring the accuracy of data verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is an application scenario diagram of a data verification method provided in an embodiment of the present application.

[0016] Figure 2 It is an interactive flow chart of the data verification method provided in the embodiment of the present application.

[0017] Figure 3 It is a flow chart of the data verification method provided in an embodiment of the present application applied to a working node of a second device cluster in a distributed system.

[0018] Figure 4 It is a flow chart of generating a data acquisition strategy provided in an embodiment of the present application.

[0019] Figure 5 It is a detailed flow chart for generating verification results provided in an embodiment of the present application.

[0020] Figure 6 It is a further detailed flowchart of generating verification results provided in an embodiment of the present application.

[0021] Figure 7 This is a flow chart of a data verification method provided by another embodiment of the present application applied to a working node of a second device cluster in a distributed system.

[0022] Figure 8 It is a functional module diagram of the data verification device provided in an embodiment of the present application.

[0023] Fig. 9 It is a structural diagram of a working node for implementing a data verification method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0026] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0027] like Figure 1 As shown, it is an application scenario diagram of a data verification method provided by an embodiment of the present application. The data verification method is applied to a distributed system 10, and the distributed system 10 includes a first device cluster 100 and a second device cluster 200. The first device cluster 100 and the second device cluster 200 can communicate with each other through an API (Application Programming Interface) interface. The first device cluster can be a Master cluster composed of multiple master nodes (for example, multiple hosts). The first device cluster 100 can be a distributed task scheduling system, and multiple master nodes can be used to control the task allocation of the distributed system 10. The second device cluster 200 can be a Worker cluster composed of multiple working nodes. The second device cluster 200 has the ability to expand horizontally, and can dynamically add or delete working nodes as needed to adapt to different workloads. For example, the working nodes can include working node 1, working node 2,..., working node m. Multiple working nodes in the second device cluster 200 can respond to the task allocation instruction sent by the first device cluster 100 to perform data processing tasks. For example, the data processing task may include a data verification task. When the working node performs the data verification task, the working node can obtain data from multiple preset databases respectively. For example, Figure 1The source database 300 and the target database 400 are shown, wherein the source database 300 may include, but is not limited to: Mysql database, TiDB database, MongoDB database and other relational or non-relational databases, and the target database 400 may include, but is not limited to: MongoDB database, Redis database, ElasticSearch database and other relational or non-relational databases.

[0028] In many business scenarios of e-commerce platforms, in order to achieve data search or data analysis on the e-commerce independent site, it is usually necessary to store the data of one database (for example, the source database 300) in another database (for example, the target database 400) through complex multi-level links. For example, the data of the source database 300 can be multi-level linked to form a wide table, and the obtained wide table can be stored in the search engine of the e-commerce independent site. The wide table can record the data stored in the source database 300 and the derivative data generated by associative calculations. However, under normal circumstances, due to reasons such as overly complex link relationships or long link links, the actual processing process usually leads to inconsistencies between the data in the source database 300 and the data in the target database 400. Based on this problem, an embodiment of the present application provides a data verification method to achieve the purpose of quality verification of the data in the target database. The following will be combined with Figure 2 The flowchart shown illustrates the data verification method provided in the embodiment of the present application.

[0029] like Figure 2 , which is an interactive flow chart of a data verification method provided in an embodiment of the present application. The data verification method is applied to a distributed system, and the distributed system includes a first device cluster (for example, Figure 1 The first device cluster 100 in the example and the second device cluster (eg, Figure 2 The second device cluster 200 in the flowchart may be changed according to different requirements, and some steps may be omitted. The data verification method provided in the embodiment of the present application includes the following steps.

[0030] S201: A first device cluster receives node indicator information reported by all working nodes of a second device cluster.

[0031] In one embodiment of the present application, all working nodes of the second device cluster can send node indicator information to the first device cluster through the API interface, and the first device cluster can also receive node indicator information reported by all working nodes of the second device cluster through the API interface. Each working node can have the same data processing capability. The node indicator information may include, but is not limited to: central processing unit (CPU) parameters and memory parameters of the working node, for example, the CPU parameter may be the CPU usage rate, and the memory parameter may be the memory usage rate. All working nodes in the second device cluster can report the corresponding node indicator information to the first device cluster according to a preset period. For example, all working nodes in the second device cluster report the current node indicator information to the first device cluster every 2 hours.

[0032] By collecting node indicator information reported by all working nodes in the second device cluster at a preset period, each master node in the first device cluster can know the load information of each working node in the second device cluster in real time, so that in subsequent task allocation, tasks can be reasonably allocated according to the load information of each working node.

[0033] S202: The first device cluster selects a target node from all working nodes of the second device cluster according to the node indicator information.

[0034] In one embodiment of the present application, a working node with the lowest node load can be selected from the second device cluster as the target node. The actual application is not limited to this. In actual applications, the working node that is most suitable for the task to be assigned can be selected according to different loads. For example, the working node with the lowest memory utilization at the current moment can be determined as the target node based on the memory utilization.

[0035] In another embodiment of the present application, the first device cluster obtains the index weight of each node index information, and performs weighted sum operation on the corresponding node index information according to the index weight to obtain the node load value of each working node, and the first device cluster determines the working node with the smallest node load value as the target node. Among them, the value of the index weight can be set according to demand. For example, the node index information includes: CPU utilization and memory utilization, the working nodes of the second device cluster include node A and node B, the CPU utilization of node A is 50%, the memory utilization of node A is 30%, the CPU utilization of node B is 60%, the memory utilization of node B is 20%, the index weight corresponding to the CPU utilization is 0.3, and the index weight corresponding to the memory utilization is 0.7. After weighted sum operation, the node load value of node A is: 0.36, and the node load value of node B is: 0.32. Therefore, the first device cluster determines node B as the target node.

[0036] In the embodiment of the present application, calculating the node indicator information according to the indicator weight can ensure the rationality of the calculation of the node load value, and then screening the target node from all the working nodes of the first device cluster through the node load value, which can improve the rationality of the screening of the target node.

[0037] S203: The first device cluster sends a data verification request to the target node.

[0038] In one embodiment of the present application, in many e-commerce platforms, in order to ensure the accuracy of the search task, it is usually necessary to verify whether the data in the target database is consistent with the data in the source database. Since the data in the source database will be updated along with the business data, the first device cluster will send a data verification request to the target node under a trigger condition. The trigger condition may be a preset time, or the trigger condition may be that the first device cluster detects that the data in the source database has been updated.

[0039] The data verification request may be a task request pre-set by the first device cluster, which is used to verify whether the data in the target database is consistent with the data in the source database, wherein the data verification request includes the library identifiers of the source database and the target database. Among them, the source database may be a self-sufficient database that directly provides original information or specific data. The self-sufficient database can directly provide original data to users without relying on other databases or systems. The self-sufficient database contains all the necessary information and tools to facilitate users to perform data access, query, processing and analysis operations. At the same time, the self-sufficient database can also expand different functions and services to meet user needs and requirements. For example, if the source database is a database of an e-commerce independent station, the data stored in the source database may include link data of goods such as clothes. The target database is an import database for data in the source database. The first device cluster sends a data verification request to the determined target node according to a preset period to verify the data in the target database, thereby controlling the data quality.

[0040] In this embodiment, by sending a data verification request to the target node with the lowest load in the second device cluster, the target node can quickly respond to the data verification request because the target node has the lowest load, thereby saving the waiting time for the data verification request.

[0041] S204: The target node generates a data acquisition strategy according to the data verification request.

[0042] In one embodiment of the present application, each source database may correspond to multiple data acquisition strategies, and the data acquisition strategy may be a strategy generated based on the configuration table information in the source database and the sampling conditions of the configuration table information, wherein the configuration table information may be a specific field in the source database that needs to be quality monitored, and the sampling conditions may be a field range value of the configuration table information. For example, the configuration table information may be sales volume, and the sampling conditions may be sales values ​​between 20,000 and 30,000. For another example, the configuration table information may be sales volume, and the sampling conditions may be a specific hash modulus range, and the specific hash modulus value in the hash modulus range is used to determine the storage location of the data that needs to be verified in the source database. Different source databases correspond to different configuration table information and sampling conditions of the configuration table information.

[0043] S205 , the target node obtains source table data from the source database based on the data acquisition strategy, and obtains target table data from the target database based on the source table data.

[0044] In one embodiment of the present application, the source table data is the data in the source database corresponding to the data acquisition strategy. The target table data is the data in the target database corresponding to the foreign key of the source table data. The foreign key of the source table data represents the identifier that establishes the relationship between the source table data and the target table data. For example, the foreign key is

[0045] FK_orders_customerId_to_customers_customerId, where orders can be the source data table in the source database, customerId can be a column in the source data table, customers can be a slave table in the target database, and customerId can be a column in the slave table. The target node instantiates the reader of the source database and the reader of the target database according to the data acquisition strategy, and then obtains the source table data and the target table data.

[0046] S206: The target node generates a verification result based on the comparison between the source table data and the target table data.

[0047] In one embodiment of the present application, the verification result may include a first preset result, wherein the first preset result may indicate that the source table data is consistent with the target table data, and the verification result may also include a second preset result, wherein the second preset result may indicate that there is inconsistent data between the target table data and the source table data, such as difference data.

[0048] In the embodiment of the present application, the above-mentioned data verification method is used to analyze the node indicator information reported by all working nodes of the second device cluster through the first device cluster to screen out the target node. Since the target node is the node with the lowest load in the second device cluster, the data verification request is sent to the target node through the first device cluster, and then the waiting time of the data verification request is saved by the target node through the processing of the data verification request. Thus, the generation efficiency of the verification result is improved. In addition, the embodiment of the present application can obtain reasonable source table data for data comparison through the generated data acquisition strategy. Since there is no need to compare the full amount of data in the source database, the generation efficiency of the verification result can be ensured, thereby avoiding the impact of excessive load on the system. At the same time, the acquisition of source table data through the data acquisition strategy can avoid the problem of uneven data sampling, thereby improving the accuracy of verification. In addition, through the source table data, the target table data can be accurately obtained for data comparison, thereby ensuring the accuracy of data verification.

[0049] like Figure 3 , is a flowchart of a data verification method provided in an embodiment of the present application applied to a working node of a second device cluster in a distributed system. The data verification method is applied to a working node of a second device cluster in a distributed system, and the distributed system also includes a first device cluster. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The data verification method provided in an embodiment of the present application includes the following steps.

[0050] S301 : Generate a data acquisition strategy in response to a data verification request received from a first device cluster.

[0051] In at least one embodiment of the present application, the data verification request includes the library identifiers of the source database and the target database. The data acquisition strategy can be generated based on the configuration table information that needs to be quality-monitored in the source database and the sampling conditions of the configuration table information, and the configuration table information is a specific field in the source database. The data acquisition strategy is a strategy generated when the verification method of the data verification request is a preset method, wherein the preset method can be set to a sampling comparison method. The verification method can be determined based on the relationship between the total amount of data that needs to be verified in the source database and the preset quantity threshold. When the total amount of data that needs to be verified is greater than the preset quantity threshold, the verification method can be set to the preset method to avoid excessive load caused by a large amount of data verification, thereby ensuring verification efficiency. If the total amount of data that needs to be verified is less than or equal to the preset quantity threshold, the working node determines that the verification method is a full comparison method to ensure verification accuracy.

[0052] S302, based on the data acquisition strategy, source table data is acquired from the source database, and target table data is acquired from the target database based on the source table data.

[0053] In at least one embodiment of the present application, the source table data may be data in the source database that corresponds to both the configuration table information and the sampling conditions. The target table data is data in the target database that corresponds to the foreign key of the source table data. Specifically, the working node locates the primary key data whose primary key is the configuration table information from the source database, and filters out the data corresponding to the sampling conditions from the primary key data as the source table data. For example, in an e-commerce scenario, the source database stores the business data of the e-commerce independent station, such as the commodity data and order data of each e-commerce independent station in the e-commerce platform. If the configuration table information is the order number, the primary key data may be the data containing the order number in the source database. If the sampling condition is from order number 00001 to order number 00100, the source table data may be the order data from order number 00001 to order number 00100 in the e-commerce independent station. This embodiment can accurately obtain the primary key data from the source database through the relationship between the configuration table information and the primary key, and then can reasonably filter out the source table data from the primary key data through the sampling condition, thereby reducing the total amount of verification of the source table data and ensuring the efficiency of data quality monitoring.

[0054] In at least one embodiment of the present application, the working node obtains the foreign key of the source table data from the source database, and obtains the data corresponding to the foreign key of the source table data from the target database as the target table data. This embodiment can accurately obtain the target table data corresponding to the source table data from the target database through the foreign key of the source table data. For example, in an e-commerce scenario, the target database stores the business data of the e-commerce independent station, such as the commodity data and order data of each e-commerce independent station in the e-commerce platform. If the source table data is the order data from order number 00001 to order number 00100, the foreign key is FK_orders_customerId_to_customers_customerId, and the target table data can be the data from order number 00001 to order number 00100.

[0055] S303: Generate a verification result based on the comparison between the source table data and the target table data.

[0056] In at least one embodiment of the present application, the verification result may include a first preset result indicating that the source table data is consistent with the target table data, and the verification result may also include a second preset result indicating that there is difference data in the target table data that is inconsistent with the source table data. In another embodiment, the verification result can also be determined by the comparison result of the source table data and the target table data and the current comparison round, wherein the current comparison round represents the comparison round between the source table data and the target table data when performing data comparison, and the comparison round is used to control the number of comparisons between the source table data and the target table data. For example, in response to a data verification request, the working node obtains the source table data from the source database and the target table data from the target database, and compares the source table data obtained this time with the target table data. If there is difference data different from the source table data in the target table data at this time, it is determined that the comparison round is 1. Assuming that the preset number threshold is set to 3, since the current comparison round 1 is less than the preset number threshold 3, data is re-obtained from the source database and the target database for comparison, and the comparison rounds are accumulated, and so on, until the current comparison round reaches the preset number threshold, or the re-obtained data is verified to be the same, the verification result is determined. The embodiment of the present application introduces the current comparison round to determine the verification result, which can avoid low accuracy of the verification result caused by erroneous acquisition of source table data or target table data. At the same time, by introducing a preset number threshold, endless comparison of source table data and target table data can be avoided, thereby ensuring data verification efficiency.

[0057] The embodiment of the present application utilizes the above-mentioned data verification method, and through the generated data acquisition strategy, reasonable source table data can be obtained for data comparison. Since there is no need to compare the full amount of data in the source database, the generation efficiency of the verification result can be ensured, thereby avoiding the impact of excessive load on the system. At the same time, the acquisition of source table data through the data acquisition strategy can avoid the problem of uneven data sampling, thereby improving the accuracy of verification. In addition, through the source table data, the target table data can be accurately obtained for data comparison, thereby ensuring the accuracy of data verification.

[0058] like Figure 4 As shown in FIG. 1 , it is a flow chart of generating a data acquisition strategy provided by an embodiment of the present application, and the method for generating a data acquisition strategy is applied to a working node. Figure 4 As shown, it specifically includes steps: S3011 to S3015.

[0059] S3011, reading data from a source database according to the request type of the data verification request, and determining the amount of the read data.

[0060] In at least one embodiment of the present application, the request type includes an incremental task type and a full task type. Specifically, when the request type is an incremental task type, the working node reads the newly added or modified data from the source database relative to the last time the second device cluster responded to the data verification request as the read data. When the request type is a full task type, the working node reads all data from the source database as the read data. The working node reads the data corresponding to the request type from the source database and counts the amount of the read data. This embodiment can determine the data to be verified based on user needs through the request type.

[0061] S3012, determine the verification method based on the data volume and the preset quantity threshold.

[0062] In at least one embodiment of the present application, the preset quantity threshold can be set according to the user's demand for the verification time, and the preset quantity threshold can also be set according to the node load of the working node. The verification method includes a sampling comparison method and a full comparison method. Specifically, the working node compares the size relationship between the data volume and the preset quantity threshold. If the data volume is greater than the preset quantity threshold, the working node determines that the verification method is a sampling comparison method. If the data volume is less than or equal to the preset quantity threshold, the working node determines that the verification method is a full comparison method.

[0063] In this embodiment, when the amount of data is greater than the preset quantity threshold, the verification method of the data verification request is determined as a sampling comparison method, which can avoid comparing all the read data and causing excessive load during business peak hours. In addition, when the amount of data is less than or equal to the preset quantity threshold, the verification method of the data verification request is determined as a full comparison method, which can ensure the accuracy of data verification.

[0064] S3013, checking whether the verification method is the preset method.

[0065] In at least one embodiment of the present application, the preset method is usually set to a sampling comparison method.

[0066] S3014: If the verification method is a preset method, a data acquisition strategy is generated according to the configuration table information in the source database and the sampling conditions of the configuration table information.

[0067] In at least one embodiment of the present application, the configuration table information is the field in the source database that needs to be monitored for data quality, the sampling condition is the field range value set for the configuration table information, and both the configuration table information and the sampling condition can be set according to the actual needs of the user. In this embodiment, when the verification method is the sampling comparison method, by combining the configuration table information and the sampling condition, a strategy for reasonable sampling can be generated.

[0068] S3015: If the verification method is not the preset method, data is obtained from the target database based on the read data, and the data obtained from the target database is fully compared with the data read from the source database.

[0069] In this embodiment, when the verification method is not the preset method, the read data can be verified by a full comparison method. By obtaining data from the target database and performing a full comparison with the read data, the accuracy of the verification result can be ensured.

[0070] The embodiment of the present application can read the data that needs to be quality monitored through the request type, and then reasonably determine the verification method by comparing the amount of data read with a preset quantity threshold.

[0071] like Figure 5 As shown in FIG. 1 , it is a detailed flow chart of generating verification results provided by an embodiment of the present application, and the method for generating a data acquisition strategy is applied to a working node. Figure 5 As shown, it specifically includes steps: S3031 to S3035.

[0072] S3031, detecting whether the data format of the source table data is the same as the data format of the target table data.

[0073] In at least one embodiment of the present application, a work node may determine the data format of source table data based on metadata of a source database, and a work node may also determine the data format of target table data based on metadata of a target database. Metadata may be structured data used to describe data features, and metadata may include metadata items and metadata content. For example, a metadata item may be a name, and the corresponding metadata content may be: Zhang San. Data formats include, but are not limited to: character format, numeric format, etc.

[0074] S3032: If the data format of the source table data is different from the data format of the target table data, perform format conversion on the source table data or the target table data.

[0075] In at least one embodiment of the present application, when the data format of the source table data is different from the data format of the target table data, the source table data and the target table data with the same data format can be obtained by format conversion of the source table data or the target table data. By format conversion, the present application can support data verification of databases with different data formats, thereby increasing the verification range.

[0076] S3033: If the data format of the source table data is the same as the data format of the target table data, compare the source table data with the target table data.

[0077] In at least one embodiment of the present application, the working node may call a preset data comparison script to compare the source table data with the target table data, wherein the data comparison script is used to compare whether the data is consistent.

[0078] S3034: If there is difference data in the target table data that is different from the source table data, determine the current comparison round between the source table data and the target table data.

[0079] In at least one embodiment of the present application, the difference data may be all data in the target table data, or may be part of the data in the target table data. For example, if the target table data includes data 1 to data 9, the difference data may be data 1 to data 9, or may be data 1, etc. The current comparison round is the comparison round between the source table data and the target table data when generating the verification result.

[0080] S3035, determine the verification result according to the current comparison round and difference data.

[0081] In at least one embodiment of the present application, the verification result is determined in combination with the comparison result of the source table data and the target table data and the current comparison round, and the verification result includes a first preset result and a second preset result. The first preset result indicates that the source table data is the same as the target table data, and the second preset result indicates that there is difference data in the target table data that is different from the source table data.

[0082] The embodiment of the present application can ensure that the verification result will not be inaccurate due to the different data formats of the source table data and the target table data by detecting the data formats of the source table data and the target table data. Furthermore, when the data formats are different, by converting the format of the source table data or the target table data, the embodiment of the present application can support the verification of databases of multiple data types. Furthermore, by combining the current comparison round and the difference data to determine the verification result, it is possible to avoid the inaccuracy of the verification result caused by an error in a certain verification, thereby improving the accuracy of the generation of the verification result.

[0083] like Figure 6 As shown, it is a further detailed flow chart of generating verification results provided by the embodiment of the present application, and the method for generating the data acquisition strategy is applied to the working node of the second device cluster. Figure 6 As shown, the specific steps include the following steps.

[0084] S30351, check whether the current comparison round is less than the preset number threshold.

[0085] In at least one embodiment of the present application, the preset number threshold can be set according to the processing efficiency and processing accuracy of the data verification request. For example, the preset number threshold can be set to 3. The current comparison round represents the comparison round between the source table data and the target table data when performing data comparison. By setting the preset number threshold, endless comparison between the source table data and the target table data can be avoided.

[0086] S30352: If the current comparison round is greater than or equal to the preset number threshold, the verification result is determined as the second preset result.

[0087] In at least one embodiment of the present application, the second preset result indicates that there is difference data in the target table data that is inconsistent with the source table data. In this embodiment, when there is difference data in the target table data that is different from the source table data, and the current comparison round is greater than or equal to the preset number threshold, the verification result is determined as the second preset result. Since the target table data and the source table data are compared for multiple rounds, the possibility of incorrect verification results due to errors in obtaining the target table data can be reduced through multiple comparisons, thereby ensuring the accuracy of the verification results. For example, when it is determined that there are difference data after multiple comparisons, the corresponding verification result can be determined. In addition, by limiting the preset number threshold, continuous and repeated comparisons can also be avoided, which can improve the efficiency of data verification.

[0088] S30353: If the current comparison round is less than the preset number threshold, first comparison data is obtained from the target database according to the data primary key of the difference data, and second comparison data is obtained from the source database according to the data foreign key of the first comparison data.

[0089] In at least one embodiment of the present application, the data primary key is the primary key information of the difference data in the target database, and the data foreign key is the foreign key information of the difference data in the target database. In this embodiment, the first comparison data and the second comparison data are re-acquired through the data primary key and the data foreign key, which can avoid the low accuracy of the verification result caused by obtaining wrong data from the source database and the target database, thereby ensuring the accuracy of the verification result.

[0090] S30354, detecting whether the first comparison data is the same as the second comparison data.

[0091] S30355: If the first comparison data is different from the second comparison data, the current comparison round is accumulated and the process returns to step S30351.

[0092] For example, in response to a data verification request, the working node obtains source table data from the source database and target table data from the target database for the first time, and compares the source table data with the target table data. If there is difference data different from the source table data in the target table data, the current comparison round of the source table data and the target table data is determined to be 1. Since the current comparison round 1 is less than the preset number threshold 3, the working node re-obtains the first comparison data from the source database based on the difference data, and re-obtains the second data from the target database based on the first comparison data, and compares the first comparison data with the second comparison data. If there is data different from the first comparison data in the second comparison data, the current comparison round is accumulated, and the current comparison round is 2 at this time. Since the current comparison round 2 is less than the preset number threshold 3, the working node obtains data from the source database and the target database again for comparison. If the comparison results at this time are the same, the verification result is determined as the first preset result. If there is still difference data different from the source table data in the target table data at this time, the current comparison round continues to be accumulated, and the current comparison round is 3 at this time. Since the current comparison round 3 is equal to the preset number threshold 3, the verification result is determined as the second preset result.

[0093] S30356: If the first comparison data is the same as the second comparison data, the verification result is determined as the first preset result.

[0094] In at least one embodiment of the present application, the first preset result indicates that the source table data is consistent with the target table data. When all data in the source table data is the same as the corresponding data in the target table data, the verification result is the first preset result.

[0095] In the embodiment of the present application, when there is difference data in the source table data that is different from the target table data, by detecting the relationship between the current comparison round and the preset number threshold, and then when the current comparison round is greater than or equal to the preset number threshold, the verification result is directly determined as the second preset result. Since there is no need to further compare the difference data when the current comparison round is greater than or equal to the preset number threshold, it is possible to ensure the accuracy of the verification result while ensuring the efficiency of generating the verification result. In addition, when the current comparison round is less than the preset number threshold, the embodiment of the present application re-acquires the first comparison data and the second comparison data through the data primary key and data foreign key of the difference data for comparison, which can ensure the accuracy of the verification result.

[0096] like Figure 7 , is a flowchart of a data verification method provided by another embodiment of the present application applied to a working node of a second device cluster in a distributed system. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The data verification method provided by the embodiment of the present application includes steps: S701 to S708.

[0097] S701, detecting whether the current comparison round is less than a preset number threshold.

[0098] S702: If the current comparison round is greater than or equal to a preset number threshold, the verification result is determined as a second preset result.

[0099] For details of steps S701-S702, please refer to the above Figure 6 The detailed description of steps S30351-S30352 is not repeated here.

[0100] S703, generating warning information of the verification result according to the difference data.

[0101] In at least one embodiment of the present application, the warning information may carry difference data, wherein the warning information may be that the order number 00100 in the target table data is different from the order number 00100 in the source table data.

[0102] S704, sending warning information to the designated user.

[0103] In at least one embodiment of the present application, the designated user may be the specific person in charge of the data verification request. The working node may send the warning information to the binding terminal of the designated user, and the working node may also send the warning information to the binding account of the designated user. Among them, the binding terminal may be a terminal device held by the designated user, and the binding account may be the account information of the designated user on a certain platform. This embodiment ensures that the designated user can receive the warning information in a timely manner by sending the warning information to the binding terminal and / or binding account of the designated user.

[0104] S705, if the current comparison round is less than the preset number threshold, obtain the first comparison data from the target database according to the data primary key of the difference data, and obtain the second comparison data from the source database according to the data foreign key of the first comparison data.

[0105] S706: Detect whether the first comparison data and the second comparison data are the same.

[0106] S707: If the first comparison data is different from the second comparison data, the current comparison round is accumulated and the process returns to step S701.

[0107] S708: If the first comparison data is the same as the second comparison data, the verification result is determined as the first preset result.

[0108] For details of steps S705-S708, please refer to the above Figure 6 The detailed description of steps S30353-S30356 is not repeated here.

[0109] like Figure 8 , is a functional module diagram of a data verification device provided in an embodiment of the present application. The data verification device 11 runs on a working node of a second device cluster in a distributed system, and the distributed system also includes a first device cluster. The data verification device 11 includes a generation unit 110, an acquisition unit 111, and a sending unit 112. The module / unit referred to in this application refers to a series of computer-readable instruction segments that can be acquired by a processor 13 and can perform fixed functions, which are stored in a memory 12.

[0110] The generation unit 110 is used to generate a data acquisition strategy in response to a data verification request received from the first device cluster, wherein the data verification request includes library identifiers of a source database and a target database; the acquisition unit 111 is used to acquire source table data from the source database based on the data acquisition strategy, and acquire target table data from the target database based on the source table data; the generation unit 110 is also used to generate a verification result based on a comparison between the source table data and the target table data.

[0111] In one embodiment, the generation unit 110 includes: a comparison subunit, which is used to compare the source table data with the target table data if the data format of the source table data is the same as the data format of the target table data; a determination subunit, which is used to determine the current comparison round of the source table data and the target table data if there is difference data in the target table data that is different from the source table data; and a determination subunit, which is used to determine the verification result according to the current comparison round and the difference data.

[0112] In one embodiment, the generation unit 110 includes: an acquisition subunit, which is used to obtain first comparison data from the target database according to the data primary key of the difference data if the current comparison round is less than a preset number threshold, and obtain second comparison data from the source database according to the data foreign key of the first comparison data; a determination subunit, which is used to determine the verification result as a first preset result if the first comparison data is the same as the second comparison data, and the first preset result indicates that the source table data is consistent with the target table data.

[0113] In one embodiment, the generation unit 110 includes: a determination subunit, configured to determine the verification result as a second preset result if the current comparison round is greater than or equal to a preset number threshold, and the second preset result indicates that inconsistent data exists between the target table data and the source table data.

[0114] Furthermore, the generating unit 110 is further configured to generate warning information of the verification result according to the difference data if the verification result is a second preset result; and the sending unit 112 is configured to send the warning information to a designated user.

[0115] The embodiment of the present application can obtain reasonable source table data for data comparison through the generated data acquisition strategy. Since there is no need to compare the full amount of data in the source database, the efficiency of generating the verification result can be ensured, thereby avoiding the impact of excessive load on the system. At the same time, the acquisition of source table data through the data acquisition strategy can avoid the problem of uneven data sampling, thereby improving the accuracy of verification. In addition, through the source table data, the target table data can be accurately obtained for data comparison, thereby ensuring the accuracy of data verification.

[0116] like Fig. 9 , which is a schematic diagram of the structure of a working node for implementing a data verification method provided in an embodiment of the present application.

[0117] In one embodiment of the present application, the working node 1000 includes, but is not limited to, a memory 12, a processor 13, and computer-readable instructions stored in the memory 12 and executable on the processor 13, such as a data verification program.

[0118] Those skilled in the art will understand that the schematic diagram is merely an example of the working node 1000 and does not constitute a limitation on the working node 1000. The working node 1000 may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the working node 1000 may also include input and output devices, network access devices, buses, etc.

[0119] The processor 13 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 13 is the computing core and control center of the working node 1000, and uses various interfaces and lines to connect various parts of the entire working node 1000, and executes the operating system of the working node 1000 and various installed applications, program codes, etc.

[0120] Exemplarily, the computer-readable instructions may be divided into one or more modules / units, one or more modules / units are stored in the memory 12, and are executed by the processor 13 to complete the present application. One or more modules / units may be a series of computer-readable instruction segments capable of completing a specific function, and the computer-readable instruction segments are used to describe the execution process of the computer-readable instructions in the work node 1000. For example, the computer-readable instructions may be divided into a generation unit 110, an acquisition unit 111, and a sending unit 112.

[0121] The memory 12 can be used to store computer-readable instructions and / or modules. The processor 13 implements various functions of the working node 1000 by running or executing the computer-readable instructions and / or modules stored in the memory 12, and calling the data stored in the memory 12. The memory 12 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the working node 1000, etc. The memory 12 can include non-volatile and volatile memories, such as: a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other storage devices.

[0122] The memory 12 may be an external memory and / or an internal memory of the working node 1000. Furthermore, the memory 12 may be a memory in a physical form, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0123] If the modules / units integrated in the working node 1000 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware to complete through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the computer-readable instructions are executed by the processor, the steps of the above-mentioned various method embodiments can be implemented.

[0124] The computer-readable instructions include computer-readable instruction codes, which may be in source code form, object code form, executable files, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer-readable instruction codes, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM).

[0125] Combination Figure 2-7 The memory 12 in the working node 1000 stores computer-readable instructions, and the processor 13 can execute the computer-readable instructions stored in the memory 12 to implement the following Figure 2-7 The data verification method shown.

[0126] Specifically, the specific implementation method of the processor 13 for the above-mentioned computer readable instructions can refer to Figure 2-7 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.

[0128] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0130] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present application is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0131] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A data verification method, applied to a distributed system, characterized in that: The distributed system includes a first device cluster and a second device cluster, and the method includes: The first device cluster receives node indicator information reported by all working nodes of the second device cluster; The first device cluster selects a target node from all working nodes of the second device cluster according to the node indicator information; The first device cluster sends a data verification request to the target node; The target node generates a data acquisition strategy according to the data verification request, wherein the data verification request includes database identifiers of the source database and the target database; The target node acquires source table data from the source database based on the data acquisition strategy, and acquires target table data from the target database based on the source table data; The target node generates a verification result based on a comparison between the source table data and the target table data.

2. A data verification method, characterized in that: The method is applied to a working node of a second device cluster in a distributed system, wherein the distributed system also includes a first device cluster. generating a data acquisition strategy in response to a data verification request received from the first device cluster, wherein the data verification request includes library identifiers of a source database and a target database; Based on the data acquisition strategy, source table data is acquired from the source database, and target table data is acquired from the target database based on the source table data; Based on the comparison between the source table data and the target table data, a verification result is generated.

3. The data verification method according to claim 2, characterized in that: The generating a verification result based on the comparison between the source table data and the target table data includes: If the data format of the source table data is the same as the data format of the target table data, comparing the source table data with the target table data; If there is difference data in the target table data that is different from the source table data, determining a current comparison round between the source table data and the target table data; The verification result is determined according to the current comparison round and the difference data.

4. The data verification method according to claim 3, characterized in that: Determining the verification result according to the current comparison round and the difference data includes: If the current comparison round is less than a preset number threshold, first comparison data is obtained from the target database according to the data primary key of the difference data, and second comparison data is obtained from the source database according to the data foreign key of the first comparison data; If the first comparison data is the same as the second comparison data, the verification result is determined as a first preset result, and the first preset result indicates that the source table data is consistent with the target table data.

5. The data verification method according to claim 4, characterized in that: The determining the verification result according to the current comparison round and the difference data further includes: If the current comparison round is greater than or equal to the preset number threshold, the verification result is determined as a second preset result, and the second preset result indicates that inconsistent data exists between the target table data and the source table data.

6. The data verification method according to claim 5, characterized in that: The method further comprises: If the verification result is the second preset result, generating warning information of the verification result according to the difference data; Send the warning information to the designated user.

7. The data verification method according to claim 2, characterized in that: The generating a data acquisition strategy in response to the data verification request received from the first device cluster includes: Reading data from the source database according to the request type of the data verification request, and determining the amount of the read data; Determining a verification method according to the data volume and a preset quantity threshold; If the verification method is a preset method, the data acquisition strategy is generated according to the configuration table information in the source database and the sampling condition of the configuration table information.

8. A data verification device, characterized in that: A working node running in a second device cluster in a distributed system, wherein the distributed system also includes a first device cluster, and the apparatus includes: A generating unit, configured to generate a data acquisition strategy in response to a data verification request received from the first device cluster, wherein the data verification request includes library identifiers of a source database and a target database; an acquisition unit, configured to acquire source table data from the source database based on the data acquisition strategy, and acquire target table data from the target database based on the source table data; The generating unit is further configured to generate a verification result based on a comparison between the source table data and the target table data.

9. A distributed system, characterized in that: include: A first device cluster and a second device cluster, wherein the second device cluster includes working nodes; The working node includes: a memory storing computer-readable instructions; and A processor executes computer-readable instructions stored in the memory to implement the data verification method according to any one of claims 2 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions are executed by a processor in a working node to implement the data verification method as described in any one of claims 2 to 7.

Images