Data verification method, device and equipment and readable storage medium

Abstract

The invention discloses a data verification method, apparatus and device, and a readable storage medium. The method comprises the steps of obtaining a general table identifier and a sub-library rule of a target general table; wherein the sub-library rule comprises a first field variable used for calculating a business data sub-library route; selecting multiple pieces of to-be-verified service information from the target general table; based on a sub-library rule, determining a target sub-table identifier of each piece of to-be-verified service information, and extracting target service data corresponding to the to-be-verified service information; checking each piece of to-be-checked business information and the corresponding target business data; if all the to-be-verified business information is consistent with the corresponding target business data, determining that the sub-library is correct; and if any piece of to-be-detected business information is different from the corresponding target business data, determining that a sub-library error exists. It can be seen that the accuracy verification of the sub-library data can be realized by comparing the target general table data with the sub-library storage data, and the resource loss is reduced and the verification efficiency is improved while the verification reliability is ensured.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data verification method, apparatus, device, and readable storage medium. Background Technology

[0002] In the business domain, the data volume of business data tables is growing explosively, with the data storage size of some core business single tables exceeding 100GB, and even reaching the TB level in some scenarios. Such giant data tables directly cause a significant decrease in data query response performance, severely restricting business response efficiency.

[0003] To address the aforementioned technical challenges, existing technologies commonly employ data table splitting, which involves dividing a massive data table and distributing it across multiple databases. However, during the splitting and distributed storage of massive data tables, issues such as data storage errors and data integrity breaches can easily arise due to system hardware and software failures, network transmission anomalies, and other factors. Because the split data is distributed across multiple storage nodes and has complex relationships, locating and troubleshooting data storage errors becomes extremely difficult. Development and maintenance teams for various business systems face the technical challenge of verifying the accuracy of the split data, necessitating a reliable data accuracy verification solution. Summary of the Invention

[0004] In view of this, this application provides a data verification method, apparatus, device, and readable storage medium for verifying the accuracy of split business data.

[0005] To achieve the above objectives, the following solution is proposed:

[0006] A data validation method, comprising:

[0007] Obtain the master table identifier of the target master table, and obtain the database sharding rules of the target master table; wherein, the database sharding rules include a first field variable used to calculate the database sharding route for business data;

[0008] Select multiple business information items to be verified from the target master table;

[0009] Based on the database sharding rules, the target sharding identifier for each piece of business information to be verified is determined;

[0010] Extract the target business data corresponding to the business information to be verified from the target database table corresponding to each target table identifier;

[0011] Each piece of business information to be verified and its corresponding target business data are checked;

[0012] If all the business information to be verified is consistent with its corresponding target business data, then the database sharding is confirmed to be correct.

[0013] If any piece of business information to be inspected differs from its corresponding target business data, then a database sharding error is identified.

[0014] Optionally, multiple pieces of business information to be verified can be selected from the target master table, including:

[0015] Determine the number of business inspections;

[0016] Based on the priority sorting of different business information in the target master table, select multiple business information items from the target master table whose quantity matches the number of business checks, as multiple business information items to be inspected.

[0017] Optionally, determining the target sharding identifier for each piece of business information to be verified based on the sharding rules includes:

[0018] When the target master table does not have corresponding mapping table matching information and associated table matching information, extract the shard key value of each business information to be verified.

[0019] Replace the first field variable in the database sharding rule with the sharding key value of each business information to be verified in turn, and construct the routing calculation statement corresponding to each business information to be verified.

[0020] Execute each route calculation statement to obtain the target sub-table identifier for the corresponding business information to be verified.

[0021] Optionally, determining the target sharding identifier for each piece of business information to be verified based on the sharding rules includes:

[0022] When there is mapping table matching information corresponding to the target master table, the mapping database sharding rule is determined according to the mapping table matching information. The mapping database sharding rule includes a second field variable for calculating the mapping data database sharding route.

[0023] Determine if each piece of business information to be verified contains a business sharding field value;

[0024] If so, the first field variable in the database sharding rule is replaced with the corresponding business database sharding field value, the first route calculation statement corresponding to the business information to be verified is constructed and executed, and the target sharding identifier of the corresponding business information to be verified is determined.

[0025] If not, then based on the mapping sharding rules, determine the business sharding field value of the corresponding business information to be verified; proceed to the step of replacing the first field variable in the sharding rules with the corresponding business sharding field value, constructing and executing the first route calculation statement of the corresponding business information to be verified, and determining the target sharding identifier of the corresponding business information to be verified.

[0026] Optionally, determining the business sharding field value corresponding to the business information to be verified based on the mapping sharding rules includes:

[0027] Determine the mapping field values ​​for the corresponding business information to be verified;

[0028] Replace the second field variable with the mapping field value, construct and execute the mapping lookup statement to find the mapping table of the corresponding business information to be verified;

[0029] Extract the business sub-database field values ​​corresponding to the business information to be verified from the mapping table.

[0030] Optionally, determining the target sharding identifier for each piece of business information to be verified based on the sharding rules includes:

[0031] When there is matching information for related tables in the target master table, the field mapping relationship is determined based on the matching information for related tables.

[0032] Determine if each piece of business information to be verified contains a business sharding field value;

[0033] If so, the first field variable in the database sharding rule is replaced with the corresponding business database sharding field value, a second routing calculation statement for the corresponding business information to be verified is constructed and executed, and the target sharding identifier for the corresponding business information to be verified is determined.

[0034] If not, analyze the field mapping relationship to determine the business sharding field value corresponding to the business information to be verified; proceed to the step of replacing the first field variable in the sharding rule with the corresponding business sharding field value, constructing and executing the second route calculation statement corresponding to the business information to be verified, and determining the target sharding identifier of the corresponding business information to be verified.

[0035] Optionally, the step of analyzing the field mapping relationship to determine the business database field value corresponding to the business information to be verified includes:

[0036] Analyze the field mapping relationship to determine the associated field types of the corresponding business information to be verified;

[0037] Extract the associated fields whose types match the corresponding business information to be verified;

[0038] Based on the associated fields, determine the values ​​of the corresponding business sub-database fields for the business information to be verified.

[0039] A data verification device, comprising:

[0040] The acquisition module is used to acquire the master table identifier of the target master table and the database sharding rules of the target master table; wherein, the database sharding rules include a first field variable used to calculate the database sharding route for business data;

[0041] The selection module is used to select multiple pieces of business information to be verified from the target master table;

[0042] The determination module is used to determine the target table identifier for each piece of business information to be verified based on the database sharding rules.

[0043] The extraction module is used to extract target business data corresponding to the business information to be verified from the target database table corresponding to each target table identifier;

[0044] The verification module is used to verify each piece of business information to be verified and its corresponding target business data. If all the business information to be verified is consistent with its corresponding target business data, the database sharding is determined to be correct. If any piece of business information to be verified is different from its corresponding target business data, the database sharding error is determined to exist.

[0045] A data verification device, comprising a memory and a processor;

[0046] The memory is used to store programs;

[0047] The processor is used to execute the program to implement each step of the above-described data verification method.

[0048] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data verification method described above.

[0049] As can be seen from the above technical solution, the data verification method provided in this application can obtain the master table identifier of the target master table and the sharding rules of the target master table; wherein, the sharding rules include a first field variable for calculating the sharding route of business data; multiple business information to be verified are selected from the target master table; thus, this application can extract key business data for verification, avoiding the problems of long verification time, high resource consumption and low verification efficiency caused by verifying all business data one by one; subsequently, this application determines the target sharding identifier of each business information to be verified based on the sharding rules; extracts the target business data of the corresponding business information to be verified from the target sharding table corresponding to each target sharding identifier; it can realize the combination of the first field variable of the sharding rules to perform routing calculation for each business information to be verified, and locate the theoretical sharding of the corresponding business information to be verified. The application involves creating a database and theoretical partitioning tables. The actual stored data of the business information to be verified is obtained from the theoretical partitioning tables, establishing a comparison data source with the original business information to ensure accurate correspondence between the obtained actual data and the information to be verified, avoiding data extraction misalignment that could affect the verification results. Then, each piece of business information to be verified is compared with its corresponding target business data. If all the business information to be verified is consistent with its corresponding target business data, the database partitioning is considered correct. If any piece of business information to be verified differs from its corresponding target business data, a database partitioning error is identified. This allows for the determination of whether the splitting and storage of individual data is accurate, transforming the correctness of database partitioning into a directly verifiable data consistency result. Based on the sample verification results, it can be inferred that the database partitioning rules were executed correctly and that there were no data errors in the partitioning process, providing a conclusion to support the reliability of the database partitioning scheme. This enables timely detection of database partitioning storage anomalies and troubleshooting of database partitioning faults. Therefore, this application can verify the accuracy of database partitioning data by comparing the target master table data with the partitioned database storage data, ensuring verification reliability while reducing resource consumption and improving verification efficiency. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a flowchart of a data verification method disclosed in an embodiment of this application;

[0052] Figure 2 This is a structural block diagram of a data verification device disclosed in an embodiment of this application;

[0053] Figure 3 This is a hardware structure block diagram of a data verification device disclosed in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] This application provides a data verification method that can be applied to various business processing systems or data management systems, as well as to various computer terminals or smart terminals. The executing entity can be the processor or server of the computer terminal or smart terminal.

[0056] Next, combine Figure 1 The data verification method of this application is described in detail, including the following steps:

[0057] Step S1: Obtain the master table identifier of the target master table and obtain the database sharding rules of the target master table.

[0058] Specifically, the target master table can be stored in the source database.

[0059] The source database can contain multiple master tables, and the validation task of each master table in the source database can be used as one of the data validation tasks.

[0060] It can determine the target database for each master table in the distributed storage source database and obtain the connection information between the source database and the target database.

[0061] Each data verification task can be completed sequentially.

[0062] The master table corresponding to the current data verification task can be used as the target master table.

[0063] The target master table can store different business information.

[0064] Each piece of business information can contain multi-dimensional business-related data of the same entity.

[0065] The subject can be of various types, such as users and products.

[0066] Different types of subjects can correspond to different dimensions of relevant data.

[0067] For example, the multi-dimensional business-related data corresponding to a product may include the product's source, manufacturer, product ID, and price.

[0068] The distribution rules of the target master table can be used as the database distribution rules.

[0069] Sharding rules can include a first field variable used to calculate the sharding route for business data.

[0070] Database sharding rules can be implemented using DML syntax.

[0071] Step S2: Select multiple business information items to be verified from the target master table.

[0072] Specifically, the various business information items to be verified in the target master table can be prioritized and selected as the business information items to be verified.

[0073] Step S3: Based on the database sharding rules, determine the target sharding identifier for each business information to be verified.

[0074] Specifically, based on the database sharding rules, the variable value of the first field can be updated, and the routing calculation can be performed on each business information to be verified to obtain the target sharding identifier for each business information to be verified.

[0075] Step S4: Extract the target business data corresponding to the business information to be verified from the target database table corresponding to each target table identifier.

[0076] Specifically, the positioning dimension can be determined based on multi-dimensional information such as data matching fields, field mapping relationships, lists of fields to be ignored, and distinguishing fields for each business information to be verified; for example, the positioning dimension can be the product ID.

[0077] Find the dimension value that matches the positioning dimension from each piece of business information to be verified;

[0078] For each business information to be verified, search the target database table corresponding to the target partition identifier and find the stored business data that matches the corresponding dimension value as the target business data.

[0079] Step S5: Verify each piece of business information to be verified and its corresponding target business data; if all the business information to be verified and its corresponding target business data are consistent, then the database sharding is confirmed to be correct; if any piece of business information to be verified is different from its corresponding target business data, then the database sharding error is confirmed to exist.

[0080] Specifically, for each piece of business information to be verified, the consistency of the business information to be verified can be compared with the dimension values ​​of the same dimension in the corresponding target business data; if the dimension value of any dimension is different, it can be determined that the business information to be verified is different from its corresponding target business data; if the dimension values ​​of all dimensions are the same, it can be determined that the business information to be verified is the same as its corresponding target business data.

[0081] If any pending business information differs from its corresponding target business data, a storage sharding error can be determined. Furthermore, the total number of different pending business information entries can be counted, and the abnormal content and abnormal dimension name of each different pending business information entry can be identified. Based on the master table identifier, abnormal content, abnormal dimension name, and total number, an abnormal alert is generated and transmitted to the administrator's terminal via email and / or SMS.

[0082] It can also record other abnormal situations such as connection timeouts, data processing errors, and routing calculation errors.

[0083] If all the business information to be verified is the same as its corresponding target business data, then it can be determined that the database sharding of the target master table is correct.

[0084] As can be seen from the above technical solution, the data verification method provided in this application can obtain the master table identifier of the target master table and the sharding rules of the target master table; wherein, the sharding rules include a first field variable for calculating the sharding route of business data; multiple business information to be verified are selected from the target master table; thus, this application can extract key business data for verification, avoiding the problems of long verification time, high resource consumption and low verification efficiency caused by verifying all business data one by one; subsequently, this application determines the target sharding identifier of each business information to be verified based on the sharding rules; extracts the target business data of the corresponding business information to be verified from the target sharding table corresponding to each target sharding identifier; it can realize the combination of the first field variable of the sharding rules to perform routing calculation for each business information to be verified, and locate the theoretical sharding of the corresponding business information to be verified. The application involves creating a database and theoretical partitioning tables. The actual stored data of the business information to be verified is obtained from the theoretical partitioning tables, establishing a comparison data source with the original business information to ensure accurate correspondence between the obtained actual data and the information to be verified, avoiding data extraction misalignment that could affect the verification results. Then, each piece of business information to be verified is compared with its corresponding target business data. If all the business information to be verified is consistent with its corresponding target business data, the database partitioning is considered correct. If any piece of business information to be verified differs from its corresponding target business data, a database partitioning error is identified. This allows for the determination of whether the splitting and storage of individual data is accurate, transforming the correctness of database partitioning into a directly verifiable data consistency result. Based on the sample verification results, it can be inferred that the database partitioning rules were executed correctly and that there were no data errors in the partitioning process, providing a conclusion to support the reliability of the database partitioning scheme. This enables timely detection of database partitioning storage anomalies and troubleshooting of database partitioning faults. Therefore, this application can verify the accuracy of database partitioning data by comparing the target master table data with the partitioned database storage data, ensuring verification reliability while reducing resource consumption and improving verification efficiency.

[0085] In some embodiments of this application, the process of selecting multiple pieces of business information to be verified from the target master table is described in detail, and the steps are as follows:

[0086] S20. Determine the number of business inspections.

[0087] Specifically, the total number of business operations that need to be inspected can be used as the number of business inspections.

[0088] Specifically, the inspection ratio can be determined based on the quantitative analysis results of accuracy and inspection efficiency corresponding to the number of historical inspections; for example, the ratio corresponding to the optimal overall performance of accuracy and inspection efficiency can be used as the inspection ratio.

[0089] Determine the total number of business transactions in the target master table, calculate the product between the total number of business transactions and the inspection ratio, and round the product to obtain the number of business transactions to be inspected.

[0090] S21. Based on the priority sorting of different business information in the target master table, select multiple business information from the target master table whose quantity matches the number of business checks, as multiple business information to be inspected.

[0091] Specifically, the modification time of each piece of business information can be determined, and the business information in the target master table can be prioritized according to the order of modification time.

[0092] Select multiple business information items whose latest modification time matches the number of business checks from the sorting results as the business information items to be inspected.

[0093] As can be seen from the above technical solution, this embodiment provides an optional method for selecting multiple business information items to be verified from the target master table. This method allows for the determination of business information to be verified based on its modification time, ensuring that verification covers the most recently modified business data, thereby enabling more timely detection of potential data storage problems.

[0094] In some embodiments of this application, considering that whether the target master table contains associated tables and mapping tables has a significant impact on the determination of the target sub-table, different execution steps S3 can be adopted for different situations to determine the target sub-table identifier for each business information to be verified based on the database sharding rules. The various processes will be described in detail below:

[0095] The first type

[0096] S30. When there is no matching information in the mapping table or the associated table in the target master table, extract the shard key value of each business information to be verified.

[0097] Specifically, it can be determined whether the target master table has matching mapping table information and related table information. If no matching mapping table information or related table information exists, the field value corresponding to the sharding field in each business information to be verified can be extracted as the sharding key value. For example, it can be the product ID.

[0098] S31. Replace the first field variable in the database sharding rule with the sharding key value of each business information to be verified in turn, and construct the routing calculation statement corresponding to each business information to be verified.

[0099] Specifically, for each piece of business information to be verified, the sharding key value corresponding to the business information to be verified can be used as the variable value of the first field variable to construct the routing calculation SQL statement corresponding to the business information to be verified.

[0100] S32. Execute each route calculation statement to obtain the target sub-table identifier of the corresponding service information to be verified.

[0101] Specifically, each route calculation statement can be executed to obtain the target sub-table identifier for each service information to be verified.

[0102] The second type

[0103] S30. When there is mapping table matching information corresponding to the target master table, the mapping database rule is determined according to the mapping table matching information. The mapping database rule includes a second field variable for calculating the mapping data database route.

[0104] Specifically, when the mapping table matching information of the target master table can be found, the mapping table matching information can be analyzed to determine the mapping database rules;

[0105] The mapping database sharding rule includes a second field variable used to calculate the mapping data sharding route.

[0106] S31. Determine if each business information to be verified has a business sharding field value. If yes, proceed to step S32; otherwise, proceed to step S33.

[0107] Specifically, it can be analyzed whether each business information to be verified contains a field value corresponding to the sharding field. If so, the field value corresponding to the sharding field can be used as the business sharding field value, and step S32 can be executed; if not, step S33 can be executed.

[0108] S32. Replace the first field variable in the database sharding rule with the corresponding business database sharding field value, construct and execute the first route calculation statement for the corresponding business information to be verified, and determine the target sharding identifier for the corresponding business information to be verified.

[0109] Specifically, the first field variable in the sharding rule can be assigned the value of the business sharding field, and the first route calculation SQL statement corresponding to the business information to be verified can be constructed and executed to determine the target sharding identifier of the corresponding business information to be verified.

[0110] S33. Based on the mapping database rules, determine the business database field values ​​corresponding to the business information to be verified; proceed to execution step S32.

[0111] Specifically, the mapping database rules can be analyzed to find the business database field values ​​of the target business information to be verified from the mapping table.

[0112] Among them, the target business information to be verified refers to the business information to be verified that does not contain the business sub-database field value in the target master table with corresponding mapping table matching information.

[0113] The third type

[0114] S30. When there is matching information for related tables in the target master table, determine the field mapping relationship based on the matching information for related tables.

[0115] Specifically, the matching information of the associated tables in the target master table can be analyzed to identify the mapping relationships between different fields.

[0116] S31. Determine if each business information to be verified has a business sharding field value. If yes, proceed to step S32; otherwise, proceed to step S33.

[0117] Specifically, it can be analyzed whether each business information to be verified contains a field value corresponding to the sharding field. If so, the field value corresponding to the sharding field can be used as the business sharding field value, and step S32 can be executed; if not, step S33 can be executed.

[0118] S32. Replace the first field variable in the database sharding rule with the corresponding business database sharding field value, construct and execute the second routing calculation statement for the corresponding business information to be verified, and determine the target sharding identifier for the corresponding business information to be verified.

[0119] Specifically, the value of each business shard field can be used as the variable value of the first field variable in the sharding rule, and the second route calculation SQL statement corresponding to the business information to be verified can be constructed and executed to determine the target shard identifier of the corresponding business information to be verified.

[0120] It should be noted that the first route calculation statement is formed by the business database field values ​​of the business information to be verified that corresponds to mapping table matching information; the second route calculation statement is formed by the business database field values ​​of the business information to be verified that corresponds to association table matching information; if any business information to be verified corresponds to both mapping table matching information and association table matching information, then the route calculation statement formed by the business database field values ​​found according to the mapping database rules is the first route calculation statement; the route calculation statement formed by the business database field values ​​found according to the association table matching information is the second route calculation statement.

[0121] S33. Analyze the field mapping relationship and determine the corresponding business sub-database field value of the business information to be verified; proceed to execution step S32.

[0122] Specifically, the field mapping relationship can be analyzed to find the business sub-database field values ​​of specific business information to be verified from the associated table.

[0123] Among them, the specific business information to be verified refers to the business information to be verified that does not contain the business sub-database field value in the target master table with corresponding related table matching information.

[0124] As can be seen from the above technical solutions, this embodiment provides three optional methods for determining the target sharding identifier for each piece of business information to be verified based on the sharding rules. Through these methods, the target sharding identifier for each piece of business information to be verified can be flexibly determined depending on whether the target master table contains related tables and mapping tables. This makes the sharding verification process more comprehensive and accurate, adaptable to data verification needs in different scenarios, and effectively improves the reliability and efficiency of data verification.

[0125] In some embodiments of this application, the process of determining the business database field value corresponding to the business information to be verified based on the mapping database rules is described in detail, and the steps are as follows:

[0126] S330. Determine the mapping field value of the corresponding business information to be verified.

[0127] Specifically, the mapping database rules can be analyzed to determine the mapping fields used to look up the values ​​of business database fields;

[0128] The field value of the corresponding business information to be verified can be found in the mapping table and used as the mapping field value. For example, if the business database field value is product ID, and the mapping database rule is that product links can be mapped to product IDs, then the product link can be specified as the mapping field to associate with the business database field value. Therefore, the specific field value of the product link corresponding to the business information to be verified can be found in the mapping table and used as the mapping field value.

[0129] S331. Replace the second field variable with the mapping field value, construct and execute the mapping lookup statement to find the mapping table of the corresponding business information to be verified.

[0130] Specifically, the second field variable in the mapping database rule can be replaced with the determined mapping field value, and an SQL statement for mapping lookup can be constructed as the mapping lookup statement. This mapping lookup statement can then be executed to locate the mapping table of the corresponding business information to be verified.

[0131] S332. Extract the business sub-database field values ​​corresponding to the business information to be verified from the mapping table.

[0132] Specifically, the product ID that matches the mapping field value can be extracted from the mapping table as the business sharding field value based on the specific mapping field value.

[0133] As can be seen from the above technical solution, this embodiment provides an optional method for determining the business shard field values ​​corresponding to the business information to be verified based on the mapping sharding rules. Through this method, when mapping table matching information exists in the target master table, the impact of the mapping table matching information on determining the business shard field values ​​can be fully considered. This allows the entire data verification process to adapt to different data structure situations, improving the universality and reliability of data verification. It effectively avoids the problem of incorrect determination of business shard field values ​​due to failure to consider mapping table matching information, which in turn affects the subsequent determination of the target shard identifier and the overall data verification result.

[0134] In some embodiments of this application, the process of step S33, analyzing the field mapping relationship, and determining the business database field value corresponding to the business information to be verified is described in detail, and the steps are as follows:

[0135] S330. Analyze the field mapping relationship to determine the associated field type of the corresponding business information to be verified.

[0136] Specifically, the field mapping relationship can be analyzed to determine the type of the associated field in the associated table for each piece of business information to be verified.

[0137] For example, if we use the product price field as the database sharding field, the associated table is the order details table, and the target master table is the product master table, the associated fields might include product ID, order number, etc. We can retrieve the product price field value from the order details table based on the product ID. In this case, we can use product ID as the associated field type.

[0138] S331. Extract the associated fields whose types match the corresponding business information to be verified.

[0139] Specifically, based on the determined associated field types, relevant content matching the associated fields can be extracted from the business information to be verified.

[0140] For example, if the associated field type is product ID, then the specific value of the product ID is extracted from the business information to be verified as the associated field.

[0141] S332. Based on the associated fields, determine the business sub-database field values ​​corresponding to the business information to be verified.

[0142] Specifically, the product price field value that matches the product ID can be found in the key table and used as the business database sharding field value.

[0143] As can be seen from the above technical solution, this embodiment provides an optional method for analyzing the field mapping relationship to determine the business sub-database field values ​​corresponding to the business information to be verified. Using this method, when there is matching information in the associated tables within the target master table, the field mapping relationship in the associated tables can be fully utilized to determine the business sub-database field values. This allows the application to adapt to various complex data structure scenarios, ensuring accurate acquisition of business sub-database field values ​​under different circumstances and avoiding errors in field value determination due to differences in data structures.

[0144] Next, we will combine Figure 2 The data verification device provided in this application is described in detail below. The data verification device described below can be compared with the data verification method provided above.

[0145] See Figure 2 It can be observed that the data verification device may include:

[0146] The acquisition module 10 is used to acquire the master table identifier of the target master table and acquire the database sharding rules of the target master table; wherein, the database sharding rules include a first field variable used to calculate the database sharding route of business data;

[0147] Selection module 20 is used to select multiple business information to be verified from the target master table;

[0148] The determination module 30 is used to determine the target table identifier for each business information to be verified based on the database sharding rules.

[0149] Extraction module 40 is used to extract target business data corresponding to the business information to be verified from the target database table corresponding to each target database table identifier;

[0150] The verification module 50 is used to verify each business information to be verified and its corresponding target business data. If all the business information to be verified and its corresponding target business data are consistent, the database sharding is determined to be correct. If any business information to be verified is different from its corresponding target business data, the database sharding error is determined to exist.

[0151] Furthermore, module 20 may include:

[0152] The first selection unit is used to determine the number of business checks;

[0153] The second selection unit is used to select multiple pieces of business information from the target master table whose quantity matches the number of business checks, based on the priority sorting of different business information in the target master table, as multiple pieces of business information to be checked.

[0154] Furthermore, module 30 may include:

[0155] The sharding key value extraction unit is used to extract the sharding key value of each business information to be verified when the target master table does not have corresponding mapping table matching information and association table matching information.

[0156] The routing calculation statement construction unit is used to replace the first field variable in the database sharding rule with the sharding key value of each business information to be verified in turn, and construct the routing calculation statement corresponding to each business information to be verified.

[0157] The target partition identifier acquisition unit is used to execute each route calculation statement to obtain the target partition identifier of the corresponding service information to be verified.

[0158] Furthermore, module 30 may also include:

[0159] The mapping database sharding rule determination unit is used to determine the mapping database sharding rule based on the mapping table matching information when there is mapping table matching information corresponding to the target master table. The mapping database sharding rule includes a second field variable for calculating the mapping data sharding route.

[0160] The business sharding field value identification unit is used to determine whether each piece of business information to be verified has a business sharding field value; if so, the field variable replacement unit is called; if not, the mapping sharding rule utilization unit is called.

[0161] The field variable replacement unit is used to replace the first field variable in the database sharding rule with the corresponding business database sharding field value, construct and execute the first route calculation statement for the corresponding business information to be verified, and determine the target sharding identifier for the corresponding business information to be verified.

[0162] The mapping sharding rule utilization unit is used to determine the business sharding field values ​​corresponding to the business information to be verified based on the mapping sharding rule; and to call the field variable replacement unit.

[0163] Furthermore, the mapping database sharding rule utilization unit may include:

[0164] The first mapping database sharding rule uses sub-units to determine the mapping field values ​​of the corresponding business information to be verified;

[0165] The second mapping database rule uses sub-units to replace the second field variable with the mapping field value, construct and execute the mapping lookup statement, and look up the mapping table of the corresponding business information to be verified.

[0166] The third mapping database sharding rule utilizes sub-units to extract the business database sharding field values ​​corresponding to the business information to be verified from the mapping table.

[0167] Furthermore, module 30 may also include:

[0168] The field mapping relationship determination unit is used to determine the field mapping relationship based on the associated table matching information when the target master table has associated table matching information.

[0169] The pending business information judgment unit is used to determine whether each pending business information has a business database field value; if so, the target distribution identifier determination unit is called; if not, the field mapping relationship analysis unit is called.

[0170] The target distribution identifier determination unit is used to replace the first field variable in the database sharding rule with the corresponding business database sharding field value, construct and execute the second routing calculation statement for the corresponding business information to be verified, and determine the target sharding identifier for the corresponding business information to be verified.

[0171] The field mapping relationship analysis unit is used to analyze the field mapping relationship, determine the business sub-database field value corresponding to the business information to be verified, and call the target distribution identifier determination unit.

[0172] Furthermore, the field mapping relationship analysis unit may include:

[0173] The first field mapping relationship determination subunit is used to analyze the field mapping relationship and determine the associated field type of the corresponding business information to be verified.

[0174] The second field mapping relationship determination subunit is used to extract the associated field matching the associated field type from the corresponding business information to be verified;

[0175] The third field mapping relationship determination subunit is used to determine the business sub-database field value of the corresponding business information to be verified based on the associated field.

[0176] The data verification device provided in this application embodiment can be applied to data verification equipment, such as PC terminals, cloud platforms, servers, and server clusters. Optionally, Figure 3 The hardware structure block diagram of the data verification device is shown below. Figure 3 The hardware structure of the data verification device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0177] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0178] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0179] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0180] The memory stores a program, which the processor can call. The program is used for:

[0181] Obtain the master table identifier of the target master table, and obtain the database sharding rules of the target master table; wherein, the database sharding rules include a first field variable used to calculate the database sharding route for business data;

[0182] Select multiple business information items to be verified from the target master table;

[0183] Based on the database sharding rules, the target sharding identifier for each piece of business information to be verified is determined;

[0184] Extract the target business data corresponding to the business information to be verified from the target database table corresponding to each target table identifier;

[0185] Each piece of business information to be verified and its corresponding target business data are checked;

[0186] If all the business information to be verified is consistent with its corresponding target business data, then the database sharding is confirmed to be correct.

[0187] If any piece of business information to be inspected differs from its corresponding target business data, then a database sharding error is identified.

[0188] Optionally, the refined and extended functions of the program can be referred to the above description.

[0189] This application embodiment also provides a readable storage medium that can store a program suitable for execution by a processor, the program being used for:

[0190] Obtain the master table identifier of the target master table, and obtain the database sharding rules of the target master table; wherein, the database sharding rules include a first field variable used to calculate the database sharding route for business data;

[0191] Select multiple business information items to be verified from the target master table;

[0192] Based on the database sharding rules, the target sharding identifier for each piece of business information to be verified is determined;

[0193] Extract the target business data corresponding to the business information to be verified from the target database table corresponding to each target table identifier;

[0194] Each piece of business information to be verified and its corresponding target business data are checked;

[0195] If all the business information to be verified is consistent with its corresponding target business data, then the database sharding is confirmed to be correct.

[0196] If any piece of business information to be inspected differs from its corresponding target business data, then a database sharding error is identified.

[0197] Optionally, the refined and extended functions of the program can be referred to the above description.

[0198] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0199] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0200] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. The various embodiments of this application can be combined with each other. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data verification method, characterized in that, include: Obtain the master table identifier of the target master table, and obtain the database sharding rules of the target master table; wherein, the database sharding rules include a first field variable used to calculate the database sharding route for business data; Select multiple business information items to be verified from the target master table; Based on the database sharding rules, the target sharding identifier for each piece of business information to be verified is determined; Extract the target business data corresponding to the business information to be verified from the target database table corresponding to each target table identifier; Each piece of business information to be verified and its corresponding target business data are checked; If all the business information to be verified is consistent with its corresponding target business data, then the database sharding is confirmed to be correct. If any piece of business information to be inspected differs from its corresponding target business data, then a database sharding error is identified.

2. The data verification method according to claim 1, characterized in that, Select multiple pieces of business information to be verified from the target master table, including: Determine the number of business inspections; Based on the priority sorting of different business information in the target master table, select multiple business information items from the target master table whose quantity matches the number of business checks, as multiple business information items to be inspected.

3. The data verification method according to claim 1, characterized in that, The step of determining the target sharding identifier for each piece of business information to be verified based on the sharding rules includes: When the target master table does not have corresponding mapping table matching information and associated table matching information, extract the shard key value of each business information to be verified. Replace the first field variable in the database sharding rule with the sharding key value of each business information to be verified in turn, and construct the routing calculation statement corresponding to each business information to be verified. Execute each route calculation statement to obtain the target sub-table identifier for the corresponding business information to be verified.

4. The data verification method according to claim 1, characterized in that, The step of determining the target sharding identifier for each piece of business information to be verified based on the sharding rules includes: When there is mapping table matching information corresponding to the target master table, the mapping database sharding rule is determined according to the mapping table matching information. The mapping database sharding rule includes a second field variable for calculating the mapping data database sharding route. Determine if each piece of business information to be verified contains a business sharding field value; If so, the first field variable in the database sharding rule is replaced with the corresponding business database sharding field value, the first route calculation statement corresponding to the business information to be verified is constructed and executed, and the target sharding identifier of the corresponding business information to be verified is determined. If not, then based on the mapping sharding rules, determine the business sharding field value of the corresponding business information to be verified; proceed to the step of replacing the first field variable in the sharding rules with the corresponding business sharding field value, constructing and executing the first route calculation statement of the corresponding business information to be verified, and determining the target sharding identifier of the corresponding business information to be verified.

5. The data verification method according to claim 4, characterized in that, The step of determining the business sharding field value corresponding to the business information to be verified based on the mapping sharding rules includes: Determine the mapping field values ​​for the corresponding business information to be verified; Replace the second field variable with the mapping field value, construct and execute the mapping lookup statement to find the mapping table of the corresponding business information to be verified; Extract the business sub-database field values ​​corresponding to the business information to be verified from the mapping table.

6. The data verification method according to claim 1, characterized in that, The step of determining the target sharding identifier for each piece of business information to be verified based on the sharding rules includes: When there is matching information for related tables in the target master table, the field mapping relationship is determined based on the matching information for related tables. Determine if each piece of business information to be verified contains a business sharding field value; If so, the first field variable in the database sharding rule is replaced with the corresponding business database sharding field value, a second routing calculation statement for the corresponding business information to be verified is constructed and executed, and the target sharding identifier for the corresponding business information to be verified is determined. If not, analyze the field mapping relationship to determine the business sharding field value corresponding to the business information to be verified; proceed to the step of replacing the first field variable in the sharding rule with the corresponding business sharding field value, constructing and executing the second route calculation statement corresponding to the business information to be verified, and determining the target sharding identifier of the corresponding business information to be verified.

7. The data verification method according to claim 6, characterized in that, The analysis of the field mapping relationship to determine the corresponding business database field values ​​for the business information to be verified includes: Analyze the field mapping relationship to determine the associated field types of the corresponding business information to be verified; Extract the associated fields whose types match the corresponding business information to be verified; Based on the associated fields, determine the values ​​of the corresponding business sub-database fields for the business information to be verified.

8. A data verification device, characterized in that, include: The acquisition module is used to acquire the master table identifier of the target master table and the database sharding rules of the target master table; wherein, the database sharding rules include a first field variable used to calculate the database sharding route for business data; The selection module is used to select multiple pieces of business information to be verified from the target master table; The determination module is used to determine the target table identifier for each piece of business information to be verified based on the database sharding rules. The extraction module is used to extract target business data corresponding to the business information to be verified from the target database table corresponding to each target table identifier; The verification module is used to verify each piece of business information to be verified and its corresponding target business data. If all the business information to be verified is consistent with its corresponding target business data, the database sharding is determined to be correct. If any piece of business information to be verified is different from its corresponding target business data, the database sharding error is determined to exist.

9. A data verification device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the data verification method as described in any one of claims 1-7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the data verification method as described in any one of claims 1-7.

Images