Data verification method, storage medium and electronic equipment
Through the data verification method, the solution of data quality problems in the enterprise data integration process is simplified, personnel requirements and time are reduced, problem positioning and resolution efficiency is improved, and the generated statistical information is easy to analyze and implement solutions, and the existing technology is solved Data quality issues are complex and time-consuming.
Patent Information
- Application Number
- CN202411049239.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-08
AI Technical Summary
During the data integration process, enterprises have data quality problems caused by business personnel input problems, process defects, model construction problems and code bugs, which require professional personnel to locate, trace, analyze, and contact the person in charge of each system for a long time and personnel requirements. High, high recurrence rate of problem, reducing customer satisfaction.
Using the data verification method, by creating the verification object configuration table and element configuration table, configuring timing tasks, calling the verification program to verify the data, generating verification record statistics table and verification log table, outputting results, making it easier to analyze and Solve the problem.
The steps of positioning, traceability, analysis, and contacting the person in charge of each system are eliminated, the problem solving process is simplified, personnel requirements and time are reduced, problem positioning and resolution efficiency is improved, and the statistical information generated is convenient for analyzing the causes of the problem and implementing solutions.
Smart Images

Figure CN118916351A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field, and specifically relates to a data verification method, a storage medium and an electronic device. Background Art
[0002] As market competition becomes increasingly fierce and information construction continues to develop in depth, the entire industry has gradually carried out the construction of data middle platform projects. The overall requirements of information construction for unified standards, unified platforms, unified databases, and unified networks have been clarified, and system integration, resource integration, and information sharing have been gradually realized. This has not only put forward urgent requirements for the construction of enterprise-level data middle platforms, but also laid a solid foundation for the construction of enterprise data centers. In this process, data quality problems caused by business personnel input problems, process defects, model construction problems, code BUG problems, etc. have always been a major problem that major companies want to solve.
[0003] Data is one of the most valuable assets of an organization. There is a direct link between the quality of an enterprise's data and business performance. High-quality data can keep an enterprise competitive and invincible during economic turmoil. With universal and in-depth data quality, enterprises can trust all data to meet all needs at all times. There are many subjective and objective reasons for the current large problems in the quality of enterprise market data, which must be solved when enterprises integrate data. For solutions to problems caused by data quality, professionals are required to locate, trace, analyze, contact the heads of various systems, research solutions, and finally implement them. The existing solutions have high requirements for personnel, long time to solve problems, high recurrence rate of problems, and difficulty in statistical problem details, which greatly reduces customer satisfaction. Summary of the invention
[0004] In view of the above problems, the present invention is proposed to provide a data verification method, storage medium and electronic device that overcome the above problems or at least partially solve the above problems.
[0005] To achieve the above object, the present invention adopts the following technical solutions:
[0006] In a first aspect of the present application, a data verification method is provided, the method comprising:
[0007] Create a verification object configuration table and an element configuration table;
[0008] Configure the table name that needs to be verified to the verification object configuration table;
[0009] Configure the fields that need to be verified to the element configuration table and fill in the configuration items;
[0010] Configure the scheduled task and call the verification program regularly. The verification program verifies the objects in the verification object configuration table according to the verification rules corresponding to the verification type in the element configuration table.
[0011] After verification, the verification record statistics table and verification log table are output.
[0012] Optionally, the fields of the verification object configuration table include a table name, a table primary key, and an association condition.
[0013] Optionally, the fields of the element configuration table include object name, verification type, verification field, unique value field, code value description field, year-on-year field, quarter-on-quarter field, end-of-last-year field, end-of-last-month field, end-of-last-quarter field, calculation rule field, comparison rule field and comparison symbol.
[0014] Optionally, the verification record statistics table is used to count data quality and summarize information, and the fields of the verification record statistics table include verification date, object name, number of verification rules, number of verification fields, total number of verification records and number of abnormal records.
[0015] Optionally, the verification log table is used to accurately locate data problems, and the fields of the verification log table include verification date, object name, verification type, verification field and primary key id.
[0016] Optionally, the verification program verifies the objects in the verification object configuration table according to the verification rules corresponding to the verification types in the element configuration table, including:
[0017] If the check type is non-empty check, the amount of data with empty input field name is calculated. If the amount of data is 0, the check passes. If the amount of data is greater than 0, the check fails. The check record statistics table and check log table of non-empty check are generated respectively.
[0018] If the verification type is a comparison verification, the customer insurance information table in the verification object configuration table is queried. The query condition uses the verification field in the element configuration table to compare with the comparison rule field. The operator uses the comparison symbol field in the element configuration table. If the data volume is 0, the verification passes. If the data volume is greater than 0, it means that the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
[0019] Optionally, after the verification, the verification record statistics table and the verification log table are output and sent to the person in charge by email.
[0020] In a second aspect of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method described in any one of the first aspects is adopted.
[0021] In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor loads and executes the computer program, the method described in any one of the first aspects is adopted.
[0022] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0023] 1. The data verification method of the present invention eliminates the steps of positioning, tracing, analyzing, and contacting the persons in charge of each system, thereby solving the problems of high requirements for personnel, long problem solving time, and high recurrence rate of problems.
[0024] 2. The results generated by the data verification method of the present invention can show the specific problem data, data sources and related statistical information. It is not necessary to be very proficient in the business system to analyze the cause of the problem based on the generated statistical information; it is convenient to finally provide a solution and implement it. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 A schematic diagram of a data verification method provided in an embodiment of the present application;
[0026] Figure 2 The verification object configuration table data model provided in the embodiment of the present invention;
[0027] Figure 3 The element configuration table data model provided by the embodiment of the present invention;
[0028] Figure 4 A data model of a verification record statistics table provided in an embodiment of the present invention;
[0029] Figure 5 This is the verification log table data model provided by the embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0031] See also Figure 1 , this embodiment provides a data verification method, including:
[0032] S1. Create a verification object configuration table and an element configuration table.
[0033] The fields of the verification object configuration table include table name, table primary key and associated conditions; specifically, the fields of the verification object configuration table in this embodiment include table A name, table A primary key, table B name, table B primary key, associated conditions, for details, please refer to Figure 2 ,Depend on Figure 2 It can be seen that the finest granularity of the verification object configuration table is the data table name. The detailed design of the verification object configuration table can be seen from the data model in Table 1.
[0034] The fields of the element configuration table include object name, check type, check field, unique value field, code value description field, year-on-year field, quarter-on-quarter field, end of last year field, end of last month field, end of last quarter field, calculation rule field, comparison rule field and comparison symbol. For details, please refer to Figure 3 ,Depend on Figure 3 It can be seen that the finest granularity of the element configuration table is the check field. The detailed design of the element configuration table can be seen from the data model in Table 1.
[0035]
[0036]
[0037] Table 1 Model design details of the configuration table
[0038] Regarding the code value table, a complete system has a code value table which is indispensable. This embodiment does not create one and directly uses the code value table of the verified system. Please refer to Table 1 and Table 2 for the code value table data model.
[0039]
[0040] Table 2 A code value table data model
[0041] S2. Configure the table name that needs to be verified to the verification object configuration table.
[0042] Configure the objects that need to be verified into the verification object configuration table. In this embodiment, the customer configures the verification in the insurance information table, as shown in Table 3.
[0043] A table name A table primary key B table name Primary key of table B Association conditions Customer Insurance Information Form Guaranteed person ID Customer Master Table Customer ID Guaranteed person ID = Customer ID
[0044] Table 3 Configuration information table of the verification object configuration table
[0045] S3. Configure the fields that need to be verified to the element configuration table and fill in the configuration items.
[0046] Configure the field names and rule types that need to be verified into the element configuration table. This implementation class uses the fields in the customer's insurance information table for verification, as shown in Table 4.
[0047]
[0048] Table 4 Configuration information table of element configuration table
[0049] S4. Configure the scheduled task and call the verification program regularly.
[0050] The verification program verifies the objects in the verification object configuration table according to the verification rules corresponding to the verification types in the element configuration table.
[0051] The specific rules are as follows:
[0052] Non-empty check: Determine the check type in the element configuration table. If it is a non-empty check, calculate the amount of data when the input field name is empty. If the amount of data is 0, the check passes. If the amount of data is greater than 0, the check fails. Generate a check record statistics table and a check log table for the non-empty check respectively. If there is an empty value in the province and city of registration, the guarantor's ID will be written into the check log table, and the number of empty values will be written into the check record statistics table.
[0053] Association check: Determine the check type in the element configuration table. If the check type is an association check, use the customer insurance information table in the check object configuration table to associate with the customer master table. The association condition uses the association condition in the check object configuration table. First, query the data volume count_1 of the customer insurance information table, and then query the data volume count_2 after the customer insurance information table is associated with the customer master table. If count_1 is equal to count_2, the check passes. If count_1 is not equal to count_2, it proves that a Cartesian product appears after the association, and the statistical information is written into the verification record statistics table and the verification log table respectively.
[0054] Row record verification: Determine the verification type in the element configuration table. If the verification type is row record verification, query the customer insurance information table in the verification object configuration table for the current day's data volume and compare it with the previous day's data volume. If the current day's data volume is greater than or equal to the previous day's data volume, the verification passes. If the current day's data volume is less than the previous day's data volume, the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
[0055] Uniqueness check: Determine the check type in the element configuration table. If the check type is uniqueness check, there can only be one related information under the same unique information. Here, take customer information as an example: the same social credit number can only have one national standard industry; use the unique value field in the element configuration table plus the field to be checked for aggregation and grouping, and use the second-level aggregation of the data after aggregation and grouping. The second aggregation uses the unique value field in the element configuration table for aggregation. If the same unique value of the data after the second aggregation is greater than or equal to 2, it means that the verification fails. Write the statistical information into the verification record statistics table and the verification log table respectively. If the same unique value of the data after the second aggregation is equal to 1, the verification passes.
[0056] Year-on-year verification: Determine the verification type in the element configuration table. If the verification type is year-on-year verification, use the verification fields in the customer insurance information table in the verification object configuration table and the element configuration table to compare the customer insurance information table in the verification object configuration table and the element configuration table in the same period of the previous year. The association condition uses the primary key of the customer insurance information table in the verification object configuration table to associate with the primary key of the customer insurance information table in the same period. If the comparison data volume is 0, the verification passes. If the comparison data volume is greater than 0, it means that the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
[0057] Month-on-month verification: determine the verification type in the element configuration table. If the verification type is month-on-month verification, use the verification fields in the customer insurance information table in the verification object configuration table and the element configuration table to compare the month-on-month fields in the customer insurance information table in the verification object configuration table and the element configuration table for the same period last month. The association condition uses the primary key of the customer insurance information table in the verification object configuration table to associate with the primary key of the customer insurance information table for the same period last month. If the comparison data volume is 0, the verification passes. If the comparison data volume is greater than 0, it means that the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
[0058] Quarterly comparison verification: Determine the verification type in the element configuration table. If the verification type is quarterly comparison verification, use the verification fields in the customer insurance information table in the verification object configuration table and the element configuration table to compare the quarterly comparison fields in the customer insurance information table in the verification object configuration table and the element configuration table for the same period of the previous quarter. The association condition uses the primary key of the customer insurance information table in the verification object configuration table to associate with the primary key of the customer insurance information table for the same period of the previous quarter. If the comparison data volume is 0, the verification passes. If the comparison data volume is greater than 0, it means that the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
[0059] Verification at the end of the previous year: determine the verification type in the element configuration table. If the verification type is verification at the end of the previous year, use the verification fields in the customer insurance information table in the verification object configuration table and the element configuration table to compare the customer insurance information table in the verification object configuration table at the end of the previous year with the fields at the end of the previous year in the element configuration table. The association condition uses the primary key of the customer insurance information table in the verification object configuration table to associate with the primary key of the customer insurance information table at the end of the previous year. If the amount of compared data is 0, the verification passes. If the amount of compared data is greater than 0, it means that the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
[0060] Verification at the end of last month: determine the verification type in the element configuration table. If the verification type is verification at the end of last month, use the verification fields in the customer insurance information table in the verification object configuration table and the element configuration table to compare the customer insurance information table in the verification object configuration table at the end of last month with the fields at the end of last month in the element configuration table. The association condition uses the primary key of the customer insurance information table in the verification object configuration table to associate with the primary key of the customer insurance information table at the end of last month. If the comparison data volume is 0, the verification passes. If the comparison data volume is greater than 0, it means that the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
[0061] Verification at the end of the previous quarter: determine the verification type in the element configuration table. If the verification type is verification at the end of the previous quarter, use the verification fields in the customer insurance information table in the verification object configuration table and the element configuration table to compare the customer insurance information table in the verification object configuration table at the end of the previous quarter with the fields at the end of the previous quarter in the element configuration table. The association condition uses the primary key of the customer insurance information table in the verification object configuration table to associate with the primary key of the customer insurance information table at the end of the previous quarter. If the amount of compared data is 0, the verification passes. If the amount of compared data is greater than 0, it means that the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
[0062] Code value check: Determine the check type in the element configuration table. If the check type is code value check, use the check field and code value meaning field in the customer's insurance information table and element configuration table in the check object configuration table to associate the code value code field and code value meaning field in the code value table respectively, and the condition is that the associated field of the code value table is empty. If the data volume is 0, the check passes. If the data volume is greater than 0, it means that the check fails, and the statistical information is written into the check record statistics table and the check log table respectively.
[0063] Calculation rule verification: determine the verification type in the element configuration table. If the verification type is calculation verification, query the customer insurance information table in the verification object configuration table. The query condition uses the verification field in the element configuration table not equal to the calculation rule field. If the data volume is 0, the verification passes. If the data volume is greater than 0, it means that the verification fails. Write the statistical information into the verification record statistics table and the verification log table respectively.
[0064] Comparison rule verification: determine the verification type in the element configuration table. If the verification type is a comparison verification, query the customer insurance information table in the verification object configuration table. The query condition uses the verification field in the element configuration table to compare with the comparison rule field. The operator uses the comparison symbol field in the element configuration table. If the data volume is 0, the verification passes. If the data volume is greater than 0, it means that the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
[0065] S5. After verification, the verification record statistics table and the verification log table are output.
[0066] The verification record statistics table is used to count the quality of data and summarize information. The fields of the verification record statistics table include verification date, object name, number of verification rules, number of verification fields, total number of verification records and number of abnormal records. Figure 4 ,Depend on Figure 4 It can be seen that the verification record statistics table is based on the object dimension; the detailed design of the verification record statistics table can be seen from the data model in Table 1.
[0067] The verification log table is used to accurately locate data problems. The fields of the verification log table include verification date, object name, verification type, verification field and primary key id. For details, please refer to Figure 5 ,Depend on Figure 5 It can be seen that the verification log table can locate a specific piece of data; the detailed design of the verification log table can be seen from the data model in Table 1.
[0068] After verification, the verification record statistics table and the verification log table are output and sent to the person in charge by email.
[0069] The data verification method of this embodiment eliminates the steps of positioning, tracing, analyzing, and contacting the persons in charge of each system, thereby solving the problems of high requirements for personnel, long problem solving time, and high problem recurrence rate.
[0070] The results generated by the data verification method of this embodiment show the specific problem data, data sources and related statistical information. It is not necessary to be very proficient in the business system to analyze the cause of the problem based on the generated statistical information; it is convenient to finally provide a solution and implement it.
[0071] In addition, it should be noted that: an embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method described in the above embodiment is executed.
[0072] In addition, it should be noted that an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor loads and executes the computer program, the method described in the above embodiment is executed.
[0073] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A data verification method, characterized in that: The method comprises: Create a verification object configuration table and an element configuration table; Configure the table name that needs to be verified to the verification object configuration table; Configure the fields that need to be verified to the element configuration table and fill in the configuration items; Configure the scheduled task and call the verification program regularly. The verification program verifies the objects in the verification object configuration table according to the verification rules corresponding to the verification type in the element configuration table. After verification, the verification record statistics table and verification log table are output.
2. A data verification method according to claim 1, characterized in that: The fields of the verification object configuration table include table name, table primary key and association condition.
3. A data verification method as claimed in claim 1, characterized in that: The fields of the element configuration table include object name, check type, check field, unique value field, code value description field, year-on-year field, quarter-on-quarter field, end-of-last-year field, end-of-last-month field, end-of-last-quarter field, calculation rule field, comparison rule field and comparison symbol.
4. A data verification method as claimed in claim 1, characterized in that: The verification record statistics table is used to count the data quality and summarize information. The fields of the verification record statistics table include verification date, object name, number of verification rules, number of verification fields, total number of verification records and number of abnormal records.
5. A data verification method as claimed in claim 1, characterized in that: The verification log table is used to accurately locate data problems. The fields of the verification log table include verification date, object name, verification type, verification field and primary key id.
6. A data verification method according to claim 1, characterized in that: The verification program verifies the objects in the verification object configuration table according to the verification rules corresponding to the verification types in the element configuration table, including: If the check type is non-empty check, the amount of data with empty input field name is calculated. If the amount of data is 0, the check passes. If the amount of data is greater than 0, the check fails. The check record statistics table and check log table of non-empty check are generated respectively. If the verification type is a comparison verification, the customer insurance information table in the verification object configuration table is queried. The query condition uses the verification field in the element configuration table to compare with the comparison rule field. The operator uses the comparison symbol field in the element configuration table. If the data volume is 0, the verification passes. If the data volume is greater than 0, it means that the verification fails. The statistical information is written into the verification record statistics table and the verification log table respectively.
7. A data verification method according to claim 1, characterized in that: After verification, the verification record statistics table and the verification log table are output and sent to the person in charge by email.
8. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is loaded and executed by a processor, the method according to any one of claims 1 to 7 is adopted.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor loads and executes the computer program, the method according to any one of claims 1 to 7 is adopted.
