Data quality verification method and data quality verification equipment

Abstract

The present application provides a data quality verification method and data quality verification device. The method includes: verifying the first data, storing the first data locally if the verification is successful, and not storing the first data locally if the verification fails; verifying the second data, updating the second data that fails the verification, until all the second data pass the verification. The data verification rules are solidified at the data entry to block incremental problem data, that is, after the second device sends the data, the first data is verified before the data is stored in the database, and the data that fails the verification is not stored in the database. This avoids the problem of poor data quality caused by subsequent data being stored in the database. At the same time, inside the first device, the problematic data is continuously searched according to the business, that is, the data is also continuously verified locally until the data meets the verification rules, forming a closed-loop data management, thereby improving data quality.

Description

Technical Field

[0001] The present application relates to the technical field of data quality assessment, and in particular to a data quality verification method, a data quality verification device, a computer-readable storage medium, and a data quality verification system. Background Art

[0002] The digitalization of the power industry has placed new demands on data quality in order to conduct effective data mining and multi-dimensional business analysis. In the digital age, data, as a production factor, has become a new driving force for economic and social development and transformation. Data quality has become a crucial factor influencing the value of this element. In practical work and digitalization efforts, it has been found that existing data quality is poor and can no longer meet the increasingly complex requirements of data analysis. Summary of the Invention

[0003] The main purpose of this application is to provide a data quality verification method, data quality verification device, computer-readable storage medium and data quality verification system, so as to at least solve the problem that the data quality in the prior art is poor and it is difficult to meet the increasingly complex data analysis requirements.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data quality verification method is provided, including: a first device obtains a data verification rule, wherein the data verification rule is a predefined rule for verifying the quality of data; when the first device obtains a first business requirement, the first device verifies the first data corresponding to the first business requirement according to the data verification rule to obtain a first verification result, the first business requirement is the requirement of the second device to request the first device to process the business, the first business requirement and the first data have a one-to-one correspondence, the first data is the data sent by the second device, the first device is a device for storing the data according to the business requirement, and the second device is a device for reporting the data to the first device; the first device verifies the first data according to the first verification rule in the first verification result When the verification result indicates that the verification is passed, the first data is stored locally; when the first verification result indicates that the verification is failed, the first data is not stored locally; when the first device obtains the second business demand, the second data corresponding to the second business demand is verified according to the data verification rule to obtain a second verification result, wherein the second business demand is the demand of the first device to process local business, the second business demand and the second data have a one-to-one correspondence, and the second data is the data stored locally by the first device; when the second verification result indicates that the verification is failed, the first device updates the second data that fails the verification, and verifies the updated second data again until all the second data pass the verification.

[0005] Optionally, the first device obtains a data verification rule, including: the first device obtains a first data verification rule, wherein the first data verification rule is a rule for verifying the type, length, and enumeration value of all fields in the same data table, and verifying the value range of the second field limited by the data element of the first field, and the first field and the second field are both fields in the same data table; the first device obtains a second data verification rule, wherein the second data verification rule is a rule for verifying the value range of the fourth field limited by the data element of the third field, the third field is a field in the first data table, and the fourth field is a field in the second data table; the first device obtains a third data verification rule, wherein the third data verification rule is a rule for logical verification between the first data record of the first data table and the second data record of the second data table; the first device obtains a fourth data verification rule, wherein the fourth data verification rule is a verification rule for whether there are duplicate data records in any data table.

[0006] Optionally, after the first device obtains the data verification rule, the method further includes: the first device generates a record template at least according to the content and number of iterations in the data verification rule, wherein the record template represents a standardized verification template for data verification; the first device determines the version of the data verification rule based on business requirements, the data verification rule and the record template.

[0007] Optionally, there are multiple data verification rules, and the first device determines the version of the data verification rules based on the business requirements, the data verification rules and the record template, including: when the first business requirement exists, the first device extracts at least part of the data verification rules from the multiple data verification rules according to the first business requirement to obtain a first target data verification rule; the first device adds the first target data verification rule to the record template, generates an incremental verification rule, and tests the incremental verification rule to obtain a first test result; when the first test result indicates that the test fails, the first device updates the incremental verification rule and tests the incremental verification rule again until the first test result indicates that the test passes, and determines that the version of the incremental verification rule that passes the test is the latest version.

[0008] Optionally, there are multiple data verification rules, and the first device determines the version of the data verification rules based on the business requirements, the data verification rules and the record template, including: when there is the second business requirement, the first device extracts at least part of the data verification rules from the multiple data verification rules according to the second business requirement to obtain a second target data verification rule; the first device adds the second target data verification rule to the record template to generate an existing verification rule, and tests the existing verification rule to obtain a second test result; when the second test result indicates that the test fails, the first device updates the existing verification rule and tests the existing verification rule again until the second test result indicates that the test passes, and determines that the version of the existing verification rule that passes the test is the latest version.

[0009] Optionally, after the first device obtains the data verification rules, the method also includes: when the first device obtains the third business requirement, the first device verifies the third data corresponding to the third business requirement according to the data verification rules to obtain a third verification result, wherein the third business requirement is the requirement for a new business in the first device, the third business requirement and the third data have a one-to-one correspondence, and the third data is the data newly added when the new business requirement is added; when the third verification result indicates that the verification has been passed, the first device verifies the third data again according to the data verification rules before storing the third data locally to obtain a fourth verification result; when the fourth verification result indicates that the verification has been passed, the first device stores the third data locally; when the fourth verification result indicates that the verification has not passed, the first device generates a prompt message; when the first verification result indicates that the verification has not passed, the first device generates a first problem data list based on the first data that has not passed the verification, and sends it to the second device.

[0010] Optionally, after verifying the second data corresponding to the second business requirement according to the data verification rule and obtaining a second verification result, the method further includes: when the second verification result indicates that the verification has failed, the first device generates a second problem data list based on the second data that has failed the verification; the first device determines whether the data verification rule is the latest version, and when the data verification rule is the latest version, determines that the data verification rule is abnormal.

[0011] According to another aspect of the present application, a data quality verification device is provided, comprising: an acquisition unit for acquiring data verification rules, wherein the data verification rules are predefined rules for data quality verification; a first verification unit for, upon obtaining a first business requirement, verifying first data corresponding to the first business requirement according to the data verification rules to obtain a first verification result, wherein the first business requirement is a requirement for a second device to request a first device to process a business, the first business requirement and the first data have a one-to-one correspondence, and the first data is data sent by the second device; a first processing unit for storing the first data locally if the first verification result indicates that the verification has passed, and processing the data locally. The first verification result indicates that the first data will not be stored locally if the verification fails; the second verification unit is used to verify the second data corresponding to the second business requirement according to the data verification rule when a second business requirement is obtained, to obtain a second verification result, wherein the second business requirement is the requirement of the first device to process local business, the second business requirement and the second data have a one-to-one correspondence, and the second data is the data stored locally by the first device; the second processing unit is used to update the second data that fails the verification if the second verification result indicates that the verification fails, and to verify the updated second data again until all the second data pass the verification.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the data quality verification methods.

[0013] According to another aspect of the present application, a data quality verification system is provided, including: a first device and a second device, the first device is a device for storing data according to business needs, the second device is a device for reporting the data to the first device, and the first device is used to execute any one of the data quality verification methods.

[0014] By applying the technical solution of the present application, the data verification rules are solidified at the data entry to block incremental problem data. That is, after the second device sends the data, the first data is verified before the data is stored in the warehouse. If the verification fails, the data will not be stored in the warehouse. This avoids the problem of poor data quality after the subsequent data is stored in the warehouse. At the same time, inside the first device, the problem data is continuously searched according to the business, that is, the data must also be continuously verified locally until the data meets the verification rules, forming a closed-loop data management, thereby improving the data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for performing a data quality verification method provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram of a process for verifying data quality according to an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram showing the classification of data verification rules is shown;

[0019] Figure 4 A schematic diagram showing the process of sorting out, establishing, and solidifying application rules;

[0020] Figure 5 shows a flow chart of the data verification process;

[0021] Figure 6 A flowchart showing version management of data validation rules is shown;

[0022] Figure 7 A flowchart illustrating version management of incremental validation rules is shown;

[0023] Figure 8 A flowchart showing version management of inventory verification rules is shown;

[0024] Figure 9 A schematic diagram of the process of solidifying the verification rules for blocking increments is shown;

[0025] Figure 10 A schematic diagram of the process of applying the verification rule for blocking increment is shown;

[0026] Figure 11 A schematic diagram of the process of solidifying the verification rules for checking inventory is shown;

[0027] Figure 12 A schematic diagram showing the flow of applying verification rules for checking inventory;

[0028] Figure 13 A structural block diagram of a data quality verification device provided according to an embodiment of the present application is shown.

[0029] The above drawings include the following reference numerals:

[0030] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0031] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] In actual work and digital construction, it is found that the quality of existing data is poor and can no longer meet the increasingly complex data analysis requirements. Therefore, it is urgent to create a long-term mechanism to study methods to improve data quality and effectively improve the quality of business analysis data.

[0035] As introduced in the background technology, the quality of data in the existing technology is poor and it is difficult to meet the increasingly complex data analysis requirements. In order to solve the above problems, the embodiments of the present application provide a data quality verification method, a data quality verification device, a computer-readable storage medium and a data quality verification system.

[0036] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0037] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG is a hardware structure block diagram of a mobile terminal for a data quality verification method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0038] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the device information display method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] In this embodiment, a method for verifying data quality running on a mobile terminal, a computer terminal, or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 2 This is a flow chart of a data quality verification method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0041] Step S201: The first device obtains a data verification rule, wherein the data verification rule is a predefined rule for verifying the quality of data;

[0042] Specifically, different data verification rules can be formulated according to different businesses, and the data verification rules can be classified later.

[0043] In step S202, upon obtaining the first service requirement, the first device verifies the first data corresponding to the first service requirement according to the data verification rule to obtain a first verification result. The first service requirement is a requirement for the second device to request the first device to process a service. The first service requirement corresponds to the first data in a one-to-one manner. The first data is data sent by the second device. The first device stores the data according to the service requirement, and the second device reports the data to the first device.

[0044] Specifically, when there are new business needs, the new data can be verified before being stored in the database.

[0045] Step S203: the first device stores the first data locally if the first verification result indicates that the verification has passed, and does not store the first data locally if the first verification result indicates that the verification has failed.

[0046] Specifically, no matter whether the data is reported through the B / S end, C / S end, third-party interface, etc., the incoming data must be verified according to the verification rules, and the data quality of the reported incremental data can be ensured by refusing to save the data with known problems into the database.

[0047] Step S204: Upon obtaining the second service requirement, the first device verifies the second data corresponding to the second service requirement according to the data verification rule to obtain a second verification result, wherein the second service requirement is a requirement for the first device to process a local service, the second service requirement corresponds to the second data in a one-to-one manner, and the second data is data locally stored by the first device.

[0048] Specifically, when there is a business with local storage, the local data can be verified regularly.

[0049] In step S205 , when the second verification result indicates that the second data fails verification, the first device updates the second data that fails verification, and verifies the updated second data again until all the second data passes verification.

[0050] Specifically, a data quality problem verification query function can be developed in the information system based on the verification rules. By performing rolling verification on the data in the database, the existing problem data can be effectively located, helping to effectively close the rectification loop of related problem data.

[0051] Through this embodiment, the data verification rules are solidified at the data entry to block incremental problem data, that is, after the second device sends the data, the first data is verified before the data is stored in the warehouse. If the verification fails, the data will not be stored in the warehouse. This avoids the problem of poor data quality caused by subsequent data being stored in the warehouse. At the same time, inside the first device, the problem data is continuously searched according to the business, that is, the data is also continuously verified locally until the data meets the verification rules, forming a closed-loop data management, thereby improving the data quality.

[0052] Currently, enterprises generally manage the data quality of their business data (i.e., data) in a relatively extensive manner, often only imposing certain restrictions on data type, length, and enumeration values ​​during data reporting. When enterprises encounter data quality issues while conducting analysis using business data, they conduct ad hoc analyses based on business needs to address identified data quality issues and implement temporary data governance only for the existing data required for analysis. There is no systematic approach to refining and solidifying data quality validation rules to address these issues. As incremental business data enters the database, lacking a robust data quality management mechanism, data quality issues can become chaotic, impacting subsequent data governance efforts. This solution establishes a systematic and scientific data quality management system. Based on a rational classification of validation rules and effective record templates, data quality validation rules are summarized and organized according to business needs. These validation rules are then solidified at the data entry point to prevent incremental problematic data. Solidified within the system, they enable business functions to continuously identify and rectify existing problematic data, effectively improving data quality through continuous closed-loop management and control. Furthermore, through version management, validation rules will be continuously updated to ensure they align with the latest business analysis needs.

[0053] There are many data validation rules, which can be classified. In the specific implementation process, such as Figure 3As shown, the first device obtains the data verification rule, which can be implemented by the following steps: the above-mentioned first device obtains the first data verification rule (intra-table validity verification), wherein the above-mentioned first data verification rule is a rule for verifying the type, length, and enumeration value of all fields in the same data table, and verifying the value range of the second field limited by the data element of the first field, and the above-mentioned first field and the above-mentioned second field are both fields in the same data table; the above-mentioned first device obtains the second data verification rule (inter-table validity verification), wherein the above-mentioned second data verification rule is a rule for verifying the value range of the fourth field limited by the data element of the third field, the above-mentioned third field is a field in the first data table, and the above-mentioned fourth field is a field in the second data table; the above-mentioned first device obtains the third data verification rule (association verification), wherein the above-mentioned third data verification rule is a rule for logical verification between the first data record of the first data table and the second data record of the second data table; the above-mentioned first device obtains the fourth data verification rule (repetition verification), wherein the above-mentioned fourth data verification rule is a verification rule for whether there are duplicate data records in any data table.

[0054] In this solution, data verification rules are divided into four types: intra-table validity verification of data values, inter-table validity verification of data values, data record association verification, and data record duplication verification. Intra-table validity verification and inter-table validity verification are data quality verifications for field values, while association verification and duplication verification are data quality verifications for data records that have been stored. In this way, the corresponding verification rules can be extracted to verify the data according to changes in business needs, thereby ensuring that the efficiency of data verification can be further improved.

[0055] Specifically, the in-table validity check of the data value is to ensure that there is no invalid data in the data table by timely checking the data after the user fills in the data. The in-table validity check of the data value includes: the valid value range of the data element limited by the business attributes of the data element itself or the valid value range of the data element limited by the values ​​of other data elements in the table. For example: 1. The type, length, and enumeration value settings of the field; 2. The valid value range of the data element limited by the business attributes of the data element itself, such as: the length of the medium-voltage line is 0-200km; 3. The valid value range of the data element limited by the values ​​of other data elements in the table, such as: the reason for the power outage is the quality of the distribution network equipment, and the power outage equipment can only select the distribution network equipment subcategory.

[0056] Specifically, the inter-table validity check of data values ​​is carried out by cross-constraining data across tables at the same time or cross-constraining data across tables at different times. The page or client will verify the data in a timely manner after the user completes the page. If the data does not meet the requirements, the problem with the filling will be directly reported using a pop-up window to ensure that there is no invalid data in the data table. Inter-table validity check of data values: The valid value range of this data element is limited by the values ​​of other data elements between tables. For example: the equipment outage period is a field in the equipment outage event table, the equipment commissioning status is a field in the equipment parameter table, and the equipment outage period must be within the valid period when the equipment is in commissioning status.

[0057] Specifically, data record relevance verification is achieved by verifying the matching relationship between the data element values ​​in the same data element items between tables to ensure that there are no unrelated data records in the data tables. For example, if Data Table 1 records Xiao Ming's age as 20, and Data Table 2 records Xiao Ming as elderly, there is a logical error and the corresponding association cannot be made.

[0058] Specifically, data record duplication verification is performed by checking the duplication of data records in the table to ensure that there are no redundant duplicate data records in the data table, which would affect the indicator calculation results. For example, if a data table records 10 devices A, and for some reason multiple records of this data are stored, the indicator calculation results will be affected.

[0059] In order to sort out and summarize data verification rules, and effectively solidify and apply data verification rules to find problematic data, and make rectifications and improvements, after the first device obtains the data verification rules, the above method also includes the following steps: the above first device generates a record template at least according to the content and number of iterations in the above data verification rules, wherein the above record template represents a standardized verification template for data verification; the above first device determines the version of the above data verification rules based on business needs, the above data verification rules and the above record template.

[0060] In this solution, in order to facilitate the record management of data verification rules, a verification rule record tracking matrix is ​​formulated to determine the content and iteration batches of the verification rule records and the changes in the activation and deactivation of the rules, which is conducive to the standardized management of the verification rules. A record template can be generated. According to the reasonable classification of data verification rules and effective record templates, the data quality verification rules can be summarized and concluded according to business needs, and the verification rules can be rolled out according to the data quality verification rule version management process based on subsequent business needs. This ensures that the data verification rules are relatively accurate and further ensures that this solution can effectively improve data quality.

[0061] The initialized record template defines information such as serial number, field name, data type, business meaning, data unit, data length, value example, value restriction rules for its own business attributes, value restriction rules for other fields, iteration batch, proposer, purpose of proposal, proposer contact information, activation time, deactivation time, etc., as shown in Table 1.

[0062] Table 1

[0063]

[0064] In actual application, two record templates are filled in, as shown in Table 2.

[0065] Table 2

[0066]

[0067]

[0068] Specifically, if Figure 4 、 Figure 5 、 Figure 6 As shown, the core of this application is to use a series of scientific methods to sort out and summarize data quality verification rules, and effectively solidify and apply the verification rules to find problematic data, and rectify and improve them. Through the data verification solution of this application, data quality problems encountered in the use of internal enterprise data can be solved, and the authenticity, accuracy and completeness of business data (i.e. data) can be improved, providing a solid and reliable data foundation for the company's precision marketing, management decision-making, risk control, etc., laying the foundation for the effective and in-depth development of the company's work.

[0069] This solution allows for the rational classification of validation rules and effective record templates, allowing for the summary and compilation of data quality validation rules based on business needs. This allows for version management and rolling iteration of validation rules based on established version management processes, as business needs evolve. By implementing and applying the solidified data quality validation rules, it is possible to verify the existing data and prevent incremental data from being added. Effective solidification of data quality validation rules is typically achieved through program code.

[0070] You can set the verification rule management field information for the responsible persons and completion time of the three different stages of design and development, testing, and online / update in the item number, data scanning verification rules, B / S-end verification rules, C / S-end verification rules, and interface verification rules to achieve full-process management of data quality verification rules, effectively clarify time nodes, locate responsible persons, and promote the process-based and standardized management of the entire life cycle of data quality verification rules. The version management table of data verification rules is shown in Table 3.

[0071] Table 3

[0072]

[0073] As the business continues to develop, when new business needs arise, the data quality verification rules can be summarized and concluded according to the reasonable classification of data verification rules and effective record templates based on the newly added business needs, and recorded in the system to form query, maintenance and other functions. In the specific implementation process, there are multiple data verification rules. The above-mentioned first device determines the version of the above-mentioned data verification rules based on the business needs, the above-mentioned data verification rules and the above-mentioned record templates. Specifically, it can be implemented through the following steps: when there is the above-mentioned first business need, the above-mentioned first device extracts at least part of the above-mentioned data verification rules from the multiple above-mentioned data verification rules according to the above-mentioned first business need to obtain a first target data verification rule; the above-mentioned first device adds the above-mentioned first target data verification rule to the above-mentioned record template, generates an incremental verification rule, and tests the above-mentioned incremental verification rule to obtain a first test result; when the above-mentioned first test result indicates that the test fails, the above-mentioned first device updates the above-mentioned incremental verification rule and tests the above-mentioned incremental verification rule again until the above-mentioned first test result indicates that the test passes, and determines that the version of the above-mentioned incremental verification rule that passes the test is the latest version.

[0074] In this solution, effectiveness testing can be carried out on the developed verification rule functions. If the verification function does not meet the requirements, it will be modified and improved until the verification rule function meets the requirements. After all verification functions have been tested and improved, the verification rule functions that have completed the test will be updated and released. This ensures that the management efficiency of data verification rules of this solution is high when blocking increments, and ensures that the incremental verification rules obtained by this solution are relatively accurate, thereby ensuring that the incremental verification rules can be used to accurately and efficiently verify the data in the future.

[0075] Specifically, if Figure 4 、 Figure 5 As shown, when the fixed verification rules are used to block incremental data, the data is reported through a third-party interface (the interface for communication with the first device), etc., and the incoming data is verified according to the verification rules. By refusing to save data with known problems into the database, the data quality of the reported incremental data is ensured.

[0076] like Figure 7As shown, in the solution of the present application, by sorting out and summarizing the data quality verification rules, the newly added B / S-side verification rules, C / S-side verification rules, interface verification rules and data quality scanning verification rules are functionally designed, and the corresponding data quality verification rule functions are implemented through code. It should be noted in this link that the development task of the data quality scanning verification rule function must be the first priority to meet the needs of scanning and verifying existing data; the development tasks of the B / S-side verification rule function, the C / S-side verification rule function, and the interface verification rule function are all the second priority to meet the needs of incremental data verification. In order to ensure the effectiveness of the newly added data quality scanning verification rule function, the B / S-side verification rule function, the C / S-side verification rule function, and the interface verification rule function, effectiveness testing can be carried out on the developed verification rule function. If the verification function does not meet the requirements, it will be modified and improved until the verification rule function meets the requirements.

[0077] In actual applications, data quality verification rules can be summarized and concluded according to the reasonable classification and effective record template of data verification rules and existing business needs, and recorded in the system to form query, maintenance and other functions. In the specific implementation process, there are multiple data verification rules. The above-mentioned first device determines the version of the above-mentioned data verification rules based on business needs, the above-mentioned data verification rules and the above-mentioned record template. It can be specifically implemented through the following steps: when there is the above-mentioned second business need, the above-mentioned first device extracts at least part of the above-mentioned data verification rules from the multiple above-mentioned data verification rules according to the above-mentioned second business need to obtain a second target data verification rule; the above-mentioned first device adds the above-mentioned second target data verification rule to the above-mentioned record template to generate an existing verification rule, and tests the above-mentioned existing verification rule to obtain a second test result; when the above-mentioned second test result indicates that the test fails, the above-mentioned first device updates the above-mentioned existing verification rule and tests the above-mentioned existing verification rule again until the above-mentioned second test result indicates that the test passes, and determines that the version of the above-mentioned existing verification rule that passes the test is the latest version.

[0078] In this solution, the effectiveness test can be carried out on the developed verification rule function. If the verification function does not meet the requirements, it will be modified and improved until the verification rule function meets the requirements. After all the inspection functions are tested and improved, the verification rule function that has completed the test will be updated and released. This ensures that the management efficiency of data verification rules in this solution is high when checking inventory, and ensures that the inventory verification rules obtained by this solution are relatively accurate, thereby ensuring that the inventory verification rules can be used to accurately and efficiently verify the data in the future.

[0079] Specifically, if Figure 4 、 Figure 5As shown, when checking the inventory by solidifying the verification rules, it is necessary to develop a data quality problem verification query function in the information system according to the verification rules. By performing rolling verification on the data in the database, the effective positioning of the inventory problem data can be achieved, and the effective rectification of the relevant problem data can be facilitated.

[0080] like Figure 8 As shown, in the scheme of the present application, by sorting out and summarizing the data quality verification rules, the B / S-side verification rules, C / S-side verification rules, interface verification rules and data quality scanning verification rules are functionally designed respectively, and the corresponding data quality verification rule functions are implemented through code. In order to ensure the effectiveness of the B / S-side verification rule functions, C / S-side verification rule functions, interface verification rule functions and data quality scanning verification rule functions, effectiveness testing can be carried out on the developed verification rule functions. If the verification function does not meet the requirements, it will be modified and improved until the verification rule function meets the requirements.

[0081] To ensure the data quality of information systems, we are promoting the development of a data quality verification rule module based on the improvement of data quality verification rules and aiming to support the diversified big data applications of cybersecurity technical supervision. This module will be used to carry out comprehensive management of verification rules. Specifically, we will be problem-oriented and continuously refine and optimize the cybersecurity technical supervision business informationization specifications and data quality verification rules. We will block incremental data and check existing data to ensure data validity, relevance, and uniqueness, and that it complies with actual scenarios. We will ensure that the data verification process forms a closed-loop verification loop from data entry to data storage. Solidified application rules are divided into two categories: solidified verification rules for blocking incremental data and solidified verification rules for checking existing data.

[0082] In some embodiments, after the first device obtains the data verification rules, the above method also includes the following steps: when the above first device obtains the third business demand, the above first device verifies the third data corresponding to the above third business demand according to the above data verification rules to obtain a third verification result, wherein the above third business demand is the demand for a new business in the above first device, the above third business demand and the above third data have a one-to-one correspondence, and the above third data is the data newly added when the new business demand is added; when the above third verification result indicates that the verification has been passed, the above first device verifies the above third data again according to the above data verification rules before storing the above third data locally to obtain a fourth verification result; when the above fourth verification result indicates that the verification has been passed, the above first device stores the above third data locally; when the above fourth verification result indicates that the verification has not passed, the above first device generates a prompt message; when the above first verification result indicates that the verification has not passed, the above first device generates a first problem data list based on the above first data that has not passed the verification, and sends it to the above second device.

[0083] In this solution, data sent from a third-party second device is directly verified. If the verification fails, the data is rejected from storage. When new business needs arise in the first device, verification is required both during data entry and before data storage. If the verification fails, the data is rejected from storage, further ensuring the high quality of the data stored in the first device. If the verification result indicates that the verification fails, a list of problematic data can be generated. This list can then be used to determine the cause of the data quality problem, allowing for a timely resolution based on the cause of the anomaly.

[0084] Specifically, if Figure 4 、 Figure 5 As shown, when the fixed verification rules are used to block incremental data, the data is reported through B / S reporting, C / S reporting, etc. The incoming data is verified according to the verification rules, and the data quality of the reported incremental data is ensured by refusing to save the data with known problems into the database.

[0085] Specifically, validation rules for intra-table validity, inter-table validity, association, and duplication are implemented in the system through code, creating a managed entry. When data is submitted to the system, it is automatically verified according to the established validation rules. If the rules are met, the data is allowed into the database; if not, it is rejected and a notification is displayed indicating any issues with the data and corrective measures are recommended.

[0086] The B / S end and the C / S end (both are first-class devices, but with different architectures) block the incremental solidification process, such as Figure 9 As shown: During the B / S and C / S reporting process, the content filled in by the user is subject to in-table validity verification, and if it does not comply with the rules, a reminder will be given through pop-up windows, highlights, etc.; after the verification is passed during entry, when the user clicks the Save button, the system will automatically perform inter-table validity verification, correlation verification, and duplication verification on the data. If it does not comply with the verification rules, the data will not be allowed to be stored, and a reminder will be given through pop-up windows, highlights, etc.; data will be allowed to be stored only when it fully complies with the verification rules.

[0087] The third-party system interface data (data sent by the second device) is blocked during the incremental solidification process when it is transmitted, such as Figure 9 As shown: When a third-party system transmits data to this system, the interface data is checked for intra-table validity, inter-table validity, correlation, and duplication. If the data does not meet the verification rules, a corresponding list of problem data is generated and returned to the third-party system. Data is allowed to be stored only when it fully meets the verification rules.

[0088] The application steps of the incremental verification rule are divided into 6 steps, such as Figure 10 As shown, the details are as follows:

[0089] Step 1: Analyze data based on business needs and develop data quality verification rules for different categories, including intra-table validity verification, inter-table validity verification, correlation verification, and duplication verification;

[0090] Step 2: Create a data view query statement based on the established validation rules, and store the defined validation rules and data view name in the database;

[0091] Step 3: Implement data quality verification rule logic judgment through code technology development, complete business modules, verification rules, data visualization Figure 3 the relationship between

[0092] In step 4, the user enters data content through the B / S and C / S terminals and receives data from third-party systems. The system automatically executes the validation rule SQL statements, calculates each validation rule based on the data view, and returns a data value, where '0' indicates no violation of the rule and '1' indicates a violation of the rule.

[0093] Step 5: Concatenate the calculation results of multiple business rules for each piece of data; filter the concatenated result data to obtain abnormal data; parse each piece of data to extract data information with an intermediate result of 1; and output the extracted abnormal data information as a return result;

[0094] Step 6: If the data verification returns '0', the system will store the data.

[0095] In some embodiments, after verifying the second data corresponding to the second business requirement according to the data verification rule and obtaining the second verification result, the method further includes the following steps: when the second verification result indicates that the verification has failed, the first device generates a second problem data list based on the second data that has failed the verification; the first device determines whether the data verification rule is the latest version, and when the data verification rule is the latest version, determines that the data verification rule is abnormal.

[0096] In this solution, if the second verification result indicates that the verification fails, a second problem data list can be generated. In this way, the cause of the data quality problem can be determined based on the second problem data list. One of the reasons is to determine whether the data verification rules are abnormal. In this way, a solution can be determined in time based on the cause of the abnormality until the data passes the verification, thereby further ensuring that the quality of the data stored in the first device is high.

[0097] like Figure 5 、 Figure 11As shown, all data in the database is subject to intra-table validity checks, inter-table validity checks, correlation checks, and duplication checks, along with a full data scan to verify data rationality. Data that does not comply with the rules is compiled into a problem data list, which users can use to rectify and correct the data. At the same time, a comparative analysis is performed between the problem data list and the latest version of the data quality verification rules to identify the causes of data quality issues. Common causes and solutions for data quality issues are as follows:

[0098] Cause 1: The incremental check rule function has not been solidified;

[0099] Solution to reason 1: It is necessary to solidify the incremental verification rule function.

[0100] Reason 2: There is a bug in the incremental verification rule function;

[0101] Solution for Cause 1: The buggy blocking increment verification rule function needs to be corrected.

[0102] The application steps of inventory verification rules are divided into 7 steps, such as Figure 12 As shown, the details are as follows:

[0103] Step A: Analyze data based on business needs and formulate data quality verification rules for different categories, including intra-table validity verification, inter-table validity verification, correlation verification, and duplication verification;

[0104] Step B: Create a data view query statement based on the established validation rules, and store the defined validation rules and data view name in the database;

[0105] Step C, through code technology development to achieve data quality verification rule logic judgment, complete business module, verification rules, data view Figure 3 the relationship between

[0106] In step D, the user executes the existing data scan function, and the system automatically executes the validation rule SQL statement, calculates each validation rule based on the data view, and returns a data value, where '0' represents no violation of the rule and '1' represents a violation of the rule;

[0107] Step E: concatenate the calculation results of multiple business rules for each piece of data; filter the concatenated result data to obtain abnormal data; parse each piece of data to extract data information with an intermediate result of 1; and output the extracted abnormal data information as a return result;

[0108] Step F: If there is a data quality problem, the user updates the data according to the problem list and then repeats step 4 until the verification is normal.

[0109] In step G, if the data verification returns '0', then there is no quality problem with the existing data and the scanning process is completed.

[0110] In summary, this solution introduces a comprehensive data quality verification rule classification: It summarizes a scientific classification method for data quality verification rules and compiles a comprehensive and detailed record template. It establishes a data quality verification rule management system to implement full-process management of data quality verification rules, effectively clarify timelines, identify responsible individuals, and promote the full lifecycle management of data quality verification rules, helping to reduce data quality management costs. This introduces a systematic, scientific, and effective overall solution for comprehensive business system data quality management.

[0111] The present application also provides a data quality verification device. It should be noted that the data quality verification device of the present application embodiment can be used to perform the verification method for data quality provided by the present application embodiment. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the details that have been described will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0112] The following introduces the data quality verification device provided in the embodiment of the present application. It should be noted that the data quality verification device here is the first device.

[0113] Figure 13 This is a structural block diagram of a data quality verification device according to an embodiment of the present application. Figure 13 As shown, the data quality verification device includes:

[0114] An acquisition unit 10 is configured to acquire a data verification rule, wherein the data verification rule is a predefined rule for verifying the quality of the data;

[0115] A first verification unit 20 is configured to, upon obtaining a first service requirement, verify the first data corresponding to the first service requirement according to the data verification rule to obtain a first verification result, wherein the first service requirement is a requirement for the second device to request the first device to process a service, the first service requirement corresponds to the first data in a one-to-one manner, and the first data is data sent by the second device;

[0116] a first processing unit 30 configured to store the first data locally if the first verification result indicates that the verification has passed, and not store the first data locally if the first verification result indicates that the verification has failed;

[0117] A second verification unit 40 is configured to, upon obtaining a second service requirement, verify the second data corresponding to the second service requirement according to the data verification rule to obtain a second verification result, wherein the second service requirement is a requirement for the first device to process a local service, the second service requirement corresponds to the second data in a one-to-one manner, and the second data is data locally stored on the first device;

[0118] The second processing unit 50 is configured to update the second data that failed verification if the second verification result indicates that the second data failed verification, and verify the updated second data again until all the second data pass verification.

[0119] Through this embodiment, the data verification rules are solidified at the data entry to block incremental problem data, that is, after the second device sends the data, the first data is verified before the data is stored in the warehouse. If the verification fails, the data will not be stored in the warehouse. This avoids the problem of poor data quality caused by subsequent data being stored in the warehouse. At the same time, inside the first device, the problem data is continuously searched according to the business, that is, the data is also continuously verified locally until the data meets the verification rules, forming a closed-loop data management, thereby improving the data quality.

[0120] There are many data verification rules, which can be classified. In the specific implementation process, the acquisition unit includes a first acquisition module, a second acquisition module, a third acquisition module and a fourth acquisition module. The first acquisition module is used to obtain the first data verification rule (intra-table validity check), wherein the first data verification rule is a rule for checking the type, length, and enumeration value of all fields in the same data table, and checking the value range of the second field limited by the data element of the first field. The first field and the second field are both fields in the same data table; the second acquisition module is used to obtain the second data verification rule (inter-table validity check). In the embodiment, the second data verification rule is a rule for verifying the value range of the fourth field limited by the data elements of the third field, the third field is a field in the first data table, and the fourth field is a field in the second data table; the third acquisition module is used to obtain the third data verification rule (association verification), wherein the third data verification rule is a rule for logical verification between the first data record of the first data table and the second data record of the second data table; the fourth acquisition module is used to obtain the fourth data verification rule (duplication verification), wherein the fourth data verification rule is a verification rule for whether there are duplicate data records in any data table.

[0121] In this solution, data verification rules are divided into four types: intra-table validity verification of data values, inter-table validity verification of data values, data record association verification, and data record duplication verification. Intra-table validity verification and inter-table validity verification are data quality verifications for field values, while association verification and duplication verification are data quality verifications for data records that have been stored. In this way, the corresponding verification rules can be extracted to verify the data according to changes in business needs, thereby ensuring that the efficiency of data verification can be further improved.

[0122] In order to sort out and summarize data verification rules, and effectively solidify and apply data verification rules to find problematic data, rectify and improve them, the data quality verification device of the present application also includes a first generation unit and a first determination unit. The first generation unit is used to generate a record template after obtaining the data verification rules, at least according to the content and number of iterations in the above data verification rules, wherein the above record template represents a standardized verification template for data verification; the first determination unit is used to determine the version of the above data verification rules according to business needs, the above data verification rules and the above record template.

[0123] In this solution, in order to facilitate the record management of data verification rules, a verification rule record tracking matrix is ​​formulated to determine the content and iteration batches of the verification rule records and the changes in the activation and deactivation of the rules, which is conducive to the standardized management of the verification rules. A record template can be generated. According to the reasonable classification of data verification rules and effective record templates, the data quality verification rules can be summarized and concluded according to business needs, and the verification rules can be rolled out according to the data quality verification rule version management process based on subsequent business needs. This ensures that the data verification rules are relatively accurate and further ensures that this solution can effectively improve data quality.

[0124] As the business continues to develop, when new business needs arise, the data quality verification rules can be summarized and concluded according to the reasonable classification of data verification rules and effective record templates based on the newly added business needs, and recorded in the system to form query, maintenance and other functions. In the specific implementation process, there are multiple data verification rules. The first determination unit includes a first extraction module, a first generation module and a first test module. The first extraction module is used to extract at least part of the above-mentioned data verification rules from multiple data verification rules according to the above-mentioned first business need, and obtain a first target data verification rule; the first generation module is used to add the above-mentioned first target data verification rule to the above-mentioned record template, generate an incremental verification rule, and test the above-mentioned incremental verification rule to obtain a first test result; the first test module is used to update the above-mentioned incremental verification rule when the above-mentioned first test result indicates that the test fails, and test the above-mentioned incremental verification rule again until the above-mentioned first test result indicates that the test passes, and determines that the version of the above-mentioned incremental verification rule that passes the test is the latest version.

[0125] In this solution, effectiveness testing can be carried out on the developed verification rule functions. If the verification function does not meet the requirements, it will be modified and improved until the verification rule function meets the requirements. After all verification functions have been tested and improved, the verification rule functions that have completed the test will be updated and released. This ensures that the management efficiency of data verification rules of this solution is high when blocking increments, and ensures that the incremental verification rules obtained by this solution are relatively accurate, thereby ensuring that the incremental verification rules can be used to accurately and efficiently verify the data in the future.

[0126] In actual applications, according to the reasonable classification of data verification rules and effective record templates, the data quality verification rules can be summarized and concluded according to existing business needs, and recorded in the system to form query, maintenance and other functions. In the specific implementation process, there are multiple data verification rules. The first determination unit includes a second extraction module, a second generation module and a second test module. The second extraction module is used to extract at least part of the above-mentioned data verification rules from multiple data verification rules according to the above-mentioned second business needs, and obtain the second target data verification rules; the second generation module is used to add the above-mentioned second target data verification rules to the above-mentioned record template, generate existing verification rules, and test the above-mentioned existing verification rules to obtain a second test result; the second test module is used to update the above-mentioned existing verification rules when the above-mentioned second test result indicates that the test fails, and test the above-mentioned existing verification rules again until the above-mentioned second test result indicates that the test passes, and determine that the version of the above-mentioned existing verification rules that pass the test is the latest version.

[0127] In this solution, the effectiveness test can be carried out on the developed verification rule function. If the verification function does not meet the requirements, it will be modified and improved until the verification rule function meets the requirements. After all the inspection functions are tested and improved, the verification rule function that has completed the test will be updated and released. This ensures that the management efficiency of data verification rules in this solution is high when checking inventory, and ensures that the inventory verification rules obtained by this solution are relatively accurate, thereby ensuring that the inventory verification rules can be used to accurately and efficiently verify the data in the future.

[0128] In some embodiments, the data quality verification device further includes a third verification unit, a fourth verification unit, a storage unit, a second generation unit, and a third processing unit. The third verification unit is configured to, after obtaining the data verification rule, verify the third data corresponding to the third business requirement according to the data verification rule when a third business requirement is obtained, to obtain a third verification result, wherein the third business requirement is a requirement for a new business in the first device, the third business requirement and the third data have a one-to-one correspondence, and the third data is data newly added when a new business requirement is added; the fourth verification unit is configured to, if the third verification result indicates that the verification has been passed, re-verify the third data according to the data verification rule before storing the third data locally, to obtain a fourth verification result; the storage unit is configured to, if the fourth verification result indicates that the verification has been passed, store the third data locally; the second generation unit is configured to, if the fourth verification result indicates that the verification has failed, generate a prompt message; the third processing unit is configured to, if the first verification result indicates that the verification has failed, generate a first problem data list based on the first data that failed the verification, and send it to the second device.

[0129] In this solution, data sent from a third-party second device is directly verified. If the verification fails, the data is rejected from storage. When new business needs arise in the first device, verification is required both during data entry and before data storage. If the verification fails, the data is rejected from storage, further ensuring the high quality of the data stored in the first device. If the verification result indicates that the verification fails, a list of problematic data can be generated. This list can then be used to determine the cause of the data quality problem, allowing for a timely resolution based on the cause of the anomaly.

[0130] In some embodiments, the above-mentioned device also includes a third generation unit and a second determination unit. The third generation unit is used to verify the second data corresponding to the above-mentioned second business demand according to the above-mentioned data verification rule, and after obtaining the second verification result, if the above-mentioned second verification result indicates that the verification fails, generate a second problem data list based on the above-mentioned second data that fails the verification; the second determination unit is used to determine whether the above-mentioned data verification rule is the latest version, and if the above-mentioned data verification rule is the latest version, determine that the above-mentioned data verification rule is abnormal.

[0131] In this solution, if the second verification result indicates that the verification fails, a second problem data list can be generated. In this way, the cause of the data quality problem can be determined based on the second problem data list. One of the reasons is to determine whether the data verification rules are abnormal. In this way, a solution can be determined in time based on the cause of the abnormality until the data passes the verification, thereby further ensuring that the quality of the data stored in the first device is high.

[0132] The data quality verification device includes a processor and a memory. The acquisition unit, first verification unit, first processing unit, second verification unit, and second processing unit are all stored as program units in the memory. The processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0133] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be configured, and kernel parameters can be adjusted to address the poor data quality of existing technologies, which is no longer able to meet the increasingly complex data analysis requirements.

[0134] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0135] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device where the computer-readable storage medium is located is controlled to execute the data quality verification method.

[0136] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data quality verification method is executed when the program is running.

[0137] The present application also provides a data quality verification system, including a first device and a second device, wherein the first device is a device for storing data according to business needs, the second device is a device for reporting the data to the first device, and the first device is used to execute any one of the above-mentioned data quality verification methods.

[0138] An embodiment of the present invention provides a device comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements at least the following steps of a data quality verification method. The device herein may be a server, a PC, a PAD, a mobile phone, or the like.

[0139] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initiating at least the following steps of a method for verifying data quality.

[0140] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0141] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0146] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0149] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0150] 1) The data quality verification method of the present application solidifies the data verification rules at the data entry to block incremental problem data. That is, after the second device sends the data, the first data is verified before the data is stored in the warehouse. If the verification fails, the data will not be stored in the warehouse. This avoids the problem of poor data quality caused by subsequent data being stored in the warehouse. At the same time, inside the first device, the problem data is continuously searched according to the business, that is, the data must also be continuously verified locally until the data meets the verification rules, forming a closed-loop data management, thereby improving the data quality.

[0151] 2) The data quality verification device of the present application solidifies the data verification rules at the data entry to block incremental problem data. That is, after the second device sends the data, the first data is verified before the data is stored in the warehouse. If the verification fails, the data will not be stored in the warehouse. This avoids the problem of poor data quality caused by subsequent data being stored in the warehouse. At the same time, inside the first device, the problem data is continuously searched according to the business, that is, the data must also be continuously verified locally until the data meets the verification rules, forming a closed-loop data management, thereby improving the data quality.

[0152] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A data quality verification method, characterized in that: include: The first device obtains a data verification rule, wherein the data verification rule is a predefined rule for verifying the quality of the data; When the first device obtains the first service requirement, the first device verifies the first data corresponding to the first service requirement according to the data verification rule to obtain a first verification result, wherein the first service requirement is a requirement for the second device to request the first device to process a service, the first service requirement and the first data have a one-to-one correspondence, the first data is data sent by the second device, the first device is a device that stores the data according to the service requirement, and the second device is a device that reports the data to the first device; The first device stores the first data locally if the first verification result indicates that the verification has passed, and does not store the first data locally if the first verification result indicates that the verification has failed; When the first device obtains the second service requirement, verifying the second data corresponding to the second service requirement according to the data verification rule to obtain a second verification result, wherein the second service requirement is a requirement for the first device to process a local service, the second service requirement and the second data have a one-to-one correspondence, and the second data is data locally stored by the first device; If the second verification result indicates that the verification fails, the first device updates the second data that failed the verification, and verifies the updated second data again until all the second data pass the verification; After the first device obtains the data verification rule, the method also includes: the first device generates a record template at least according to the content and number of iterations in the data verification rule, wherein the record template represents a standardized verification template for data verification; the first device determines the version of the data verification rule based on business requirements, the data verification rule and the record template.

2. The method according to claim 1, characterized in that The first device obtains data verification rules, including: The first device obtains a first data verification rule, wherein the first data verification rule is a rule for verifying the type, length, and enumeration value of all fields in the same data table, and verifying the value range of a second field limited by a data element of the first field, and the first field and the second field are both fields in the same data table; The first device obtains a second data verification rule, wherein the second data verification rule is a rule for verifying a value range of a fourth field limited by a data element of a third field, the third field is a field in the first data table, and the fourth field is a field in the second data table; The first device acquires a third data verification rule, wherein the third data verification rule is a rule for logic verification between a first data record of the first data table and a second data record of the second data table; The first device obtains a fourth data verification rule, wherein the fourth data verification rule is a verification rule for determining whether there are duplicate data records in any data table.

3. The method according to claim 1, characterized in that There are multiple data verification rules, and the first device determines a version of the data verification rule according to business requirements, the data verification rule, and the record template, including: When the first service requirement is met, the first device extracts at least part of the data verification rules from the plurality of data verification rules according to the first service requirement to obtain a first target data verification rule; The first device adds the first target data verification rule to the record template to generate an incremental verification rule, and tests the incremental verification rule to obtain a first test result; When the first test result indicates that the first device fails the test, the first device updates the incremental verification rule and tests the incremental verification rule again until the first test result indicates that the incremental verification rule passes the test, and determines that the version of the incremental verification rule that passes the test is the latest version.

4. The method according to claim 1, wherein There are multiple data verification rules, and the first device determines a version of the data verification rule according to business requirements, the data verification rule, and the record template, including: When the second service requirement is met, the first device extracts at least part of the data verification rules from the plurality of data verification rules according to the second service requirement to obtain a second target data verification rule; The first device adds the second target data verification rule to the record template to generate an existing verification rule, and tests the existing verification rule to obtain a second test result; When the second test result indicates that the first device fails the test, the first device updates the existing verification rules and tests the existing verification rules again until the second test result indicates that the test passes, and determines that the version of the existing verification rules that passes the test is the latest version.

5. The method according to claim 1, wherein After the first device obtains the data verification rule, the method further includes: When the first device obtains the third service requirement, verifying the third data corresponding to the third service requirement according to the data verification rule to obtain a third verification result, wherein the third service requirement is a requirement of a new service added in the first device, the third service requirement and the third data have a one-to-one correspondence, and the third data is data newly added when the new service requirement is added; If the third verification result indicates that the verification has passed, the first device re-verifies the third data according to the data verification rule before storing the third data locally to obtain a fourth verification result; If the fourth verification result indicates that the verification has been passed, the first device stores the third data locally; The first device generates a prompt message when the fourth verification result indicates that the verification has failed; When the first verification result indicates that the verification has failed, the first device generates a first problem data list based on the first data that has failed the verification, and sends the list to the second device.

6. The method according to claim 1, characterized in that After verifying the second data corresponding to the second business requirement according to the data verification rule to obtain a second verification result, the method further includes: When the second verification result indicates that the verification has failed, the first device generates a second problematic data list based on the second data that has failed the verification; The first device determines whether the data verification rule is the latest version, and if the data verification rule is the latest version, determines that the data verification rule is abnormal.

7. A data quality verification device, characterized in that: include: An acquisition unit, configured to acquire a data verification rule, wherein the data verification rule is a predefined rule for verifying the quality of data; a first verification unit, configured to, upon obtaining a first service requirement, verify first data corresponding to the first service requirement according to the data verification rule to obtain a first verification result, wherein the first service requirement is a requirement for the second device to request the first device to process a service, the first service requirement and the first data have a one-to-one correspondence, and the first data is data sent by the second device; a first processing unit, configured to store the first data locally if the first verification result indicates that the verification has passed, and not store the first data locally if the first verification result indicates that the verification has failed; a second verification unit, configured to, upon obtaining a second business requirement, verify second data corresponding to the second business requirement according to the data verification rule to obtain a second verification result, wherein the second business requirement is a requirement for the first device to process a local business, the second business requirement and the second data have a one-to-one correspondence, and the second data is data locally stored by the first device; a second processing unit configured to, if the second verification result indicates that the second data fails verification, update the second data that fails verification, and verify the updated second data again until all the second data passes verification; The data quality verification device also includes a first generation unit and a first determination unit. The first generation unit is used to generate a record template after obtaining the data verification rule, at least according to the content and number of iterations in the data verification rule, wherein the record template represents a standardized verification template for data verification; the first determination unit is used to determine the version of the data verification rule based on business requirements, the data verification rule and the record template.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the data quality verification method according to any one of claims 1 to 6.

9. A data quality verification system, characterized in that: include: A first device and a second device, wherein the first device is a device for storing data according to business needs, the second device is a device for reporting the data to the first device, and the first device is used to execute the data quality verification method described in any one of claims 1 to 6.

Images