Data verification method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data verification method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining to-be-verified data, and determining a verification priority corresponding to the to-be-verified data based on a data type of the to-be-verified data; dividing the to-be-verified data into a plurality of to-be-verified data sets according to the verification priority; determining a target verification mechanism corresponding to each to-be-verified data set based on the verification priority corresponding to each to-be-verified data set, and determining a target verification strategy corresponding to each to-be-verified data set based on a corresponding relationship between the verification mechanism and a preset verification strategy; and according to the target verification mechanism and the target verification strategy corresponding to each to-be-verified data set, performing data consistency verification on each to-be-verified data to obtain a data verification result. By adopting the method, the efficiency of data verification can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data verification, and in particular to a data verification method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of business, the number of information systems and the amount of data they need to process have increased dramatically. At the same time, there has also been a large demand for statistical analysis reports. Therefore, a unified statistical caliber is needed.

[0003] In traditional technology, maintenance personnel use forward unit testing to check data consistency. First, for different statistical data, maintenance personnel manually write test scripts, simulate various input conditions, run ETL (Extract-Transform-Load, data warehouse) tasks, and use assertions to verify whether the test data meets expectations, thereby determining the data consistency of the verified data.

[0004] However, in current traditional technologies, the efficiency of data consistency testing is low due to the rapid increase in data volume. Summary of the invention

[0005] Based on this, it is necessary to provide a data verification method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a data verification method, comprising:

[0007] Acquire the data to be verified, and determine the verification priority corresponding to the data to be verified based on the data type of the data to be verified;

[0008] Dividing the data to be verified into a plurality of data sets to be verified according to the verification priority;

[0009] Based on the verification priority corresponding to each of the data sets to be verified, determining the target verification mechanism corresponding to each of the data sets to be verified, and based on the correspondence between the verification mechanism and the preset verification strategy, determining the target verification strategy corresponding to each of the data sets to be verified;

[0010] According to the target verification mechanism and the target verification strategy corresponding to each of the data sets to be verified, a data consistency check is performed on each of the data sets to be verified to obtain a data verification result.

[0011] In one embodiment, obtaining the data to be verified includes:

[0012] Obtain candidate data from each system;

[0013] The data to be verified is determined from the candidate data according to the data form and data source of the candidate data.

[0014] In one of the embodiments, dividing the data to be verified into a plurality of data sets to be verified according to the verification priority includes:

[0015] Determining the verification priority of each of the data to be verified according to the risk level and data source corresponding to the data to be verified;

[0016] The data to be verified are divided based on the verification priority and each preset priority threshold to obtain data sets to be verified with multiple verification priorities.

[0017] In one embodiment, determining the target verification mechanism corresponding to each of the data sets to be verified based on the verification priority corresponding to each of the data sets to be verified includes:

[0018] Determining a verification order of the data set to be verified according to the verification priority of the data set to be verified;

[0019] The target verification mechanism corresponding to each of the to-be-verified data sets is determined according to the verification sequence and the preset data verification indicator dimension.

[0020] In one embodiment, the target verification mechanism includes a verification method and a verification order corresponding to the verification method; the determining the target verification mechanism corresponding to each of the to-be-verified data sets according to the verification order and the preset data verification indicator dimension includes:

[0021] The verification method corresponding to the data set to be verified and the verification order corresponding to each verification method are determined according to the matching relationship between the verification mechanism identifier in the verification sequence and the verification mechanism identifier in the preset data distant indicator dimension.

[0022] In one of the embodiments, the verification mechanism includes data migration testing, data warehouse testing, report testing and model query risk control system testing; the test type, data format, data range, data correlation relationship and historical data continuity each correspond to a verification mechanism.

[0023] In one embodiment, based on the correspondence between the verification mechanism and the preset verification strategy, determining the target verification strategy corresponding to each of the to-be-verified data sets includes:

[0024] When the first target verification mechanism includes data content consistency verification and data format consistency verification, based on the correspondence between the verification mechanism and the preset verification strategy, determining the first target verification strategy corresponding to each of the to-be-verified data sets;

[0025] When the first target verification mechanism includes data format consistency verification, the second target verification strategy corresponding to each of the data sets to be verified is determined based on the correspondence between the verification mechanism and the preset verification strategy; the verification frequency, depth and range in the first target verification strategy are greater than those in the second target verification strategy.

[0026] In one embodiment, after performing data consistency verification on each of the data sets to be verified according to the target verification mechanism and the target verification strategy corresponding to each of the data sets to be verified and obtaining the data verification result, the method further includes:

[0027] Storing abnormal results in the verification results;

[0028] Generate a data verification report based on the data source, verification strategy and preset report template corresponding to the abnormal result.

[0029] In a second aspect, the present application also provides a data verification device, comprising:

[0030] An acquisition module, used for acquiring data to be verified, and determining a verification priority corresponding to the data to be verified based on a data type of the data to be verified;

[0031] A division module, used for dividing the data to be verified into a plurality of data sets to be verified according to the verification priority;

[0032] A determination module, configured to determine a target verification mechanism corresponding to each of the data sets to be verified based on the verification priority corresponding to each of the data sets to be verified, and determine a target verification strategy corresponding to each of the data sets to be verified based on the correspondence between the verification mechanism and the preset verification strategy;

[0033] The verification module is used to perform data consistency verification on each of the data sets to be verified according to the target verification mechanism and the target verification strategy corresponding to each of the data sets to be verified, so as to obtain a data verification result.

[0034] In one of the embodiments, the acquisition module is specifically used to acquire candidate data of each system;

[0035] The data to be verified is determined from the candidate data according to the data form and data source of the candidate data.

[0036] In one of the embodiments, the division module is specifically used to determine the verification priority of each of the data to be verified according to the risk level and data source corresponding to the data to be verified;

[0037] The data to be verified are divided based on the verification priority and each preset priority threshold to obtain data sets to be verified with multiple verification priorities.

[0038] In one of the embodiments, the determination module is specifically used to determine the verification order of the data set to be verified according to the verification priority of the data set to be verified;

[0039] The target verification mechanism corresponding to each of the to-be-verified data sets is determined according to the verification sequence and the preset data verification indicator dimension.

[0040] In one of the embodiments, the target verification mechanism includes a verification method and a verification order corresponding to the verification method; the determination module is specifically used to determine the verification method corresponding to the data set to be verified and the verification order corresponding to each verification method based on a matching relationship between the verification mechanism identifier in the verification order and the verification mechanism identifier in a preset data distant indicator dimension.

[0041] In one of the embodiments, the verification mechanism includes data migration testing, data warehouse testing, report testing and model query risk control system testing; the test type, data format, data range, data correlation relationship and historical data continuity each correspond to a verification mechanism.

[0042] In one of the embodiments, the determination module is specifically used to determine the first target verification strategy corresponding to each of the to-be-verified data sets based on the correspondence between the verification mechanism and the preset verification strategy when the first target verification mechanism includes data content consistency verification and data format consistency verification;

[0043] When the first target verification mechanism includes data format consistency verification, the second target verification strategy corresponding to each of the data sets to be verified is determined based on the correspondence between the verification mechanism and the preset verification strategy; the verification frequency, depth and range in the first target verification strategy are greater than those in the second target verification strategy.

[0044] In one embodiment, the device further comprises:

[0045] A storage module, used for storing abnormal results in the verification results;

[0046] A generation module is used to generate a data verification report according to the data source, verification strategy and preset report template corresponding to the abnormal result.

[0047] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0048] Acquire the data to be verified, and determine the verification priority corresponding to the data to be verified based on the data type of the data to be verified;

[0049] Dividing the data to be verified into a plurality of data sets to be verified according to the verification priority;

[0050] Based on the verification priority corresponding to each of the data sets to be verified, determining the target verification mechanism corresponding to each of the data sets to be verified, and based on the correspondence between the verification mechanism and the preset verification strategy, determining the target verification strategy corresponding to each of the data sets to be verified;

[0051] According to the target verification mechanism and the target verification strategy corresponding to each of the data sets to be verified, a data consistency check is performed on each of the data sets to be verified to obtain a data verification result.

[0052] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0053] Acquire the data to be verified, and determine the verification priority corresponding to the data to be verified based on the data type of the data to be verified;

[0054] Dividing the data to be verified into a plurality of data sets to be verified according to the verification priority;

[0055] Based on the verification priority corresponding to each of the data sets to be verified, determining the target verification mechanism corresponding to each of the data sets to be verified, and based on the correspondence between the verification mechanism and the preset verification strategy, determining the target verification strategy corresponding to each of the data sets to be verified;

[0056] According to the target verification mechanism and the target verification strategy corresponding to each of the data sets to be verified, a data consistency check is performed on each of the data sets to be verified to obtain a data verification result.

[0057] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0058] Acquire the data to be verified, and determine the verification priority corresponding to the data to be verified based on the data type of the data to be verified;

[0059] Dividing the data to be verified into a plurality of data sets to be verified according to the verification priority;

[0060] Based on the verification priority corresponding to each of the data sets to be verified, determining the target verification mechanism corresponding to each of the data sets to be verified, and based on the correspondence between the verification mechanism and the preset verification strategy, determining the target verification strategy corresponding to each of the data sets to be verified;

[0061] According to the target verification mechanism and the target verification strategy corresponding to each of the data sets to be verified, a data consistency check is performed on each of the data sets to be verified to obtain a data verification result.

[0062] The above-mentioned data verification method, device, computer equipment, computer-readable storage medium and computer program product obtain the data to be verified, determine the verification priority corresponding to the data to be verified based on the data type of the data to be verified; divide the data to be verified into multiple data sets of verification priority according to the verification priority; determine the target verification mechanism corresponding to each of the verification data sets in the verification mechanism based on the correspondence between the data set to be verified and the preset verification strategy; perform data consistency verification on the data to be verified according to the target verification mechanism and the target verification strategy corresponding to the data set to be verified, and obtain the data verification result. By adopting this method, the target verification mechanism of each data set is determined by the preset verification strategy, and the most appropriate target verification mechanism can be selected for each data set to be verified, ensuring that the requirements of the data sets to be verified of different data types and priorities are met. By determining the verification priority of the data to be verified and dividing the data to be verified into the data sets to be verified according to the verification priority, the data of the key data source and the high-risk data can be processed first, and divided into multiple data sets to be verified, and the efficiency of data verification can be improved and resource allocation can be optimized. And through the verification priority of the data to be verified, the corresponding target verification strategy is selected, and the target verification mechanism corresponding to the data set to be verified is determined among a large number of verification mechanisms through the verification mechanism identifier, so as to meet the verification requirements of different types of data sets to be verified, thereby improving the targetedness of data consistency verification and thus improving the efficiency of data verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 A schematic diagram of a flow chart of a data verification method in an embodiment;

[0065] Figure 2 A schematic diagram of a data verification application process environment in one embodiment;

[0066] Figure 3 A schematic diagram of a process for determining data to be verified in one embodiment;

[0067] Figure 4 A schematic diagram of a flow chart of determining a data set to be verified in another embodiment;

[0068] Figure 5 A schematic diagram of a process for determining a target verification mechanism in one embodiment;

[0069] Figure 6 A flowchart of the steps of generating a report in one embodiment;

[0070] Figure 7 is a structural block diagram of a data verification device in an embodiment;

[0071] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0073] In one embodiment, Figure 1 As shown, a data verification method is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0074] Step 102: Acquire the data to be verified, and determine the verification priority corresponding to the data to be verified based on the data type of the data to be verified.

[0075] In the embodiments of the present application, in multiple business systems, the development process of each business system may be independent, resulting in different meanings of the field names, data extraction interfaces or indicators of reports in different business systems, which may easily lead to confusion in data meaning and confusion in usage. Taking financial institutions as an example, among the multiple business systems used in each financial institution, there is a problem of inconsistent statistical calibers for fields and names that refer to the same meaning in each business system. In addition, for data supervision of financial institutions in different regions, the corresponding data extraction requirements are different in different regions. Therefore, financial institutions need to perform data consistency checks on the data content, data structure and status in the databases of different business systems to ensure that data transmission and storage between different business systems will not be erroneous or lost, so as to ensure the accuracy and integrity of the data. Figure 2As shown in the figure, the source data layer includes databases of different business systems, such as core system databases, credit system databases and other business source system databases. When data from different business systems are transmitted and shared, the data exchange platform extracts data from databases of different business systems through NFS (Network File System) sharing, and transfers the extracted data to the data warehouse for data processing, that is, the ETL (Extract-Transform-Load, data warehouse technology) layer performs different tasks and operations on the data according to the knowledge base and detectors to obtain the business data in the form required by the application system, and then the business users process the business data through the application system.

[0076] The terminal can start the data consistency verification process according to a pre-set data verification cycle, or, in response to the user's startup operation, perform data consistency verification on the business system specified by the user, or, before performing data-related operations such as data integration synchronization, data migration, system upgrade, data backup, etc. in multiple business systems, trigger a data consistency check on the business systems involved in the operation.

[0077] Taking the triggering of data consistency check in response to data-related operations as an example, first, the terminal determines the target business system corresponding to the operation. The terminal may pre-store the data processed by each target business system. For example, the loan system contains fields under different classification rules, such as loan balance fields under different credit intervals. The terminal can determine the data to be verified corresponding to the current target business system based on the pre-stored data processed by each target business system. Each piece of data to be verified has a corresponding data type, and each data type has a corresponding verification priority. For example, when the data type is core data, the verification priority corresponding to the data to be verified is high; when the data type is non-core data, the verification priority corresponding to the data to be verified is medium; when the data type is a functional element, the verification priority corresponding to the data to be verified is low.

[0078] Step 104: divide the data to be verified into a plurality of data sets to be verified according to the verification priority.

[0079] In an embodiment of the present application, the terminal divides the data set to be verified according to the verification priority corresponding to each piece of data in the data to be verified, and then obtains all the data sets to be verified under multiple different verification priority partitions, forming multiple data sets to be verified. For example, all the data to be verified with a verification priority of "high" will be classified into one data set, all the data to be verified with a verification priority of "medium" will be classified into another data set, and so on. In this way, the data to be verified that were originally mixed together are organized in order according to the verification priority. Each divided data set to be verified contains data with the same verification priority. Then, when performing data consistency verification later, those data to be verified that are critical to the business or have a higher risk of errors can be processed first, thereby ensuring the accuracy and integrity of key data.

[0080] Step 106, based on the verification priority corresponding to each data set to be verified, determine the target verification mechanism corresponding to each data set to be verified, and based on the correspondence between the verification mechanism and the preset verification strategy, determine the target verification strategy corresponding to each data set to be verified.

[0081] In an embodiment of the present application, on the basis of determining the verification priority and the data set to be verified corresponding to each verification priority, in order to select the most suitable verification strategy for the data sets to be verified with different verification priorities, the terminal further determines the target verification mechanism and verification strategy corresponding to each data set to be verified. Among them, the verification mechanism is applied to multiple test scenarios such as data migration testing, data warehouse testing, report testing, and model query risk control system testing. Each test scenario may include different test purposes in checking data consistency, integrity or accuracy. The target verification mechanism refers to a specific method for checking data consistency, integrity and accuracy. For example, the verification method for the data to be verified of different data types includes full comparison, repeatability verification, integrity verification, format verification and consistency verification. The preset verification strategy refers to the specific operation method and rules taken when executing each target verification mechanism. For example, different target verification mechanisms correspond to different preset verification strategies, and different preset verification strategies have different verification frequencies, verification depths and verification ranges.

[0082] Each data set to be verified contains data to be verified in different data formats. The terminal sorts the verification order of the data to be verified according to the verification priority. Data sets with high verification priority usually correspond to stricter and more complex verification mechanisms. Since data sets with high verification priority may have a strong correlation with other data, data consistency verification needs to be prioritized, while data sets with low verification priority can use relatively simple and fast verification mechanisms, that is, the verification priority corresponds to a part of the target verification mechanism. For example, there are multiple target verification mechanisms corresponding to data sets with a "high" verification priority, including full comparison, repeatability verification, integrity verification and other target verification mechanisms. On this basis, the target verification strategies covered by multiple target verification mechanisms also include multiple. Data sets with a "low" verification priority may only need basic format verification. Finally, the terminal can determine the target verification mechanism and target verification strategy corresponding to each data set to be verified with each verification priority.

[0083] Step 108 , according to the target verification mechanism and target verification strategy corresponding to each data set to be verified, perform data consistency verification on each data set to be verified to obtain a data verification result.

[0084] In an embodiment of the present application, the terminal performs data consistency verification and executes a target verification mechanism on the data to be verified according to the target verification strategy to achieve data consistency verification for the data to be verified. The terminal can obtain the data verification result. If there is a problem of data inconsistency, the terminal can determine the detailed reasons for the inconsistent data fields or data formats.

[0085] In the above data verification method, the target verification mechanism of each data set is determined by presetting the verification strategy, and the most appropriate target verification mechanism can be selected for each data set to be verified, ensuring that the needs of data sets to be verified of different data types and priorities are met, thereby improving the targetedness of data consistency verification and thus improving the efficiency of data verification.

[0086] In an exemplary embodiment, before determining the priority of the data to be verified, it is necessary to first clarify the content of the data to be verified, such as Figure 3 As shown, step 102 includes steps 302 to 304. Among them:

[0087] Step 302: Obtain candidate data for each system.

[0088] In the embodiment of the present application, first, the terminal determines the scope, object and target (data source) of the data consistency check, and clarifies which systems' data will be used as candidate data (data source). The system that needs to perform data consistency check can be a core system, a credit system, or other business-related source system. Each system may have its own independent database, storing data in different formats and structures, and after establishing a connection with the database of each system, each system table record, field value or other business-related data element is extracted as candidate data.

[0089] Step 304: determine the data to be verified from the candidate data according to the data format and data source of the candidate data.

[0090] In an embodiment of the present application, the terminal performs a formal analysis on the extracted candidate data, including the data type (for example, numeric value, text, date, etc.), structure (for example, table, key-value pair, etc.) and possible format (for example, XML, JSON, CSV, etc.), and identifies the data source of the candidate data, that is, determines which system's database the candidate data comes from. Different systems process different business data, and the importance of business data varies. The terminal can identify according to the importance parameters and data formats set for different systems, determine which candidate data need to be checked for consistency, and then obtain the data to be checked.

[0091] In this embodiment, by determining the data to be verified among the candidate data through data form and data source, the candidate data can be effectively obtained from each system, and the data to be verified that needs to be verified for consistency can be determined therefrom, which serves as the basis for subsequent data verification, helping to ensure the accuracy and completeness of the data consistency verification.

[0092] In an exemplary embodiment, Figure 4 As shown, step 104 includes steps 402 to 404. Among them:

[0093] Step 402: Determine the verification priority of each data to be verified according to the risk level and data source corresponding to the data to be verified.

[0094] In the embodiment of the present application, different risk levels are predefined in the terminal, such as high, medium, low, etc. These risk levels are generally based on the business impact, data importance, or the impact of potential errors. Specifically, as shown in Table 1 below, the risk level can be reflected by the data level, and the terminal pre-stores the priority levels corresponding to different business systems and test types.

[0095] Table 1

[0096]

[0097] For each data to be verified, the terminal assigns a corresponding risk level based on its field, data type, business background, etc. For example, core business data (e.g., loan balance, account information), financial-related reports, risk-related reports, and regulatory reporting reports are usually considered high-risk.

[0098] For the data sources corresponding to the data to be verified, the data from different data sources differ in format, naming and business logic, and the data to be verified from different data sources also have corresponding priorities. The terminal can perform a comprehensive assessment based on the risk level and data source to obtain a priority score as the verification priority. For example, the terminal calculates the verification priority for each piece of data to be verified based on a weighted scoring method.

[0099] Step 404 , divide the data to be verified based on the verification priority and each preset priority threshold to obtain a plurality of data sets to be verified with verification priorities.

[0100] In an embodiment of the present application, a preset priority threshold for dividing the verification priority is pre-set in the terminal. The preset priority threshold can be a fixed score, ratio or other quantitative indicator for distinguishing different verification priorities. For example, the terminal divides the verification priority into different intervals according to a fixed score interval, so that 0-20 is divided into low priority, 21-70 is divided into medium priority, and 71-100 is divided into high priority. Then, according to the verification priority corresponding to each piece of data to be verified, each piece of data to be verified is classified into its corresponding priority interval. For example, the terminal classifies data with a verification priority of 85 into a high-priority data set to be verified.

[0101] After completing the division of each data to be verified, the terminal stores the divided data sets to be verified in the system, marks the verification priority of each data set to be verified, and sorts each data set to be verified according to the verification priority to ensure that high-priority data sets to be verified can be given priority for data consistency verification.

[0102] In this embodiment, by determining the verification priority of the data to be verified and dividing the data to be verified into data sets to be verified according to the verification priority, data of key data sources and high-risk data can be processed preferentially and divided into multiple data sets to be verified. At the same time, the efficiency of data verification can be improved, resource allocation can be optimized, and data consistency can be ensured.

[0103] In an exemplary embodiment, Figure 5 As shown, step 106 includes steps 502 to 504. Among them:

[0104] Step 502: Determine the verification order of the data sets to be verified according to the verification priorities of the data sets to be verified.

[0105] Among them, the target verification strategy includes a verification mechanism identifier.

[0106] In an embodiment of the present application, the terminal pre-stores the verification priorities corresponding to different types of data to be verified, and retrieves the data to be verified contained in the data to be verified that needs to be selected for verification order in the data set to be verified, and then the terminal obtains the verification priority corresponding to each data to be verified in the data set to be verified based on the correspondence between the data to be verified in the data set to be verified and the verification priority. The verification priority can be a numerical identifier or a character identifier, so that the terminal can perform secondary sorting of the data set to be verified according to the verification priority. Finally, the terminal maps the verification priorities of all the data to be verified in the data set to be verified to the data set to be verified according to the numerical values ​​of the verification priority from high to low, or according to the order of "high priority", "medium priority" and "low priority" in the character identifier. For example, the terminal can redefine the verification priority of the data set to be verified according to the proportion of data to be verified with different verification priorities contained in the data set to be verified.

[0107] According to the verification priority of the data set to be verified, the terminal selects the verification strategy of the corresponding verification priority level from the preset verification strategy library as the target verification strategy of the data set to be verified. Each target verification strategy contains relevant verification mechanism identifiers, such as structured data verification mechanism identifiers, semi-structured data verification mechanism identifiers, etc. For example, the verification strategy corresponding to a high-priority data set may be very strict, covering verification of multiple verification dimensions such as data content consistency, data length, data range, and data structure type, while the verification strategy corresponding to a low-priority data set may be relatively simplified, and may only include verification of a small number of verification dimensions such as data length or whether data is missing.

[0108] Step 504: determine the target verification mechanism corresponding to each data set to be verified according to the verification sequence and the preset data verification indicator dimension.

[0109] In an embodiment of the present application, the terminal selects each target verification mechanism corresponding to the verification mechanism identifier from all the verification mechanisms contained in the preset verification mechanism library in accordance with the verification order according to the verification mechanism identifier contained in the preset data verification indicator dimension as the specific data consistency verification method for the data set to be verified. The preset data verification indicator dimension mainly includes consistency verification of data content and data structure, wherein the consistency verification of data content is to check whether the content of the data is consistent, including the fields, values ​​and formats of the data; the consistency verification of data structure is to check whether the structure of the data is consistent, including the tables, fields and relationships of the data.

[0110] For the data forms of the data to be verified that are detailed in the preset data verification indicator dimension, including structured data, semi-structured data, metadata, etc., structured data, semi-structured data, and metadata correspond to a verification mechanism identifier of the preset data verification indicator dimension. Structured data refers to data that has a clear, predefined data model and follows a consistent order. Structured data is usually stored in a relational database and strictly follows the data format and length specifications, such as enterprise reporting system data, enterprise ERP system, financial system, and data in medical HIS databases; semi-structured data refers to data that is between structured data and unstructured data, has certain structured characteristics, but does not fully conform to structured characteristics, such as log files, XML documents, JSON documents, Email, HTML documents, etc. Semi-structured data usually contains data that describes the structure and actual data, and the data structure is tightly coupled with the data itself.

[0111] The verification mechanism pre-stored in the terminal has a corresponding verification mechanism identifier. Taking the verification mechanism of structured data and unstructured data as an example, the verification mechanism carrying the structured data verification mechanism identifier can be to obtain the information of structured data through inherent key values, and the data format of structured data is fixed. For example, in a relational database, SQL query statements can be used to perform consistency verification on structured data between different systems. The verification mechanism carrying the semi-structured data verification mechanism identifier can be to obtain the corresponding information through flexible key value adjustment, and the data format of semi-structured data is not fixed. Semi-structured data usually requires the use of a specific parser (for example, XML parser, JSON parser, etc.) to extract and process the information in the semi-structured data, and then perform consistency verification on the semi-structured data between different systems.

[0112] In this embodiment, the corresponding target verification strategy is selected by the verification priority of the data to be verified, and the target verification mechanism corresponding to the data set to be verified is determined from a large number of verification mechanisms through the verification mechanism identifier, thereby meeting the verification requirements of different types of data sets to be verified, thereby ensuring the accuracy of data consistency verification.

[0113] In an exemplary embodiment, the target verification mechanism includes a verification method and a verification order corresponding to the verification method, and step 504 includes:

[0114] Step 5041, according to the matching relationship between the verification mechanism identifier in the verification sequence and the verification mechanism identifier in the preset data remote indicator dimension, determine the verification method corresponding to the data set to be verified and the verification order corresponding to each verification method.

[0115] In an exemplary embodiment, the verification mechanism includes data migration testing, data warehouse testing, report testing, and model query risk control system testing; the test type, data format, data range, data correlation relationship, and historical data continuity each correspond to a verification mechanism.

[0116] In an embodiment of the present application, the terminal can also verify the data format, data value range, and data association relationship of the data to be verified. Specifically, the data format can be a consistency check of the data length of the data to be verified in different systems, and a consistency check of the data value range of the data to be verified in different systems; for data association relationships, the terminal can detect whether there is a synchronous operation of a certain data operation task in the databases of different systems. For example, in a product trading system, when a new order record is added, the database will check the inserted customer_id and customer_no (order number) values ​​to generate an order data in the customers table; in the accounting system, the account record and association information of the corresponding order customer_no will also be generated at the same time to maintain the data relationship of the order in different systems or different business tables; when deleting a record in the customers table (if there is an associated order record), it may be decided how to handle the associated order record according to the database settings (such as cascade deletion, restricted deletion, etc.), that is, the terminal can maintain the consistency of the data association relationship between the data to be verified in different systems according to the preset setting rules.

[0117] In addition, by analyzing the trend of historical data changes, the terminal can also determine the business data of tables, fields or systems that may have data consistency problems, and compare the target data and historical data based on the pre-calculation of business logic to see if they are processed in accordance with the consistency of business logic. For example, in the deposit system, a savings customer conducted an account cancellation process on day T. In the entire data link, the associated system and table data must also be processed on day T+1. The balance field corresponding to this customer in the financial system should be 0, and the customer information status of the corresponding table in the customer system should be displayed as account cancellation.

[0118] At the same time, the terminal can also pre-set verification mechanisms corresponding to different test scenarios. Data migration consistency testing refers to the process of migrating data from one storage system, database or data format to another storage system, database or data format to maintain consistency. Database table consistency testing is a test process to ensure that data between multiple tables in a database and between different copies of a table (such as tables in a master-slave database) are consistent in terms of logical relationships, data integrity and accuracy. Report consistency testing is a test process to ensure that reports generated under different conditions (such as different times, different systems, different data sources, etc.) are consistent in terms of data, format and logic. Model query system consistency testing refers to the process of verifying that the results returned by the model query system are consistent in terms of data accuracy, integrity and logic under different conditions.

[0119] In an exemplary embodiment, step 106 further includes:

[0120] Step 1061, when the first target verification mechanism includes data content consistency verification and data format consistency verification, based on the correspondence between the verification mechanism and the preset verification strategy, determine the first target verification strategy corresponding to each data set to be verified.

[0121] Step 1062, when the first target verification mechanism includes data format consistency verification, based on the correspondence between the verification mechanism and the preset verification strategy, determine the second target verification strategy corresponding to each data set to be verified; the verification frequency, depth, and range in the first target verification strategy are greater than the second target verification strategy.

[0122] In an exemplary embodiment, Figure 6 As shown, after step 108, the method further includes steps 602 to 604. Among them:

[0123] Step 602: store the abnormal results in the verification results.

[0124] In an embodiment of the present application, the terminal stores the abnormal results detected during the verification process for subsequent analysis and processing. Abnormal results refer to data that does not conform to expectations or rules found during the verification process. Specifically, during the verification process, the terminal performs a data consistency check on the verification data according to the target verification mechanism and the target verification strategy, obtains the data verification result, identifies any abnormal data items that do not conform to the rules, and stores them.

[0125] Step 604: Generate a data verification report based on the data source, verification strategy and preset report template corresponding to the abnormal result.

[0126] In an embodiment of the present application, the terminal generates a detailed data verification report based on the stored abnormal results, in combination with the corresponding data source, verification strategy, and a pre-set report template. At the same time, the terminal can also associate the abnormal results with the executed verification strategy, indicating the rules or mechanisms based on which each abnormality is detected, to help understand the nature of the problem. Specifically, a variety of report templates are pre-set in the terminal, and each template contains a specific structure and format for presenting different types of verification results. The template may include a title, an abnormality list, a detailed description, an impact analysis, and recommended measures. Furthermore, based on the analysis of the abnormal results, the terminal fills the relevant information into the preset report template to generate a final data verification report. The data verification report should include a specific description of the abnormality, data source, verification strategy, abnormal time, impact range, and recommended solutions.

[0127] After the data verification report is generated, it can be output in multiple formats, such as PDF, Word, HTML, etc., to facilitate the use of different users and systems. The generated report will be automatically sent to relevant personnel or departments through emails, system notifications, etc., to ensure that key personnel are aware of the verification results and actions to be taken in a timely manner.

[0128] In this embodiment, a data verification report is generated by using the data source, verification strategy and preset report template corresponding to the abnormal results. This can not only effectively store the abnormal results found during the verification process, but also automatically generate a detailed data verification report based on these results, providing clear and accurate information to assist operation and maintenance personnel in solving the problem of inconsistent data in different systems.

[0129] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0130] Based on the same inventive concept, the embodiment of the present application also provides a data verification device for implementing the data verification method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data verification device embodiments provided below can refer to the limitations on the data verification method above, and will not be repeated here.

[0131] In an exemplary embodiment, Figure 7 As shown, a data verification device 700 is provided, comprising: an acquisition module 701, a division module 702, a determination module 703 and a verification module 704, wherein:

[0132] An acquisition module 701 is used to acquire data to be verified, and determine a verification priority corresponding to the data to be verified based on the data type of the data to be verified;

[0133] A division module 702 is used to divide the to-be-verified data into a plurality of to-be-verified data sets of verification priorities according to the verification priority;

[0134] The determination module 703 is used to determine the target verification mechanism corresponding to each data set to be verified based on the verification priority corresponding to each data set to be verified, and determine the target verification strategy corresponding to each data set to be verified based on the correspondence between the verification mechanism and the preset verification strategy;

[0135] The verification module 704 is used to perform data consistency verification on the data to be verified according to the target verification mechanism and target verification strategy corresponding to the data set to be verified, and obtain a data verification result.

[0136] In one embodiment, the acquisition module 701 is specifically used to acquire candidate data of each system;

[0137] The data to be verified is determined from the candidate data according to the data form and data source of the candidate data.

[0138] In one embodiment, the division module 702 is specifically used to determine the verification priority of each data to be verified according to the risk level and data source corresponding to the data to be verified;

[0139] The data to be verified are divided based on the verification priority and each preset priority threshold to obtain a plurality of data sets to be verified with verification priorities.

[0140] In one embodiment, the determination module 703 is specifically used to determine the verification order of the data set to be verified according to the verification priority of the data set to be verified;

[0141] Determine the target verification mechanism corresponding to each data set to be verified based on the verification order and the preset data verification indicator dimensions.

[0142] In one of the embodiments, the target verification mechanism includes a verification method and a verification order corresponding to the verification method; the determination module 703 is specifically used to determine the verification method corresponding to the data set to be verified and the verification order corresponding to each verification method based on the matching relationship between the verification mechanism identifier in the verification order and the verification mechanism identifier in the preset data farther indicator dimension.

[0143] In one of the embodiments, the verification mechanism includes data migration testing, data warehouse testing, report testing, and model query risk control system testing; the test type, data format, data range, data correlation relationship, and historical data continuity each correspond to a verification mechanism.

[0144] In one embodiment, the determination module 703 is specifically used to determine the first target verification strategy corresponding to each to-be-verified data set based on the correspondence between the verification mechanism and the preset verification strategy when the first target verification mechanism includes data content consistency verification and data format consistency verification;

[0145] When the first target verification mechanism includes data format consistency verification, the second target verification strategy corresponding to each data set to be verified is determined based on the correspondence between the verification mechanism and the preset verification strategy; the verification frequency, depth and range in the first target verification strategy are greater than those in the second target verification strategy.

[0146] In one embodiment, the apparatus 700 further includes:

[0147] A storage module, used for storing abnormal results in the verification results;

[0148] The generation module is used to generate a data verification report based on the data source, verification strategy and preset report template corresponding to the abnormal result.

[0149] Each module in the above data verification device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0150] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be verified. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data verification method is implemented.

[0151] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0152] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0153] Acquire the data to be verified, and determine the verification priority corresponding to the data to be verified based on the data type of the data to be verified;

[0154] Dividing the data to be verified into multiple data sets to be verified according to the verification priority;

[0155] Based on the verification priority corresponding to each data set to be verified, determine the target verification mechanism corresponding to each data set to be verified, and based on the correspondence between the verification mechanism and the preset verification strategy, determine the target verification strategy corresponding to each data set to be verified;

[0156] According to the target verification mechanism and target verification strategy corresponding to each data set to be verified, data consistency verification is performed on each data set to obtain a data verification result.

[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0158] Obtain candidate data from each system;

[0159] The data to be verified is determined from the candidate data according to the data form and data source of the candidate data.

[0160] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0161] Determine the verification priority of each data to be verified according to the risk level and data source corresponding to the data to be verified;

[0162] The data to be verified are divided based on the verification priority and each preset priority threshold to obtain a plurality of data sets to be verified with verification priorities.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] Determine the verification order of the data set to be verified according to the verification priority of the data set to be verified;

[0165] Determine the target verification mechanism corresponding to each data set to be verified based on the verification order and the preset data verification indicator dimensions.

[0166] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0167] According to the matching relationship between the verification mechanism identifier in the verification sequence and the verification mechanism identifier in the preset data distant indicator dimension, the verification method corresponding to the data set to be verified and the verification sequence corresponding to each verification method are determined.

[0168] In one embodiment, the verification mechanism includes data migration testing, data warehouse testing, report testing, and model query risk control system testing; the test type, data format, data range, data association relationship, and historical data continuity each correspond to a verification mechanism.

[0169] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0170] When the first target verification mechanism includes data content consistency verification and data format consistency verification, based on the correspondence between the verification mechanism and the preset verification strategy, determining the first target verification strategy corresponding to each data set to be verified;

[0171] When the first target verification mechanism includes data format consistency verification, the second target verification strategy corresponding to each data set to be verified is determined based on the correspondence between the verification mechanism and the preset verification strategy; the verification frequency, depth and range in the first target verification strategy are greater than those in the second target verification strategy.

[0172] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0173] Storing abnormal results in the verification results;

[0174] Generate a data verification report based on the data source, verification strategy and preset report template corresponding to the abnormal result.

[0175] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0176] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0178] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0180] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data verification method, characterized in that: The method comprises: Acquire the data to be verified, and determine the verification priority corresponding to the data to be verified based on the data type of the data to be verified; Dividing the data to be verified into a plurality of data sets to be verified according to the verification priority; Based on the verification priority corresponding to each of the data sets to be verified, determining the target verification mechanism corresponding to each of the data sets to be verified, and based on the correspondence between the verification mechanism and the preset verification strategy, determining the target verification strategy corresponding to each of the data sets to be verified; According to the target verification mechanism and the target verification strategy corresponding to each of the data sets to be verified, a data consistency check is performed on each of the data sets to be verified to obtain a data verification result.

2. The method according to claim 1, characterized in that The obtaining of the data to be verified comprises: Obtain candidate data from each system; The data to be verified is determined from the candidate data according to the data form and data source of the candidate data.

3. The method according to claim 1, characterized in that The step of dividing the data to be verified into a plurality of data sets to be verified according to the verification priority level includes: Determining the verification priority of each of the data to be verified according to the risk level and data source corresponding to the data to be verified; The data to be verified are divided based on the verification priority and each preset priority threshold to obtain data sets to be verified with multiple verification priorities.

4. The method according to claim 1, characterized in that: The determining, based on the verification priority corresponding to each of the data sets to be verified, a target verification mechanism corresponding to each of the data sets to be verified, includes: Determining a verification order of the data set to be verified according to the verification priority of the data set to be verified; The target verification mechanism corresponding to each of the to-be-verified data sets is determined according to the verification sequence and the preset data verification indicator dimension.

5. The method according to claim 4, characterized in that The target verification mechanism includes a verification method and a verification order corresponding to the verification method; the target verification mechanism corresponding to each of the to-be-verified data sets is determined according to the verification order and the preset data verification indicator dimension, including: The verification method corresponding to the data set to be verified and the verification order corresponding to each verification method are determined according to the matching relationship between the verification mechanism identifier in the verification sequence and the verification mechanism identifier in the preset data distant indicator dimension.

6. The method according to claim 1 or 4, characterized in that: The verification mechanism includes data migration testing, data warehouse testing, report testing and model query risk control system testing; the test type, data format, data range, data association relationship and historical data continuity each correspond to a verification mechanism.

7. The method according to claim 1, characterized in that The step of determining the target verification strategy corresponding to each of the to-be-verified data sets based on the correspondence between the verification mechanism and the preset verification strategy includes: When the first target verification mechanism includes data content consistency verification and data format consistency verification, based on the correspondence between the verification mechanism and the preset verification strategy, determining the first target verification strategy corresponding to each of the to-be-verified data sets; When the first target verification mechanism includes data format consistency verification, the second target verification strategy corresponding to each of the data sets to be verified is determined based on the correspondence between the verification mechanism and the preset verification strategy; the verification frequency, depth and range in the first target verification strategy are greater than those in the second target verification strategy.

8. The method according to claim 1, characterized in that After performing data consistency verification on each of the data sets to be verified according to the target verification mechanism and the target verification strategy corresponding to each of the data sets to be verified and obtaining the data verification result, the method further includes: Storing abnormal results in the verification results; Generate a data verification report based on the data source, verification strategy and preset report template corresponding to the abnormal result.

9. A data verification device, characterized in that: The device comprises: An acquisition module, used for acquiring data to be verified, and determining a verification priority corresponding to the data to be verified based on a data type of the data to be verified; A division module, used for dividing the data to be verified into a plurality of data sets to be verified according to the verification priority; A determination module, configured to determine a target verification mechanism corresponding to each of the data sets to be verified based on the verification priority corresponding to each of the data sets to be verified, and determine a target verification strategy corresponding to each of the data sets to be verified based on the correspondence between the verification mechanism and the preset verification strategy; The verification module is used to perform data consistency verification on each of the data sets to be verified according to the target verification mechanism and the target verification strategy corresponding to each of the data sets to be verified, so as to obtain a data verification result.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images