Data quality verification method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of data processing technology, and discloses a data quality verification method, device, computer equipment and storage medium. The data quality verification method obtains a data verification request including verification configuration information and request condition information; determines the target verification type and the data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; determines the target query statement according to the query configuration information and the request condition information, determines the target data source according to the data source connection information, and executes the target query statement in the target data source to obtain the data to be verified; performs data processing on the data to be verified according to the verification rule information to obtain the data verification result corresponding to the target verification type; if the data verification result is a data verification abnormality, generates data verification warning information. The present invention realizes automatic query and verification of data, improves the convenience of use and operation and maintenance, and has good versatility.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data quality verification method, device, computer equipment and storage medium. Background Art

[0002] In the information age, data is crucial to the operation and decision-making of enterprises. Data quality directly affects the efficiency of business operations and the accuracy of decision-making. Data quality verification uses a series of technical means and rules to check data to ensure the integrity, consistency and accuracy of data, which helps enterprises to more accurately analyze market trends, optimize resource allocation and improve customer satisfaction.

[0003] In the prior art, data quality verification is divided into two methods: manual verification and tool verification, both of which have certain defects. The manual verification method requires a lot of manpower and time to gradually proofread the intermediate data and result data. The proofreading process is inefficient and cannot meet the real-time requirements. It is difficult to process large-scale data sets, and it is prone to human errors and omissions, and the accuracy and consistency of the data cannot be guaranteed. The tool verification method can improve processing efficiency by relying on open source software tools. For example, the open source tool Griffin performs the data quality verification process by providing a complete data quality management framework. Although existing open source software tool products provide powerful functions, the software deployment depends on big data ecosystems such as Hadoop and Spark. The use and configuration process is often very complicated, which increases the difficulty of use and operation and maintenance. In addition, it can only verify a single data table and cannot meet the application requirements across data sources. Summary of the invention

[0004] Based on this, it is necessary to provide a data quality verification method, device, computer equipment and storage medium to address the above technical problems, so as to solve the problems of low processing efficiency, complex operation process and poor versatility of existing data quality verification methods.

[0005] A data quality verification method, comprising: Obtaining a data verification request, wherein the data verification request includes verification configuration information and request condition information; Determine the target verification type and the data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; Determine a target query statement according to the query configuration information and the request condition information, determine a target data source according to the data source connection information, and execute the target query statement in the target data source to obtain data to be verified; Performing data processing on the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type; If the data verification result is a data verification abnormality, a data verification warning message is generated.

[0006] A data quality verification device, comprising: A request acquisition module, used to acquire a data verification request, wherein the data verification request includes verification configuration information and request condition information; A configuration information parsing module, used to determine a target verification type and data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; A data query module, used to determine a target query statement according to the query configuration information and the request condition information, determine a target data source according to the data source connection information, and execute the target query statement in the target data source to obtain data to be verified; A data verification module, used to process the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type; The verification warning module is used to generate data verification warning information if the data verification result is a data verification abnormality.

[0007] A computer device comprises a memory, a processor and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data quality verification method when executing the computer-readable instructions.

[0008] A computer-readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the data quality verification method as described above.

[0009] In the above-mentioned data quality verification method, device, computer equipment and storage medium, the data quality verification method obtains a data verification request, and the data verification request includes verification configuration information and request condition information; determines the target verification type and the data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; determines the target query statement according to the query configuration information and the request condition information, determines the target data source according to the data source connection information, and executes the target query statement in the target data source to obtain the data to be verified; performs data processing on the data to be verified according to the verification rule information to obtain the data verification result corresponding to the target verification type; if the data verification result is a data verification abnormality, a data verification warning information is generated. Based on the verification configuration information and request condition information extracted from the data verification request, the present invention can realize automatic query and verification of data after instant configuration, does not need to rely on the big data environment, simplifies the use and configuration process, and improves the convenience of use and operation and maintenance. Based on the target verification type and its associated configuration in the verification configuration information, the present invention can realize both single data source verification and cross-data source verification, increases the flexibility of data verification, and has good scalability and versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 is a flow chart of a data quality verification method in one embodiment of the present invention; Figure 2 is a structural schematic diagram of a data quality verification device in one embodiment of the present invention; Figure 3 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The data quality verification method provided in this embodiment can be applied to an application scenario in which a user configures verification requirements through a client, and a server performs data query and verification according to the configured verification requirements, and the client communicates with the server. The client includes but is not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0014] In one embodiment, if Figure 1 As shown, a data quality verification method is provided, comprising the following steps S10-S50: S10. Obtain a data verification request, where the data verification request includes verification configuration information and request condition information.

[0015] Understandably, the data verification request is a request information generated after the user configures the verification requirements to verify whether the data is normal. The data verification request is used to trigger the data quality verification process. The data verification request includes verification configuration information and request condition information. Among them, the verification configuration information refers to the specific information configured by the user according to needs to characterize how to query and verify the data, such as verification type, data source, query configuration and verification rules. Request condition information refers to the specific conditions used to specify the query data, such as time range, value of specific attributes, etc.

[0016] S20. Determine a target verification type and data source connection information, query configuration information, and verification rule information corresponding to the target verification type according to the verification configuration information.

[0017] Understandably, after the server obtains the data verification request, it parses the verification configuration information in the data verification request to determine the target verification type and related information. The target verification type refers to the specific verification type determined based on the verification configuration information, that is, the data source verification type selected by the user according to needs. Verification types include single data source verification and cross-data source verification. Single data source verification refers to data verification of a specific single database table. Cross-data source verification refers to comparing and verifying data between at least two different database tables. The target verification type is one of single data source verification and cross-data source verification.

[0018] The relevant information of the target verification type includes the data source connection information, query configuration information and verification rule information corresponding to the target verification type, which are the information required to perform the subsequent query and verification steps. The data source connection information refers to the detailed information used to connect to the database, file or other data source storing data, such as the connection address string, user name and password of the database, or the path and access rights of the file. The query configuration information refers to the retrieval method and parameter information used to find data from the data source, such as the Structured Query Language (SQL) statement. The query configuration information represents how to locate and extract the data to be verified. The verification rule information refers to the specific verification logic or conditions used to determine whether the data meets the expected requirements, such as data type check, value range verification and uniqueness constraint. Specifically, when the target verification type is single data source verification, the data source connection information and query configuration information corresponding to one data source, and the verification rule information for verifying the found data. When the target verification type is cross-data source verification, the data source connection information and query configuration information corresponding to at least two data sources, and the verification rule information for combining and verifying the found data.

[0019] S30. Determine a target query statement according to the query configuration information and the request condition information, determine a target data source according to the data source connection information, and execute the target query statement in the target data source to obtain data to be verified.

[0020] Understandably, the server can construct a complete target query statement based on the query configuration information and request condition information. The target query statement refers to an SQL statement used to retrieve the data to be verified from the data source, and the target query statement includes the field name to be queried. The server uses the data source connection information to connect to the target data source, and executes the target query statement in the target data source to obtain the data to be verified. The target data source refers to a database table used to store data that needs to be verified. The data to be verified refers to the query result of the target query statement, that is, the indicator value corresponding to the field name in the target query statement found from the target data source.

[0021] S40: Process the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type.

[0022] Understandably, after obtaining the data to be verified, the server parses the verification rule information, and can obtain specific verification methods and parameters (such as comparing the value of the data to be verified with the expected value), and processes the data to be verified according to the verification method to obtain the data verification result corresponding to the target verification type. The data verification result refers to the information used to characterize whether the data to be verified meets the set quality requirements. The data verification results include normal data verification and abnormal data verification. Normal data verification is used to characterize that the data to be verified meets the quality requirements, and abnormal data verification is used to characterize that the data to be verified does not meet the quality requirements.

[0023] S50: If the data verification result is a data verification abnormality, a data verification warning message is generated.

[0024] Understandably, after obtaining the data verification result, the server further determines whether the data verification result is a data verification abnormality. When the data verification result is that the data verification is normal, it indicates that the data to be verified meets the quality requirements, and the data quality verification process ends. When the data verification result is that the data verification is normal, it indicates that the data to be verified does not meet the quality requirements, and the server will generate a data verification warning message. The data verification warning information is a prompt message used to characterize that the data does not meet the quality requirements, including detailed information of the abnormal data, the type of abnormality, possible causes, and recommended repair measures. The server can also send the data verification warning information to the preset notification recipient, so that the preset notification recipient can handle the data with verification abnormalities. The preset notification recipient refers to a pre-set person used to adjust the data that does not meet the quality requirements, such as a data operation and maintenance manager.

[0025] This embodiment obtains a data verification request, which includes verification configuration information and request condition information; determines the target verification type and the data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; determines the target query statement according to the query configuration information and the request condition information, determines the target data source according to the data source connection information, and executes the target query statement in the target data source to obtain the data to be verified; processes the data to be verified according to the verification rule information to obtain the data verification result corresponding to the target verification type; if the data verification result is a data verification abnormality, a data verification warning information is generated. Based on the verification configuration information and request condition information extracted from the data verification request, this embodiment can realize automatic query and verification of data after instant configuration, without relying on the big data environment, simplifying the use and configuration process, and improving the convenience of use and operation and maintenance. Based on the target verification type and its associated configuration in the verification configuration information, this embodiment can realize both single data source verification and cross-data source verification, increase the flexibility of data verification, and have good scalability and versatility. The data quality verification method of this embodiment can automatically execute the data verification process based on the data verification request, reduce manual intervention, and improve verification efficiency and accuracy. In addition, by generating data verification warning information, operation and maintenance personnel can be promptly reminded to pay attention to data quality issues and take corresponding measures.

[0026] In one embodiment, step S10, i.e., before obtaining the data verification request, includes: S101, receiving a target verification type entered by a client, generating a connection configuration request according to the target verification type, and sending the connection configuration request to the client; S102: upon receiving data source connection information corresponding to the connection configuration request sent by the client, generating a verification configuration request according to the data source connection information, and sending the verification configuration request to the client; S103: upon receiving query configuration information corresponding to the verification configuration request sent by the client, generating a verification rule configuration request according to the query configuration information, and sending the verification rule configuration request to the client; S104, upon receiving the verification rule information corresponding to the verification rule configuration request sent by the client, determining the verification configuration information according to the target verification type, data source connection information, query configuration information and verification rule information; S105, obtaining request time information, and determining the request condition information according to the request time information; S106. Generate the data verification request according to the verification configuration information and the request condition information.

[0027] Understandably, before obtaining a data verification request, the user can configure the verification requirements through data interaction between the client and the server, so that the server can generate a data verification request based on the configured requirement information. First, the client displays the configuration page to the user through the human-computer interaction interface. The user selects single data source verification or cross-data source verification as the target verification type through the configuration page, and the client sends the target verification type to the server. The server generates a connection configuration request based on the received target verification type and sends it to the client. The connection configuration request is used to prompt the user to enter the data source connection requirements through the client.

[0028] Then, the user fills in the data source connection information corresponding to the target verification type through the configuration page, and the client sends the data source connection information to the server. The server generates a verification configuration request based on the received data source connection information and sends it to the client. The verification configuration request is used to prompt the user to enter the data query requirements through the client. When the target verification type is single data source verification, the data source connection information is the connection information of a single target data source; when the target verification type is cross-data source verification, the data source connection information includes the connection information of at least two target data sources. Preferably, this embodiment connects to the data source through the JDBC (Java Database Connectivity) driver, and the data source connection information is the API interface connection property of the database. In addition, for some special databases, the configuration items of the data source connection information can be expanded according to actual needs. For example, Elastic Search supports Restful query, and the data source connection information needs to be configured as HTTP connection properties.

[0029] Next, the user fills in the query configuration information of the target data source corresponding to the data source connection information through the configuration page. The client sends the query configuration information to the server. The server generates a verification rule configuration request based on the received query configuration information and sends it to the client. The verification rule configuration request is used to prompt the user to enter the data verification requirements through the client. Specifically, the client receives the custom query statement entered through the client, and determines the custom query statement as the query configuration information (including the database table name, field name and query condition of the query, etc.). For example, when the user needs to verify whether the revenue and expenditure of a certain day meet expectations, enter the custom query statement "select sum(revenue) astotal_revenue, sum(disburse) as total_disburse from tab where biz_date = '${date}'", that is, select the total revenue (total_revenue) and total expenditure (total_disburse) under a specific business date (biz_date) from the database table named "tab". The aggregate function sum() is used to calculate the total revenue and total expenditure respectively. ${date} is a variable placeholder, which is used to replace the actual date value or the specified date value before executing the query statement. For data query during cross-data source verification, users need to fill in the corresponding query statements for different data sources, and the number of fields and field names in the query configuration information between different data sources must be consistent, so that the data to be verified can be obtained according to the field name for comparison and verification. For example, when it is necessary to perform cross-data source verification on two target data sources, it is necessary to determine whether the number of query fields and query field names of the two target data sources are consistent. If they are inconsistent, a query configuration error message is generated. That is, the select fields of the two target data sources must include the total_revenue and total_disburse fields, so that the total revenue and total expenditure data between the two data sources can be compared and verified in the future. They cannot be missing or increased. In the process of generating a verification rule configuration request based on the received query configuration information, the server needs to parse the query configuration information, extract the field indicators, and generate a verification rule request based on the field indicators. The parsing methods include but are not limited to AST syntax tree parsing or secondary encapsulation of SQL requests to obtain response columns. For example, the custom query statement "select sum(revenue) as total_revenue, sum(disburse) astotal_disburse from tab where biz_date = '${date}'" is parsed to obtain two field indicators, total_revenue and total_disburse.Field metrics are information used to measure and evaluate the performance of data in one or more fields in a database table.

[0030] Next, the user fills in the verification rule information corresponding to the field indicators in the query configuration information through the configuration page, and the client sends the verification rule information to the server. The server determines the verification configuration information according to the target verification type, data source connection information, query configuration information and verification rule information. For example, the user fills in the subsequent verification method and parameters for the two field indicators of total_revenue and total_disburse as needed, that is, the configuration calculates and compares the total revenue and total expenditure values ​​found from the database table. In addition, after the user fills in the verification rule information corresponding to the field indicators in the query configuration information through the configuration page, before the client sends the verification rule information to the server, the user can also fill in the alarm configuration information corresponding to the verification rule information through the configuration page, and send the verification rule information and alarm configuration information to the server, so that the server can determine the verification configuration information according to the target verification type, data source connection information, query configuration information, verification rule information and alarm configuration information. The alarm configuration information is the method and parameters used to trigger an alarm when the data to be verified does not meet the verification rules, such as sending a text message and a text message template.

[0031] Finally, the server obtains the request time information and determines the request condition information based on the request time information. The request time information refers to the real-time date generated by the data verification request, that is, the date of the day. Before the server determines the request condition information based on the request time information, the user can also fill in the target query time used to characterize the verification time requirement through the configuration page (such as specifying a specific date or date range based on the actual time requirement of data verification), so the client needs to monitor whether the user enters the target query time. If the client detects that the user enters the target query time, it will send the target query time to the server instead of the request time information, so that the server can determine the request condition information based on the target query time. If the client does not detect that the user enters the target query time, it will directly send the request time information to the server, so that the server can determine the request condition information based on the request time information. Based on the request condition information, the server generates a data verification request based on the verification configuration information and the request condition information.

[0032] Based on the information interaction between the client and the server, this embodiment realizes the complete process from the client inputting information to the server generating a data verification request based on the input information. Each step depends on the output of the previous step and serves as the input of the next step to ensure the integrity of the verification configuration. In addition, this embodiment can customize and expand different verification types and configuration items according to actual needs, thereby improving the flexibility of the verification configuration.

[0033] In one embodiment, in step S30, determining the target query statement according to the query configuration information and the request condition information includes: S301, determining an initial query statement according to the query configuration information; S302: Determine a query variable parameter according to the request condition information, and determine a target query statement according to the initial query statement and the query variable parameter.

[0034] Understandably, on the one hand, the server parses the query configuration information to obtain the initial query statement. The initial query statement refers to a basic query statement that has not yet been filled with specific query conditions, such as the custom query statement entered by the user when configuring the verification requirements: select sum(revenue) as total_revenue, sum(disburse) astotal_disburse from tab where biz_date = '${date}'. The initial query statement contains the basic structure of the database query, such as which fields to select (select), which table or tables to query from (from), and possible query conditions ('${date}'). The initial query statement cannot be executed in the target data source because it lacks the specific value of the variable placeholder ('${date}') in the query condition. On the other hand, the server determines the specific date and time contained in the request condition information as the query variable parameter, and determines the target query statement based on the combination of the initial query statement and the query variable parameter. The query variable parameter refers to the value assigned to the variable placeholder ('${date}') in the initial query statement. That is, the variable placeholders in the initial query statement are replaced with the specific dates in the request condition information to obtain a complete target query statement that can be executed in the target data source.

[0035] This embodiment configures the data source connection information based on the JDBC driver or custom driver, and supports the verification of data from a single data source or across data sources. In subsequent queries, by parsing the select field of the SQL query statement, the date variable in the query statement is replaced according to the data date, and the queried data is subjected to a fixed value or volatility data quality verification according to the verification rules. This embodiment not only simplifies the operation process and lowers the threshold for use, but also significantly reduces resource usage compared to deployment in a big data environment, is easy to integrate into other systems, and is also suitable for a wider range of data quality verification scenarios, improves verification flexibility, and can be expanded in terms of data source connection, verification type, and verification method. For example, the reporting platform detects whether there are abnormal fluctuations in daily report data, and the workflow system is used to detect whether the output data meets expectations.

[0036] This embodiment can flexibly generate a query statement that meets user needs based on query configuration information and request condition information, improves the flexibility and customizability of the query, and can dynamically construct a query statement and execute the query based on the user's customized configuration.

[0037] In one embodiment, in step S302, determining the query variable parameter according to the request condition information includes: S3021, determining whether there is a target query time in the request condition information; S3022: If the request condition information contains a target query time, determine the target query time as a query variable parameter; S3023: If the target query time does not exist in the request condition information, obtain the request time information, and process the request time information according to a preset variable rule to obtain a query variable parameter.

[0038] Understandably, when the server determines the query variable parameters according to the request condition information, it is first necessary to determine whether a specific target query time is explicitly specified in the request condition information. The target query time refers to the specific time point or time period that the user specifies when configuring the verification requirements to query the data. If there is a target query time in the request condition information, the target query time is directly determined as the query variable parameter. If the target query time is not specified in the request condition information, the request time information when the data verification request is generated will be obtained, and the request time information will be processed according to the preset variable rules to obtain the query variable parameters. The preset variable rules are pre-set processing methods for time transformation, such as time offset, time window determination, etc. The purpose of the preset variable rules is to derive a reasonable query time range based on the request time. For example, when the request time information is the date of the day, the date of the previous day is determined as the query variable parameter to facilitate querying the business data of the previous day.

[0039] In this embodiment, if the user specifies a target query time, the time is directly used for query; if the user does not specify a target query time, a reasonable query time range is derived through a preset rule. This embodiment can ensure that the query operation can be flexibly adjusted according to the user's time requirements, so as to accurately find the required data.

[0040] In one embodiment, in step S40, that is, performing data processing on the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type, includes: S401, determining the data type, comparison method and data expected value corresponding to the data to be verified according to the verification rule information; S402, determining a target verification method according to the data type and the target verification type, and performing data processing on the data to be verified according to the target verification method to obtain a data verification value; S403: Compare the data verification value with the data expected value according to the comparison method to obtain the data verification result.

[0041] Understandably, the verification rule information includes the data type, verification method, comparison method and data expected value set for the field indicator. The data type refers to the format type of the data to be verified corresponding to the field indicator, and the data type is divided into numerical type and fluctuating type. Numerical data refers to specific numerical values, which can be directly mathematically operated. Fluctuating data refers to data used to characterize trends, periodicity and random fluctuations in time series, such as the rate of change of the business data of the next day compared to the business data of the previous day. The verification method refers to the method and parameters for pre-processing different types of data before verification. The expected value of data refers to the critical value used to determine whether the data to be verified meets the quality requirements. For example, if the expected daily revenue is > 1 million, then the expected value of the data corresponding to the total daily revenue is 1,000,000. The comparison method is a result used to characterize the size relationship between the actual value and the expected value, such as less than, less than or equal to, equal to, greater than or equal to, greater than and not equal to, etc.

[0042] Different data types and target verification types correspond to different target verification methods. For example, when the target verification type is "single data source verification" and the data type is "numeric type", the target verification method is to directly use the original data for subsequent verification. Determine the target verification method according to the data type and target verification type, and process the data to be verified according to the target verification method to obtain the data verification value. The data verification value refers to the result of the data to be verified after being processed by the target verification method. Compare the data verification value with the data expected value according to the comparison method to obtain the data verification result.

[0043] After parsing the verification rule information, this embodiment performs verification processing in different ways on the data to be verified of different data types and target verification types, and obtains the verification result by comparing with the expected value, thereby ensuring the accuracy and flexibility of data verification.

[0044] In one embodiment, the data type includes a numerical type and a fluctuating type, and the target verification type includes a single data source verification and a cross-data source verification; in step S402, that is, determining a target verification method according to the data type and the target verification type, performing data processing on the data to be verified according to the target verification method to obtain a data verification value, includes: S4021. When the target verification type is single data source verification and the data type is a numeric type, a fixed value verification is determined as the target verification mode, and the data to be verified is determined as a data verification value; S4022. When the target verification type is single data source verification and the data type is a fluctuation type, determine the mean fluctuation verification as the target verification mode, obtain the period average value corresponding to the data to be verified, perform fluctuation calculation processing on the data to be verified and the period average value, and obtain a data verification value; S4023: When the target verification type is cross-data source verification and the data type is a numeric type, a sum-difference verification is determined as the target verification mode, and a sum-difference calculation process is performed based on the to-be-verified data obtained from at least two target data sources to obtain a data verification value; S4024. When the target verification type is cross-data source verification and the data type is a fluctuation type, cross-fluctuation verification is determined as the target verification method, and fluctuation calculation is performed based on the data to be verified from at least two target data sources to obtain a data verification value.

[0045] Understandably, when performing verification, first determine whether the target verification type is single data source verification or cross-data source verification, and then perform subsequent verification based on the data type. When the target verification type is single data source verification and the verification type is numerical, the fixed value verification is determined as the target verification method, and the data to be verified is determined as the data verification value. Fixed value verification refers to a method of keeping the original value of the data to be verified, that is, the value actually queried is compared with the expected value of the data without changing the value through preprocessing. When the target verification type is single data source verification and the verification type is fluctuation type, the mean fluctuation verification is determined as the target verification method, and the period average value corresponding to the data to be verified is obtained, and the fluctuation rate of the verification data and the period average value is calculated to obtain the data verification value. Mean fluctuation verification refers to a preprocessing method that requires the verification data to be averaged and the fluctuation rate calculated. The period average value corresponding to the data to be verified refers to the average value within several data statistical periods starting from the date corresponding to the query variable parameter. For example, the data to be verified is the daily total revenue data, the data statistical period is 1 day, the query variable parameter is the date of the previous day, and the period average value is the average value of the total business revenue data in the past 7 days. The volatility is calculated as: volatility = (sample value - benchmark value) / benchmark value × 100%, so that the absolute value of the calculated volatility can be compared with the expected value in the subsequent comparison method. The sample value is the data to be verified for this query, and the benchmark value is the average value of the last n periods, where n is an integer greater than 2, preferably n=7.

[0046] When the target verification type is cross-data source verification and the verification type is numerical, the sum-difference verification is determined as the target verification method, and the sum-difference calculation is performed based on the data to be verified obtained from at least two target data sources to obtain a data verification value. The sum-difference verification refers to a preprocessing method that requires summing or difference calculation of data between different data sources. For example, when the target data sources of the cross-data source verification are data source 1 and data source 2, the data verification value is "data source 1-data source 2" or "data source 1+data source 2". When the target verification type is cross-data source verification and the verification type is fluctuation type, the cross-fluctuation verification is determined as the target verification method, and the volatility calculation is performed based on the data to be verified from at least two target data sources to obtain a data verification value. Cross-fluctuation verification refers to a preprocessing method that requires cross-volatility calculation of data between different data sources. The cross-volatility calculation method is volatility = (data source 1-data source 2) / data source 2×100%, so that the absolute value of the calculated volatility can be compared with the expected value in the subsequent comparison method.

[0047] This embodiment performs different preprocessing methods on the data to be verified of different data types and target verification types to obtain data verification values, thereby improving the effectiveness and efficiency of subsequent comparisons between the data verification values ​​and the data expected values. In addition, this embodiment isolates query verification from calculation, and only requires a single server to meet the operating conditions, which is convenient for use and operation and maintenance.

[0048] In one embodiment, in step S403, that is, comparing the data verification value with the data expected value according to the comparison method to obtain the data verification result, includes: S4031, comparing the data check value with the data expected value to obtain a comparison result, and determining whether the comparison result is consistent with the comparison method; S4032: If the comparison result is consistent with the comparison method, determining that the data verification result is normal; S4033: If the comparison result is inconsistent with the comparison method, determine that the data verification result is a data verification anomaly.

[0049] Understandably, the server compares the data verification value with the data expected value, obtains the comparison result, and determines whether the comparison result is consistent with the comparison method. The comparison result refers to the actual size relationship between the data verification value and the data expected value. For example, when the expected daily revenue is > 1 million, the data expected value corresponding to the daily total revenue is 1,000,000, and the comparison method is greater than. If the target verification type is "single data source verification" and the data type is "numeric type", the request time information is the date of the day, and the data to be verified (the total daily revenue data of the previous day) is 900,000, then the data to be verified (900,000) is directly used as the data verification value to compare with the data expected value (1,000,000), and the comparison result is less than. At this time, the comparison result is inconsistent with the comparison method, and the data verification result is determined to be a data verification abnormality. Further, the data verification warning information can also contain more detailed information (such as the difference between the data verification value and the data expected value).

[0050] This embodiment ensures the validity and accuracy of data verification by setting a clear comparison method and comparing the data verification value with the data expected value, can promptly discover errors or anomalies in the data, and improves the efficiency of data quality verification.

[0051] In practical applications, data verification is often used to ensure the accuracy, integrity and consistency of data, especially during data transmission, storage and processing. By implementing an effective data verification mechanism, the reliability and stability of the system can be greatly improved.

[0052] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0053] In one embodiment, a data quality verification device is provided, and the data quality verification device corresponds one-to-one to the data quality verification method in the above embodiment. Figure 2 As shown, the data quality verification device includes a request acquisition module 10, a configuration information parsing module 20, a data query module 30, a data verification module 40 and a verification warning module 50. The functional modules are described in detail as follows: A request acquisition module 10, used to acquire a data verification request, wherein the data verification request includes verification configuration information and request condition information; A configuration information parsing module 20, configured to determine a target verification type and data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; A data query module 30, configured to determine a target query statement according to the query configuration information and the request condition information, determine a target data source according to the data source connection information, and execute the target query statement in the target data source to obtain data to be verified; A data verification module 40 is used to process the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type; The verification warning module 50 is used to generate data verification warning information if the data verification result is a data verification abnormality.

[0054] In one embodiment, the request acquisition module 10 includes: A data type configuration unit, configured to receive a target verification type entered by a client, generate a connection configuration request according to the target verification type, and send the connection configuration request to the client; a connection information configuration unit, configured to generate a verification configuration request according to the data source connection information when receiving the data source connection information corresponding to the connection configuration request sent by the client, and send the verification configuration request to the client; a query configuration unit, configured to, upon receiving query configuration information corresponding to the verification configuration request sent by the client, generate a verification rule configuration request according to the query configuration information, and send the verification rule configuration request to the client; A verification rule configuration unit, configured to determine the verification configuration information according to the target verification type, data source connection information, query configuration information and verification rule information when receiving the verification rule information corresponding to the verification rule configuration request sent by the client; A request condition information determining unit, used to obtain request time information, and determine the request condition information according to the request time information; A data verification request generating unit is used to generate the data verification request according to the verification configuration information and the request condition information.

[0055] In one embodiment, the data query module 30 includes: An initial query statement determining unit, configured to determine an initial query statement according to the query configuration information; The target query statement determining unit is used to determine the query variable parameters according to the request condition information, and determine the target query statement according to the initial query statement and the query variable parameters.

[0056] In one embodiment, the data query module 30 further includes: A target query time determination unit, used to determine whether there is a target query time in the request condition information; a first query variable determining unit, configured to determine the target query time as a query variable parameter if the request condition information contains a target query time; The second query variable determination unit is used to obtain request time information if there is no target query time in the request condition information, and process the request time information according to a preset variable rule to obtain a query variable parameter.

[0057] In one embodiment, the data verification module 40 includes: A verification rule information parsing unit, used to determine the data type, comparison method and data expected value corresponding to the data to be verified according to the verification rule information; A data verification value determination unit, configured to determine a target verification method according to the data type and the target verification type, and perform data processing on the data to be verified according to the target verification method to obtain a data verification value; A comparison processing unit is used to compare the data verification value with the data expected value according to the comparison method to obtain the data verification result.

[0058] In one embodiment, the data verification module 40 further includes: A first data check value determination unit, configured to determine a fixed value check as a target check mode and determine the data to be checked as a data check value when the target check type is a single data source check and the data type is a numerical type; A second data verification value determination unit is used to determine mean fluctuation verification as a target verification method when the target verification type is single data source verification and the data type is a fluctuation type, and obtain a period average value corresponding to the data to be verified, and perform fluctuation rate calculation processing on the data to be verified and the period average value to obtain a data verification value; a third data check value determination unit, configured to determine a sum-difference check as a target check mode when the target check type is cross-data source check and the data type is a numeric type, and perform sum-difference calculation processing based on the to-be-checked data obtained from at least two target data sources to obtain a data check value; The fourth data verification value determination unit is used to determine the cross-fluctuation verification as the target verification method when the target verification type is cross-data source verification and the data type is a fluctuation type, and to obtain the data verification value based on the fluctuation rate calculation processing of the data to be verified from at least two target data sources.

[0059] In one embodiment, the data verification module 40 further includes: A comparison result judgment unit, used for comparing the data verification value with the data expected value to obtain a comparison result, and judging whether the comparison result is consistent with the comparison method; a data verification normality determination unit, configured to determine that the data verification result is normal if the comparison result is consistent with the comparison mode; The data verification anomaly determination unit is used to determine that the data verification result is a data verification anomaly if the comparison result is inconsistent with the comparison mode.

[0060] For the specific definition of the data quality verification device, please refer to the definition of the data quality verification method above, which will not be repeated here. Each module in the above-mentioned data quality verification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0061] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer-readable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database of the computer device is used to store data involved in the data quality verification method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, a data quality verification method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0062] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor executes the computer-readable instructions to implement the following steps: Obtaining a data verification request, wherein the data verification request includes verification configuration information and request condition information; Determine the target verification type and the data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; Determine a target query statement according to the query configuration information and the request condition information, determine a target data source according to the data source connection information, and execute the target query statement in the target data source to obtain data to be verified; Performing data processing on the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type; If the data verification result is a data verification abnormality, a data verification warning message is generated.

[0063] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media store computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the following steps are implemented: Obtaining a data verification request, wherein the data verification request includes verification configuration information and request condition information; Determine the target verification type and the data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; Determine a target query statement according to the query configuration information and the request condition information, determine a target data source according to the data source connection information, and execute the target query statement in the target data source to obtain data to be verified; Performing data processing on the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type; If the data verification result is a data verification abnormality, a data verification warning message is generated.

[0064] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0065] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0066] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data quality verification method, characterized in that: include: Obtaining a data verification request, wherein the data verification request includes verification configuration information and request condition information; Determine the target verification type and the data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; Determine a target query statement according to the query configuration information and the request condition information, determine a target data source according to the data source connection information, and execute the target query statement in the target data source to obtain data to be verified; Performing data processing on the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type; If the data verification result is a data verification abnormality, a data verification warning message is generated.

2. The data quality verification method according to claim 1, characterized in that: Before obtaining the data verification request, the following steps are included: receiving a target verification type entered by a client, generating a connection configuration request according to the target verification type, and sending the connection configuration request to the client; Upon receiving the data source connection information corresponding to the connection configuration request sent by the client, generating a verification configuration request according to the data source connection information, and sending the verification configuration request to the client; Upon receiving query configuration information corresponding to the verification configuration request sent by the client, generating a verification rule configuration request according to the query configuration information, and sending the verification rule configuration request to the client; Upon receiving the verification rule information corresponding to the verification rule configuration request sent by the client, determining the verification configuration information according to the target verification type, data source connection information, query configuration information and verification rule information; Acquire request time information, and determine the request condition information according to the request time information; The data verification request is generated according to the verification configuration information and the request condition information.

3. The data quality verification method according to claim 1, characterized in that: The determining the target query statement according to the query configuration information and the request condition information includes: Determine an initial query statement according to the query configuration information; A query variable parameter is determined according to the request condition information, and a target query statement is determined according to the initial query statement and the query variable parameter.

4. The data quality verification method according to claim 3, characterized in that: The determining of the query variable parameter according to the request condition information includes: Determining whether a target query time exists in the request condition information; If the request condition information contains a target query time, the target query time is determined as a query variable parameter; If the target query time does not exist in the request condition information, the request time information is obtained, and the request time information is processed according to a preset variable rule to obtain a query variable parameter.

5. The data quality verification method according to claim 1, characterized in that: The performing data processing on the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type includes: Determine the data type, comparison method and data expected value corresponding to the data to be verified according to the verification rule information; Determine a target verification method according to the data type and the target verification type, and perform data processing on the data to be verified according to the target verification method to obtain a data verification value; The data verification value and the data expected value are compared and processed according to the comparison method to obtain the data verification result.

6. The data quality verification method according to claim 5, characterized in that: The data types include numerical type and fluctuation type, and the target verification types include single data source verification and cross-data source verification; The step of determining a target verification method according to the data type and the target verification type, and performing data processing on the data to be verified according to the target verification method to obtain a data verification value includes: When the target verification type is single data source verification and the data type is a numeric type, a fixed value verification is determined as the target verification mode, and the data to be verified is determined as a data verification value; When the target verification type is single data source verification and the data type is a fluctuation type, the mean fluctuation verification is determined as the target verification mode, and the period average value corresponding to the data to be verified is obtained, and the fluctuation rate of the data to be verified and the period average value is calculated to obtain a data verification value; When the target verification type is cross-data source verification and the data type is a numeric type, a sum-difference verification is determined as the target verification mode, and a sum-difference calculation process is performed based on the data to be verified obtained from at least two target data sources to obtain a data verification value; When the target verification type is cross-data source verification and the data type is a fluctuation type, cross-fluctuation verification is determined as the target verification method, and fluctuation calculation is performed based on the data to be verified from at least two target data sources to obtain a data verification value.

7. The data quality verification method according to claim 5, characterized in that: The step of comparing the data verification value with the data expected value according to the comparison method to obtain the data verification result includes: Compare the data check value with the data expected value to obtain a comparison result, and determine whether the comparison result is consistent with the comparison method; If the comparison result is consistent with the comparison method, determining that the data verification result is normal; If the comparison result is inconsistent with the comparison method, it is determined that the data verification result is a data verification abnormality.

8. A data quality verification device, characterized in that: include: A request acquisition module, used to acquire a data verification request, wherein the data verification request includes verification configuration information and request condition information; A configuration information parsing module, used to determine a target verification type and data source connection information, query configuration information and verification rule information corresponding to the target verification type according to the verification configuration information; A data query module, used to determine a target query statement according to the query configuration information and the request condition information, determine a target data source according to the data source connection information, and execute the target query statement in the target data source to obtain data to be verified; A data verification module, used to process the data to be verified according to the verification rule information to obtain a data verification result corresponding to the target verification type; The verification warning module is used to generate data verification warning information if the data verification result is a data verification abnormality.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, the data quality verification method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors execute the data quality verification method according to any one of claims 1 to 7.

Images