Data quality checking method, device and computer readable storage medium

Abstract

The present application relates to the technical field of Fintech (Fintech), and discloses a data quality checking method, device and computer readable storage medium, wherein the checking logic for checking the rule expression required by each field in a data table is first encapsulated at a bottom layer, and then a rule information definition table adapted to the description information of the data table field is configured based on the checking logic, so that when there is a data checking demand, only the parameters for describing the data table need to be given, and the system can automatically load the corresponding rule configuration and data table according to the parameters, and then the rule matching is used to automatically generate the rule expression required by each field checking, and finally, each field is checked according to the corresponding rule expression, so that the manual configuration and SQL statement writing are not required in the whole process, the checking personnel do not need to learn the writing of the checking rule expression, the professional requirement of the checking personnel is reduced, the configuration of the field checking is more convenient and fast, and the efficiency of the data quality checking is improved.

Description

Technical Field

[0001] This invention relates to the field of financial technology (Fintech), and in particular to a data quality verification method, device, and computer-readable storage medium. Background Technology

[0002] With the development of computer technology, more and more technologies (big data, distributed systems, blockchain, artificial intelligence, etc.) are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher demands are being placed on data processing technology.

[0003] In daily data processing, it's common to perform quality checks on collected data. Since the amount of data to be checked is usually large, massive amounts of data are stored in a Hive database (a data warehouse tool based on Hadoop; Hadoop is a distributed system infrastructure capable of high-speed computation and storage of massive amounts of data). Then, leveraging Hive's capabilities, the data validation rules are translated into SQL statements. Finally, these validation rules are executed on the data in the database to filter out abnormal data. However, traditional validation methods involve manually configuring validation rules based on field logic and manually writing SQL statements. When dealing with large amounts of data requiring validation, this method is clearly inefficient. Summary of the Invention

[0004] The main objective of this invention is to propose a data quality verification method, device, and computer-readable storage medium, aiming to solve the technical problem of low efficiency in existing manual data quality verification methods.

[0005] To achieve the above objectives, the present invention provides a data quality verification method, the data quality verification method comprising:

[0006] The system obtains input parameters and queries the rule configuration information related to the input parameters from the pre-configured rule information definition table. The rule information definition table is adapted to the description information of each data table and has several validation logic for validation rule expressions encapsulated at the underlying level.

[0007] The rule configuration information is converted into a first key-value pair set, and the data table to be verified is obtained according to the input parameters, so as to convert the data table to be verified into a second key-value pair set;

[0008] The first set of key-value pairs is matched with the second set of key-value pairs to obtain the target verification expression corresponding to each field in the data table to be verified.

[0009] The data in the data table to be verified is verified according to the target verification expression to obtain the verification result.

[0010] Optionally, the rule information definition table includes a basic information table and a verification rule definition table.

[0011] The steps of obtaining input parameters and querying rule configuration information related to the input parameters from a pre-configured rule information definition table include:

[0012] Upon receiving a data verification instruction, the database name and table name are obtained from the data verification instruction and used as the input parameters.

[0013] The rule configuration information related to the database name and data table name is retrieved from the pre-configured basic information table and the verification rule definition table. The database name parameter and the data table name parameter are defined in both the basic information table and the verification rule definition table.

[0014] Optionally, the step of converting the rule configuration information into a first set of key-value pairs includes:

[0015] The rule configuration information is converted into several first key-value pairs, wherein the key in the first key-value pair is a field name defined in the verification rule definition table, and the value in the first key-value pair is the field name corresponding to the basic information and verification rule information in the basic information table and the verification rule definition table, respectively.

[0016] Several of the first key-value pairs are aggregated into the first key-value pair set.

[0017] Optionally, the step of matching the first set of key-value pairs with the second set of key-value pairs to obtain the target validation expression corresponding to each field in the data table to be validated includes:

[0018] Match the key of each second key-value pair in the second key-value pair set with the key of each first key-value pair in the first key-value pair set;

[0019] Based on the basic information and validation rule information in the first successfully matched key-value pair, a corresponding validation rule expression is generated, which serves as the target validation expression for the field corresponding to the second successfully matched key-value pair.

[0020] Optionally, the step of obtaining the data table to be verified based on the input parameters and converting the data table to be verified into a second set of key-value pairs includes:

[0021] Generate a data query SQL statement based on the input parameters, and use the SQL statement to query the data table to be verified.

[0022] The data table to be verified is loaded into the elastic distributed dataset using a distributed computing engine;

[0023] All partitions in the elastic distributed dataset are traversed in parallel to convert each row of data in all partitions into a second key-value pair, wherein the key of the second key-value pair is the actual field name in the data table to be verified, and the value in the second key-value pair is the field value corresponding to the actual field name;

[0024] The second key-value pairs in all the partitions are aggregated into the second key-value pair set.

[0025] Optionally, the step of performing data validation on each field of the data table to be validated according to the target validation expression to obtain the validation result includes:

[0026] Determine the dependencies between the target validation expressions matched by each field in the data table to be validated;

[0027] According to the aforementioned dependency relationship, the matched target validation expression is executed on each of the fields in sequence to perform data validation on each of the fields and obtain the validation result corresponding to each field.

[0028] Optionally, after the step of performing data validation on each field of the data table to be validated according to the target validation expression to obtain the validation result, the method further includes:

[0029] If the verification result is an abnormal verification result, then a data abnormality prompt message is generated based on the abnormal verification result, and the alarm level of the prompt message is determined.

[0030] The abnormal data prompts and alarm levels of each field in the data table to be verified are summarized into an abnormal data summary table, and the abnormal data summary table is pushed to the relevant processing personnel.

[0031] Optionally, after the step of performing data validation on each field of the data table to be validated according to the target validation expression to obtain the validation result, the method further includes:

[0032] Configure the basic information and validation rules of the data table fields to generate the initial rule information definition table;

[0033] When the front end obtains the class and / or function of the custom validation rule, the class and / or function of the custom validation rule are integrated into the initial rule information definition table to obtain the rule information definition table.

[0034] Furthermore, to achieve the above objectives, the present invention also provides a data quality verification system, the data quality verification system comprising:

[0035] The rule configuration query module is used to obtain input parameters and query the rule configuration information related to the input parameters from the pre-configured rule information definition table. The rule information definition table is adapted to the description information of each data table and has several validation logic for validating rule expressions encapsulated at the underlying level.

[0036] The key-value pair set conversion module is used to convert the rule configuration information into a first key-value pair set, and obtain the data table to be verified according to the input parameters, so as to convert the data table to be verified into a second key-value pair set;

[0037] The target rule matching module is used to match the first key-value pair set with the second key-value pair set to obtain the target verification expression corresponding to each field in the data table to be verified.

[0038] The target field validation module is used to validate each field in the data table to be validated according to the target validation expression to obtain the validation result.

[0039] In addition, to achieve the above objectives, the present invention also provides a data quality verification device, which includes: a memory, a processor, and a data quality verification program stored in the memory and executable on the processor. When the data quality verification program is executed by the processor, it implements the steps of the data quality verification method as described above.

[0040] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a data quality verification program, which, when executed by a processor, implements the steps of the data quality verification method as described above.

[0041] In addition, to achieve the above objectives, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data quality verification method described above.

[0042] This invention encapsulates the validation logic of the rule expressions required for each field in a data table at the underlying level. Then, based on this, a rule information definition table adapted to the description information of the data table fields is configured. When a user has a data validation requirement, they only need to provide the relevant parameters describing the data table. The system can automatically load the corresponding rule configuration and the data table itself based on the input parameters. Then, it automatically generates the rule expressions required for validating each field in the data table through rule matching. Finally, it validates each field according to the corresponding rule expressions to obtain the validation results. The entire process requires no manual configuration or SQL statement writing, thus eliminating the need for validation personnel to learn how to write validation rule expressions, reducing the professional requirements for validation personnel. Furthermore, it eliminates the need to parse each rule into SQL before execution, making field validation configuration more convenient and faster, thereby improving the efficiency of data quality validation and solving the technical problem of low efficiency in existing manual data quality validation methods. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention;

[0044] Figure 2 This is a flowchart illustrating the first embodiment of the data quality verification method of the present invention;

[0045] Figure 3 This is an overall flowchart of data verification in a specific embodiment of the third embodiment of the data quality verification method of the present invention;

[0046] Figure 4 This is a schematic diagram of the functional modules of the data quality verification system of the present invention.

[0047] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0049] In daily data processing, it's common to perform quality checks on collected data. Since the amount of data to be checked is usually large, massive amounts of data are stored in a Hive database (a data warehouse tool based on Hadoop; Hadoop is a distributed system infrastructure capable of high-speed computation and storage of massive amounts of data). Then, leveraging Hive's capabilities, the data validation rules are translated into SQL statements. Finally, these validation rules are executed on the data in the database to filter out abnormal data. However, traditional validation methods involve manually configuring validation rules based on field logic and manually writing SQL statements. When dealing with large amounts of data requiring validation, this method is clearly inefficient.

[0050] To address the aforementioned issues, this invention provides a data quality verification method. This method first encapsulates the verification logic of rule expressions required for verifying each field in a data table at the underlying level. Then, based on this, a rule information definition table adapted to the description information of the data table fields is configured. When a user has a data verification requirement, they only need to provide relevant parameters describing the data table. The system can automatically load the corresponding rule configuration and the data of the table to be verified based on the input parameters. Then, rule matching automatically generates the rule expressions required for verifying each field in the data table. Finally, each field is verified according to the corresponding rule expressions to obtain the verification result. The entire process requires no manual configuration or SQL statement writing. Therefore, verification personnel do not need to learn how to write verification rule expressions, reducing the professional requirements for verification personnel. Furthermore, there is no need to parse each rule into SQL before execution, making field verification configuration more convenient and faster, thereby improving the efficiency of data quality verification and solving the technical problem of low efficiency in existing manual data quality verification methods.

[0051] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0052] like Figure 1As shown, the data quality verification system may include: a processor 1001, such as a CPU; a user interface 1003; a network interface 1004; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0053] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0054] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data quality verification program.

[0055] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the client (programmer's end) and communicate with the client; while the processor 1001 can be used to call the data quality verification program stored in the memory 1005 and execute the operations in the following data quality verification method.

[0056] Based on the above hardware structure, an embodiment of the data quality verification method of the present invention is proposed.

[0057] Reference Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of the data quality verification method of the present invention. The data quality verification method includes:

[0058] Step S10: Obtain the input parameters and query the rule configuration information related to the input parameters from the pre-configured rule information definition table. The rule information definition table is adapted to the description information of each data table and has several validation logic for validation rule expressions encapsulated at the underlying level.

[0059] In this embodiment, the present invention is applied to a terminal device. The data table to be verified refers to the data in the database that needs to be verified; the data is stored in the database in tables as the organizational unit. Input parameters are used to describe the data table, and the description may specifically include the database name where the data table to be verified resides, the table name of the data table to be verified, the data date, etc. The rule information definition table refers to a definition table that has been pre-configured with basic information about table fields and table field verification rules. This basic information and table field verification rules are defined for all fields in the data table to be verified, thus ensuring adaptability to these data tables. Even for some special data tables, corresponding definitions can be added to the definition table to adapt them, thus achieving high scalability. Furthermore, the underlying verification logic for these verification rules is already encapsulated, so users do not need to worry about how to write the verification rule expressions themselves. Rule configuration information refers to some rule configurations in the rule information definition table that are applicable to the current data table to be verified.

[0060] Specifically, if a user has a data verification requirement, they can specify on the terminal the database where the data to be verified is located, the name of the data table, and which items need to be verified. After receiving the user's specified content, the terminal uses it as the above input parameters, and then finds the rule configuration information applicable to the data table to be verified from the pre-configured rule information definition table according to these input parameters.

[0061] Step S20: The rule configuration information is converted into a first key-value pair set, and the data table to be verified is obtained according to the input parameters, so as to convert the data table to be verified into a second key-value pair set;

[0062] In this embodiment, the first key-value pair set refers to a set composed of several first key-value pairs, which are key-value pairs transformed from rule configuration information. The second key-value pair set refers to a set composed of several second key-value pairs, which are key-value pairs transformed from the data table to be verified.

[0063] Specifically, the input parameters include at least the database name and the table name. The terminal converts each piece of information in the rule configuration information into key-value pairs, obtaining several first key-value pairs, and then aggregates them into a first key-value pair set. Based on the database name and table name in the input parameters, the terminal retrieves the data table to be verified from the corresponding database, then converts each row of records in the table into a second key-value pair, obtaining several second key-value pairs, and then aggregates them into a second key-value pair set. It should be noted that the rule configuration information conversion step and the retrieval and conversion steps of the data table to be verified can be performed simultaneously or sequentially, depending on the actual situation; this embodiment does not impose specific limitations.

[0064] Step S30: Match the first key-value pair set with the second key-value pair set to obtain the target verification expression corresponding to each field in the data table to be verified;

[0065] Step S40: Perform data validation on each field in the data table to be validated according to the target validation expression to obtain the validation result.

[0066] In this embodiment, the matching method includes performing keyword matching on each of the two types of key-value pairs. If a consistent relationship, subordinate relationship, or association relationship exists, then the first key-value pair can be considered to match the second key-value pair. The target validation expression refers to the validation expression applicable to each field in the data table to be validated. The target expressions for different fields may be the same or different.

[0067] Specifically, the terminal performs keyword matching between each first key-value pair in the first key-value pair set and each second key-value pair in the second key-value pair set to obtain matching results. Then, based on these matching results, it retrieves the target validation expression corresponding to each field in the data table (equivalent to each second key-value pair). The terminal executes data validation logic on each field according to the target validation expression. If the data conforms to the expression, the validation passes; otherwise, it fails, indicating that the data value is abnormal. Therefore, the validation result may be a normal validation result or an abnormal validation result.

[0068] It should be noted that the validation expressions can be generated before step S10, then associated with the corresponding rule configuration, and finally assigned according to the matching results; or they can be not generated in advance, but after the rule configuration information applicable to the data table to be validated is determined, the validation expressions corresponding to the rule configuration information are generated based on the underlying encapsulated validation logic, and then these validation expressions are assigned to the corresponding fields according to the matching results.

[0069] This embodiment provides a data quality verification method. The method first encapsulates the verification logic of the rule expressions required for verifying each field in a data table at the underlying level. Then, based on this, a rule information definition table adapted to the description information of the data table fields is configured. When a user has a data verification requirement, they only need to provide relevant parameters describing the data table. The system can automatically load the corresponding rule configuration and the data table itself based on the input parameters. Then, rule matching automatically generates the rule expressions required for verifying each field in the data table. Finally, each field is verified according to the corresponding rule expressions to obtain the verification result. The entire process requires no manual configuration or SQL statement writing. Therefore, verification personnel do not need to learn how to write verification rule expressions, reducing the professional requirements for verification personnel. Furthermore, there is no need to parse each rule into SQL before execution, making field verification configuration more convenient and faster, thereby improving the efficiency of data quality verification and solving the technical problem of low efficiency in existing manual data quality verification methods.

[0070] Furthermore, based on the above Figure 2 The first embodiment shown illustrates a second embodiment of the data quality verification method of the present invention. In this embodiment, the rule information definition table includes a basic information table and a verification rule definition table, and step S10 includes:

[0071] Step S11: Upon receiving a data verification instruction, obtain the database name and data table name from the data verification instruction to serve as the input parameters;

[0072] Step S12: Query the rule configuration information related to the database name and data table name from the pre-configured basic information table and the verification rule definition table. The database name parameter and the data table name parameter are defined in both the basic information table and the verification rule definition table.

[0073] In this embodiment, the basic information table is a table that defines various attribute parameters of the table fields (such as database name, table name, field name, validation format, etc.), and the validation rule definition table is a table that defines the validation rule format (such as database name, table name, field name, rule name, rule type, etc.).

[0074] When data validation is required, the user can input the specified database name and table name into the terminal, as well as the data date and other predefined parameters. The terminal uses this information as the input parameters. Alternatively, the user can open the corresponding description information selection interface on the terminal, select the appropriate database name and table name, and choose other parameter information. The terminal uses these selected information as the input parameters. After determining the database and table names, the terminal uses these names to query the pre-configured basic information table and validation rule definition table to retrieve the relevant rule configuration information.

[0075] As a specific embodiment, the basic information table is shown in the table below.

[0076]

[0077] The table defines basic information for its fields, including: database name, table name, field name, Chinese field name, field type, field length, nullability, primary key status, fixed length, null precision, null range, default value, and validation format. The simplified definition of the validation format is as follows:

[0078]

[0079]

[0080] The validation rule definition table is shown in the table below:

[0081]

[0082] The rule type needs to refer to the predefined rule type table, which covers a variety of common validation rule types and can be defined according to actual needs.

[0083] This embodiment pre-configures the basic information and validation rules of the table fields to be used during validation. This allows users to simply specify the various descriptive information of the fields, and the system can automatically convert them into corresponding validation rule expressions based on the pre-configured information. Users only need to select the various dimensions of information for the fields in the table to be validated, making configuration more convenient and improving configuration efficiency.

[0084] Further, step S20, which involves converting the rule configuration information into a first set of key-value pairs, includes:

[0085] Step S21: Convert the rule configuration information into several first key-value pairs, wherein the key in the first key-value pair is a field name defined in the verification rule definition table, and the value in the first key-value pair is the field name corresponding to the basic information and verification rule information in the basic information table and the verification rule definition table, respectively.

[0086] Step S22: Summarize the first key-value pairs into the first key-value pair set.

[0087] In this embodiment, since both the basic information table and the verification rule definition table define field names, the terminal can convert the rule configuration information into key-value pairs. Based on the different field names, each field name is used as a key, and the corresponding basic information in the basic information table and the corresponding verification rule in the verification rule definition table are used as the value for that key, forming a first key-value pair. This process is repeated until all the first key-value pairs are aggregated into a first key-value pair set.

[0088] This embodiment converts rule configuration information into key-value pairs, which facilitates subsequent matching with the fields to be verified, thereby improving the matching efficiency of the verification rules corresponding to the fields to be verified.

[0089] Further, step S30 includes:

[0090] Step S31: Match the key of each second key-value pair in the second key-value pair set with the key of each first key-value pair in the first key-value pair set;

[0091] Step S32: Generate a corresponding validation rule expression based on the basic information and validation rule information in the first successfully matched key-value pair, and use it as the target validation expression for the field corresponding to the second successfully matched key-value pair.

[0092] In this embodiment, the terminal performs keyword matching on each second key-value pair in the second key-value pair set and each first key-value pair in the first key-value pair set to match the basic information and rule information corresponding to each field in the table. Then, based on the matched basic information and rule information, it generates the target verification expression corresponding to each field, or directly obtains the expression associated with the matched basic information and rule information from the pre-generated verification expression combination as the target verification expression.

[0093] This embodiment converts the data table to be verified into key-value pairs, enabling the data table to be verified and the rule configuration information to be directly matched through the format conversion, thereby improving the matching efficiency of the verification rules corresponding to the fields.

[0094] Further, step S20, which involves obtaining the data table to be verified based on the input parameters and converting the data table to be verified into a second set of key-value pairs, includes:

[0095] Step S23: Generate a data query SQL statement based on the input parameters, and use the SQL statement to query the data table to be verified;

[0096] Step S24: Load the data table to be verified into the elastic distributed dataset through the distributed computing engine;

[0097] Step S25: Parallel traversal of all partitions in the elastic distributed dataset to convert each row of data in all partitions into a second key-value pair, wherein the key of the second key-value pair is the actual field name in the data table to be verified, and the value in the second key-value pair is the field value corresponding to the actual field name;

[0098] Step S26: Summarize the second key-value pairs in all partitions into the second key-value pair set.

[0099] In this embodiment, specifically, the terminal generates a corresponding data query SQL statement based on the database name, table name, and data date in the input parameters, and executes the statement to retrieve the data table to be verified. Then, using the distributed computing engine Spark, the data of the data table to be verified is loaded into an Elastic Distributed Dataset (RDD). All partitions in an RDD are traversed in parallel, and a task is generated for each partition to execute the tasks in parallel. Each task traverses and scans all data in its assigned partition. Each row of data in the partition is converted into a second key-value pair, where the key in each second key-value pair is the field name of the row (i.e., the actual field name mentioned above), and the value corresponding to the key is the field value of the row. The terminal aggregates all generated second key-value pairs into a second key-value pair set.

[0100] In existing methods, each validation rule requires the generation of a corresponding SQL statement before execution. However, in this embodiment, by introducing the distributed computing engine Spark and combining it with expressions, only a full table scan is needed to validate all fields in the table. This consumes fewer resources, has a faster validation speed, and is more efficient than traditional methods. Furthermore, even when using Spark to execute these SQL statements in existing technologies, performance issues may arise. Since this embodiment does not require writing and executing SQL statements, performance problems will not occur.

[0101] Furthermore, based on the above Figure 2 The first embodiment shown presents a third embodiment of the data quality verification method of the present invention. In this embodiment, step S40 includes:

[0102] Step S41: Determine the dependency relationships between the target validation expressions matched by each field in the data table to be validated;

[0103] Step S42: Execute the matched target validation expression on each field according to the dependency relationship to perform data validation on each field and obtain the validation result corresponding to each field.

[0104] In this embodiment, all fields in the data table to be verified are traversed, and the values ​​of each column of each row of data are executed sequentially according to the corresponding verification rules. The execution content is the target verification expression that is matched, and the parameter is the value of the field. The execution result (i.e. the above verification result) may be true or false. If the result is false, it means that the rule does not match the data and the data does not conform to the verification logic.

[0105] This embodiment further improves the overall verification efficiency by performing data verification in the order of dependencies between rules, thus avoiding the possibility of verification failure that may occur if data verification is performed directly without considering dependencies.

[0106] Furthermore, after step S40, the following steps are also included:

[0107] Step S50: If the verification result is an abnormal verification result, generate a data abnormality prompt message based on the abnormal verification result, and determine the alarm level of the prompt message;

[0108] Step S60: Summarize the data anomaly prompts and alarm levels of each field in the data table to be verified into an anomaly data summary table, and push the anomaly data summary table to the relevant processing personnel.

[0109] In this embodiment, the output format for data anomalies can be predefined. If the terminal receives an anomaly verification result (i.e., the result is false), it indicates that the field value is abnormal data. The terminal outputs the anomaly verification result to the anomaly data result table according to the error message format and alarm level defined by the rules, and takes the highest alarm level among all alarm levels of the anomaly rules for that row of data as the anomaly alarm level for that row of data. After all data in the table has been verified, the data in the anomaly result table is summarized, and the summary information is sent to the corresponding database / table manager via email or other means.

[0110] As a specific implementation method, the predefined data anomaly output format is shown in the table below:

[0111] Data Date Which data date's data validation failed? library name Hive library name to be verified Table name Validated data table name Primary key column name The primary key column of the data table comes from the primary key field defined in section 1, which provides basic information about the field. Primary key column values Which specific row of primary key data is incorrect? Verification Category Rule type definition Alarm Categories Take the highest alert level from all alert levels of the exception rules executed in this row of data.

[0112] This embodiment further improves the convenience for responsible persons to be informed of abnormal data by setting up an alarm process, which enables timely notification of data anomalies after verification.

[0113] Furthermore, prior to step S10, the following steps are also included:

[0114] Step S01: Configure the basic information and validation rules of the data table fields to generate the initial rule information definition table;

[0115] Step S02: When the front end obtains the class and / or function of the custom validation rule, integrate the class and / or function of the custom validation rule into the initial rule information definition table to obtain the rule information definition table.

[0116] In this embodiment, the terminal is pre-configured with the database and table information to be verified, including the database name, table name, person in charge, etc., as well as basic information descriptions of the Hive table data, defining the basic information of the table fields to be verified, which can be referred to in the basic information table above; it is also necessary to define the format of the table field rules, which can be referred to in the rule definition table above.

[0117] In addition, for specific rule types, rules need to be defined for all fields in the table to be validated. The specific rule type table is shown below:

[0118]

[0119]

[0120] For logical condition validation types, Aviator is a high-performance, lightweight Java expression evaluation engine. Because it can utilize all built-in functions of Aviator expressions, as well as custom-loaded Java classes / methods, and uses logical symbols to combine various complex validation logics, users can integrate their custom Java classes / methods into tables in the above manner to combine row data validation logic, thus achieving high extensibility in logical condition configuration. Furthermore, Aviator expressions can be replaced with EL expressions or Drools expressions.

[0121] Existing methods that convert validation rules into SQL statements are only suitable for simple validation logic. For complex validation logic, manual writing of complex SQL statements is required, highlighting the significant limitations of current data quality validation methods. This embodiment, however, pre-encapsulates the logic for various validation types and provides a way to add custom rules. This allows users to combine more complex validation logic according to their actual validation needs without writing SQL statements themselves, thus achieving high scalability.

[0122] As a specific embodiment, such as Figure 3 As shown, Figure 3 This is the overall flowchart for data verification. The first stage is the rule configuration phase. In this phase, the terminal needs to complete the configuration of basic table field information and table field rules. The specific configuration definition method is as described above and will not be repeated here. After configuration, the rule matching phase begins. If there is a data verification requirement, the data to be verified needs to be loaded, and the target rule expression suitable for the data to be verified is matched from the pre-configured verification rules according to the table fields of the data to be verified and the pre-configured rule definitions. After matching is completed, the rule execution phase begins. In this phase, the terminal executes the corresponding rule expressions according to the dependency relationships defined by the rules matched by each field to perform data verification. After the rules are executed, the final result processing phase begins. In this phase, the terminal summarizes and outputs the abnormal results and sends alerts to relevant personnel based on the actual alarm level.

[0123] like Figure 4 As shown, the present invention also provides a data quality verification system, the data quality verification system comprising:

[0124] The rule configuration query module 10 is used to obtain input parameters and query rule configuration information related to the input parameters from a pre-configured rule information definition table. The rule information definition table is adapted to the description information of each data table and has several validation logic for validating rule expressions encapsulated at the underlying level.

[0125] The key-value pair set conversion module 20 is used to convert the rule configuration information into a first key-value pair set, and obtain the data table to be verified according to the input parameters, so as to convert the data table to be verified into a second key-value pair set;

[0126] The target rule matching module 30 is used to match the first key-value pair set with the second key-value pair set to obtain the target verification expression corresponding to each field in the data table to be verified.

[0127] The target field validation module 40 is used to validate each field in the data table to be validated according to the target validation expression to obtain the validation result.

[0128] Optionally, the rule configuration query module 10 includes:

[0129] The input parameter acquisition unit is used to obtain the database name and data table name from the data verification instruction when a data verification instruction is received, and use them as the input parameters.

[0130] The relevant information query unit is used to query rule configuration information related to the database name and data table name from the pre-configured basic information table and the verification rule definition table. The basic information table and the verification rule definition table both define database name parameters and data table name parameters.

[0131] Optionally, the key-value pair set conversion module 20 includes:

[0132] The first key-value pair conversion unit is used to convert the rule configuration information into several first key-value pairs, wherein the key in the first key-value pair is a field name defined in the verification rule definition table, and the value in the first key-value pair is the field name corresponding to the basic information and verification rule information in the basic information table and the verification rule definition table, respectively.

[0133] The first key-value pair aggregation unit is used to aggregate several first key-value pairs into the first key-value pair set.

[0134] Optionally, the target rule matching module 30 includes:

[0135] The keyword matching unit is used to match the keyword of each second key-value pair in the second key-value pair set with the keyword of each first key-value pair in the first key-value pair set;

[0136] The target expression generation unit is used to generate a corresponding validation rule expression based on the basic information and validation rule information in the first successfully matched key-value pair, which serves as the target validation expression for the field corresponding to the second successfully matched key-value pair.

[0137] Optionally, the key-value pair set conversion module 20 includes:

[0138] The query statement unit is used to generate a data query SQL statement based on the input parameters, and to use the SQL statement to query the data table to be verified.

[0139] The computing engine loading unit is used to load the data table to be verified into the elastic distributed dataset through the distributed computing engine.

[0140] The second key-value pair conversion unit is used to traverse all partitions in the elastic distributed dataset in parallel to convert each row of data in all partitions into a second key-value pair, wherein the key of the second key-value pair is the actual field name in the data table to be verified, and the value in the second key-value pair is the field value corresponding to the actual field name.

[0141] The second key-value pair aggregation unit is used to aggregate the second key-value pairs in all partitions into the second key-value pair set.

[0142] Optionally, the target field verification module 40 includes:

[0143] The dependency determination unit is used to determine the dependency relationships between the target validation expressions matched by each field in the data table to be validated;

[0144] The verification result generation unit is used to execute the matched target verification expression on each of the fields in sequence according to the dependency relationship, so as to perform data verification on each of the fields and obtain the verification result corresponding to each field.

[0145] Optionally, the data quality verification system further includes:

[0146] An anomaly information generation module is used to generate data anomaly prompt information based on the anomaly verification result if the verification result is an anomaly verification result, and to determine the alarm level of the prompt information.

[0147] The abnormal data push module is used to summarize the abnormal data prompts and alarm levels of each field in the data table to be verified into an abnormal data summary table, and push the abnormal data summary table to the relevant processing personnel.

[0148] Optionally, the data quality verification system further includes:

[0149] The initial definition generation module is used to configure the basic information of the data table fields and the validation rules to generate the initial rule information definition table;

[0150] The custom rule integration module is used to integrate the custom validation rule classes and / or functions obtained from the front end into the initial rule information definition table to obtain the rule information definition table.

[0151] The present invention also provides a data quality verification device.

[0152] The data quality verification device includes a processor, a memory, and a data quality verification program stored in the memory and executable on the processor. When the data quality verification program is executed by the processor, it implements the steps of the data quality verification method described above.

[0153] The method implemented when the data quality verification program is executed can be referred to in various embodiments of the data quality verification method of the present invention, and will not be repeated here.

[0154] The present invention also provides a computer-readable storage medium.

[0155] The present invention provides a data quality verification program stored on a computer-readable storage medium, which, when executed by a processor, implements the steps of the data quality verification method described above.

[0156] The method implemented when the data quality verification program is executed can be referred to in various embodiments of the data quality verification method of the present invention, and will not be repeated here.

[0157] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data quality verification method described above.

[0158] The method implemented when the computer program is executed can be referred to in various embodiments of the data quality verification method of the present invention, and will not be repeated here.

[0159] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0160] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0162] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data quality verification method, characterized in that, The data quality verification method includes: The process involves obtaining input parameters and retrieving rule configuration information related to the input parameters from a pre-configured rule information definition table. The rule information definition table is adapted to the description information of each data table and encapsulates several validation rule expressions at its underlying level. The rule information definition table includes a basic information table and a validation rule definition table. The steps of obtaining input parameters and retrieving rule configuration information related to the input parameters from the pre-configured rule information definition table include: upon receiving a data validation instruction, obtaining the database name and data table name from the data validation instruction as the input parameters; and retrieving rule configuration information related to the database name and data table name from the pre-configured basic information table and validation rule definition table. Both the basic information table and the validation rule definition table define database name parameters and data table name parameters. The rule configuration information is converted into a first key-value pair set, and the data table to be verified is obtained according to the input parameters, so as to convert the data table to be verified into a second key-value pair set; wherein, the step of converting the rule configuration information into the first key-value pair set includes: converting the rule configuration information into several first key-value pairs, wherein the key in the first key-value pair is a field name defined in the verification rule definition table, and the value in the first key-value pair is the field name corresponding to the basic information and verification rule information in the basic information table and the verification rule definition table, respectively; summarizing the several first key-value pairs into the first key-value pair set; the step of obtaining the data table to be verified according to the input parameters... The step of validating a data table to transform it into a second key-value pair set includes: generating a data query SQL statement based on the input parameters, and using the SQL statement to query the data table to be validated; loading the data table to be validated into an elastic distributed dataset through a distributed computing engine; traversing all partitions in the elastic distributed dataset in parallel to convert each row of data in all partitions into a second key-value pair, wherein the key of the second key-value pair is the actual field name in the data table to be validated, and the value in the second key-value pair is the field value corresponding to the actual field name; and summarizing the second key-value pairs in all partitions into a second key-value pair set. The first set of key-value pairs is matched with the second set of key-value pairs to obtain the target verification expression corresponding to each field in the data table to be verified. The data in the data table to be verified is verified according to the target verification expression to obtain the verification result.

2. The data quality verification method as described in claim 1, characterized in that, The step of matching the first set of key-value pairs with the second set of key-value pairs to obtain the target validation expression corresponding to each field in the data table to be validated includes: Match the key of each second key-value pair in the second key-value pair set with the key of each first key-value pair in the first key-value pair set; Based on the basic information and validation rule information in the first successfully matched key-value pair, a corresponding validation rule expression is generated, which serves as the target validation expression for the field corresponding to the second successfully matched key-value pair.

3. The data quality verification method as described in claim 1, characterized in that, The step of performing data validation on each field of the data table to be validated according to the target validation expression to obtain the validation result includes: Determine the dependencies between the target validation expressions matched by each field in the data table to be validated; According to the aforementioned dependency relationship, the matched target validation expression is executed on each of the fields in sequence to perform data validation on each of the fields and obtain the validation result corresponding to each field.

4. The data quality verification method as described in claim 1, characterized in that, After the step of performing data validation on each field of the data table to be validated according to the target validation expression to obtain the validation result, the method further includes: If the verification result is an abnormal verification result, then a data abnormality prompt message is generated based on the abnormal verification result, and the alarm level of the prompt message is determined. The abnormal data prompts and alarm levels of each field in the data table to be verified are summarized into an abnormal data summary table, and the abnormal data summary table is pushed to the relevant processing personnel.

5. The data quality verification method according to any one of claims 1-4, characterized in that, Before the step of obtaining input parameters, the method further includes: Configure the basic information and validation rules of the data table fields to generate the initial rule information definition table; When the front end obtains the class and / or function of the custom validation rule, the class and / or function of the custom validation rule are integrated into the initial rule information definition table to obtain the rule information definition table.

6. A data quality verification device, characterized in that, The data quality verification device includes: a memory, a processor, and a data quality verification program stored in the memory and executable on the processor. When the data quality verification program is executed by the processor, it implements the steps of the data quality verification method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data quality verification program, which, when executed by a processor, implements the steps of the data quality verification method as described in any one of claims 1 to 5.

Images