A multi-rule data quality inspection method and system based on dynamic expression

Abstract

The present invention relates to the technical field of financial data processing, and in particular to a multi-rule data quality inspection method and system based on dynamic expressions; the present invention classifies inspection tasks according to each inspection table, generates execution logic dynamic expressions for all inspection points to be executed in the table, obtains the inspection tasks of the table, and performs parallel inspections on the inspection tasks of multiple inspection tables to obtain inspection results of multiple tables; the present invention significantly shortens the task execution time, improves data inspection efficiency, and ensures the timeliness and accuracy of data quality.

Description

Technical Field

[0001] The present invention relates to the technical field of financial data processing, and in particular to a multi-rule data quality inspection method and system based on dynamic expressions. Background Art

[0002] In the banking industry, data quality checks are crucial. Banks process a large amount of customer information and transaction records every day. Regulators have strict requirements on data quality, and ensuring the accuracy and integrity of data is a basic requirement for compliance. Banks need to conduct data quality checks regularly to prevent fines or legal liabilities due to data problems. Therefore, establishing an effective data quality check mechanism to ensure the accuracy and consistency of data has become an important task for banks.

[0003] Data quality issues mainly come from two aspects: one is the data quality problems of the business system, such as input errors, inconsistencies and missing values; the other is the problems generated during the data processing process, especially in the data extraction, transformation and loading (ETL) process, which may lead to business misunderstanding errors, code loopholes or improper processing logic, all of which may have a negative impact on the final data quality.

[0004] Regardless of the type of data problem, it is crucial to detect and rectify it as early as possible. To meet this challenge, the industry generally adopts the establishment of independent data quality inspection subsystems or modules. These subsystems usually consist of several key parts: data quality inspection task configuration, inspection task execution, and inspection result processing. However, when faced with massive data and numerous inspection rules, how to effectively improve the efficiency of data quality task inspections and reduce inspection execution time has always been an important issue that needs to be solved urgently. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention aims to provide a multi-rule data quality inspection method and system based on dynamic expressions to improve the efficiency of data quality inspection.

[0006] In order to solve the above problems, the present invention adopts the following technical solutions:

[0007] In one aspect, the present invention provides a multi-rule data quality inspection method based on dynamic expressions, comprising:

[0008] Configure the inspection task;

[0009] Parse and execute the inspection tasks for the checklist to be checked, including:

[0010] S21, classifying the inspection tasks according to each inspection sheet to be checked, selecting any inspection sheet from the inspection sheets to be checked, and generating an inspection sheet data range for the inspection sheet;

[0011] S22, obtaining a checkpoint to be executed in the checklist;

[0012] S23, according to the obtained checkpoint to be executed, according to the data range logic configured in the check rule of the checkpoint, and the mapping relationship of the check parameters, the checkpoint check range is replaced with parameters through the template engine to obtain a data check range fragment;

[0013] S24, according to the obtained checkpoint to be executed, and according to the execution logic in the check rule of the checkpoint, the template engine is used to replace the check parameters to obtain the data check logic fragment of the checkpoint;

[0014] S25, generating an execution logic dynamic expression of the checkpoint according to the data check range fragment and the data check logic fragment;

[0015] S26, repeat S22-S25 until the execution logic dynamic expressions of all the checkpoints in the checklist are generated;

[0016] S27, combining the checklist data range, merging the execution logic dynamic expressions of all checkpoints, and generating a checklist check expression;

[0017] S28, repeat S21-S27 until the checklist check expressions of all the checklists to be checked are generated;

[0018] S29. Based on the generated checklist check expressions of all the checklists to be checked, the checklist check expressions of all the checklists to be checked are executed in parallel to obtain the check results of all the checklists to be checked.

[0019] As an implementable method, the check result processing logic is added to the dynamic expression of the execution logic of the checkpoint to monitor whether the checkpoint is triggered. If the checkpoint is detected to be triggered, the information of the check item and the trigger rule is spliced.

[0020] As an implementable method, it is characterized in that the generation of the execution logic dynamic expression of the checkpoint includes generating a case when SQL execution expression as the execution logic dynamic expression of the checkpoint through a data check range fragment and a data check logic fragment.

[0021] As an implementable method, configuring the inspection task includes:

[0022] S11. Define inspection rules;

[0023] S12. Select the checklist to be checked;

[0024] S13, select the inspection items to be checked;

[0025] S14. According to one of the inspection items to be checked, configure the inspection point through the mapping relationship between the inspection item and the parameters of the inspection rule;

[0026] S15, repeat S13-S14 until all the checkpoints of the check items to be checked in the selected checklist are configured;

[0027] S16. Repeat S12-S15 until all the checkpoints of the checklist to be checked are configured.

[0028] As an implementable embodiment, the inspection rule includes basic information, execution logic, data range logic and inspection parameters.

[0029] On the other hand, the present invention provides a multi-rule data quality inspection system based on dynamic expressions, including an inspection task configuration module and an inspection task parsing and execution module;

[0030] The inspection task configuration module is used to configure the inspection task;

[0031] The inspection task parsing and execution module is used to parse and execute the inspection task on the inspection table to be checked, including:

[0032] S21, classifying the inspection tasks according to each inspection sheet to be checked, selecting any inspection sheet from the inspection sheets to be checked, and generating an inspection sheet data range for the inspection sheet;

[0033] S22, obtaining a checkpoint to be executed in the checklist;

[0034] S23, according to the obtained checkpoint to be executed, according to the data range logic configured in the check rule of the checkpoint, and the mapping relationship of the check parameters, the checkpoint check range is replaced with parameters through the template engine to obtain a data check range fragment;

[0035] S24, according to the obtained checkpoint to be executed, and according to the execution logic in the check rule of the checkpoint, the template engine is used to replace the check parameters to obtain the data check logic fragment of the checkpoint;

[0036] S25, generating an execution logic dynamic expression of the checkpoint according to the data check range fragment and the data check logic fragment;

[0037] S26, repeat S22-S25 until the execution logic dynamic expressions of all the checkpoints in the checklist are generated;

[0038] S27, combining the checklist data range, merging the execution logic dynamic expressions of all checkpoints, and generating a checklist check expression;

[0039] S28, repeat S21-S27 until the checklist check expressions of all the checklists to be checked are generated;

[0040] S29. Based on the generated checklist check expressions of all the checklists to be checked, the checklist check expressions of all the checklists to be checked are executed in parallel to obtain the check results of all the checklists to be checked.

[0041] As an implementable method, the check result processing logic is added to the dynamic expression of the execution logic of the checkpoint to monitor whether the checkpoint is triggered. If the checkpoint is detected to be triggered, the information of the check item and the trigger rule is spliced.

[0042] As an implementable method, generating the execution logic dynamic expression of the checkpoint includes generating a case when SQL execution expression as the execution logic dynamic expression of the checkpoint through a data check range fragment and a data check logic fragment.

[0043] As an implementable method, configuring the inspection task includes:

[0044] S11. Define inspection rules;

[0045] S12. Select the checklist to be checked;

[0046] S13, select the inspection items to be checked;

[0047] S14. According to one of the inspection items to be checked, configure the inspection point through the mapping relationship between the inspection item and the parameters of the inspection rule;

[0048] S15, repeat S13-S14 until all the checkpoints of the check items to be checked in the selected checklist are configured;

[0049] S16. Repeat S12-S15 until all checkpoints of the checklist to be checked are configured.

[0050] As an implementable embodiment, the inspection rule includes basic information, execution logic, data range logic and inspection parameters.

[0051] The beneficial effects of the present invention are as follows: the present invention uses a dynamic expression method to merge checkpoints and generate dynamic tasks, while retaining the original multi-node task parallel execution method; it not only optimizes the management of checkpoints, but also improves the flexibility and adaptability of tasks. Under the conditions of the same data inspection amount, inspection table, inspection scope, inspection rules and checkpoints, this solution significantly shortens the task execution time, improves the data inspection efficiency, and ensures the timeliness and accuracy of data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention is a flowchart of a multi-rule data quality inspection method based on dynamic expressions in an embodiment of the present invention.

[0053] Figure 2 The figure is a flow chart for configuring an inspection task in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below in conjunction with specific embodiments.

[0055] It should be noted that these embodiments are only used to illustrate the present invention rather than to limit the present invention. Simple improvements to the method based on the concept of the present invention all fall within the scope of protection claimed by the present invention.

[0056] The ultimate goal of data quality inspection is to identify potential problem data from massive data. The method of discovering problems is called inspection rules, for example: the name is all numbers, the ID number is inconsistent with the age, the ID number is inconsistent with the customer's gender, etc. The specific inspection content is called inspection items, such as customer name, ID number, etc. A check rule under a check item is called a checkpoint, for example, the "name is all numbers" check is performed on the "customer name", and the execution of the checkpoint is called an inspection task. Usually, data inspection is performed on a data table basis. A data table contains multiple inspection items, and a inspection item can contain multiple checkpoints. Inspection rules may be reused by multiple checkpoints.

[0057] See also Figure 1 , which is a multi-rule data quality inspection method based on dynamic expressions, including:

[0058] S10. Configure the inspection task.

[0059] S20, parsing and executing the inspection task for the inspection list to be checked, including:

[0060] S21. Classify the inspection tasks according to each inspection sheet to be checked, select any inspection sheet from the inspection sheets to be checked, and generate an inspection sheet data range for the inspection sheet.

[0061] First, classify the check tasks. Considering that although the data range of each checkpoint in the same table is different, the data range of the checkpoint will not exceed the data range of the table. Therefore, dynamically create data check expressions based on tables. First, classify all checkpoints (the execution of checkpoints is a check task) based on tables.

[0062] In order to improve the efficiency of inspection execution, first generate the inspection table data range. The table data range is for all inspection points and will be spliced ​​into the where condition when the expression is generated. The inspection table data range supports four categories: data date range, inspection object, inspection table custom conditions, and organization range. This information comes from the inspection table information configuration.

[0063] Generate checklist data range:

[0064] 1. Data date [Incremental date field] filtering. If configured as [Full], it is ignored. If configured as [Incremental], it is generated based on the incremental date field configuration combined with the current date field. If the incremental date field is DATA_DATE, the field type is character type, and the current date is 2024-01-01, then the expression DATA_DATE='2024-01-01' is generated. If the field type is date type, the expression is generated based on the database type. For Oracle database, the expression generated is DATA_DATE=to_date('2024-01-01','yyyy-MM-dd').

[0065] 2. Check object filtering. Generate based on the configured subject field. If the subject field is configured as CUST_ID, the generated expression is CUST_ID IN (SELECT chk_id FROM CHK_TBL). If not configured, the generated expression is empty.

[0066] 3. Check the table custom condition filtering. If custom conditions are configured, they are directly superimposed. If not configured, they are ignored. For example, CHK_TYPE='1'.

[0067] 4. Organization range [all, part] filtering. If the organization is all, it is ignored, otherwise it is generated according to the configuration organization field and the organization selected by the user. For example, if the organization field is ORG, and the user selects organization A (code 101) and organization B (code 102), the generated expression is ORG IN ('101', '102').

[0068] The final checklist data range is:

[0069] DATA_DATE='2024-01-01'

[0070] AND CUST_ID IN (SELECT chk_id FROM CHK_TBL)

[0071] AND CHK_TYPE='1'

[0072] AND ORG IN ('101','102').

[0073] S22. Obtain a checkpoint to be executed in the checklist.

[0074] The purpose is to change the way one checkpoint generates one inspection task, and to splice the checkpoints of the same data table into one expression without affecting the data inspection scope and inspection logic, so as to reduce the number of inspection tasks.

[0075] S23. According to the obtained checkpoint to be executed, according to the data range logic configured in the checkpoint check rule, and the mapping relationship of the check parameters, the checkpoint check range is replaced by the template engine to obtain the data check range fragment, and it is temporarily saved for subsequent use in generating the checkpoint expression. The process of replacing the checkpoint check range with the actual checkpoint data range in the check table to obtain the data check range fragment is the process of replacing the check parameter with the actual value. For example, if there is a parameter of the system date, then in actual execution, this parameter will be replaced with the computer system date when the task is executed.

[0076] Checkpoints are generated based on check rules. One check rule may generate multiple checkpoints. For example, if the check rule is a non-empty check, it may generate a non-empty check for field A or a non-empty check for field B. The data range logic is defined in the rule, and the checkpoint data range replaces the check range logic defined in the check rule with parameters such as the check field and is actually executable.

[0077] Since each checkpoint checks a different data range, the checkpoint data range must be generated first. The checkpoint data range is within the check table data range, and then the following processing is performed:

[0078] Generate checkpoint data range:

[0079] 1. Checkpoint’s own filtering:

[0080] Advanced: special SQL conditions for the main table and the sub-table.

[0081] Graphical configuration: Master table: id = '' ... Slave table: id = '' ... .

[0082] 2. Check the filtering of the SQL validation corresponding to the rule.

[0083] 3. Check the filter in the rule to see if the field is empty by default.

[0084] By filtering the above-mentioned data ranges of the checklist and checkpoints, normal data can be filtered out and problematic data can be retained, providing an accurate target data set for subsequent data inspection based on the data range.

[0085] According to the inspection range selection logic defined in the inspection rule and the inspection parameter mapping relationship, the template engine is used to generate the data inspection range expression of the inspection point, such as the document type inspection range, and its rules are defined as:

[0086] ${Parm_chk_col} = ${parm_val_id_type}

[0087] In the check item configuration, the parameter column in the parameter mapping is configured as id_type, and the parameter value is 10.

[0088] The result after replacement is Id_type = '10'.

[0089] Replace the checkpoint check range with the data check range fragment of the actual checkpoint data range in the check table.

[0090] S24, according to the obtained checkpoint to be executed, according to the execution logic in the checkpoint check rule, the template engine is used to replace the check parameters to obtain the data check logic fragment of the checkpoint. For example:

[0091] According to the execution logic defined in the check rule and the mapping relationship of the check parameters, the template engine is used to generate the check logic. If there is erroneous data, the result returned is true, otherwise the result is false.

[0092] For example, in the rule definition for name full-digit check, the template check rule is:

[0093] REGEXP_LIKE(${Parm_chk_col} , '^[0-9]+$') .

[0094] In the inspection item configuration, the inspection field is: cust_name.

[0095] After using the rule engine to replace, the generated data check logic fragment is:

[0096] REGEXP_LIKE(cust_name , '^[0-9]+$') .

[0097] S25. Generate the execution logic dynamic expression of the checkpoint according to the data check range fragment and the data check logic fragment. Generating the execution logic dynamic expression of the checkpoint includes generating the SQL execution expression of case when as the execution logic dynamic expression of the checkpoint through the data check range fragment and the data check logic fragment. The basic logic is as follows:

[0098] CASE

[0099] WHEN Data check scope THEN Data check logic

[0100] ELSE FALSE

[0101] END.

[0102] When the checkpoint meets the data check range condition, the data check logic judgment is executed. If there is erroneous data, the result is returned as true, otherwise the result is returned as false. In addition, each checkpoint expression contains a data check range fragment and a data check logic fragment. The data check expression of each checkpoint is placed in a list queue so that it can be used in the subsequent table generation expression.

[0103] For example:

[0104] CASE WHEN A.AOSP IN ('@N','@I','@E') OR A.AOSP IS NULL OR LENGTH(A.AOSP) = 0 THEN TRUE ELSE FALSE END AS ITEM_ID_1.

[0105] In this expression, if any of the above conditions is met (that is, the value of field A.AOSP is equal to one of '@N', '@I', '@E', or A.AOSP is null, or the length of A.AOSP is zero), then the entire CASE WHEN statement will return TRUE and assign this TRUE value to the newly generated column ITEM_ID_1.

[0106] S26. Repeat S22-S25 until the execution logic dynamic expressions of all the checkpoints in the checklist are generated.

[0107] S27. Combine the checklist data range, merge the execution logic dynamic expressions of all checkpoints, and generate the checklist check expression.

[0108] Merge all checkpoint expressions to generate a unified check expression. Specifically, use the SQL execution expression table of casewhen to concatenate the execution logic expressions of all checkpoints. From the above content, it can be seen that when a checkpoint has abnormal data, the dynamic logic expression execution result of the checkpoint is true, and the result is output. Therefore, the check results must be processed uniformly to generate a unified expression based on the specified data table. As shown below:

[0109] SELECT

[0110] Case when checkpoint 1 THEN check column 1||check column value 1||check rule 1 END

[0111] ||','||

[0112] Case when check point 2 THEN check column 2||check column value 2||check rule 2 END

[0113] ||','||

[0114] Case when check point N THEN check column N||check column value N||check rule N END

[0115] FROM Checklist

[0116] WHERE Check table data range

[0117] AND (checkpoint 1 OR checkpoint 2 OR checkpoint 3 OR checkpoint N).

[0118] It can be understood that the checkpoint 1 in the above checklist expression is a dynamic expression of the execution logic of the checkpoint 1, and the execution of the checkpoint verification is performed in a parallel manner.

[0119] In addition, the check result processing logic is added to the dynamic expression of the execution logic of the checkpoint to monitor whether the checkpoint is triggered. If the checkpoint is detected to be triggered, the information of the check item and the trigger rule are spliced. In an inspection, as long as any checkpoint is triggered, it means that there is a problem with the data. It is necessary to clarify which checkpoint found the problem and splice the information of the check item and the trigger rule for subsequent analysis.

[0120] S28. Repeat S21-S27 until the checklist check expressions for all the checklists to be checked are generated.

[0121] S29. Based on the generated checklist check expressions of all the checklists to be checked, the checklist check expressions of all the checklists to be checked are executed in parallel to obtain the check results of all the checklists to be checked.

[0122] From the above steps, we can see that data checking is done in units of check tables, and a unified expression is executed to complete data checking of all checkpoints.

[0123] In the above solution, the overall data set range that needs to be checked is determined through the data table in the expression. Then, by transferring the inspection range and inspection logic of the segmented data of each checkpoint to the expression, the problem of inconsistent inspection range and logic between different checkpoints is cleverly solved. A checkpoint only generates a dynamic expression of execution logic, and all checkpoints are merged into a large expression / check table inspection expression. Covering multiple checkpoints with one expression not only reduces the number of inspection tasks, improves execution efficiency, and shortens task execution time, but also ensures the independence of each checkpoint.

[0124] See also Figure 2 , S10, configure the inspection task, including:

[0125] S11. Define inspection rules, where the inspection rules include basic information, execution logic, data range logic and inspection parameters. The basic information includes rule code, rule name, execution type, inspection type, description, etc. One inspection rule corresponds to one inspection range, which may include multiple filter conditions. One filter condition includes one range logic and multiple inspection parameters. One inspection point corresponds to one data inspection range segment.

[0126] Check parameters are parameters encountered in executing logic and data scope, such as check items and check scope. Check rules in different rows contain different check parameters. For example, for the "ID number and age are inconsistent" rule, its parameters include the document type, document number, and age field. A check rule may be used by multiple check items, such as name check can be used for name check in customer table and account table.

[0127] S12. Select the checklist to be checked. For example: [Customer Information Sheet] data check, and maintain the relevant check contents:

[0128] Whether it is a dynamic table: Applicable to scenarios where the data table is dynamically generated.

[0129] Subject field, subject name: used for inspection result analysis and inspection object filtering.

[0130] Line department: used to distinguish data ownership.

[0131] Full or incremental date field: used to distinguish the data date range of the checklist.

[0132] Inspection object type and inspection object field: used to filter inspection objects.

[0133] Whether all institutions, inspection institutions field: used for inspection institution filtering

[0134] Check table custom conditions: Allow users to control the detailed inspection range of the data table.

[0135] S13. Select the inspection item to be checked. For example, select the data inspection item [ID number] in the customer information table.

[0136] S14. According to one of the inspection items to be checked, a checkpoint is configured through a mapping relationship between parameters of the inspection item and the inspection rule.

[0137] For example, when configuring the [Are the ID number and gender consistent?] checkpoint, configure the mapping relationship between the check items and the parameters of the check rules. First, select the check rule, and then configure the mapping relationship between the check data and the check parameters according to the check parameters in the check rule. For the [Are the ID number and gender consistent?] checkpoint, set the check parameter column to [ID type], the parameter value to [00-ID card], and the auxiliary check column to [Gender].

[0138] S15. Repeat S13-S14 until all the checkpoints of the selected checklist (such as [ID number]) to be checked are configured.

[0139] S16. Repeat S12-S15 until all the checkpoints of the checklists to be checked (such as the [Customer Information Table]) are configured. After the configuration is completed, submit the check rules.

[0140] In the above configuration, the data table / check table, check items, and checkpoints are independent of each other, the checkpoints are decoupled, and the check rules are reusable between checkpoints.

[0141] In order to improve the usability of the system, visual management is adopted for inspection task configuration. The data table to be inspected is maintained in the metadata management module, including the management of permissions and basic information.

[0142] A multi-rule data quality inspection system based on dynamic expressions, comprising an inspection task configuration module and an inspection task parsing and execution module;

[0143] The inspection task configuration module is used to configure the inspection task;

[0144] The inspection task parsing and execution module is used to parse and execute the inspection task for the inspection list to be checked, including:

[0145] S21, classifying the inspection tasks according to each inspection sheet to be checked, selecting any inspection sheet from the inspection sheets to be checked, and generating an inspection sheet data range for the inspection sheet;

[0146] S22, obtaining a checkpoint to be executed in the checklist;

[0147] S23, according to the obtained checkpoint to be executed, according to the data range logic configured in the check rule of the checkpoint, and the mapping relationship of the check parameters, the checkpoint check range is replaced with parameters through the template engine to obtain a data check range fragment;

[0148] S24, according to the obtained checkpoint to be executed, and according to the execution logic in the check rule of the checkpoint, the template engine is used to replace the check parameters to obtain the data check logic fragment of the checkpoint;

[0149] S25, generating an execution logic dynamic expression of the checkpoint according to the data check range fragment and the data check logic fragment;

[0150] S26, repeat S22-S25 until the execution logic dynamic expressions of all the checkpoints in the checklist are generated;

[0151] S27, combining the checklist data range, merging the execution logic dynamic expressions of all checkpoints, and generating a checklist check expression;

[0152] S28, repeat S21-S27 until the checklist check expressions of all the checklists to be checked are generated;

[0153] S29. Based on the generated checklist check expressions of all the checklists to be checked, the checklist check expressions of all the checklists to be checked are executed in parallel to obtain the check results of all the checklists to be checked.

[0154] Among them, the inspection result processing logic is added to the dynamic expression of the execution logic of the checkpoint to monitor whether the checkpoint is triggered. If it is monitored that the checkpoint is triggered, the information of the inspection item and the triggering rule is spliced.

[0155] The generating of the execution logic dynamic expression of the checkpoint includes generating a case when SQL execution expression as the execution logic dynamic expression of the checkpoint through the data check range fragment and the data check logic fragment.

[0156] The configuration of the inspection task includes:

[0157] S11. Define inspection rules, where the inspection rules include basic information, execution logic, data range logic, and inspection parameters;

[0158] S12. Select the checklist to be checked;

[0159] S13, select the inspection items to be checked;

[0160] S14. According to one of the inspection items to be checked, configure the inspection point through the mapping relationship between the inspection item and the parameters of the inspection rule;

[0161] S15, repeat S13-S14 until all the checkpoints of the check items to be checked in the selected checklist are configured;

[0162] S16. Repeat S12-S15 until all the checkpoints of the checklist to be checked are configured.

[0163] In a bank's actual production scenario, a total of 25 data tables were checked, with 514 checkpoints. With a total data volume of 1 TB (excluding replicas), 6 nodes were used for parallel execution, and it took about 8.36 hours to complete all data check tasks.

[0164] The method of the present invention was applied to the anti-money laundering data quality inspection project of a commercial bank. Compared with the original solution, under the same hardware environment, the data quality inspection time for the same 1T data volume, 25 data tables and 514 inspection points was greatly reduced from 8.36 hours to 1.38 hours, and the efficiency was improved by more than 6 times.

[0165] Data quality check is the cornerstone of data application, and its importance is self-evident. The present invention can be widely used in data projects such as data warehouses, middle platforms and platforms. Through this efficient method, data problems can be quickly identified and the effect of data application can be significantly improved. It provides data managers with a faster and more reliable means of data quality assurance.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A multi-rule data quality inspection method based on dynamic expression, characterized in that: include: Configure the inspection task; Parse and execute the inspection tasks for the checklist to be checked, including: S21, classifying the inspection tasks according to each inspection sheet to be checked, selecting any inspection sheet from the inspection sheets to be checked, and generating an inspection sheet data range for the inspection sheet; S22, obtaining a checkpoint to be executed in the checklist; S23, according to the obtained checkpoint to be executed, according to the data range logic configured in the check rule of the checkpoint, and the mapping relationship of the check parameters, the checkpoint check range is replaced with parameters through the template engine to obtain a data check range fragment; S24, according to the obtained checkpoint to be executed, and according to the execution logic in the check rule of the checkpoint, the template engine is used to replace the check parameters to obtain the data check logic fragment of the checkpoint; S25, generating an execution logic dynamic expression of the checkpoint according to the data check range fragment and the data check logic fragment; S26, repeat S22-S25 until the execution logic dynamic expressions of all the checkpoints in the checklist are generated; S27, combining the checklist data range, merging the execution logic dynamic expressions of all checkpoints, and generating a checklist check expression; S28, repeat S21-S27 until the checklist check expressions of all the checklists to be checked are generated; S29, based on the generated checklist check expressions of all the checklists to be checked, executing the checklist check expressions of all the checklists to be checked in parallel, and obtaining the check results of all the checklists to be checked; The configuration of the inspection task includes: S11. Define inspection rules; S12. Select the checklist to be checked; S13, select the inspection items to be checked; S14. According to one of the inspection items to be checked, configure the inspection point through the mapping relationship between the inspection item and the parameters of the inspection rule; S15, repeat S13-S14 until all the checkpoints of the check items to be checked in the selected checklist are configured; S16. Repeat S12-S15 until all checkpoints of the checklist to be checked are configured.

2. The multi-rule data quality inspection method based on dynamic expression according to claim 1 is characterized in that: The check result processing logic is added to the execution logic dynamic expression of the checkpoint to monitor whether the checkpoint is triggered. If it is monitored that the checkpoint is triggered, the information of the check item and the trigger rule is spliced.

3. The multi-rule data quality inspection method based on dynamic expression according to claim 1 is characterized in that: The generating of the execution logic dynamic expression of the checkpoint includes generating a case when SQL execution expression as the execution logic dynamic expression of the checkpoint through the data check range fragment and the data check logic fragment.

4. The multi-rule data quality inspection method based on dynamic expression according to claim 1 is characterized in that: The inspection rules include basic information, execution logic, data range logic and inspection parameters.

5. A multi-rule data quality inspection system based on dynamic expressions, characterized in that: It includes an inspection task configuration module and an inspection task parsing and execution module; The inspection task configuration module is used to configure the inspection task; The inspection task parsing and execution module is used to parse and execute the inspection task on the inspection table to be checked, including: S21, classifying the inspection tasks according to each inspection sheet to be checked, selecting any inspection sheet from the inspection sheets to be checked, and generating an inspection sheet data range for the inspection sheet; S22, obtaining a checkpoint to be executed in the checklist; S23, according to the obtained checkpoint to be executed, according to the data range logic configured in the check rule of the checkpoint, and the mapping relationship of the check parameters, the checkpoint check range is replaced with parameters through the template engine to obtain a data check range fragment; S24, according to the obtained checkpoint to be executed, and according to the execution logic in the check rule of the checkpoint, the template engine is used to replace the check parameters to obtain the data check logic fragment of the checkpoint; S25, generating an execution logic dynamic expression of the checkpoint according to the data check range fragment and the data check logic fragment; S26, repeat S22-S25 until the execution logic dynamic expressions of all the checkpoints in the checklist are generated; S27, combining the checklist data range, merging the execution logic dynamic expressions of all checkpoints, and generating a checklist check expression; S28, repeat S21-S27 until the checklist check expressions of all the checklists to be checked are generated; S29, based on the generated checklist check expressions of all the checklists to be checked, executing the checklist check expressions of all the checklists to be checked in parallel, and obtaining the check results of all the checklists to be checked; The configuration of the inspection task includes: S11. Define inspection rules; S12. Select the checklist to be checked; S13, select the inspection items to be checked; S14. According to one of the inspection items to be checked, configure the inspection point through the mapping relationship between the inspection item and the parameters of the inspection rule; S15, repeat S13-S14 until all the checkpoints of the check items to be checked in the selected checklist are configured; S16. Repeat S12-S15 until all checkpoints of the checklist to be checked are configured.

6. The multi-rule data quality inspection system based on dynamic expression according to claim 5, characterized in that: The check result processing logic is added to the execution logic dynamic expression of the checkpoint to monitor whether the checkpoint is triggered. If it is monitored that the checkpoint is triggered, the information of the check item and the trigger rule is spliced.

7. The multi-rule data quality inspection system based on dynamic expression according to claim 5, characterized in that: The generating of the execution logic dynamic expression of the checkpoint includes generating a case when SQL execution expression as the execution logic dynamic expression of the checkpoint through the data check range fragment and the data check logic fragment.

8. The multi-rule data quality inspection system based on dynamic expression according to claim 5, characterized in that: The inspection rules include basic information, execution logic, data range logic and inspection parameters.

Images