Bank data quality verification and tracking treatment method and system

Abstract

The invention discloses a bank data quality verification and tracking treatment method and system. The tracking treatment method comprises the steps of performing parameter configuration; functions are realized; and presenting a data quality result. According to the method, data item result accuracy checking is achieved, data quality influencing result accuracy can be rapidly positioned, the problem of data quality directly points to a corresponding system through a configuration table, and the condition of data quality is fed back to a data governance department for governance.

Description

Technical Field

[0001] The present invention relates to the field of data quality verification, and in particular to a bank data quality verification and tracking management method and system. Background Art

[0002] For data projects, the output report results and product run results are often inaccurate due to missing fields, inaccurate code values, unreasonable values, etc. in the source system data, which affects customer use;

[0003] When the system report results are abnormal, technical personnel are required to investigate. In order to quickly locate the cause of the inaccurate report, this verification method is invented for daily maintenance and data governance of data projects.

[0004] For data quality issues, it is usually easy to find problems only when there are large deviations in the issued reports. Then the report reasons are queried in sequence from report results to application layer results, to integration layer results, and finally to source layer data. This is very time-consuming to deal with the problems.

[0005] For data projects, the existing technology makes it difficult for business users to find errors when subtle errors appear in reports. When large differences occur, technical personnel can only query the specific causes of the problems layer by layer through the flow of data, which is very time-consuming. It is also impossible to quickly connect with the data governance department for the problems found. Summary of the invention

[0006] In view of the above problems, the present invention is proposed to provide a bank data quality verification and tracking management method and system that overcomes the above problems or at least partially solves the above problems.

[0007] According to one aspect of the present invention, a bank data quality verification and tracking management method is provided, and the tracking management method comprises:

[0008] Perform parameter configuration;

[0009] Function realization;

[0010] Data quality results are presented.

[0011] Optionally, the parameter configuration specifically includes:

[0012] Parameter definition description;

[0013] Parameter table structure design;

[0014] Parameter configuration is performed according to the parameter definition and the parameter table structure.

[0015] Optionally, the function implementation specifically includes:

[0016] Generate verification SQL based on the configuration splicing in the verification rule parameter table;

[0017] Execute SQL and insert the results into the verification result detail table CS_VALIDATA_RESULT. Summarize the data in the verification result detail table into the engine verification result-table-level summary information table CS_VALIDATE_TABLE according to the verification rules. Use reporting tools to develop reports based on the detail table and summary table, and display the data quality verification summary and details in the data quality result presentation.

[0018] Optionally, the data quality result presentation specifically includes:

[0019] Data quality check results;

[0020] Data quality tracking and governance.

[0021] Optionally, the parameter definition description specifically includes:

[0022] The tables, fields, effective range of field values, business scope of verification, and default values ​​to be checked for data quality are determined in advance and configured in the data table.

[0023] Optionally, the data quality check result specifically includes:

[0024] Design summary data quality presentation solutions and detailed data quality presentation solutions;

[0025] In the summary plan, it shows which data quality rules have problems, in which tables, and how many transactions have problems;

[0026] The detailed data quality presentation table will display specific business information to facilitate system maintenance personnel to use business details to communicate with upstream systems on data quality. The detailed quality presentation table will display fields with data quality issues.

[0027] Optionally, the data quality tracking and governance specifically includes:

[0028] Design reports to show the data quality of the past six months at one time, making it easier for system maintenance personnel to analyze changes in data quality;

[0029] Design reports to enable business personnel to understand the business scope affected by data quality issues and the status of data quality tracking and governance.

[0030] The present invention also provides a bank data quality verification and tracking management system, which uses the above-mentioned bank data quality verification and tracking management method, and the management system includes:

[0031] Parameter configuration module, used for parameter configuration;

[0032] Function realization module, used for function realization;

[0033] The data quality result presentation module is used to present data quality results.

[0034] The present invention provides a bank data quality verification and tracking management method and system, the tracking management method includes: parameter configuration; function realization; data quality result presentation. It can solve the problem of checking the accuracy of data project results, quickly locate the data quality that affects the accuracy of the results, directly point the data quality problems to the corresponding system through the configuration table, and feed back the data quality situation to the data management department for management.

[0035] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0037] Figure 1 A flow chart of a bank data quality verification and tracking management method provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the parameter table structure design provided for an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of parameter configuration provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of a summary data verification report provided by an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of a detailed data verification report provided by an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of a data governance report provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0044] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0045] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0046] like Figure 1 As shown, the present invention provides a bank data quality verification and tracking management method, and the tracking management method includes: parameter configuration; function implementation; data quality result presentation.

[0047] The method of data quality verification and tracking governance is to identify all upstream fields used through project business demand analysis and the needs of the measurement engine. In the absence of verification methods, field-level data quality is usually difficult to detect. This is an inspection method designed to prevent field-level data quality problems in the upstream from affecting product measurement and regulatory reporting.

[0048] The core of the design is to sort out the validation logic of all fields, such as the account number cannot be empty, the five-level classification cannot be empty and must be in the code value range of 301, 302, 303, 304, and 305. After all the validation rules are designed, all the rules are configured with parameters through the configuration table to facilitate program development more concisely and efficiently. In terms of program implementation, you only need to associate the table to be validated with the configuration table to realize the identification of data quality details and data aggregation.

[0049] Data quality verification is merely a means of identification. By presenting data quality details and summaries, the design of data quality tracking tables can effectively track and manage data with quality issues, thus forming a closed-loop management of identification and management.

[0050] Parameter configuration module

[0051] 1. Parameter setting instructions

[0052] To facilitate program development, the tables, fields, effective range of field values, business scope of verification, and default values ​​for data quality checks must be determined in advance and configured in the data table.

[0053] For example, the accuracy of the system's measurement results depends on the credit limit field of a single enterprise in the counterparty information table provided by the upstream system. If this field has a null value, the measurement result is inaccurate. To address this data quality issue, define

[0054] Verification rule name: The credit limit of a single enterprise cannot be empty

[0055] Rule Type: Null Value

[0056] Verification table name: Counterparty information table

[0057] Verification field name: Credit limit for a single enterprise

[0058] Effective scope: Corporate customers

[0059] Rule activation status: Enabled

[0060] Default value: 9990000

[0061] Source system: Credit system

[0062] The name of the validation rule is mainly to define the content of the check so that system users can quickly identify the content of the check; the name of the validation table and the name of the validation field are mainly to specify the target of the validation, and the rule type is to distinguish whether the validation is a null value or a value range, etc.; the effective range refers to a constraint on the data in the checked table, and only part of the data is validated, and unnecessary data is not validated; the purpose of setting the default value is mainly to set the default value when the upstream system data does not meet the requirements, so that the system can run the result; the configuration of the source system is presented in the data quality results, which is convenient for quickly locating which system has a problem.

[0063] Parameter table structure design

[0064] Table name: CS_VALIDATE_RULE, such as Figure 2 shown.

[0065] Parameter configuration, such as Figure 3 The following is an example of parameter configuration.

[0066] The functional implementation includes:

[0067] Generate verification SQL based on the configuration splicing in the verification rule parameter table;

[0068] Execute SQL and insert the results into the verification result detail table CS_VALIDATA_RESULT

[0069] Summarize the data in the validation result detail table into the engine validation result-table level summary information table CS_VALIDATE_TABLE according to the validation rules;

[0070] Use reporting tools to develop reports based on detail tables and summary tables, and display data quality verification summaries and details in the data quality results presentation.

[0071] Data quality result presentation module

[0072] Data quality inspection results: To facilitate system maintenance personnel and business personnel to quickly understand the data quality problems in the system, the solution has designed a summary data quality presentation solution and a detailed data quality presentation solution. The summary solution shows which data quality rules have problems, which tables have problems, and how many businesses have problems. The detailed data quality presentation table will display specific business information to facilitate system maintenance personnel to use the business details to communicate data quality with the upstream system. In addition, the detailed quality presentation table will also display fields with data quality problems. Specific examples are as follows:

[0073] Summary data verification report such as Figure 4 shown.

[0074] Detailed data verification report, such as Figure 5 shown.

[0075] Data quality tracking and management, the design report shows the data quality status of the past 6 months at one time, which is convenient for system maintenance personnel to analyze the changes in data quality. In addition, the design report can enable business personnel to understand the business scope affected by data quality issues and the situation of data quality tracking and management.

[0076] Data governance reports show that, for example Figure 6 shown.

[0077] A bank data quality verification and tracking management system, comprising:

[0078] Parameter configuration module: parameter definition description, parameter table structure design, parameter configuration;

[0079] Function realization module;

[0080] Data quality result presentation module: data quality inspection results, data quality tracking and governance.

[0081] Beneficial effects: It solves the problem of checking the accuracy of data project results, can quickly locate the data quality that affects the accuracy of the results, and directly points the data quality problems to the corresponding system through the configuration table, and then feeds back the data quality situation to the data governance department for governance.

[0082] Effectively solve the rapid positioning and data governance of data quality in data projects.

[0083] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A bank data quality verification and tracking management method, characterized in that: The tracking and governance method comprises: Perform parameter configuration; Function realization; Data quality results are presented.

2. A bank data quality verification and tracking management method according to claim 1, characterized in that: The parameter configuration specifically includes: Parameter definition description; Parameter table structure design; Parameter configuration is performed according to the parameter definition and the parameter table structure.

3. A bank data quality verification and tracking management method according to claim 1, characterized in that: The functional implementation specifically includes: Generate verification SQL based on the configuration splicing in the verification rule parameter table; Execute SQL and insert the results into the verification result detail table CS_VALIDATA_RESULT. Summarize the data in the verification result detail table into the engine verification result-table-level summary information table CS_VALIDATE_TABLE according to the verification rules. Use reporting tools to develop reports based on the detail table and summary table, and display the data quality verification summary and details in the data quality result presentation.

4. A bank data quality verification and tracking management method according to claim 1, characterized in that: The data quality results presentation specifically includes: Data quality check results; Data quality tracking and governance.

5. A bank data quality verification and tracking management method according to claim 2, characterized in that: The parameter definition description specifically includes: The tables, fields, effective range of field values, business scope of verification, and default values ​​to be checked for data quality are determined in advance and configured in the data table.

6. A bank data quality verification and tracking management method according to claim 4, characterized in that: The data quality check results specifically include: Design summary data quality presentation solutions and detailed data quality presentation solutions; In the summary plan, it shows which data quality rules have problems, in which tables, and how many transactions have problems; The detailed data quality presentation table will display specific business information to facilitate system maintenance personnel to use business details to communicate with upstream systems on data quality. The detailed quality presentation table will display fields with data quality issues.

7. A bank data quality verification and tracking management method according to claim 4, characterized in that: The data quality tracking and governance specifically includes: Design reports to show the data quality of the past six months at one time, making it easier for system maintenance personnel to analyze changes in data quality; Design reports to enable business personnel to understand the business scope affected by data quality issues and the status of data quality tracking and governance.

8. A bank data quality verification and tracking management system, using a bank data quality verification and tracking management method as described in any one of claims 1 to 7, characterized in that: The governance system includes: Parameter configuration module, used for parameter configuration; Function realization module, used for function realization; The data quality result presentation module is used to present data quality results.

Images