A data quality analysis method and system

Abstract

The embodiment of the present invention provides a data quality analysis method and system, including: constructing a data full-link map; determining the data to be checked, and filtering out the problem data that does not meet the preset data quality check rules from the data to be checked; searching for the associated problem data associated with the problem data according to the full-link map; correcting the problem data; and synchronously updating the associated problem data according to the corrected problem data. Through this technical solution, after discovering a problem data, the system can automatically update or supplement all related data according to the full-link map without repeated searches, which greatly improves work efficiency and effectively ensures the consistency of data modification.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality management, and in particular to a data quality analysis method and system. Background Art

[0002] Data quality management involves a variety of technologies, including data analysis and mining, determining data models and data, data quality metrics and indicators, data specification and verification, and management and monitoring. The combination of these technologies can support a comprehensive solution for data quality analysis, repair, and management. In the existing technology, there are some mature data quality management engine products that can integrate the above functions.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] In existing data quality management engines, although problematic data can be found according to preset rules, the data usually also appears in other data tables or databases outside the current location, so it is necessary to manually find and correct them one by one, which is very inefficient. Therefore, how to speed up the speed of finding and correcting problematic data is a problem that needs to be solved. Summary of the invention

[0005] The embodiments of the present invention provide a data quality analysis method and system to solve the problem of low efficiency in finding problems in existing data quality analysis work.

[0006] To achieve the above-mentioned purpose, on the one hand, an embodiment of the present invention provides a data quality analysis method, including: constructing a full-link data map; determining the data to be checked, and filtering out problem data that does not meet preset data quality check rules from the data to be checked; searching for related problem data associated with the problem data according to the full-link data map; correcting the problem data; and synchronously updating the related problem data according to the corrected problem data.

[0007] On the other hand, an embodiment of the present invention provides a data quality analysis system, including: a data full-link map construction module, used to construct a data full-link map; a data quality verification module, used to determine the data to be verified, and filter out problem data that does not meet preset data quality verification rules from the data to be verified; a problem correction module, used to correct the problem data; and a synchronous update module, used to synchronously update the associated problem data according to the corrected problem data.

[0008] At the same time, an embodiment of the present invention also provides a computer device, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the data quality analysis method as described above.

[0009] The above technical solution has the following beneficial effects:

[0010] In this technical solution, when problem data is found through preset data quality check rules, the system can search for upstream and downstream related data associated with the problem data according to the established data full-link map, thereby automatically finding all problem data; after correcting the problem data, the data flow processing is completed according to the data full-link map to achieve synchronous updating of all related problem data, without the need to search and correct one by one as in the prior art, thereby greatly improving work efficiency and ensuring the consistency of data modification.

[0011] In addition, the present invention also has the following characteristics:

[0012] This technical solution also supports the customization and scenario definition of data quality check rules. For example, users only need to enter the Chinese description of the check target in the preset input interface. The system will analyze the corresponding check rules based on the user input content and convert it into database language to update the existing data quality check rules, thereby meeting different user needs and effectively improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 is a flow chart of a data quality analysis method according to an embodiment of the present invention;

[0015] Figure 2 is a framework diagram of a data quality analysis system according to an embodiment of the present invention;

[0016] Figure 3 It is an architecture diagram of a specific embodiment of the data quality management engine of an embodiment of the present invention;

[0017] Figure 4 is a schematic diagram of searching for associated problem data in a specific embodiment of the present invention;

[0018] Figure 5It is a schematic diagram of an input interface in a specific embodiment of the present invention;

[0019] Figure 6 It is a schematic diagram of the table structure in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] like Figure 1 As shown, an embodiment of the present invention provides a data quality analysis method, including:

[0022] S101, build a full-link data map;

[0023] S102, determining the data to be checked, and screening out problematic data that does not meet preset data quality check rules from the data to be checked;

[0024] S103, searching for related problem data associated with the problem data according to the data full-link map;

[0025] S104, correct the problematic data;

[0026] S105. Synchronously update the associated question data according to the corrected question data.

[0027] To solve the above problems, this technical solution no longer sets different data quality inspection rules to find all problematic data one by one as in the prior art. Instead, a full-link data map is created at the same time when the database is constructed. When a problematic data is subsequently discovered, it is traced back through the full-link data map to find the upstream and downstream data directly related to the problematic data. In this way, all the related problematic data can be found at one time, greatly reducing the workload of data quality management personnel. Since there is a blood relationship between each data and the related data in the full-link data map, that is, the related data are all obtained by data flow processing starting from a certain data, in this way, as long as the problematic data is modified, all other related problematic data can be modified at the same time according to the full-link data map to achieve synchronous update.

[0028] Furthermore, step S103 specifically includes:

[0029] S1031. According to the data full-link map, all associated data associated with the problematic data are traced within the scope of database-level lineage, table-level lineage, and field-level lineage;

[0030] S1032. Filter out the associated data that do not meet the preset data quality check rules from all the associated data as associated problem data.

[0031] The problem data usually found are all fields in a certain worksheet. When searching for related problem data, in addition to searching in the table, you also need to search in different data tables with blood relationships and different databases with blood relationships.

[0032] Furthermore, step S101 specifically includes:

[0033] S1011. Obtaining blood relationship data for building a full-link data map;

[0034] S1012, analyzing the hierarchical relationship of blood relationship data;

[0035] S1013. Construct a full-link data map based on the hierarchical relationship.

[0036] In this technical solution, the blood relationship data of the data full-link map is a kind of relationship. The basic idea of ​​constructing the data full-link map is: first, obtain the SQL script corresponding to ETL (Extract-Transform-Load, which is used to describe the process of extracting, transforming, and loading data from the source to the destination) through collection, and parse the SQL in it through the blood relationship parsing tool. By parsing the ETL script, a data processing link corresponding to the database, table, and field is formed, that is, the full link, and the blood relationship between the table and the field can be seen by drilling, where the field is the smallest granularity.

[0037] Furthermore, before step S101, the method further includes:

[0038] S011. Develop custom input logic rules based on the table structure of the data table;

[0039] S012. Determine the input interface according to the custom input logic rule;

[0040] S013, obtaining user-defined input content through the input interface;

[0041] S014, parsing the user-defined input content to obtain a logical expression for the user-defined verification rule;

[0042] S015, converting the logical expression into an input instruction in a database language form;

[0043] S016. Complete user customization of data quality check rules by inputting instructions.

[0044] In common data quality management engines, there is only a single logic of discovering problematic data through quality rules. If the quality rules need to be adjusted, technical personnel need to readjust the program in the database language to update the quality rules. However, data quality management personnel are usually not familiar with programming and it is difficult for them to complete the adjustment work alone. Therefore, in order to facilitate the operation of data management personnel, this technical solution also designs a method that supports rule customization and scenario definition, that is, the user only needs to enter the content description for the verification target that meets the preset input logic rules (preferably by entering in a preset entry interface, or by selecting from a drop-down menu in the table of the entry interface. In this way, since the entry interface is designed based on the custom input logic rules, it has a certain constraint on user input, which can prevent the user's input from being too arbitrary and cannot be accurately identified). The system will automatically generate a logical expression for the user-defined verification rule based on the input content (that is, used to express the user's intention), and automatically generate SQL language code based on the logical expression, thereby realizing user customization of data quality verification rules and improving user usage scenarios.

[0045] Furthermore, the data quality check rules specifically include:

[0046] Data quality checking rules for data integrity check, data quality checking rules for data accuracy verification, data quality checking rules for data consistency check, and data quality checking rules for abnormal data identification are used to find problematic data that does not meet these checking rules.

[0047] like Figure 2 As shown, an embodiment of the present invention further provides a data quality analysis system, including:

[0048] A data full-link map construction module 21 is used to construct a data full-link map;

[0049] The data quality checking module 22 is used to determine the data to be checked and filter out problematic data that does not meet the preset data quality checking rules from the data to be checked;

[0050] The associated problem data search module 23 is used to search for associated problem data associated with the problem data according to the data full-link map;

[0051] A problem correction module 24, used to correct the problem data;

[0052] The synchronous updating module 25 is used to synchronously update the associated problem data according to the corrected problem data.

[0053] Furthermore, the data quality check module 22 is specifically used to: trace back all related data associated with the problem data within the scope of database-level lineage, table-level lineage and field-level lineage according to the data full-link map; and filter out the related data that does not meet the preset data quality check rules from all the related data as the related problem data.

[0054] Furthermore, the data full-link map construction module 21 is specifically used to: obtain blood relationship data used to construct the data full-link map; parse the hierarchical relationship of the blood relationship data; and construct the data full-link map according to the hierarchical relationship.

[0055] Furthermore, the data quality analysis system also includes a data quality check rule customization module 01, which is used to: formulate customized input logic rules according to the table structure of the data table; determine the input interface according to the customized input logic rules; obtain user-defined input content through the input interface; parse the user-defined input content to obtain a logical expression about the user-defined check rule; convert the logical expression into an input instruction in the form of a database language; and complete user customization of data quality check rules through input instructions.

[0056] The following is a detailed description of the technical solution of the embodiment of the present invention in conjunction with a specific application example in the banking field. Figure 3 As shown, it is a data quality management engine that adopts the above data quality analysis method. The data quality management engine receives the business-based data quality verification rules provided by the business department, the technology department and the branch bank, and automatically converts the business-based data quality verification rules into the technical-based data quality verification script rules; uses the converted technical-based data quality verification script rules to perform automated data quality checks on the simulated production environment data; automatically issues quality problem details and verification reports for the problem data; obtains the management personnel's review and confirmation results of the quality problem details and verification reports; automatically analyzes the problems and generates improvement plans based on the quality problem details and verification reports; and automatically adjusts the data problems at the source based on the details of the problem data. In the step of confirming the verification results, if it is found that there are problems with the details and verification reports of the problem data, feedback is given to the step of providing the business-based data quality verification rules or the step of converting the business-based data quality verification rules for reconfirmation.

[0057] Should Figure 3 The actual workflow of the data quality management engine of the example is as follows:

[0058] 1) Business departments, technology departments, and branches provide data quality verification rules for business purposes;

[0059] 2) Data quality management personnel formulate technical-caliber data quality verification rules based on business-caliber data quality verification rules and enter them into the data quality system; specifically, these business-caliber data quality verification rules are converted into technical-caliber verification script rules, which involves converting business logic and data standards into technical languages, such as SQL queries, programming scripts, etc., so that automated data quality checks can be implemented in the data quality system.

[0060] 3) Data quality management personnel conduct data quality checks on simulated production environment data migrated from the production environment; the production environment refers to the environment where the application or system is actually running, including hardware, software, network and other related components. In this production environment, users can access and use the application or system to perform actual tasks and operations. The production environment is rigorously tested and verified to ensure its stability, security and performance. Simulated production environment data refers to data copied or generated from the production environment for testing, verification or training purposes. This data is similar to the real data in the production environment, but may have been processed or modified to meet the needs of testing or verification.

[0061] It is not recommended to perform frequent data verification operations directly in the production environment, as this may interfere with actual user usage and may even cause data corruption or system crashes. Therefore, migrate the data in the production environment to an independent test environment for detailed verification and testing. The following is a detailed explanation:

[0062] Data migration: First, export the data that needs to be verified from the production environment and import it into the test environment. This ensures that the data in the test environment is consistent with the data in the production environment, making the test results more representative.

[0063] Verification in a test environment: In a test environment, you can perform various verification operations on the migrated data, such as data integrity check, performance test, security test, etc. Since the test environment is isolated from the production environment, these operations will not affect actual users.

[0064] Start QA: After completing data validation in the test environment, the QA process can be started. This includes a series of automated test scripts and tools to check the accuracy of the data and the stability of the system. If any problems are found during this process, they can be fixed without affecting users.

[0065] Re-verification: After fixing the issues found in the quality check, you need to verify it again in the test environment to ensure that all issues have been properly resolved.

[0066] Quality check in the production environment: After all verifications are completed in the test environment and confirmed to be problem-free, the updated code and data can be deployed to the production environment. In the production environment, a final quality check is also required to ensure stability and performance in actual use.

[0067] Through this process of migration first, testing second, and deployment last, the risk of problems in the production environment can be minimized, thereby ensuring the stability of the system and the security of the data.

[0068] 4) Issue quality problem details and verification report of problematic data based on data quality verification tools;

[0069] This data quality check tool first performs preliminary screening and identification of data based on business caliber when working. Business caliber refers to data standards or specifications related to specific business logic, business rules and business requirements.

[0070] Specifically, data quality checking tools first use business criteria to identify which data is relevant to a specific business or which data may have problems. For example, based on the definition and rules of the business, some data may be considered abnormal or non-compliant.

[0071] Then, this data quality check tool will "translate" the problematic data identified based on the business caliber into the technical caliber. The technical caliber is the data standard and specification defined from a technical perspective, which is closely related to the underlying data structure, data format, data type and other technical details.

[0072] By translating into technical terms, data quality verification tools can more deeply verify and identify problematic data from a technical perspective. This includes checking the integrity, accuracy, consistency, and other aspects of the data, as well as checking whether there are technical issues such as data redundancy, data missing, and data format errors.

[0073] In general, this data quality verification tool combines the advantages of business and technical perspectives. It first screens and identifies data from a business perspective, and then conducts in-depth problem data verification from a technical perspective to ensure that the data is comprehensive and accurate.

[0074] 5) Business department personnel and data quality management personnel confirm whether the verification results are correct based on the details of the problem data and the verification report; if correct, the business department personnel check whether the previous business rules are consistent with the problem data given by the tool. If they are consistent, then these verification rules are compliant and can be entered into the production environment for verification; if incorrect, then it is necessary to adjust the business scope description or optimize the technical scope and re-verify the data with quality problems.

[0075] 6) Data quality management personnel analyze problems and organize improvement plans or suggestions;

[0076] First, identify the specific issues found during the data quality verification process. This may include issues with data accuracy, completeness, consistency, timeliness, etc. The issues are classified and summarized by automatically identifying the details of the problematic data and the verification report.

[0077] Then, for each identified problem, an analysis is automatically performed to find out its root cause. This may include technical reasons (such as system failure, data processing errors, etc.) and business reasons (such as unreasonable business processes, irregular data entry, etc.).

[0078] Then, according to the nature and root cause of the problem, corresponding solutions are automatically formulated. This includes technical solutions (such as fixing system errors, optimizing data processing processes, etc.) and business solutions (such as improving business processes, improving data entry quality, etc.).

[0079] Finally, for each solution, develop a detailed implementation plan, including specific steps, timelines, required resources, and expected outcomes.

[0080] 7) If business department personnel and data quality management personnel find problems with the details and verification reports of the problem data, they will feedback to link 1 and link 2 to reconfirm the business and technical calibers; the above problems include: incorrect data format, inconsistent data, non-compliance with the value range, duplicate data, incorrect data, foreign key reference errors, and any one or more of non-compliance with business rules.

[0081] 8) Business department personnel adjust data problems at the source (the source refers to the earliest data with problems found according to the data full-link map, which generally points to the business department) according to the details of the problem data. The data quality management engine of this specific embodiment can realize the following functions:

[0082] Data analysis and mining: Use data analysis and mining techniques to identify data quality issues. This includes techniques such as statistical analysis of data, anomaly detection, pattern recognition, and data mining to find errors, omissions, duplications, or inconsistencies in the data.

[0083] Determine the data model and data: Build a data model and initialize the source data records through data deduplication, merging, format conversion, etc., or directly use the source data records.

[0084] Data quality metrics and indicators: Use data quality metrics and indicators to assess the quality level of data. This includes defining and calculating data quality indicators such as accuracy, completeness, consistency, uniqueness, and timeliness to quantify data quality and compare and monitor it.

[0085] Data specification and validation: Use data specification and validation techniques to ensure that data meets predefined specifications and standards. This includes defining data specifications, developing data validation rules and constraints, and using validation algorithms and logic checks to verify the format, scope, uniqueness, and consistency of data. Here are some examples of validation algorithms and logic:

[0086] Format validation: This validation method is used to ensure that the data conforms to a specific format requirement. For example, for a phone number, you can write a regular expression to validate that the input follows a specific phone number format.

[0087] Range validation: This type of validation is used to ensure that data falls within a predetermined range. For example, a field may require that the value entered be between 1 and 100.

[0088] Uniqueness validation: This validation is used to ensure that the values ​​in the data set are unique and there are no duplicates. For example, in a database, a user ID must be unique and there cannot be two identical user IDs.

[0089] Consistency validation: This validation is used to ensure that data is consistent across multiple fields or tables. For example, in an order processing system, the order total must be consistent with the total amount of the order items.

[0090] Existence validation: This validation is used to ensure that the referenced data exists. For example, in an order processing system, when creating a new order item, it is necessary to verify whether the referenced product ID exists in the product table.

[0091] Dependency validation: This validation is used to ensure that data meets specific business rules or dependencies. For example, in an employee management system, an employee's start date cannot be later than the end date.

[0092] Custom Validation: When the predefined validation rules do not meet your specific needs, you can write custom validation logic. This can include complex business rules or domain-specific rules.

[0093] Management and monitoring: Supports the management, tracking and monitoring of data quality. It can provide data quality reports, alerts, dashboards and visualizations to help users understand data quality status and changes in real time.

[0094] The outstanding feature of this data quality management engine is that when a problem is found in the data corresponding to a field in a table, combined with the pre-built data full-link map, it can quickly find the upstream and downstream data fields that are directly related to the current problem data. This method allows users to no longer repeatedly sort out the verification rules and associate upstream and downstream data. In other words, through a point problem, find the source and destination of the data.

[0095] The data quality management engine finds problems mainly in data integrity check, data accuracy verification, data consistency check, data change monitoring, and abnormal data identification, specifically:

[0096] 1) Data integrity check: By tracing the relationship of data in the data full-link map, it is possible to check whether the data has gone through the expected processing process and whether there is any data loss or omission. If the data is interrupted or missing on the data full-link map, it can be considered that there is a data integrity problem.

[0097] 2) Data accuracy verification: By working backwards along the data full-link map, we can analyze whether the data has undergone the correct conversion and calculation steps during the processing. If errors, anomalies, or inconsistencies are found in the data full-link map, data accuracy issues can be identified.

[0098] 3) Data consistency check: Through the data full link map, you can compare the data in different links and verify whether they are consistent. If you find that the data is inconsistent or contradictory on the data full link map, you can determine that there is a data consistency problem.

[0099] 4) Data change monitoring: Through the data full-link map, you can track the data change process and identify data modification, update or deletion operations. You can check whether the data has been changed as expected and verify the accuracy and completeness of the change operation. If the data full-link map shows abnormal changes or unauthorized modifications, data change problems can be discovered.

[0100] 5) Abnormal data identification: By tracing along the data full-link map, abnormal values, missing values, duplicate values, etc. of the data can be monitored, and the data processing link that causes the abnormality can be located. This helps to detect and handle abnormal data early and improve data quality.

[0101] Specifically, for how to implement the aforementioned "searching for associated problem data associated with the problem data according to the data full-link map" and "correcting the problem data and synchronously updating the associated problem data according to the corrected problem data" in this specific embodiment, the specific process can be found in Figure 4 :The rectangular box in the figure refers to the table name and field name where data is stored in the database. Implementation refers to the data full-link map formed by parsing ETL. The dotted arrow refers to finding all fields related to this field based on a certain point (for example, a field of problem data) based on the data full-link map. For example, now in the account table, it is found that the data corresponding to an ID card number does not meet the data quality verification rules and there is a problem (for example, the ID card number does not meet 18 digits, or contains illegal characters, or the last check digit is incorrect, or the birthday information is wrong). Then, the problem data corresponding to the table and field related to the ID card number in the account table can be found through the data full-link map. For example, if the ID card number is in the account table, user table, legal person table, and summary detail table, then find them all (in the actual process, the field name may have changed, but the link relationship is there, and it can also be found, such as the name in the user table is called the name, and in the legal person table it can be called the legal representative name, etc.). In this way, users no longer need to check the data in the user table, legal person table, and summary detail table. At the same time, if it is also possible to base it on the data's very beginning point, such as the ID number in the user table, when the user modifies the corresponding problem data (ID number) in the user table, the system can also start scheduling based on the ETL script to complete the data flow processing and realize the synchronous update of data from the user table to the account table, legal person table, and summary detail table. In other words, after adjusting the data at the source, the data can be updated synchronously, thereby greatly improving work efficiency.

[0102] Specifically, as to how to implement the aforementioned "user customization of data quality check rules" in this specific embodiment, the idea in this technical solution is to parse the content based on the text filled in by the user (comments on tables and fields in the database) and the calculations between the texts, and map the parsed content to the SQL script language that the database can recognize. In short, based on the predetermined quality check grammar (or custom input logic rules), the text content that the user can understand is parsed into the SQL script language that the computer can recognize. The predetermined quality check grammar can support the following capabilities:

[0103] 1) Expression parsing: Able to parse the input mathematical expression and convert it into an executable computing language;

[0104] 2) Operator support: Supports various mathematical operators, such as addition, subtraction, multiplication, division, remainder, etc.

[0105] 3) Variable and function support: allows the definition and use of variables and functions for calculation in expressions;

[0106] 4) Data type support: supports common data types such as integers, floating-point numbers, Boolean values, etc.

[0107] For the detailed process of user-defined data quality check rules, please refer to Figure 5 , Figure 6 An example is shown: Users can Figure 5 The table shown in the figure (one of the input interfaces) is used for custom input (or operation such as selection through a preset drop-down menu is performed), and the bank information table (the table name stored in the database is bank_info) is used for storing comments and field names in the database as shown in the figure. Figure 6 As shown, now Figure 5 Taking the second row of data in the table as an example, it can be understood that the "Bank Address" of the "Bank Information Table" cannot be empty, and the prerequisite for it to be empty is "Postal Code! = '450000'"; then, according to the preset quality check syntax, combined with Figure 5 and Figure 6 The content of the text ["Bank address" of the "Bank information table" cannot be empty, and the prerequisite for being empty is "Postal code! = '450000'"] can be converted into a database script SQL that can be executed by the computer. The SQL is [select id, bank_name, bank_addr, post_no from bank_info where post_no! = '450000' and bank_name is null]. Finally, when applied, the problem data with a postal code that is not 450000 and a bank address that is empty can be found by executing the parsed SQL.

[0108] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0109] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data quality analysis method, characterized in that: include: Build a full-link data map; Determine the data to be checked, and filter out problematic data that does not meet preset data quality check rules from the data to be checked; Searching for associated problem data associated with the problem data according to the data full-link map; Correct the problematic data; Synchronously updating the associated problem data according to the corrected problem data; Wherein, searching for associated problem data associated with the problem data according to the data full-link map includes: According to the data full-link map, all associated data associated with the problematic data are traced back within the scope of database-level lineage, table-level lineage, and field-level lineage; Filter out the associated data that do not meet the preset data quality check rules from all the associated data as associated problem data; Before constructing the data full-link map, it also includes: Create custom input logic rules based on the table structure of the data table; Determine the input interface according to the custom input logic rule; Obtaining user-defined input content through the input interface; Parsing the user-defined input content to obtain a logical expression for the user-defined checking rule; Converting the logical expression into an input instruction in a database language form; The user customization of the data quality check rules is completed through the input instructions.

2. The data quality analysis method according to claim 1, characterized in that: The construction of the data full-link map includes: Obtain blood relationship data for building a full-link data map; Analyzing the hierarchical relationship of the blood relationship data; The data full-link map is constructed according to the hierarchical relationship.

3. The data quality analysis method according to claim 1, characterized in that: The data quality check rules specifically include: Data quality checking rules for implementing data integrity checks, data quality checking rules for implementing data accuracy verification, data quality checking rules for implementing data consistency checks, and data quality checking rules for implementing abnormal data identification.

4. A data quality analysis system, characterized in that: include: Data full-link map construction module, used to build data full-link map; A data quality check module is used to determine the data to be checked and filter out problematic data that does not meet preset data quality check rules from the data to be checked; A related problem data search module, used to search for related problem data associated with the problem data according to the data full-link map; A problem correction module, used for correcting the problem data; A synchronous updating module, used for synchronously updating the associated problem data according to the corrected problem data; The data quality checking module is specifically used to: trace back all associated data associated with the problematic data within the scope of database-level kinship, table-level kinship, and field-level kinship, respectively, according to the data full-link map; Filter out the associated data that do not meet the preset data quality check rules from all the associated data as associated problem data; The data quality analysis system also includes a data quality check rule customization module, which is used to: formulate customized input logic rules based on the table structure of the data table; determine the input interface based on the customized input logic rules; obtain user-defined input content through the input interface; parse the user-defined input content to obtain a logical expression for the user-defined check rule; convert the logical expression into an input instruction in the form of a database language; and complete the user customization of the data quality check rule through the input instruction.

5. The data quality analysis system according to claim 4, characterized in that: The data full-link map construction module is specifically used to: obtain blood relationship data used to construct the data full-link map; parse the hierarchical relationship of the blood relationship data; and construct the data full-link map according to the hierarchical relationship.

6. A computer device, characterized in that: It includes: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data quality analysis method according to any one of claims 1 to 3.

Images