Big data platform data quality abnormity reason positioning method and system

Abstract

The invention provides a big data platform data quality abnormity reason positioning method, and belongs to the field of data analysis. Comprising the following steps: analyzing a service DAG data flow, generating a DAG task dependency relationship, forming a workflow, and constructing a DAG job unified scheduling platform; generating a field-level data blood relationship; establishing an association relationship between DAG tasks and data consanguinity, and analyzing a data table and an operation type cited in each DAG task; recording an operation log and a running state of each DAG task node; when the data is abnormal, all the upstream dependency tables are traced through the data consanguinity, and the abnormal source is further determined after a suspicious abnormal source is positioned through the running state and the operation log of the DAG task node correspondingly associated with each upstream dependency table. According to the method, an incidence relation system of the DAG job link, the data consanguinity, the job running state and the job operation log is constructed, when a data quality problem occurs, related DAG jobs, data tables and operation logs can be rapidly associated, and abnormal links can be rapidly positioned.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a method and system for locating causes of abnormal data quality on a big data platform. Background Art

[0002] In today's era of rapid information development, big data platforms play a vital role across all industries. These platforms support decision-making by collecting, processing, and analyzing massive amounts of data. However, data quality issues frequently arise during the operation of big data platforms, such as empty tables, empty daily data increments, and data calculation errors, seriously impacting data accuracy and availability. These issues can arise from a variety of reasons, including but not limited to abnormal big data cluster scheduling, computing resource congestion, and job modifications. Especially when processing complex big data computing tasks, due to the numerous computing links involved and the complex data flow, quickly and accurately locating the source of data quality anomalies becomes crucial. This not only affects the timeliness and reliability of data but also directly impacts the results of data analysis and the accuracy of business decisions.

[0003] Traditional methods for troubleshooting data quality anomalies often rely on manual experience, examining each step one by one to locate the problem. This approach is not only time-consuming and labor-intensive, but also inefficient and prone to missing critical information. Furthermore, with the continuous growth of data volumes and the increasing complexity of computing tasks, traditional troubleshooting methods are increasingly unable to meet practical needs.

[0004] In the existing big data platform environment, the scheduling and management of computing jobs faces a series of challenges, mainly reflected in the following aspects:

[0005] Fragmented scheduling and management: Job scheduling relies heavily on scripted operations, which are spread across numerous servers. There is a lack of effective centralized management and a unified view. Different development teams or individuals independently schedule and maintain jobs according to their own habits and rules. This not only leads to fragmented management but also increases the overall complexity and maintenance difficulty of the system.

[0006] Lack of transparency in dependencies: The lack of visualization tools to directly display the DAG dependencies and data flows between jobs makes it extremely difficult to examine the execution paths and dependency chains of DAG jobs from a global perspective. This makes it difficult to quickly grasp the logical connections and data flows between jobs, hindering the system's understandability and maintainability.

[0007] Weak lineage tracking capabilities: The current system has significant deficiencies in tracking data lineage relationships and associating them with DAG jobs. When data quality issues arise, locating the source becomes extremely cumbersome. Developers must manually trace the source of data tables and check their upstream dependent tables one by one. This process not only requires a deep understanding of the business but also takes a significant amount of time to work backwards, resulting in low efficiency.

[0008] Lack of change monitoring: Tracking which DAG jobs execute specific data tables, their current status, and any recent modifications or changes to the data is difficult and time-consuming. Due to the lack of automated monitoring and auditing, developers are forced to rely on manual log queries and code reviews, which not only prolongs problem resolution time but also increases the risk of misjudgment.

[0009] The invention document, publication number CN117708103A, provides a method for tracing the cause of data anomalies based on data lineage. This method registers the transmission rules between parent and child tables in lineage management as record-level lineage relationships. When a data anomaly is discovered during a quality audit, the method automatically generates audit tasks for related records in the parent table based on the record-level lineage relationship, and then performs a quality audit on the parent table. The method continues recursively upward until the origin of the anomaly is found, and a snapshot of the anomaly record is saved for quality analysis and reporting. If poor data quality is discovered in a table during a data quality audit, the method can quickly query the anomaly cause based on the lineage chain to determine which table and data portion the problematic data originated from. However, the method still lacks a link between data lineage and DAG tasks, making it impossible to quickly locate anomaly DAG task nodes based on data lineage. Furthermore, it is difficult to promptly track which DAG jobs execute specific data tables, the current running status of these jobs, and whether any modifications or online changes have been made. This lack of change monitoring makes it difficult to quickly locate the source of data anomalies. Summary of the Invention

[0010] The technical problem to be solved by the present invention is how to quickly locate the source of the problem when data quality problems occur on a big data platform.

[0011] To solve the above technical problems, the present invention provides the following technical solution: a method for locating the cause of abnormal data quality on a big data platform, comprising the following steps:

[0012] S1: Analyze the business's DAG data flow, generate DAG task dependencies, and form workflows. A workflow contains multiple DAG task nodes, building a unified DAG job scheduling platform.

[0013] S2: Generate field-level data lineage relationships through SQL or Python scripts;

[0014] S3: Establishes the relationship between DAG tasks and data lineage, and analyzes the data tables and operation types referenced in each DAG task;

[0015] S4: Record the operation log and running status of each DAG task node;

[0016] S5: Provides a data quality anomaly query portal. When data anomalies are detected, enter the data table name and field name, trace all upstream dependent tables through data lineage relationships, and locate the suspected anomaly source through the running status and operation logs of the DAG task nodes associated with each upstream dependent table.

[0017] The present invention realizes centralized management of DAG jobs by constructing a unified DAG job scheduling platform. By establishing the association between data lineage and DAG tasks and recording the operation log and running status of each DAG task node, a comprehensive and efficient association system of DAG job links, data lineage, job running status and operation logs is constructed. When data quality problems occur, the system can track the upstream dependencies and downstream impacts of the data table based on the lineage relationship from the data field, and then track the specific DAG job based on the association between data lineage and DAG tasks, and further track the execution log of the DAG task, which can greatly shorten the time for problem investigation and quickly locate the source of the abnormality.

[0018] Preferably, in step S1, the construction of a unified DAG job scheduling platform includes: defining a workflow data table, defining a DAG task node data table of the workflow, and defining a DAG task node relationship data table of the workflow.

[0019] The present invention can improve the reliability and maintainability of the scheduling platform by structured storage of workflows, nodes and data relationships, and can support rapid query and analysis of complex dependency relationships.

[0020] Preferably, in step S2, generating field-level data lineage relationships includes: defining a data lineage relationship data table, and recording information including a source table, source table fields, a target table, and target table fields.

[0021] The present invention meticulously records field-level mapping relationships to ensure the accuracy of lineage tracing, and is particularly suitable for abnormal scenarios with multi-field associations.

[0022] Preferably, in step S3, establishing the association relationship between the DAG task and the data lineage includes: defining the relationship between the DAG task and the data table, and recording information including the node task and the corresponding target table.

[0023] The present invention can deeply integrate DAG tasks and data lineage, not only establishing the connection between DAG tasks and data tables, but also further tracing the dependent data tables of the data tables referenced in the DAG tasks through lineage relationships.

[0024] Preferably, in step S3, the operation type includes DDL and DML statement operations.

[0025] The present invention clarifies the operation type of a task on a data table and can quickly identify task behaviors that may cause abnormalities, including structural changes or data updates.

[0026] Preferably, in step S4, recording the operation log includes defining a DAG task operation log data table, and the recorded information includes the operator who modified or deleted the DAG task node, the operation time, and the task snapshot information.

[0027] Preferably, in step S4, recording the running status includes defining a workflow node task instance data table, and recording information includes task instance, node task, corresponding workflow and task execution status.

[0028] Preferably, the task execution status includes successful execution, failed execution and running.

[0029] By recording operation logs and operation status, the present invention can trace the operation history and analyze problems after building a correlation system of DAG operation links, data lineage, operation operation status and operation logs, thereby accelerating fault location and problem solving.

[0030] Preferably, the specific process of step S5 is: providing a data quality anomaly query entry, when the data is abnormal, inputting the data table name and field name, tracing back all upstream dependent tables through data lineage relationships, when the task execution status recorded in the running status of the DAG task node corresponding to each upstream dependent table has a run failure, or there is a modification record in the operation log of the DAG task node corresponding to each upstream dependent table, then mark the current DAG task node as a suspicious anomaly source, determine the workflow to which the suspicious anomaly source belongs, and the running status of the DAG task node it depends on, if the task execution status recorded in the running status of the dependent DAG task node is a run failure, further check the operation log of the dependent DAG task node, if there is a modification record, then mark it as an anomaly source.

[0031] The present invention also provides a system for locating the cause of abnormal data quality on a big data platform, comprising the following modules:

[0032] Build a unified DAG job scheduling platform module: This module is used to analyze the DAG data flow of the business, generate DAG task dependencies, and form workflows. A workflow contains multiple DAG task nodes, thus building a unified DAG job scheduling platform.

[0033] Generate blood relationship module: used to generate field-level data blood relationships through SQL or Python scripts;

[0034] The module for associating DAG tasks with data lineage is used to establish the relationship between DAG tasks and data lineage, and to analyze the data tables and operation types referenced in each DAG task.

[0035] Recording module: used to record the operation log and running status of each DAG task node;

[0036] Abnormal Source Determination Module: This module provides a data quality abnormality query entry. When data abnormalities occur, the data table name and field name are entered, and all upstream dependent tables are traced back through data lineage relationships. The operating status and operation logs of the DAG task nodes corresponding to each upstream dependent table are used to locate the suspected abnormality source and further determine the abnormality source.

[0037] Compared to existing technologies, this invention offers the advantage of building a system that integrates DAG job links, data lineage, job status, and job operation logs, building upon centralized DAG job management and data lineage analysis. This allows for rapid linkage to related DAG jobs, data tables, and operation logs when data quality issues arise, enabling rapid identification of anomalies and efficient problem resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of Example 1 of the present invention. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0040] Example 1

[0041] like Figure 1 As shown, this embodiment provides a method for locating the cause of abnormal data quality on a big data platform, including the following steps:

[0042] S1: Analyze the business's DAG data flow, generate DAG task dependencies, and form workflows. A workflow contains multiple DAG task nodes, building a unified DAG job scheduling platform.

[0043] The construction of the unified DAG job scheduling platform includes: defining a workflow data table, defining a DAG task node data table of the workflow, and defining a DAG task node relationship data table of the workflow.

[0044] In this embodiment, the workflow data table is as follows:

[0045] Field encoding Field Name process_id Workflow primary key process_code Workflow English Code process_name Workflow Name ..... Other business attributes

[0046] The DAG task node data table of the workflow is:

[0047] Field encoding Field Name task_id Node task primary key task_code Node task English code task_name Node task name process_id Workflow ..... Other business attributes

[0048] The DAG task node relationship data table of the workflow is:

[0049]

[0050]

[0051] S2: Use interpreters to translate SQL and Python development scripts, automatically identifying field-level metadata relationships and defining data lineage tables based on preset specifications.

[0052] In this embodiment, the data lineage relationship data table is:

[0053] Field encoding Field Name lineage_id Blood relationship primary key source_table_name Data Source Table source_table_col_name Data source table fields target_table_name Target table target_table_col_name Target table fields ..... Other business attributes

[0054] S3: Establishes the relationship between DAG tasks and data lineage, and parses the data tables referenced in each DAG task, including SQL tasks. Other tasks can be configured by performing invariant analysis. This requires only parsing DDL (Create, Alter, Drop, Truncate, Rename) and DML (Insert, Update, Delete, Call) statements—those that modify table structures or data—and defining the relationship between DAG tasks and data tables.

[0055] The relationship between DAG tasks and data tables in this embodiment is as follows:

[0056]

[0057]

[0058] S4: Record the operation log and running status of each DAG task node, specifically:

[0059] S401: Record the operation log of each DAG task node, including defining a DAG task operation log data table. The recorded information includes the operator who modified or deleted the DAG task node, the operation time, and the task snapshot information. The DAG task operation log data table defined in this embodiment is:

[0060] Field encoding Field Name operate_id Operation primary key operate_userId Operator ID operate_userName Operator Name process_id Workflow primary key task_id Node task primary key operate_time Operation time ..... Other business attributes

[0061] At the same time, the big data platform storage database is monitored. For all data metadata and manual modifications to business data, the system checks in real time whether the above binding relationships are designed. If so, the system also records them in the log.

[0062] S402: Record the running status of each DAG task node, including defining a workflow node task instance data table. The recorded information includes task instance, node task, workflow to which it belongs, and task execution status. The workflow node task instance data table defined in this embodiment is:

[0063]

[0064]

[0065] S5: Provides a data quality anomaly query portal and a visual interface. When data anomalies are detected, enter the data table name and field name. After clicking Query, the page first traces back all upstream dependent tables through data lineage relationships. At the same time, it checks for each upstream dependent table to see whether the DAG task node where the upstream dependent table is located is running normally and whether it has been modified recently. If the task execution status recorded in the running status of the DAG task node associated with each upstream dependent table indicates a run failure, or if there is a modification record in the operation log of the DAG task node associated with each upstream dependent table, the current DAG task node is marked as a suspected anomaly source and a red highlight is displayed.

[0066] Clicking a red icon will drill down to reveal the workflow to which the suspected anomaly source belongs, along with the running status of the DAG task nodes it depends on. If the task execution status recorded in the running status of the dependent DAG task node indicates a failure, a red highlight will be displayed. Clicking a red icon will reveal the operation log of the dependent DAG task node. If a task previously ran stably but failed this time and has a recent modification record, it will be marked as an anomaly source. A professional comparison function will then be provided to compare the differences between the snapshot and the current version of the job, allowing for quick troubleshooting and feedback to the editor, enabling timely task adjustments.

[0067] Based on the in-depth development and construction of a unified DAG job scheduling and management system and metadata management system, this invention aims to centrally manage and integrate all computing tasks and data lineage information to build a comprehensive and efficient system for the association of DAG job links, data lineage, job running status, and job operation logs. This system can quickly and accurately locate abnormal links, effectively solving the following core technical problems:

[0068] Constructing an analysis of the correlation between data lineage and DAG job tasks: To solve the problem of missing correlation between data lineage tracking and DAG job tasks, the present invention uses a metadata management system to deeply integrate data lineage analysis and DAG job execution information, and establishes a precise correlation between data tables, DAG jobs, and their running status. The establishment of this correlation relationship enables the rapid location of the source of data quality issues when they occur, tracking the upstream dependencies and downstream impacts of the data tables, and at the same time associating with specific DAG jobs and their execution logs, greatly shortening the time for troubleshooting and improving the efficiency and accuracy of data governance.

[0069] Correlation Analysis of Job Operation Logs: To quickly locate problems, this method correlates job operation logs with DAG job links, data lineage, and other information. By establishing a correlation between job operation logs and job scheduling and execution, we can trace job history and analyze problems, accelerating fault location and resolution.

[0070] In particular, the present invention achieves a deep association between data lineage and DAG job tasks: Traditional data lineage only focuses on the data source of the data table, that is, which data tables the data in the data table is associated with, and which fields in which data tables affect the data fields in the table, without knowing the generation process. The present invention not only establishes an association relationship between the data source, but also establishes a dependency relationship between the generation process of each data, thereby establishing a precise association between the data table, DAG job and its running status. The establishment of this association relationship not only solves the problem of missing association between data lineage tracking and DAG job tasks, but also provides strong support for the rapid location of data quality issues.

[0071] This invention also enables the construction and utilization of multi-dimensional relationships: Building on the centralized management of DAG jobs and data lineage analysis, it further establishes a relationship system for DAG job links, data lineage, job status, and job operation logs. This innovation enables rapid linkage to related DAG jobs, data tables, and operation logs when data quality issues arise, enabling rapid identification of anomalies and efficient problem resolution.

[0072] This invention also improves the stability and reliability of big data platforms: Through its technological innovations, the platform's capabilities in DAG job scheduling and data lineage management are significantly enhanced. This not only improves data processing efficiency and accuracy, but also enhances the platform's stability and reliability, providing stronger support for data-driven business decision-making.

[0073] Example 2

[0074] Corresponding to Example 1, this embodiment further provides a system for locating the cause of abnormal data quality on a big data platform, including the following modules:

[0075] Build a unified DAG job scheduling platform module: used to analyze the DAG data flow of the business, generate DAG task dependencies and form a workflow. A workflow contains multiple DAG task nodes.

[0076] This includes building a unified DAG job scheduling platform unit, which is used to define a workflow data table, a DAG task node data table for defining workflows, and a DAG task node relationship data table for defining workflows;

[0077] Generate Lineage Relationship Module: This module is used to translate SQL and Python development scripts through interpreters, automatically identify field-level metadata relationships, and define data lineage relationship tables based on preset specifications.

[0078] The DAG task and data lineage association module is used to establish the relationship between DAG tasks and data lineage, and parse the data tables referenced in each DAG task, including SQL tasks. Other tasks for invariant analysis can be configured. Only DDL (Create, Alter, Drop, Truncate, Rename) and DML (Insert, Update, Delete, Call) statements (that is, statements that modify table structure or table data) need to be parsed, and the relationship between DAG tasks and data tables needs to be defined.

[0079] Recording module: used to record the operation log and running status of each DAG task node, including the following units:

[0080] Operation log recording unit: used to record the operation log of each DAG task node, including defining the DAG task operation log data table. The recorded information includes the operator who modified or deleted the DAG task node, the operation time, and the task snapshot information.

[0081] Running status recording unit: used to record the running status of each DAG task node, including defining the workflow node task instance data table, recording information including task instance, node task, workflow to which it belongs, and task execution status;

[0082] Abnormal Source Determination Module: This module provides a data quality abnormality query entry and a visual interface. When data abnormalities occur, enter the data table name and field name. After clicking Query, the page first traces back all upstream dependent tables through data lineage relationships. At the same time, it checks for each upstream dependent table to see whether the DAG task node where each upstream dependent table is located is running normally and whether it has been modified recently. If the task execution status recorded in the running status of the DAG task node associated with each upstream dependent table indicates a run failure, or if there is a modification record in the operation log of the DAG task node associated with each upstream dependent table, the current DAG task node is marked as a suspected abnormality source and a red highlight is displayed.

[0083] Clicking a red icon will drill down to reveal the workflow to which the suspected anomaly source belongs, along with the running status of the DAG task nodes it depends on. If the task execution status recorded in the running status of the dependent DAG task node indicates a failure, a red highlight will be displayed. Clicking a red icon will reveal the operation log of the dependent DAG task node. If a task previously ran stably but failed this time and has a recent modification record, it will be marked as an anomaly source. A professional comparison function will then be provided to compare the differences between the snapshot and the current version of the job, allowing for quick troubleshooting and feedback to the editor, enabling timely task adjustments.

[0084] Through the operation and coordination of five modules, this embodiment establishes a comprehensive link between the generation, execution, and modification of DAG jobs and data storage. This enables centralized, visualized, and intelligent management of all computing tasks and data lineage relationships in the big data platform. This brings significant and far-reaching benefits, including the following three aspects:

[0085] Significantly improves problem location and resolution efficiency: Through the visual display of DAG job chains, developers can intuitively understand the dependencies between tasks. This allows them to quickly locate the source of problems when faced with data quality anomalies, significantly shortening the time period for problem identification and resolution and improving data processing efficiency.

[0086] Enhanced traceability of data lineage: By deeply integrating the metadata management system with the DAG job scheduling system, this invention achieves full-link tracking of data from generation to consumption, ensuring that every step of data flow can be accurately recorded and traced, providing a solid foundation for continuous monitoring and optimization of data quality.

[0087] Improve system stability and reliability: It can effectively identify and resolve data quality issues, reduce system failures and business interruptions caused by data anomalies, and enhance the stability and reliability of the entire big data platform.

[0088] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for locating the cause of abnormal data quality on a big data platform, characterized in that: The following steps are involved: S1: Analyze the business's DAG data flow, generate DAG task dependencies, and form workflows. A workflow contains multiple DAG task nodes, building a unified DAG job scheduling platform. S2: Generate field-level data lineage relationships through SQL or Python scripts; S3: Establishes the relationship between DAG tasks and data lineage, and analyzes the data tables and operation types referenced in each DAG task; S4: Record the operation log and running status of each DAG task node; S5: Provides a data quality anomaly query portal. When data anomalies are detected, enter the data table name and field name, trace all upstream dependent tables through data lineage relationships, and locate the suspected anomaly source through the running status and operation logs of the DAG task nodes associated with each upstream dependent table.

2. The method for locating the cause of abnormal data quality on a big data platform according to claim 1 is characterized in that: In step S1, the construction of a unified DAG job scheduling platform includes: defining a workflow data table, defining a DAG task node data table of the workflow, and defining a DAG task node relationship data table of the workflow.

3. The method for locating the cause of abnormal data quality on a big data platform according to claim 1, characterized in that: In step S2, generating field-level data lineage relationships includes: defining a data lineage relationship data table, and recording information including a source table, source table fields, a target table, and target table fields.

4. The method for locating the cause of abnormal data quality on a big data platform according to claim 3 is characterized in that: In step S3, establishing the association relationship between the DAG task and the data lineage includes: defining the relationship between the DAG task and the data table, and recording information including the node task and the corresponding target table.

5. The method for locating the cause of abnormal data quality on a big data platform according to claim 4 is characterized in that: In step S3, the operation types include DDL and DML statement operations.

6. The method for locating the cause of abnormal data quality on a big data platform according to claim 1, characterized in that: In step S4, recording the operation log includes defining a DAG task operation log data table, and the recorded information includes the operator who modified or deleted the DAG task node, the operation time, and the task snapshot information.

7. The method for locating the cause of abnormal data quality on a big data platform according to claim 6, characterized in that: In step S4, the recording of the running status includes defining a workflow node task instance data table, and the recorded information includes task instance, node task, belonging workflow and task execution status.

8. The method for locating the cause of abnormal data quality on a big data platform according to claim 7, characterized in that: The task execution status includes successful execution, failed execution, and running.

9. The method for locating the cause of abnormal data quality on a big data platform according to claim 8, characterized in that: The specific process of step S5 is as follows: provide a data quality anomaly query entry; when data anomalies occur, enter the data table name and field name; trace back all upstream dependent tables through data lineage relationships; when the task execution status recorded in the running status of the DAG task node associated with each upstream dependent table indicates a run failure, or there is a modification record in the operation log of the DAG task node associated with each upstream dependent table, mark the current DAG task node as a suspected anomaly source; determine the workflow to which the suspected anomaly source belongs, and the running status of the DAG task node to which it depends; if the task execution status recorded in the running status of the dependent DAG task node indicates a run failure, further check the operation log of the dependent DAG task node; if there is a modification record, mark it as an anomaly source.

10. A system for locating the cause of abnormal data quality on a big data platform, characterized in that: Includes the following modules: Build a unified DAG job scheduling platform module: This module is used to analyze the DAG data flow of the business, generate DAG task dependencies, and form workflows. A workflow contains multiple DAG task nodes, thus building a unified DAG job scheduling platform. Generate blood relationship module: used to generate field-level data blood relationships through SQL or Python scripts; The module for associating DAG tasks with data lineage is used to establish the relationship between DAG tasks and data lineage, and to analyze the data tables and operation types referenced in each DAG task. Recording module: used to record the operation log and running status of each DAG task node; Abnormal Source Determination Module: This module provides a data quality abnormality query entry. When data abnormalities occur, the data table name and field name are entered, and all upstream dependent tables are traced back through data lineage relationships. The operating status and operation logs of the DAG task nodes corresponding to each upstream dependent table are used to locate the suspected abnormality source and further determine the abnormality source.

Images