Self-healing data governance intelligent agent control system based on large language model

Abstract

The application discloses a self-healing data governance intelligent agent control system based on a large language model, relates to the technical field of semantic retrieval, and comprises an abnormality detection and reporting module, a root cause reasoning context construction module, a root cause reasoning module, a repair scheme generation and dispatching module and a repair verification and closed loop module.The abnormality detection and reporting module generates a structured abnormality event when data quality inspection finds data abnormality.The root cause reasoning context construction module searches upstream node information of each level along a blood relationship graph from an abnormality object identifier as a starting point, searches change records of the abnormality object and within a preset time window, and obtains root cause reasoning context.The root cause reasoning module inputs the root cause reasoning context into a large language model to obtain abnormality root cause positioning results.The repair scheme generation and dispatching module inputs the abnormality root cause positioning results into the large language model to generate a repair scheme and perform a repair operation.The repair verification and closed loop module re-executes quality inspection on the abnormality object after the repair operation is completed.The technical problem that the data governance cycle is too long due to reliance on manual troubleshooting in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of semantic retrieval technology, and more specifically to a self-healing data governance intelligent agent control system based on a large language model. Background Technology

[0002] Data quality inspection, as a crucial component of the data governance system, automates the detection of field-level indicators in data tables by setting preset quality thresholds, and can trigger alarm mechanisms when data anomalies occur. However, existing data governance systems have significant automation gaps between anomaly detection and anomaly repair.

[0003] In traditional data governance systems, alarms can be automatically generated when data quality checks detect data anomalies. However, the root cause reasoning process between anomaly detection and anomaly repair relies on manual intervention. This requires manual review of metadata lineage diagrams to trace upstream nodes, retrieve relevant change records, analyze the causes of anomalies, and formulate repair plans. As a result, the manual investigation process is time-consuming and depends on personal experience. Consequently, data governance cannot form a fully automated closed-loop control from anomaly perception, root cause location, repair execution, to repair verification, which seriously affects the timeliness and scalability of data governance.

[0004] Therefore, an intelligent agent control system that can integrate anomaly detection and anomaly repair processes to achieve self-healing data governance is needed. Summary of the Invention

[0005] This invention provides a self-healing intelligent control system for data governance based on a large language model, which solves the technical problem that the data governance cycle is too long due to reliance on manual investigation in the prior art.

[0006] This invention provides a self-healing data governance intelligent agent control system based on a large language model, the system comprising: The anomaly detection and reporting module is used to generate structured anomaly events containing anomaly object identifiers, anomaly types, and anomaly representation values ​​when data quality checks detect data anomalies. The root cause reasoning context construction module is used to retrieve upstream node information at each level along the lineage graph starting from the abnormal object identifier, retrieve change records of the abnormal object and related upstream nodes within a preset time window, and assemble the structured abnormal event, the upstream node information at each level, and the change records into a root cause reasoning context. The root cause reasoning module is used to input the root cause reasoning context into the large language model and output the abnormal root cause localization result. The repair solution generation and dispatch module is used to input the abnormal root cause localization results into the large language model, generate a repair solution, and perform repair operations based on the repair solution; The repair verification and closed-loop module is used to re-perform quality checks on the abnormal object after the repair operation is completed. If the check fails, the abnormal detection and reporting module is triggered again to enter the next round of self-healing loop.

[0007] One or more technical solutions provided in this invention have at least the following technical effects or advantages: This invention provides a self-healing data governance intelligent agent control system based on a large language model. First, the anomaly detection and reporting module transforms the original inspection results into structured anomaly events, providing a unified processing foundation for subsequent stages. Second, the root cause reasoning context construction module uses lineage graph tracing and change record retrieval to extract anomaly object identifiers, shortening information collection time. Third, the root cause reasoning module utilizes the semantic understanding capabilities of the large language model for root cause reasoning, quantifying the reasoning effect through reasoning confidence scoring, avoiding the subjectivity and uncertainty of manual judgment. Further, the repair plan generation and dispatch module inputs the anomaly root cause location results into the large language model to generate repair plans and executes repair operations based on these plans, eliminating the time delay of manual repair plans. Finally, the repair verification and closed-loop module ensures the reliability of the repair effect and provides automatic retry capability in case of repair failure through re-checking and cyclic triggering mechanisms.

[0008] In summary, this approach achieves closed-loop control of data governance from anomaly detection to anomaly elimination, reducing the frequency of manual intervention and shortening the cycle from data anomaly discovery to repair. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the structure of the self-healing data governance intelligent agent control system based on a large language model provided by the present invention; Figure 2 This is a logical schematic diagram of the self-healing data governance intelligent agent control system based on a large language model provided by the present invention.

[0010] In the attached diagram, the components represented by each number are as follows: Anomaly detection and reporting module 11; Root cause reasoning context construction module 12; Root cause reasoning module 13; Repair scheme generation and dispatch module 14; Repair verification and closure module 15. Detailed Implementation

[0011] This invention provides a self-healing intelligent control system for data governance based on a large language model, which solves the technical problem that the data governance cycle is too long due to reliance on manual investigation in the prior art.

[0012] The present invention will now be described in detail with reference to the accompanying drawings.

[0013] Example 1, as Figure 1 , Figure 2 As shown, the present invention relates to a self-healing data governance intelligent agent control system based on a large language model, the system comprising: The anomaly detection and reporting module 11 is used to generate a structured anomaly event containing an anomaly object identifier, anomaly type, and anomaly representation value when data quality inspection finds data anomalies. In this embodiment of the application, data quality inspection refers to the process of automatically detecting the data integrity, consistency, validity and other quality dimensions at the field level in the data table according to the preset data quality assessment rules; structured anomaly events refer to standardized data records formed by organizing anomaly-related information according to a predefined data structure.

[0014] Specifically, the anomaly detection and reporting module is configured to continuously monitor or proactively execute data quality checks. When the data quality check result indicates that a certain quality indicator of a data object deviates from the normal range, it is determined that a data anomaly has been detected. After a data anomaly is detected, a structured anomaly event is generated. This structured anomaly event is used to encapsulate the scattered anomaly detection results into a unified format that can be processed programmatically by subsequent modules.

[0015] In this embodiment of the application, the anomaly detection and reporting module is specifically used for: Data quality inspection tasks are triggered periodically according to a preset inspection cycle. For each field of each data table within the inspection scope, the null value rate, duplication rate, value range compliance rate, and format compliance rate of the field under inspection are calculated in the current inspection cycle. Compare any one of the actual detected values ​​of the null value rate, the duplication rate, the value range compliance rate, and the format compliance rate with the corresponding preset quality threshold. When any one of the actual detected values ​​exceeds the corresponding preset quality threshold, it is determined that data anomalies have been found. A structured exception event is generated, which includes an exception object identifier, exception type, exception characterization value, exception discovery timestamp, and current self-healing cycle identifier. The exception object identifier is a combination of the data table name and field name where the data exception occurred. The exception type is the name of the indicator type that triggers the preset quality threshold. The exception characterization value is the actual detection value that triggers the preset quality threshold. The exception discovery timestamp is the time point when the current quality check task is executed. The current self-healing cycle identifier is a unique identifier assigned by the exception detection and reporting module when generating the structured exception event.

[0016] In this embodiment, a preset inspection cycle is first set. The preset inspection cycle is a pre-configured time interval parameter used to control the triggering frequency of the quality inspection task. Its specific value can be configured by the data governance administrator according to the business requirements for data timeliness; for example, it can be set to trigger once every hour, once every 6 hours, or once every morning. The data quality inspection task is then triggered periodically according to the preset inspection cycle.

[0017] After the audit task is triggered, the audit scope is determined. The audit scope refers to the set of data objects covered by this quality audit task. This can be pre-configured by the data governance administrator, such as specifying all data tables in a specific database instance, all data tables under a specific data domain, or specific core business data tables. For each audited field in each data table within the audit scope, the null value rate, duplication rate, value range compliance rate, and format compliance rate of the audited field within the current audit period are calculated. The audited field refers to the column field in the data table that is included in the quality monitoring scope; typically, all business fields in the data table can be used as audited fields.

[0018] Specifically, the null value rate is the proportion of records with null values ​​in the checked field out of the total number of records in that field; the duplication rate is the proportion of records with identical values ​​in the checked field out of the total number of records in that field; the value range compliance rate is the proportion of records with values ​​in the checked field falling within a preset legal value range out of the total number of records in that field; and the format compliance rate is the proportion of records with values ​​in the checked field conforming to preset format regular expression rules out of the total number of records in that field. These four indicators are used to quantitatively evaluate data quality from four dimensions: data integrity, uniqueness, validity, and format compliance.

[0019] Secondly, for each field being inspected, the measured values ​​of the aforementioned four indicators are obtained. Simultaneously, the preset quality threshold corresponding to that field is read from internal storage.

[0020] Specifically, preset quality thresholds are normal range boundary values ​​pre-set for each indicator type. Their values ​​primarily depend on the patterns of business data and the system's fault tolerance. These thresholds can be pre-configured by those skilled in the art for specific application scenarios and can be dynamically adjusted according to different verification cycles or changes in data sources. The principle behind their setting is to objectively reflect the maximum allowable proportion of data anomalies for the field during normal business operations, balancing verification sensitivity with the avoidance of false alarms. For example, preset quality thresholds can be set as follows: no more than 5% for the null value rate, no more than 3% for the duplication rate, no less than 95% for the value range compliance rate, and no less than 90% for the format compliance rate. The direction of the preset quality thresholds may differ for different types of indicators: for the null value rate and duplication rate, exceeding the preset quality threshold means the actual detected value is greater than the preset quality threshold; for the value range compliance rate and format compliance rate, exceeding the preset quality threshold means the actual detected value is less than the preset quality threshold. The actual detected value of any one of the null value rate, duplication rate, value range compliance rate, and format compliance rate is compared with its corresponding preset quality threshold. When any actual detected value exceeds its corresponding preset quality threshold, a data anomaly is determined.

[0021] Finally, after identifying a data anomaly, a structured anomaly event is generated, containing an anomaly object identifier, anomaly type, anomaly representation value, anomaly discovery timestamp, and a self-healing loop identifier. The anomaly object identifier is a combination of the field name and table name where the data anomaly occurred, uniquely locating the field-level data object in the entire data governance system. The anomaly type is the name of the indicator type that triggered the preset quality threshold; that is, the type name corresponding to the indicator whose actual detected value exceeds the threshold. For example, if the triggering indicator is the null value rate, the anomaly type is null value rate. If multiple indicators exceed the threshold simultaneously, the indicator type with the largest exceedance can be selected as the anomaly type, or a combination of multiple indicator names can be used. The anomaly representation value is the actual detected value that triggered the preset quality threshold. The anomaly discovery timestamp is the time point when this quality check task was executed. The self-healing loop identifier is a unique identifier assigned by the anomaly detection and reporting module when generating the structured anomaly event, linking this anomaly event to all subsequent related processing steps.

[0022] In this embodiment of the application, for each field of each data table within the scope of the inspection, the null value rate, duplication rate, value range compliance rate, and format compliance rate of the field under inspection in the current inspection period are calculated, including: The total number of all sampled records for the field being inspected within the current inspection period is counted and used as the total number of sampled records. The number of records in the current inspection period where the field being inspected has a null value is counted. The null value rate is obtained by dividing the number of records with null values ​​by the total number of sampled records. The number of records with identical field values ​​in all sampled records within the current inspection period is counted, and the duplication rate is obtained by dividing the number of records with identical field values ​​by the total number of sampled records. The number of records in the current inspection period whose field values ​​fall within the preset value range is counted. The value range compliance rate is obtained by dividing the number of records whose field values ​​fall within the preset value range by the total number of sampled records. The number of records in the current inspection period whose field values ​​conform to a preset regular expression is counted. The number of records whose field values ​​conform to the preset regular expression is divided by the total number of sampled records to obtain the format compliance rate.

[0023] In this embodiment, firstly, the total number of all sampled records for the field being checked within the current checking period is counted, which is taken as the total number of sampled records. The total number of sampled records, which is the total number of all sampled records within the current checking period, serves as the unified denominator for the four indicators.

[0024] Secondly, count the number of records with null values ​​in the field being checked within the current checking period. Divide this number by the total number of samples to obtain the null value rate. The null value rate is calculated as: Null Value Rate = Number of records with null values ​​ / Total number of samples. A higher null value rate indicates poorer data integrity for that field and a more severe problem with missing values.

[0025] Next, count the number of records with identical field values ​​in all sampled records within the current inspection period. Divide this number by the total number of sampled records to obtain the duplication rate. The number of records with identical field values ​​refers to the total frequency of a specific value for that field across all sampled records. If the frequency of a value is greater than 1, the number of duplicate records corresponding to that value is the frequency value.

[0026] The calculation formula is: Duplicate rate = Number of records with identical field values ​​ / Total number of sampled records. The duplicate rate reflects the proportion of redundant and duplicate data in that field. An excessively high duplicate rate may indicate that unexpected data duplication occurred during data entry or data integration.

[0027] Furthermore, the number of records in the current inspection period where the field value falls within the preset value range is counted. This number is then divided by the total number of sampled records to obtain the value range compliance rate. Specifically, the value range compliance rate = number of records where the field value falls within the preset value range / total number of sampled records. The value range compliance rate reflects the validity of the field value at the business semantic level; a low value range compliance rate indicates the presence of a large amount of invalid data that does not conform to business rules.

[0028] Specifically, the preset value range refers to the legal value boundaries predefined by the data governance administrator for the field being checked, based on specific business rules and established data standards. It can take the form of a continuous numerical interval or a discrete enumerated set of values. When calculating the value range compliance rate, each sampled record for that field within the current check period is extracted, and its field value is compared with the preset value range: for continuous intervals, it is determined whether the field value is greater than or equal to the lower limit of the interval and less than or equal to the upper limit; for discrete enumerated sets, it is determined whether the field value is an element in the set. The number of records determined to fall within the preset value range is accumulated to obtain the total number of records whose field values ​​fall within the preset value range. This number is then divided by the total number of sampled records in the current check period, and the resulting quotient is the value range compliance rate for that field.

[0029] A higher value for the value range compliance rate indicates that the vast majority of sampled records in this field comply with the preset business rules and value range constraints, and the data business validity is higher; conversely, a lower value range compliance rate indicates that there are a large number of invalid data records in this field whose values ​​are outside the legal range.

[0030] Finally, the number of records in all sampled records within the current inspection period whose field values ​​conform to the preset format regular expression is counted. This number is then divided by the total number of sampled records to obtain the format compliance rate. Specifically, the format compliance rate is calculated as: (Number of records whose field values ​​conform to the preset format regular expression) / (Total number of sampled records).

[0031] Specifically, all four indicators range from 0 to 1. The closer the null value rate and duplication rate are to 1, the worse the data quality; conversely, the closer the value range compliance rate and format compliance rate are to 1, the better the data quality. A data anomaly detection is triggered when the actual detected value of any indicator exceeds its corresponding preset quality threshold. The preset quality threshold is a pre-defined value, and different preset quality thresholds can be set for different indicators.

[0032] In this embodiment, based on the anomaly detection and reporting module, a data quality inspection task is triggered periodically according to a preset inspection cycle. For each field of each data table within the inspection scope, the null value rate, duplication rate, value range compliance rate, and format compliance rate of the inspected field in the current inspection cycle are calculated. Then, the actual detection value is compared with the corresponding preset quality threshold. When any actual detection value exceeds the corresponding preset quality threshold, a data anomaly is determined to be found. Finally, a structured anomaly event is generated, which includes an anomaly object identifier, anomaly type, anomaly representation value, anomaly discovery timestamp, and a self-healing cycle identifier, to ensure the numerical benchmark for subsequent root cause inference.

[0033] The root cause reasoning context construction module 12 is used to retrieve upstream node information at each level along the lineage graph starting from the abnormal object identifier, retrieve the change records of the abnormal object and related upstream nodes within a preset time window, and assemble the structured abnormal event, the upstream node information at each level and the change records into a root cause reasoning context. In this embodiment of the application, the change record refers to the operation log record automatically retained by the system when the metadata information such as data extraction-transformation-loading task configuration, data table structure definition, and data standard specifications are modified during the operation of the data governance system; the preset time window refers to a fixed period of time that is traced back from the current time of anomaly discovery.

[0034] Specifically, structured anomaly events are used as input to extract anomaly object identifiers. Then, starting from the data node indicated by the anomaly object identifier, a hierarchical retrieval operation is performed in the reverse direction of the data flow in the metadata lineage graph. Simultaneously, a change record retrieval operation is performed, using the anomaly object identifier and the node identifiers of each retrieved upstream node as query conditions to retrieve all relevant change records in the change record repository whose change times fall within a preset time window. After information collection is complete, the structured anomaly events, upstream node information at each level, and change records are assembled and organized according to a preset assembly format to form a root cause reasoning context.

[0035] In this embodiment of the application, the root cause reasoning context construction module is specifically used for: Receive the structured exception event and extract the exception object identifier from the structured exception event; In the metadata lineage diagram, starting from the abnormal object identifier, the direct upstream nodes are searched level by level in the opposite direction of the data flow until the data source table node is searched or the number of search levels reaches the preset upper limit of the search level. The node identifiers and node types of all upstream nodes extracted during the search process are used to form the upstream node information of each level. Using the abnormal object identifier and the node identifiers of all upstream nodes in the upstream node information of each level as query conditions, retrieve ETL task change records and data standard change records whose change time falls within a preset time window in the change record storage, and arrange the retrieved change records in order from the latest to the earliest change time to form a change record sequence. The retrieved upstream nodes are filtered for influence, and low-relevance upstream nodes are filtered out to obtain the filtered upstream node information at each level and the filtered change record sequence. The structured abnormal events, the filtered upstream node information at each level, and the filtered change record sequence are assembled into a root cause reasoning context.

[0036] In this embodiment, firstly, a structured exception event is received, and a parsing operation is performed on the structured exception event to extract the exception object identifier of each field according to a predefined data structure. The exception object identifier serves as the starting index for all subsequent retrieval operations. This identifier is extracted and stored in a memory variable as the starting point for the lineage graph retrieval.

[0037] Secondly, in a metadata lineage graph, nodes represent data entities; directed edges represent data flow dependencies, pointing from upstream nodes to downstream nodes, indicating that the data of downstream nodes originates from upstream nodes. The metadata lineage graph is a well-known term in the data governance field, referring to a directed acyclic graph (DAG) that uses data entities such as tables, ETL tasks, and databases as nodes, and the data flow between these entities as directed edges. It describes the complete flow path of data from the source to the consumer and the upstream and downstream dependencies. In data governance practice, the metadata lineage graph is automatically built and continuously maintained by metadata management tools by parsing ETL task scripts, database table structures, and data integration links. It serves as the standard infrastructure for data tracing, impact analysis, and root cause investigation.

[0038] Specifically, starting with the anomaly object identifier, the direct upstream node is retrieved level by level in the reverse direction of the data flow. The reverse direction of the data flow is tracing back from the current node along the direction of the directed edge, that is, searching from the downstream node to the upstream node. The direct upstream node is the node above the current node that is directly connected to the current node by a directed edge and provides data to the current node. For example, if there is a directed edge from node U to node V, then for node V, node U is its direct upstream node.

[0039] Starting from the initial node, query the list of direct upstream nodes, extract the node identifier and node type, and add them to the upstream node set. Then, starting from the node with the highest current level in the upstream node set, continue querying direct upstream nodes, incrementing the level number by 1 for each level above. Continue executing the search until a node in the data source table is retrieved or the search level reaches the preset search level limit. Then, combine the node identifiers and node types of all upstream nodes extracted during the search process to form the upstream node information for each level.

[0040] Secondly, using the anomaly object identifier and the node identifiers of all upstream nodes in the upstream node information at each level as query conditions, a time range constraint is added to the query conditions. Only ETL task change records and data standard change records whose change times fall within a preset time window are retrieved from the change record storage. ETL task change records are those generated when modifications are made to the SQL script, transformation logic, scheduling configuration, etc., of the ETL task; data standard change records are those generated when modifications are made to the data dictionary, standard field definitions, standard value ranges, etc., in the data standard specification. The retrieved change records are arranged in order of change time from newest to earliest, forming a change record sequence.

[0041] Specifically, the duration of the preset time window is configured by the data governance administrator based on empirical values ​​of data link change propagation delays, such as 24 hours, 72 hours, or 7 days. During retrieval, the starting boundary of the preset time window is set to the anomaly discovery timestamp minus the preset window duration, and the ending boundary is set to the anomaly discovery timestamp itself. Then, all ETL task change records and data standard change records with change timestamps greater than or equal to the starting boundary and less than or equal to the ending boundary are filtered from the change record storage. For example, if the anomaly discovery time is x and the preset time window is 7 days, then ETL task change records and data standard change records with change times between [x-7, x] will be retrieved.

[0042] Furthermore, the upstream nodes retrieved are subjected to influence screening. Nodes with low relevance to the current anomaly are filtered out from all retrieved upstream nodes, resulting in the filtered upstream node information at each level and the filtered change record sequence.

[0043] Finally, the three parts of data—structured abnormal events, filtered upstream node information at each level, and filtered change record sequences—are assembled into a root cause reasoning context according to a preset template format.

[0044] In this embodiment of the application, in the metadata lineage graph, starting from the abnormal object identifier, the direct upstream nodes are retrieved level by level in the opposite direction of the data flow until the data source table node is retrieved or the number of retrieval levels reaches the preset upper limit of the retrieval level. The node identifiers and node types of all upstream nodes extracted during the retrieval process are used to form the upstream node information of each level, including: In the metadata lineage graph, locate the abnormal node corresponding to the abnormal object identifier; Query the list of direct upstream nodes of the abnormal node, extract the node identifier and node type of each direct upstream node, add each direct upstream node to the upstream node set, and record the level of each direct upstream node in the lineage graph with the abnormal node as the first level; For each node with the largest number of levels in the upstream node set, continue to query the list of direct upstream nodes of each node, extract the node identifier and node type of the next higher level node, add the next higher level node to the upstream node set, and record the number of levels of the next higher level node as the number of levels of each node with the largest number of levels plus one. Repeatedly execute the operation of querying the list of direct upstream nodes of each node with the largest search level, increasing the search level by one level for each upward search level, until the search stops when any of the following termination conditions are met: the currently retrieved node is a data source table node, or the current search level reaches the preset search level upper limit, wherein the preset search level upper limit is five levels. The upstream node information for each level is composed of the node identifier, node type, and number of levels of all nodes in the upstream node set.

[0045] In this embodiment of the application, firstly, the abnormal object identifier is received as an input parameter, and then a matching query is performed in the node set of the metadata lineage graph to find the node whose value of the node identifier field is completely consistent with the input abnormal object identifier, and the node is located as the abnormal node corresponding to the abnormal object identifier.

[0046] Secondly, query the list of direct upstream nodes of the abnormal node. In the lineage graph, direct upstream nodes are those nodes that are directly linked to the current node via a directed edge. Obtain the direct upstream nodes by querying the incoming edges in the metadata lineage graph with the current node as the endpoint. For each retrieved direct upstream node, extract its node identifier and node type, add each direct upstream node to the upstream node set, and record the level of each direct upstream node in the lineage graph relative to the abnormal node as level one, indicating that the node and the abnormal node are separated by one hop in the upstream lineage path.

[0047] Secondly, since nodes with different levels may exist in the upstream node set, the same retrieval operation is performed on each node with the highest level in the upstream node set, continuing to query the list of direct upstream nodes for each node. For each higher-level node retrieved, the node identifier and node type are extracted, the higher-level node is added to the upstream node set, and the level of the higher-level node is recorded as the level of each node with the highest level plus one. For example, if the level of the boundary node is the second level, then the level of the higher-level node is recorded as the third level. Through the above operations, the boundary of the upstream retrieval expands upstream level by level, and the retrieval depth increases by one level in each iteration.

[0048] Furthermore, the process of repeatedly querying the list of direct upstream nodes for each node with the highest search level is repeated, increasing the search level by one level for each subsequent level. After completing each level of retrieval, a termination condition is checked. The retrieval stops if either of the following conditions is met: the currently retrieved node is a data source table node, or the current search level reaches the preset search level limit. A data source table node refers to a data table at the very top of the lineage graph with no further upstream nodes, typically corresponding to the original table in the business system's source database or an external data source access table. The preset search level limit is a configurable positive integer, set to control the search depth and prevent excessively long backtracking paths in complex lineage graph scenarios. It can be set according to the topological complexity of the metadata lineage graph; for example, a preset search level limit of five levels can be set for a relatively simple metadata lineage graph. After completing the retrieval of the fifth-level upstream node, if the data source table node has not yet been reached, the retrieval stops, and the search does not continue to the sixth level.

[0049] Finally, when the recursive search stops at any termination condition, the upstream node set has collected all the upstream nodes that have been retrieved. Traverse each node element in the set, extract the node identifier, node type and the number of recorded levels for each node element, and arrange them in ascending order of the number of levels to form the upstream node information for each level.

[0050] In this embodiment of the application, the retrieved upstream nodes are subjected to influence filtering to filter out low-relevance upstream nodes, resulting in filtered upstream node information at each level and a filtered sequence of change records, including: For each upstream node in the upstream node information of each level, perform the following steps: Check whether there are any associated change records for the upstream node within the preset time window; If the upstream node does not have any associated change records within the preset time window, the upstream node will be filtered and removed from the upstream node information at each level, and the change records associated with the removed upstream node in the change record sequence will be removed simultaneously. If the upstream node has associated change records within the preset time window, the reciprocal of the number of levels between the upstream node and the abnormal object in the lineage graph is calculated as the influence score of the upstream node. Upstream nodes with influence scores lower than the preset influence threshold are filtered and removed from the upstream node information at each level, and change records associated with the removed upstream nodes in the change record sequence are removed simultaneously. The upstream nodes retained after the removal operation are completed are used to form the filtered upstream node information for each level, and the change records retained after the removal operation are used to form the filtered change record sequence.

[0051] In this embodiment, an independent correlation check is performed on each upstream node included in the upstream node information of each level. For each upstream node in the upstream node information of each level, the following steps are performed: First, the node identifier of the upstream node to be checked is used as the query key to match and search the previously retrieved change record sequence to check whether there are any related change records of the upstream node within a preset time window.

[0052] If there are no associated change records for the upstream node within the preset time window, it means that the upstream node has not undergone any ETL task logic modification or data standard definition adjustment within the time period from the current anomaly discovery timestamp back to the starting boundary of the preset time window, and its data output logic remains stable. In this case, the upstream node will be filtered and removed from the upstream node information at each level, and the change records associated with the removed upstream node in the change record sequence will be removed simultaneously.

[0053] If an upstream node has associated change records within a preset time window, the inverse of the number of levels between the upstream node and the abnormal object in the lineage graph is calculated as the influence score of the upstream node.

[0054] Specifically, the larger the number of levels, the more likely the impact of changes will be transformed or covered by intermediate nodes in the hierarchical transmission, and its influence will gradually decrease. Therefore, upstream nodes with influence scores lower than the preset influence threshold are filtered out, and only high-influence nodes with close blood ties and actual changes are retained for subsequent analysis.

[0055] For example, suppose the abnormal object is located at level 5, and its upstream nodes are at levels 4, 3, and 2, with levels 1, 2, and 3 respectively. The corresponding influence scores are 1, 1 / 2, and 1 / 3. If the preset influence threshold is set to 0.5, the influence score of level 1 (1) and the influence score of level 2 (1 / 2) are greater than or equal to the preset influence threshold and are therefore retained. However, the influence score of level 3 is approximately 0.33, which is less than the preset influence threshold and is therefore filtered out and removed.

[0056] Therefore, upstream nodes with influence scores below the preset influence threshold are considered as remote weak influence nodes. Although their changes exist, they are insufficient to produce observable abnormal behavior on the abnormal object. They are also filtered out from the upstream node information at each level, and only nodes with higher influence that are closely related and have indeed changed are retained for subsequent anomaly attribution analysis.

[0057] In one feasible embodiment, a preset influence threshold is used to filter upstream nodes. The specific setting process needs to comprehensively consider the target tracing depth of the data lineage, the signal attenuation characteristics of anomaly propagation, and the actual needs of the business scenario for root cause localization accuracy. A value between 0 and 1 is pre-configured by technical personnel. The setting principle is as follows: upstream nodes are divided into high-influence and low-influence upstream nodes by the preset influence threshold. The former are considered potential sources of anomalies with direct causal relationship due to their closer lineage and smaller signal attenuation during change transmission, and are retained for subsequent attribution analysis; the latter are filtered out because their lineage is farther and the change signal is greatly reduced by the transformation operations of intermediate nodes during the step-by-step transmission process.

[0058] Preferably, the preset influence threshold is determined using an adaptive calculation method, specifically including: obtaining the actual maximum number of levels L traced upstream from the abnormal node in the current kinship graph. max And the total number of upstream nodes N contained in the upstream node information at each level; based on the actual maximum number of levels L max The baseline threshold T is calculated according to the following piecewise function: when L max When ≤3, T=0.5; when 3 <L max When ≤6, T=0.4; when 6 <L max When L ≤ 10, T = 0.3; when L max When the value is >10, T=0.25. Obtain the historical self-healing success rate R of the data domain to which the current abnormal object belongs, and the historical average root cause localization level L of that data domain; based on the historical self-healing success rate R and the historical average root cause localization level L... g Calculate the dynamic adjustment ΔS using the following formula: Where α is a preset coefficient, 0 < α < 1, with a preferred value of 0.6; the final preset influence threshold is set to T. f =T+ΔS, and T f The value is constrained between the preset minimum threshold of 0.15 and the preset maximum threshold of 0.85.

[0059] The value of the preset influence threshold directly determines the strictness of the screening. A preset influence threshold closer to 1 allows only upstream nodes directly adjacent to the anomalous object (i.e., level 1) to pass the screening, resulting in the shortest tracing distance and the most focused analysis. This is suitable for scenarios with fewer data link levels, simpler intermediate transformation logic, and anomalies typically caused by direct upstream changes. A lower preset influence threshold, such as 0.3, allows upstream nodes with a tracing distance of 2 or 3 levels to also be included in the retention scope, correspondingly increasing the tracing depth. This is suitable for scenarios with longer data links, where anomalies may originate from changes in distant upstream data sources and manifest after multiple layers of transmission. For example, if statistical analysis reveals that over 80% of the root cause changes in historical data anomalies occurred on upstream nodes within 2 levels of the anomalous object, the preset influence threshold can be set to 1 / 2 = 0.5. This retains all upstream nodes with a tracing distance of 1 level and an influence score of 1, and those with a tracing distance of 2 levels and an influence score of 0.5, while filtering out upstream nodes with a tracing distance of 3 levels or more and an influence score ≤ 0.33.

[0060] Furthermore, the upstream nodes retained after the removal operation are reorganized into filtered upstream node information of each level in ascending order of their hierarchical number. At the same time, all retained change records are rearranged into a filtered change record sequence in ascending order of change time.

[0061] In this embodiment, a structured anomaly event is received and an anomaly object identifier is extracted from it. The data is then searched level by level in the reverse direction of the data flow in the metadata lineage graph until the preset search level limit is reached. Information about upstream nodes at each level is obtained. ETL task change records and data standard change records whose change times fall within a preset time window are then retrieved from the change record storage. The retrieved change records are arranged according to change time to form a change record sequence. The upstream nodes are then subjected to influence filtering to remove low-relevance upstream nodes, resulting in the filtered upstream node information at each level and the filtered change record sequence. Finally, the structured anomaly event, the filtered upstream node information at each level, and the filtered change record sequence are assembled into a root cause reasoning context. This process retains the upstream lineage paths and change information that have a potential causal relationship with the current anomaly while effectively compressing the context volume, providing a high signal-to-noise ratio input for root cause reasoning.

[0062] Root cause reasoning module 13 is used to input the root cause reasoning context into the large language model and output the abnormal root cause localization result; In this embodiment of the application, the large language model refers to a generative language model based on the Transformer architecture and pre-trained on a large-scale corpus, such as GPT and DeepSeek, which has the ability to extract key information from complex contexts and perform logical reasoning; the root cause localization result is a structured description of the root cause of the anomaly.

[0063] Specifically, the root cause reasoning context output by the root cause reasoning context construction module is received and used as a component of the input prompt words. It is then input into the large language model. After receiving the root cause reasoning context containing the description of the abnormal event, the information of upstream link nodes at each level, and the sequence of change records, the model performs semantic encoding and association analysis on the context information through its internal attention mechanism and feedforward network. This identifies the upstream node most relevant to the current data anomaly and the most likely cause of the change, thereby outputting the anomaly root cause localization result.

[0064] In this embodiment of the application, the root cause reasoning module is specifically used for: The abnormal object identifier, abnormal type, and abnormal representation value in the structured abnormal event are concatenated into a query text string according to a preset delimiter, and the query text string is transformed into a fixed-dimensional query vector through a pre-trained text embedding model. Retrieve a predetermined number of historical governance cases from the historical governance case database that have the highest cosine similarity to the query vector, and use the retrieved historical governance cases as reference cases. The historical anomaly event descriptions, historical root cause localization results, and historical remediation solutions of the reference case are combined with the root cause inference context and confidence assessment instructions to form input prompts. These input prompts are then input into a large language model, which performs analogical inference based on the reference case, outputting anomaly root cause localization results and inference basis. Simultaneously, a reasoning confidence score between 0 and 1 is output according to the confidence assessment instructions. The anomaly root cause localization results include the identified upstream node and an anomaly cause description. The inference basis includes a reference index pointing to relevant change records in the change record sequence and a case reference index pointing to the reference case. When the reasoning confidence score is lower than the preset confidence threshold, the structured abnormal event and the root cause reasoning context are forwarded to the manual review terminal, the current self-healing loop is paused, and the manual review terminal is waited for the manual root cause localization result to be returned. Upon receiving the artificial root cause localization result, the abnormal root cause localization result output by the large language model is replaced with the artificial root cause localization result, and the repair plan generation and dispatch continue.

[0065] In this embodiment, firstly, the anomaly object identifier, anomaly type, and anomaly representation value in the structured anomaly event are concatenated according to a preset delimiter to obtain a query text string consisting of anomaly object identifier - preset delimiter - anomaly type - preset delimiter - anomaly representation value. Then, this query text string is input into a pre-trained text embedding model. The text embedding model is a neural network model that maps natural language text into a dense vector representation. Its output is a fixed-dimensional floating-point vector that encodes the semantic information of the input text in a high-dimensional semantic space.

[0066] Secondly, a predetermined number of historical governance cases with the highest cosine similarity to the query vector are retrieved from the historical governance case library and used as reference cases. The historical governance case library is a pre-built database within the data governance system, storing successfully handled data anomaly events and their complete processing records. Each historical governance case includes at least three fields: a description of the historical anomaly event, the historical root cause location result, and the historical remediation plan.

[0067] Specifically, using the generated query vector as the retrieval condition, vector similarity retrieval is performed in the historical governance case database. The similarity metric uses cosine similarity: it measures the directional consistency by calculating the cosine of the angle between two vectors, with a value ranging from -1 to 1. The closer the value is to 1, the more consistent the directions of the two vectors in the semantic space, and the higher the semantic similarity. Then, each case in the historical governance case database is traversed, and the cosine similarity is calculated between the pre-stored historical abnormal event description's corresponding embedded vector and the query vector. All cases are sorted from high to low according to their cosine similarity, and a predetermined number of the top-ranked cases are selected as reference cases. The specific value of the predetermined number can be configured in advance and can be combined with similarity scores during implementation. For example, if the similarity scores are all below a certain threshold, such as 0.5, then either the retrieval of reference cases is abandoned or only the historical governance case with the highest similarity is selected.

[0068] Next, the historical anomaly descriptions, root cause localization results, and historical remediation plans from the reference cases are assembled with the root cause inference context and confidence assessment instructions into input prompts. These prompts are then input into a large language model, which performs analogical inference based on the reference cases, outputting the anomaly root cause localization result and the inference basis. Simultaneously, a confidence score ranging from 0 to 1 is output according to the confidence assessment instructions. The root cause inference context provides a comprehensive description of the current anomaly to be diagnosed. The confidence assessment instructions can be a natural language instruction appended to the input prompts, such as: "When outputting the anomaly root cause localization result, please simultaneously assess the inference confidence score of this inference, represented by a value between 0 and 1, where 0 indicates complete uncertainty and 1 indicates complete certainty." The anomaly root cause localization result includes the identified upstream node and a description of the anomaly cause. The inference basis includes reference indices pointing to relevant change records in the change record sequence and case reference indices pointing to the reference cases. The reference index of relevant change records illustrates the specific change evidence upon which the root cause conclusions of the large language model are based, and the case reference index pointing to the reference cases illustrates the sources of historical experience drawn by the large language model.

[0069] In one specific embodiment, the large language model employs an autoregressive architecture for the decoder, consisting of multiple stacked Transformer decoder layers. Each decoder layer contains three sub-modules: a masked multi-head self-attention mechanism, a feedforward neural network, and residual connections and layer normalization. The masked multi-head self-attention mechanism introduces a causal mask matrix, ensuring that when processing each word in the sequence, it only focuses on all historical words preceding that word. This guarantees that the large language model can progressively generate text in an autoregressive manner, predicting the next word based on the generated context. Each prediction appends the newly generated word to the end of the input sequence as new context for subsequent predictions until an end marker is generated. The feedforward neural network performs independent nonlinear transformations on each position vector processed by the attention mechanism, extracting higher-order feature representations. Taking DeepSeek as an example, it adds group query attention or multi-head latent attention to the basic architecture, uses rotational position encoding instead of traditional embedding, models relative positional information in long sequences, and uses gated linear units instead of simple feedforward networks. The core capabilities of large language models stem from pre-training on massive unlabeled text corpora, employing a self-supervised learning task to predict the next lexical unit. After inputting pre-trained and fine-tuned input prompts into this type of large language model, it leverages its internally learned semantic understanding and analogical reasoning abilities to perform correlation analysis between reference cases and the current root cause reasoning context, outputting results including anomaly root cause localization, reasoning basis, and reasoning confidence score. The confidence score is a floating-point number between 0 and 1; a confidence score closer to 1 indicates more reliable reasoning, while a confidence score closer to 0 indicates less reliable reasoning.

[0070] The inference confidence score output by the large language model is compared with a preset confidence threshold. The preset confidence threshold is a pre-configured floating-point parameter, for example, it can be set to 0.7. When the inference confidence score is lower than the preset confidence threshold, the structured anomaly event and root cause inference context are forwarded to a human review terminal, such as the data governance engineer's computer client or web management interface. At the same time, the current self-healing loop is paused and waits until the human review terminal returns the human root cause localization result.

[0071] Upon receiving the artificial root cause localization result, the abnormal root cause localization result output by the large language model is discarded. The artificial root cause localization result is used as the input for the subsequent repair plan generation and dispatch module. Self-healing resumes execution from the paused state and continues to execute the repair plan generation and dispatch steps.

[0072] In this embodiment of the application, the pre-training process of the text embedding model includes: Construct a data governance corpus dataset, which includes data table structure description text, data standard specification text, ETL task log text, and data quality inspection report text; Field names, table names, indicator names, and data governance terms are extracted from the data governance domain corpus dataset to construct a data governance domain thesaurus; Each word in the data governance domain vocabulary is used as the center word, and other words that co-occur with the center word in the text within a preset sliding window size are used as context words. Word pairs of the center word and context words are constructed as training positive samples. Each word in the data governance domain lexicon is used as the center word. Words that do not co-occur with the center word in the text are randomly sampled from the data governance domain lexicon as negative sample context words. Word pairs between the center word and the negative sample context words are constructed as training negative samples. A text embedding model is constructed, which includes an input word embedding layer and an output word embedding layer. The input word embedding layer is used to map the center word to an input word embedding vector, and the output word embedding layer is used to map the context words to an output word embedding vector. Using the training positive samples and the training negative samples as training data, the text embedding model is trained with the training objective of maximizing the prediction probability of the center word and the context word and minimizing the prediction probability of the center word and the context word of the negative sample. The training continues until convergence is verified, resulting in a pre-trained text embedding model. The prediction probability is calculated by applying the sigmoid function to the inner product of the input word embedding vector of the center word and the output word embedding vector of the context word or the context word of the negative sample.

[0073] In this embodiment, firstly, a data governance corpus dataset is constructed. This dataset includes data table structure description text, data standard specification text, ETL task log text, and data quality inspection report text. Specifically, the data table structure description text is a natural language description of structured metadata such as table creation statements, field comments, and table description information within the data governance system; the data standard specification text is a normative document defined in the data standard management module, including data dictionaries, standard field definitions, and value range constraint rules; the ETL task log text is a log record generated during ETL task execution, containing information such as task name, transformation logic description, and execution status; and the data quality inspection report text is a report document generated after the quality inspection task is executed, containing inspection results, anomaly indicators, and threshold comparisons.

[0074] Secondly, text preprocessing operations, including word segmentation, removal of stop words and punctuation, are performed on the data governance domain corpus dataset. Then, field names, table names, indicator names and data governance terms are extracted from the data governance domain corpus dataset. After deduplication of all extracted domain terms, a data governance domain lexicon is constructed.

[0075] Next, using each word in the data governance lexicon as the center word, the text positions where the center word appears in the corpus dataset are located. For each position, all other words within a range of a preset sliding window size (number of words before and after the center word) are extracted as context words. The preset sliding window size is a configurable positive integer parameter, for example, it can be set to 5, meaning 5 words are taken before and after the center word as the context range. For each context word co-occurring with the center word within the window, a word pair between the center word and the context word is generated and labeled as a positive sample, thus constructing training positive samples. These positive samples are used to train the text embedding model to recognize semantically related word pairs.

[0076] Furthermore, each word in the data governance domain lexicon is used as the center word. Words that do not co-occur with the center word in the text are randomly sampled from the data governance domain lexicon as negative sample context words. These sampled words are required to have never co-occurred with the center word within a preset sliding window size in the entire text of the corpus dataset. Each sampled negative sample context word and the center word form a word pair, labeled as a negative sample, thus constructing training negative samples. These negative samples are used to train the text embedding model to identify semantically unrelated word pairs, serving as a mechanism to prevent all word vectors from converging during embedding model training.

[0077] Furthermore, a text embedding model with a dual-embedding-layer architecture is constructed. The text embedding model includes an input word embedding layer and an output word embedding layer. The input word embedding layer is a weight matrix of dimension V×D, where V is the size of the vocabulary in the data governance domain and D is the preset embedding vector dimension. This layer is used to map the index position of the center word in the vocabulary to a D-dimensional input word embedding vector. The output word embedding layer is a weight matrix of dimension V×D, used to map the index position of the context words in the vocabulary to a D-dimensional output word embedding vector, and to map the context words to the output word embedding vector.

[0078] Finally, the text embedding model is trained using training positive and negative samples as training data. For each training sample (center word, context word, label), the input word embedding vector of the center word is obtained through the input word embedding layer, and the output word embedding vector of the context word is obtained through the output word embedding layer. The inner product of the two is calculated, and the result is input into the sigmoid function. The sigmoid function maps any real number to a probability value between 0 and 1, which is used to represent the semantically related prediction probability of the word pair. The text embedding model is trained with the goal of maximizing the prediction probability of the center word and the context words and minimizing the prediction probability of the center word and the negative sample context words, until validation convergence is achieved, resulting in a pre-trained text embedding model. The prediction probability is calculated by taking the inner product of the input word embedding vector of the center word and the output word embedding vector of the context word or the negative sample context word and then applying it to the sigmoid function.

[0079] Specifically, a stochastic gradient descent optimization algorithm is used to iteratively update all weight parameters of the input and output word embedding layers. Each iteration processes a batch of training samples, calculates the gradient of the loss function with respect to all weights, and updates the weights along the gradient descent direction. The system continues iterative training until the loss value on the validation set stops decreasing. At this point, the weight parameters of the trained input and output word embedding layers are saved, resulting in a pre-trained text embedding model. In practical use, the word embedding vectors output by the input word embedding layer are either average pooled or directly concatenated to serve as the vector representation of the entire query text.

[0080] In this embodiment, the query text string is obtained by concatenation, and the query vector is obtained by text embedding model. A preset number of historical governance cases with the highest cosine similarity are retrieved from the historical governance case library. The retrieved historical governance cases are used as reference cases and assembled into input prompt words for input into the large language model to obtain inference confidence score, anomaly root cause location result, and inference basis. The decision boundary of human-machine collaboration is established by quantitative evaluation of inference confidence score, which improves the accuracy of anomaly root cause location result and avoids the repair risk caused by the propagation of low inference confidence score downstream.

[0081] The repair scheme generation and dispatch module 14 is used to input the abnormal root cause localization result into the large language model, generate a repair scheme, and perform repair operations based on the repair scheme; In this application embodiment, the repair scheme refers to an executable sequence of operations or a set of operation instructions designed for the identified abnormal cause to eliminate the abnormal state.

[0082] Specifically, the anomaly root cause localization results output by the root cause reasoning module are input again into the large language model. After receiving the anomaly root cause localization results, the large language model generates a remediation plan based on data governance domain knowledge. After the remediation plan is generated, it is dispatched to the corresponding execution module based on the remediation operation type.

[0083] In this embodiment of the application, the repair scheme generation and dispatch module is specifically used for: Obtain the technology stack information of the abnormal object, which includes the type of the database where the abnormal object is located, the execution engine type of the ETL task associated with the abnormal object, and the version information of the data standard specification associated with the abnormal object; The abnormal root cause localization result and the technology stack information are input into the large language model, and the large language model generates a structured repair solution, wherein the structured repair solution includes at least a repair operation type field, a repair object identifier field, and a repair operation content field; Based on the value of the repair operation type field, the structured repair scheme is dispatched to the corresponding execution module to perform the repair operation.

[0084] In this embodiment, firstly, since the remediation plan is highly dependent on the technical environment of the abnormal object, the remediation plan generation and dispatch module obtains the technical stack information of the abnormal object from the metadata storage of the data governance system. This technical stack information includes the type of the database where the abnormal object resides, the execution engine type of the ETL task associated with the abnormal object, and the version information of the data standard specification associated with the abnormal object. Specifically, the type of the database where the abnormal object resides refers to the product type of the database containing the data table storing the abnormal object. Different database types have significant differences in SQL syntax, data type conversion rules, and function support, requiring the remediation plan to generate corresponding operation statements for specific database syntax. The execution engine type of the ETL task associated with the abnormal object is the execution engine running the ETL task responsible for processing the data table data. Different execution engines differ in task configuration modification and job submission methods. The version information of the data standard specification associated with the abnormal object is the version number of the data standard document that constrains the value specification of the field. Different versions of the standard may have changes in value range definitions and format requirements.

[0085] Secondly, the root cause analysis results and technology stack information are input into the large language model, which generates a structured repair plan. A structured repair plan is a set of repair operation instructions organized according to a predefined data structure. This structured repair plan includes at least a repair operation type field, a repair object identifier field, and a repair operation content field. The repair operation type field indicates the category of the repair operation, and its value determines which execution module the repair plan should be dispatched to. The repair object identifier field uniquely identifies the metadata object affected by the repair operation, such as the task ID of the ETL task to be modified or the table name of the data table to be corrected. The repair operation content field carries the specific repair operation instructions, such as the modified ETL task configuration parameters or the SQL correction statement to be executed.

[0086] Finally, based on the value of the repair operation type field, the structured repair solution is dispatched to the corresponding execution module to perform the repair operation. For example, if the repair operation type is "modify ETL task configuration", the repair solution is dispatched to the ETL scheduling management module, which then updates the ETL task configuration according to the configuration parameters in the repair operation content field.

[0087] In this embodiment, the technology stack information of the abnormal object is obtained, and the root cause localization result and technology stack information are input together into a large language model to generate a structured repair plan. Finally, the structured repair plan is dispatched to the corresponding execution module to perform the repair operation according to the value of the repair operation type field. The database type, ETL task execution engine type, and data standard specification version information of the abnormal object are used as constraints for the generation of the repair plan and input into the large language model to ensure that the generated repair operation content is accurately matched with the actual technology stack of the abnormal object. At the same time, the standardized value of the repair operation type field enables the automatic dispatch of the repair plan to different execution modules, avoiding the intermediate links of manual translation and forwarding and the operational deviations introduced therefrom.

[0088] The repair verification and closed-loop module 15 is used to re-perform quality checks on the abnormal object after the repair operation is completed. If the check fails, the abnormal detection and reporting module is re-triggered to enter the next round of self-healing loop.

[0089] In this embodiment, an execution operation is triggered after the repair operation is completed. A re-check instruction for the original abnormal object is sent to the anomaly detection and reporting module, triggering the re-execution of the same quality check rules as the initial check on the same data object. The purpose of re-executing the quality check is to verify whether the implemented repair operation has indeed eliminated the data anomaly. After obtaining the re-check result, a judgment is made: if the re-check result shows that the anomaly characteristic value has fallen back to within the preset quality threshold, it indicates that the repair operation is effective and the self-healing loop ends successfully; if the re-check result shows that the anomaly characteristic value has not fallen back to within the preset quality threshold, it indicates that the repair operation has failed to effectively eliminate the cause of the anomaly. At this time, the anomaly detection and reporting module is re-triggered to generate a new structured anomaly event and enter the next round of self-healing loop. Through the feedback mechanism, the repair effect is automatically verified, and automatic retry is performed when the repair is ineffective, forming a closed-loop control.

[0090] In this embodiment of the application, the repair verification and closed-loop module is specifically used for: After the repair operation is completed, a re-check instruction for the abnormal object identifier is sent to the anomaly detection and reporting module. The anomaly detection and reporting module then re-executes the quality check on the data table field corresponding to the abnormal object identifier to obtain the re-check result. When the abnormal characterization value in the re-check result falls back to within the preset quality threshold, a successful repair record is generated, and the structured abnormal event, abnormal root cause localization result, structured repair plan and successful repair record of this self-healing cycle are stored as a historical governance case in the historical governance case database. When the abnormal characterization value in the re-check result still has not fallen back to within the preset quality threshold, a re-trigger instruction containing the abnormal object identifier and the current self-healing cycle identifier is generated, and the re-trigger instruction is sent to the abnormal detection and reporting module. The abnormal detection and reporting module generates a new structured abnormal event carrying the current self-healing cycle identifier, triggering a new round of self-healing cycle.

[0091] In this embodiment, firstly, after the repair operation is completed, a re-check instruction for the abnormal object identifier is sent to the anomaly detection and reporting module. After receiving the re-check instruction, the anomaly detection and reporting module re-executes the quality check on the data table field corresponding to the abnormal object identifier. That is, it uses the above calculation method to recalculate the null value rate, duplication rate, value range compliance rate, and format compliance rate of the field at the current time point to obtain the re-check result. The re-check result includes the measured values ​​of the four quality indicators of the field at the current time.

[0092] The abnormality representation values ​​corresponding to the original abnormality type in the re-verification results are compared with the preset quality threshold. When the abnormality representation values ​​in the re-verification results fall back to within the preset quality threshold, the repair operation is deemed successful, a repair success record is generated, and the structured abnormality event, abnormal root cause localization results, structured repair plan, and repair success record of this self-healing cycle are stored as a historical governance case in the historical governance case database.

[0093] If the abnormality values ​​in the re-verification results still do not fall below the preset quality threshold, it is determined that the repair operation has failed to effectively eliminate the cause of the abnormality. A re-trigger command containing the abnormal object identifier and the current self-healing cycle identifier is then generated. This re-trigger command is sent to the anomaly detection and reporting module, which generates a new structured abnormal event carrying the current self-healing cycle identifier, triggering a new round of self-healing. The new cycle will re-execute all steps from root cause reasoning context construction to repair plan generation and dispatch, adjusting the anomaly root cause localization results and structured repair plans based on the updated state until the repair is successful.

[0094] In this embodiment, the repair effect is quantitatively evaluated by a re-verification machine, replacing the manual verification method, to ensure the accuracy and real-time nature of the verification results; each successful self-healing experience is stored as a historical governance case and stored in the historical governance case library; the self-healing cycle is iterated by re-triggering after repair failure, adaptively responding to complex abnormal scenarios until the abnormality is completely eliminated, thus ensuring the reliability of the self-healing process.

[0095] Through the above specific implementation methods, the embodiments of the present invention achieve the following technical effects: In this embodiment of the invention, firstly, the anomaly detection and reporting module triggers a data quality inspection task periodically according to a preset inspection cycle. For each field of each data table within the inspection scope, the null value rate, duplication rate, value range compliance rate, and format compliance rate of the inspected field in the current inspection cycle are calculated. Then, the actual detection values ​​are compared with the corresponding preset quality thresholds. When any actual detection value exceeds the corresponding preset quality threshold, a data anomaly is determined to be found. Finally, a structured anomaly event is generated, which includes an anomaly object identifier, anomaly type, anomaly representation value, anomaly discovery timestamp, and a self-healing cycle identifier, to ensure the numerical benchmark for subsequent root cause inference.

[0096] Secondly, the system receives structured anomaly events and extracts anomaly object identifiers from them. It then searches the metadata lineage graph in the reverse direction of the data flow, level by level, until the preset search level limit is reached. This process obtains upstream node information at each level. Next, it retrieves ETL task change records and data standard change records whose change times fall within a preset time window from the change record storage. The retrieved change records are arranged according to change time to form a change record sequence. The upstream nodes are then filtered for influence to remove low-relevance upstream nodes, resulting in filtered upstream node information at each level and a filtered change record sequence. Finally, the structured anomaly events, the filtered upstream node information at each level, and the filtered change record sequence are assembled into a root cause reasoning context. This process retains upstream lineage paths and change information that have a potential causal relationship with the current anomaly while effectively compressing the context volume, providing a high signal-to-noise ratio input for root cause reasoning.

[0097] Secondly, the query text string is obtained by concatenation, and the query vector is obtained through a text embedding model. A preset number of historical governance cases with the highest similarity to cosine are retrieved from the historical governance case library. The retrieved historical governance cases are used as reference cases and assembled into input prompt words for input into a large language model to obtain inference confidence scores, anomaly root cause localization results, and inference basis. The decision boundary of human-machine collaboration is established through the quantitative evaluation of inference confidence scores, which improves the accuracy of anomaly root cause localization results and avoids the repair risk caused by the downstream propagation of low inference confidence scores.

[0098] Finally, the repair effect is quantitatively evaluated by a re-verification machine, replacing manual verification to ensure the accuracy and real-time nature of the verification results; each successful self-healing experience is stored as a historical governance case and added to the historical governance case library; the self-healing cycle is iterated by re-triggering after repair failure, adaptively responding to complex abnormal scenarios until the anomaly is completely eliminated, ensuring the reliability of the self-healing process.

Claims

1. A self-healing data governance intelligent agent control system based on a large language model, characterized in that, The system includes: The anomaly detection and reporting module is used to generate structured anomaly events containing anomaly object identifiers, anomaly types, and anomaly representation values ​​when data quality checks detect data anomalies. The root cause reasoning context construction module is used to retrieve upstream node information at each level along the lineage graph starting from the abnormal object identifier, retrieve change records of the abnormal object and related upstream nodes within a preset time window, and assemble the structured abnormal event, the upstream node information at each level, and the change records into a root cause reasoning context. The root cause reasoning module is used to input the root cause reasoning context into the large language model and output the abnormal root cause localization result. The repair solution generation and dispatch module is used to input the abnormal root cause localization results into the large language model, generate a repair solution, and perform repair operations based on the repair solution; The repair verification and closed-loop module is used to re-perform quality checks on the abnormal object after the repair operation is completed. If the check fails, the abnormal detection and reporting module is triggered again to enter the next round of self-healing loop.

2. The self-healing data governance intelligent agent control system based on a large language model according to claim 1, characterized in that, The anomaly detection and reporting module is specifically used for: Data quality inspection tasks are triggered periodically according to a preset inspection cycle. For each field of each data table within the inspection scope, the null value rate, duplication rate, value range compliance rate, and format compliance rate of the field under inspection are calculated in the current inspection cycle. Compare any one of the actual detected values ​​of the null value rate, the duplication rate, the value range compliance rate, and the format compliance rate with the corresponding preset quality threshold. When any one of the actual detected values ​​exceeds the corresponding preset quality threshold, it is determined that data anomalies have been found. A structured exception event is generated, which includes an exception object identifier, exception type, exception characterization value, exception discovery timestamp, and current self-healing cycle identifier. The exception object identifier is a combination of the data table name and field name where the data exception occurred. The exception type is the name of the indicator type that triggers the preset quality threshold. The exception characterization value is the actual detection value that triggers the preset quality threshold. The exception discovery timestamp is the time point when the current quality check task is executed. The current self-healing cycle identifier is a unique identifier assigned by the exception detection and reporting module when generating the structured exception event.

3. The self-healing data governance intelligent agent control system based on a large language model according to claim 2, characterized in that, For each data table within the inspection scope and for each field under inspection, calculate the null value rate, duplication rate, value range compliance rate, and format compliance rate of the field under inspection within the current inspection period, including: The total number of all sampled records for the field being inspected within the current inspection period is counted and used as the total number of sampled records. The number of records in the current inspection period where the field being inspected has a null value is counted. The null value rate is obtained by dividing the number of records with null values ​​by the total number of sampled records. The number of records with identical field values ​​in all sampled records within the current inspection period is counted, and the duplication rate is obtained by dividing the number of records with identical field values ​​by the total number of sampled records. The number of records in the current inspection period whose field values ​​fall within the preset value range is counted. The value range compliance rate is obtained by dividing the number of records whose field values ​​fall within the preset value range by the total number of sampled records. The number of records in the current inspection period whose field values ​​conform to a preset regular expression is counted. The number of records whose field values ​​conform to the preset regular expression is divided by the total number of sampled records to obtain the format compliance rate.

4. The self-healing data governance intelligent agent control system based on a large language model according to claim 1, characterized in that, The root cause reasoning context construction module is specifically used for: Receive the structured exception event and extract the exception object identifier from the structured exception event; In the metadata lineage diagram, starting from the abnormal object identifier, the direct upstream nodes are searched level by level in the opposite direction of the data flow until the data source table node is searched or the number of search levels reaches the preset upper limit of the search level. The node identifiers and node types of all upstream nodes extracted during the search process are used to form the upstream node information of each level. Using the abnormal object identifier and the node identifiers of all upstream nodes in the upstream node information of each level as query conditions, retrieve ETL task change records and data standard change records whose change time falls within a preset time window in the change record storage, and arrange the retrieved change records in order from the latest to the earliest change time to form a change record sequence. The retrieved upstream nodes are filtered for influence, and low-relevance upstream nodes are filtered out to obtain the filtered upstream node information at each level and the filtered change record sequence. The structured abnormal events, the filtered upstream node information at each level, and the filtered change record sequence are assembled into a root cause reasoning context.

5. The self-healing data governance intelligent agent control system based on a large language model according to claim 4, characterized in that, In the metadata lineage graph, starting from the abnormal object identifier, the direct upstream nodes are retrieved level by level in the reverse direction of the data flow until the data source table node is reached or the retrieval level reaches the preset upper limit. The node identifiers and node types of all upstream nodes extracted during the retrieval process are used to form the upstream node information for each level, including: In the metadata lineage graph, locate the abnormal node corresponding to the abnormal object identifier; Query the list of direct upstream nodes of the abnormal node, extract the node identifier and node type of each direct upstream node, add each direct upstream node to the upstream node set, and record the level of each direct upstream node in the lineage graph with the abnormal node as the first level; For each node with the largest number of levels in the upstream node set, continue to query the list of direct upstream nodes of each node, extract the node identifier and node type of the next higher level node, add the next higher level node to the upstream node set, and record the number of levels of the next higher level node as the number of levels of each node with the largest number of levels plus one. Repeatedly execute the operation of querying the list of direct upstream nodes of each node with the largest search level, increasing the search level by one level for each upward search level, until the search stops when any of the following termination conditions are met: the currently retrieved node is a data source table node, or the current search level reaches the preset search level upper limit, wherein the preset search level upper limit is five levels. The upstream node information for each level is composed of the node identifier, node type, and number of levels of all nodes in the upstream node set.

6. The self-healing data governance intelligent agent control system based on a large language model according to claim 4, characterized in that, The retrieved upstream nodes are filtered for influence, removing low-relevance upstream nodes, resulting in filtered upstream node information at each level and a filtered sequence of change records, including: For each upstream node in the upstream node information of each level, perform the following steps: Check whether there are any associated change records for the upstream node within the preset time window; If the upstream node does not have any associated change records within the preset time window, the upstream node will be filtered and removed from the upstream node information at each level, and the change records associated with the removed upstream node in the change record sequence will be removed simultaneously. If the upstream node has associated change records within the preset time window, the reciprocal of the number of levels between the upstream node and the abnormal object in the lineage graph is calculated as the influence score of the upstream node. Upstream nodes with influence scores lower than the preset influence threshold are filtered and removed from the upstream node information at each level, and change records associated with the removed upstream nodes in the change record sequence are removed simultaneously. The upstream nodes retained after the removal operation are completed are used to form the filtered upstream node information for each level, and the change records retained after the removal operation are used to form the filtered change record sequence.

7. The self-healing data governance intelligent agent control system based on a large language model according to claim 1, characterized in that, The root cause reasoning module is specifically used for: The abnormal object identifier, abnormal type, and abnormal representation value in the structured abnormal event are concatenated into a query text string according to a preset delimiter, and the query text string is transformed into a fixed-dimensional query vector through a pre-trained text embedding model. Retrieve a predetermined number of historical governance cases from the historical governance case database that have the highest cosine similarity to the query vector, and use the retrieved historical governance cases as reference cases. The historical anomaly event descriptions, historical root cause localization results, and historical remediation solutions of the reference case are combined with the root cause inference context and confidence assessment instructions to form input prompts. These input prompts are then input into a large language model, which performs analogical inference based on the reference case, outputting anomaly root cause localization results and inference basis. Simultaneously, a reasoning confidence score between 0 and 1 is output according to the confidence assessment instructions. The anomaly root cause localization results include the identified upstream node and an anomaly cause description. The inference basis includes a reference index pointing to relevant change records in the change record sequence and a case reference index pointing to the reference case. When the reasoning confidence score is lower than the preset confidence threshold, the structured abnormal event and the root cause reasoning context are forwarded to the manual review terminal, the current self-healing loop is paused, and the manual review terminal is waited for the manual root cause localization result to be returned. Upon receiving the artificial root cause localization result, the abnormal root cause localization result output by the large language model is replaced with the artificial root cause localization result, and the repair plan generation and dispatch continue.

8. The self-healing data governance intelligent agent control system based on a large language model according to claim 7, characterized in that, The pre-training process of a text embedding model includes: Construct a data governance corpus dataset, which includes data table structure description text, data standard specification text, ETL task log text, and data quality inspection report text; Field names, table names, indicator names, and data governance terms are extracted from the data governance domain corpus dataset to construct a data governance domain thesaurus; Each word in the data governance domain vocabulary is used as the center word, and other words that co-occur with the center word in the text within a preset sliding window size are used as context words. Word pairs of the center word and context words are constructed as training positive samples. Each word in the data governance domain lexicon is used as the center word. Words that do not co-occur with the center word in the text are randomly sampled from the data governance domain lexicon as negative sample context words. Word pairs between the center word and the negative sample context words are constructed as training negative samples. A text embedding model is constructed, which includes an input word embedding layer and an output word embedding layer. The input word embedding layer is used to map the center word to an input word embedding vector, and the output word embedding layer is used to map the context words to an output word embedding vector. Using the training positive samples and the training negative samples as training data, the text embedding model is trained with the training objective of maximizing the prediction probability of the center word and the context word and minimizing the prediction probability of the center word and the context word of the negative sample. The training continues until convergence is verified, resulting in a pre-trained text embedding model. The prediction probability is calculated by applying the sigmoid function to the inner product of the input word embedding vector of the center word and the output word embedding vector of the context word or the context word of the negative sample.

9. The self-healing data governance intelligent agent control system based on a large language model according to claim 1, characterized in that, The repair scheme generation and dispatch module is specifically used for: Obtain the technology stack information of the abnormal object, which includes the type of the database where the abnormal object is located, the execution engine type of the ETL task associated with the abnormal object, and the version information of the data standard specification associated with the abnormal object; The abnormal root cause localization result and the technology stack information are input into the large language model, and the large language model generates a structured repair solution, wherein the structured repair solution includes at least a repair operation type field, a repair object identifier field, and a repair operation content field; Based on the value of the repair operation type field, the structured repair scheme is dispatched to the corresponding execution module to perform the repair operation.

10. The self-healing data governance intelligent agent control system based on a large language model according to claim 1, characterized in that, The repair verification and closed-loop module is specifically used for: After the repair operation is completed, a re-check instruction for the abnormal object identifier is sent to the anomaly detection and reporting module. The anomaly detection and reporting module then re-executes the quality check on the data table field corresponding to the abnormal object identifier to obtain the re-check result. When the abnormal characterization value in the re-check result falls back to within the preset quality threshold, a successful repair record is generated, and the structured abnormal event, abnormal root cause localization result, structured repair plan and successful repair record of this self-healing cycle are stored as a historical governance case in the historical governance case database. When the abnormal characterization value in the re-check result still has not fallen back to within the preset quality threshold, a re-trigger instruction containing the abnormal object identifier and the current self-healing cycle identifier is generated, and the re-trigger instruction is sent to the abnormal detection and reporting module. The abnormal detection and reporting module generates a new structured abnormal event carrying the current self-healing cycle identifier, triggering a new round of self-healing cycle.

Images