Data governance method based on intelligent decision engine

Abstract

The application relates to the technical field of data governance, and discloses a data governance method based on an intelligent decision engine, which comprises the following steps: classifying and marking the business semantics of real-time streaming data after preliminary processing, and constructing a data portrait library; constructing a dynamic topology graph, learning abnormal propagation rules based on a graph neural network, analyzing the influence range, and establishing an influence grading mechanism; performing multi-dimensional quality evaluation on the data, generating a dynamic data quality score and a grading strategy; constructing a data quality historical problem and reason case library, using a large language model agent to generate a quality abnormal root cause judgment and influence quantization report; generating a candidate strategy set, selecting an optimal governance strategy from the candidate strategy set by establishing a multi-objective optimization model; using a large language model agent to perform compliance testing and conflict identification on the optimal governance strategy, and dynamically adjusting the decision weights of the rule engine by using a reinforcement learning algorithm, so that a closed loop of data governance and dynamic learning is realized.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, and in particular to a data governance method based on an intelligent decision engine. Background Technology

[0002] With the acceleration of global digital transformation, the amount of data generated by enterprises and other organizations in their daily operations is exploding, and the types of data are expanding from traditional structured data to diverse and heterogeneous data such as logs, sensor IoT streaming data, etc. Core business and management activities are increasingly reliant on high-quality, reliable data, and data quality is directly related to operational efficiency, risk control, and decision intelligence.

[0003] Currently, mainstream data governance solutions primarily rely on rule-based data quality platforms, master data management platforms, and some tools based on data lineage analysis. They typically employ predefined static rules for data cleaning and quality control, which can be effective when data scale is limited and business scenarios are simple. However, in the face of modern data environments characterized by multi-source heterogeneity, high concurrency, and frequent changes in business logic, existing solutions reveal several significant shortcomings: ① Existing systems heavily rely on manually pre-defined rules, failing to dynamically adjust and optimize based on data distribution and changes in business context. When data source structures change or new business scenarios emerge, significant manpower is required for rule maintenance and manual intervention, resulting in slow response times and high operational costs. ② Existing solutions lack the ability to integrate multi-dimensional information for joint analysis and intelligent reasoning. Their governance processes are mostly post-event reporting and manual tracking, unable to achieve automatic problem attribution, intelligent recommendation of repair paths, or support real-time decision-making needs under complex business scenarios. Furthermore, the decision-making process is often a "black box," lacking transparency and explainability. ③ Traditional governance methods often remain at the table, field, or rule level, failing to establish a comprehensive quality map management system encompassing "indicators-processes-applications." This makes it difficult to identify and respond to the propagation of anomalies across systems and business domains, and governance measures are often piecemeal, failing to address the root causes of problems. ④ Existing methods primarily rely on technical indicators as measurement standards, resulting in a disconnect between governance outcomes and business effectiveness. The lack of automatic linkage and intelligent evaluation of the business value brought by governance measures makes it difficult to quantify the return on investment in governance work and to obtain sustained business support.

[0004] Therefore, developing a new generation of data governance methods that can achieve intelligent sensing, dynamic optimization, and deep closed-loop integration with business operations is not only an inevitable trend in technological development, but also an urgent business need to ensure the safe and stable operation of the power grid, improve operational efficiency, and realize data-driven business. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] Therefore, this invention provides a data governance method based on an intelligent decision engine, which solves the problem that existing technologies mainly rely on static, predefined data governance rules, making it difficult to adaptively handle modern data environments that are multi-source, heterogeneous, highly concurrent, and have ever-changing business needs. This results in low governance efficiency, weak decision-making capabilities, and the inability to form a dynamically optimized governance closed loop.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] This invention provides a data governance method based on an intelligent decision engine, comprising:

[0009] Real-time streaming data is accessed from multiple heterogeneous data sources and subjected to preliminary cleaning and format conversion.

[0010] Perform business semantic classification and tagging on the pre-processed real-time streaming data to build a data profile library;

[0011] By identifying data entities and their corresponding relationships, a dynamic topology graph is constructed. Based on graph neural networks, the anomaly propagation pattern is learned, the impact of entity failures on related entities is analyzed, and an influence grading mechanism is established.

[0012] Based on business rules and machine learning models, the data quality is assessed in multiple dimensions to generate dynamic data quality scores and grading strategies; a database of historical data quality problems and causes is built, and a large language model agent is used to generate a report on the root cause judgment and impact quantification of quality anomalies.

[0013] A set of candidate strategies is generated based on the scope of impact, severity, and causes of anomalies in data quality issues. The optimal governance strategy is then selected from the set of candidate strategies by establishing a multi-objective optimization model.

[0014] The large-scale language model agent is used to perform compliance checks and conflict identification on the optimal governance strategy, and a reinforcement learning algorithm is used to dynamically adjust the decision weights of the rule engine to achieve a closed loop of data governance and dynamic learning.

[0015] As a preferred embodiment of the data governance method based on an intelligent decision engine described in this invention, the step of learning the anomaly propagation law based on graph neural networks, analyzing the impact of entity failures on related entities, and establishing an influence grading mechanism includes:

[0016] Based on the dynamic topology graph, the four types of features of each entity are combined into a node feature vector and input into the spatiotemporal graph neural network model, which outputs the probability that each node is affected by an anomaly. The spatiotemporal graph neural network model uses historical anomaly propagation cases as training samples, defines the anomaly propagation task as a node classification or regression task, and trains the model to learn and simulate the propagation path and impact range of anomalies in the topology graph.

[0017] Based on the probability of each node being affected by an anomaly, the influence value of a node is quantified by analyzing the propagation score distribution of other nodes when a certain node is the starting point of a fault.

[0018] Based on the first quantitative standard and the obtained influence value of the node, the node influence is divided into three levels: high, medium and low, to clarify the priority of fault handling; the first quantitative standard considers the number of affected entities, the time it takes for the anomaly to spread to the entire area, and the business criticality of the affected entities.

[0019] As a preferred embodiment of the data governance method based on an intelligent decision engine described in this invention, the step of performing multi-dimensional quality assessment of data based on business rules and machine learning models to generate dynamic data quality scores and grading strategies includes:

[0020] We define three core data quality assessment dimensions: completeness, accuracy, and consistency. For each dimension, we build an executable verification rule library that includes pre-event rules, in-event rules, and post-event rules. The rules are derived from the specific content written by business personnel for each business domain and functional module.

[0021] The data extracts indicators reflecting the problems of each quality dimension as features, and obtains manually labeled or historical case data as labels. These are then input into the extreme gradient boosting model to calculate the quality score. Based on the output of the extreme gradient boosting model, the contribution weights of completeness, accuracy, and consistency are automatically calculated.

[0022] The rule engine loads all rules from the executable validation rule library, performs batch validation on the preprocessed data, and outputs the rule violation status for each piece of data.

[0023] Based on the verification results of the rule engine and the contribution weight, a comprehensive quality score is calculated for each piece of data. According to the preset dynamic threshold, the comprehensive quality score is divided into three levels: urgent defect, general defect and high quality, and corresponding governance priority labels of high risk, medium risk and low risk are generated.

[0024] As a preferred embodiment of the data governance method based on an intelligent decision engine described in this invention, the step of generating a report on the root cause judgment and impact quantification of quality anomalies using a large-scale language model agent includes:

[0025] The identified high-risk abnormal data and the topological relationships of abnormal data are used as input to the large-scale language model agent. Utilizing language understanding and reasoning capabilities, combined with information from the data quality historical problem and cause case library, the contextual information of the abnormal data is analyzed, and similar historical cases are retrieved based on the characteristics of the current abnormality. By integrating the dynamic topology graph, contextual information, and similar historical cases, the root cause judgment of the data quality anomaly is inferred and generated.

[0026] Based on the topology impact analysis results, identify and quantify the specific datasets, tables, fields, and number of records affected by the current anomaly; generate an impact quantification report, which includes the technical and business reasons for the data quality anomaly, a list of affected datasets and records and their degree of impact, and a preliminary risk assessment.

[0027] As a preferred embodiment of the data governance method based on an intelligent decision engine described in this invention, the step of generating a candidate strategy set based on the impact scope, severity, and causes of anomalies in data quality issues includes:

[0028] Data governance strategies are divided into automated repair strategies, semi-automated repair strategies, and manual intervention / source governance strategies, and the impact scope parameter R, severity parameter Q, and anomaly cause parameter P of data quality issues are defined respectively.

[0029] Based on the actual values ​​of the influence range parameter R, severity parameter Q, and abnormal cause parameter P, candidate strategies that meet the constraints are selected from the three types of strategies.

[0030] Based on the results of the filtering, a candidate strategy set containing all strategies that satisfy the constraints is generated.

[0031] As a preferred embodiment of the data governance method based on an intelligent decision engine described in this invention, the step of selecting the optimal governance strategy from the candidate strategy set by establishing a multi-objective optimization model includes:

[0032] Establish a multi-objective optimization model with the goals of minimizing repair costs, maximizing reliability, and minimizing risks;

[0033] Based on the input impact range, severity, abnormal cause parameters and impact weights, the Pareto optimality algorithm is used to solve the multi-objective optimization model and output the Pareto optimal solution set, which includes all non-dominated solutions and is visualized on the Pareto front curve.

[0034] From the Pareto optimal solution set, a final governance strategy is selected based on the priority and risk preference of the current business scenario. The selection process is implemented through a rule engine, which compares the values ​​of different solutions in terms of cost, reliability and risk objectives according to preset business rules, and outputs the recommended optimal governance strategy.

[0035] As a preferred embodiment of the data governance method based on an intelligent decision engine described in this invention, the compliance verification and conflict identification of the optimal governance strategy includes:

[0036] The optimal governance strategy is input into the large language model agent for compliance checks. The compliance checks include verifying whether the data correction scope and rule adjustment suggestions comply with power data standards and security protocols, and analyzing the performance of the strategy in similar historical cases.

[0037] The large language model agent uses its reasoning ability to identify high-risk operations in the optimal governance strategy that may lead to business interruption, data inconsistency, or coupling conflicts with existing systems. For strategies whose impact is difficult to judge directly, the large language model agent automatically generates supplementary data verification or simulation test instructions to obtain more comprehensive risk assessment data.

[0038] For the optimal governance strategy identified as high-risk or high-conflict, the system forcibly triggers a manual review process, providing the detailed diagnostic report, compliance analysis, and risk assessment results generated by the large language model agent to business experts for final decision-making; the business experts' adoption, rejection, or modification opinions are recorded and fed back to the large language model agent as training data.

[0039] As a preferred embodiment of the data governance method based on an intelligent decision engine described in this invention, the preliminary cleaning and format conversion includes:

[0040] Convert the source system's data types to the standard types required by the target system;

[0041] Standardize the processing of different date formats, numerical precision, and text case;

[0042] Perform preliminary repairs or mark obvious null values ​​and simple formatting errors to ensure that the data is parsable;

[0043] Historical change information is provided by capturing the facts of data manipulation language changes and the changed data itself;

[0044] Use regular expressions to perform preliminary data identification and data format conversion.

[0045] As a preferred embodiment of the data governance method based on an intelligent decision engine described in this invention, the construction of the data profile library includes:

[0046] Associate and map the collected physical metadata with business metadata;

[0047] Model experts and business personnel label data tables, fields, and datasets based on the business object ID, business object name, first-level business category, second-level business category, business capability, business process, and operation-level process module of each business domain, and form a data business catalog.

[0048] Information about each data asset is stored according to the data service catalog to form a data profile library.

[0049] As a preferred embodiment of the data governance method based on an intelligent decision engine described in this invention, the step of dynamically adjusting the decision weights of the rule engine using a reinforcement learning algorithm includes:

[0050] The adoption of governance strategies by human experts in the review process is defined as a positive reward signal, and rejection is defined as a negative penalty signal; the specific corrective opinions proposed by human experts are quantified into reward signals of different intensities; the status, actions and corresponding reward signals in historical review records are collected to construct a reinforcement learning experience replay dataset.

[0051] A Q-function is established, taking the decision state of the rule engine as input and aiming to maximize long-term cumulative rewards; the Q-function is updated through a temporal difference learning algorithm to dynamically evaluate the expected utility of each rule under different data quality problem scenarios; and the decision weights of the corresponding rules in the rule engine are adjusted according to the updated Q-value.

[0052] The reinforcement learning agent is deployed in an online learning mode to absorb newly generated review feedback data in real time and update decision weights immediately. A rule version control system is established to record and archive the history of weight adjustments, reasons for adjustments, and adjustment times for each rule. The rule base is scanned regularly to mark or archive rules that have been in a low-weight state for a long time and are ineffective.

[0053] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a data governance method based on an intelligent decision engine. By constructing a collaborative decision-making framework that integrates a rule engine, a machine learning model, and a large language model intelligence agent, it achieves intelligent and adaptive closed-loop management of the entire data governance process. This invention can perceive the state changes of multi-source heterogeneous data in real time, deeply trace the root causes of data anomalies, and dynamically generate precise governance strategies based on business scenarios. Its advantage lies in upgrading governance actions from passive response to proactive prediction and dynamic optimization, significantly improving the refinement level and reliability of data quality management and decision-making. Simultaneously, this invention possesses high interpretability and business relevance; all governance decisions support reverse tracing and logical verification, ensuring the transparency and credibility of the governance process. Through continuous learning and feedback optimization mechanisms, it continuously accumulates governance experience and internalizes it into rule knowledge, forming a spiraling governance capability, ultimately providing a solid and reliable data foundation for business innovation and risk control. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of the overall process logic of a data governance method based on an intelligent decision engine, provided as an embodiment of the present invention. Detailed Implementation

[0056] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0057] Example 1, referring to Figure 1 As one embodiment of the present invention, a data governance method based on an intelligent decision engine is provided, such as... Figure 1 The specific steps shown are as follows:

[0058] S100: Intakes real-time streaming data from multiple heterogeneous data sources and performs preliminary cleaning and format conversion;

[0059] S200: Perform business semantic classification and tagging on the pre-processed real-time streaming data to build a data profile library;

[0060] S300: By identifying data entities and their corresponding relationships, a dynamic topology graph is constructed. Based on graph neural networks, the anomaly propagation law is learned, the impact of entity failures on related entities is analyzed, and an influence grading mechanism is established.

[0061] S400: Based on business rules and machine learning models, perform multi-dimensional quality assessment of data to generate dynamic data quality scores and grading strategies;

[0062] S500: Construct a case library of historical data quality problems and causes, and use a large language model agent to generate a report on the root cause judgment and impact quantification of quality anomalies.

[0063] S600: Generate a set of candidate strategies based on the scope of impact, severity and cause of anomalies of data quality issues, and select the optimal governance strategy from the set of candidate strategies by establishing a multi-objective optimization model;

[0064] S700: It uses a large language model agent to perform compliance checks and conflict identification on the optimal governance strategy, and adopts reinforcement learning algorithms to dynamically adjust the decision weights of the rule engine, thereby realizing a closed loop of data governance and dynamic learning.

[0065] It should be noted that steps S100-S700 above construct a collaborative decision-making framework integrating a rule engine, machine learning model, and large language model intelligence agent to achieve intelligent and adaptive closed-loop management of the entire data governance process. This invention can perceive the state changes of multi-source heterogeneous data in real time, deeply trace the root causes of data anomalies, and dynamically generate precise governance strategies based on business scenarios. Its advantage lies in upgrading governance actions from passive response to proactive prediction and dynamic optimization, significantly improving the refinement level and reliability of data quality management and decision-making. Simultaneously, this invention possesses high interpretability and business relevance; all governance decisions support reverse tracing and logical verification, ensuring the transparency and credibility of the governance process. Through continuous learning and feedback optimization mechanisms, governance experience is continuously accumulated and internalized into rule knowledge, forming a spiraling governance capability, ultimately providing a solid and reliable data foundation for business innovation and risk control.

[0066] In this embodiment of the invention, step S100 includes the following sub-steps A1 and A2:

[0067] In A1: Real-time streaming data is accessed from multiple heterogeneous data sources;

[0068] Specifically, multiple data sources include power grid dispatch automation systems, relay protection and safety automatic devices, and equipment monitoring systems. The industry mainly uses the Kafka message bus protocol for real-time streaming data acquisition.

[0069] Specifically, data is collected from power grid equipment in JSON standard format, preprocessed, and then sent to a Kafka cluster; the JSON format data accessed by the Kafka cluster has the following specific field descriptions:

[0070] table: Table name, including schema and table name. Value must be uppercase, e.g., USER1.TABLE1;

[0071] op_type: Operation type, with values ​​I, U, D, and DDL, representing whether the data record is inserted, updated, deleted, or modified via DDL, respectively. The value must be in uppercase.

[0072] op_ts: Database operation time, e.g., 2021-02-04 12:17:49.00006;

[0073] current_ts: The time when the message was sent to Kafka, e.g., 2021-02-04T20:16:55.018000;

[0074] pos: The offset of the music compressed OGG file record, which is monotonically increasing globally within the Kafka topic, e.g. 00000000010000112006;

[0075] primary_keys: The primary key field names of the source records. Multiple primary key fields are separated by commas. It is an array. The value must be uppercase, for example: ["ID","BI_SFDM"];

[0076] before: This structure is used to store information about the original table before the changes. It only exists when op_type is U or D. It is a JSON structure: {"field name 1:field value 1", "field name 2:field value 2"}. The field name contains all the fields of the table. The field name is uppercase and the value is the value of the corresponding field.

[0077] after: Used to store information after changes to the original table. This structure only exists when op_type is I, U, or DDL. It is a JSON structure: {"field name 1:field value 1", "field name 2:field value 2"}. The field name contains all the fields of the table. The field name is uppercase and the value is the value of the corresponding field.

[0078] Specifically, for systems such as power grid management platforms, digital operation and control platforms, renewable energy subsidy contract management systems, and invoice centers, batch data extraction is performed on relational databases using JDBC (Application Programming Interface for Java programs to access relational databases) on a timed or triggered basis, supporting full extraction and incremental extraction based on timestamps or update time fields.

[0079] In A2: the collected real-time streaming data undergoes preliminary cleaning and format conversion; specific steps include:

[0080] Convert the source system's data types to the standard types required by the target system;

[0081] Standardize different date formats, numerical precision, and text case;

[0082] Perform preliminary repairs or mark obvious null values ​​and simple formatting errors to ensure that the data is parsable;

[0083] Historical change information is provided by capturing the facts of data manipulation language changes and the changed data itself;

[0084] Regular expressions are used for initial identification and conversion of data formats.

[0085] In this embodiment of the invention, step S200, which involves classifying and tagging the pre-processed real-time streaming data according to business semantics to construct a data profile library, includes:

[0086] Associate and map the collected physical metadata with business metadata;

[0087] Model experts and business personnel tagged data tables, fields, and datasets based on modules such as business object ID, business object name, primary business category, secondary business category, business capability, business process, and operational process of each business domain, and formed a data business catalog.

[0088] Store detailed metadata, business tags, lineage, access permissions, and other information for each data asset to form a data profile library.

[0089] In this embodiment of the invention, step S300 includes the following sub-steps C1 to C3:

[0090] In C1: A dynamic topology graph is constructed by identifying data entities and their corresponding relationships;

[0091] Specifically, the most critical entities in the business are identified from contextualized data and used as nodes in a graph convolutional neural network (CNN). The connections between these entities are then identified as edges connecting the nodes in the CNN. These connections include: ① Explicit relationships: Directly observable relationships, such as: the substation field in a transmission line table in a database, linked to the primary key ID in a substation equipment table → the corresponding line includes that substation. Data lineage relationships, such as: the upstream of identified equipment data is a substation real-time acquisition system → the corresponding equipment data depends on the substation system. ② Implicit relationships: Relationships that need to be inferred through rules or analysis, such as: business rule: customers in the same area usually use the same substation for power supply → inferring that customers are associated with a substation. Data value association: the real-time current data of device A and device B always fluctuate synchronously → inferring that device A and device B have a transmission connection.

[0092] Specifically, assign values ​​to the relationships between entities, using numerical values ​​to represent the importance of the association between the two entities; if the entities are device A and device B, the weight can be the frequency with which both of them experience anomalies, the higher the frequency, the closer the association, and the greater the weight; if the entities are a customer and a device, the weight can be the frequency with which the customer uses the device, the more frequently it is used, the greater the weight.

[0093] Specifically, the graph structure of a dynamic topology graph is not fixed and will be updated in real time as business data changes.

[0094] For example, if a new substation is put into operation, a new substation node is added to the graph, and its contained lines are automatically associated; if a certain line is switched to be powered by another substation, the original line-substation edge is deleted, and a new line-new substation edge is added; if the synchronization fluctuation frequency of device A and device B decreases, the weight of the edge between them is reduced.

[0095] In C2: the anomaly propagation pattern is learned through a graph neural network based on a dynamic topology graph; specifically, this includes the following sub-steps C21~C23:

[0096] In C21: Based on the dynamic topology graph, the four types of features of each entity are combined into a node feature vector and input into the spatiotemporal graph neural network model, which outputs the probability that each node is affected by anomalies.

[0097] Specifically, each entity has four types of characteristics, including initial data quality score, historical anomaly rate, business attributes, and time series characteristics.

[0098] Initial data quality scores, such as data integrity and accuracy, reflect the reliability of the entity's data. Nodes with poor quality are more likely to trigger anomaly propagation. For example, if the real-time current field integrity of a line table is less than 80%, it may indicate a data collection anomaly or a source database failure. Historical anomaly rate is the frequency of past anomalies. For example, if multiple instances in the past month show that the warehouse entry time of a certain supply chain domain table is longer than the warehouse exit time, and the data accuracy is low, it reflects the entity's vulnerability, making this node more likely to become the starting point for propagation. Business attributes and inherent business characteristics determine the ability of anomaly propagation. For example, anomalies in a 220kV substation may have a wider impact than anomalies in a 10kV line because of the higher voltage level and the larger number of users covered. Time-series characteristics changing over time, such as the current fluctuation trend of a line over the past hour and the peak load period of a substation, reflect the "timing characteristics" of anomalies. For example, line anomalies during peak electricity consumption periods, such as 6-8 PM, may have a higher probability of occurrence and greater propagation impact than those during off-peak periods because the higher load makes chain reactions more likely.

[0099] In C22: The spatiotemporal graph neural network model uses historical anomaly propagation cases as training samples, defines the anomaly propagation task as a node classification or regression task, and trains the model to learn and simulate the propagation path and impact range of anomalies in the topology graph.

[0100] It should be noted that the spatiotemporal graph neural network model uses historical anomaly propagation cases as training samples, enabling it to learn and simulate the propagation path and impact range of data anomalies in the topology graph. For example, when the data quality of a certain line device becomes abnormal, it can predict the substations, distribution areas, and related customer metering data that may be affected. This is accomplished through simulation training of anomaly propagation on the graph, or by defining anomaly propagation as a node classification and regression task.

[0101] Specifically, training the spatiotemporal graph neural network model includes:

[0102] Collect complete cases of abnormal propagation from historical data, such as short circuit in line L1 → tripping of substation S1 → power outage in distribution area C1, and convert them into training samples for a spatiotemporal graph neural network model. Input the features and topology structure of the abnormal starting node into the model, output the affected nodes, and label them as affected or unaffected.

[0103] Predict whether each node will be affected by anomalies (binary classification: 0 = unaffected, 1 = affected); predict the probability of each node being affected, such as the probability that substation S1 is affected is 80%. Example: In a power grid, when line L2 experiences data anomalies such as a surge in current, the model can learn to output the propagation path and probability of substation S2 → distribution line D1 (probability 60%) → residential area C1.

[0104] In C23: Based on the probability that each node is affected by an anomaly, the influence value of a node is quantified by analyzing the propagation score distribution of other nodes when a certain node is the starting point of a fault.

[0105] It should be noted that adding time and space dimensions makes the analysis more accurate. Anomaly propagation is not only affected by topological relationships, but also closely related to when and where it occurs. Therefore, it is necessary to combine spatiotemporal information.

[0106] Specifically, the speed of anomaly propagation varies over time, with anomalies spreading faster during peak periods. This can be addressed by incorporating temporal features into the model, such as using LSTM to process the historical state sequences of nodes, or by directly using a spatiotemporal graph neural network, allowing the model to learn time dependencies.

[0107] Specifically, the propagation of anomalies is limited by geographical location; for example, equipment in adjacent areas is more likely to affect each other. Geographic Information System (GIS) data can be introduced, with physical distance included as part of the edge weights, allowing the model to learn the impact of spatial proximity on propagation. For example, a substation within 1 kilometer of a faulty line is more likely to be affected than one more than one 5 kilometers away.

[0108] In C3: Analyze the impact of entity failures on related entities and establish an impact grading mechanism;

[0109] It should be noted that the goal of node impact assessment is to numerically measure the destructive power of a single entity's failure on adjacent entities and even the entire system—that is, the degree to which it causes data distortion and functional abnormalities in other entities. This assessment relies on the output of a spatiotemporal graph neural network model, and there are two commonly used methods:

[0110] ① Evaluation of Node Embeddings Based on Graph Convolutional Neural Networks (GNNs): During training, node embedding vectors are generated. These vectors encode information such as the node's topological position in the graph, its association strength with other nodes, and its own characteristics. The importance of the embedding vector directly reflects the node's influence: if a node's embedding vector has high similarity to the embedding vectors of more nodes in the entire graph, or if it occupies a core position in the vector space, it indicates that the node holds a pivotal position in the topology, and its impact after a fault is wider. For example, in a 220kV hub substation in a power grid, the node embeddings connecting multiple transmission lines and distribution areas are usually more core than those of ordinary 10kV lines, and its fault impact is greater.

[0111] ② Propagation Score Evaluation Based on Graph Convolutional Neural Network (GNN): When predicting anomaly propagation, the probability of each node being affected (i.e., propagation score) is output. By analyzing the distribution of propagation scores of other nodes when a node is the starting point of a fault, its influence can be quantified: if after node A fails, 10 related nodes have an impact probability exceeding 80%, while after node B fails, only 2 related nodes have an impact probability exceeding 80%, then the influence of node A is greater than that of node B. Example: When transmission line L1 fails, GNN predicts that 5 nodes, including substation S1 and distribution area C1, have an impact probability > 90%; while when branch line L2 fails, only 1 user node has an impact probability > 50%, so the influence of L1 is significantly higher than that of L2.

[0112] In this embodiment of the invention, based on a first quantification standard and the obtained node influence value, the node influence is divided into three levels: high, medium, and low, to clarify the priority of fault handling. The first quantification standard considers the number of affected entities, the time it takes for the anomaly to spread to the entire area, and the business criticality of the affected entities. Examples of the classification in the power grid scenario are shown in Table 1.

[0113] Table 1: Examples of hierarchical classification in power grid scenarios;

[0114]

[0115] It should be noted that: high-impact entities require priority resource allocation (such as deploying real-time monitoring, redundant backups, and regular maintenance) to reduce their failure probability; if a failure occurs, an emergency response should be initiated immediately. For medium-impact entities, a regular inspection plan should be developed, and moderate resource support should be allocated; after a failure, procedures should be followed, but the scope of spread must be controlled. For low-impact entities, conventional governance strategies should be adopted, resource allocation can be appropriately reduced, and after a failure, standard procedures should be followed for repair.

[0116] It should be noted that step S300 above constructs dynamic topological relationships and analyzes the anomaly propagation patterns through graph neural networks, achieving a leap from isolated data monitoring to interconnected network governance. It can accurately identify key entities and predict potential risk propagation paths, greatly enhancing the system's ability to recognize and warn of complex data relationships.

[0117] In this embodiment of the invention, step S400 includes the following sub-steps D1 to D3:

[0118] In D1: Define three core data quality assessment dimensions: completeness, accuracy, and consistency. For each dimension, build an executable verification rule library containing pre-event rules, in-event rules, and post-event rules. The rules are derived from the specific content written by business personnel for each business domain and functional module.

[0119] Specifically, completeness involves determining whether data is missing; for example, the contractor field in the basic contract information cannot be empty, the validity of the associated investment project ID in the single annual investment plan information in the project center must be verified, and the project name, project establishment year, construction address, and funding nature of distribution network infrastructure projects cannot be empty.

[0120] Specifically, accuracy refers to whether data values ​​conform to actual business facts and are within a reasonable range. For example, the weekly maintenance and repair plan in the safety monitoring domain must be derived from the monthly work plan in the production technology domain; otherwise, it is considered a temporary work and requires a special approval process.

[0121] Specifically, consistency refers to consistency in business rules and cross-system data. Examples include: distribution transformer consistency: no data in the equipment center, but available in marketing; station / line / transformer / customer consistency: consistent between the equipment center and marketing line / transformer; distribution transformer consistency: inconsistent rated capacity between the equipment center and marketing distribution transformers; and distribution transformer consistency: available in the equipment center, but not in marketing.

[0122] Specifically, pre-process rules are rules that take effect before data enters the system, with the core objective of preventing quality issues at the source and reducing subsequent governance costs. During-process rules are rules that take effect in real time during data transmission, processing, and use, with the core objective of promptly identifying and intervening in ongoing quality issues to prevent their spread and impact on downstream business. Post-process rules are rules that take effect after data is generated and business processing is completed, with the core objective of analyzing the root causes of quality issues, optimizing preceding rules, and forming a closed loop of "discovery-improvement-prevention."

[0123] In D2: Indicators reflecting the problems of each quality dimension are extracted from the data as features, and manually labeled or historical case data are obtained as labels. These are then input into the extreme gradient boosting model to calculate the quality score. The contribution weights of completeness, accuracy and consistency are automatically calculated based on the output of the extreme gradient boosting model.

[0124] Specifically, indicators that reflect quality issues are extracted from the data and used as input to the extreme gradient boosting model, as shown in Table 2:

[0125] Table 2: Indicators Reflecting Quality Issues;

[0126]

[0127] Specifically, the model needs standard answers to learn from, and the label data comes from manual annotation and historical cases; among them, manual annotation is business experts labeling a small amount of data as good or problematic; historical cases are confirmed problematic data as negative samples (0) and normal data as positive samples (1).

[0128] Specifically, indicators reflecting problems in each quality dimension are extracted from the data as features, and manually labeled or historical case data are obtained as labels. These are then input into the extreme gradient boosting model to calculate the quality score. The contribution of the three dimensions is automatically calculated through the feature importance output of the model.

[0129] In D3: The rule engine loads all rules in the executable verification rule library, performs batch verification on the preprocessed data, and outputs the rule violation status of each data item; the comprehensive quality score of each data item is calculated by combining the verification results and contribution weight of the rule engine; based on the preset dynamic threshold, the comprehensive quality score is divided into three levels: urgent defect, general defect and high quality, and corresponding governance priority labels of high risk, medium risk and low risk are generated.

[0130] Specifically, based on the comprehensive score, and considering the real-time nature and importance of the power business data, quality levels are dynamically assigned:

[0131] When the overall score is less than 60, it is considered an emergency defect (high risk): for example, "the accuracy score of the current monitoring data of a certain substation is only 50 points, and it involves the main line", which needs to be repaired immediately, otherwise it may cause dispatch misjudgment.

[0132] When the overall score is between 60 and 80, it is considered a general defect (medium risk): for example, "the integrity score of a certain branch line is 70 (a small amount of data is missing)", it needs to be investigated within 24 hours to prevent the problem from escalating.

[0133] A comprehensive score >80 indicates high quality (low risk): for example, "the comprehensive score of user measurement data is 85 points". Only continuous monitoring is required, and no emergency action is needed.

[0134] It should be noted that step S400 above performs multi-dimensional quality assessment by integrating the rule engine and machine learning model, realizing the evolution of data quality quantitative evaluation from static threshold judgment to dynamic intelligent scoring, making the quality assessment results more accurate, comprehensive and able to reflect business priorities.

[0135] In this embodiment of the invention, step S500 includes the following sub-steps E1 to E4:

[0136] In E1: Build a case library of historical data quality issues and their causes;

[0137] Specifically, define a unified system for classifying data quality issues and their causes, including:

[0138] The business reasons include data entry errors due to human error, inconsistent data labels for the same field across multiple systems, unsynchronized metadata changes, and changes in business processes.

[0139] Technical reasons include excessive workload of strategy execution leading to data update delays, network or hardware failures causing data loss, data acquisition failures, data format compatibility issues, data conversion errors, system code defects, interface definition inconsistencies, and database crashes.

[0140] Specifically, information is extracted from historical data quality defect work orders, operation and maintenance logs, and fault reports. This includes: storing each case in a structured manner, including at least: problem description, occurrence time, involved data tables and fields, quality score, topology impact, diagnostic cause, repair process, and repair result. Some case information, especially entities, relationships, and events, is integrated into a knowledge graph to enhance its reasoning capabilities.

[0141] Specifically, structured case data is converted into a text format suitable for large language model (LLM) learning, serving as corpus for LLM fine-tuning or contextual learning, enabling it to understand and reason about the root causes of data quality problems.

[0142] In E2: The identified high-risk abnormal data and the topological relationships of abnormal data are used as input to a large language model agent. By utilizing language understanding and reasoning capabilities, combined with information from the database of historical data quality problems and cause cases, the contextual information of abnormal data is analyzed, and similar historical cases are retrieved based on the characteristics of the current abnormality.

[0143] Specifically, the large-scale intelligent agent architecture possesses the following core capabilities: ① It decomposes diagnostic tasks into sub-tasks, such as querying details of abnormal data, querying topological relationships, and querying historical cases. ② It can call external tools or APIs, such as calling influence analysis result APIs, calling historical case library query APIs, and calling data exploration APIs to obtain data context. ③ It possesses knowledge and experience in short-term dialogue history and persistent storage.

[0144] Specifically, the intelligent agent leverages its language understanding and reasoning capabilities, combined with a knowledge graph, to analyze the context of anomalous data, such as: which business process the data belongs to, related maintenance records, weather information at the time, and operation and maintenance logs, to determine whether the data conforms to the expected semantics and logic. It also accesses a database of historical data quality issues and causes, retrieving similar historical cases based on the current anomaly's data type, fields, anomaly pattern, scope of impact, and historical business scenarios.

[0145] In E3: By combining dynamic topology graphs, contextual information, and similar historical cases, the root cause judgment of data quality anomalies is inferred and generated;

[0146] For example, "The equipment monitoring system failed to collect data, resulting in the loss of operating data for distribution network equipment A. The reason is that there was a sudden network failure in area B yesterday." or "The customer information was entered incorrectly because the front-end system did not perform format verification on the mobile phone number, resulting in the customer's mobile phone number being in an incorrect format."

[0147] In E4: Based on the topology impact analysis results, identify and quantify the specific datasets, tables, fields, and number of records affected by the current anomaly; generate an impact quantification report, which includes the technical and business reasons for the data quality anomaly, a list of affected datasets and records and their impact, and a preliminary risk assessment;

[0148] It should be noted that step S500 above transforms traditional fault diagnosis, which relies on human experience, into automated and intelligent accurate attribution by building a historical case library and using a large model intelligent agent for root cause analysis, thereby significantly improving the efficiency of problem location and the reliability of diagnostic conclusions.

[0149] In this embodiment of the invention, step S600 includes the following sub-steps F1 to F6:

[0150] In F1: Data governance strategies are divided into automated remediation strategies, semi-automated remediation strategies, and manual intervention / source governance strategies;

[0151] In this embodiment of the invention, the steps for triggering the automated repair strategy are as follows:

[0152] ① The system automatically calls the corresponding repair method based on the type of abnormal data it determines;

[0153] If the missing value is an anomaly, the "Context Completion" method is called to automatically complete the missing context;

[0154] If the value is invalid or redundant, the "normal cleaning" method will be called to clean the abnormal data.

[0155] If the data structure format is abnormal, the "format conversion" method is called to convert the non-standard format data into standard format data.

[0156] ② Steps to execute the automatic repair method:

[0157] For missing values ​​with clear contextual dependencies, imputation is derived based on historical or related data;

[0158] Standardized cleaning is performed on invalid, redundant, and duplicate values.

[0159] Transform non-standard data into standard data;

[0160] ③ The repaired data must pass the pre-set verification rules to ensure that the repaired data meets the business standards. If the verification passes, the data is updated; if it fails, it is automatically marked with the "requires semi-automatic processing" label.

[0161] ④ Log recording and strategy optimization: Automatically store the repair process and complete the historical case library; regularly analyze the repair effect through machine learning models, optimize repair rules, improve the accuracy of various parameter definitions, adjust weight parameters, and improve the accuracy of automated repair.

[0162] In this embodiment of the invention, the steps for triggering the semi-automatic repair strategy are as follows:

[0163] ① Anomaly identification and range localization: The system locates abnormal data through real-time monitoring tools, analyzes the scope of the anomaly's impact by combining the constructed topology map information, and quickly confirms the cause of the abnormal data through large-scale intelligent agent anomaly detection and diagnosis, thus assisting human decision-making.

[0164] ②The system performs initial repairs based on the scenario:

[0165] In cases involving complex data repair, the system generates preliminary repair suggestions by pre-defining business logic rules; in cases involving data link breakpoint resumption, the system automatically detects the breakpoint location, attempts to reconnect the link, and executes breakpoint resumption.

[0166] ③ After the initial system repair, manually review and process the data that the system cannot repair, including: confirming the rationality of the system repair logic; checking whether the results of the initial system repair are consistent with the actual business; and manually handling steps that the system cannot complete.

[0167] ④ Repair record archiving and strategy optimization: Automatically store the repair process and complete the historical case library; regularly analyze the repair effect through machine learning models, optimize repair rules, improve the accuracy of various parameter definitions, adjust weight parameters, and improve the accuracy of automated repair.

[0168] In this embodiment of the invention, the steps for triggering the manual intervention / source control strategy are as follows:

[0169] ① Locate the area where abnormal data is generated by constructing a topology map and identifying problems, combined with data monitoring tools; rule out technical reasons as the cause of the anomaly by combining factors such as business rules, table data specifications, and abnormal system log records; and detect whether the identified abnormal data is high-risk business data.

[0170] ② Root cause analysis to identify the fundamental reasons at the business level (such as repetitive processes in single-domain business scenarios or inconsistent standards in cross-domain business scenarios), and resolve the causes of data anomalies at their source:

[0171] In cross-departmental collaboration, the node where the anomaly first appears and its neighboring nodes are located by constructing a topology map to determine whether the upstream system of the data is abnormal; business analysis methods are used to delve into the causes, optimize business processes, and resolve inconsistencies or conflicts in cross-domain business processes.

[0172] ③ Develop and optimize the rules for system entry, strengthen employee training on data entry, and reduce the generation of abnormal data during system entry; upgrade existing system functions and fix system defects; optimize existing cross-domain conflict and inconsistent standard processes.

[0173] ④ Solution Implementation: The business department takes the lead in organizing operational training and updating data entry standards; the digitalization department schedules development, and the business department participates in requirements review and testing; the new process is confirmed through cross-departmental meetings to form a standard operating procedure.

[0174] In F2: Define the impact range parameter R, severity parameter Q, and anomaly cause parameter P for data quality issues respectively; based on the actual values, select candidate strategies that meet the constraints from the three types of strategies;

[0175] In F3: Based on the filtering results, a candidate strategy set containing all strategies that satisfy the constraints is generated;

[0176] In F4: Establish a multi-objective optimization model with the goals of minimizing repair costs, maximizing reliability, and minimizing risk;

[0177] In F5: Based on the input impact range, severity, abnormal cause parameters and impact weights, the Pareto optimality algorithm is used to solve the multi-objective optimization model and output the Pareto optimal solution set, which contains all non-dominated solutions and is visualized on the Pareto frontier curve.

[0178] In F6: From the Pareto optimal solution set, the final governance strategy is selected based on the priority and risk appetite of the current business scenario. The selection process is implemented through a rule engine, which compares the values ​​of different solutions in terms of cost, reliability, and risk objectives according to preset business rules, and outputs the recommended optimal governance strategy. Specifically, the basic attributes of the governance strategy are preset according to Table 3.

[0179] Table 3: Basis for Pre-setting Basic Attributes of Governance Strategies;

[0180]

[0181] For example, the following is the calculation process for the output recommended optimal governance strategy:

[0182] The steps for defining strategy-related parameters are as follows: ① Strategy A (Automatic Repair): Cost: 10, Integrity Compliance Rate I: 0.7, Accuracy Compliance Rate A: 0.65, Compliance Compliance Rate C: 0.75, Integrity Risk Control Rate 0.4, accuracy risk control rate 0.5, compliance risk control rate :0.3; ② Strategy B (Semi-automatic Repair): Cost: 30, Completeness Compliance Rate I: 0.85, Accuracy Compliance Rate A: 0.8, Compliance Compliance Rate C: 0.9, Completeness Risk Control Rate 0.6, accuracy risk control rate Compliance risk control rate: 0.7 Strategy C (Manual Intervention / Source Governance): Cost: 60, Completeness Compliance Rate I: 0.95, Accuracy Compliance Rate A: 0.9, Compliance Compliance Rate C: 1, Completeness Risk Control Rate Accuracy risk control rate: 0.8 Compliance risk control rate: 0.9 0.9.

[0183] A certain data anomaly scenario has an uncontrollable impact range, is assessed as medium risk, and is determined to be caused by technical reasons. The impact weight (accuracy priority) is K=(0.2,0.5,0.3).

[0184] Candidate strategies were selected: Strategy A requires: controllable scope of impact and low risk → met; Strategy B requires: uncontrollable scope of impact, medium risk, technical reasons → met; Strategy C requires: uncontrollable scope of impact, high risk, business reasons → not met; the candidate strategies obtained were B and A.

[0185] The steps to calculate the policy objective value A are as follows: Cost(A): directly take the base value = 10; Reliability(A) = ·I+ ·A+ • C; Input data: Reliability(A) = 69; Risk(B) = 100 − ( · + · + · ); Substitute the data Risk(A) = 59. Wherein, The weight representing the impact of the completeness compliance rate. This indicates the weighting of the impact on the accuracy achievement rate. This indicates the weighting of the impact on the compliance rate.

[0186] The steps to calculate the strategy target value B are as follows: Cost(B): directly take the base value = 30; Reliability(B) = ·I+ ·A+ •C; Reliability(B)=84; Risk(B)=100−( · + · + · Risk(B) = 38.

[0187] Since A and B are mutually exclusive, both strategies are Pareto optimal solutions.

[0188] Furthermore, based on the boundary scenarios and weights, strategy B has higher reliability (84>69), so strategy B is recommended.

[0189] It should be noted that strategy A is an option for low-risk scenarios (Q>80) due to its lower cost; strategy B is an option for medium-risk scenarios, which balances cost and reliability; and strategy C is the mandatory option for high-risk scenarios, which focuses on restoring reliability but has a higher cost.

[0190] It should be noted that step S600 above generates the optimal governance strategy through a multi-objective optimization model, which can scientifically select the best solution under multiple constraints such as cost, risk, and efficiency, and significantly improve the scientific nature of decision-making and the efficiency of resource utilization.

[0191] In this embodiment of the invention, step S700 includes the following sub-steps G1 to G3:

[0192] In G1: A large language model agent is used to perform compliance checks and conflict identification on the optimal governance strategy; specific steps include:

[0193] The optimal governance strategy is input into a large language model agent for compliance checks. The compliance checks include verifying whether the scope of data correction and rule adjustment suggestions comply with power data standards and security protocols, and analyzing the performance of the strategy in similar historical cases.

[0194] Large language model agents leverage their reasoning capabilities to identify high-risk operations in optimal governance strategies that could lead to business disruptions, data inconsistencies, or coupling conflicts with existing systems. For strategies whose impact is difficult to assess directly, the agents automatically generate supplementary data verification or simulation test instructions to obtain more comprehensive risk assessment data.

[0195] For optimal governance strategies identified as high-risk or high-conflict, the system forcibly triggers a manual review process, providing detailed diagnostic reports, compliance analyses, and risk assessment results generated by the agent to business experts for final decision-making; the business experts' adoption, rejection, or modification opinions are recorded and fed back as training data to the large language model agent.

[0196] In G2: reinforcement learning is used to dynamically adjust the decision weights of the rule engine; the specific steps include:

[0197] When human experts approve and adopt the remediation strategies recommended by LLM experts, it clearly indicates that the rules upon which the strategy is based are effective and align with actual business expectations. The system will then strengthen the decision weight of these rules in the rule engine, giving them higher priority and stronger influence when handling similar data quality issues and recommending remediation solutions in the future. When human experts reject the remediation strategies recommended by LLM experts, it usually suggests that the rules upon which the strategy relies are flawed or no longer applicable. The system will, based on the specific reasons for rejection, downgrade the relevant rules or mark them as requiring optimization, and immediately initiate a rule relearning or adjustment process. Specific modification suggestions from human experts regarding remediation strategies are directly used to guide parameter adjustments or logic optimization in the rule engine, allowing the implicit knowledge of business experts to be efficiently absorbed and utilized by the algorithm model.

[0198] The adoption of a repair solution by a human expert is explicitly defined as a positive reward, while the rejection is set as a negative penalty. Furthermore, the "corrective suggestions" proposed by human experts can be further refined into different levels of reward signals to more precisely guide the reinforcement learning model in adjusting and optimizing the rules.

[0199] The decision weights of different rules are dynamically adjusted based on the scope and severity of the data quality issues and the actual effectiveness of historical remediation. For example, for data quality issues with a wide impact and high potential risk, the reinforcement learning model will proactively learn and increase the weights of relevant rules to ensure that these rules are given priority when generating remediation solutions. Through a continuous reinforcement learning process, the rule engine can gradually internalize the rich experience, professional judgment, and decision-making logic of power business experts into the algorithm model itself. This means that the system no longer relies solely on preset fixed rules, but can flexibly adapt to new business scenarios and challenges through continuous learning, ultimately achieving a fundamental shift from "rule-driven" to "experience-driven" systems.

[0200] When the system detects significant changes in data distribution, feature correlation, or business behavior, the rule engine will automatically trigger a process of relearning and adjusting the rules.

[0201] In G3: To comprehensively and objectively measure the actual effectiveness of data repair, a multi-dimensional and comprehensive evaluation indicator system has been carefully constructed. The system mainly includes the following key indicators:

[0202] Accuracy: The degree to which the repaired data matches the true value or expert-annotated value, calculated using the formula:

[0203] ;

[0204] Where Accuracy represents the accuracy rate, TP represents correctly identified and corrected abnormal data, TN represents correctly identified and retained normal data, FP represents normally corrected data, and FN represents abnormal data that was not corrected.

[0205] Recall rate: The proportion of all actual anomalous data that is successfully corrected, calculated using the following formula:

[0206] ;

[0207] Here, Recall represents the recall rate.

[0208] F1 score: The harmonic mean of precision and recall, which comprehensively measures the effectiveness of the repair. The calculation formula is as follows:

[0209] ;

[0210] ;

[0211] Precision refers to the accuracy rate.

[0212] Business impact: The impact of the repair operation on downstream business systems or decisions, such as whether it causes business interruption or affects the accuracy of reports;

[0213] Repair efficiency: The time and resources required to complete the repair;

[0214] Explainability: The transparency of the remediation decisions and processes, and whether they can provide a clear basis for remediation.

[0215] Furthermore, if the calculated accuracy rate is lower than the preset 95% threshold, the system will immediately activate the feedback mechanism.

[0216] Specifically, the evaluation results are immediately fed back to LLM experts and the dynamic rule evolution module. LLM experts will conduct a thorough re-analysis of the root causes of the poor repair results; simultaneously, the dynamic rule evolution module will review the relevant rules used for voltage data repair based on the received feedback, attempting to adjust rule parameters or introduce entirely new rules. Through multiple rounds of iterative optimization and continuous learning, the system will eventually find the optimal repair strategy and rules for this type of equipment fault. The system will also automatically generate a detailed repair report, fully recording the entire repair process, all evaluation results, and optimization measures taken, providing a solid basis for subsequent auditing work and knowledge transfer.

[0217] It should be noted that the evaluation data primarily comes from the following sources: ① Manual review and precise labeling of a portion of the restored data by experienced business experts or professional data governance personnel. ② In-depth comparative analysis of the restored data, the original data before restoration, and historical normal data to quantify the specific improvements brought about by the restoration operation. ③ Real-time feedback information from downstream business systems, including data usage and trends in key business indicators, directly reflecting the actual impact of data quality on business operations. ④ Actively collecting direct opinions and suggestions from data users regarding data quality to gain a deeper understanding of pain points in practical applications and propose targeted improvement needs accordingly.

[0218] It should be noted that the above step S700 forms a self-optimization closed loop of data governance through large model intelligent agent verification and reinforcement learning optimization, enabling the system to continuously learn and improve from governance practices, and ultimately realize the evolution from "rule-driven" to "experience-driven", ensuring the continuous improvement of governance capabilities.

[0219] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A data governance method based on an intelligent decision engine, characterized in that, include: Real-time streaming data is accessed from multiple heterogeneous data sources and subjected to preliminary cleaning and format conversion; Perform business semantic classification and tagging on the pre-processed real-time streaming data to build a data profile library; By identifying data entities and their corresponding relationships, a dynamic topology graph is constructed. Based on graph neural networks, the anomaly propagation pattern is learned, the impact of entity failures on related entities is analyzed, and an influence grading mechanism is established. Based on business rules and machine learning models, the data quality is assessed in multiple dimensions to generate dynamic data quality scores and grading strategies; a database of historical data quality problems and causes is built, and a large language model agent is used to generate a report on the root cause judgment and impact quantification of quality anomalies. A set of candidate strategies is generated based on the scope of impact, severity, and causes of anomalies in data quality issues. The optimal governance strategy is then selected from the set of candidate strategies by establishing a multi-objective optimization model. The large language model agent is used to perform compliance checks and conflict identification on the optimal governance strategy, and a reinforcement learning algorithm is used to dynamically adjust the decision weights of the rule engine to achieve a closed loop of data governance and dynamic learning. The method of learning anomaly propagation patterns based on graph neural networks, analyzing the impact of entity failures on related entities, and establishing an influence grading mechanism includes: Based on the dynamic topology graph, the four types of features of each entity are combined into a node feature vector and input into the spatiotemporal graph neural network model, which outputs the probability that each node is affected by an anomaly. The spatiotemporal graph neural network model uses historical anomaly propagation cases as training samples, defines the anomaly propagation task as a node classification or regression task, and trains the model to learn and simulate the propagation path and impact range of anomalies in the topology graph. Based on the probability of each node being affected by an anomaly, the influence value of a node is quantified by analyzing the propagation score distribution of other nodes when a certain node is the starting point of a fault. Based on the first quantitative standard and the obtained influence value of the node, the node influence is divided into three levels: high, medium and low, to clarify the priority of fault handling; the first quantitative standard considers the number of affected entities, the time it takes for the anomaly to spread to the entire area, and the business criticality of the affected entities; The method of generating a quality anomaly root cause judgment and impact quantification report using a large language model agent includes: The identified high-risk abnormal data and the topological relationships of abnormal data are used as input to the large-scale language model agent. Utilizing language understanding and reasoning capabilities, combined with information from the data quality historical problem and cause case library, the contextual information of the abnormal data is analyzed, and similar historical cases are retrieved based on the characteristics of the current abnormality. By integrating the dynamic topology graph, contextual information, and similar historical cases, the root cause judgment of the data quality anomaly is inferred and generated. Based on the topology impact analysis results, identify and quantify the specific datasets, tables, fields, and number of records affected by the current anomaly; generate an impact quantification report, which includes the technical and business reasons for the data quality anomaly, a list of affected datasets and records and their degree of impact, and a preliminary risk assessment.

2. The data governance method based on an intelligent decision engine as described in claim 1, characterized in that, The process of performing multi-dimensional quality assessment of data based on business rules and machine learning models to generate dynamic data quality scores and grading strategies includes: We define three core data quality assessment dimensions: completeness, accuracy, and consistency. For each dimension, we build an executable verification rule library that includes pre-event rules, in-event rules, and post-event rules. The rules are derived from the specific content written by business personnel for each business domain and functional module. The data extracts indicators reflecting the problems of each quality dimension as features, and obtains manually labeled or historical case data as labels. These are then input into the extreme gradient boosting model to calculate the quality score. Based on the output of the extreme gradient boosting model, the contribution weights of completeness, accuracy, and consistency are automatically calculated. The rule engine loads all rules from the executable validation rule library, performs batch validation on the preprocessed data, and outputs the rule violation status for each piece of data. Based on the verification results of the rule engine and the contribution weight, a comprehensive quality score is calculated for each piece of data. According to the preset dynamic threshold, the comprehensive quality score is divided into three levels: urgent defect, general defect and high quality, and corresponding governance priority labels of high risk, medium risk and low risk are generated.

3. The data governance method based on an intelligent decision engine as described in claim 2, characterized in that, The process of generating a candidate strategy set based on the scope, severity, and cause of the data quality problem includes: Data governance strategies are divided into automated repair strategies, semi-automated repair strategies, and manual intervention or source governance strategies, and the impact scope parameter R, severity parameter Q, and anomaly cause parameter P of data quality issues are defined respectively. Based on the actual values ​​of the influence range parameter R, severity parameter Q, and abnormal cause parameter P, candidate strategies that meet the constraints are selected from the three types of strategies. Based on the results of the screening, a candidate strategy set containing all strategies that satisfy the constraints is generated.

4. The data governance method based on an intelligent decision engine as described in claim 3, characterized in that, The step of selecting the optimal governance strategy from the candidate strategy set by establishing a multi-objective optimization model includes: Establish a multi-objective optimization model with the goals of minimizing repair costs, maximizing reliability, and minimizing risks; Based on the input impact range, severity, abnormal cause parameters and impact weights, the Pareto optimality algorithm is used to solve the multi-objective optimization model and output the Pareto optimal solution set, which includes all non-dominated solutions and is visualized on the Pareto front curve. From the Pareto optimal solution set, a final governance strategy is selected based on the priority and risk preference of the current business scenario. The selection process is implemented through a rule engine, which compares the values ​​of different solutions in terms of cost, reliability and risk objectives according to preset business rules, and outputs the recommended optimal governance strategy.

5. The data governance method based on an intelligent decision engine as described in claim 4, characterized in that, The compliance verification and conflict identification of the optimal governance strategy includes: The optimal governance strategy is input into the large language model agent for compliance checks. The compliance checks include verifying whether the data correction scope and rule adjustment suggestions comply with power data standards and security protocols, and analyzing the performance of the strategy in similar historical cases. The large language model agent uses its reasoning ability to identify high-risk operations in the optimal governance strategy that may lead to business interruption, data inconsistency, or coupling conflicts with existing systems. For strategies whose impact is difficult to judge directly, the large language model agent automatically generates supplementary data verification or simulation test instructions to obtain more comprehensive risk assessment data. For the optimal governance strategy identified as high-risk or high-conflict, the system forcibly triggers a manual review process, providing the detailed diagnostic report, compliance analysis, and risk assessment results generated by the large language model agent to business experts for final decision-making; the business experts' adoption, rejection, or modification opinions are recorded and fed back to the large language model agent as training data.

6. The data governance method based on an intelligent decision engine as described in claim 1, characterized in that, The preliminary cleaning and format conversion includes: Convert the source system's data types to the standard types required by the target system; Standardize the processing of different date formats, numerical precision, and text case; Perform preliminary repairs or mark obvious null values ​​and simple formatting errors to ensure that the data is parsable; Historical change information is provided by capturing the facts of data manipulation language changes and the changed data itself; Use regular expressions to perform preliminary data identification and data format conversion.

7. The data governance method based on an intelligent decision engine as described in claim 1, characterized in that, The construction of the data profiling library includes: Associate and map the collected physical metadata with business metadata; Model experts and business personnel label data tables, fields, and datasets based on the business object ID, business object name, first-level business category, second-level business category, business capability, business process, and operation-level process module of each business domain, and form a data business catalog. Information about each data asset is stored in the data service catalog to form a data profile library.

8. The data governance method based on an intelligent decision engine as described in claim 1, characterized in that, The method of dynamically adjusting the decision weights of the rule engine using reinforcement learning algorithms includes: The adoption of governance strategies by human experts in the review process is defined as a positive reward signal, and rejection is defined as a negative penalty signal; the specific corrective opinions proposed by human experts are quantified into reward signals of different intensities; the status, actions and corresponding reward signals in historical review records are collected to construct a reinforcement learning experience replay dataset. A Q-function is established, taking the decision state of the rule engine as input and aiming to maximize long-term cumulative rewards; the Q-function is updated through a temporal difference learning algorithm to dynamically evaluate the expected utility of each rule under different data quality problem scenarios; and the decision weights of the corresponding rules in the rule engine are adjusted according to the updated Q-value. The reinforcement learning agent is deployed in an online learning mode to absorb newly generated review feedback data in real time and update decision weights immediately. A rule version control system is established to record and archive the history of weight adjustments, reasons for adjustments, and adjustment times for each rule. The rule base is scanned regularly to mark or archive rules that have been in a low-weight state for a long time and are ineffective.

Images