An adaptive data governance method and system based on a large model and multiple agents

Abstract

This application relates to the field of data governance technology, providing an adaptive data governance method and system based on large models and multiple agents. The method includes: semantic extraction from multi-source heterogeneous data to construct a semantic feature map; a scheduling and coordination center dynamically calculates the demand weights of each data entity for multiple specialized agents based on the explicit semantic association strength between entities according to the semantic feature map, and arranges the agents participating in governance and their collaboration order based on the results of comparing scheduling activation thresholds; the scheduled agents execute governance tasks and interact with a dynamic governance knowledge base; a policy generation agent comprehensively discovers and generates governance strategies and executes them; the execution results are evaluated, and if the target is met, the knowledge increment is updated to the multi-dimensional knowledge base using a unified framework and differentiated learning rate; otherwise, the results are fed back to the scheduling and coordination center for re-arrangement. This application achieves a deep integration of semantic understanding and professional division of labor, as well as the continuous self-evolution of the knowledge base.

Description

Technical Field

[0001] This application relates to the field of data governance technology, specifically to an adaptive data governance method and system based on large models and multiple agents. Background Technology

[0002] In enterprises and organizations, data governance is a fundamental task to ensure the quality, security, and availability of data assets. Traditional data governance relies on manual rule setting, manual labeling, classification, and grading, and manual quality inspection, which is inefficient and struggles to address the governance needs of massive, multi-source, and heterogeneous data. In recent years, although data governance solutions based on large models and multi-agent systems have emerged, as discussed in Chapter 1, existing solutions suffer from problems such as a disconnect between semantic understanding and professional division of labor, fixed collaboration patterns, and static or one-dimensional knowledge bases.

[0003] In recent years, artificial intelligence (AI) technology has been gradually introduced into the field of data governance to improve the automation and intelligence of governance. Currently, AI-based data governance technologies are developing mainly along the following two directions: Direction 1: Data governance based on large models.

[0004] This approach primarily leverages the powerful semantic understanding and reasoning capabilities of pre-trained large models to perform governance tasks. For example, Chinese invention patent CN121166662B discloses an "Automated Data Governance Method and System Based on Large Models." This scheme uses a pre-trained large model to perform cross-domain feature association mining on the data domain to be governed, generating semantic feature maps and implicit association links of data entities; it then performs quality dimension evaluation based on the semantic feature maps, outputs quality defect vectors, and performs root cause analysis; finally, it drives the policy evolution module in the large model to generate adaptive governance policies and deploy and execute them.

[0005] However, this type of approach has obvious limitations: it uses a single pre-trained large model as the core engine, and all governance tasks are completed by the same model, lacking a specialized division of labor and collaboration mechanism for different governance sub-tasks; at the same time, the generation of strategies depends on the internal reasoning of the large model, lacking a structured governance knowledge base as support for experience accumulation and reuse, making it difficult to achieve the continuous accumulation and evolution of governance experience.

[0006] Direction 2: Data governance based on multi-agent systems.

[0007] This approach utilizes the collaboration of multiple agents with specific functions to accomplish governance tasks. For example, Chinese invention patent application CN120542460A discloses a "multi-agent data governance method and system based on negative feedback optimization." This scheme governs the data to be governed based on a multi-agent system, creating a task execution path graph in real time; it evaluates the governance results based on evaluation indicators to obtain negative feedback data; and it queries and optimizes the task processing path and corresponding agent parameters based on the negative feedback data, thereby re-governing the data.

[0008] However, this type of solution also has limitations: the core optimization mechanism relies on post-event negative feedback data and lacks the deep semantic perception capability provided by the semantic understanding of large models beforehand; the cooperation methods between agents are fixed (such as preset task execution paths) and lack the ability to orchestrate and schedule tasks in real time based on semantic understanding; in addition, its governance knowledge is managed through a parameter matching library, which is a static rule library and does not have the ability to automatically accumulate experience and achieve self-evolution of knowledge during the governance execution process.

[0009] In summary, existing data governance technologies suffer from common drawbacks, including a disconnect between semantic understanding and specialized division of labor, rigid and inflexible collaboration models, static governance knowledge, and incomplete coverage. Currently, no existing technology can organically integrate the deep semantic understanding capabilities of large models, the advantages of specialized collaboration among multiple agents, and the self-evolutionary mechanism of dynamic knowledge bases into a single adaptive closed-loop system to achieve end-to-end intelligent data governance. Summary of the Invention

[0010] To help solve the above-mentioned technical problems, this application provides an adaptive data governance method and system based on large models and multiple agents.

[0011] Firstly, this application provides an adaptive data governance method based on a large model and multiple agents, wherein the method is based on multiple specialized agents, including a classification and grading agent, a quality detection agent, a lineage construction agent, and a strategy generation agent; the method includes: Semantic extraction is performed on multi-source heterogeneous data to construct a semantic feature graph containing entity nodes, edges, and implicit association links; Based on the semantic feature map, the scheduling and coordination center dynamically calculates the demand weight of each data entity for the multiple specialized intelligent agents based on the explicit semantic association strength between entities and the task expertise of each intelligent agent, and compares the demand weight with the preset scheduling activation threshold. Based on the comparison result, the intelligent agents participating in this governance and their cooperation order are arranged. The scheduled agents execute their respective governance tasks. During execution, the classification and grading agent, the quality detection agent, and the lineage construction agent interact with the corresponding classification and grading library, quality rule library, and lineage relationship graph in the dynamic governance knowledge base, respectively, and output governance findings. The policy generation agent integrates the governance findings and combines them with the policy template library in the dynamic governance knowledge base to generate and execute governance policies; The effectiveness of governance implementation is evaluated. If the evaluation results meet the standards, a unified self-evolutionary update framework is adopted. For multiple dimensions such as classification and grading knowledge, quality rule knowledge, lineage graph knowledge, and strategy template knowledge, the knowledge increments generated during the governance process are updated to the knowledge base of each dimension with differentiated learning rates and knowledge merging operations corresponding to each dimension. If the evaluation results do not meet the standards, the evaluation results are fed back to the scheduling and coordination center to trigger re-orchestration.

[0012] The scheduling and coordination center dynamically calculates the demand weights as follows: Based on the explicit semantic association strength g between entities in the semantic feature map dj The semantic adaptation parameters a of entity d to agent k′ dk′ And the task expertise vector e of agent k′ k′ The demand weight w of entity d for agent k′ is calculated according to the following formula. dk′ : ; Where k′ is the agent type index, It is the sigmoid activation function. Let w represent the inner product, and N be the total number of entities in the semantic feature graph; when w dk′ Exceeding the preset scheduling activation threshold At that time, the scheduling agent k′ participates in the governance of entity d.

[0013] The unified self-evolutionary update framework is represented as follows: Where k represents the knowledge dimension index, Let k be the state of the knowledge base after the t-th round of governance, representing the knowledge dimension k. These correspond to classification and grading knowledge, quality rule knowledge, lineage chart knowledge, and strategy template knowledge, respectively. For the differentiated learning rate of knowledge dimension k, ; The knowledge increment generated in knowledge dimension k during the t-th round of governance is jointly determined by the results of the effect evaluation and the structured knowledge generated during the governance process; This refers to the knowledge merging operation corresponding to knowledge dimension k.

[0014] The differentiated learning rate The setting is: the differentiated learning rate used for high-confidence knowledge dimensions that have undergone multiple rounds of validation is less than the differentiated learning rate used for knowledge dimensions that are in the rapid accumulation stage.

[0015] When k=cls, knowledge merging operation This includes updating the prototype vector for new categories using a weighted average of semantic vectors. Where n is the original number of samples, vproto_old is the original prototype vector, and vnew is the semantic vector of the new sample; for existing categories, the prototype vector is updated using an exponential moving average: ,in For smoothing coefficients; When k=graph, the knowledge merging operation Includes: Incremental updates of graph nodes and edges to add new blood relations: The confidence level of existing blood-related edges is updated as follows: ,in is the decay factor, and evidence_score is the score for new evidence.

[0016] In the semantic feature graph, nodes correspond to data entities, and node attributes include a unique entity identifier, entity type, semantic vector, and metadata tag set; edges correspond to explicit semantic associations between entities, and edge attributes include source node, target node, association type, and association strength; the implicit association links are discovered by large model inference and stored in the form of triples containing relationship description, confidence level, and evidence anchor.

[0017] The classification and grading agent performs the following tasks: matching the semantic vector of the data entity with the prototype vector in the classification and grading library to obtain candidate categories, and combining the original content of the data entity and the context of the implicit association link to confirm the category and determine the sensitivity level, and output the category label, sensitivity level label and classification confidence score of the data entity. The quality inspection agent performs the following tasks: adaptively selects the detection dimension set according to the category of the data entity, performs rule matching and anomaly detection dimension by dimension, summarizes and generates a quality defect report containing defect vector, root cause type, severity level and association rule number, and outputs a comprehensive quality score of the data entity. The tasks performed by the lineage construction agent include: parsing the explicit dependencies of data entities to construct preliminary lineage edges, supplementing implicit lineage edges using the implicit association links, evaluating the confidence of lineage relationships and merging and deduplicating them, and outputting the incremental update of the field-level / table-level lineage relationship graph, the confidence label of newly added lineage edges, and the lineage change log.

[0018] The strategy generation agent integrates the governance findings output by the classification and grading agent, the quality detection agent, and the lineage construction agent, and generates a governance strategy by combining the matching strategy template set in the strategy template library. The governance strategy is a hierarchical governance strategy, including a root repair strategy that corrects at the upstream source of the lineage, and a derivative repair strategy that synchronously updates the affected downstream fields.

[0019] The dynamic governance knowledge base includes a classification and grading library, a quality rule library, a lineage relationship graph, and a strategy template library; the evaluation indicators for the effect assessment include classification accuracy, quality improvement magnitude, and lineage recall rate.

[0020] Secondly, this application provides an adaptive data governance system based on large models and multi-agent systems, comprising: The semantic understanding module is used to extract semantics from multi-source heterogeneous data and construct a semantic feature graph containing entity nodes, edges, and implicit association links. The multi-agent collaboration module includes a scheduling and coordination center and multiple specialized agents, including a classification and grading agent, a quality detection agent, a lineage building agent, and a strategy generation agent. The scheduling and coordination center is used to dynamically calculate the demand weight of each data entity to the multiple specialized agents based on the semantic feature map, the explicit semantic association strength between entities, and the task expertise of each agent. The demand weight is then compared with a preset scheduling activation threshold, and the agents participating in this governance and their collaboration order are arranged according to the comparison results. The dynamic governance knowledge base includes a classification and grading library, a quality rule library, a lineage graph, and a strategy template library. The governance execution and evaluation module is used to execute the governance strategy generated by the strategy generation agent and evaluate the effectiveness of the governance execution results. The classification and grading agent, quality detection agent, and lineage construction agent interact with the classification and grading library, quality rule library, and lineage graph respectively during execution, outputting governance findings. The policy generation agent integrates the governance findings and combines them with the policy template library to generate governance policies. If the evaluation results meet the standards, a unified self-evolutionary update framework is adopted. For multiple dimensions of classification and grading knowledge, quality rule knowledge, lineage graph knowledge, and policy template knowledge, differentiated learning rates and knowledge merging operations are used to update the knowledge increments generated during the governance process to the knowledge bases of each dimension. If the evaluation results do not meet the standards, the evaluation results are fed back to the scheduling and coordination center to trigger re-orchestration.

[0021] In summary, this application constructs a two-layer architecture of a large-scale semantic foundation and multi-agent professional division of labor, which deeply integrates deep semantic perception and task execution, thereby improving the accuracy and efficiency of governance. Based on the semantic feature map, it adaptively calculates the required weights to dynamically orchestrate agent collaboration, replacing the fixed pipeline and improving the system's ability to flexibly adapt to complex scenarios. At the same time, it uses effect evaluation feedback to drive the automatic accumulation and self-evolution of governance knowledge, ultimately forming a closed-loop system of execution, evaluation, feedback, and optimization. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an adaptive data governance method based on a large model and multiple agents proposed in this application. Detailed Implementation

[0023] The present application will be further described below with reference to the accompanying drawings. The principles of the present application are very clear to those skilled in the art. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

[0024] System overall architecture.

[0025] The overall architecture of this application system is divided into a data access layer, a large model semantic understanding layer, a multi-agent collaboration layer, a dynamic governance knowledge base, and a governance execution and evaluation layer.

[0026] The data access layer is responsible for receiving and preprocessing structured, semi-structured, and unstructured data from different data sources. The data preprocessing module performs format standardization, noise filtering, and preliminary feature extraction on the raw data, providing standardized input for the semantic understanding layer.

[0027] The large-scale model semantic understanding layer serves as the semantic foundation of the entire system, comprising a semantic encoder, a feature map construction module, and an implicit association reasoning module. The semantic encoder transforms the preprocessed data into high-dimensional semantic vector representations. The feature map construction module constructs semantic feature maps of data entities based on these semantic vectors. The implicit association reasoning module utilizes the large-scale model's reasoning capabilities to discover potential implicit associations between data entities.

[0028] The multi-agent collaboration layer comprises a scheduling and coordination center and multiple specialized agents, including classification and grading agents, quality inspection agents, lineage building agents, and policy generation agents. The scheduling and coordination center receives the output from the semantic understanding layer and dynamically orchestrates the agents to be invoked and their collaboration order for this governance task. Each agent interacts with the dynamic governance knowledge base during execution.

[0029] The dynamic governance knowledge base includes a classification and grading library, a quality rule library, a lineage graph, and a strategy template library, which are used to store and continuously evolve governance knowledge.

[0030] The governance execution and evaluation layer comprises a governance execution engine and an effect evaluation module. The governance execution engine executes specific governance operations based on the governance strategies output by the policy-generating agent, while the effect evaluation module performs multi-dimensional evaluations of the governance results and outputs feedback signals, forming a closed loop.

[0031] Figure 1 This is a flowchart illustrating an adaptive data governance method based on a large model and multiple agents, as proposed in this application. The method of this application may include the following parts: Semantic extraction and feature map construction.

[0032] First, after preprocessing, the multi-source heterogeneous data undergoes unified semantic encoding by a large-model semantic encoder. It should be noted that this invention does not specifically limit the large-model used; any general-purpose large-model with semantic encoding and text understanding capabilities can be used, such as the GPT series, BERT series, LLaMA series, ChatGLM series, or Tongyi Qianwen series of general pre-trained language models. In actual deployment, a suitable large-model architecture can be flexibly selected based on the application scenario's requirements for semantic understanding depth, inference ability, deployment cost, etc. For each data entity d (d=1,2,…,N), a D-dimensional semantic vector vd∈RD is generated. In this embodiment, the value of D can be 768, 1024, 2048, or 4096, depending on the selected large-model architecture.

[0033] Based on semantic vectors, the feature graph construction module builds semantic feature graphs of data entities. This graph consists of three parts: nodes, edges, and implicit association links. Node: Each data entity (such as a table, field, file, API interface, etc.) is mapped to a node in the graph. Node attributes include the entity's unique identifier id, entity type type (type∈{TABLE,FIELD,FILE,API,…}), semantic vector v∈RD, and metadata tag set tags.

[0034] Edges: Represent explicit semantic associations between entities. Edge attributes include source node src_id, target node tgt_id, association type rel_type (rel_type∈{CONTAINS,REFERENCES,DEPENDS_ON,DERIVES_FROM,…}), and association strength ∈[0,1]. The entire semantic feature graph can be represented as a matrix Gsem∈RN×N, where the element gij∈[0,1] represents the explicit semantic association strength between entity i and entity j.

[0035] Implicit association links: Discovered by the implicit association inference module of the large model, these are deep semantic connections across domains and tables, which cannot be directly obtained from metadata. Each implicit association link is stored in the implicit association index of the graph as a triple (entity_i, relation_desc, entity_j, confidence), where relation_desc is the semantic description of the association in natural language, and confidence is the confidence score of the large model's inference. Each implicit association link is also associated with one or more evidence anchors, i.e., the original semantic feature fragments that triggered the inference, to support the interpretability and traceability of the association.

[0036] Multi-agent dynamic scheduling part.

[0037] After receiving the semantic feature graph and implicit association links, the scheduling and coordination center analyzes the data type, semantic complexity, and governance requirements of the current governance object, and dynamically orchestrates the types of intelligent agents to be invoked, the invocation order, and the cooperation strategy. The core scheduling method is a requirement weight calculation method based on the semantic feature graph.

[0038] The scheduling and coordination center dynamically calculates the demand weights as follows: Based on the explicit semantic association strength g between entities in the semantic feature map dj The semantic adaptation parameters a of entity d to agent k′ dk′ And the task expertise vector e of agent k′ k′ The demand weight w of entity d for agent k′ is calculated according to the following formula. dk′ : ; Where k′ is the agent type index, It is the sigmoid activation function. Let w represent the inner product, and N be the total number of entities in the semantic feature graph; when w dk′ Exceeding the preset scheduling activation threshold At that time, the scheduling agent k′ participates in the governance of entity d.

[0039] The calculated weight w dk′ With the preset scheduling activation threshold Comparison: When w dk′ > At that time, the scheduling agent k′ participates in the governance of entity d. Threshold The threshold value is between (0,1), and the lower the threshold, the easier it is for the agent to be scheduled. By setting different thresholds for different agents, the scheduling strategy can be flexibly controlled. For example, for structured database tables, the quality inspection agent and the lineage building agent usually have higher weights and are scheduled first; for unstructured documents, the classification and grading agent has more prominent weights.

[0040] This ultimately forms the agent scheduling vector s for entity d. d =(w dc ,w dq ,w dl These correspond to the demand weights of the classification and grading agent, the quality inspection agent, and the lineage building agent, respectively. The policy generation agent is typically activated in the later comprehensive decision-making stage. This semantically driven dynamic orchestration method replaces the fixed pipeline or serial collaboration mode in existing solutions.

[0041] The collaborative execution part of intelligent agents.

[0042] The scheduled agents execute their respective tasks in an ordered sequence and interact with the dynamic governance knowledge base.

[0043] The processing procedure of the classification and grading agent is as follows: First, the semantic vector (vd) of the data entity is matched with the prototype vectors of existing categories in the classification and grading library to obtain Top-K candidate categories. Then, combining the original content of the data entity (such as table structure, field values, and document text) and contextual information of implicit association links, a large model inference is used to confirm and refine the candidate categories. For sensitivity level determination, a multi-factor weighted determination is performed based on the semantics of the fields (such as whether they contain ID card numbers, mobile phone numbers, or financial information keywords), data domain affiliation, and compliance rules, outputting the category label, sensitivity level label (such as L1-L4), and classification confidence score of the data entity.

[0044] The quality inspection agent's processing procedure is as follows: It adaptively selects the set of inspection dimensions based on the data entity category. For structured data, four dimensions—completeness, consistency, accuracy, and timeliness—are enabled by default; for unstructured data, semantic completeness and compliance dimensions are additionally enabled. Rule matching and anomaly detection are performed dimension by dimension: Completeness checks the field null value rate and record duplication rate; consistency checks cross-table / cross-field value range constraints and format specifications; accuracy verifies whether field values ​​are within a reasonable semantic space based on a semantic benchmarking model; and timeliness checks the deviation between the data timestamp and the expected update cycle. The inspection results for each dimension are summarized to generate a quality defect report, which includes the defect vector, root cause type, severity level, and association rule number, and provides a comprehensive quality score for the data entity.

[0045] The process of building a lineage-based intelligent agent involves: parsing explicit dependencies of data entities (such as JOIN and subqueries in SQL statements) and constructing preliminary field-level / table-level lineage edges. Explicit dependencies have a high confidence level (≥0.95). Simultaneously, implicit relationship chains are used to discover non-explicit lineage relationships; for example, two fields may not have direct SQL references but are semantically highly similar and originate from the same business concept, thus supplementing implicit lineage edges. Confidence assessments are performed on all lineage relationships, and the newly discovered lineages are merged with existing structures in the lineage graph to remove duplicates. The final output includes the incremental updates to the field-level / table-level lineage graph, confidence labels for newly added lineage edges, and a lineage change log.

[0046] The strategy generation agent synthesizes the outputs of the aforementioned agents to make decisions. First, it matches candidate strategy templates from the strategy template library based on the governance need type (data repair / compliance processing / quality improvement / standards implementation). Then, based on the classification and grading results, it determines the scope and priority of the strategy (e.g., prioritizing highly sensitive data). Finally, combining the root cause type in the quality defect report and the impact range (downstream dependent fields) in the lineage diagram, it generates hierarchical governance strategies: root cause repair strategies correct upstream sources in the lineage, while derivative repair strategies synchronously update affected downstream fields. The generated strategies include strategy type, scope, execution steps, estimated impact range, and priority, and undergo feasibility verification and side effect evaluation.

[0047] The self-evolving part of the dynamically governed knowledge base.

[0048] The dynamic governance knowledge base includes a classification and grading library, a quality rule library, a lineage graph, and a strategy template library. Unlike traditional static rule bases, it can achieve continuous self-evolution through performance evaluation feedback. The self-evolution process follows a unified framework: The unified self-evolutionary update framework is represented as follows: Where k represents the knowledge dimension index, Let k be the state of the knowledge base after the t-th round of governance, representing the knowledge dimension k. These correspond to classification and grading knowledge, quality rule knowledge, lineage chart knowledge, and strategy template knowledge, respectively. For the differentiated learning rate of knowledge dimension k, ; The knowledge increment generated in knowledge dimension k during the t-th round of governance is jointly determined by the results of the effect evaluation and the structured knowledge generated during the governance process; This refers to the knowledge merging operation corresponding to knowledge dimension k.

[0049] The strategy for setting differentiated learning rates is as follows: for high-confidence knowledge dimensions that have undergone multiple rounds of validation (such as explicit dependency edges in a pedigree graph), a smaller learning rate is used. To maintain stability; for dimensions in a rapid accumulation phase (such as the classification and labeling of new business scenarios), a larger [size / size] is used. To accelerate convergence.

[0050] When k=cls, knowledge merging operation This includes updating the prototype vector for new categories using a weighted average of semantic vectors. Where n is the original number of samples, vproto_old is the original prototype vector, and vnew is the semantic vector of the new sample; for existing categories, the prototype vector is updated using an exponential moving average: ,in For smoothing coefficients; When k=graph, the knowledge merging operation Includes: Incremental updates of graph nodes and edges to add new blood relations: The confidence level of existing blood-related edges is updated as follows: ,in is the decay factor, and evidence_score is the score for new evidence.

[0051] When k=rule, newly added rules are stored in a structured manner according to "conditions → conclusions → confidence level"; existing rules are updated with priority and applicable conditions through rule validity scores (based on the hit rate and false alarm rate of the most recent N detections); long-term inefficient rules are automatically downgraded or archived.

[0052] When k=tmpl, the newly added successful strategy is abstracted into a parameterized template (extracting variable parameters and invariant skeleton); the existing templates are dynamically adjusted in recommendation order based on the governance success rate; similar templates are automatically merged and deduplicated.

[0053] Governance implementation and closed-loop evaluation section.

[0054] The governance execution engine executes specific operations based on the hierarchical governance strategies output by the strategy-generating agent, such as data repair, tagging, and compliance processing. After execution, the effectiveness evaluation module performs a multi-dimensional quantitative evaluation of the governance results, with evaluation indicators including classification accuracy, quality improvement, and lineage recall, generating a comprehensive effectiveness score E. (t) ∈[0,1].

[0055] If the assessment results meet the standards (e.g., E) (t)If the evaluation result exceeds a set threshold, a knowledge feedback mechanism is triggered. Effective annotations, successful remediation plans, and newly discovered lineage relationships from the governance process are automatically and structurally stored in the knowledge base across various dimensions. This knowledge base is then updated according to the unified self-evolutionary update framework described above, enabling continuous optimization. If the evaluation result fails to meet the standards, it is fed back to the scheduling and coordination center, triggering a re-orchestration process. This includes adjusting the agent composition, changing the collaboration order, or correcting key parameters, and the process re-enters the governance cycle until the expected quality standards are met.

[0056] Overall process.

[0057] The entire governance process can be summarized in the following steps: S1: Multi-source data access and preprocessing; S2: Large-scale model semantic encoding and feature extraction; S3: Semantic feature graph and implicit association construction; S4: Semantic analysis of the scheduling and coordination center; S5: Dynamic orchestration of agent tasks, which enter the execution phase of classification and grading agents, quality inspection agents, and lineage building agents according to requirements; S6: Each agent performs tasks and interacts with the dynamic governance knowledge base; S7: The strategy-generating intelligent agent makes comprehensive decisions and generates hierarchical governance strategies; S8: Governance execution engine execution strategy; S9: Multi-dimensional evaluation of effectiveness; S10: Evaluation results are determined. If the standard is met, proceed to S11 Knowledge Flow and Self-Evolution; otherwise, regress to S4 for rearrangement. S11: Knowledge reflux and self-evolution, governance completed.

[0058] These processes can be summarized as follows: Part 1 (semantic extraction and graph construction), Part 2 (semantic-driven dynamic scheduling), Part 3 (multi-agent collaborative execution), Part 4 (policy generation and execution), and Part 5 (effect evaluation and knowledge self-evolution). For details, please refer to [link to relevant documentation]. Figure 1 .

[0059] The technical scope of this application is not limited to the contents of the above specification. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this application, and all such modifications and variations should fall within the scope of this application.

Claims

1. An adaptive data governance method based on large models and multi-agent systems, characterized in that, The method is based on multiple specialized intelligent agents, including a classification and grading agent, a quality inspection agent, a lineage establishment agent, and a strategy generation agent; the method includes: Semantic extraction is performed on multi-source heterogeneous data to construct a semantic feature graph containing entity nodes, edges, and implicit association links; Based on the semantic feature map, the scheduling and coordination center dynamically calculates the demand weight of each data entity for the multiple specialized intelligent agents based on the explicit semantic association strength between entities and the task expertise of each intelligent agent, and compares the demand weight with the preset scheduling activation threshold. Based on the comparison result, the intelligent agents participating in this governance and their cooperation order are arranged. The scheduled agents execute their respective governance tasks. During execution, the classification and grading agent, the quality detection agent, and the lineage construction agent interact with the corresponding classification and grading library, quality rule library, and lineage relationship graph in the dynamic governance knowledge base, respectively, and output governance findings. The policy generation agent integrates the governance findings and combines them with the policy template library in the dynamic governance knowledge base to generate and execute governance policies; The effectiveness of governance implementation is evaluated. If the evaluation results meet the standards, a unified self-evolutionary update framework is adopted. For multiple dimensions such as classification and grading knowledge, quality rule knowledge, lineage graph knowledge, and strategy template knowledge, the knowledge increments generated during the governance process are updated to the knowledge base of each dimension with differentiated learning rates and knowledge merging operations corresponding to each dimension. If the evaluation results do not meet the standards, the evaluation results are fed back to the scheduling and coordination center to trigger re-orchestration.

2. The method according to claim 1, characterized in that, The scheduling and coordination center dynamically calculates the demand weights as follows: Based on the explicit semantic association strength g between entities in the semantic feature map dj The semantic adaptation parameters a of entity d to agent k′ dk′ And the task expertise vector e of agent k′ k′ The demand weight w of entity d for agent k′ is calculated according to the following formula. dk′ : ； Where k′ is the agent type index, It is the sigmoid activation function. Let w represent the inner product, and N be the total number of entities in the semantic feature graph; when w dk′ Exceeding the preset scheduling activation threshold At that time, the scheduling agent k′ participates in the governance of entity d.

3. The method according to claim 1, characterized in that, The unified self-evolutionary update framework is represented as follows: Where k represents the knowledge dimension index, Let k be the state of the knowledge base after the t-th round of governance, representing the knowledge dimension k. These correspond to classification and grading knowledge, quality rule knowledge, lineage chart knowledge, and strategy template knowledge, respectively. For the differentiated learning rate of knowledge dimension k, ; The knowledge increment generated in knowledge dimension k during the t-th round of governance is jointly determined by the results of the effect evaluation and the structured knowledge generated during the governance process; This refers to the knowledge merging operation corresponding to knowledge dimension k.

4. The method according to claim 3, characterized in that, The differentiated learning rate The setting is: the differentiated learning rate used for high-confidence knowledge dimensions that have undergone multiple rounds of validation is less than the differentiated learning rate used for knowledge dimensions that are in the rapid accumulation stage.

5. The method according to claim 3, characterized in that, When k=cls, knowledge merging operation This includes: For newly added categories, updating the prototype vector using a weighted average of semantic vectors. Where n is the original number of samples, vproto_old is the original prototype vector, and vnew is the semantic vector of the new sample; for existing categories, the prototype vector is updated using an exponential moving average: ,in For smoothing coefficients; When k=graph, the knowledge merging operation Includes: Incremental updates of graph nodes and edges to add new blood relations: The confidence level of existing blood-related edges is updated as follows: ,in is the decay factor, and evidence_score is the score for new evidence.

6. The method according to claim 1, characterized in that, In the semantic feature graph, nodes correspond to data entities, and node attributes include a unique entity identifier, entity type, semantic vector, and metadata tag set; edges correspond to explicit semantic associations between entities, and edge attributes include source node, target node, association type, and association strength; the implicit association links are discovered by large model inference and stored in the form of triples containing relationship description, confidence level, and evidence anchor.

7. The method according to claim 1, characterized in that, The classification and grading agent performs the following tasks: matching the semantic vector of the data entity with the prototype vector in the classification and grading library to obtain candidate categories, and combining the original content of the data entity and the context of the implicit association link to confirm the category and determine the sensitivity level, and output the category label, sensitivity level label and classification confidence score of the data entity. The quality inspection agent performs the following tasks: adaptively selects the detection dimension set according to the category of the data entity, performs rule matching and anomaly detection dimension by dimension, summarizes and generates a quality defect report containing defect vector, root cause type, severity level and association rule number, and outputs a comprehensive quality score of the data entity. The tasks performed by the lineage construction agent include: parsing the explicit dependencies of data entities to construct preliminary lineage edges, supplementing implicit lineage edges using the implicit association links, evaluating the confidence of lineage relationships and merging and deduplicating them, and outputting the incremental update of the field-level / table-level lineage relationship graph, the confidence label of newly added lineage edges, and the lineage change log.

8. The method according to claim 1, characterized in that, The strategy generation agent integrates the governance findings output by the classification and grading agent, the quality detection agent, and the lineage construction agent, and generates a governance strategy by combining the matching strategy template set in the strategy template library. The governance strategy is a hierarchical governance strategy, including a root repair strategy that corrects at the upstream source of the lineage, and a derivative repair strategy that synchronously updates the affected downstream fields.

9. The method according to claim 1, characterized in that, The dynamic governance knowledge base includes a classification and grading library, a quality rule library, a lineage relationship graph, and a strategy template library; the evaluation indicators for the effect assessment include classification accuracy, quality improvement magnitude, and lineage recall rate.

10. An adaptive data governance system based on large models and multi-agent systems, characterized in that, include: The semantic understanding module is used to extract semantics from multi-source heterogeneous data and construct a semantic feature graph containing entity nodes, edges, and implicit association links. The multi-agent collaboration module includes a scheduling and coordination center and multiple specialized agents, including a classification and grading agent, a quality detection agent, a lineage building agent, and a strategy generation agent. The scheduling and coordination center is used to dynamically calculate the demand weight of each data entity to the multiple specialized agents based on the semantic feature map, the explicit semantic association strength between entities, and the task expertise of each agent. The demand weight is then compared with a preset scheduling activation threshold, and the agents participating in this governance and their collaboration order are arranged according to the comparison results. The dynamic governance knowledge base includes a classification and grading library, a quality rule library, a lineage graph, and a strategy template library. The governance execution and evaluation module is used to execute the governance strategy generated by the strategy generation agent and evaluate the effectiveness of the governance execution results. The classification and grading agent, the quality detection agent, and the lineage construction agent interact with the classification and grading library, the quality rule library, and the lineage relationship graph respectively during execution, and output governance findings; the strategy generation agent integrates the governance findings and combines them with the strategy template library to generate governance strategies. If the evaluation results meet the standards, a unified self-evolutionary update framework is adopted. For multiple dimensions such as classification and grading knowledge, quality rule knowledge, lineage graph knowledge, and strategy template knowledge, the knowledge increments generated during the governance process are updated to the knowledge base of each dimension with differentiated learning rates and knowledge merging operations corresponding to each dimension. If the evaluation results do not meet the standards, the evaluation results are fed back to the scheduling and coordination center to trigger re-orchestration.

Images