Method and system for building a data governance agent

Abstract

The present application relates to the field of data governance, and more particularly, to a method and system for building a data governance agent. The method comprises: converting a knowledge base related to data governance into a dataset for fine-tuning; fine-tuning a base large model through low-rank adaptation using the dataset to generate a domain large model; providing a callable extended small model for the domain large model, and building a data governance agent based on the domain large model and the extended small model; updating the data governance agent according to user feedback for the data governance agent and upgrading of the base large model.

Description

Technical Field

[0001] This application relates to the field of data governance, and more specifically, to methods and systems for constructing data governance agents and methods for constructing semantic benchmarking models. Background Technology

[0002] In today's digital age, massive amounts of business data are generated and processed. The metrics of this business data vary across different departments, projects, and business operations, leading to inconsistent standards and making data governance extremely difficult. Traditional data governance methods largely rely on human intervention to develop and implement data governance strategies. This results in data analysts on the service side struggling to find accurate and comprehensive data, while on the operations side, labor costs are high and operational efficiency is insufficient.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] To address or at least mitigate one or more of the above problems, this application provides a method for constructing a data governance agent, a system, a computer-readable storage medium, and a computer program product for implementing the method, as well as a method for constructing a semantic benchmarking model to be used in the data governance agent, thereby enabling the data governance process to be carried out in an efficient, comprehensive, and accurate manner without relying on manual operation.

[0005] According to a first aspect of this application, a method for constructing a data governance agent is provided, the method comprising: converting a knowledge base related to data governance into a dataset for fine-tuning; using the dataset to fine-tune a base large model via low-rank adaptation (LoRA) to generate a domain large model; providing callable extended small models for the domain large model, and constructing a data governance agent based on the domain large model and the extended small models; and updating the data governance agent based on user feedback to the data governance agent and upgrades to the base large model.

[0006] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, the extended small model includes a semantic benchmarking model and an anomaly detection model.

[0007] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, the semantic mapping model includes a feature encoding layer, a similarity calculation layer, and an intelligent interaction layer for interacting with the domain-wide model.

[0008] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, the semantic benchmarking model is constructed through the following steps: obtaining a standard data set and a field set, and determining the standard data corresponding to each field in the field set; performing feature encoding on each standard data in the standard data set and each field in the field set through the feature encoding layer to obtain a standard data encoding set and a field encoding set; calculating the similarity between each field encoding and each standard data encoding in the standard data encoding set based on an initial segmentation weight through the similarity calculation layer; adjusting the initial segmentation weight to obtain a final segmentation weight, wherein the final segmentation weight minimizes the difference between the standard data corresponding to the standard data encoding with the highest similarity to each field encoding and the standard data corresponding to that field.

[0009] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, the similarity calculation layer calculates the similarity between each field code and the standard data code using the following formula: Where S is the similarity, k is the number of segments between the field encoding and the standard data encoding, and λ is the similarity score. i It is the segment weight, a i It is a segment in standard data encoding, b i It refers to a segment in the field encoding.

[0010] As an alternative or supplement to the above scheme, in a method according to an embodiment of this application, the field encoding set is further divided into a training set, a validation set, and a test set for determining the final segment weights.

[0011] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, the standard data includes: the standard English name of the data, the standard Chinese name of the data, the business definition, and the value-added specifications; and wherein the fields include: the English name of the field, the Chinese name of the field, the field description, the field type, and the field length.

[0012] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, the field type and the field length are encoded into the same segment.

[0013] As an alternative or supplement to the above scheme, in a method according to an embodiment of this application, the standard English name and standard Chinese name of the data, as well as the English name and Chinese name of the field, are encoded by word frequency-based One-Hot encoding in the feature encoding layer.

[0014] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, the business definition and field description are encoded by Base64 encoding in the feature encoding layer.

[0015] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, fine-tuning the basic large model using the dataset through low-rank adaptation includes: fine-tuning the low-rank matrix using a general data governance dataset; fine-tuning the low-rank matrix again using an additional dataset related to the business scenario; and scoring the output of the fine-tuned domain large model, and supplementing the correct output corresponding to the low-scoring output results.

[0016] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, converting a knowledge base related to data governance into a dataset for fine-tuning includes: collecting current data standard specification files, statistical indicator caliber files, and structured metadata files; de-identifying the collected data standard specification files, statistical indicator caliber files, and structured metadata files; and converting the de-identified files into a dataset for fine-tuning, wherein each piece of data in the dataset includes input content and response content.

[0017] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, updating the data governance agent includes: in response to a version upgrade of the basic large model, reusing the dataset to fine-tune the upgraded basic large model to generate an upgraded domain large model; and reconstructing the updated data governance agent based on the upgraded domain large model and the extended small model.

[0018] As an alternative or supplement to the above solutions, in a method according to an embodiment of this application, providing a callable extended small model for the domain large model includes: connecting the extended small model to the domain large model through a model context protocol.

[0019] According to a second aspect of this application, a method for constructing a semantic benchmarking model is provided. The semantic benchmarking model includes a feature encoding layer, a similarity calculation layer, and an intelligent interaction layer. The method includes: acquiring a standard data set and a field set, and determining the standard data corresponding to each field in the field set; in the feature encoding layer, performing feature encoding on each standard data in the standard data set and each field in the field set to obtain a standard data encoding set and a field encoding set; in the similarity calculation layer, for each field encoding, calculating the similarity between the field encoding and each standard data encoding in the standard data encoding set based on an initial segmentation weight; adjusting the initial segmentation weight to obtain a final segmentation weight, wherein the final segmentation weight minimizes the difference between the standard data corresponding to the standard data encoding with the highest similarity to each field encoding and the standard data corresponding to that field.

[0020] According to a third aspect of this application, a system for constructing a data governance agent is provided, the system comprising: a memory; a processor coupled to the memory; and a computer program stored on the memory and running on the processor, the execution of the computer program causing the following operations: converting a knowledge base related to data governance into a dataset for fine-tuning; using the dataset to fine-tune a base large model through low-rank adaptation to generate a domain large model; providing callable extended small models for the domain large model, and constructing a data governance agent based on the domain large model and the extended small models; and updating the data governance agent based on user feedback for the data governance agent and upgrades to the base large model.

[0021] According to a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed, perform any one of the methods for constructing a data governance intelligent agent according to a first aspect of this application.

[0022] According to a fifth aspect of this application, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements any one of the methods for constructing a data governance intelligent agent according to a first aspect of this application.

[0023] The scheme for constructing a data governance intelligent agent according to one or more embodiments of this application, based on the natural language processing (NLP) and understanding capabilities of a large language model, fine-tunes it through low-rank adaptation to acquire enterprise or industry-related domain knowledge without requiring extensive parameter modifications. Furthermore, by constructing a semantic benchmarking model and integrating it as an extended small model into the fine-tuned domain large model, the domain large model can understand business data of different standards or specifications and provide standardized output results. Therefore, data governance can be performed efficiently, comprehensively, and accurately without relying on manual querying or traditional rule engine matching. Attached Figure Description

[0024] The above and / or other aspects and advantages of this application will become clearer and more readily understood from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are denoted by the same reference numerals. In the drawings: Figure 1 This is a flowchart of a method 100 for constructing a data governance intelligent agent according to an embodiment of this application; Figure 2 This is a flowchart of a process 200 for fine-tuning a base large model through low-rank adaptation according to an embodiment of this application; Figure 3 This is a hierarchical architecture diagram of a semantic benchmarking model 300 according to an embodiment of this application; Figure 4 This is a schematic diagram illustrating feature encoding of standard data and fields according to an embodiment of this application; Figure 5 This is a block diagram of a system 50 for constructing a data governance intelligent agent according to an embodiment of this application. Detailed Implementation

[0025] The following detailed description is merely exemplary in nature and is not intended to limit the disclosed technology or its application and use. Furthermore, it is not intended to be bound by any express or implied theory presented in the foregoing technical fields, background art, or the following detailed description.

[0026] In the following detailed description of the embodiments, numerous specific details are set forth in order to provide a more thorough understanding of the disclosed technology. However, it will be apparent to those skilled in the art that the disclosed technology can be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0027] Terms such as "possessing" and "comprising" indicate that, in addition to the units (modules) and steps directly and explicitly stated in the specification and claims, the technical solution of this application does not exclude the presence of other units (modules) and steps not directly or explicitly stated. Terms such as "first" and "second" do not indicate the order of the units in terms of time, space, size, etc., but are merely used to distinguish the units. Furthermore, the steps in this document are not limited to being performed in the order they are written; a step written later may be performed simultaneously with or before a step written earlier.

[0028] Furthermore, it should be noted that the user information and data involved in one or more embodiments of this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0029] Currently, enterprise data governance primarily relies on single keyword matching or manual data processing, which may not yield comprehensive data. While some rule-based automated data governance solutions exist, these rules or systems may lack flexibility or accuracy. With the development of artificial intelligence technology, applying the natural language processing capabilities of large language models to data governance can significantly improve efficiency.

[0030] The proposed solution for constructing a data governance agent can be widely applied to data governance scenarios in enterprises or industries, especially for enterprise scenarios with complex and inconsistent business data. Building upon the natural language processing capabilities of a large language model, this solution improves the insufficient accuracy of the basic large model in vertical domains through low-rank adaptation fine-tuning. Furthermore, based on existing data standards and business fields, a semantic benchmarking model is constructed (which can be called by the fine-tuned domain large model as an extended small model), thereby achieving accurate and comprehensive matching of fields or business data across different formats, specifications, and usage habits. Moreover, the agent constructed based on the aforementioned fine-tuned domain large model and extended small models possesses continuous iteration and evolution capabilities. When user feedback is received or new functions need to be introduced, flexible customization of new functions can be achieved by connecting additional extended small models (e.g., anomaly detection models). When a new version of the adopted basic large model is released, low-rank adaptation fine-tuning can be performed again for the new version of the basic model, and a new version of the data governance agent can be reconstructed.

[0031] In the following, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings.

[0032] refer to Figure 1 , Figure 1 This is a flowchart of a method 100 for constructing a data governance intelligent agent according to an embodiment of this application.

[0033] like Figure 1 As illustrated in the diagram, in step 102, the knowledge base related to data governance is converted into a dataset for fine-tuning. Step 102 can be understood as the data preparation and preprocessing steps required to build a data governance intelligent agent. Specifically, converting the knowledge base related to data governance into a dataset for fine-tuning may include the following steps: collecting current data standard specification files, statistical indicator caliber files, and structured metadata files; de-identifying the collected data standard specification files, statistical indicator caliber files, and structured metadata files; and converting the de-identified files into a dataset for fine-tuning, wherein each piece of data in the dataset includes input content and response content.

[0034] In some embodiments, the knowledge base related to data governance can be an enterprise knowledge base, which includes currently effective enterprise data standards, unified statistical indicator definitions, the meaning and interpretation of structured metadata, business scenarios related to key enterprise operations, and fundamental knowledge such as data usage. Files containing the above content can be stored in the enterprise's database in different formats as an enterprise or industry knowledge base. To build a data governance intelligent agent, this type of data needs to be collected first.

[0035] Next, sensitive information (such as customer names, mobile phone numbers, ID card numbers, or card numbers) included in the collected documents needs to be anonymized to avoid violating data privacy regulations or harming customer rights due to data breaches. Furthermore, these documents need to be uniformly converted into text with clear headings and accurate content (e.g., lightweight text in Markdown format).

[0036] Next, the anonymized lightweight files can be converted into a dataset for LoRA fine-tuning. The data in this dataset must have a uniform format; for example, each data point should include both input and response content. Furthermore, for the upcoming LoRA fine-tuning, the dataset needs to avoid erroneous or contradictory information, irrelevant content (which may dilute the target features), and duplicate data. In practice, the dataset for LoRA fine-tuning has relatively low requirements for data size, but very high requirements for data quality and data matching.

[0037] As an example, for companies in the financial payments sector, datasets used for LoRA fine-tuning could include the following "single-turn instruction-response dialogue data": Example 1: {"messages": [{"content": "What is a data standard?", "role": "user"}, {"content": "Refers to a consistent convention for the expression, format, and definition of data, including data business attributes, technical attributes, and..."} Management attributes. Business attributes mainly include business definitions and business rules; technical attributes mainly include data types, data... According to format, etc.; management attributes mainly include standard definition, standard status, etc., "role": "assistant"}]} Example 2: {"messages": [{"content": "What is the business definition of the card number?", "role": " user"}, {"content": "Bank cards are issued to the public by commercial banks and other financial institutions, and have consumer credit, transfer..." A bank card is a payment tool that provides all or some of the functions of account settlement, cash deposit and withdrawal. The bank card number identifies the issuing institution and the cardholder. The information number consists of the issuing institution identification code, a custom digit, and a check digit. Example 3: {"messages": [{"content": "What are the guidelines for filling in the merchant code?", "role": "user"}, {"content": "Enter a 15-character fixed-length set of numbers and letters. Letters should be uppercase.", "role": "assistant"}]} Each data message includes content corresponding to the user role and content corresponding to the assistant role. This data is characterized by its consistent format, clear objectives, and high accuracy, making it suitable for subsequent domain knowledge learning through LoRA fine-tuning. Based on this data, the foundational model can quickly focus on the mapping relationship between "user questions → correct answers," and thereby grasp the fundamental knowledge of the financial payment field.

[0038] Continue to refer to Figure 1 In step 104, the dataset is used to fine-tune the basic large model through low-rank adaptation to generate a domain-specific large model. Low-rank adaptation is an efficient technique for fine-tuning large model parameters. Its core idea is to reduce the parameter scale during fine-tuning through low-rank matrix factorization, thereby achieving rapid adaptation of the model to a specific task without modifying the backbone parameters of the pre-trained model. Specifically, current basic large models (e.g., DeepSeek, GPT, LLama, Qianwen, etc.) typically include a huge parameter matrix W. Directly adjusting the parameter matrix W requires significant GPU resources, storage resources, and time, and may cause the large model to lose some pre-trained knowledge. Therefore, low-rank adaptation can solve these problems through lightweight parameter updates without sacrificing fine-tuning effectiveness.

[0039] As an example, the parameter matrix W of the chosen underlying large model may have a large dimension (e.g., W∈R). d ×k d and k can be large values, for example, 2048 or 4096. During fine-tuning, the original parameter matrix W remains frozen, with only a pair of low-rank matrices U and V added next to it, where U ∈ R. d×r , V∈R r×k r is typically much smaller than the values ​​of d and k, for example, r = 8, 16, 32. Then, during LoRA fine-tuning, only the parameters of matrices U and V are optimized, without changing the parameters of the original matrix W, and the final parameter matrix W is formed based on the adjusted matrices U and V and the original matrix W. final W final=W + V × U. Compared to adjusting all parameters of the entire parameter matrix W (e.g., 2048 × 4096 parameters), LoRA fine-tuning only requires adjusting the parameters of the low-rank matrices U and V (e.g., 2048 × 8 and 8 × 4096 parameters respectively), which greatly reduces the required computational and storage resources as well as training time.

[0040] By fine-tuning the base model using the dataset obtained in step 102, the base model can learn industry knowledge in relevant fields (e.g., financial payments), thereby further meeting the capability requirements of subsequent data governance tasks. During the fine-tuning process, low-rank adaptation is used to significantly reduce the resources and time required for the fine-tuning process with almost no performance loss.

[0041] In some embodiments, the fine-tuning process can be further refined based on the partitioning of the dataset. (See reference) Figure 2 , Figure 2 This is a flowchart of a process 200 for fine-tuning a base large model through low-rank adaptation according to an embodiment of this application. Figure 2 As illustrated in the diagram, fine-tuning the basic large model through low-rank adaptation includes: in step 202, fine-tuning the low-rank matrix using a general data governance dataset; in step 204, fine-tuning the low-rank matrix again using an additional dataset related to the business scenario; and in step 206, scoring the output of the fine-tuned domain large model and supplementing the correct output corresponding to the low-scoring output results.

[0042] Specifically, in step 202, which is the first fine-tuning phase, the general data governance dataset includes some common concepts and logic of data governance (e.g., basic definitions of data standards, statistical indicators, etc.). Based on these general datasets, the basic big model can learn the definitions and background related to data governance.

[0043] Next, in step 204, the second fine-tuning stage, further fine-tuning can be performed on key scenarios. For example, for financial payment companies, these key scenarios may include, but are not limited to, "preferential subsidies," "overseas visitors to China," "trade-in programs," and similar situations. The additional dataset used in step 204 may differ from the dataset formed in step 102. The purpose of using this additional dataset for secondary fine-tuning is to enhance the performance of the base model for specific scenarios.

[0044] Finally, in step 206, the output of the fine-tuned domain model can be scored and supplemented. For example, after completing the fine-tuning in steps 202 and 204, the output of the large model under a given input (with a corresponding expected output) can be scored. A higher score is obtained when the output of the large model is close to the expected output; a lower score is obtained when the output of the large model is far from the expected output. Then, the outputs with the lowest scores (e.g., below 60 points) are filtered out, and the inputs corresponding to the lower outputs and the expected outputs of these inputs are used as a supplementary dataset. The large model is then fine-tuned in the third stage using the supplementary dataset.

[0045] return Figure 1 Next, in step 106, callable extended small models are provided for the domain-wide model, and a data governance agent is constructed based on the domain-wide model and the extended small models. Specifically, after LoRA fine-tuning, the generated domain-wide model, in addition to possessing the natural language processing and understanding capabilities of the basic model, can also understand the professional knowledge of vertical domains (e.g., financial payments). However, the domain-wide model still has shortcomings in its ability or efficiency in performing certain specific functions. To address this issue, extended small models can be provided for the domain-wide model. When the large model needs to perform certain functions, it does not process them itself, but instead calls the corresponding extended small models through an interface, provides them with the required input, and obtains the output from these extended small models. In this way, the professional capabilities of various extended small models in specific tasks can be coupled with the basic capabilities of the domain-wide model, thereby enhancing the flexibility and scalability of the agent and broadening the application boundaries of the agent.

[0046] In some embodiments, the extended small model includes a semantic mapping model and an anomaly detection model. Specifically, the semantic mapping model can be a functional model specifically designed to find planned data. As an example, in the data governance process, the task instructions provided by the operator may include data or fields that differ from standard data. In this case, the data governance agent is expected to be able to map the data or fields in the task instructions to standard data, and then search and process them in the database using the standard data. As an example, the operator's instruction might be "to count the issuing outlets of IC cards issued within a certain week." For the domain large model, "IC card" or "issuing outlet" may not be standard data, and it may be difficult to map it to data or fields in the database. In this case, by invoking the semantic mapping model, the domain large model can obtain standard data (e.g., "bank card") corresponding to "IC card" that can be used for matching queries in the database. Similarly, the semantic mapping model can also determine labeled data (e.g., "branch office") corresponding to "issuing outlet".

[0047] On the other hand, anomaly detection models can identify logical contradictions based on context awareness. For example, the discount amount calculated according to a certain discount rule might be greater than the transaction amount. Then, based on the output of the anomaly detection model, the domain-wide model can identify logical contradictions in the calculation results.

[0048] Next, as an example, the architecture and training process of the semantic mapping model will be described in detail with reference to one or more embodiments. References Figure 3 , Figure 3 This is a hierarchical architecture diagram of a semantic benchmarking model 300 according to an embodiment of this application.

[0049] It should be understood that, in addition to being integrated into larger domain models as extensions of smaller models to construct data governance intelligent agents, Figure 3 The semantic benchmarking model 300 and its construction method illustrated in the figure can also be used independently as a tool or module. That is, the semantic benchmarking model 300 can operate independently of the basic large model, domain large model, or data governance intelligent agent for purposes such as standardizing and normalizing business data. Therefore, this application also provides a semantic benchmarking model and a method for constructing the semantic benchmarking model.

[0050] like Figure 3 As illustrated in the diagram, the semantic mapping model 300 may include an intelligent interaction layer 302, a similarity calculation layer 304, and a feature encoding layer 306. Specifically, the intelligent interaction layer 302 is located at the top layer of the semantic mapping model and is used to interact with the accessed domain large model. In some embodiments, the semantic mapping model 300 can be encapsulated as a dedicated tool to access the domain large model described above through the Model Context Protocol (MCP) in the intelligent interaction layer. This allows the semantic mapping model 300 to obtain input from the domain large model via the intelligent interaction layer 302, process it, and return the resulting output to the domain large model. MCP is an open protocol designed to achieve seamless integration between large language model applications and external data sources, tools, and services. In some embodiments, the semantic mapping model 300 may be deployed in the same domain as the enterprise's database or knowledge base, and the domain large model may be deployed in another domain. The domain large model can only obtain standard data corresponding to the provided fields or the similarity between fields and standard data from the semantic mapping model (e.g., as an MCP server) via an MCP client, and cannot directly access the enterprise's database or knowledge base.

[0051] Next, in the feature encoding layer 306, feature encoding operations can be performed on the input fields from the domain large model to obtain field encodings.

[0052] Then, in the similarity calculation layer 304, the similarity between the field code and all standard data codes can be calculated, and the standard data code with the highest similarity to the field code can be determined. Based on the standard data code, the standard data corresponding to the input field can be determined. The construction process of the semantic benchmarking model will be described in detail below.

[0053] In some embodiments, such as Figure 3 The semantic mapping model 300 of the illustrated architecture can be constructed through three stages: data definition, feature encoding, and model training. In the data definition stage, a standard data set Std and a field set F are obtained, and each field f in the field set F is determined. i The corresponding standard data std i Then, in the feature encoding stage, the feature encoding layer processes each standard data point std in the standard data set Std. i And feature encoding f for each field in the field set F. i This is done to obtain a standard data encoding set A and a field encoding set B; finally, during the model training phase, the similarity calculation layer is used to encode each field B. i The field code B is calculated based on the initial segment weights. i The similarity S with each standard data code in the standard data encoding set; and the adjustment of the initial segmentation weights to obtain final segmentation weights, the final segmentation weights being such that the similarity B with each field code is... i The standard data y corresponding to the standard data encoding with the highest similarity i ’ The standard data y corresponding to this field i The difference is minimal.

[0054] First, in the data definition phase, define a standard data set Std={std1,std2,...,std...} n}, where std i This represents a standard data item; similarly, it retrieves the set of fields F={f1,f2,...,f...} used in the company's existing business operations. m}, where f i This indicates the field used in a specific business function. In fact, f i This can correspond to a specific item in the standard data set Std, and therefore, the reference relationship between the field set F and the standard data set Std can be represented as Y = {y1, y2, ..., y}. m}, y i∈ Std. As an example, in the example of the bank card described above, in the data specifications or standards of an enterprise, "bank card" is a standard data (for example, the standard Chinese name of std1 is "bank card"). However, in various documents or reports, different fields may be used to represent the meaning of "bank card" (for example, the Chinese name of the field f3 from a certain document is "payment card"). Then, the standard data referred to by this field can be determined (for example, y3 = std1, with the standard Chinese name being "bank card").

[0055] Next, for the feature encoding stage, refer to Figure 4 , Figure 4 which is a schematic diagram of feature encoding for standard data and fields according to an embodiment of the present application. As Figure 4 shown in the figure, the standard data std i includes four subsections: the standard English name of the data, the standard Chinese name of the data, the business definition, and the value-added specification. The field f i includes the field English name, the field Chinese name, the field description, the field type, and the field length.

[0056] Continue to refer to Figure 4 , and perform feature encoding on each standard data in the standard data set Std to obtain the standard data encoding set A = {A1, A2,..., A n}, and perform feature encoding on each field in the field set F to obtain the field encoding set B = {B1, B2,..., B m}. As Figure 4 shown in the figure, in some embodiments, in order to correspond to the subsections of the standard data, the field type and the field length (collectively referred to as the field format) can be encoded into the same subsection.

[0057] In some embodiments, the standard English name and the standard Chinese name of the data, as well as the field English name and the Chinese name, are encoded by One-Hot encoding based on word frequency. This encoding method assigns values in the 0-1 interval of different sizes according to different word segmentation probabilities. The higher the frequency of the data standard, the closer its encoding value is to 1, and the higher the subsection weight in the subsequent feature similarity calculation. In some embodiments, the business definition and the field description are encoded by Base64 encoding, and the value-added specification and the field format are encoded by encoding the meanings of English and digital characters to reduce semantic loss.

[0058] The data definition stage and the feature encoding stage are described above. Next, the model training stage for the similarity calculation layer will be described in detail. Specifically, in the model training stage, for the field encoding B iCalculate the similarity between the field code and each standard data code in the standard data code set A, and denote the standard data corresponding to the standard data code with the highest similarity as y. i ’ The similarity can be calculated using the following formula: Where S is the similarity, k is the number of segments between the field encoding and the standard data encoding, and λ is the similarity score. i It is the segment weight, a i It is a segment in standard data encoding, b i It refers to a segment in the field encoding.

[0059] The above description Figure 4 In this embodiment, each standard data code and each field code comprises four sections (i.e., k=4). Specifically, λ1 is the weight of the Chinese name section (i.e., the first section), λ2 is the weight of the English name section (i.e., the second section), λ3 is the weight of the business definition and field description section (i.e., the third section), and λ4 is the weight of the value-added specification or field format section (i.e., the third section). Section weight λ i The initial values ​​can be set to equal weights (e.g., each segment has a weight of 0.25). Then, as training progresses, the segment weights λ are gradually adjusted. i Until it matches field code B i The standard data y corresponding to the standard data encoding with the highest similarity i ’ The standard data y corresponding to this field i The difference is minimal. For example, due to differences in terminology and expression habits, the section weight corresponding to the Chinese and English names of fields may be smaller in the final section weight, while the section weight corresponding to the field description may be larger. In this case, even if an unconventional description such as "payment card" refers to "bank card," the content in its field description (which may include the issuing institution code, such as...) will still have a significant impact. Figure 4 The high similarity between "payment card" and "bank card" in the diagram shows that the semantic benchmarking model can also find the standard data of "bank card" based on "payment card".

[0060] Specifically, in the embodiments of the bank card described above, after data definition and feature encoding, the field code of field f3 with the Chinese name "payment card" may be B3, the standard data code of standard data std1 with the Chinese name "bank card" may be A1, and the field code of another standard data std5 field with the Chinese name "payment code" may be A5. By gradually adjusting the section weights until the similarity between the field code B3 and the standard data code A1 is the highest similarity in the similarity ranking of B3 and all standard data codes (for example, higher than the similarity between the field code B3 and the standard data code A5), it can be determined that the semantic alignment model can correctly align the field to the standard data, otherwise continue to adjust the section weights. In some embodiments, the field code set B can be further divided into a training set, a validation set, and a test set. In each round of training, the section weight λ i is adjusted, and the accuracy rate on the validation set is calculated. After multiple rounds of training, the section weight λ with the maximum accuracy rate is selected i as the final section weight.

[0061] Return Figure 1 , after completing the construction of the data governance agent based on the extended small model and the domain large model through step 106, it can be combined with the data governance platform of the traditional B / S architecture and deployed to various application scenarios. The application scenarios include but are not limited to metadata management, data quality control, and data standardization and normalization. Then, the operator can interact with the data governance agent through the intelligent application. The data governance agent uses the domain large model to understand the natural language input by the operator, and can call the semantic alignment model to find appropriate standard data, and finally process, analyze, and / or associate based on these standard data to obtain an output that meets the operator's requirements or complete the data governance task proposed by the operator. Next, in step 108, the data governance agent can also be updated according to the user feedback on the data governance agent and the upgrade of the base large model.

[0062] In some embodiments, updating the data governance agent includes: in response to the version upgrade of the base large model, reusing the data set to fine-tune the upgraded base large model to generate an upgraded domain large model; and reconstructing the updated data governance agent based on the upgraded domain large model and the extended small model.

[0063] Specifically, since the data governance agent in this application is built based on a domain-wide model and extended small models, on the one hand, additional extended small models for specific functions can be built according to user needs and feedback, and integrated into the domain-wide model through a model context protocol, thereby improving the performance of the data governance agent for the specific functions. On the other hand, when a new version of the base model is released (e.g., Llama 4 replacing Llama 3), the new version of the base model can be fine-tuned using LoRA to generate a new version of the domain-wide model, which can then be combined with the extended small models to construct a new version of the data governance agent, ensuring the agent's technological leadership. Similarly, during the updating and upgrading of the data governance agent, if improved versions of LoRA technology emerge (e.g., more efficient low-rank decomposition algorithms), they can also be used in the generation process of the domain-wide model.

[0064] Figure 5 This is a block diagram of a system 50 for constructing a data governance intelligent agent according to an embodiment of this application. The system 50 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520, the execution of which causes the method 100 described above to be performed.

[0065] Additionally, as described above, this application can also be implemented as a computer-readable storage medium storing information for making such... Figure 1 The process of method 100 shown is the program executed. Here, computer-readable storage media can be various types of computer-readable storage media, such as disks (e.g., magnetic disks, optical disks, etc.), cards (e.g., memory cards, optical cards, etc.), semiconductor memory (e.g., ROM, non-volatile memory, etc.), and tapes (e.g., magnetic tape, cassette tape, etc.).

[0066] This application can also be implemented as a computer program product, comprising a computer program that, when executed by a processor, implements, as described above. Figure 1 The steps of method 100 shown.

[0067] Where applicable, the various embodiments provided in this application may be implemented using hardware, software, or a combination of hardware and software. Furthermore, where applicable, without departing from the scope of this application, the various hardware and / or software components described herein may be combined into composite components comprising software, hardware, and / or both. Where applicable, without departing from the scope of this application, the various hardware and / or software components described herein may be divided into sub-components comprising software, hardware, or both. Additionally, where applicable, it is contemplated that software components may be implemented as hardware components, and vice versa.

[0068] The software (such as program code and / or data) according to this application can be stored on one or more computer storage media. It is also contemplated that the software identified herein can be implemented using one or more networked and / or otherwise general-purpose or special-purpose computers and / or computer systems. Where applicable, the order of the various steps described herein can be changed, combined into compound steps, and / or divided into sub-steps to provide the features described herein.

[0069] The embodiments and examples presented herein are provided to best illustrate embodiments of this application and its particular applications, thereby enabling those skilled in the art to implement and use this application. However, those skilled in the art will understand that the above description and examples are provided for ease of illustration and example only. The descriptions presented are not intended to cover all aspects of this application or to limit this application to the precise forms disclosed.

Claims

1. A method for constructing a data governance intelligent agent, characterized in that, The method includes: Transform the knowledge base related to data governance into a dataset for fine-tuning; The dataset is used to fine-tune the base large model through low-rank adaptation to generate a domain-specific large model; Provide callable extended small models for the large domain model, and construct a data governance intelligent agent based on the large domain model and the extended small models; The data governance agent is updated based on user feedback regarding the data governance agent and upgrades to the underlying big model.

2. The method according to claim 1, wherein, The extended mini-models include a semantic benchmarking model and an anomaly detection model.

3. The method according to claim 2, wherein, The semantic benchmarking model includes a feature encoding layer, a similarity calculation layer, and an intelligent interaction layer for interacting with the domain-wide model.

4. The method according to claim 3, wherein, The semantic benchmarking model is constructed through the following steps: Obtain the standard data set and field set, and determine the standard data corresponding to each field in the field set; Through the feature encoding layer, feature encoding is performed on each standard data in the standard data set and each field in the field set to obtain a standard data encoding set and a field encoding set; Through the similarity calculation layer, for each field encoding, the similarity between the field encoding and each standard data encoding in the standard data encoding set is calculated based on the initial segmentation weight; The initial segment weights are adjusted to obtain the final segment weights, which minimize the difference between the standard data corresponding to the standard data encoding with the highest similarity to each field encoding and the standard data corresponding to that field.

5. The method according to claim 4, wherein, In the similarity calculation layer, the similarity between each field encoding and the standard data encoding is calculated using the following formula: Where S is the similarity, k is the number of segments between the field encoding and the standard data encoding, and λ is the similarity score. i It is the segment weight, a i It is a segment in standard data encoding, b i It refers to a segment in the field encoding.

6. The method according to claim 4, wherein, The field encoding set is further divided into a training set, a validation set, and a test set to determine the final segment weights.

7. The method according to claim 4, wherein, The standard data includes: the standard English name of the data, the standard Chinese name of the data, the business definition, and the value-added specifications; and the fields include: the English name of the field, the Chinese name of the field, the field description, the field type, and the field length.

8. The method according to claim 7, wherein, The field type and the field length are encoded into the same segment.

9. The method according to claim 7, wherein, In the feature encoding layer, the standard English and standard Chinese names of the data, as well as the English and Chinese names of the fields, are encoded using One-Hot encoding based on word frequency.

10. The method according to claim 7, wherein, In the feature encoding layer, the business definition and field description are encoded using Base64 encoding.

11. The method according to claim 1, wherein, Using the aforementioned dataset to fine-tune the base large model via low-rank adaptation includes: Fine-tuning of low-rank matrices using a common data governance dataset; The low-rank matrix was fine-tuned again using an additional dataset relevant to the business scenario; and The output of the finely tuned domain model is scored, and the correct output corresponding to the low-scoring output is supplemented.

12. The method according to claim 1, wherein, Converting a knowledge base related to data governance into a dataset for fine-tuning includes: Collect current data standard specifications, statistical indicator definitions, and structured metadata files; The collected data, including standard specification documents, statistical indicator documents, and structured metadata files, are anonymized. The anonymized files are converted into a dataset for fine-tuning, where each data entry includes input and response content.

13. The method according to claim 1, wherein, Updating the data governance agent includes: In response to the version upgrade of the base model, the upgraded base model is fine-tuned using the dataset to generate an upgraded domain model; and The updated data governance agent is reconstructed based on the upgraded domain big model and the extended small model.

14. The method according to claim 1, wherein, Providing callable extended small models for the large domain model includes: connecting the extended small models to the large domain model through a model context protocol.

15. A method for constructing a semantic benchmarking model, characterized in that, The semantic alignment model includes a feature encoding layer, a similarity calculation layer, and an intelligent interaction layer; the method includes: Obtain the standard data set and field set, and determine the standard data corresponding to each field in the field set; In the feature encoding layer, feature encoding is performed on each standard data in the standard data set and each field in the field set to obtain a standard data encoding set and a field encoding set; In the similarity calculation layer, for each field code, the similarity between the field code and each standard data code in the standard data code set is calculated based on the initial segmentation weight; The initial segment weights are adjusted to obtain the final segment weights, which minimize the difference between the standard data corresponding to the standard data encoding with the highest similarity to each field encoding and the standard data corresponding to that field.

16. The method according to claim 15, wherein, In the similarity calculation layer, the similarity between each field encoding and the standard data encoding is calculated using the following formula: Where S is the similarity, k is the number of segments between the field encoding and the standard data encoding, and λ is the similarity score. i It is the segment weight, a i It is a segment in standard data encoding, b i It refers to a segment in the field encoding.

17. The method according to claim 15, wherein, The field encoding set is further divided into a training set, a validation set, and a test set to determine the final segment weights.

18. The method according to claim 15, wherein, The standard data includes: the standard English name of the data, the standard Chinese name of the data, the business definition, and the value-added specifications; and the fields include: the English name of the field, the Chinese name of the field, the field description, the field type, and the field length.

19. The method according to claim 18, wherein, The field type and the field length are encoded into the same segment.

20. The method according to claim 18, wherein, In the feature encoding layer, the standard English and standard Chinese names of the data, as well as the English and Chinese names of the fields, are encoded using One-Hot encoding based on word frequency.

21. The method according to claim 18, wherein, In the feature encoding layer, the business definition and field description are encoded using Base64 encoding.

22. The method according to claim 15, wherein, The semantic benchmarking model is connected to the large language model in the intelligent interaction layer through the model context protocol for the purpose of constructing a data governance intelligent agent.

23. The method according to claim 22, wherein, The large language model is a domain-specific large model generated by performing low-rank adaptation fine-tuning on a basic large model.

24. A system for constructing a data governance intelligent agent, characterized in that, The system includes: a memory; a processor coupled to the memory; and a computer program stored on the memory and running on the processor, the execution of which causes the following operations: Transform the knowledge base related to data governance into a dataset for fine-tuning; The dataset is used to fine-tune the base large model through low-rank adaptation to generate a domain-specific large model; Provide callable extended small models for the large domain model, and construct a data governance intelligent agent based on the large domain model and the extended small models; The data governance agent is updated based on user feedback regarding the data governance agent and upgrades to the underlying big model.

25. The system according to claim 24, wherein, The extended mini-models include a semantic benchmarking model and an anomaly detection model.

26. The system according to claim 24, wherein, Using the aforementioned dataset to fine-tune the base large model via low-rank adaptation includes: Fine-tuning of low-rank matrices using a common data governance dataset; The low-rank matrix was fine-tuned again using an additional dataset relevant to the business scenario; and The output of the finely tuned domain model is scored, and the correct output corresponding to the low-scoring output is supplemented.

27. The system according to claim 24, wherein, Converting a knowledge base related to data governance into a dataset for fine-tuning includes: Collect current data standard specifications, statistical indicator definitions, and structured metadata files; The collected data, including standard specification documents, statistical indicator documents, and structured metadata files, are anonymized. The anonymized files are converted into a dataset for fine-tuning, where each data entry includes input and response content.

28. The system according to claim 24, wherein, Updating the data governance agent includes: In response to the version upgrade of the base model, the upgraded base model is fine-tuned using the dataset to generate an upgraded domain model; and The updated data governance agent is reconstructed based on the upgraded domain big model and the extended small model.

29. The system according to claim 24, wherein, Providing callable extended small models for the large domain model includes: connecting the extended small models to the large domain model through a model context protocol.

30. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed, perform the method according to any one of claims 1-14.

31. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-14.

Images