A standardized data governance system based on RAG and large model warehouse

Abstract

The present application relates to the technical field of data governance, and particularly relates to a standardized data governance system based on RAG and a large model warehouse, which comprises a data fusion module, a standard knowledge evolution module, a preliminary inspection module and a RAG enhancement and model warehouse module; the data fusion module is used for obtaining a multi-source unified semantic data set; the standard knowledge evolution module is used for forming a standard theme library; the preliminary inspection module is used for generating a first search result based on an external task request; the RAG enhancement and model warehouse module is used for generating a second search result based on a preset multi-modal model, the external task request and the first search result. By constructing a standard theme library to form a first search result, the present application verifies the result formed by the RAG enhancement and model warehouse module as a boundary condition, improves the accuracy of the data generation result, and prevents model reasoning deviation and error knowledge diffusion.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, specifically to a standardized data governance system based on RAG and a large model repository. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, enterprises and institutions have accumulated a large amount of data from diverse sources, in complex formats, and with inconsistent semantics. This data is widely distributed across different business systems and databases, including both structured data such as tables and records, and unstructured data such as documents, images, and log files. Due to the lack of unified data standards and significant differences in semantic expression, traditional data governance systems face significant bottlenecks in achieving cross-source data integration, standardized management, and knowledge unification.

[0003] In existing technologies, data governance mainly relies on rule matching and static ontology alignment for semantic mapping and standardization. However, when faced with dynamically evolving business knowledge systems and complex external task requests, it often fails to achieve high-precision knowledge reconstruction and semantic reasoning. Furthermore, traditional model invocation mechanisms are relatively simple, lacking multimodal collaborative reasoning and adaptive enhancement capabilities. This results in low accuracy and poor response efficiency when the system responds to complex task requests (such as cross-domain data analysis, intelligent question answering, and governance decisions).

[0004] In recent years, the development of Retrieval-Augmented Generation (RAG) technology has provided new ideas for knowledge retrieval and generation in data governance. By combining the advantages of information retrieval and generative models, RAG can achieve dynamic information retrieval and semantically enhanced generation in large-scale knowledge bases. However, existing RAG applications are mostly limited to single-model architectures and lack deep integration mechanisms with domain-specific model repositories, making it difficult to support complex and multi-scenario data governance tasks. Summary of the Invention

[0005] (a) Purpose of the invention

[0006] The purpose of this invention is to provide a standardized data governance system based on RAG and a large model repository. By constructing a standard topic library, a first retrieval result is formed, which serves as a boundary condition to verify the results generated by the RAG enhancement and model repository modules, thereby improving the accuracy of the data generation results and preventing model inference bias and the spread of erroneous knowledge.

[0007] (II) Technical Solution

[0008] To address the aforementioned issues, this invention provides a standardized data governance system based on RAG and a large model repository, comprising: a data fusion module, a standard knowledge evolution module, an initial inspection module, and a RAG enhancement and model repository module;

[0009] The data fusion module is used to extract structured and unstructured data from multiple distributed databases and obtain a unified semantic dataset from multiple sources based on a cross-source semantic alignment algorithm.

[0010] The standard knowledge evolution module is used to reconstruct the semantic dataset using a preset corpus algorithm to form a standard topic library.

[0011] The initial detection module is used to search the standard topic library based on an external task request and generate the first search result of the task request;

[0012] The RAG enhancement and model repository module is used to perform inference generation based on a preset multimodal model, external task requests, and the first retrieval result to obtain the second retrieval result.

[0013] In another aspect of the present invention, preferably, the data fusion module extracts structured and unstructured data from multiple distributed databases, and obtains a unified semantic dataset from multiple sources based on a cross-source semantic alignment algorithm, including:

[0014] Configure connections to multiple distributed databases and establish data access channels;

[0015] Perform field mapping and primary key identification on the structured data to generate a preliminary structured index table;

[0016] Feature extraction is performed on the unstructured data, and semantic vectors are extracted using a text embedding model;

[0017] Based on the cross-source semantic alignment algorithm, the structured index table and the semantic vector are semantically aggregated and matched to obtain the cross-source entity alignment result;

[0018] Based on the cross-source entity alignment results, a multi-source semantic dataset in a unified semantic space is generated.

[0019] In another aspect of the present invention, preferably, the standard knowledge evolution module is used to perform standard semantic reconstruction on the semantic dataset using a preset corpus algorithm to form a standard topic library, including:

[0020] Domain feature recognition and term extraction are performed on the semantic dataset to obtain topics and related term sets;

[0021] Based on the topic, the semantic similarity between terms in the relevant term set is calculated using a preset standard domain corpus algorithm, and the semantically similar terms are reconstructed according to the similarity threshold to generate a standardized term set.

[0022] Based on the semantic hierarchy, logical dependencies, and association strength among the terms in the standardized terminology set, semantic relationship edges are constructed;

[0023] Based on the standardized terminology set and semantic relationship edges, a standard topic library is generated.

[0024] In another aspect of the present invention, preferably, based on the topic, the semantic similarity between terms in a related term set is calculated using a preset standard domain corpus algorithm, and semantically similar terms are reconstructed using standard semantics based on a similarity threshold to generate a standardized term set, including:

[0025] Based on the topic, a corresponding preset standard domain corpus algorithm is determined, and the preset standard domain corpus algorithm is constructed based on the topic corpus;

[0026] Each term in the relevant term set is vectorized to obtain a semantic embedding vector for the term;

[0027] The cosine similarity calculation model is used to calculate the semantic similarity score between any two terms;

[0028] When the semantic similarity score is greater than or equal to the set similarity threshold, the corresponding terms are divided into the same semantic cluster. The similarity threshold is determined based on the topic.

[0029] Identify the main terms within each semantic cluster, the identification of which is based on the frequency of occurrence of each term within the semantic cluster in a topic-based corpus;

[0030] The main term is used to replace synonyms or near-synonyms within the cluster to form a standardized term set.

[0031] In another aspect of the present invention, preferably, the initial detection module is used to search the standard topic library based on an external task request, and generate a first search result for the task request, including:

[0032] Semantic parsing is performed on received external task requests to extract task intent vectors and determine the corresponding standard topic library;

[0033] Based on the task intent vector, the corresponding standard topic library is searched to obtain an initial candidate set;

[0034] Based on the initial candidate set, a structured answer is generated, which is the first search result.

[0035] In another aspect of the present invention, preferably, the step of semantically parsing the received external task request, extracting the task intent vector, and determining the corresponding standard topic library includes:

[0036] External task requests are preprocessed with text, including word segmentation, stop word removal, and part-of-speech tagging.

[0037] Extract the contextual semantic features of the preprocessed text to identify the semantic components and key entities in the task request;

[0038] The semantic components and key entities are vectorized to generate semantic vectors, which serve as task intent vectors.

[0039] Based on the task intent vector, the similarity of the semantic vectors of each topic in the standard topic library is calculated to determine the corresponding standard topic library.

[0040] In another aspect of the present invention, preferably, the step of generating structured answer content based on the initial candidate set, wherein the structured answer content is a first search result, includes:

[0041] The initial candidate set is semantically encoded to generate candidate semantic vectors;

[0042] The candidate semantic vectors are fused with the task intent vectors to obtain the fused semantic representation;

[0043] Content generation is performed on the fused semantic representation to generate structured answer content, which is the first search result.

[0044] In another aspect of the present invention, preferably, the preset multimodal model includes at least one general large model and several topic models.

[0045] In another aspect of the present invention, preferably, the model repository module performs reasoning generation based on a preset multimodal model, external task requests, and the first search result to obtain a second search result, including:

[0046] Based on the external task request, determine the corresponding topic model;

[0047] Based on the external task request, joint semantic reasoning is performed on the external task request using the general large model and the corresponding topic model to obtain the reasoning result;

[0048] The first search result is used as a boundary condition to verify the reasoning result, and a second search result is obtained.

[0049] In another aspect of the present invention, preferably, the first search result is used as a boundary condition to verify the reasoning result to obtain a second search result, including:

[0050] Construct a constraint set based on the first search results;

[0051] The reasoning results are verified using the constraint set.

[0052] When the reasoning result satisfies the constraint set, the reasoning result is confirmed to be valid and output as the second search result;

[0053] When the reasoning result deviates from the constraint set, the reasoning result is adaptively corrected based on the degree of deviation, and the second search result is output after the update.

[0054] (III) Beneficial Effects

[0055] The above-described technical solution of the present invention has the following beneficial technical effects:

[0056] This invention improves the accuracy of cross-source data fusion by constructing a standard topic library and a multi-source semantic dataset, achieving unified data management and standardized processing. The system uses the first retrieval result generated by the initial detection module as boundary conditions to verify the inference results of the RAG enhancement and model repository modules, effectively preventing model inference bias and the spread of erroneous knowledge, thereby improving the accuracy and reliability of the generated results. Through joint inference of multimodal models and external task requests, the system can comprehensively process multi-source information such as text, structured data, and images, achieving high-precision responses to complex task requests. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall structure of one embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0059] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0060] In the description of this invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0061] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0062] The invention will now be described in more detail with reference to the accompanying drawings. In the various drawings, the same elements are indicated by similar reference numerals. For clarity, the various parts in the drawings are not drawn to scale.

[0063] Example 1

[0064] A standardized data governance system based on RAG and a large model repository. Figure 1 A schematic diagram of the overall structure of an embodiment of the present invention is shown, as follows. Figure 1 As shown, it includes: a data fusion module, a standard knowledge evolution module, a preliminary inspection module, and a RAG enhancement and model repository module;

[0065] The data fusion module extracts structured and unstructured data from multiple distributed databases and obtains a unified semantic dataset based on a cross-source semantic alignment algorithm. These distributed databases include relational databases, NoSQL databases, document repositories, log systems, and multimodal data sources such as images and videos. The cross-source semantic alignment algorithm maps heterogeneous information from different data sources to a unified semantic space, generating a unified semantic dataset. This module can also perform data cleaning, deduplication, and format standardization based on the characteristics of the data sources, ensuring the integrity, consistency, and parsability of the fused data.

[0066] Furthermore, in this embodiment, the data fusion module extracts structured and unstructured data from multiple distributed databases and obtains a unified semantic dataset from multiple sources based on a cross-source semantic alignment algorithm, including:

[0067] Configure connections to multiple distributed databases and establish data access channels. The connection configuration process can be completed through a unified interface or driver, ensuring the real-time performance and stability of data access, while also supporting data permission management and access control.

[0068] Field mapping and primary key identification are performed on the structured data to generate a preliminary structured index table. Field mapping achieves attribute standardization by matching fields with the same meaning but different names in different databases. Primary key identification is used to determine the unique identifier of data records to form a preliminary structured index table for subsequent accurate matching and aggregation of cross-source entities.

[0069] Feature extraction is performed on the unstructured data, and semantic vectors are extracted using a text embedding model. For unstructured data, including text, documents, logs, and images, feature extraction techniques are used to transform the original content into a processable semantic representation. Text-based data generates semantic vectors through a text embedding model, while image-based data obtains feature vectors through a visual feature extraction model, thereby achieving a unified semantic representation for different types of data and providing a foundation for subsequent cross-source alignment.

[0070] Based on the cross-source semantic alignment algorithm, the structured index table and the semantic vector are semantically aggregated and matched to obtain cross-source entity alignment results. The cross-source semantic alignment algorithm can identify multiple expressions of the same entity in different data sources, and generate cross-source entity alignment results through semantic similarity calculation, entity disambiguation and cluster analysis to ensure the consistency of the same entity in multi-source data.

[0071] Based on the cross-source entity alignment results, a multi-source semantic dataset under a unified semantic space is generated. During the fusion process, not only are the key attributes and feature information of each data source preserved, but redundancy and conflicts are also eliminated through a unified semantic standard, forming a high-quality, unified multi-source dataset usable by the standard knowledge evolution module, thus achieving semantic unification and integrated management of cross-source data.

[0072] The standard knowledge evolution module is used to reconstruct the semantic dataset using a pre-defined corpus algorithm, forming a standard topic library. This standard topic library enables the system to form a reusable and scalable standardized knowledge structure. Simultaneously, it allows for dynamic knowledge evolution, meaning that the standard topic library is continuously updated and optimized based on new data or external knowledge sources during system operation, ensuring the timeliness and completeness of the knowledge base.

[0073] Furthermore, in this embodiment, the standard knowledge evolution module is used to perform standard semantic reconstruction on the semantic dataset using a preset corpus algorithm to form a standard topic library, including:

[0074] The semantic dataset is subjected to domain feature recognition and term extraction to obtain a set of topics and related terms. The semantic dataset is labeled with features using a pre-set domain dictionary, statistical co-occurrence model or word vector embedding model, such as Word2Vec, BERT or SimCSE. High-frequency professional words and terms in the text are extracted based on word frequency statistics, TF-IDF weighting and dependency parsing. The extracted terms are initially grouped according to topic clustering algorithms, such as LDA or K-Means, to obtain a set of topics and related terms.

[0075] Based on the stated topic, a preset standard domain corpus algorithm is used to calculate the semantic similarity between terms in the relevant term set. Then, based on a similarity threshold, semantically similar terms are reconstructed using standard semantics to generate a standardized term set. The corpus algorithm may include a semantic embedding algorithm based on a pre-trained language model, which quantifies the semantic similarity relationship between terms by calculating cosine similarity, Euclidean distance, or KL divergence in the standard domain corpus. Based on a preset similarity threshold, such as 0.8, semantically close term pairs are selected and mapped to a unified standard term representation.

[0076] Based on the semantic hierarchy, logical dependency, and association strength among the terms in the standardized terminology set, semantic relationship edges are constructed. The standard semantic reconstruction unit systematically models the semantic hierarchy, logical dependency, and association strength among the terms based on the standardized terminology set, including: identifying the hierarchical levels of terms based on hierarchical clustering algorithms or graph structure analysis methods (such as PageRank or GNN); establishing logical dependency chains according to semantic dependency rules (such as "belongs to", "contains", "depends on"); and calculating the association strength index among terms (such as mutual information MI or co-occurrence probability) to determine the edge weights of semantic relationships.

[0077] Based on the standardized terminology set and semantic relationship edges, a standard topic library is generated. Each standard topic node includes a topic identifier, a core terminology set, a semantic hierarchy structure, and relationship weight attributes. Nodes are connected by semantic relationship edges, forming a multi-layered semantic association network. The standard topic library can be persistently stored in a graph database (such as Neo4j or GraphDB) to support subsequent semantic retrieval, knowledge evolution, and model reasoning. By performing domain feature recognition and term extraction on the semantic dataset, combined with standard corpus algorithms to calculate semantic similarity between terms, and performing standard semantic reconstruction based on similarity thresholds, data from different databases and formats achieves unified semantic mapping, significantly reducing semantic bias caused by terminology differences and inconsistent expressions. The construction of the standard topic library makes semantic relationships more explicit and hierarchical, facilitating the rapid location of highly relevant topics in subsequent reasoning and question-answering tasks, significantly improving the accuracy and response efficiency of semantic retrieval, and enhancing the accuracy of boundary conditions for subsequent retrieval.

[0078] Furthermore, in this embodiment, based on the topic, the semantic similarity between terms in the relevant term set is calculated using a preset standard domain corpus algorithm, and semantically similar terms are reconstructed using standard semantics based on a similarity threshold to generate a standardized term set, including:

[0079] Based on the stated topic, a corresponding pre-defined standard domain corpus algorithm is determined. This pre-defined standard domain corpus algorithm is constructed based on a topic-specific corpus. The standard domain corpus algorithm is a set of semantic analysis and similarity calculation rules established for a specific topic, and its core includes three parts: domain corpus construction, vectorized model training, and similarity calculation model definition. The domain corpus can consist of public datasets, domain knowledge bases, and manually annotated text, preferably including relevant professional terms, technical documents, standards and specifications, and patent texts. During the model construction stage, word embedding-based natural language processing algorithms, such as Word2Vec, FastText, BERT, or SimCSE, can be used to pre-train the corpus to obtain a vectorized model capable of expressing semantic similarity, providing a foundation for subsequent semantic similarity calculations.

[0080] Each term in the relevant term set is vectorized to obtain a semantic embedding vector. Each term is then input into the pre-trained vectorization model to obtain its corresponding semantic embedding vector. The semantic embedding vector is a multi-dimensional floating-point vector used to represent the term's position in the semantic space. Furthermore, each vector has a dimension between 100 and 1024, which can be set according to the corpus size and topic complexity. Through word vectorization, the semantic differences between terms can be quantified using distance in the vector space.

[0081] The cosine similarity calculation model is used to calculate the semantic similarity score between any two terms;

[0082] When the semantic similarity score is greater than or equal to a set similarity threshold, the corresponding terms are grouped into the same semantic cluster. The similarity threshold is determined based on the topic; the similarity threshold can be adaptively determined according to topic features, and can be in the range of 0.75 to 0.9. For terms with semantic similarity scores below the threshold, they are retained as independent term nodes to prevent excessive semantic merging. Clustering can be performed using density-based DBSCAN algorithms, hierarchical clustering algorithms, or K-means algorithms to ensure stable term clustering results and clear semantic boundaries.

[0083] Identify the main term within each semantic cluster. The identification of the main term is based on the frequency of occurrence of each term within the semantic cluster in a topic-based corpus. Count the number of occurrences of each term in the topic corpus. When the frequency of occurrence of term i is greater than the average frequency of occurrence of terms within the cluster, it is preferentially marked as a candidate main term. If there are multiple candidate terms, the final main term is further determined based on its centrality index in the semantic space (such as average similarity or vector center deviation).

[0084] The terminology is used to replace synonyms or near-synonyms within a cluster, forming a standardized terminology set. All terms belonging to the same cluster are uniformly replaced with their corresponding terminology. During the replacement process, the mapping relationship between the original terms and the terminology is recorded, generating a standardized terminology mapping table for subsequent semantic consistency maintenance and automatic knowledge updates.

[0085] The initial search module is used to search the standard topic library based on an external task request, and generate the first search result of the task request, including:

[0086] The received external task requests are semantically parsed to extract task intent vectors and determine the corresponding standard topic library. The task intent vector is a multi-dimensional semantic embedding vector used to quantitatively represent the topic direction and semantic requirements of the external task. The task request is mapped into a semantic vector representation based on a pre-trained language model (such as BERT, ERNIE, RoBERTa or SimCSE).

[0087] Based on the task intent vector, the corresponding standard topic library is searched to obtain an initial candidate set; after the target topic library is determined, the topic matching unit performs semantic search on the corresponding standard topic library based on the task intent vector to obtain an initial candidate set.

[0088] Based on the initial candidate set, structured response content is generated, which becomes the first search result. The structured response content includes information such as topic name, key terms, semantic explanations, related topics, and hierarchical relationships.

[0089] Furthermore, in this embodiment, the step of semantically parsing the received external task request, extracting the task intent vector, and determining the corresponding standard topic library includes:

[0090] The text preprocessing for external task requests includes word segmentation, stop word removal, and part-of-speech tagging. Word segmentation divides continuous natural language text into independent word sequences using statistical models or neural network-based algorithms to ensure accurate recognition of complex technical terms. Stop word removal removes functional words with low semantic contribution from the segmentation results to highlight key semantic information. The stop word list can be customized and expanded based on domain characteristics to avoid erroneous deletion of domain terms. Part-of-speech tagging uses a part-of-speech tagging model (e.g., HMM or BERT-CRF joint model) to add part-of-speech information to each segmentation result, thereby distinguishing semantic functions such as nouns, verbs, and adjectives, providing grammatical support for subsequent semantic component recognition.

[0091] Extract the contextual semantic features of the preprocessed text to identify semantic components and key entities in the task request; use a pre-trained language model (such as BERT, RoBERTa, ERNIE, or ELECTRA) to bidirectionally encode the input text, generating context-related word vector representations so that the semantic information of each word simultaneously includes contextual dependencies; determine the subject-verb-object structure, modification relations, and semantic roles of each word in the sentence based on a dependency parsing model, and identify semantic components such as "task objective," "object of action," and "conditional constraints"; use the Named Entity Recognition (NER) algorithm to identify proper nouns, domain terms, or structured entities (such as earthquake event names, equipment types, geographical locations, time parameters, etc.) contained in the task request, and perform semantic classification and standardization on the recognition results.

[0092] The semantic components and key entities are vectorized to generate semantic vectors, which serve as task intent vectors. Multiple semantic vectors are weighted and aggregated, with weights calculated based on part-of-speech importance, entity category, and attention score in the context. An overall semantic representation vector, denoted as the task intent vector, is generated through vector averaging or weighted pooling to quantify the semantic requirements and topic direction of external task requests.

[0093] Based on the task intent vector, the semantic vectors of each topic in the standard topic library are similar to calculate the similarity and the corresponding standard topic library is determined. When multiple topics meet the similarity condition, the system further calculates the final confidence score based on the context matching score, entity co-occurrence rate and domain weight, and selects the topic library with the highest confidence score as the target standard topic library.

[0094] Furthermore, in this embodiment, the generation of structured answer content based on the initial candidate set, wherein the structured answer content is the first search result, includes:

[0095] Semantic encoding is performed on the initial candidate set to generate candidate semantic vectors; each topic node and its subordinate terms in the candidate set are input into a pre-trained semantic embedding model (such as BERT, Sentence-BERT, RoBERTa or SimCSE) for vectorization processing to obtain a high-dimensional semantic representation of each node and term.

[0096] The candidate semantic vectors are fused with the task intent vectors to obtain a fused semantic representation. During the fusion process, an attention weighting mechanism can be introduced to assign different weights to different dimensions of the candidate semantic vectors according to the task intent vector, thereby highlighting knowledge nodes related to the core intent of the task. The semantic representation generated after fusion is a high-dimensional vector that can simultaneously express the semantic requirements of the task request and the semantic content of the candidate set knowledge, thereby achieving semantic alignment between the task intent and the knowledge base information.

[0097] The fused semantic representation is used to generate structured response content, which serves as the first search result. The fused semantic vector is decoded or semantically aggregated, mapping it to an understandable structured knowledge representation, including topic names, key terms, semantic relationships, attribute descriptions, and hierarchical structures. Template filling methods, graph structure generation methods, or neural network-based generative models (such as Seq2Seq, Transformer decoders, or Graph Neural Networks (GNNs)) can be used to transform the fused semantic representation into structured response content. This embodiment enables efficient semantic fusion of the initial candidate set and the task intent vector, generating structured response content and realizing the processing from task request to standardized search results.

[0098] The RAG enhancement and model repository module is used to generate a second search result by reasoning based on a preset multimodal model, external task requests, and the first search result. The preset multimodal model includes at least one general large model and several topic models.

[0099] In this embodiment, the model repository module performs reasoning based on a preset multimodal model, external task requests, and the first search result to obtain a second search result, including:

[0100] Based on the external task request, a corresponding topic model is determined. The topic model contains professional terms, knowledge structures, rules and logical relationships related to a specific field, which can enable in-depth understanding and accurate reasoning of the professional content in the task request. If the task request involves knowledge from multiple fields, the system can select multiple relevant topic models and perform joint processing according to topic weights.

[0101] Based on the external task request, joint semantic reasoning is performed on the external task request using the general large model and the corresponding topic model to obtain the reasoning result. The general large model is responsible for handling the natural language understanding and cross-domain semantic logic reasoning of the task request, generating preliminary reasoning results. The topic model provides domain knowledge constraints, injecting relevant terms, concepts, and logical relationships from the standard topic library into the reasoning process to achieve professional knowledge enhancement. During the joint semantic reasoning process, attention mechanisms, multimodal information fusion, or graph neural network technology can be used to enable the task intent vector, the first retrieval result vector, and the topic knowledge representation to interact in a high-dimensional semantic space, generating a fused semantic reasoning vector. During the reasoning process, semantic completion, logical verification, and hierarchical relationship correction can be performed on key entities to ensure the professionalism and accuracy of the reasoning results.

[0102] The first search result is used as a boundary condition to verify the reasoning result, resulting in a second search result, including:

[0103] A constraint set is constructed based on the first search result; the constraint set includes entity existence constraint (the reasoning result must contain the core entity in the first search result), relation consistency constraint (the entity relations in the reasoning result must be logically consistent with the first search result), semantic range constraint (the attribute values ​​in the reasoning result must be within a reasonable range), etc.

[0104] The reasoning results are verified using the constraint set; the verification unit performs constraint checks on the reasoning results to ensure that they conform to the semantic boundaries provided by the first retrieval results; during the verification process, semantic similarity calculation can be combined to perform fault tolerance processing on items with slight differences in expression but consistent semantics, such as synonyms or semantically equivalent substitutions.

[0105] When the reasoning result satisfies the constraint set, the reasoning result is confirmed to be valid and output as the second retrieval result; when the reasoning result fully satisfies the constraint set conditions, the verification unit confirms its validity and directly outputs it as the second retrieval result.

[0106] When the inference result deviates from the constraint set, the inference result is adaptively corrected based on the degree of deviation, and a second retrieval result is output after the update. Adaptive correction includes: supplementing missing core entities or relationships to ensure the inference result covers the knowledge boundaries provided by the first retrieval result; adjusting attribute values ​​or logical relationships that deviate from the constraints, for example, by minimizing semantic bias or applying topic model knowledge rules; optimizing and unifying structurally inconsistent or repetitive information to ensure the structured output result is complete, standardized, and interpretable; after correction, an updated second retrieval result is generated, ensuring that it satisfies both the constraints provided by the first retrieval result and includes new knowledge generated through RAG enhancement and model repository module inference.

[0107] The system rigorously verifies the reasoning results through a constraint set to ensure that the second retrieval result is semantically and logically consistent with the first retrieval result. The first retrieval result is used as a boundary condition so that the reasoning results generated by RAG enhancement can introduce new knowledge without deviating from the existing knowledge system. When there is a deviation in the reasoning result, the system can automatically correct it and generate an updated result to improve the accuracy and coverage of the reasoning result.

[0108] This invention improves the accuracy of cross-source data fusion by constructing a standard topic library and a multi-source semantic dataset, achieving unified data management and standardized processing. The system uses the first retrieval result generated by the initial detection module as boundary conditions to verify the inference results of the RAG enhancement and model repository modules, effectively preventing model inference bias and the spread of erroneous knowledge, thereby improving the accuracy and reliability of the generated results. Through joint inference of multimodal models and external task requests, the system can comprehensively process multi-source information such as text, structured data, and images, achieving high-precision responses to complex task requests.

[0109] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

[0110] The present invention has been described above with reference to embodiments thereof. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

[0111] Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions, and modifications can be made to the embodiments of the present invention without departing from the spirit and scope of the invention.

[0112] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A standardized data governance system based on RAG and a large model repository, characterized in that, include: The module includes a data fusion module, a standard knowledge evolution module, a preliminary inspection module, and a RAG enhancement and model repository module. The data fusion module is used to extract structured and unstructured data from multiple distributed databases and obtain a unified semantic dataset from multiple sources based on a cross-source semantic alignment algorithm. The standard knowledge evolution module is used to reconstruct the semantic dataset using a preset corpus algorithm to form a standard topic library. The initial detection module is used to search the standard topic library based on an external task request and generate the first search result of the task request; The RAG enhancement and model repository module is used to perform reasoning and generation based on a preset multimodal model, external task requests and the first retrieval result to obtain a second retrieval result. The preset multimodal model includes at least one general large model and several topic models. The second search result is generated by reasoning based on a preset multimodal model, external task requests, and the first search result, including: Based on the external task request, determine the corresponding topic model; Based on the external task request, joint semantic reasoning is performed on the external task request using the general large model and the corresponding topic model to obtain the reasoning result; The first search result is used as a boundary condition to verify the reasoning result, and a second search result is obtained.

2. The standardized data governance system based on RAG and large model warehouse according to claim 1, characterized in that, The data fusion module extracts structured and unstructured data from multiple distributed databases and, based on a cross-source semantic alignment algorithm, obtains a unified semantic dataset from multiple sources, including: Configure connections to multiple distributed databases and establish data access channels; Perform field mapping and primary key identification on the structured data to generate a preliminary structured index table; Feature extraction is performed on the unstructured data, and semantic vectors are extracted using a text embedding model; Based on the cross-source semantic alignment algorithm, the structured index table and the semantic vector are semantically aggregated and matched to obtain the cross-source entity alignment result; Based on the cross-source entity alignment results, a multi-source semantic dataset in a unified semantic space is generated.

3. The standardized data governance system based on RAG and large model repository according to claim 1, characterized in that, The standard knowledge evolution module is used to reconstruct the semantic dataset using a preset corpus algorithm, forming a standard topic library, including: Domain feature recognition and term extraction are performed on the semantic dataset to obtain topics and related term sets; Based on the topic, the semantic similarity between terms in the relevant term set is calculated using a preset standard domain corpus algorithm, and the semantically similar terms are reconstructed according to the similarity threshold to generate a standardized term set. Based on the semantic hierarchy, logical dependencies, and association strength among the terms in the standardized terminology set, semantic relationship edges are constructed; Based on the standardized terminology set and semantic relationship edges, a standard topic library is generated.

4. The standardized data governance system based on RAG and large model warehouse according to claim 3, characterized in that, Based on the stated topic, the semantic similarity between terms in the relevant term set is calculated using a preset standard domain corpus algorithm. Then, based on a similarity threshold, semantically similar terms are reconstructed using standard semantics to generate a standardized term set, including: Based on the topic, a corresponding preset standard domain corpus algorithm is determined, and the preset standard domain corpus algorithm is constructed based on the topic corpus; Each term in the relevant term set is vectorized to obtain a semantic embedding vector for the term; The cosine similarity calculation model is used to calculate the semantic similarity score between any two terms; When the semantic similarity score is greater than or equal to the set similarity threshold, the corresponding terms are divided into the same semantic cluster. The similarity threshold is determined based on the topic. Identify the main terms within each semantic cluster, the identification of which is based on the frequency of occurrence of each term within the semantic cluster in a topic-based corpus; The main term is used to replace synonyms or near-synonyms within the cluster to form a standardized term set.

5. The standardized data governance system based on RAG and large model warehouse according to claim 1, characterized in that, The initial search module is used to search the standard topic library based on an external task request, and generate the first search result of the task request, including: Semantic parsing is performed on received external task requests to extract task intent vectors and determine the corresponding standard topic library; Based on the task intent vector, the corresponding standard topic library is searched to obtain an initial candidate set; Based on the initial candidate set, a structured answer is generated, which is the first search result.

6. The standardized data governance system based on RAG and large model warehouse according to claim 5, characterized in that, The step of semantically parsing the received external task request, extracting the task intent vector, and determining the corresponding standard topic library includes: External task requests are preprocessed with text, including word segmentation, stop word removal, and part-of-speech tagging. Extract the contextual semantic features of the preprocessed text to identify the semantic components and key entities in the task request; The semantic components and key entities are vectorized to generate semantic vectors, which serve as task intent vectors. Based on the task intent vector, the similarity of the semantic vectors of each topic in the standard topic library is calculated to determine the corresponding standard topic library.

7. The standardized data governance system based on RAG and large model warehouse according to claim 6, characterized in that, The process involves generating structured response content based on the initial candidate set. This structured response content constitutes the first search result and includes: The initial candidate set is semantically encoded to generate candidate semantic vectors; The candidate semantic vectors are fused with the task intent vectors to obtain the fused semantic representation; Content generation is performed on the fused semantic representation to generate structured answer content, which is the first search result.

8. The standardized data governance system based on RAG and large model warehouse according to claim 1, characterized in that, The first search result is used as a boundary condition to verify the reasoning result, resulting in a second search result, including: Construct a constraint set based on the first search results; The reasoning results are verified using the constraint set. When the reasoning result satisfies the constraint set, the reasoning result is confirmed to be valid and output as the second search result; When the reasoning result deviates from the constraint set, the reasoning result is adaptively corrected based on the degree of deviation, and the second search result is output after the update.

Images