Database metadata information processing method and system based on RAG and large model
By constructing a standard knowledge base based on RAG and large models and combining it with multi-source information processing, the efficiency and accuracy issues in sensitive information identification in databases are solved, achieving efficient and automated sensitive data identification and classification, which is suitable for complex and ever-changing data environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANFU JIANGXI LAB
- Filing Date
- 2026-04-13
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies for identifying sensitive information in databases suffer from low efficiency, high subjectivity, low standardization, and difficulty in handling fields with non-standard naming or lack of comments, resulting in a high rate of missed detection of sensitive data, increased compliance costs, and an inability to adapt to rapidly changing data environments.
A standard knowledge base is constructed using a RAG-based and large-model approach. The large language model is used to parse regulatory documents and business data, automatically extract sensitive tags, combine multi-source information for semantic understanding and recognition, and use the embedding model of the Transformer architecture for vectorized representation to achieve efficient and automated identification and classification of database metadata.
It improves the accuracy and generalization ability of sensitive information identification, reduces missed detections and false judgments, lowers the cost of manual intervention, is suitable for complex and ever-changing data scenarios, and provides cross-industry and multi-system adaptability.
Smart Images
Figure CN122019734A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and system for processing database metadata information based on RAG and large models. Background Technology
[0002] With the widespread application of big data technology and the strengthening of data security regulatory policies, the identification of sensitive information in databases has become a core requirement for data governance in fields such as finance, government affairs, and healthcare. Some regulations explicitly require enterprises to classify and grade their stored data; however, the traditional, manually-led method of defining sensitive labels suffers from low efficiency, strong subjectivity, and low standardization, leading to high rates of missed detection of sensitive data and a surge in compliance costs. Furthermore, massive historical databases commonly suffer from legacy issues such as non-standard table / field names, missing comments, and semantic ambiguity, further complicating the identification of sensitive information.
[0003] Current technologies primarily rely on manual rule-driven, mapping table matching, and shallow model matching solutions to process database metadata. Manual rule-driven methods involve manually interpreting regulatory documents and business materials to define sensitive tags and build a knowledge base. This method relies on expert experience, resulting in high labor costs and a lack of standardized tag definitions, making it prone to subjective bias. Mapping table matching uses predefined dictionaries or regular expressions to match keywords in database names, table names, and field names. However, it cannot handle fields without annotations or with unusual names (such as pinyin abbreviations or fields containing mixed Chinese and English characters), and requires frequent updates and maintenance of the rule base, making it unsuitable for rapidly changing data environments. Shallow model matching uses keyword indexing, regular expressions, or traditional classification models for sensitive data identification. However, it is limited by the single nature of features, making it difficult to capture complex semantic relationships (such as sensitive scenarios involving cross-table field value combinations), and it cannot dynamically infer sensitive attributes based on context, leading to insufficient accuracy in sensitive data identification.
[0004] Therefore, researching and designing a database metadata information processing method and system based on RAG and large models that can overcome the above-mentioned shortcomings is an urgent problem that we need to solve. Summary of the Invention
[0005] To address the shortcomings of existing technologies, the present invention aims to provide a database metadata information processing method and system based on RAG and large models. This method utilizes large models to perform semantic parsing of regulatory documents and business data, automatically extracting and constructing a sensitive label knowledge base. It leverages large models to perform semantic understanding of field names and contextual information, enabling automatic sensitivity inference for fields without annotations. Furthermore, by introducing RAG technology and combining multi-source information such as field values, structure, and historical data for semantic completion and enhanced recognition, the method improves recognition accuracy, generalization ability, and interpretability. It is suitable for complex and ever-changing data scenarios and provides a feasible path for the application of large language models in the field of sensitive data recognition.
[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution: Firstly, a method for processing database metadata information based on RAG and large models is provided, including the following steps: A standard knowledge base is constructed based on regulatory requirements and domain business experience information, including a tag library, sensitive identifier definitions, value range descriptions, explanations of common business terms, and interpretability knowledge. Extract the metadata of the data table to be processed. The metadata includes at least the data table name, field comment information, field data type, and the association between the field and the external dictionary / mapping table. After formatting the input of the data table to be processed, the large language model is called to perform semantic restoration or translation operations to achieve understanding and correction of the contextual semantics of the data table; Based on the standard knowledge base, the semantically processed data table is tagged and classified. Combined with retrieval enhancement generation technology, regulatory clause fragments, industry guidance documents and historical compliance records are retrieved, and structured evidence chains are integrated to generate standardized compliance record tags. The data, after being classified and categorized, is processed in a structured manner and represented in a vectorized form to generate structured vector results associated with compliance tags, compliance reasons, and evidence citations. These results are then stored in the database, written back to the data directory, and used for decision processing.
[0007] Furthermore, the construction of the standard knowledge base includes: The system uses a large language model to parse legal provisions, automatically extracts keywords, and maps them to a tag library. The sensitive identifiers and value ranges are formally defined to form an expandable sensitive feature library; Establish relationships between legal provisions, business terms, and sensitive tags using knowledge graph technology.
[0008] Furthermore, the step of invoking the large language model to perform semantic restoration or translation operations includes: Based on the explanations of common business terms and the descriptions of value ranges in the standard knowledge base, the contextual semantic correction is performed on the ambiguous field annotations and non-standardized expressions in the data table; By combining field type and historical data distribution characteristics, semantic disambiguation is performed on fields with abnormal naming.
[0009] Furthermore, the labeling and classification process includes: Multi-source information retrieval is performed using retrieval enhancement generation technology. The multi-source information includes applicable clause fragments in the legal clause knowledge base, business classification and process information in the industry compliance guidance library, field-level tag annotation information of historical compliance records, and structured tables of similar business scenarios. The retrieved information is fused and processed to generate a structured evidence package, which includes contextual compliance evidence of metadata, value range constraint evidence, business similarity matching evidence, and mapping relationship of compliance standard items. The structured evidence package is used to generate labels and determine classification and grading.
[0010] Furthermore, the structured processing and vectorized representation include: Perform formatting transformation, missing value filling, and business logic completion on unstructured fields; An embedding model is used to generate high-dimensional feature vectors by combining compliance tags, field relationships, and evidence citations. Hash indexing technology enables fast retrieval and storage of structured vector results in a database.
[0011] Furthermore, the embedded model is built on the Transformer architecture.
[0012] Furthermore, the method also includes: The structured evidence package is input into the reasoning module of the large language model, which then outputs candidate labels, compliance classification results, reasoning explanations, evidence citation identifiers, and decision confidence levels based on the evidence package. The execution strategy and routing operation are determined by the data processing strategy through automatic threshold determination, collaborative scheduling between the end side and the cloud side, multi-model routing selection, canary release mechanism, A / B test traffic splitting and cost constraint evaluation. When the confidence level of the inference result is in the preset low confidence interval, the low confidence branch processing flow is triggered to perform a re-judgment operation or organize domain experts to collaboratively review the judgment logic.
[0013] Furthermore, the method also includes: Build a continuous learning closed-loop mechanism to feed back misjudgment samples from business decisions into the knowledge base to complete the incremental update of the knowledge base; Generate interpretation and audit output results, including rule ID identifiers, complete evidence chain traceability, clause mapping relationships, liability tracing information, and verifiable judgment path information for data source tracing.
[0014] Furthermore, the method also includes: Model distillation and quantization techniques are used to process the updated knowledge base and model parameters, and the updated content is then distributed to the reasoning and rule engine via over-the-air download.
[0015] Secondly, it provides a database metadata information processing system based on RAG and large models, including: The knowledge base construction module is configured to build a standard knowledge base. The standard knowledge base is built based on regulatory requirements and domain business experience information, and includes a tag library, sensitive identifier definitions, value range descriptions, explanations of common business terms, and interpretability knowledge. The metadata extraction module is configured to extract metadata from the data table to be processed. The metadata includes at least the data table name, field annotation information, field data type, and the association between the field and the external dictionary / mapping table. The large model calling module is configured to perform formatted input processing on the data table to be processed, and then call the large language model to perform semantic restoration or translation operations to realize the understanding and correction of the contextual semantics of the data table; The RAG retrieval module is configured to tag and classify the semantically processed data table based on the standard knowledge base, and combine retrieval enhancement generation technology to retrieve legal clause fragments, industry guidance documents and historical compliance records, and integrate structured evidence chains to generate standardized compliance record tags. The structured processing module is configured to perform structured processing and vectorized representation on the data after hierarchical classification, generate structured vector results with compliance tags, compliance reasons and evidence references, and store the results in the database, write them back to the data directory and decision processing.
[0016] Thirdly, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the database metadata information processing method based on RAG and large models as described in any one of the first aspects.
[0017] Fourthly, a computer-readable medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement the database metadata information processing method based on RAG and large models as described in any one of the first aspects.
[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. The database metadata information processing method based on RAG and a large model provided by this invention utilizes the collaborative work of the large model and RAG. The RAG accurately retrieves relevant information from the database as a reference, assisting the large model in judging sensitive information. This achieves efficient and automated identification of sensitive information in the database, effectively reducing the "illusion" problem of large models and significantly lowering the cost of manual intervention. Furthermore, by combining semantic reasoning and multi-source information retrieval, such as multi-model data joint retrieval, and leveraging the contextual understanding and semantic expansion capabilities of the large model, it effectively identifies fields with non-standard naming, lack of annotations, or complex semantics. Additionally, by implementing joint retrieval of vectors and text, its accuracy exceeds that of traditional single-model retrieval, thereby reducing missed detections and false positives and improving retrieval accuracy. 2. The output results in this invention include label definitions, judgment criteria, and reference information, which facilitates auditing and compliance verification.
[0019] 3. This invention does not require frequent updates to keywords, mapping tables, or training samples, making it suitable for cross-industry and multi-system data environments, thus improving system adaptability and scalability. Attached Figure Description
[0020] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart from Embodiment 1 of the present invention; Figure 2 This is a system block diagram in Embodiment 2 of the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.
[0022] Example 1: A database metadata information processing method based on RAG and large models, such as... Figure 1 As shown, it includes the following steps: S1: Construct a standard knowledge base. The standard knowledge base is built based on regulatory requirements and domain business experience information. It includes a tag library, sensitive identifier definitions, value range descriptions, explanations of common business terms, and interpretability knowledge. S2: Extract the metadata of the data table to be processed. The metadata includes at least the data table name, field comment information, field data type, and the relationship between the field and the external dictionary / mapping table. S3: After formatting the input of the data table to be processed, call the large language model to perform semantic restoration or translation operations to achieve understanding and correction of the contextual semantics of the data table; S4: Based on the standard knowledge base, the semantically processed data table is tagged and classified. Combined with the retrieval enhancement generation technology, regulatory clause fragments, industry guidance documents and historical compliance records are retrieved, and structured evidence chains are integrated to generate standardized compliance record tags. S5: Perform structured processing and vectorization on the classified data to generate structured vector results with compliance tags, compliance reasons and evidence references, and store the results in the database, write them back to the data directory and decision processing.
[0023] In step S1, the construction of the standard knowledge base specifically includes the following steps: using a Large Language Model (LLM) to parse the legal provisions, automatically extract keywords and map them to a tag library; formally defining sensitive identifiers and value ranges to form an expandable sensitive feature library; and establishing the relationship between legal provisions, business terms and sensitive tags through knowledge graph technology.
[0024] In step S2, the system first needs to extract the metadata of the data table to be processed. This metadata is the basic data that comprehensively describes the structure and attributes of the data table. Specifically, the metadata should at least include the name of the data table to uniquely identify it; annotation information for each field to explain the actual business meaning and purpose of the field; the data type of the field, clarifying the data format and constraints stored in the field; and the association between the field and external dictionaries or mapping tables to ensure data consistency and traceability. By extracting this metadata, the necessary context and structural support can be provided for subsequent data processing, analysis, and integration.
[0025] In step S3, calling the large language model to perform semantic restoration or translation operations specifically includes the following steps: based on the explanation of common business terms and the description of value range in the standard knowledge base, perform contextual semantic correction on ambiguous field annotations and non-standardized expressions in the data table; and combine field types and historical data distribution characteristics to perform semantic disambiguation on fields with abnormal naming.
[0026] Based on the established explanations of common business terms and their corresponding standardized value ranges within the enterprise standard knowledge base, the annotations of fields in the data table that are ambiguous in expression or inconsistent in meaning are refined and corrected at the contextual semantic level. At the same time, combined with the data type characteristics of each field and the distribution patterns of historical data, fields with ambiguous names, abnormal naming logic, or names that do not match the content are subjected to in-depth semantic analysis and disambiguation processing to improve the standardization, consistency, and usability of the data.
[0027] In step S4, the tagging and hierarchical classification process specifically includes the following steps: Multi-source information retrieval is performed using Retrieval-Augmented Generation (RAG) technology. This multi-source information includes applicable clause fragments from the regulatory knowledge base, business classification and process information from the industry compliance guidance library, field-level tagging information from historical compliance records, and structured tables of similar business scenarios. The retrieved information is then fused to generate a structured evidence package. This package contains contextual compliance evidence of metadata, value range constraint evidence, business similarity matching evidence, and mapping relationships of compliance standard entries. Tag generation and classification / hierarchical determination are then completed based on the structured evidence package.
[0028] In step S5, the structured processing and vectorized representation specifically include the following steps: formatting and converting unstructured fields, filling in missing values, and completing business logic; using an embedding model based on the Transformer architecture, combined with compliance tags, field relationships, and evidence references to generate high-dimensional feature vectors; and using hash indexing technology to achieve fast retrieval and storage of structured vector results in the database.
[0029] This invention also designs a reasoning and rule engine for reasoning and analysis. Specifically, the structured evidence package is imported as input into the large language model's reasoning module. This module performs logical reasoning and analysis based on the structured information contained in the evidence package, enabling the large language model to automatically output a series of key results based on the input data. These results include the generation of candidate labels, specific compliance classification results, detailed explanations of the judgment reasons, the evidence citation identifiers used, and the confidence level assessment of the judgment results. Subsequently, the system executes strategy and routing operations, achieving automatic judgment by setting thresholds. Combining the collaborative scheduling mechanism between the edge and cloud sides, a multi-model routing selection strategy is adopted, and multiple evaluation dimensions such as canary release mechanism, A / B testing diversion method, and cost constraints are used to comprehensively determine the optimal data processing strategy. When the confidence level of the model's reasoning result is in the preset low confidence interval, the system automatically triggers the low confidence branch processing flow, performs a re-judgment operation, or organizes experts in relevant fields for collaborative review to deeply analyze and verify the rationality and accuracy of the judgment logic.
[0030] Furthermore, this invention innovatively constructs a continuous learning closed-loop mechanism, which encompasses several key stages. The specific implementation steps are as follows: First, a complete continuous learning closed-loop process is established. By systematically identifying and filtering misjudgment samples generated during business judgment, these samples are effectively fed back into the knowledge base, achieving dynamic incremental updates of knowledge content. Second, advanced model distillation and quantization techniques are used to efficiently compress and optimize the updated knowledge base and related model parameters, ensuring a simultaneous improvement in knowledge representation and reasoning efficiency. Subsequently, over-the-air (OTA) download technology is used to securely and reliably distribute the optimized updated content to the reasoning and rule engine deployed on the front end, enabling real-time iteration and deployment of model capabilities. Finally, highly interpretable and auditable output results are generated. The output content includes not only the unique identifier of the rule ID but also the complete evidence chain trajectory, the mapping and correlation between clauses, clear accountability information, and verifiable data source traceability information, thus forming a systematic, transparent, and verifiable judgment path.
[0031] For example, the Housing Provident Fund Center: Identification of sensitive data in employer contribution and withdrawal transactions.
[0032] 1) Business and data background.
[0033] The system comprises two subsystems: employer contribution management and individual account management. After historical migration, it suffers from inconsistent naming and missing field notes.
[0034] Objective: To automatically identify sensitive personal information, financial account information, and sensitive identifiers of entities, and to output an auditable chain of evidence and classification suggestions.
[0035] 2) Typical tables and "problem fields".
[0036] UNIT_INFO (Unit Information Table): unit_id, uscid, pay_bank_acct, contact_tel; EMP_ACCOUNT (Personal Account Table): acc_no, pid_no, mobile, base_salary; CONTRIB_DETAIL (Deposit Details): acc_no, contrib_month, base_amt, company_amt, personal_amt; WITHDRAW_APPLY (withdrawal request): pid_no, bank_card, recv_name, loan_contract_no; Example of missing notes / name anomalies: USCID (Unified Social Credit Code) has no remarks; the old name for pay_bank_acct was collection_account or pay_acc. In the history database, pid_no is written as gmsfhm; acc_no is written as gjj_zh in another database.
[0037] 3) Operations on the MCE (Model Context Engine) side.
[0038] (a) Semantic reduction Based on the context and fields in the same table: uscid is located in the unit table and its length / format matches "18-digit social credit code"; If the field acc_no / gjj_zh is located in the personal account table or contribution details, and has a strong correlation with the month and amount, it is determined to be "housing provident fund personal account"; The pid_no / gmsfhm string contains both the name and mobile phone number, and its format matches that of a second-generation ID card.
[0039] (b) Portrait and Value Features pay_bank_acct / bank_card: The values are all 16–19 digits, which is consistent with the number of digits in the card numbers of major domestic banks, and they pass Luhn verification (bank card fingerprint). According to relevant statistics, the Luhn verification accuracy rate of 16-digit USD personal user bank cards can reach 99.84%. `base_salary / base_amt`: These are continuous values that remain stable throughout the month, consistent with the characteristics of housing provident fund contribution base / amount. According to regulations, the housing provident fund contribution base changes annually in principle, and the monthly contribution amount is calculated based on the employee's average monthly salary from the previous year. This further confirms the characteristic of this value remaining stable throughout the month.
[0040] (c) RAG search and evidence fusion The recall includes items such as "sensitive personal information" (ID card number, mobile phone number, bank card number), "financial account identifiers" (corporate collection accounts), and "unique identifiers of organizations" (unified social credit code); The evidence package includes: {context (table / column position & business semantics), value fingerprint (length / Luhn / regular expression), Top-K similar structures, and standard entry mapping}.
[0041] 4) LLM reasoning (with structured evidence package) → Labels and graded candidates.
[0042] Here are three representative outputs (each with a short link and verifiable): A. Field: pid_no (WITHDRAW_APPLY); Judgment: ID card number → Sensitive personal information (high); Chain of Evidence (Summary): Value fingerprint: 18 bits, valid check bits; Context: Co-occurs with recv_name / bank_card in the withdrawal application process; Similar structures: consistent annotation across historical databases gmsfhm; Entry mapping: KB#PII-IDCARD (Personal Identifier Class).
[0043] Recommended handling: When displaying information externally, retain the first 3 and last 2, and mask the rest; desensitize across different lineages simultaneously.
[0044] B. Field: pay_bank_acct (UNIT_INFO) Judgment: Corporate bank collection account → Financial account identifier (medium-high); Chain of Evidence (Summary): Value fingerprint: 16–19 bits + Luhn passed; Context: The unit information table contains foreign keys / business references related to the deposit deduction relationship; Similar structure: another database, pay_acc / collection_account, is labeled as "unit collection account"; Entry mapping: KB#FIN-ACCT (Financial Accounts).
[0045] Recommended solutions: encrypt storage within the database; only output the last digit; require a whitelist for export.
[0046] C. Field: acc_no (EMP_ACCOUNT / CONTRIB_DETAIL) Judgment: Individual Housing Provident Fund Account → Key Business Identifier (Medium); Chain of Evidence (Summary): Context: Stable co-occurrence with contrib_month / base_amt; Similar structure: The old database gjj_zh is labeled as "housing provident fund account"; Entry mapping: KB#GJJ-ACC (Industry-specific sensitive identifier).
[0047] Recommended action: Display part of the mask publicly; additional approval is required for searches combining it with pid_no.
[0048] 5) Decision-making and implementation.
[0049] Thresholds: Confidence ≥ 0.85, drop directly; 0.7–0.85, trigger duplicate testing; < 0.7, enter manual review pool.
[0050] Output: {Tag, Classification, Evidence Chain ID, Processing Suggestion (Anonymization / Encryption / Admission), Policy Version} Write back to the directory and policy engine.
[0051] Audit: Automatically generates a "Sensitive Element Extraction Application Review Form", including fields → evidence → item mapping → accountability tracking.
[0052] Example 2: A database metadata information processing system based on RAG and large models, such as Figure 2 As shown, the system is used for the database metadata information processing method based on RAG and large models described in Example 1, including a knowledge base construction module, a metadata extraction module, a large model calling module, a RAG retrieval module, and a structured processing module.
[0053] Specifically, the knowledge base construction module is configured to build a standard knowledge base. The standard knowledge base is built based on regulatory requirements and domain business experience information, and includes a tag library, sensitive identifier definitions, value range descriptions, explanations of common business terms, and interpretability knowledge.
[0054] The metadata extraction module is configured to extract metadata from the data table to be processed. The metadata includes at least the data table name, field comment information, field data type, and the association between the field and the external dictionary / mapping table.
[0055] The large model calling module is configured to perform semantic restoration or translation operations on the large language model after formatting the input of the data table to be processed, thereby realizing the understanding and correction of the contextual semantics of the data table.
[0056] The RAG retrieval module is configured to tag and classify semantically processed data tables based on a standard knowledge base, and combine retrieval enhancement generation technology to retrieve legal clause fragments, industry guidance documents, and historical compliance records, and integrate structured evidence chains to generate standardized compliance record tags.
[0057] The structured processing module is configured to perform structured processing and vectorized representation on the data after hierarchical classification, generate structured vector results with compliance tags, compliance reasons and evidence references, and store the results in the database, write them back to the data directory and decision processing.
[0058] Example 3: A computer terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the database metadata information processing method based on RAG and large models as described in Example 1.
[0059] Example 4: A computer-readable medium having a computer program stored thereon, which, when executed by a processor, enables a database metadata information processing method based on RAG and large models as described in the example.
[0060] Working Principle: This system leverages a large model in collaboration with RAG (Reference Aggregator). RAG accurately retrieves relevant information from the database as a reference, assisting the large model in judging sensitive information. This achieves efficient and automated identification of sensitive information in the database, effectively reducing the "illusion" problem of large models and significantly lowering the cost of manual intervention. Furthermore, by combining semantic reasoning and multi-source information retrieval, such as multi-model data joint retrieval, and utilizing the large model's contextual understanding and semantic expansion capabilities, it effectively identifies fields with non-standard naming, lack of annotations, or complex semantics. Additionally, by implementing joint vector and text retrieval, its accuracy surpasses that of traditional single-model retrieval, thereby reducing missed detections and false positives and improving retrieval precision.
[0061] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0062] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0065] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A database metadata information processing method based on RAG and large models, characterized in that, Includes the following steps: A standard knowledge base is constructed based on regulatory requirements and domain business experience information, including a tag library, sensitive identifier definitions, value range descriptions, explanations of common business terms, and interpretability knowledge. Extract the metadata of the data table to be processed. The metadata includes at least the data table name, field comment information, field data type, and the association between the field and the external dictionary / mapping table. After formatting the input of the data table to be processed, the large language model is called to perform semantic restoration or translation operations to achieve understanding and correction of the contextual semantics of the data table; Based on the standard knowledge base, the semantically processed data table is tagged and classified. Combined with retrieval enhancement generation technology, regulatory clause fragments, industry guidance documents and historical compliance records are retrieved, and structured evidence chains are integrated to generate standardized compliance record tags. The data, after being classified and categorized, is processed in a structured manner and represented in a vectorized form to generate structured vector results associated with compliance tags, compliance reasons, and evidence citations. These results are then stored in the database, written back to the data directory, and used for decision processing.
2. The database metadata information processing method based on RAG and large models according to claim 1, characterized in that, The construction of the standard knowledge base includes: The system uses a large language model to parse legal provisions, automatically extracts keywords, and maps them to a tag library. The sensitive identifiers and value ranges are formally defined to form an expandable sensitive feature library; Establish relationships between legal provisions, business terms, and sensitive tags using knowledge graph technology.
3. The database metadata information processing method based on RAG and large models according to claim 1, characterized in that, The process of calling a large language model to perform semantic restoration or translation includes: Based on the explanations of common business terms and the descriptions of value ranges in the standard knowledge base, the contextual semantic correction is performed on ambiguous field annotations and non-standardized expressions in the data table; By combining field type and historical data distribution characteristics, semantic disambiguation is performed on fields with abnormal naming.
4. The database metadata information processing method based on RAG and large models according to claim 1, characterized in that, The labeling and classification process includes: Multi-source information retrieval is performed using retrieval enhancement generation technology. The multi-source information includes applicable clause fragments in the legal clause knowledge base, business classification and process information in the industry compliance guidance library, field-level tag annotation information of historical compliance records, and structured tables of similar business scenarios. The retrieved information is fused and processed to generate a structured evidence package, which includes contextual compliance evidence of metadata, value range constraint evidence, business similarity matching evidence, and mapping relationship of compliance standard items. The structured evidence package is used to generate labels and determine classification and grading.
5. The database metadata information processing method based on RAG and large models according to claim 1, characterized in that, The structured processing and vectorized representation include: Perform formatting transformation, missing value filling, and business logic completion on unstructured fields; An embedding model is used to generate high-dimensional feature vectors by combining compliance tags, field relationships, and evidence citations. Hash indexing technology enables fast retrieval and storage of structured vector results in a database.
6. The database metadata information processing method based on RAG and large models according to claim 5, characterized in that, The embedded model is built on the Transformer architecture.
7. The database metadata information processing method based on RAG and large models according to claim 1, characterized in that, The method also includes: The structured evidence package is input into the reasoning module of the large language model, which then outputs candidate labels, compliance classification results, reasoning explanations, evidence citation identifiers, and decision confidence levels based on the evidence package. The execution strategy and routing operation are determined by the data processing strategy through automatic threshold determination, collaborative scheduling between the end side and the cloud side, multi-model routing selection, canary release mechanism, A / B test traffic splitting and cost constraint evaluation. When the confidence level of the inference result is in the preset low confidence interval, the low confidence branch processing flow is triggered to perform a re-judgment operation or organize domain experts to collaboratively review the judgment logic.
8. The database metadata information processing method based on RAG and large models according to claim 7, characterized in that, The method also includes: Build a continuous learning closed-loop mechanism to feed back misjudgment samples from business decisions to the knowledge base to complete the incremental update of the knowledge base; Generate interpretation and audit output results, including rule ID identifiers, complete evidence chain traceability, clause mapping relationships, liability tracing information, and verifiable judgment path information for data source tracing.
9. The database metadata information processing method based on RAG and large models according to claim 8, characterized in that, The method also includes: Model distillation and quantization techniques are used to process the updated knowledge base and model parameters, and the updated content is then distributed to the reasoning and rule engine via over-the-air download.
10. A database metadata information processing system based on RAG and large models, characterized in that, include: The knowledge base construction module is configured to build a standard knowledge base. The standard knowledge base is built based on regulatory requirements and domain business experience information, and includes a tag library, sensitive identifier definitions, value range descriptions, explanations of common business terms, and interpretability knowledge. The metadata extraction module is configured to extract metadata from the data table to be processed. The metadata includes at least the data table name, field annotation information, field data type, and the association between the field and the external dictionary / mapping table. The large model calling module is configured to perform formatted input processing on the data table to be processed, and then call the large language model to perform semantic restoration or translation operations to realize the understanding and correction of the contextual semantics of the data table; The RAG retrieval module is configured to tag and classify the semantically processed data table based on the standard knowledge base, and combine retrieval enhancement generation technology to retrieve legal clause fragments, industry guidance documents and historical compliance records, and integrate structured evidence chains to generate standardized compliance record tags. The structured processing module is configured to perform structured processing and vectorized representation on the data after hierarchical classification, generate structured vector results with compliance tags, compliance reasons and evidence references, and store the results in the database, write them back to the data directory and decision processing.