Master data automatic governance method and system based on large model and Dores

Abstract

The invention provides a main data automatic governance method and system based on a large model and Dores, and belongs to the field of data governance. The method comprises the following steps: constructing an RAG main data management knowledge base stored in Dores, and exploring multi-source heterogeneous data by utilizing federated query and MCP service of the Dores; driving an end-to-end process from main data automatic identification, asset checking, source layer modeling and data access, and main data standard model definition and gold attribute set generation by using a large language model; deep cleaning is carried out on the data by adopting a rule and AI cooperation mode, and heterogeneous data mapping and ETL task generation and arrangement are completed; finally, in combination with a graph database and an authority system, intelligent data security sharing based on dynamic desensitization is realized. According to the method, automation, intelligence and adaptive optimization of the whole process of main data management are realized, and the problems of dependence on manpower, low efficiency, incomplete coverage and insufficient safety compliance of a traditional scheme are solved.

Description

Technical Field

[0001] This invention belongs to the field of data governance technology, and in particular relates to an automated master data governance method and system based on large models and Doris implementation. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the deep integration and rapid popularization of big data, cloud computing, and artificial intelligence technologies, enterprise data is growing exponentially. Data has become a key strategic asset supporting business decision optimization, operational model innovation, and the construction of core competitiveness. However, the enterprise data ecosystem exhibits complex characteristics of being multi-sourced, heterogeneous, and dynamic, posing unprecedented challenges to data governance. Enterprises not only need to meet increasingly stringent data security and compliance regulations (such as GDPR and the Personal Data Protection Act), but also need to address the problem of data silos across departments and systems, and ensure the high quality and reliability of data assets to support accurate business insights.

[0004] Existing master data governance solutions are mainly divided into two categories: traditional rule engine solutions and intelligent governance platforms. However, both have significant shortcomings and are difficult to meet the large-scale and intelligent governance needs of enterprises. Traditional rule engine solutions rely on data governance experts to predefine and configure a large number of static rules to perform master data cleaning, standardization, merging, and distribution. However, this model's rule configuration is highly dependent on human experience, making the process cumbersome, time-consuming, and costly. Secondly, the rule base is rigid and cannot adapt to rapid changes in business logic. Once the business scenario or data pattern changes, a large amount of manpower is required to re-examine and configure the rules, resulting in low governance efficiency. Finally, traditional rules are difficult to effectively cover unstructured or semi-structured data, as well as complex and implicit data relationships, leading to blind spots in governance effectiveness.

[0005] While intelligent governance platforms can achieve some automated functions by introducing machine learning and AI technologies, their machine learning models are mostly black boxes with poor interpretability. Governance personnel find it difficult to understand the specific basis for the model's judgment, which is a major flaw in highly regulated fields such as finance and healthcare. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides an automated master data governance method and system based on a large model and Doris. By constructing a RAG knowledge base, utilizing Doris federated queries and MCP services, and calling a large model, it automates the entire process from data discovery, standard definition, model design, ETL task generation to secure sharing. This solves the problems of traditional governance solutions, such as reliance on manual labor, poor adaptability, and inadequate security mechanisms, achieving intelligent, adaptive, and scalable implementation of master data governance.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides an automated master data governance method based on a large model and Doris implementation; Automated master data governance methods based on large models and Doris include: Based on the master data governance scenario, multiple large models were evaluated and selected, and the data governance standard documents were semantically sliced ​​and vectorized to build the RAG master data governance knowledge base; The metadata of heterogeneous data sources is obtained by using Doris's federated query and MCP service. Combined with the query results of the RAG master data governance knowledge base, the large model is called to automatically identify master data entity information. Based on the master data entity information, the large model is called to generate source layer table creation statements and the tables are created through the Doris MCP service; based on multi-source metadata and RAG knowledge base, the large model is called to perform semantic aggregation and normalization to generate standardized master data model definitions and table creation statements. The system performs quality checks and data cleaning on the incoming raw data, completes field mapping between the source data and the master data model based on semantic understanding, and generates an ETL task workflow. Sensitive information in the ETL task workflow is encrypted by calling a custom Doris encryption UDF function and passing in an encryption authentication code. For data that has completed ETL process governance, based on user permissions and data classification information, a large language model is invoked to generate secure data query statements, enabling permission-based data sharing and dynamic anonymization.

[0008] As a further technical solution, the RAG master data governance knowledge base includes business dimension, data characteristic dimension, and management dimension.

[0009] As a further technical solution, Doris's federated queries and MCP service are used to obtain metadata from heterogeneous data sources. Combined with the results of querying the RAG master data governance knowledge base, the large model is invoked to automatically identify master data entity information, including: Obtain metadata information for all data sources, including databases, tables, rows, columns, and the Comment database, by calling the MCP service provided by Doris; Based on the acquired metadata information, the RAG master data governance knowledge base is constructed through vector retrieval to obtain professional knowledge related to master data identification and form prompt words for obtaining public entity information from different data sources; Call the inference model to retrieve the master data entities and corresponding table information from all data sources and output them in a formatted manner; The Doris MCP service is called sequentially for different data sources to query the data in the master data table that has been obtained, and to perform statistics on the core fields of the relevant data. At the same time, a list of relevant dictionary items is sorted out according to the characteristics of the data, and the data is scored according to the established rules. A formatted asset inventory report is then output, reviewed, and modified as the basis for subsequent data access.

[0010] As a further technical solution, based on the master data entity information, the large model is invoked to generate the source layer table creation statement and the table is created through the Doris MCP service, including: Extract the table information in the source system corresponding to the master data entity information after review and modification, call the large model to generate Doris table creation statements for the source layer, and create the corresponding tables in the data warehouse by calling Doris's MCP service; By traversing the asset inventory report information in the source system and reviewing the modified master data entity information, an access task is automatically created; the source system database connection information, source table metadata, data volume, commonly used incremental fields, call cycle, and existing Datax access template information are used as prompt words to call the large model to generate Datax data access tasks, and persist them to the data service platform; Execute the created access task. If an error occurs during execution, the error information is fed back to the large model for access task correction, and the corrected task is updated in the data service platform. Call Doris's MCP service to compare data accuracy and verify the accuracy of the automatically created task.

[0011] As a further technical solution, based on multi-source metadata and the RAG knowledge base, a large model is invoked for semantic aggregation and normalization to generate standardized master data model definitions and table creation statements, including: Based on the master data inventory report and corresponding metadata information extracted from each data source that has passed manual review, as well as the metadata information already available in the data platform, prompt words are used to call the large model, and the large model is used to identify the core business entities. The system queries the RAG Master Data Governance Knowledge Base to obtain industry knowledge related to master data. It combines this knowledge with the returned list of entity attributes and the existing metadata list obtained through Doris's MCP to form prompt words. The system then calls the large model again to select the most core and stable attributes as the golden attribute set. For each golden attribute, the system recommends the most suitable data type length and constraints, and designs tables and relationships between tables according to database normalization. Combining the existing metadata database information, the system outputs a standardized master data model definition document and the corresponding database table creation statements. The newly added metadata information and table creation statements will be pushed for review, and Doris's MCP service will be called to create master data entity related tables in the standard layer of the data warehouse.

[0012] As a further technical solution, the raw data being accessed is subjected to quality inspection and data cleaning, including: The system calls upon a large model to generate data cleaning rules, and uses a combination of rule engine and large model semantic analysis to collaboratively clean up issues such as missing data, duplication, inconsistent formats, and outliers. Remove data with missing values ​​in core fields and fill in data with missing values ​​in non-core fields; For duplicate data, first use the ID or name unique field to accurately remove duplicates, then use a large model to match and perform deep semantic analysis through semantic similarity, associate similar records and tagged them with similarity; For issues with inconsistent formats, we first use regular expressions to standardize fields with clear standards and specifications, and then use a large model to intelligently standardize fields with complex specification mappings based on data tags in the knowledge base. For outlier issues, corrections are made based on outlier detection rules. Then, all field values ​​of the records are linked through a large model, and the data standard knowledge base is used as the evaluation standard to analyze whether the records are outliers. Outlier records are marked with an outlier tag and their confidence level is indicated.

[0013] As a further technical solution, semantic understanding is used to complete the field mapping between source data and master data model, generating an ETL task workflow, including: The system invokes a large model to perform field mapping between source data and the master data model based on semantic understanding; it also automatically identifies task dependencies and generates and executes an ETL task workflow that includes data cleaning, mapping relationship construction, and foreign key population. After data cleaning, the different structure tables from different source systems are mapped to the master data model to obtain entity mapping master data entities. For entity mapping to master data entities, create the original master data relationship mapping table and generate the mapping governance task SQL; By associating the relationship table generated by the standard names of the dependent entities, the corresponding standard foreign key information is obtained. With the help of the large model, the dependencies between tasks are automatically identified. The data cleaning, mapping, and mapping table generation processes are combined with workflow knowledge to automatically generate the ETL task orchestration process.

[0014] The second aspect of this invention provides an automated master data governance system based on a large model and Doris implementation.

[0015] An automated master data governance system based on large models and Doris includes: The knowledge base construction module is configured to: evaluate and select multiple large models based on the master data governance scenario, and semantically slice and vectorize the data governance standard documents to build the RAG master data governance knowledge base; The master data entity information identification module is configured to: use Doris's federated query and MCP service to obtain metadata from heterogeneous data sources, combine the results of querying the RAG master data governance knowledge base, and call the large model to automatically identify master data entity information; The master data model definition and table creation statement generation module is configured to: generate source layer table creation statements based on master data entity information by calling the large model and creating tables through the Doris MCP service; and generate standardized master data model definition and table creation statements based on multi-source metadata and RAG knowledge base by calling the large model for semantic aggregation and normalization. Model standardization, or task execution module, is configured to: perform quality checks and data cleaning on the incoming raw data; complete the field mapping between the source data and the master data model based on semantic understanding; and generate an ETL task workflow. It also encrypts sensitive information in the ETL task workflow by calling a custom Doris encryption UDF function and passing in an encryption authentication code. The secure data sharing service module is configured to: generate secure data query statements by calling a large language model based on user permissions and data classification information for data that has completed ETL process governance, thereby achieving permission-based data sharing and dynamic desensitization.

[0016] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the master data automated governance method based on a large model and Doris as described in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the master data automated governance method based on a large model and Doris as described in the first aspect of the present invention.

[0018] The above one or more technical solutions have the following beneficial effects: This invention introduces Retrieval-Enhanced Generation (RAG) technology, using a professional knowledge base for data governance as the "external brain" of a large model, effectively compensating for the knowledge gaps of general-purpose large models in vertical fields. Based on this, the system can intelligently deduce and generate initial drafts of data governance standards that conform to industry norms. This not only frees experts from tedious rule writing but also forms a highly efficient new collaborative model of "intelligent initial draft, expert final review," revolutionizing the efficiency and quality of data governance specification.

[0019] By building a highly automated data governance pipeline, the large model is positioned as the "intelligent scheduling center" for the process. From the initial stage, the large model automatically explores the entire data domain through Doris federated queries to identify potential master data tables. Subsequently, based on its built-in governance knowledge base, it automatically completes the logical model design and source-layer physical modeling. Finally, it directly generates executable DataX data synchronization tasks and Airflow workflow scheduling scripts. The entire process compresses the manual sorting, design, and development work that previously required weeks or even months into a few hours of automated completion. This end-to-end automation not only frees up manpower but also ensures the consistency of governance standards and the standardization of processes, enabling the large-scale and agile implementation of master data governance.

[0020] This invention achieves significant innovation in the data sharing process by deeply integrating knowledge graphs and permission systems to create an intelligent data service API. When a user initiates a query, the system first parses the user's intent, locating the main table and all its related tables through the Neo4j knowledge graph, forming a complete data relationship network. Subsequently, the system queries the user's data permissions and data hierarchical tags in real time, providing this contextual information to the large model. Based on this, the large model generates SQL query statements that meet user needs while strictly adhering to permission boundaries, and can intelligently determine whether to call Doris's UDF to decrypt sensitive information. This mechanism achieves automated and refined data sharing with "data visible within permissions and sensitive data anonymized," greatly improving data utilization efficiency and security.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of the method in the first embodiment.

[0024] Figure 2 This is a system structure diagram of the second embodiment. Detailed Implementation

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] Example 1 This embodiment discloses an automated master data governance method based on large models and Doris; like Figure 1 As shown, the automated master data governance method based on the large model and Doris includes: Step S1: Evaluate and select multiple large models based on the master data governance scenario, and semantically slice and vectorize the data governance standard documents to build the RAG master data governance knowledge base stored in the Doris database.

[0029] In this embodiment, a multi-dimensional evaluation is conducted on open-source large models such as the DeepSeek series, Qwen series (Alibaba), Kimi K2 series, and ChatGLM series based on the master data governance scenario to select the optimal large model in this field. Specifically, the accuracy of the model's understanding of DAMA standards, data lineage, and quality rules is evaluated through manual evaluation; the model performance is quantified through automated benchmark testing (such as data cleaning F1 score and classification accuracy); and finally, the model with the best overall score is selected as the core governance engine.

[0030] Furthermore, documents such as the "Big Data Standardization White Paper," "DAMA Data Management Body of Knowledge Guide," "Master Data-Driven Data Governance," and "Master Data Management: The Foundation for Enterprise Datafication," as well as actual enterprise internal data governance standard documents and industry knowledge documents, are semantically sliced ​​and embedded before being stored in Doris's vector database as an external knowledge base for the large model. Its content includes: business dimensions (customer, product, employee); data characteristic dimensions (data structure, sensitivity, lifecycle); and management dimensions (scope of sharing, compliance requirements). Combining the advantages of vector retrieval and keyword retrieval, a re-ranking model is used to normalize the results, improving recall quality.

[0031] Step S2: Use Doris's federated query and MCP service to obtain metadata from heterogeneous data sources, and combine the results of querying the RAG master data governance knowledge base to call the large model to automatically identify master data entity information.

[0032] Organize the data source information that needs to be connected, and leverage Doris's Multi-Catalog multi-data source federated query capabilities to perform unified data access configuration for the connected data sources (such as MySQL, PostgreSQL, Hive, Iceberg, ES, etc.).

[0033] Furthermore, a master data entity identification workflow is created. Specifically, firstly, the database metadata information, including databases, tables, rows, columns, and comments, from all data sources is obtained by calling the MCP service provided by Doris. Secondly, based on the obtained metadata information, and using the master data governance knowledge base built in the vector retrieval step, relevant professional knowledge for master data identification is acquired to form prompt words for obtaining common entity information from different data sources. In addition, the master data entities and corresponding table information from all data sources are obtained by calling the inference big model and formatted for output. Finally, the Doris MCP service is called again for different data sources to query the data in the obtained master data tables. The core fields of the relevant data, such as the device model, device number, MAC address, and IP address in the device table, are statistically analyzed based on dimensions such as value rate, data type, whether it is non-nullable, data format accuracy, and the amount of data in the table. At the same time, a list of relevant dictionary items is compiled according to the data characteristics and scored according to established rules. Then, a formatted asset inventory report is output and entered into the database, and simultaneously pushed to the front end for display on the review page for manual review and modification, serving as the basis for subsequent data integration.

[0034] Step S3: Based on the approved master data entity information, call the large model to generate source layer table creation statements and create tables through the Doris MCP service; based on multi-source metadata and RAG knowledge base, call the large model to perform semantic aggregation and normalization to generate standardized master data model definitions and table creation statements.

[0035] Extract the table information from the source system corresponding to the master data entity information after manual review and modification, call the large model to generate the Doris table creation statement for the source layer (the table name is stg_{source system number}_{original table name}), and then call the Doris MCP service to create the corresponding table in the data warehouse.

[0036] In addition, the system iterates through the master data asset report information in the source system and the data warehouse standard master data entity information after manual review and modification, and automatically creates access tasks. It uses information such as the source system database connection information, source table metadata, data volume, commonly used incremental fields, call cycle, and existing Datax access templates as prompts to call the large model to generate Datax data access tasks, and persists them to the data service platform.

[0037] The system automatically executes the access task. If an error occurs, the error information is fed back to the large model for task correction, and the corrected task is updated in the data service platform. It calls Doris's MCP service and leverages its federated query capabilities to retrieve the top 10% of data from both the source and target tables for accuracy comparison, thus verifying the accuracy of the automated task creation process.

[0038] Furthermore, based on the master data inventory reports and corresponding metadata information extracted from various data sources that have passed manual review, as well as the metadata information already available in the data platform, prompt words are used to invoke the large model. The large model analyzes all input tables and identifies core business entities, such as "customer," "product," "supplier," and "employee." For example, it can identify that tables such as crm_customer_info, order_customer, and marketing_user_list are all related to the "customer" entity. For each identified entity (such as "customer"), the large model aggregates all its attributes from all related tables. For example, it interprets fields such as cust_name, user_nm, and client_name as "customer name"; and interprets phone_no and mobile as "contact number," listing the identified core entities and their full attribute list aggregated from each source table and returning it in a formatted format.

[0039] The system queries the knowledge base to obtain industry knowledge related to the master data and combines it with the returned list of entity attributes and existing metadata obtained through Doris's MCP to form prompt words, which are then invoked again in the main model. A "Golden Record" is defined for each master data entity. The most core and stable attributes are selected, such as a customer's "unique identifier, name, ID number, mobile phone number, gender, and date of birth." The system recommends the most suitable data type length and constraints for each Golden Record and designs tables and relationships between tables according to database normalization. Finally, based on existing metadata database information, if semantically similar attribute names exist, the existing metadata names are used; otherwise, custom names are created according to standards. The system outputs DDL table creation statements and a standardized "Master Data Model Definition Document," including an entity relationship diagram (the logical description of the ERD), a list of fields for each table, data types, constraints, and comments.

[0040] The newly added metadata information and table creation statements are pushed to the front end for manual review. After the review is approved, Doris's MCP service is called to create the master data entity related tables in the standard layer of the data warehouse.

[0041] Step S4 involves quality inspection and data cleaning of the incoming raw data, mapping the fields of the source data to the master data model based on semantic understanding, and generating an ETL task workflow. Sensitive information in the ETL task workflow is encrypted by calling a custom Doris encryption UDF function and passing in an encryption authentication code.

[0042] After data access, the raw data undergoes quality analysis. Great Expectations is used to automatically scan the raw data, generating field-level statistical reports (empty rate, unique value distribution). These reports are then translated into understandable prompts for the large-scale model, incorporating relevant data cleaning knowledge, and the model is invoked to generate corresponding cleaning rules. A rule-based, AI-driven, and manual approach is employed to clean different types of problematic data. For issues involving missing core field values: this data is marked as low quality and moved to the dirty data log table; for issues involving missing non-core field values, missing values ​​are filled. For duplicate data issues, precise deduplication is first performed using the unique ID or name field, followed by deep semantic analysis using the large-scale model based on semantic similarity, extracting core feature vectors and calculating the cosine similarity between feature vectors of different records. When the similarity exceeds a preset threshold (e.g., 0.95), it is judged as a suspected same entity. Similar records are associated and tagged with similarity for subsequent manual review. For inconsistent formatting issues, fields with clear standard specifications are first standardized using regular expressions. For fields with complex specification mappings, a large model is used for intelligent standardization based on data markers in the knowledge base. Data standardized by the large model is tagged with AI standardization for subsequent manual review. For outlier issues, corrections are made based on rules, and then the large model is used to associate all field values ​​of the records, using the data standard knowledge base as the evaluation standard to analyze whether the records are abnormal. Abnormal records are tagged with anomalies and their confidence levels are indicated. After data cleaning for the above five types of issues, manual review of the intelligently cleaned content by the large model is required (to avoid the negative impact of "phantom reads" from the large model). The reviewed data is stored in the standard layer of the data warehouse.

[0043] After data cleaning, different structured tables from different source systems are mapped to the master data model. A data intelligent mapping workflow is built to handle this, inputting table structure information from different source systems; the master data model, through semantic understanding of table fields and standard metadata, associates fields with a meaning similarity >90% to fields in the master data model; the mapping relationships of the master data model are output in a structured manner; the formatted results are stored in the database and pushed to the front end for display on the review page.

[0044] For entities that have passed the review and are mapped to master data entities, the original-master data relationship mapping table is created, and the mapping governance task SQL is generated. Field attributes such as original data identifier, standard data ID, standard name, source system name, and source system identifier are used as prompts to generate relevant table creation DDL statements and data governance insertion SQL statements for the large model. The output is formatted and then Doris is invoked to execute the table creation.

[0045] The process involves populating foreign key standard IDs by querying the relational table generated from the dependent entity's standard name to obtain the corresponding standard foreign key information and then populating it. Leveraging a large model, dependencies between tasks are automatically identified. The data cleaning, mapping, and mapping table generation processes, combined with AirFlow workflow knowledge, are automatically generated to create an ETL task orchestration process. An AirFlow execution agent is then invoked to perform pre-execution testing. If errors occur, the model is used to analyze the errors and further correct the orchestration file until the task executes successfully. Finally, the corresponding orchestration task is submitted for manual review.

[0046] Furthermore, the generated ETL task workflow undergoes data tiering and classification, and sensitive fields are encrypted. By calling Doris's MCP service, metadata information from each master data table and the first 100 data entries are obtained. Security standards and industry standards are retrieved from the knowledge base as prompts to invoke the large model. Attributes in each master data entity are categorized (e.g., public, internal, secret, confidential), and sensitive fields requiring encryption (e.g., ID card information, mobile phone number login) are listed. The attribute classification labels are stored in a Doris relationship table and pushed to the front-end for manual review and correction. For fields confirmed to require encryption, a custom Doris encryption UDF function is called, passing in parameters such as the encryption authentication code to encrypt the sensitive information.

[0047] Step S5: For the data that has been processed through the ETL process, based on user permissions and data classification information, call the large language model to generate secure data query statements, thereby achieving permission-based data sharing and dynamic desensitization.

[0048] Establish a shared API-SQL workflow. Based on the relationships between the master data and the metadata information of the master data, publish it to the Neo4j graph database. Based on the query intent entered by the user, call Doris's Mcp to obtain the relevant master table information. Based on the master table, query the Neo4j graph database to obtain all related tables and foreign keys. Query the data hierarchy corresponding to the user's permissions and query the attribute classification hierarchy relationship of the generated table.

[0049] Using the inverted index feature provided by Doris, the master data name is segmented into words. This allows the generated query SQL to perform word-segmented queries on the master name, providing more flexible and accurate information. Based on permissions, it determines whether to call Doris's UDF function for sensitive information decryption. These requirements are used as prompts to survey the large model and generate SQL statements for relevant shared APIs. The generated SQL is then executed using Doris. If errors occur, they are submitted to the large model for correction. Finally, the query results are rendered on the webpage.

[0050] Example 2 This embodiment discloses an automated master data governance system based on large models and Doris; like Figure 2 As shown, the automated master data governance system based on the large model and Doris includes: The knowledge base construction module is configured to: evaluate and select multiple large models based on the master data governance scenario, and semantically slice and vectorize the data governance standard documents to build the RAG master data governance knowledge base; The master data entity information identification module is configured to: use Doris's federated query and MCP service to obtain metadata from heterogeneous data sources, combine the results of querying the RAG master data governance knowledge base, and call the large model to automatically identify master data entity information; The master data model definition and table creation statement generation module is configured to: generate source layer table creation statements based on master data entity information by calling the large model and creating tables through the Doris MCP service; and generate standardized master data model definition and table creation statements based on multi-source metadata and RAG knowledge base by calling the large model for semantic aggregation and normalization. Model standardization, or task execution module, is configured to: perform quality checks and data cleaning on the incoming raw data; complete the field mapping between the source data and the master data model based on semantic understanding; and generate an ETL task workflow. It also encrypts sensitive information in the ETL task workflow by calling a custom Doris encryption UDF function and passing in an encryption authentication code. The secure data sharing service module is configured to: generate secure data query statements by calling a large language model based on user permissions and data classification information for data that has completed ETL process governance, thereby achieving permission-based data sharing and dynamic desensitization.

[0051] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0052] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the master data automated governance method based on a large model and Doris implementation as described in Example 1.

[0053] Example 4 The purpose of this embodiment is to provide an electronic device.

[0054] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the master data automated governance method based on a large model and Doris as described in Embodiment 1.

[0055] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0056] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0057] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A master data automated governance method based on large models and Doris, characterized in that, include: Based on the master data governance scenario, multiple large models were evaluated and selected, and the data governance standard documents were semantically sliced ​​and vectorized to build the RAG master data governance knowledge base; The metadata of heterogeneous data sources is obtained by using Doris's federated query and MCP service. Combined with the query results of the RAG master data governance knowledge base, the large model is called to automatically identify master data entity information. Based on the master data entity information, the large model is called to generate source layer table creation statements and the tables are created through the Doris MCP service; based on multi-source metadata and RAG knowledge base, the large model is called to perform semantic aggregation and normalization to generate standardized master data model definitions and table creation statements. The system performs quality checks and data cleaning on the incoming raw data, completes field mapping between the source data and the master data model based on semantic understanding, and generates an ETL task workflow. Sensitive information in the ETL task workflow is encrypted by calling a custom Doris encryption UDF function and passing in an encryption authentication code. For data that has completed ETL process governance, based on user permissions and data classification information, a large language model is invoked to generate secure data query statements, enabling permission-based data sharing and dynamic anonymization.

2. The automated master data governance method based on large models and Doris as described in claim 1, characterized in that, The RAG master data governance knowledge base includes business dimensions, data characteristic dimensions, and management dimensions.

3. The automated master data governance method based on large models and Doris as described in claim 1, characterized in that, Using Doris's federated queries and MCP service to obtain metadata from heterogeneous data sources, and combining the results of querying the RAG master data governance knowledge base, the large model is invoked to automatically identify master data entity information, including: Obtain metadata information for all data sources, including databases, tables, rows, columns, and the Comment database, by calling the MCP service provided by Doris; Based on the acquired metadata information, the RAG master data governance knowledge base is constructed through vector retrieval to obtain professional knowledge related to master data identification and form prompt words for obtaining public entity information from different data sources; Call the inference model to retrieve the master data entities and corresponding table information from all data sources and output them in a formatted manner; The Doris MCP service is called sequentially for different data sources to query the data in the master data table that has been obtained, and to perform statistics on the core fields of the relevant data. At the same time, a list of relevant dictionary items is sorted out according to the characteristics of the data, and the data is scored according to the established rules. A formatted asset inventory report is then output, reviewed, and modified as the basis for subsequent data access.

4. The automated master data governance method based on large models and Doris as described in claim 1, characterized in that, Based on the master data entity information, the large model is invoked to generate the source layer table creation statement and the table is created through the Doris MCP service, including: Extract the table information in the source system corresponding to the master data entity information after review and modification, call the large model to generate Doris table creation statements for the source layer, and create the corresponding tables in the data warehouse by calling Doris's MCP service; By traversing the asset inventory report information in the source system and reviewing the modified master data entity information, an access task is automatically created; the source system database connection information, source table metadata, data volume, commonly used incremental fields, call cycle, and existing Datax access template information are used as prompt words to call the large model to generate Datax data access tasks, and persist them to the data service platform; Execute the created access task. If an error occurs during execution, the error information is fed back to the large model for access task correction, and the corrected task is updated in the data service platform. Call Doris's MCP service to compare data accuracy and verify the accuracy of the automatically created task.

5. The automated master data governance method based on large models and Doris as described in claim 1, characterized in that, Based on multi-source metadata and the RAG knowledge base, the system calls a large model for semantic aggregation and normalization, generating standardized master data model definitions and table creation statements, including: Based on the master data inventory report and corresponding metadata information extracted from each data source that has passed manual review, as well as the metadata information already available in the data platform, prompt words are used to call the large model, and the large model is used to identify the core business entities. The system queries the RAG Master Data Governance Knowledge Base to obtain industry knowledge related to master data. It combines this knowledge with the returned list of entity attributes and the existing metadata list obtained through Doris's MCP to form prompt words. The system then calls the large model again to select the most core and stable attributes as the golden attribute set. For each golden attribute, the system recommends the most suitable data type length and constraints, and designs tables and relationships between tables according to database normalization. Combining the existing metadata database information, the system outputs a standardized master data model definition document and the corresponding database table creation statements. The newly added metadata information and table creation statements will be pushed for review, and Doris's MCP service will be called to create master data entity related tables in the standard layer of the data warehouse.

6. The automated master data governance method based on large models and Doris as described in claim 1, characterized in that, The raw data received undergoes quality inspection and data cleaning, including: The system calls upon a large model to generate data cleaning rules, and uses a combination of rule engine and large model semantic analysis to collaboratively clean up issues such as missing data, duplication, inconsistent formats, and outliers. Remove data with missing values ​​in core fields and fill in data with missing values ​​in non-core fields; For duplicate data, first use the ID or name unique field to accurately remove duplicates, then use a large model to match and perform deep semantic analysis through semantic similarity, associate similar records and tagged them with similarity; For issues with inconsistent formats, we first use regular expressions to standardize fields with clear standards and specifications, and then use a large model to intelligently standardize fields with complex specification mappings based on data tags in the knowledge base. For outlier issues, corrections are made based on outlier detection rules. Then, all field values ​​of the records are linked through a large model, and the data standard knowledge base is used as the evaluation standard to analyze whether the records are outliers. Outlier records are marked with an outlier tag and their confidence level is indicated.

7. The automated master data governance method based on large models and Doris as described in claim 1, characterized in that, Based on semantic understanding, the field mapping between the source data and the master data model is completed, generating an ETL task workflow, including: The system invokes a large model to perform field mapping between source data and the master data model based on semantic understanding; it also automatically identifies task dependencies and generates and executes an ETL task workflow that includes data cleaning, mapping relationship construction, and foreign key population. After data cleaning, the different structure tables from different source systems are mapped to the master data model to obtain entity mapping master data entities. For entity mapping to master data entities, create the original-master data relationship mapping table and generate the mapping governance task SQL; By associating the relationship table generated by the standard names of the dependent entities, the corresponding standard foreign key information is obtained. With the help of the large model, the dependencies between tasks are automatically identified. The data cleaning, mapping, and mapping table generation processes are combined with workflow knowledge to automatically generate the ETL task orchestration process.

8. A master data automated governance system based on large models and Doris, characterized in that: include: The knowledge base construction module is configured to: evaluate and select multiple large models based on the master data governance scenario, and semantically slice and vectorize the data governance standard documents to build the RAG master data governance knowledge base; The master data entity information identification module is configured to: use Doris's federated query and MCP service to obtain metadata from heterogeneous data sources, combine the results of querying the RAG master data governance knowledge base, and call the large model to automatically identify master data entity information; The master data model definition and table creation statement generation module is configured to: generate source layer table creation statements based on master data entity information by calling the large model and creating tables through the Doris MCP service; and generate standardized master data model definition and table creation statements based on multi-source metadata and RAG knowledge base by calling the large model for semantic aggregation and normalization. Model standardization, or task execution module, is configured to: perform quality checks and data cleaning on the incoming raw data; complete the field mapping between the source data and the master data model based on semantic understanding; and generate an ETL task workflow. It also encrypts sensitive information in the ETL task workflow by calling a custom Doris encryption UDF function and passing in an encryption authentication code. The secure data sharing service module is configured to: generate secure data query statements by calling a large language model based on user permissions and data classification information for data that has completed ETL process governance, thereby achieving permission-based data sharing and dynamic desensitization.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps of the master data automated governance method based on a large model and Doris as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the master data automated governance method based on large models and Doris as described in any one of claims 1-7.

Images