Method and system for improving data governance capability based on large model
By parsing specification documents and generating SQL code using a large model, the time-consuming and labor-intensive problem in data governance is solved, enabling efficient and accurate data development and standard definition.
Patent Information
- Application Number
- CN202511614465.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-06
- Publication Date
- 2026-02-24
Smart Images
Figure CN121560967A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of data governance and artificial intelligence large model application technology, specifically a method and system for improving data governance capabilities based on large models. Background Technology
[0002] With the rapid advancement of generative AI technology, various industries are facing significant opportunities to revolutionize user experience and enhance product intelligence. Faced with massive amounts of heterogeneous data and cumbersome processing workflows, many enterprises often need to invest huge amounts of manpower in data governance. Therefore, the industry is actively exploring the application of large-scale AI models in the field of data governance, aiming to significantly improve operational efficiency. Currently, mainstream AI-enabled data governance solutions mainly cover the following aspects:
[0003] (1) Optimize data access: Use intelligent algorithms to automatically identify data sources, data types and formats; use machine learning models to optimize data conversion logic; and improve data storage and retrieval performance through intelligent scheduling mechanisms.
[0004] (2) Optimize metadata management: Metadata includes technical metadata and business metadata. Technical metadata is mainly used to describe the physical structure and storage method of the data. AI can intelligently identify and optimize the database architecture, transformation logic, and related historical records to ensure the standardization and correctness of the data. Business metadata is mainly used to describe the business meaning and purpose of the data. AI can help parse the definition of indicators and calculation rules to improve the interpretability and ease of application of the data.
[0005] (3) Optimize data quality assessment: The intelligent data quality assessment system covers multiple dimensions such as completeness, consistency, accuracy, and timeliness. Utilizing machine learning technologies, including classification algorithms, function approximation, and regression analysis, AI can automatically extract key quality assessment features. Simultaneously, AI can automate the data verification process, quickly identifying data anomalies through real-time monitoring and intelligent analysis.
[0006] While the current three optimization solutions cover some aspects of data governance, they still lack effective countermeasures for the time-consuming standardization and data development stages. In practical data governance scenarios, enterprises need to organize professionals to spend considerable time sorting out business data elements and model architectures and manually entering them into the governance platform. Even if industry standard documents exist, these materials are usually in unstructured formats (such as PDFs or images), resulting in manual data entry. Meanwhile, at the data development level, technical personnel often need to write complex SQL statements to complete data integration and correlation analysis, especially in scenarios involving multi-table joins, which places particularly stringent demands on their professional skills. Existing solutions are mainly based on traditional machine learning techniques and have not fully utilized the potential of next-generation generative AI, meaning that there is still significant potential for breakthroughs in improving governance efficiency. Summary of the Invention
[0007] The purpose of this invention is to provide a method and system for improving data governance capabilities based on large models, so as to solve the problems mentioned in the background art.
[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for improving data governance capabilities based on large models, comprising the following steps:
[0009] Extracting data standard information from specification documents and updating it to the data governance platform to assist users in decision-making involves reading data standard specification documents, extracting data elements, and solidifying the data standard extraction results.
[0010] By connecting to the data governance model interface, the user's requirements described in natural language are transformed into executable SQL development statements. The transformation process includes selecting data sources, calculating metrics and query examples, parsing the natural language descriptions, and performing SQL validation and output.
[0011] Preferably, the process of extracting data standard information from the specification document is as follows:
[0012] Reading data standard specification files: The document loading function is implemented through Python's PyPDF2 parsing library. The text content is extracted in page order, and the document content is divided into blocks according to the rules of headings, paragraphs, and tables, decomposing the entire standard document into a set of fine-grained text units.
[0013] Data element extraction: Design a prompt template to identify whether a text block contains the data element to be extracted. Call the large model to determine whether the text block contains the data element to be extracted. If it is determined to be yes, design a data element extraction prompt template to extract detailed information of the data element, including the name, definition, data type and value range of the data element.
[0014] Extraction result consolidation: Design the database table structure, mainly including the data element table, parse the extracted data element information and consolidate it into the database.
[0015] Preferably, in the process of translating user requirements described in natural language into executable SQL development statements, the specific details of selecting the data source, calculating metrics, and query examples are as follows:
[0016] Data metadata is created uniformly by users on the data source management page; data metrics are defined in the data model, including the field names, calculation methods, and descriptions of the metrics; query examples are optional, and providing suitable examples can improve the accuracy of generated SQL code.
[0017] Preferably, in the process of converting user requirements described in natural language into executable SQL development statements, the specific process of natural language description parsing is as follows:
[0018] Users input the content they want to query using natural language. The backend interface obtains relevant data source information, combines the selected calculation indicators and sample queries, injects the relevant information into the previously trained prompt template, calls the large code model, and returns automatically generated SQL code.
[0019] The preferred SQL validation and output process is as follows:
[0020] Users test the SQL code and judge its accuracy based on the test results. If there are any problems, they continue to adjust the natural language description or manually modify the SQL. The large model is a pre-trained model based on natural language processing, including but not limited to GPT and BERT models.
[0021] A system for improving data governance capabilities based on large models includes:
[0022] The data standard extraction module is used to extract data standard information from specification documents and update the extraction results to the data governance platform to assist user decision-making; and
[0023] The SQL statement generation module, by connecting to the data governance model interface, transforms the user's requirements described in natural language into executable SQL development statements;
[0024] The system operates based on a large AI model, which is a pre-trained model based on natural language processing, including but not limited to GPT and BERT models.
[0025] Preferably, the data standard extraction module includes:
[0026] The file reading unit is used to read data standard specification files. Specifically, it uses Python's PyPDF2 parsing library to implement document loading, extracts text content in page order, and divides the document content into blocks according to the rules of headings, paragraphs, and tables, decomposing the entire standard document into a set of fine-grained text units.
[0027] The data element extraction unit is used to extract data elements. Specifically, it is designed to identify whether a text block contains the data element content to be extracted. It calls the large model to determine whether the text block contains the data element content to be extracted. If the determination is yes, it designs a data element extraction prompt template to extract detailed information of the data element and calls the large model to extract the name, definition, data type and value range of the data element.
[0028] The result consolidation unit is used to solidify the data standard extraction results. It specifically designs the database table structure, mainly including the data element table, which parses the extracted data element information and solidifies it into the database.
[0029] Preferably, the SQL statement generation module includes:
[0030] The selection unit is used to select the data source, calculation metrics, and query examples. The data metadata is created uniformly by the user on the data source management page. The data metrics are defined in the data model, including the field names, calculation methods, and descriptions of the metrics. The query examples are optional, and providing appropriate examples can improve the accuracy of the generated SQL code.
[0031] The parsing unit is used to parse natural language descriptions. Specifically, the user inputs the content they want to query through natural language, the backend interface obtains relevant data source information, combines the selected calculation indicators and sample queries, injects the relevant information into the previously trained prompt template, calls the code big model, and returns automatically generated SQL code.
[0032] The verification output unit is used to perform SQL verification and output the results. Specifically, it allows users to test SQL code and determine the accuracy of the SQL based on the test results. If there are any problems, the natural language description can be adjusted or the SQL can be manually modified.
[0033] Preferably, when reading a standard file, the file reading unit further includes:
[0034] The document loading subunit utilizes Python's PyPDF2 parsing library to implement document loading functionality;
[0035] The text extraction sub-unit extracts text content from the loaded document in page number order;
[0036] The content is divided into sub-units, and the extracted text content is processed into blocks according to the rules of headings, paragraphs, and tables, decomposing the standard document into a set of fine-grained text units.
[0037] Preferably, the parsing unit, when parsing natural language descriptions, further includes:
[0038] The input retrieval subunit retrieves the content the user wants to query, inputted in natural language.
[0039] The information integration subunit obtains relevant data source information from the backend interface and integrates the relevant information by combining the calculation indicators selected by the selection unit and the query examples.
[0040] The template injection subunit injects the integrated relevant information into the pre-trained prompt template;
[0041] The model calls the sub-unit, which in turn calls the larger code model. Based on the prompt template of the injected information, it automatically generates SQL code and returns it.
[0042] Compared with the prior art, the beneficial effects of the present invention are:
[0043] This invention proposes a method and system for improving data governance capabilities based on large-scale models. Leveraging the multimodal and code generation capabilities provided by large-scale models, it interfaces with large-scale model inference services during the data standard definition and data development stages of data governance. This provides intelligent parsing of data standard documents, automatically extracting data standard information from these documents. Furthermore, it offers natural language requirement parsing to quickly generate corresponding SQL code, and provides code review and testing functions. This method significantly lowers the technical barrier to data development and improves its efficiency and accuracy. Attached Figure Description
[0044] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0046] Example 1: This invention provides a technical solution: a method for improving data governance capabilities based on large models, comprising the following steps:
[0047] Data standard information is extracted from standard documents and updated to the data governance platform to assist user decision-making. The process includes: reading the standard documents, extracting data elements, and solidifying the extraction results.
[0048] By connecting to the data governance model interface, user requirements described in natural language are transformed into executable SQL statements. The process includes: selecting data sources, calculating metrics and query examples, parsing natural language descriptions, and validating and outputting the SQL.
[0049] The specific method for extracting standard data information included in this invention is as follows:
[0050] 1) Read the specification file.
[0051] Standard documents typically exist in various formats, such as Word documents (.doc, .docx) and PDF documents (.pdf). This invention primarily uses PDF documents and implements document loading functionality through Python's PyPDF2 parsing library, extracting text content in page-number order to ensure the integrity of the original document structure.
[0052] The document content is divided into blocks according to certain rules such as headings, paragraphs, and tables, breaking down the entire standard document into a set of fine-grained text units. This structured processing approach facilitates accurate analysis of each text unit in subsequent steps, thereby significantly improving the accuracy of data element recognition and processing efficiency.
[0053] 2) Data element extraction.
[0054] a) First, determine whether the text block contains the data element to be extracted. If it does, then further extract the detailed information of the data element. The specific process is as follows:
[0055] A prompt template is designed to identify whether a text block contains the data element to be extracted. This template includes clear instructions and examples to help large models make accurate judgments. The prompt template is as follows:
[0056]
[0057] An iterative processing mechanism is adopted to inject each text block in 1) into the prompt template, call the large model for intelligent analysis, and determine whether to continue extracting data meta information based on the output results.
[0058] b) For the text block in the output of step a) that contains the data elements to be extracted, design a data element extraction prompt template to extract detailed data element information. This prompt template should have a clear description and example of the data element extraction task, and explicitly require the extracted results to be output in a specific format to facilitate...
[0059] The subsequent processing and storage, and the data element extraction prompt template are designed as follows:
[0060] An iterative processing mechanism is adopted. The text block containing the data element content to be extracted in the output result of step a) is injected into the data element extraction prompt template. The large model is called to extract data elements and return the extraction results in a fixed format, which facilitates subsequent result processing and storage.
[0061] 2) Solidify the extraction results.
[0062] After the first two steps, the data elements have been extracted and generated from the data standard document. The extracted data element information is then parsed and solidified into the database so that subsequent data modeling and data quality inspection processes can directly reference the data standard.
[0063] The specific method for generating SQL data development code included in this invention is as follows:
[0064] 3) Select data source, calculate metrics, and query examples.
[0065] Users select relevant data sources, calculated metrics, and query examples. Data metadata is created uniformly by users on the data source management page; data metrics are defined in the data model, including the corresponding field names, calculation methods, and descriptive information; query examples are optional, and having suitable examples can improve the accuracy of generated SQL code.
[0066] 4) Natural Language Description and Parsing
[0067] Users input their query content in natural language, and the backend interface obtains relevant data source information, including database table field definitions, data types, comments, lineage, and constraint relationships. Combining the selected calculation indicators and sample queries, the relevant information is injected into the previously trained prompt template. The large code model is then invoked, and automatically generated SQL code is returned.
[0068] 5) SQL validation and output
[0069] For the SQL code returned in 2), users can perform SQL code testing and judge whether the SQL is accurate based on the test results. If there are any problems, users can continue to adjust the natural language description or manually modify the SQL. This query will be automatically recorded as a historical query example.
[0070] Example 2, based on Example 1, proposes a system for improving data governance capabilities based on large models, including:
[0071] The data standard extraction module is used to extract data standard information from standard documents and update the extraction results to the data governance platform to assist user decision-making. It includes: a file reading unit, used to read data standard specification documents, specifically using Python's PyPDF2 parsing library to load the document, extracting text content in page order, and dividing the document content into blocks according to headings, paragraphs, and tables, decomposing the entire standard document into fine-grained sets of text units; a data element extraction unit, used to extract data elements, specifically designed to identify whether a text block contains the data element to be extracted, calling a large model to determine if the text block contains the data element to be extracted, and if so, designing a data element extraction prompt template to extract detailed data element information, calling the large model to extract the name, definition, data type, and value range of the data element; and a result consolidation unit, used to solidify the data standard extraction results, specifically designed a database table structure, mainly including a data element table, parsing the extracted data element information and solidifying it into the database.
[0072] The SQL statement generation module, through its interface with the data governance model, transforms user-described requirements in natural language into executable SQL statements. It includes: a selection unit for choosing the data source, calculation metrics, and query examples. Data metadata is created by the user on the data source management page, data metrics are defined in the data model, including field names, calculation methods, and descriptions, and query examples are optional; providing suitable examples improves the accuracy of the generated SQL code. A parsing unit parses the natural language description. Specifically, the user inputs the desired query in natural language, the backend interface retrieves relevant data source information, combines the selected calculation metrics and the query example, injects the relevant information into a pre-trained prompt template, calls the code model, and returns automatically generated SQL code. A verification and output unit verifies and outputs the SQL. Specifically, the user tests the SQL code, and the test results determine its accuracy. If problems are found, the user can further adjust the natural language description or manually modify the SQL.
[0073] The system operates based on a large AI model, which is a pre-trained model based on natural language processing, including but not limited to GPT and BERT models.
[0074] The file reading unit, when reading a standard document, further includes: a document loading subunit, which uses Python's PyPDF2 parsing library to implement document loading; a text extraction subunit, which extracts text content from the loaded document in page order; and a content segmentation subunit, which segments the extracted text content into blocks according to the rules of headings, paragraphs, and tables, decomposing the standard document into a set of fine-grained text units.
[0075] The parsing unit, when parsing natural language descriptions, further includes: an input acquisition subunit, which acquires the content the user wants to query in natural language; an information integration subunit, which acquires relevant data source information from the backend interface and integrates the relevant information by combining the calculation indicators selected by the selection unit and the query examples; a template injection subunit, which injects the integrated relevant information into a pre-trained prompt template; and a model invocation subunit, which invokes the code model, automatically generates SQL code based on the prompt template with injected information, and returns it.
[0076] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for improving data governance capabilities based on large models, characterized in that: Includes the following steps: Extracting data standard information from specification documents and updating it to the data governance platform to assist users in decision-making involves reading data standard specification documents, extracting data elements, and solidifying the data standard extraction results. By connecting to the data governance model interface, the user's requirements described in natural language are transformed into executable SQL development statements. The transformation process includes selecting data sources, calculating metrics and query examples, parsing the natural language descriptions, and performing SQL validation and output.
2. The method for improving data governance capabilities based on large models according to claim 1, characterized in that: The process of extracting data standard information from standard documents is as follows: Reading data standard specification files: The document loading function is implemented through Python's PyPDF2 parsing library. The text content is extracted in page order, and the document content is divided into blocks according to the rules of headings, paragraphs, and tables, decomposing the entire standard document into a set of fine-grained text units. Data element extraction: Design a prompt template to identify whether a text block contains the data element to be extracted. Call the large model to determine whether the text block contains the data element to be extracted. If it is determined to be yes, design a data element extraction prompt template to extract detailed information of the data element, including the name, definition, data type and value range of the data element. Extraction result consolidation: Design the database table structure, mainly including the data element table, parse the extracted data element information and consolidate it into the database.
3. The method for improving data governance capabilities based on a large model according to claim 2, characterized in that: In the process of translating user requests described in natural language into executable SQL statements, the specific details of selecting the data source, calculating metrics, and query examples are as follows: Data metadata is created uniformly by users on the data source management page; data metrics are defined in the data model, including the field names, calculation methods, and descriptions of the metrics; query examples are optional, and providing suitable examples can improve the accuracy of generated SQL code.
4. The method for improving data governance capabilities based on a large model according to claim 3, characterized in that: The specific process of parsing natural language descriptions in converting user requests into executable SQL statements is as follows: Users input the content they want to query using natural language. The backend interface obtains relevant data source information, combines the selected calculation indicators and sample queries, injects the relevant information into the previously trained prompt template, calls the large code model, and returns automatically generated SQL code.
5. The method for improving data governance capabilities based on a large model according to claim 4, characterized in that: The SQL validation and output process is as follows: Users test the SQL code and judge its accuracy based on the test results. If there are any problems, they continue to adjust the natural language description or manually modify the SQL. The large model is a pre-trained model based on natural language processing, including but not limited to GPT and BERT models.
6. A system for improving data governance capabilities based on large models, employing the method described in claim 5, characterized in that: include: The data standard extraction module is used to extract data standard information from standard documents and update the extraction results to the data governance platform to assist users in decision-making. as well as The SQL statement generation module, by connecting to the data governance model interface, transforms the user's requirements described in natural language into executable SQL development statements; The system operates based on a large AI model, which is a pre-trained model based on natural language processing, including but not limited to GPT and BERT models.
7. The system for improving data governance capabilities based on a large model according to claim 6, characterized in that: The data standard extraction module includes: The file reading unit is used to read data standard specification files. Specifically, it uses Python's PyPDF2 parsing library to implement document loading, extracts text content in page order, and divides the document content into blocks according to the rules of headings, paragraphs, and tables, decomposing the entire standard document into a set of fine-grained text units. The data element extraction unit is used to extract data elements. Specifically, it is designed to identify whether a text block contains the data element content to be extracted. It calls the large model to determine whether the text block contains the data element content to be extracted. If the determination is yes, it designs a data element extraction prompt template to extract detailed information of the data element and calls the large model to extract the name, definition, data type and value range of the data element. The result consolidation unit is used to solidify the data standard extraction results. It specifically designs the database table structure, mainly including the data element table, which parses the extracted data element information and solidifies it into the database.
8. The system for improving data governance capabilities based on a large model according to claim 7, characterized in that: The SQL statement generation module includes: The selection unit is used to select the data source, calculation metrics, and query examples. The data metadata is created uniformly by the user on the data source management page. The data metrics are defined in the data model, including the field names, calculation methods, and descriptions of the metrics. The query examples are optional, and providing appropriate examples can improve the accuracy of the generated SQL code. The parsing unit is used to parse natural language descriptions. Specifically, the user inputs the content they want to query through natural language, the backend interface obtains relevant data source information, combines the selected calculation indicators and sample queries, injects the relevant information into the previously trained prompt template, calls the code big model, and returns automatically generated SQL code. The verification output unit is used to perform SQL verification and output the results. Specifically, it allows users to test SQL code and determine the accuracy of the SQL based on the test results. If there are any problems, the natural language description can be adjusted or the SQL can be manually modified.
9. A system for improving data governance capabilities based on a large model according to claim 8, characterized in that: When reading a standard file, the file reading unit further includes: The document loading subunit utilizes Python's PyPDF2 parsing library to implement document loading functionality; The text extraction sub-unit extracts text content from the loaded document in page number order; The content is divided into sub-units, and the extracted text content is processed into blocks according to the rules of headings, paragraphs, and tables, decomposing the standard document into a set of fine-grained text units.
10. A system for improving data governance capabilities based on a large model according to claim 9, characterized in that: The parsing unit, when parsing natural language descriptions, further includes: The input retrieval subunit retrieves the content the user wants to query, inputted in natural language. The information integration subunit obtains relevant data source information from the backend interface and integrates the relevant information by combining the calculation indicators selected by the selection unit and the query examples. The template injection subunit injects the integrated relevant information into the pre-trained prompt template; The model calls the sub-unit, which in turn calls the larger code model. Based on the prompt template of the injected information, it automatically generates SQL code and returns it.