Method for enhancing data governance capability based on AI large model

Abstract

A method for realizing data governance capability enhancement based on an AI large model comprises the following steps: extracting data standard information from a standard file, assisting a user to make a decision, and then automatically synchronizing the decision to a data governance system; and generating an SQL data development code according to a requirement input by a user through a natural language by butting a programming model interface. According to the method, the multi-mode capability and the code generation capability provided by the large model are fully utilized, a large model reasoning service interface is connected in the data standard definition and data development links of data governance, the time of data governance landing implementation is remarkably shortened, and the enhancement of the data governance capability is realized.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and specifically to a method for enhancing data governance capabilities based on an AI big model. Background Art

[0002] The rapid development of generative artificial intelligence has brought opportunities for all walks of life to reshape product interaction methods and improve product intelligence. For many large and medium-sized enterprises, the data governance process requires a lot of manpower costs due to the diversity of data and the complexity of data processing processes. Therefore, many manufacturers hope to significantly improve the efficiency of data governance by combining AI big models with data governance scenarios. At present, the methods for optimizing data governance based on AI technology in the industry mainly include:

[0003] (1) AI-based optimization of data collection: The AI-based optimization of data collection processes includes: using AI to automatically identify data sources, data formats, and data types to improve access efficiency; using AI pattern recognition and predictive analysis to optimize data conversion rules; and using AI to optimize the data loading process to ensure efficient storage and fast access to data.

[0004] (2) Optimizing metadata management based on AI: Technical metadata involves the physical structure and storage method of data, including database table structure, conversion rules, and data history. AI can identify and optimize these metadata in an automated way to ensure data consistency and accuracy. Business metadata describes the business meaning and purpose of data, including business meaning, data standards, indicator meaning, and measurement methods. AI can help understand and interpret these metadata, making data easier to understand and use. Metadata, also known as intermediary data or relay data, is information about the organization, data domain, and its relationships. In simple terms, it is data about data. Metadata mainly describes the property information of data, which is used to support functions such as indicating storage location, historical data, resource search, and file records. In the context of data governance, metadata can be divided into multiple types, such as technical metadata, business metadata, and management metadata.

[0005] (3) Optimizing data quality assessment based on AI: Intelligent data quality assessment management involves multiple indicators, including completeness, standardization, consistency, accuracy, uniqueness, and timeliness. AI extracts effective data quality assessment indicators through classification learning, function learning, regression technology and other methods in machine learning. AI can also automate the data audit process and detect data quality problems in a timely manner through continuous monitoring and analysis of data.

[0006] The above three solutions cover part of the data governance process, but no clear solutions are given for the data standard definition and data development links, which are very time-consuming in the data governance process. Moreover, the above solutions basically rely on traditional AI machine learning capabilities and fail to give full play to the advantages of current generative artificial intelligence. Therefore, there is still huge room for improving data governance efficiency. Summary of the invention

[0007] In view of this, the problem to be solved by the present invention is to provide a method for enhancing data governance capabilities based on an AI big model.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for enhancing data governance capabilities based on an AI big model, comprising the following steps:

[0009] Extract data standard information from specification documents, assist users in making decisions, and then automatically synchronize to the data governance system;

[0010] By connecting to the programming model interface, SQL data development code is generated based on the user's requirements input through natural language.

[0011] In the present invention, preferably, the data extraction process is specifically as follows: upload the data standard specification file, then click the extract button, return the formatted standard data, determine whether it needs to be entered, if so, manually enter the standard data, and then perform the repeated viewing steps, otherwise end.

[0012] In the present invention, preferably, the process of generating SQL data development code is specifically: selecting a data source, selecting calculation indicators, using sample code, inputting a natural language description, returning to automatically generate SQL, SQL testing and verifying SQL accuracy based on SQL test results, manually proofreading and outputting the final SQL result.

[0013] In the present invention, preferably, the SQL accuracy verification process specifically determines whether the SQL is correct, and if so, generates sample code and outputs the final SQL result; otherwise, outputs the final SQL result after manual proofreading.

[0014] In the present invention, preferably, the data standard specification file is set as a PDF file, and the background program uses the path of the PDF file as a parameter to call the corresponding large model reasoning service interface. After receiving the request, the interface first uses the multimodal large model to read and semantically parse the PDF file; then calls the large model interface based on the pre-trained prompt words as input, and the prompt word content includes the content to be extracted and the format of the returned result.

[0015] In the present invention, preferably, the data returned by the multimodal large model is returned to the front end for pop-up display, and the data format is JSON array format.

[0016] In the present invention, preferably, the data source is uniformly created by the user on the data source management page, and the data source to be queried is selected. The data source definition is entered in advance in the data model function, including the field name, calculation method and explanation information corresponding to the data source.

[0017] The advantages and positive effects of the present invention are: the present invention makes full use of the multimodal capabilities and code generation capabilities provided by the big model, connects to the big model reasoning service interface in the data standard definition and data development links of data governance, significantly reduces the time for data governance implementation, and realizes the enhancement of data governance capabilities. It provides an intelligent parsing function for data standard files, which can automatically extract data standard information from specification files, assist users in making decisions, and then automatically synchronize to the data governance system. Thereby saving days for the standard file sorting and entry that previously took weeks; in addition, by connecting to the company's self-developed programming big model interface, it can understand the needs of users input through natural language, quickly generate corresponding SQL code, and provide code interpretation and code testing capabilities. By introducing this method, the technical threshold for data development can be greatly reduced, and the efficiency and accuracy of data development can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0019] Figure 1 This is an overall architecture diagram of a method for enhancing data governance capabilities based on an AI big model of the present invention;

[0020] Figure 2 It is a specific process diagram of extracting data standard information of a method for enhancing data governance capability based on an AI big model of the present invention;

[0021] Figure 3 It is a specific process diagram of generating SQL data development code for a method of enhancing data governance capability based on an AI big model of the present invention;

[0022] Figure 4 It is a schematic diagram of the process of extracting data standard information of a method for enhancing data governance capability based on an AI big model of the present invention;

[0023] Figure 5 It is a flow chart of generating SQL data development code according to a method for enhancing data governance capability based on an AI big model of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be noted that when a component is referred to as being "fixed to" another component, it may be directly on the other component or there may also be a component centered. When a component is considered to be "connected to" another component, it may be directly connected to the other component or there may also be a component centered. When a component is considered to be "set on" another component, it may be directly set on the other component or there may also be a component centered. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0027] The present invention embeds the large model service interface into the workflow of the data governance product to improve the user experience and work efficiency. Figure 1 The overall functional architecture of the current data governance product is explained. The following will explain the entire AI enhancement method in detail with the accompanying drawings.

[0028] Please see attached Figure 2, the user first uploads a PDF document with data standards, and then clicks the smart extraction button. The background program uses the PDF document path as a parameter to call the corresponding large model reasoning service interface. After receiving the request, the interface first uses the multimodal large model to read and semantically parse the PDF document; multimodality refers to the collaborative reasoning of multiple heterogeneous modal data. In the field of artificial intelligence, it often refers to the collaboration of perceptual information, such as images, text, and voice, to help artificial intelligence understand the external world more accurately; then the large model interface is called according to the previously pre-trained prompt words as input. The prompt word is an input text used to guide or stimulate the artificial intelligence model to generate specific outputs. These prompt words can be natural language or structured instructions or templates. The structured prompt word framework is a specific instruction method used to guide AI models, such as ChatGPT, to generate specific types of answers. This framework ensures clear and purposeful communication by structuring prompt words into fixed patterns. The prompt words include the content you want to extract and the format of the returned results. The data in JSON array format returned by the large model is returned to the front end for pop-up display. The user determines the accuracy of the extraction result based on the term and the position of the term in the original document, and then selects the standard data to be imported for batch import. The background import interface performs duplicate verification on the relevant data, and imports the terms that do not exist in the system into the database table. After the import is completed, the results are returned to the front end, and the extraction process ends.

[0029] Please refer to the work logic flow chart and attached Figure 3 , the user selects the data source, and the data source information is created uniformly by the user on the data source management page; then select the relevant data indicators to be queried. The definitions of these indicators are entered in advance in the data model function, including the field name, calculation method and explanation information corresponding to the indicator; then the user can choose whether to use a certain historical sample query, which is optional here. If there is a suitable sample, the accuracy of the generated SQL code can be improved; then the user enters the content to be queried in natural language description, clicks the button for auxiliary code generation, and passes the above selected content as a parameter to the background interface; the background interface obtains the library table information under the specified data source by querying the data governance database, including library table field definition, type, annotation, association relationship, etc.; then, combined with the selected indicator content and sample query, the prompt word content is spliced ​​according to the previously trained prompt word template; the prompt word is passed to the code big model, and the automatically generated SQL code is returned; the front-end user can judge whether this SQL is accurate based on the SQL code test results. If there is a problem, you can continue to adjust the natural language description or manually modify it to proofread the SQL. If there is no problem, you can click the approval button and use this case as one of the samples for subsequent queries; at this point, the entire data development process assisted by the big model ends.

[0030] like Figure 4 As shown, the present invention provides a method for enhancing data governance capabilities based on an AI big model, comprising the following steps: extracting data standard information from a specification file to assist user decision-making and then automatically synchronizing it to a data governance system; by docking with a programming model interface, generating SQL data development code based on user requirements input through natural language.

[0031] In this embodiment, further, the data extraction process is specifically as follows: upload the data standard specification file, then click the extract button, return the formatted standard data, determine whether it needs to be entered, if so, manually enter the standard data, and then perform the repeated viewing steps, otherwise end.

[0032] like Figure 5 As shown, in this embodiment, further, the process of generating SQL data development code is specifically: selecting a data source, selecting calculation indicators, using sample code, entering a natural language description, returning to automatically generate SQL, SQL testing and verifying SQL accuracy based on SQL test results, manual proofreading and outputting the final SQL result.

[0033] In this embodiment, further, the SQL accuracy verification process specifically determines whether the SQL is correct. If so, a sample code is generated and the final SQL result is output; otherwise, the final SQL result is output after manual proofreading.

[0034] In this embodiment, further, the data standard specification file is set as a PDF file, and the background program uses the path of the PDF file as a parameter to call the corresponding large model reasoning service interface. After receiving the request, the interface first uses the multimodal large model to read and semantically parse the PDF file; then the large model interface is called based on the pre-trained prompt words as input, and the prompt word content includes the content to be extracted and the format of the returned results.

[0035] In this embodiment, further, the data returned by the multimodal large model is returned to the front end for pop-up display, and the data format is JSON array format.

[0036] In this embodiment, further, the data source is uniformly created by the user on the data source management page, and the data source to be queried is selected. The data source definition is entered in advance in the data model function, including the field name, calculation method and explanation information corresponding to the data source.

[0037] The present invention makes full use of the multimodal capabilities and code generation capabilities provided by the big model, and connects the big model reasoning service interface in the data standard definition and data development links of data governance, which significantly reduces the time for data governance implementation and enhances data governance capabilities.

[0038] Unified data standard definition is the premise of data governance. In the scenario of data governance, a large amount of manpower is often required to organize and define the data elements and data models involved in the current business, and enter them into the data governance system. Even if there are relevant industry specification documents, since most of the documents are provided in the form of PDF or pictures, the entry work still needs to be completed manually. The present invention provides an intelligent parsing function for data standard files, which can automatically extract data standard information from specification files, assist users in making decisions, and then automatically synchronize them to the data governance system. Thereby saving days for the standard file organization and entry that previously took weeks.

[0039] The data development process often requires data management personnel to write a large amount of SQL code to realize data association, aggregation and other operations, especially in data analysis scenarios involving multiple tables, which places high demands on the professional capabilities of data management personnel. The present invention can understand the needs of users input through natural language by connecting to the company's self-developed programming large model interface, quickly generate corresponding SQL code, and provide code interpretation and code testing capabilities. By introducing this method, the technical threshold for data development can be greatly reduced, and the efficiency and accuracy of data development can be improved.

[0040] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of this patent.

Claims

1. A method for enhancing data governance capabilities based on an AI big model, characterized in that: The steps include: Extract data standard information from specification documents, assist users in making decisions, and then automatically synchronize to the data governance system; By connecting to the programming model interface, SQL data development code is generated based on the user's requirements input through natural language.

2. According to claim 1, a method for enhancing data governance capabilities based on an AI big model is characterized in that: The data extraction process is specifically as follows: upload the data standard specification file, then click the extract button, return the formatted standard data, determine whether it needs to be entered, if so, manually enter the standard data, then perform the repeated viewing steps, otherwise end.

3. According to claim 1, a method for enhancing data governance capabilities based on an AI big model is characterized in that: The process of generating SQL data development code is specifically as follows: selecting a data source, selecting calculation indicators, using sample code, entering a natural language description, returning to automatically generate SQL, SQL testing and verifying SQL accuracy based on SQL test results, manual proofreading and outputting the final SQL results.

4. According to claim 3, a method for enhancing data governance capabilities based on an AI big model is characterized in that: The SQL accuracy verification process specifically determines whether the SQL is correct. If so, a sample code is generated and the final SQL result is output; otherwise, the final SQL result is output after manual proofreading.

5. According to claim 1, a method for enhancing data governance capabilities based on an AI big model is characterized in that: The data standard specification file is set as a pdf file. The background program uses the path of the pdf file as a parameter and calls the corresponding large model reasoning service interface. After receiving the request, the interface first uses the multimodal large model to read and semantically parse the pdf file; then calls the large model interface based on the pre-trained prompt words as input. The prompt word content includes the content to be extracted and the format of the returned result.

6. According to claim 5, a method for enhancing data governance capabilities based on an AI big model is characterized in that: The data returned by the multimodal large model is returned to the front end for pop-up display, and the data format is JSON array format.

7. According to claim 3, a method for enhancing data governance capabilities based on an AI big model is characterized in that: The data source is created uniformly by the user on the data source management page, and the data source to be queried is selected. The data source definition is entered in advance in the data model function, including the field name, calculation method and explanation information corresponding to the data source.

Images