Data management and development integrated management system based on AI

Abstract

The invention provides an AI-based data governance and development integrated management system. The AI-based data governance and development integrated management system comprises a data standard module, a data quality module, a data service module, a data asset module, an index management module, a data modeling and data development module and a metadata module. According to the method, an embedded data governance concept is introduced, data governance related activities are embedded in the links of data modeling and data development, and when a data table and related fields of a physical model are created, a data standard is forcibly referenced, so that the data standard is ensured to effectively fall into the ground, and the data quality is substantially improved. Meanwhile, by means of AI, a large model and a knowledge base technology, SQL statements needed by development are automatically generated, the standardization of SQL writing is improved, and the data development efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, and in particular to an AI-based integrated management system for data governance development. Background Technology

[0002] The fundamental purpose of data governance is to ensure that enterprise data is visible, searchable, usable, and easy to use. In the process of data governance, factories often spend a lot of energy on the construction of the so-called data governance system due to a lack of overall understanding and implementation experience. They have formulated a lot of documents such as data standards, data specifications, and data management systems, but have neglected the implementation of these data standards and specifications. That is, to truly follow and implement these standards in the process of data development and application, or even to enforce data standards and specifications through tools in the process of data development. The current situation has the following defects: (1) Metadata governance is not thorough. The metadata and data dictionaries of some business systems are incomplete and non-standard; (2) Data governance and data development are two separate things. In the process of data modeling or data development, when data developers write SQL statements, they often use independent SQL writing tools. When creating data tables and fields, they cannot refer to data standards and data specifications, which leads to the ineffective implementation of data standards; (3) Data development efficiency is low. The SQL statements written lack standardization and the SQL running performance is not high. Summary of the Invention

[0003] To address the aforementioned problems, this invention provides an AI-based integrated data governance and development management system. This invention introduces the concept of "embedded" data governance, embedding data governance-related activities into the data modeling and data development stages. Specifically, it designs and develops functions for data modeling and data development, mandating the use of data standards when creating data tables and related fields in the physical model, ensuring the effective implementation of data standards and substantially improving data quality. Simultaneously, leveraging AI, large-scale models, and knowledge base technologies, it automatically generates the necessary SQL statements for development, improving the standardization of SQL writing and increasing data development efficiency.

[0004] This invention provides an AI-based integrated data governance and development management system, comprising: a data standard module, a data quality module, a data service module, a data asset module, an indicator management module, a data modeling and data development module, and a metadata module. The metadata module stores metadata. The data standard module establishes data standards, including basic data standards and indicator data standards. The data modeling and data development module is used for data modeling and data development, and the basic data standards are implemented through data modeling. The data quality module has data quality checking rules for data quality checking during data development. The indicator management module provides indicator processing logic support for data modeling and data development, and the indicator data standards are implemented through indicator management. The indicator management module receives indicators created by the metadata module and provides indicator lineage analysis for the metadata. The data service module provides data service assets to the data asset module. The data asset module acquires indicator data assets output by the indicator management module, acquires data service assets output by the data service module, acquires metadata from the metadata module, and provides indicator lineage analysis for the metadata.

[0005] Furthermore, in the AI-based integrated data governance and development management system solution provided by this invention: the basic data standard includes code value standards.

[0006] Furthermore, in the AI-based integrated data governance and development management system solution provided by this invention: during the data modeling and data development module, when creating a physical table, the table fields can select the data standard through a drop-down box, including code value standards and indicator data standards.

[0007] Furthermore, in the AI-based integrated data governance and development management system solution provided by this invention: the data modeling and data development module specifically includes the following in the data development process: before data development, configuring the AI ​​business knowledge base through business metadata and technical metadata to ensure that the business meaning, business rules and association rules of each data table are clear during data development; after configuring the AI ​​business knowledge, writing SQL for data development.

[0008] Furthermore, in the AI-based integrated data governance and development management system solution provided by this invention: the SQL writing is based on user requests, the system automatically writes SQL through AI, corrects syntax errors, and finally provides complete SQL statements that conform to data standards and specifications.

[0009] Furthermore, in the AI-based integrated data governance and development management system solution provided by this invention: during the SQL writing process, the system automatically provides performance optimization suggestions for the SQL through AI.

[0010] Furthermore, in the AI-based integrated data governance and development management system solution provided by this invention: the development process of the AI ​​SQL agent adopted by the data modeling and data development module in the data development process includes the following steps: data development is carried out through the integrated data governance and development platform, and the AI ​​large model is automatically developed with one click, and the AI ​​SQL automatically writes the initial SQL requirements; data governance is carried out through the integrated data governance and development platform, and combined with RAG knowledge base retrieval enhancement technology, the AI ​​SQL agent automatically generates SQL, and after the generated SQL is confirmed to be correct, it is submitted to the big data platform; human-computer interaction is carried out based on AI feedback to improve the SQL writing requirements and realize the closed-loop management of the SQL agent.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] (1) The AI-based integrated data governance and development management system of the present invention creates an integrated platform for data governance and development, and realizes the automatic reference of data standards and data specifications in the process of data modeling and data development, thus avoiding the problem of data governance and data development being separate entities;

[0013] (2) The AI-based integrated data governance and development management system of the present invention utilizes AI, big data models, knowledge base and other technologies to create an AI SQL intelligent agent, which automatically generates SQL and performs SQL syntax error checks by automatically identifying and parsing technical metadata, indicator data, business metadata, etc.

[0014] (3) The AI-based integrated data governance and development management system of the present invention has excellent performance in the SQL statements written by AI, avoiding database performance problems caused by non-standard SQL.

[0015] (4) The AI-based integrated data governance and development management system of the present invention uses AI SQL intelligent agents to guide questions based on AI feedback to conduct human-computer interaction, improve SQL requirements, form a closed loop, thereby reducing the knowledge cost of developers and improving development efficiency. Attached Figure Description

[0016] Figure 1 This is a structural framework diagram of the AI-based integrated data governance and development management system according to an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of code value standard management in an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of data standard management in an embodiment of the present invention;

[0019] Figure 4This is a schematic diagram illustrating the data standards referenced during the data modeling process in this embodiment of the invention;

[0020] Figure 5 This is a schematic diagram of SQL writing for AI one-click development in an embodiment of the present invention;

[0021] Figure 6 This is a schematic diagram of the SQL performance optimization suggestions developed by AI with one click in an embodiment of the present invention;

[0022] Figure 7 This is a schematic diagram of the AI ​​SQL agent development process in an embodiment of the present invention. Detailed Implementation

[0023] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following embodiments, in conjunction with the accompanying drawings, will specifically illustrate the technical solution of this invention.

[0024] See Figure 1 This document describes an AI-based integrated data governance and development management system, comprising a data standards module, a data quality module, a data service module, a data asset module, an indicator management module, a data modeling and development module, and a metadata module. The metadata module includes a metadata database storing metadata, specifically schema-level information such as data table structure, field types, primary and foreign key relationships, and a field lineage graph. The data standards module establishes data standards, including basic data standards and indicator data standards. The data modeling and development module is used for data modeling and development; basic data standards are implemented through data modeling. The data quality module contains data quality check rules for data quality checks during the data development process. The indicator management module provides indicator processing logic support for data modeling and development; indicator data standards are implemented through indicator management. The indicator management module receives indicators created by the metadata module and provides indicator lineage analysis for metadata. The indicator management module includes an indicator library for defining business indicator calculation formulas and dimensional constraints, supporting nested indicator formula parsing. The data service module provides data service assets to the data asset module. The data asset module is used to acquire indicator data assets output by the indicator management module and data service assets output by the data service module. It also acquires business-type and technical-type metadata from the metadata module and provides indicator lineage analysis for the metadata. This embodiment's AI-based integrated data governance and development management system employs the anti-illusion Prompt engine.

[0025] The basic data standards include code value standards (data standards based on table fields). See also Figure 2 and Figure 3 , Figure 2 This illustrates the management of code value standards. Figure 3It illustrates the management of data standards (including basic data and indicator data standards).

[0026] In the data modeling process, such as Figure 4 As shown, when creating a physical table, table fields allow users to select data standards via dropdown menus, including code value standards and indicator data standards. By mandating the reference of data standards during data modeling, the effective implementation and execution of data standards are ensured, preventing a disconnect between data development and data governance, and effectively resolving subsequent data development chaos and data inconsistency quality issues.

[0027] The data development process is as follows: Before data development, an AI business knowledge base is configured using business metadata and technical metadata to ensure that the business meaning, business rules, and relationship rules of each data table are clear during data development. After configuring the AI ​​business knowledge base, the SQL for data development is written. For example... Figure 5 As shown, SQL writing is based on user requests (after selecting the target table, click "AI One-Click Development"). The system automatically writes SQL using AI, corrects syntax errors, and ultimately provides a complete SQL statement that conforms to data standards. Figure 6 As shown, during the SQL writing process, the system automatically provides performance optimization suggestions for the SQL using AI.

[0028] See Figure 7 The development process of the AI ​​SQL agent in the data modeling and data development module includes the following steps: data development is carried out through the integrated data governance and development platform, and the AI ​​large model is automatically developed with one click, and the AI ​​SQL automatically writes the initial SQL requirements; at the same time, data governance is carried out through the integrated data governance and development platform, and combined with RAG knowledge base retrieval enhancement technology, the AI ​​SQL agent automatically generates SQL, and after the generated SQL is confirmed to be correct, it is submitted to the big data platform; in this process, human-computer interaction based on AI feedback is supported to improve the SQL writing requirements and realize the closed-loop management of the SQL agent.

[0029] The data modeling and data development module has a business rule base that stores SQL writing specifications (such as JOIN relationships).

[0030] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An AI-based data governance development integration management system, characterized by, Comprise: Data standard module, data quality module, data service module, data asset module, index management module, data modeling and data development module, metadata module, Among them, the metadata module stores metadata; The data standard module is provided with data standards, including basic data standards and index data standards; The data modeling and data development module is used for data modeling and data development, and the basic data standard is marked by data modeling; The data quality module is provided with data quality checking rules for data quality checking during data development; The index management module provides index processing logic support for data modeling and data development, and the index data standard is marked by index management; The index management module receives the indexes created by the metadata module and provides index blood relation analysis for metadata; The data service module is used for providing data service assets for the data asset module; The data asset module is used for obtaining the index data assets output by the index management module, obtaining the data service assets output by the data service module, obtaining the metadata of the metadata module, and providing index blood relation analysis for metadata.

2. The AI-based data governance and development integrated management system of claim 1, wherein: wherein, The basic data standard includes code value standard.

3. The AI-based data governance and development integrated management system of claim 2, wherein: wherein In the data modeling and data development module, when creating a physical table in the data modeling process, the table field selects the data standard through a drop-down box, including code value standard and index data standard.

4. The AI-based data governance and development integrated management system of claim 1, wherein: wherein The data modeling and data development module in the data development process specifically includes: Before data development, AI business knowledge base configuration is performed through business metadata and technical metadata to ensure the business meaning, business rules and correlation rules of each data table during data development are clear; After AI business knowledge configuration, SQL writing for data development is performed.

5. The AI-based data governance and development integrated management system of claim 4, wherein: wherein The SQL writing is based on user request, and the system automatically writes SQL through AI, corrects syntax errors, and finally gives a complete SQL statement that meets the data standard specification.

6. The AI-based data governance and development integrated management system of claim 5, wherein: wherein During the SQL writing process, the system automatically gives performance optimization suggestions for SQL through AI.

7. The AI-based data governance and development integrated management system of any one of claims 4-6, wherein: wherein The development process of AI SQL intelligent agent adopted by the data modeling and data development module in the data development process includes the following steps: Through the data governance and development integrated platform, AI big model is automatically developed by one key, and SQL is automatically written by AI for initial requirements; Through data governance development integrated platform for data governance, combined with RAG knowledge base retrieval enhancement technology, AISQL intelligent body is used to automatically generate SQL, and the generated SQL is submitted to the big data platform after being confirmed to be correct; Based on AI feedback, human-computer interaction is carried out, SQL writing requirements are improved, and closed-loop management of the SQL intelligent body is realized. 8.The AI-based data governance development-integrated management system of claim 1, wherein: The AI-based data governance development integrated management system adopts a Prompt engine.

Images