Data management method and system based on large model

Abstract

The invention particularly relates to a data governance method and system based on a large model. According to the data governance method based on the large model, data are automatically collected, and the data are preliminarily cleaned and preprocessed; carrying out deep analysis, mining and prediction on the data by utilizing the large model, and carrying out data conversion and formatting according to business requirements; a data quality standard, a data security policy and a data management process are formulated and executed according to service requirements, so that the accuracy, integrity and security of data are ensured; the data quality and the data flow are monitored in real time, and if a data quality problem is found, the data quality problem is fed back to the large model so that the large model can solve the data quality problem in time. According to the data governance method and system based on the large model, intelligentization and automation of data governance are achieved, the burden of manual operation is relieved, the data governance efficiency is improved, the data security and compliance are optimized, intelligent decision support is achieved, and the data value mining capacity is improved.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and in particular to a data governance method and system based on a big model. Background Art

[0002] Data governance plays a vital role in modern enterprises. It is not only related to the quality and security of data, but also directly affects the operational efficiency and decision-making ability of enterprises. However, in actual operation, data governance faces many problems and challenges:

[0003] (1) Organizational structure and departmental barriers:

[0004] Data governance requires cross-departmental collaboration, but there may be information islands between different departments, making data sharing and integration difficult. The lack of a unified organizational structure and coordination mechanism makes it difficult for data governance to form a joint force.

[0005] (2) Inconsistent data standards:

[0006] Different departments within an enterprise may use different data standards and formats, which may lead to compatibility issues when integrating data. Inconsistent data standards between industries also increase the difficulty of data exchange between enterprises.

[0007] (3) Difficulties in technology selection and implementation:

[0008] Choosing the right data governance tools and technology platform is a complex process that requires consideration of multiple factors such as cost, ease of use, and scalability. Various unexpected problems may occur during the implementation of technology, such as system compatibility and data migration.

[0009] (4) Data quality management:

[0010] High-quality data is one of the core goals of data governance, but it is challenging to achieve. Data cleaning, verification, and updating require continuous investment of resources, and the results may not be immediate.

[0011] (5) User privacy and security protection:

[0012] With the frequent occurrence of data leakage incidents, users' awareness of protecting personal privacy has been strengthened. How to effectively protect user privacy and data security while ensuring data availability is a problem that data governance must face.

[0013] In addition, data governance at the current stage still has problems such as data quality, inconsistent data standards, data security risks, mismatch between technology and business needs, organizational fragmentation and duplicate management.

[0014] In order to solve the above problems, the present invention proposes a data governance method and system based on a big model, aiming to meet the data governance challenges in the big data era and improve the efficiency, quality and intelligence level of data governance. Summary of the invention

[0015] In order to remedy the deficiencies of the prior art, the present invention provides a simple and efficient data governance method and system based on a large model.

[0016] The present invention is achieved through the following technical solutions:

[0017] A data governance method based on a big model includes the following steps:

[0018] Step S1: Data collection

[0019] Automatically collect data from multiple data sources and perform preliminary cleaning and preprocessing of the data;

[0020] Step S2: Data aggregation

[0021] The preprocessed data is stored in a data storage module, and efficient data access and retrieval functions are provided through the data storage module;

[0022] Use big models to conduct in-depth analysis, mining, and prediction of stored data, as well as data conversion and formatting based on business needs;

[0023] Step S3: Data governance

[0024] Develop and implement data quality standards, data security policies and data management processes based on business needs to ensure data accuracy, integrity and security;

[0025] Step S4: Data quality monitoring

[0026] Monitor data quality and data flow in real time. If any data quality problems are found, feedback will be given to the big model so that the big model can solve the data quality problems in a timely manner.

[0027] In step S2, a deep learning model or a machine learning model is used to automatically identify patterns, relationships and anomalies in the data, and perform intelligent classification, labeling and recommendation.

[0028] In step S3, data access permissions are set according to user roles and business requirements to ensure compliance with data usage.

[0029] In step S3, the data management process includes the following steps:

[0030] Step S3.1: Data cleaning and integration based on large models

[0031] Use natural language processing and pattern recognition technology to automatically identify and correct errors and outliers in data, and improve data accuracy and consistency;

[0032] At the same time, the data sources are cleaned, integrated and verified to ensure that the data input into the model is accurate and reliable, providing a solid foundation for subsequent data analysis and mining;

[0033] Step S3.2: Data standard management based on large model

[0034] Automatically formulate data standards suitable for the industry based on existing standards, improvement requirements and external standards; gradually empower peripheral systems or manage existing data in accordance with the enterprise's data standard implementation strategy, and automatically recommend / match appropriate data standards for them in batches;

[0035] Step S3.3: Metadata management based on large models

[0036] In the metadata collection process, the big model automatically fills in other core metadata information based on the most basic technical metadata information and business sample data; based on the big model, it realizes heterogeneous lineage analysis of multi-source data and realizes intelligent metadata management; using the code interpretation ability of the big model, it improves the success rate and accuracy of lineage analysis in complex code, heterogeneous databases and multi-type processing script scenarios;

[0037] Step S3.4: Data quality management based on large models

[0038] By analyzing the data lineage links and quality verification results, the source of data quality anomalies can be automatically located, and anomalies in the data can be automatically repaired based on normal business sample data;

[0039] Step S3.5: Data security management based on large models

[0040] Automatically identify the sensitivity level of data objects based on the metadata of data objects, business sample data, and the enterprise's data classification strategy; automatically recommend / match desensitization or encryption rules for data objects based on the sensitivity level and data security strategy;

[0041] Step S3.6: Data visualization and analysis based on large models

[0042] Through the analysis and prediction capabilities of large models, we can more accurately grasp the changing trends of data and business needs; at the same time, generate intuitive and easy-to-understand charts and reports to help users better understand the data and provide strong support for business decision-making and strategic planning.

[0043] A data governance system based on a big model, comprising:

[0044] The data collection module is responsible for automatically collecting data from multiple data sources and performing preliminary cleaning and preprocessing of the data;

[0045] The data storage module is responsible for storing preprocessed data and providing efficient data access and retrieval functions;

[0046] The data processing module is responsible for using large models to conduct in-depth analysis, mining, and prediction of stored data, as well as data conversion and formatting according to business needs;

[0047] The data governance module is responsible for formulating and implementing data quality standards, data security policies and data management processes based on business needs to ensure the accuracy, integrity and security of data.

[0048] The data processing module uses a deep learning model or a machine learning model to automatically identify patterns, relationships and anomalies in the data, and perform intelligent classification, labeling and recommendation.

[0049] The data governance module also includes a data permission management function, which is used to set data access permissions based on user roles and business needs to ensure the compliance of data use.

[0050] It also includes a data quality monitoring module, which is used to monitor data quality and data flow in real time, and calculate and feedback to the big model when data quality problems are found, so that the big model can solve data quality problems in a timely manner.

[0051] A data governance device based on a large model, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0052] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0053] The beneficial effects of the present invention are: the data governance method and system based on big models realize the intelligence and automation of data governance, reduce the burden of manual operations, improve data governance efficiency, optimize data security and compliance, promote data governance innovation and development, realize intelligent decision-making support, and enhance data value mining capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Attached Figure 1 This is a schematic diagram of the data governance method based on a big model of the present invention.

[0056] Attached Figure 2 This is a schematic diagram of the data governance system architecture based on the big model of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0058] The data governance method based on the big model includes the following steps:

[0059] Step S1: Data collection

[0060] Automatically collect data from multiple data sources and perform preliminary cleaning and preprocessing of the data;

[0061] Step S2: Data aggregation

[0062] The preprocessed data is stored in a data storage module, and efficient data access and retrieval functions are provided through the data storage module;

[0063] Use big models to conduct in-depth analysis, mining, and prediction of stored data, as well as data conversion and formatting based on business needs;

[0064] Step S3: Data governance

[0065] Develop and implement data quality standards, data security policies and data management processes based on business needs to ensure data accuracy, integrity and security;

[0066] Step S4: Data quality monitoring

[0067] Monitor data quality and data flow in real time. If any data quality problems are found, feedback will be given to the big model so that the big model can solve the data quality problems in a timely manner.

[0068] In step S2, a deep learning model or a machine learning model is used to automatically identify patterns, relationships and anomalies in the data, and perform intelligent classification, labeling and recommendation.

[0069] The big model uses deep learning and natural language technology to automatically identify sensitive data, intelligently review data usage behavior, and has flexible and scalable features. Centralized data management and coordination, cross-departmental collaboration and information sharing can solve existing problems in data governance at this stage.

[0070] In step S3, data access permissions are set according to user roles and business requirements to ensure compliance with data usage.

[0071] In step S3, the data management process includes the following steps:

[0072] Step S3.1: Data cleaning and integration based on large models

[0073] Use natural language processing and pattern recognition technology to automatically identify and correct errors and outliers in data, and improve data accuracy and consistency;

[0074] At the same time, the data sources are cleaned, integrated and verified to ensure that the data input into the model is accurate and reliable, providing a solid foundation for subsequent data analysis and mining;

[0075] Step S3.2: Data standard management based on large model

[0076] Automatically formulate data standards suitable for the industry based on existing standards, improvement requirements and external standards; gradually empower peripheral systems or manage existing data in accordance with the enterprise's data standard implementation strategy, and automatically recommend / match appropriate data standards for them in batches;

[0077] Step S3.3: Metadata management based on large models

[0078] In the metadata collection process, the big model automatically fills in other core metadata information based on the most basic technical metadata information and business sample data; based on the big model, it realizes heterogeneous lineage analysis of multi-source data and realizes intelligent metadata management; using the code interpretation ability of the big model, it improves the success rate and accuracy of lineage analysis in complex code, heterogeneous databases and multi-type processing script scenarios;

[0079] Step S3.4: Data quality management based on large models

[0080] By analyzing the data lineage links and quality verification results, the source of data quality anomalies can be automatically located, and anomalies in the data can be automatically repaired based on normal business sample data;

[0081] Step S3.5: Data security management based on large models

[0082] Automatically identify the sensitivity level of data objects based on the metadata of data objects, business sample data, and the enterprise's data classification strategy; automatically recommend / match desensitization or encryption rules for data objects based on the sensitivity level and data security strategy;

[0083] Step S3.6: Data visualization and analysis based on large models

[0084] Through the analysis and prediction capabilities of large models, we can more accurately grasp the changing trends of data and business needs; at the same time, generate intuitive and easy-to-understand charts and reports to help users better understand the data and provide strong support for business decision-making and strategic planning.

[0085] The data governance system based on the big model includes:

[0086] The data collection module is responsible for automatically collecting data from multiple data sources and performing preliminary cleaning and preprocessing of the data;

[0087] The data storage module is responsible for storing preprocessed data and providing efficient data access and retrieval functions;

[0088] The data processing module is responsible for using large models to conduct in-depth analysis, mining, and prediction of stored data, as well as data conversion and formatting according to business needs;

[0089] The data governance module is responsible for formulating and implementing data quality standards, data security policies and data management processes based on business needs to ensure the accuracy, integrity and security of data.

[0090] The data processing module uses a deep learning model or a machine learning model to automatically identify patterns, relationships and anomalies in the data, and perform intelligent classification, labeling and recommendation.

[0091] The data governance module also includes a data permission management function, which is used to set data access permissions based on user roles and business needs to ensure the compliance of data use.

[0092] It also includes a data quality monitoring module, which is used to monitor data quality and data flow in real time, and calculate and feedback to the big model when data quality problems are found, so that the big model can solve data quality problems in a timely manner.

[0093] Example

[0094] This big model-based data governance system supports cross-organizational and cross-domain data sharing and collaboration, and can exchange and integrate data with other systems.

[0095] Step 1: Access large model capabilities:

[0096] Step 1.1: The data governance platform is connected to the big model capability and uses the big model capability to empower data governance.

[0097] Step 2: Data Governance:

[0098] Step 2.1: After data aggregation is completed, data cleaning and integration are performed on the data governance platform through natural language processing, data labeling, etc.

[0099] Step 2.2: Access national data standards, industry data standards, and local data standards, and intelligently formulate data standards. When citing data standards, the big model will automatically update the data standards.

[0100] Step 2.3: Define data quality rules, accurately locate data anomalies through large models, and automatically repair data anomalies.

[0101] Step 2.4: Implement heterogeneous lineage analysis of multi-source data based on the big model to achieve intelligent metadata management.

[0102] Step 2.5: Through the analysis and prediction capabilities of the big model, accurately grasp the changing trends of data and business needs, and generate intuitive and easy-to-understand charts and reports to support intelligent decision-making.

[0103] Step 3: Data Security:

[0104] Step 3.1: Automatically identify the sensitivity level of data objects throughout the data governance process.

[0105] Step 3.2: Based on the sensitivity level and data security policy, automatically recommend / match desensitization or encryption rules for data objects to achieve automatic data desensitization / encryption.

[0106] The data governance device based on the big model includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0107] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.

[0108] Compared with the existing technology, this big model-based data governance method and system uses big model technology in data governance construction, empowers data governance through big model analysis and computing capabilities, realizes the intelligence and automation of data governance, improves data governance efficiency, enhances data governance accuracy, optimizes data security and compliance, promotes data governance innovation and development, realizes intelligent decision support, and improves data value mining capabilities. The above-mentioned performance improvement not only improves the efficiency and quality of data governance, but also enhances the consistency and standardization of data, optimizes data security and privacy protection, and improves decision support and business insight capabilities. At the same time, through cross-organizational data sharing and collaboration, it promotes the circulation and use of data, provides strong support for the digital transformation and sustainable development of enterprises, and reduces data governance costs.

[0109] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A data governance method based on a big model, characterized by: The following steps are involved: Step S1: Data collection Automatically collect data from data sources and perform preliminary cleaning and preprocessing on the data; Step S2: Data aggregation The preprocessed data is stored in a data storage module, and efficient data access and retrieval functions are provided through the data storage module; Use big models to conduct in-depth analysis, mining, and prediction of stored data, as well as data conversion and formatting based on business needs; Step S3: Data governance Develop and implement data quality standards, data security policies and data management processes based on business needs to ensure data accuracy, integrity and security; Step S4: Data quality monitoring Monitor data quality and data flow in real time. If any data quality problems are found, feedback will be given to the big model so that the big model can solve the data quality problems in a timely manner.

2. The data governance method based on a large model according to claim 1 is characterized in that: In step S2, a deep learning model or a machine learning model is used to automatically identify patterns, relationships and anomalies in the data, and perform intelligent classification, labeling and recommendation.

3. The data governance method based on a large model according to claim 1 is characterized in that: In step S3, data access permissions are set according to user roles and business requirements to ensure compliance with data usage.

4. The data governance method based on a large model according to claim 1 is characterized in that: In step S3, the data management process includes the following steps: Step S3.1: Data cleaning and integration based on large models Use natural language processing and pattern recognition technology to automatically identify and correct errors and outliers in data, and improve data accuracy and consistency; At the same time, the data sources are cleaned, integrated and verified to ensure that the data input into the model is accurate and reliable; Step S3.2: Data standard management based on large model Automatically formulate data standards suitable for the industry based on existing standards, improvement requirements and external standards; gradually empower peripheral systems or manage existing data in accordance with the enterprise's data standard implementation strategy, and automatically recommend / match appropriate data standards for them in batches; Step S3.3: Metadata management based on large models In the metadata collection process, the big model automatically fills in other core metadata information based on the most basic technical metadata information and business sample data; based on the big model, it realizes heterogeneous lineage analysis of multi-source data and realizes intelligent metadata management; using the code interpretation ability of the big model, it improves the success rate and accuracy of lineage analysis in complex code, heterogeneous databases and multi-type processing script scenarios; Step S3.4: Data quality management based on large models By analyzing the data lineage links and quality verification results, the source of data quality anomalies can be automatically located, and anomalies in the data can be automatically repaired based on normal business sample data; Step S3.5: Data security management based on large models Automatically identify the sensitivity level of data objects based on the metadata of data objects, business sample data, and the enterprise's data classification strategy; automatically recommend / match desensitization or encryption rules for data objects based on the sensitivity level and data security strategy; Step S3.6: Data visualization and analysis based on large models Through the analysis and prediction capabilities of large models, we can understand the changing trends of data and business needs; at the same time, generate intuitive charts and reports to help users better understand the data.

5. A data governance system based on a big model, characterized by: include: The data collection module is responsible for automatically collecting data from the data source and performing preliminary cleaning and preprocessing of the data; The data storage module is responsible for storing preprocessed data and providing efficient data access and retrieval functions; The data processing module is responsible for using large models to conduct in-depth analysis, mining, and prediction of stored data, as well as data conversion and formatting according to business needs; The data governance module is responsible for formulating and implementing data quality standards, data security policies and data management processes based on business needs to ensure the accuracy, integrity and security of data.

6. The data governance system based on a large model according to claim 5 is characterized in that: The data processing module uses a deep learning model or a machine learning model to automatically identify patterns, relationships and anomalies in the data, and perform intelligent classification, labeling and recommendation.

7. The data governance system based on a large model according to claim 5, characterized in that: The data governance module also includes a data permission management function, which is used to set data access permissions based on user roles and business needs to ensure the compliance of data use.

8. The data governance system based on a large model according to claim 5, characterized in that: It also includes a data quality monitoring module, which is used to monitor data quality and data flow in real time, and calculate and feedback to the big model when data quality problems are found, so that the big model can solve data quality problems in a timely manner.

9. A data management device based on a large model, characterized in that: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 6 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images