Multi-modal large model data set construction method and system based on data governance

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a multi-modal large model data set construction method and system based on data governance, and the method comprises the following steps: data source collection: collecting various multi-modal data sources related to the industry; data governance: carrying out data governance work such as cleaning, conversion and labeling on the collected data to form a high-quality data set; large model training: performing large model training in combination with the large model data engineering capability and the base model capability; in the large model training process, an industry knowledge base and an industry large model base are constructed; iteratively updating the data set; the beneficial effects are that the multi-modal large model data set construction method and system based on data governance can effectively collect, process, analyze and manage multi-modal data from different channels and different types, and utilize the data to train a large model with industry characteristics; therefore, the system can meet the requirements of industry knowledge management, intelligent analysis, decision support and the like.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to a method and system for constructing a multimodal large model dataset based on data governance. Background Art

[0002] With the rapid development of generative artificial intelligence, large multimodal models such as large language models, text-based graphs, and video generation continue to emerge and are widely used in various industries.

[0003] However, in the process of promoting big model projects and products, it is found that the existing model often ignores the accumulation and use of high-value data resources when using big models to empower industries. These data resources are crucial for the in-depth application of big models in specific industries. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for constructing a multimodal large model dataset based on data governance to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for constructing a multimodal large model dataset based on data governance, the method comprising the following steps:

[0006] Data source collection: collect various multimodal data sources related to the industry;

[0007] Data governance: cleaning, converting, labeling and other data governance work on the collected data to form a high-quality data set;

[0008] Large model training: combining large model data engineering capabilities with base model capabilities to train large models;

[0009] During the large model training process, build an industry knowledge base and industry large model library;

[0010] The data set is iteratively updated, and the industry subject data warehouse, high-quality large model data set, industry knowledge base and industry large model library are iteratively updated.

[0011] Preferably, the specific operations of data source collection include:

[0012] Multimodal data is collected from multiple data sources through API interfaces, database connections, sensors and monitoring equipment, and data cleaning, conversion and integration are performed. A special preprocessing process is designed for each modal data, such as word segmentation and sentiment analysis of text, denoising and feature extraction of images, key frame extraction and action recognition of videos, to ensure that each modal data is effectively processed and the most representative features are extracted.

[0013] Preferably, the specific operations of data governance include:

[0014] Ensure data security through data quality assessment, data encryption, and access control technologies, implement data lifecycle management, conduct deep cleaning, conversion, and standardization of data, and improve data quality, including preprocessing and labeling of unstructured data.

[0015] Optimally, for large model training, a distributed computing cloud that supports massive data processing is built to provide efficient and stable infrastructure to meet the high computing power requirements of data set construction and model training scenarios;

[0016] Build an algorithm platform that includes a general large language model, a computer vision model, and a multimodal fusion model to support extracting knowledge from data, identifying patterns, making predictions and decisions, and enhancing the intelligence level of the model; during the training process, introduce a joint optimization strategy so that the model can promote mutual understanding and synergy between modalities while learning the characteristics of a single modality, thereby improving the model's depth of understanding of multimodal inputs; the adaptive learning mechanism can adjust model parameters and optimize fusion strategies according to the characteristics of new data to adapt to multimodal data analysis needs in different scenarios.

[0017] Preferably, the specific operations of iteratively updating the data set include:

[0018] Based on the governed high-quality data, we build a multimodal large model data set, and continuously iterate and update the data warehouse, model library, and knowledge base as the application scenarios expand and the needs change to ensure the timeliness and effectiveness of the data set. We establish a data quality feedback mechanism, and use the model training effect as part of the data quality assessment to reversely guide the adjustment of the data processing strategy. By continuously iterating and optimizing the data processing process, we ensure that the quality of the data set is gradually improved and the model performance is enhanced.

[0019] A multimodal large model dataset construction system based on data governance, the system comprising:

[0020] Data source collection module, which collects various multimodal data sources related to the industry;

[0021] The data governance module cleans, converts, and annotates the collected data to form a high-quality data set;

[0022] The large model training module combines the large model data engineering capabilities with the base model capabilities to perform large model training;

[0023] Industry knowledge base and large model library construction module: During the large model training process, the industry knowledge base and industry large model library are constructed;

[0024] The dataset iteration update module iteratively updates the industry subject data warehouse, high-quality large model dataset, industry knowledge base and industry large model library.

[0025] Preferably, the data source collection module collects multimodal data from multiple data sources through API interfaces, database connections, sensors and monitoring equipment collection methods, and performs data cleaning, conversion and integration; a special preprocessing process is designed for each modal data, such as word segmentation and sentiment analysis of text, denoising and feature extraction of images, key frame extraction and action recognition of videos, to ensure that each modal data is effectively processed and the most representative features are extracted.

[0026] Preferably, the data governance module ensures data security through data quality assessment, data encryption, and access control technical means, and implements data lifecycle management to deeply clean, convert, and standardize data to improve data quality, including preprocessing and labeling of unstructured data.

[0027] Optimally, the large model training module builds a distributed computing cloud that supports massive data processing, provides efficient and stable infrastructure, and meets the high computing power requirements of data set construction and model training scenarios;

[0028] Build an algorithm platform that includes a general large language model, a computer vision model, and a multimodal fusion model to support extracting knowledge from data, identifying patterns, making predictions and decisions, and enhancing the intelligence level of the model; during the training process, introduce a joint optimization strategy so that the model can promote mutual understanding and synergy between modalities while learning the characteristics of a single modality, thereby improving the model's depth of understanding of multimodal inputs; the adaptive learning mechanism can adjust model parameters and optimize fusion strategies according to the characteristics of new data to adapt to multimodal data analysis needs in different scenarios.

[0029] Preferably, the dataset iteration update module builds a multimodal large model dataset based on the governed high-quality data, and continuously iterates and updates the data warehouse, model library and knowledge base as the application scenarios expand and the needs change, to ensure the timeliness and effectiveness of the dataset; establishes a data quality feedback mechanism, takes the model training effect as part of the data quality assessment, and reversely guides the adjustment of the data processing strategy, and ensures that the dataset quality is gradually improved and the model performance is enhanced by continuously iterating and optimizing the data processing process.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The method and system for constructing a multimodal large model dataset based on data governance proposed in the present invention effectively collect, process, analyze and manage multimodal data (including structured data, text, images, videos, etc.) from different channels and different types, and use these data to train a large model with industry characteristics, thereby serving the needs of industry knowledge management, intelligent analysis, decision support, etc. This method and system not only focuses on the comprehensiveness and diversity of data, but also emphasizes the improvement of data quality. Through data cleaning, conversion, annotation and other governance steps, the availability and accuracy of the data set are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a structural schematic diagram and method flow chart of the present invention;

[0033] Figure 2 This is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] For example, see Figure 1 The present invention provides a technical solution: a method for constructing a multimodal large model dataset based on data governance, the method comprising the following steps:

[0036] Data source collection: collect various multimodal data sources related to the industry; collect multimodal data from multiple data sources through API interfaces, database connections, sensors and monitoring equipment collection methods, and perform data cleaning, conversion and integration; design special preprocessing processes for each modal data, such as text segmentation and sentiment analysis, image denoising and feature extraction, video key frame extraction and action recognition, to ensure that each modal data is effectively processed and the most representative features are extracted.

[0037] Data governance involves cleaning, converting, and labeling the collected data to form high-quality data sets; ensuring data security through data quality assessment, data encryption, and access control technologies, and implementing data lifecycle management to deeply clean, convert, and standardize data to improve data quality, including preprocessing and labeling of unstructured data.

[0038] Large model training: Combine large model data engineering capabilities with base model capabilities to train large models; build a distributed computing cloud that supports massive data processing, provide efficient and stable infrastructure, and meet the high computing power requirements of data set construction and model training scenarios;

[0039] Build an algorithm platform that includes a general large language model, a computer vision model, and a multimodal fusion model to support extracting knowledge from data, identifying patterns, making predictions and decisions, and enhancing the intelligence level of the model; during the training process, introduce a joint optimization strategy so that the model can promote mutual understanding and synergy between modalities while learning the characteristics of a single modality, thereby improving the model's depth of understanding of multimodal inputs; the adaptive learning mechanism can adjust model parameters and optimize fusion strategies according to the characteristics of new data to adapt to multimodal data analysis needs in different scenarios.

[0040] During the large model training process, build an industry knowledge base and industry large model library;

[0041] Iteratively update the data set, iteratively update the industry theme data warehouse, high-quality large model data set, industry knowledge base and industry large model library, build a multimodal large model data set based on the governed high-quality data, and continuously iteratively update the data warehouse, model library and knowledge base as the application scenarios expand and the needs change to ensure the timeliness and effectiveness of the data set; establish a data quality feedback mechanism, take the model training effect as part of the data quality assessment, reversely guide the adjustment of the data processing strategy, and optimize the data processing process through continuous iteration to ensure that the data set quality is gradually improved and the model performance is enhanced.

[0042] Embodiment 2, based on embodiment 1, proposes a multimodal large model dataset construction system based on data governance, the system comprising:

[0043] The data source collection module collects various types of multimodal data sources related to the industry; then, the data governance module is used to clean, convert, and annotate the collected data to form a high-quality data set; then, the large model training module is used to combine the large model data engineering capabilities and the base model capabilities to perform large model training; during the large model training process, an industry knowledge base and an industry large model library are constructed; finally, with the enrichment of application scenarios and the advancement of the project, the data set iteration update module is used to iteratively update the industry subject data warehouse, high-quality large model data set, industry knowledge base, and industry large model library. The characteristics of the present invention and the specific functions of each module are described as follows:

[0044] 1. Data source collection module: This module collects multimodal data from multiple data sources through API interfaces, database connections, sensors and monitoring equipment collection, and performs data cleaning, conversion and integration. Design a special preprocessing process for each modal data (text, image, video), such as word segmentation and sentiment analysis of text, denoising and feature extraction of images, key frame extraction and action recognition of videos, to ensure that each modal data is effectively processed and the most representative features are extracted.

[0045] 2. Data governance module: Ensure data security through technical means such as data quality assessment, data encryption, access control, and implement data lifecycle management, deeply clean, convert and standardize data, and improve data quality, including preprocessing and labeling of unstructured data (such as images and videos).

[0046] The data quality assessment uses a real-time data quality monitoring system and an integrated data quality rule engine to analyze data in the data stream in real time and detect problems such as outliers, missing values, and inconsistencies. The quality threshold is adaptively adjusted using a machine learning algorithm to adapt to changes in data quality at different stages. Based on the monitoring results, the system automatically starts the data cleaning task and uses preset cleaning rules and self-learning algorithms to denoise the data, fill in missing values, and correct inconsistencies. At the same time, external data sources are introduced for data supplementation to improve data integrity and accuracy.

[0047] 3. Distributed computing power support: Build a distributed computing power cloud that supports massive data processing, provide efficient and stable infrastructure, and meet the high computing power requirements of scenarios such as data set construction and model training.

[0048] 4. Multimodal algorithm platform: Build an algorithm platform including a general large language model, a computer vision model, and a multimodal fusion model to support the extraction of knowledge, identification of patterns, prediction and decision-making from data, and enhance the intelligence level of the model. During the training process, a joint optimization strategy is introduced so that the model can promote mutual understanding and synergy between modalities while learning the characteristics of a single modality, thereby improving the model's depth of understanding of multimodal inputs. The adaptive learning mechanism can adjust model parameters and optimize fusion strategies according to the characteristics of new data to meet the needs of multimodal data analysis in different scenarios.

[0049] 5. Dataset construction and iterative update: Based on the governed high-quality data, build a multimodal large model dataset, and continuously iterate and update the data warehouse, model library and knowledge base as the application scenarios expand and the needs change to ensure the timeliness and effectiveness of the dataset.

[0050] Establish a data quality feedback mechanism, use the model training effect as part of the data quality assessment, and reversely guide the adjustment of the data processing strategy. By continuously iterating and optimizing the data processing process, ensure that the quality of the data set is gradually improved and the model performance is enhanced.

[0051] 6. Evaluation and optimization mechanism: An evaluation module is introduced during the model training and data set construction process. Through a predefined evaluation model, the difference between the model-generated content and the target features is quantified, and the feedback results are used to optimize model training to ensure that the content generated by the model is closer to user needs and industry standards.

[0052] 7. Data encryption and access control method: Before each data packet or data block is stored or transmitted, the system generates a one-time session key for encryption. This session key is then encrypted by the user's public key. Only authorized users can use their private keys to decrypt the session key and access the original data, ensuring the security of the data transmission process;

[0053] Establish a role-based access control (RBAC) model and further refine it to the data element level to ensure that users can only access the minimum necessary data within their responsibilities. Use tag or attribute-based access control (ABAC) to dynamically adjust permissions based on data sensitivity, user identity, time and other factors to achieve flexible and strict data protection;

[0054] Implement comprehensive logging and auditing functions to track all data access behaviors to facilitate monitoring and tracing back potential illegal access. At the same time, establish data protection policies that meet industry standards and regulatory requirements to ensure compliance of system operations.

[0055] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a multimodal large model dataset based on data governance, characterized in that: The method comprises the following steps: Data source collection: collect various multimodal data sources related to the industry; Data governance: cleaning, converting, labeling and other data governance work on the collected data to form a high-quality data set; Large model training: combining large model data engineering capabilities with base model capabilities to train large models; During the large model training process, build an industry knowledge base and industry large model library; The data set is iteratively updated, and the industry subject data warehouse, high-quality large model data set, industry knowledge base and industry large model library are iteratively updated.

2. According to a method for constructing a multimodal large model dataset based on data governance according to claim 1, it is characterized by: The specific operations of data source collection include: Multimodal data is collected from multiple data sources through API interfaces, database connections, sensors and monitoring equipment, and data cleaning, conversion and integration are performed. A special preprocessing process is designed for each modal data, such as word segmentation and sentiment analysis of text, denoising and feature extraction of images, key frame extraction and action recognition of videos, to ensure that each modal data is effectively processed and the most representative features are extracted.

3. The method for constructing a multimodal large model dataset based on data governance according to claim 1 is characterized in that: The specific operations of data governance include: Ensure data security through data quality assessment, data encryption, and access control technologies, implement data lifecycle management, conduct deep cleaning, conversion, and standardization of data, and improve data quality, including preprocessing and labeling of unstructured data.

4. The method for constructing a multimodal large model dataset based on data governance according to claim 1 is characterized in that: Large model training: building a distributed computing cloud that supports massive data processing, providing efficient and stable infrastructure, and meeting the high computing power requirements of data set construction and model training scenarios; Build an algorithm platform that includes a general large language model, a computer vision model, and a multimodal fusion model to support extracting knowledge from data, identifying patterns, making predictions and decisions, and enhancing the intelligence level of the model; during the training process, introduce a joint optimization strategy so that the model can promote mutual understanding and synergy between modalities while learning the characteristics of a single modality, thereby improving the model's depth of understanding of multimodal inputs; the adaptive learning mechanism can adjust model parameters and optimize fusion strategies according to the characteristics of new data to adapt to multimodal data analysis needs in different scenarios.

5. The method for constructing a multimodal large model dataset based on data governance according to claim 1, characterized in that: The specific operations of iterative update of the dataset include: Based on the governed high-quality data, we build a multimodal large model data set, and continuously iterate and update the data warehouse, model library, and knowledge base as the application scenarios expand and the needs change to ensure the timeliness and effectiveness of the data set. We establish a data quality feedback mechanism, and use the model training effect as part of the data quality assessment to reversely guide the adjustment of the data processing strategy. By continuously iterating and optimizing the data processing process, we ensure that the quality of the data set is gradually improved and the model performance is enhanced.

6. A system for constructing a multimodal large model dataset based on data governance according to the method for constructing a multimodal large model dataset based on data governance according to any one of claims 1 to 5, characterized in that: The system comprises: Data source collection module, which collects various multimodal data sources related to the industry; The data governance module cleans, converts, and annotates the collected data to form a high-quality data set; The large model training module combines the large model data engineering capabilities with the base model capabilities to perform large model training; Industry knowledge base and large model library construction module: During the large model training process, the industry knowledge base and industry large model library are constructed; The dataset iteration update module iteratively updates the industry subject data warehouse, high-quality large model dataset, industry knowledge base and industry large model library.

7. A multimodal large model dataset construction system based on data governance according to claim 6, characterized in that: The data source collection module collects multimodal data from multiple data sources through API interfaces, database connections, sensors and monitoring equipment, and performs data cleaning, conversion and integration. It designs a special preprocessing process for each modal data, such as text segmentation and sentiment analysis, image denoising and feature extraction, video key frame extraction and action recognition, to ensure that each modal data is effectively processed and the most representative features are extracted.

8. The multimodal large model dataset construction system based on data governance according to claim 6 is characterized by: The data governance module ensures data security through data quality assessment, data encryption, and access control technologies, and implements data lifecycle management to deeply clean, convert, and standardize data to improve data quality, including preprocessing and labeling of unstructured data.

9. The multimodal large model dataset construction system based on data governance according to claim 6 is characterized by: Large model training module, building a distributed computing cloud that supports massive data processing, providing efficient and stable infrastructure to meet the high computing power requirements of data set construction and model training scenarios; Build an algorithm platform that includes a general large language model, a computer vision model, and a multimodal fusion model to support extracting knowledge from data, identifying patterns, making predictions and decisions, and enhancing the intelligence level of the model; during the training process, introduce a joint optimization strategy so that the model can promote mutual understanding and synergy between modalities while learning the characteristics of a single modality, thereby improving the model's depth of understanding of multimodal inputs; the adaptive learning mechanism can adjust model parameters and optimize fusion strategies according to the characteristics of new data to adapt to multimodal data analysis needs in different scenarios.

10. The multimodal large model dataset construction system based on data governance according to claim 6, characterized in that: The dataset iteration update module builds a multimodal large model dataset based on the governed high-quality data, and continuously iterates and updates the data warehouse, model library, and knowledge base as the application scenarios expand and the needs change to ensure the timeliness and effectiveness of the dataset. It establishes a data quality feedback mechanism, takes the model training effect as part of the data quality assessment, and reversely guides the adjustment of the data processing strategy. By continuously iterating and optimizing the data processing process, it ensures that the quality of the dataset is gradually improved and the model performance is enhanced.

Images