Multi-modal data set construction method and system

Abstract

The invention discloses a multi-modal data set construction method and system, the multi-modal data set comprises structured data and unstructured data, the method comprises the following steps: carrying out offline processing and fusion on the unstructured data to generate a structured knowledge graph; performing off-line mapping processing on the structured data, and simultaneously inputting the non-structured data after off-line processing and the structured data after off-line mapping processing into a preset structured model to generate a database; in combination with the structured knowledge graph and a database, constructing an offline model and training the offline model to obtain a target large language model; and obtaining and analyzing user demands on line, making an analysis plan, executing the analysis plan by using the target large language model, generating a multi-modal data set, and outputting the multi-modal data set to the user. The method supports multi-modal data fusion analysis, has large-scale real-time data processing capability, and can efficiently meet the processing and updating requirements of dynamic data.

Description

Technical Field

[0001] The present invention relates to the field of multimodal data processing, and in particular to a method and system for constructing a multimodal dataset. Background Art

[0002] Traditional insight analysis primarily focuses on structured data and statistical methods. These methods are limited to the data's inherent patterns. However, in real-world scenarios, a vast amount of information is distributed across various modalities and in unstructured data, such as news or business documents. This information can further explain structured data while also providing additional context for insight analysis.

[0003] While large language models provide semantic understanding of multimodal data, their generative approach cannot be directly applied to insight analysis. The main reasons are: 1) Large language models are still limited by the length of context, making them incapable of direct analysis on all data; 2) Continuous training of large language models remains expensive and lacks quality assurance, and updates based on real-time data are not guaranteed. 3) Due to inherent limitations of large models, high accuracy of analysis results cannot be guaranteed. Summary of the Invention

[0004] (1) Purpose of the invention The purpose of the present invention is to provide a method and system for constructing a multimodal data set, which supports multimodal data fusion analysis, has large-scale real-time data processing capabilities, and can efficiently respond to the processing and updating needs of dynamic data.

[0005] (2) Technical solution To solve the above problem, a first aspect of the present invention provides a method for constructing a multimodal dataset, wherein the multimodal dataset includes structured data and unstructured data, and the method includes: Offline processing and fusion of unstructured data to generate structured knowledge graphs; After the structured data is mapped offline, the unstructured data after offline processing and the structured data after offline mapping processing are simultaneously input into the preset structured model to generate a database; Combining the structured knowledge graph and the database, building an offline model and training it to obtain a target large language model; Obtain and analyze user needs online, formulate an analysis plan, execute the analysis plan using the target large language model, generate a multimodal dataset and output it to the user.

[0006] Furthermore, the offline processing and fusion of unstructured data to generate a structured knowledge graph includes: Extracting the pictures and tables using optical character recognition technology, and converting the extraction results into non-text data features; wherein the unstructured data includes pictures and tables; Performing text analysis on the non-text data features, extracting concepts and themes, and generating text data features; The non-text data features and the text data features are fused to generate a structured knowledge graph.

[0007] Furthermore, the structured data is subjected to offline mapping processing using traditional machine learning algorithms, including decision trees, support vector machines, and random forests.

[0008] Furthermore, combining the structured knowledge graph and the database to construct an offline model and train it to obtain a target large language model includes: Converting the structured knowledge graph and the database into the same tensor, and dividing the converted tensor data into a training set, a validation set, and a test set; Building an offline model based on the tensor data and initializing the offline model; Inputting the training set data into the offline model to obtain a prediction result, and calculating a loss value based on the prediction result; Based on the loss value, the gradient of the parameters of each layer in the offline model is calculated by the chain rule, and the parameters are optimized and updated in combination with the validation set, and iterative training is continued; The trained offline model is evaluated using the test set to obtain the target large language model.

[0009] Furthermore, converting the extraction results into non-text data features includes: using a deep learning algorithm to extract image features of the extraction results as non-text data features, and the deep learning algorithm includes: a convolutional neural network, a recurrent neural network and a converter.

[0010] Furthermore, OCR is used to identify and extract concepts and themes of the non-text data features to generate text data features.

[0011] Furthermore, the fusing of the non-text data features and the text data features to generate a structured knowledge graph includes: Calculating semantic similarity between the non-text data feature and the text data feature; generating a cross-modal adjacency matrix based on the semantic similarity; A dynamic graph convolution layer is used to perform convolution operations on the corresponding nodes in the cross-modal adjacency matrix to obtain a structured knowledge graph.

[0012] Furthermore, the calculation formula of the convolution operation is as follows: ; in is the node feature matrix of the l+1th layer, is the adjacency matrix after preprocessing, For the The feature matrix of the layer, , for The degree matrix of For the The weight matrix of the layer, is the activation function.

[0013] A second aspect of the present invention provides a system for constructing a multimodal dataset, wherein the multimodal dataset includes structured data and unstructured data. The system includes an offline module and an online module: The offline module is used to: perform offline processing and fusion of unstructured data to generate a structured knowledge graph; perform offline mapping processing on structured data, and simultaneously input the unstructured data after offline processing and the structured data after offline mapping processing into a preset structured model to generate a database; combine the structured knowledge graph and the database to build an offline model and train it to obtain a target large language model; The online module is used to: acquire and analyze user needs online, formulate an analysis plan, use the target large language model to execute the analysis plan, and obtain multimodal data; acquire and analyze user needs online, formulate an analysis plan, use the target large language model to execute the analysis plan, generate a multimodal data set and output it to the user.

[0014] Further: using optical character recognition technology to extract the pictures and tables, and converting the extraction results into non-text data features; performing text analysis on the non-text data features, extracting concepts and themes, and generating text data features; fusing the non-text data features with the text data features to generate a structured knowledge graph; wherein the unstructured data includes pictures and tables.

[0015] (3) Beneficial effects The above-mentioned technical solution of the present invention has the following beneficial technical effects: In an enterprise environment, a large amount of knowledge exists in an unstructured form. Therefore, the present invention provides a method and system for constructing a multimodal dataset. The method processes unstructured data and structured data separately through offline processing, wherein the unstructured data is processed and fused offline to generate a structured knowledge graph, and the structured data is mapped offline. The offline processed unstructured data and the structured data after offline mapping are simultaneously input into a preset structured model to generate a database. This processing method can apply the semantic information in the unstructured data to the analysis of the structured data, which can further improve the effectiveness and reliability of the insight analysis of the data. In combination with the structured knowledge graph and the database, an offline model is constructed and trained to obtain a target large language model. The online inference model is a lightweight version or functional subset of the offline training model. During online inference, the core inference logic stored in the offline training model (such as feature mapping rules and semantic association weights) is called, and the parameters trained by the offline model (such as the encoding matrix of text features and cross-modal attention weights) are used to quickly calculate the output results without retraining, thereby achieving efficient inference. This method supports multimodal data fusion analysis, can achieve correlation insights between structured and unstructured data (such as topic association, abnormal pattern recognition, etc.), has large-scale real-time data processing capabilities, and can efficiently respond to the processing and updating needs of dynamic data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a method for constructing a multimodal dataset of the present invention; Figure 2 is a schematic diagram of a system for constructing a multimodal dataset according to the present invention; Figure 3 It is a specific embodiment corresponding to the construction method of the present invention. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0018] like Figure 1 As shown, the first aspect of the present invention provides a method for constructing a multimodal dataset, wherein the multimodal dataset includes structured data and unstructured data. The method mainly adopts two methods: offline processing and online processing. The method includes: S1, processes and integrates unstructured data offline to generate structured knowledge graphs, combined with Figure 3 As shown, specifically including: S11, extracting the images and tables using optical character recognition technology, and converting the extraction results into non-text data features; wherein the unstructured data includes images and tables. A deep learning algorithm is used to extract image features from the extraction results as non-text data features. The deep learning algorithm includes: a convolutional neural network, a recurrent neural network, and a converter. In addition, the unstructured data also includes audio and video, and non-text information is extracted from the audio and video using natural language recognition technology. For example, CNN is mainly used for feature extraction of image non-text data (such as extracting edge and texture features of images through convolutional layers); RNN (including LSTM and GRU) is used to process time-series text data after audio and video conversion to capture contextual semantic relationships; for tabular data, a table structure-aware encoder is used to first parse the row and column structure of the table (such as identifying the relationship between table headers and cells), then extract text features of the cell content, and ultimately generate table structured features containing structural information and content features.

[0019] S12: Perform text analysis on the non-text data features to extract concepts and themes, generating text data features. Concepts and themes extracted from the non-text data features are identified using optical character recognition (OCR) to generate text data features. Transformers (such as the BERT model) are used for semantic analysis of text data to generate deep text feature vectors.

[0020] S13, fusing the non-text data features with the text data features to generate a structured knowledge graph, specifically including: S131, calculating the semantic similarity between the non-text data feature and the text data feature. For example, the similarity between text and image can be calculated by the cosine similarity between the text feature vector (text data feature) and the image feature vector, and the similarity between table and text can be calculated by the matching degree between the table content feature and the text feature. S132, generating a cross-modal adjacency matrix according to the semantic similarity; S133, using a dynamic graph convolution layer to perform a convolution operation on the corresponding nodes in the cross-modal adjacency matrix to obtain a structured knowledge graph. The calculation formula of the convolution operation is as follows: ; (1) in is the node feature matrix of the l+1th layer, is the adjacency matrix after preprocessing, For the The feature matrix of the layer, , for The degree matrix of For the The weight matrix of the layer, is the activation function.

[0021] S2: Perform offline mapping processing on the structured data. Both the unstructured data and the structured data after offline mapping are simultaneously input into a pre-set structured model to generate a database. This step utilizes traditional machine learning algorithms, such as decision trees, support vector machines, and random forests, to perform offline mapping processing on the structured data. These algorithms are primarily used for preliminary classification and feature screening of structured data. In this step, they are used to classify and map the structured data, assist in building a knowledge base, and are used for feature selection in the early stages of model training, thereby improving the training efficiency of subsequent deep learning models.

[0022] S3, combining the structured knowledge graph and the database, building an offline model and training it to obtain a target large language model, specifically including: S31, converting the structured knowledge graph and the database into the same tensor, and dividing the converted tensor data into a training set, a validation set, and a test set. Based on the structured knowledge graph and the database, convert text, images, tables, and other data into tensors in a unified format, for example, converting text into a sequence of word vectors, images into a pixel matrix, and tables into a structured feature matrix.

[0023] S32: Build an offline model based on the tensor data and initialize the offline model, initialize the weight parameters of the neural network (such as using Xavier initialization), and set hyperparameters such as the number of network layers and activation function (such as ReLU, Softmax).

[0024] S33: Input the training set data into the offline model to obtain a prediction result, and calculate a loss value based on the prediction result. Input the training set data into the offline model, and obtain a prediction result (such as the topic label of the text and the matching probability between the image and the text) through calculations at each layer (such as convolution and pooling of CNN and time series iteration of RNN). Calculate the loss value based on the predicted result and the true label. For example, the loss value is calculated based on the predicted result and the true label. For text classification tasks, the loss value is calculated using cross entropy loss.

[0025] S34, based on the loss value, calculating the gradient of the parameters of each layer in the offline model by the chain rule, optimizing and updating the parameters in combination with the validation set, and continuing iterative training until the model converges; S35, evaluate the trained offline model through the test set to obtain the target large language model. Test and evaluate the accuracy and effect of the trained model, and judge whether the model meets the expectations or requirements based on the evaluation results. If it meets the requirements, the training is terminated to obtain the target large language model. Finally, the adjusted target large language model is stored in the model library of the system. The offline training model is preprocessed, feature extracted, classified and predicted, as well as evaluated and parameter adjusted, and stored to form a data set, and is used for online reasoning analysis; the offline training model is trained in a multi-task manner, that is, based on the same data set, multiple categories are trained and learned at the same time; and it is called through the semantic analysis module of online processing to complete the analysis.

[0026] S4, online acquisition and analysis of user needs, formulation of an analysis plan, implementation of the analysis plan using the target large language model, generation of a multimodal dataset, and output to the user. The S4 step specifically includes: 1) identifying the intent in the user's statement; 2) locating the subject of the user's statement based on the identified intent, i.e., locating the data involved in the analysis request, forming an analysis dataset, and cleaning and processing the relevant data; 3) determining the analysis method based on the intent and subject, formulating an analysis process, and submitting the dataset to the target large language model for analysis and calculation; 4) formatting the results to form a result set, returning it to the user, and visually displaying it through the user interface.

[0027] In addition, if Figure 2 As shown, the second aspect of the present invention provides a system for constructing a multimodal dataset, wherein the multimodal dataset includes structured data and unstructured data. The system includes an offline module 21 and an online module 22: The offline module 21 is used to: perform offline processing and fusion of unstructured data to generate a structured knowledge graph; perform offline mapping processing on structured data, and simultaneously input the unstructured data after offline processing and the structured data after offline mapping processing into a preset structured model to generate a database; combine the structured knowledge graph and the database to build an offline model and train it to obtain a target large language model; The online module 22 is used to: acquire and analyze user needs online, formulate an analysis plan, execute the analysis plan using the target large language model, and obtain multimodal data; acquire and analyze user needs online, formulate an analysis plan, execute the analysis plan using the target large language model, generate a multimodal data set and output it to the user.

[0028] Furthermore, the offline module 21 is also used to: extract the pictures and tables using optical character recognition technology, and convert the extraction results into non-text data features; perform text analysis on the non-text data features, extract concepts and themes, and generate text data features; and fuse the non-text data features with the text data features to generate a structured knowledge graph; wherein the unstructured data includes pictures and tables.

[0029] It should be understood that the above-mentioned specific embodiments of the present invention are merely used to illustrate or explain the principles of the present invention and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the scope of protection of the present invention. In addition, the claims attached to the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries. Those of ordinary skill in the art will understand that all or part of the steps in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the following process of the embodiments of the above-mentioned methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM). The steps in the method of the embodiment of the present invention can be adjusted in order, merged and deleted according to actual needs. The modules in the system of the embodiment of the present invention can be merged, divided and deleted according to actual needs.

Claims

1. A method for constructing a multimodal dataset, characterized in that: The multimodal dataset includes structured data and unstructured data, and the method includes: Offline processing and fusion of unstructured data to generate structured knowledge graphs; Perform offline mapping processing on structured data, input the unstructured data after offline processing and the structured data after offline mapping processing into the preset structured model at the same time to generate a database; Combining the structured knowledge graph and the database, building an offline model and training it to obtain a target large language model; Obtain and analyze user needs online, formulate an analysis plan, execute the analysis plan using the target large language model, generate a multimodal dataset and output it to the user.

2. The method for constructing a multimodal dataset according to claim 1, wherein: The offline processing and fusion of unstructured data to generate a structured knowledge graph includes: Extracting images and tables using optical character recognition technology, and converting the extraction results into non-text data features; wherein the unstructured data includes images and tables; Performing text analysis on the non-text data features, extracting concepts and themes, and generating text data features; The non-text data features and the text data features are fused to generate a structured knowledge graph.

3. The method for constructing a multimodal dataset according to claim 1, wherein: The structured data is mapped offline using traditional machine learning algorithms, including decision trees, support vector machines, and random forests.

4. The method for constructing a multimodal dataset according to claim 1, wherein: The combining of the structured knowledge graph and the database to construct an offline model and train the model to obtain a target large language model includes: Converting the structured knowledge graph and the database into the same tensor, and dividing the converted tensor data into a training set, a validation set, and a test set; Building an offline model based on the tensor data and initializing the offline model; Inputting the training set into the initialized offline model to obtain a prediction result, and calculating a loss value based on the prediction result; Based on the loss value, the gradient of the parameters of each layer in the offline model is calculated by the chain rule, and the parameters of each layer are optimized and updated in combination with the validation set, and iterative training is continued; The trained offline model is evaluated using the test set to obtain the target large language model.

5. The method for constructing a multimodal dataset according to claim 2, wherein: The converting of the extraction results into non-text data features includes: extracting image features of the extraction results as non-text data features using a deep learning algorithm, wherein the deep learning algorithm includes: a convolutional neural network, a recurrent neural network and a converter.

6. The method for constructing a multimodal dataset according to claim 2, wherein: Optical character recognition technology is used to identify concepts and themes from which the non-text data features are extracted to generate text data features.

7. The method for constructing a multimodal dataset according to claim 2, wherein: The fusing of the non-text data features and the text data features to generate a structured knowledge graph includes: Calculating semantic similarity between the non-text data feature and the text data feature; generating a cross-modal adjacency matrix based on the semantic similarity; A dynamic graph convolution layer is used to perform convolution operations on the corresponding nodes in the cross-modal adjacency matrix to obtain a structured knowledge graph.

8. The method for constructing a multimodal dataset according to claim 7, wherein: The calculation formula of the convolution operation is as follows: ； in is the node feature matrix of the l+1th layer, is the adjacency matrix after preprocessing, For the The feature matrix of the layer, , for The degree matrix of For the The weight matrix of the layer, is the activation function.

9. A system for constructing a multimodal dataset, characterized in that: The multimodal dataset includes structured data and unstructured data, and the system includes an offline module and an online module: The offline module is used to: perform offline processing and fusion of unstructured data to generate a structured knowledge graph; perform offline mapping processing on structured data, and simultaneously input the unstructured data after offline processing and the structured data after offline mapping processing into a preset structured model to generate a database; combine the structured knowledge graph and the database to build an offline model and train it to obtain a target large language model; The online module is used to: acquire and analyze user needs online, formulate an analysis plan, use the target large language model to execute the analysis plan, and obtain multimodal data; acquire and analyze user needs online, formulate an analysis plan, use the target large language model to execute the analysis plan, generate a multimodal data set and output it to the user.

10. The multimodal dataset construction system according to claim 9, characterized in that: The offline module is further used to: extract the pictures and tables using optical character recognition technology, and convert the extraction results into non-text data features; perform text analysis on the non-text data features, extract concepts and themes, and generate text data features; and fuse the non-text data features with the text data features to generate a structured knowledge graph; wherein the unstructured data includes pictures and tables.

Images