Multimodal large model training data acquisition method and system

Abstract

The present application relates to the field of computer technology, and specifically discloses a method and system for collecting data for training a multimodal large model, which uses data processing technology based on deep learning to extract semantic features and perform cross-modal joint encoding on semantically aligned image data and image content text description data, so as to capture the semantic association between modalities, realize cross-modal semantic information fusion, and further generate image samples on this basis, and then, by measuring the semantic offset between the generated image samples and the original image data, intelligently identify whether the generated image samples are qualified enhanced samples. In this way, the multimodal training data set can be effectively enriched, the quality of the data can be ensured, and the problem of cross-modal semantic consistency in the process of multimodal data enhancement can be solved, thereby improving the training efficiency of the multimodal large model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a method and system for collecting multimodal large model training data. Background Art

[0002] With the rapid development of deep learning technology, multimodal learning has become an important research direction. Multimodal learning aims to improve the understanding and generalization capabilities of machine learning models by combining different types of perceptual data (such as images, text, etc.). In this process, high-quality multimodal training data is crucial to building efficient and accurate models.

[0003] However, in practical applications, high-quality multimodal datasets are often difficult to obtain, and there is a common problem of insufficient data volume, so data augmentation is needed to generate enough samples. However, due to the multimodal data involved, data augmentation can easily maintain consistency only within the modality while causing large differences in cross-modal semantics, affecting the subsequent training effect of large multimodal models.

[0004] Therefore, an optimized multi-modal large model training data acquisition method and system are expected. Summary of the invention

[0005] In order to solve the above-mentioned technical problems, the present application is proposed. The embodiment of the present application provides a multimodal large model training data acquisition method and system, which uses deep learning-based data processing technology to perform semantic feature extraction and cross-modal joint encoding on semantically aligned image data and image content text description data, so as to capture the semantic association between modalities, realize cross-modal semantic information fusion, and further generate image samples on this basis, and then, by measuring the semantic offset between the generated image samples and the original image data, it is possible to intelligently identify whether the generated image samples are qualified enhanced samples. In this way, the multimodal training data set can be effectively enriched, the quality of the data can be ensured, and the problem of cross-modal semantic consistency in the process of multimodal data enhancement can be solved, thereby improving the training efficiency of the multimodal large model.

[0006] According to one aspect of the present application, a multimodal large model training data acquisition method is provided, which includes:

[0007] Acquire first modality data and second modality data marked as aligned, wherein the first modality data is an image, and the second modality data is a text description of the image content;

[0008] Performing semantic feature extraction on the first modal data and the second modal data respectively to obtain an image semantic coding feature map and an image content text description semantic coding feature vector;

[0009] Performing cross-modal joint encoding on the image semantic encoding feature map and the image content text description semantic encoding feature vector to obtain an image-image text description cross-modal prompt guidance joint encoding feature map;

[0010] Based on the image-image text description cross-modal prompt, the joint encoding feature map is guided to generate an image to obtain a generated image sample;

[0011] Based on the semantic offset between the generated image sample and the first modality data, it is determined whether the generated image sample is a qualified enhancement sample.

[0012] In the above-mentioned multimodal large model training data acquisition method, the image semantic coding feature map and the image content text description semantic coding feature vector are cross-modally jointly encoded, including: fine-grained local query encoding of the image semantic coding feature map and the image content text description semantic coding feature vector to obtain a set of image-image text description local query prompt semantic coding vectors; using the set of image-image text description local query prompt semantic coding vectors as prompt information, fine-grained feature modulation of the image semantic coding feature map to obtain the image-image text description cross-modal prompt guided joint coding feature map.

[0013] In the above-mentioned multimodal large model training data acquisition method, semantic feature extraction is performed on the first modal data and the second modal data respectively to obtain an image semantic coding feature map and an image content text description semantic coding feature vector, including: performing image feature extraction based on a deep separable convolutional neural network model on the first modal data to obtain the image semantic coding feature map; and performing semantic coding on the second modal data to obtain the image content text description semantic coding feature vector.

[0014] In the above-mentioned multimodal large model training data acquisition method, fine-grained local query encoding is performed on the image semantic encoding feature map and the image content text description semantic encoding feature vector to obtain a set of image-image text description local query prompt semantic encoding vectors, including: performing local feature decomposition along the channel dimension on the image semantic encoding feature map to obtain a set of image local semantic feature matrices; using the image content text description semantic encoding feature vector as a query vector, and using each image local semantic feature matrix in the set of image local semantic feature matrices as a key matrix, and inputting the query vector and the key matrix into a prompt learning network to obtain the set of image-image text description local query prompt semantic encoding vectors.

[0015] In the above-mentioned multimodal large model training data acquisition method, the image content text description semantic encoding feature vector is used as the query vector, and each image local semantic feature matrix in the set of image local semantic feature matrices is used as a key matrix, and the query vector and the key matrix are input into the prompt learning network to obtain the set of the image-image text description local query prompt semantic encoding vectors, including: calculating the cosine similarity between the query vector and each key vector in the key matrix to obtain a set of image-image text description local semantic similarities; inputting the set of image-image text description local semantic similarities into the softmax function for normalization processing to obtain a set of normalized image-image text description local semantic similarities; using the set of normalized image-image text description local semantic similarities as the weight distribution, calculating the position-weighted sum of each key vector in the key matrix to obtain the image-image text description local query prompt semantic encoding vector.

[0016] In the above-mentioned multimodal large model training data acquisition method, the set of the image-image text description local query hint semantic coding vectors is used as hint information, and the image semantic coding feature map is fine-grained feature modulated to obtain the image-image text description cross-modal hint guided joint coding feature map, including: inputting the image content text description semantic coding feature vector and each image-image text description local query hint semantic coding vector in the set of the image-image text description local query hint semantic coding vector into a cross-modal mask weaving network based on hint information to obtain a set of image local semantic feature mask weight matrices based on hint information; based on the set of image local semantic feature mask weight matrices based on hint information, the set of image local semantic feature matrices is feature modulated and aggregated to obtain the image-image text description cross-modal hint guided joint coding feature map.

[0017] In the above-mentioned multimodal large model training data acquisition method, each image-image text description local query prompt semantic coding vector in the set of the image content text description semantic coding feature vector and the image-image text description local query prompt semantic coding vector is input into a cross-modal mask weaving network based on prompt information to obtain a set of image local semantic feature mask weight matrices based on prompt information, including: calculating the vector product between the image content text description semantic coding feature vector and the transposed vector of the image-image text description local query prompt semantic coding vector to obtain the image-image text description cross-modal query attention matrix; dividing the image-image text description cross-modal query attention matrix by the square root of the feature scale of the image-image text description local query prompt semantic coding vector, and then inputting the softmax function for normalization processing to obtain the image local semantic feature mask weight matrix based on prompt information.

[0018] In the above-mentioned multimodal large model training data acquisition method, based on the set of image local semantic feature mask weight matrices based on prompt information, the set of image local semantic feature matrices is feature modulated and aggregated to obtain the image-image text description cross-modal prompt-guided joint encoding feature map, including: calculating the position point multiplication between the set of image local semantic feature mask weight matrices based on prompt information and the set of image local semantic feature matrices to obtain a set of image-image text description cross-modal local granularity significant interaction matrices; and performing feature cascade aggregation on the set of image-image text description cross-modal local granularity significant interaction matrices to obtain the image-image text description cross-modal prompt-guided joint encoding feature map.

[0019] In the above-mentioned multimodal large model training data acquisition method, image generation is performed based on the image-image text description cross-modal prompt-guided joint encoding feature map to obtain a generated image sample, including: inputting the image-image text description cross-modal prompt-guided joint encoding feature map into a sample enhancer based on a diffusion model to obtain the generated image sample.

[0020] In the above-mentioned multimodal large model training data acquisition method, based on the semantic offset between the generated image sample and the first modal data, determining whether the generated image sample is a qualified enhanced sample includes: performing image feature extraction based on a deep separable convolutional neural network model on the generated image sample to obtain a semantic encoding feature map of the generated image sample; calculating the cosine similarity between the semantic encoding feature map of the generated image sample and the image semantic encoding feature map as a cross-modal generated semantic offset coefficient; in response to the cross-modal generated semantic offset coefficient being less than or equal to a preset threshold, determining that the generated image sample is a qualified enhanced text.

[0021] According to another aspect of the present application, a multimodal large model training data acquisition system is provided, comprising:

[0022] A modality data acquisition module, configured to acquire first modality data and second modality data marked as aligned, wherein the first modality data is an image and the second modality data is a text description of the image content;

[0023] A semantic feature extraction module, used to extract semantic features from the first modal data and the second modal data respectively to obtain an image semantic coding feature map and an image content text description semantic coding feature vector;

[0024] A cross-modal joint encoding module, used for cross-modal joint encoding of the image semantic encoding feature map and the image content text description semantic encoding feature vector to obtain an image-image text description cross-modal prompt guidance joint encoding feature map;

[0025] An image generation module, configured to generate an image based on the image-image text description cross-modal prompt-guided joint encoding feature map to obtain a generated image sample;

[0026] A sample quality assessment module is used to determine whether the generated image sample is a qualified enhanced sample based on the semantic offset between the generated image sample and the first modality data.

[0027] Compared with the prior art, the multimodal large model training data acquisition method and system provided by the present application uses deep learning-based data processing technology to perform semantic feature extraction and cross-modal joint encoding of semantically aligned image data and image content text description data, so as to capture the semantic association between modalities and realize cross-modal semantic information fusion, and further generate image samples on this basis, and then, by measuring the semantic offset between the generated image samples and the original image data, it is possible to intelligently identify whether the generated image samples are qualified enhanced samples. In this way, the multimodal training data set can be effectively enriched, the quality of the data can be ensured, and the problem of cross-modal semantic consistency in the process of multimodal data enhancement can be solved, thereby improving the training efficiency of the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0029] Figure 1It is a flowchart of a multimodal large model training data acquisition method according to an embodiment of the present application.

[0030] Figure 2 Schematic diagram of data flow of a multimodal large model training data acquisition method according to an embodiment of the present application.

[0031] Figure 3 This is a flowchart of sub-step S2 of the multimodal large model training data acquisition method according to an embodiment of the present application.

[0032] Figure 4 This is a flowchart of sub-step S3 of the multimodal large model training data acquisition method according to an embodiment of the present application.

[0033] Figure 5 This is a flowchart of sub-step S31 of the multimodal large model training data acquisition method according to an embodiment of the present application.

[0034] Figure 6 This is a flowchart of sub-step S32 of the multimodal large model training data acquisition method according to an embodiment of the present application.

[0035] Figure 7 This is a flowchart of sub-step S5 of the multimodal large model training data acquisition method according to an embodiment of the present application.

[0036] Figure 8 It is a block diagram of a multimodal large model training data acquisition system according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0038] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0039] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0040] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0041] In response to the technical problems described in the above background technology, this application proposes an optimized multimodal large model training data acquisition method, which uses deep learning-based data processing technology to perform semantic feature extraction and cross-modal joint encoding of semantically aligned image data and image content text description data, so as to capture the semantic association between modalities and realize cross-modal semantic information fusion, and further generate image samples on this basis, and then, by measuring the semantic offset between the generated image samples and the original image data, it is possible to intelligently identify whether the generated image samples are qualified enhanced samples. In this way, the multimodal training data set can be effectively enriched, the quality of the data can be ensured, and the problem of cross-modal semantic consistency in the process of multimodal data enhancement can be solved, thereby improving the training efficiency of multimodal large models.

[0042] Figure 1 It is a flowchart of a multimodal large model training data acquisition method according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of a multi-modal large model training data collection method according to an embodiment of the present application. Figure 1 and Figure 2 As shown, the multimodal large model training data acquisition method includes the steps of: S1, obtaining first modal data and second modal data marked as aligned, wherein the first modal data is an image and the second modal data is a text description of the image content; S2, performing semantic feature extraction on the first modal data and the second modal data respectively to obtain an image semantic coding feature map and an image content text description semantic coding feature vector; S3, performing cross-modal joint encoding on the image semantic coding feature map and the image content text description semantic coding feature vector to obtain an image-image text description cross-modal prompt-guided joint coding feature map; S4, performing image generation based on the image-image text description cross-modal prompt-guided joint coding feature map to obtain a generated image sample; S5, determining whether the generated image sample is a qualified enhanced sample based on the semantic offset between the generated image sample and the first modal data.

[0043] In the above-mentioned multimodal large model training data acquisition method, the step S1 obtains the first modal data and the second modal data marked as aligned, wherein the first modal data is an image and the second modal data is a text description of the image content. It should be understood that in the training process of the multimodal large model, in order to ensure that the model can correctly learn the semantic correspondence between the image and the text, it is necessary to use semantically aligned cross-modal data as training data. Therefore, when generating new image samples through data enhancement, in order to ensure that the generated image samples maintain semantic consistency with the original image data and text description, the present application uses the first modal data and the second modal data marked as aligned to generate image samples, thereby further expanding the training data set.

[0044] In the above-mentioned multimodal large model training data acquisition method, the step S2 performs semantic feature extraction on the first modal data and the second modal data respectively to obtain an image semantic coding feature map and an image content text description semantic coding feature vector. It should be understood that in order to utilize the joint semantic information of the first modal data and the second modal data to generate image samples to ensure that the generated image samples have cross-modal semantic consistency, first, single-modal semantic feature extraction is performed on the first modal data and the second modal data respectively. Among them, Figure 3 FIG. 1 is a flowchart of sub-step S2 of the multi-modal large model training data collection method according to an embodiment of the present application. Figure 3 As shown, the step S2 includes the steps of: S21, performing image feature extraction based on a deep separable convolutional neural network model on the first modal data to obtain the image semantic coding feature map; S22, performing semantic coding on the second modal data to obtain the semantic coding feature vector of the image content text description.

[0045] Specifically, the step S21 performs image feature extraction based on a deep separable convolutional neural network model on the first modal data to obtain the image semantic coding feature map. Those of ordinary skill in the art should know that the deep separable convolutional neural network is an efficient neural network architecture that reduces the amount of computation while maintaining the comprehensiveness of feature extraction by subdividing the standard convolution operation into two steps: deep convolution and point-by-point convolution. In the deep convolution stage, the network focuses on learning the independent spatial features of each feature channel of the image, while in the point-by-point convolution stage, cross-channel feature integration is performed by applying a 1x1 convolution kernel to each feature channel, thereby achieving a balance between the depth and breadth of the features. This method makes the model more suitable for running on resource-constrained devices while maintaining high performance. In this way, the deep separable convolutional neural network model can effectively capture the subtle features of the image to generate the image semantic coding feature map.

[0046] Specifically, the step S22 semantically encodes the second modal data to obtain the semantic encoding feature vector of the text description of the image content. Specifically, for the second modal data, the present application uses a pre-trained language model based on natural language processing technology to semantically encode the text description of the image content, so as to capture the key semantic information in the text, and convert the complex semantic structure therein into a numerical vector representation, thereby obtaining the semantic encoding feature vector of the text description of the image content. In a specific example of the present application, the BERT model is used as a pre-trained language model to achieve an in-depth understanding of the text description. It should be known to those of ordinary skill in the art that the BERT model is a pre-trained language model that is based on a bidirectional Transformer architecture and can capture the contextual information of the text. During the pre-training process, BERT uses a large amount of unlabeled text data to learn the deep meaning of words, phrases and sentences in the text through a self-supervised learning method. This learning mechanism enables BERT to consider the context of the text at the same time, so as to more accurately understand the meaning of the text. After pre-training, the BERT model is fine-tuned on the labeled image content text description dataset for this application, so that the BERT model can better understand the deep contextual semantic information in the image content text description, thereby providing high-quality semantic representation for subsequent cross-modal joint encoding and image generation.

[0047] In the above-mentioned multimodal large model training data acquisition method, the step S3 performs cross-modal joint encoding on the image semantic coding feature map and the image content text description semantic coding feature vector to obtain an image-image text description cross-modal prompt-guided joint encoding feature map. Specifically, the present application further performs cross-modal joint encoding on the image semantic coding feature map and the image content text description semantic coding feature vector to fuse the semantic information of the image data and the image content text description, thereby enriching the source of semantic information in the image sample generation process. It is worth mentioning that in order to improve the semantic fusion effect of cross-modal data, the present application proposes a feature fusion coding method based on cross-modal prompt learning, which can dynamically guide the model to focus on the feature parts of the image features that have significant semantic associations with the text data in the joint encoding stage by introducing a continuous prompt mechanism, thereby enhancing the fusion and understanding of cross-modal information. Among them, Figure 4 FIG. 4 is a flowchart of sub-step S3 of the multi-modal large model training data collection method according to an embodiment of the present application. Figure 4As shown, the step S3 includes the steps of: S31, performing fine-grained local query encoding on the image semantic encoding feature map and the image content text description semantic encoding feature vector to obtain a set of image-image text description local query prompt semantic encoding vectors; S32, using the set of image-image text description local query prompt semantic encoding vectors as prompt information, performing fine-grained feature modulation on the image semantic encoding feature map to obtain the image-image text description cross-modal prompt guided joint encoding feature map.

[0048] Figure 5 FIG. 4 is a flowchart of sub-step S31 of the multi-modal large model training data collection method according to an embodiment of the present application. Figure 5 As shown, the step S31 includes the steps of: S311, performing local feature decomposition along the channel dimension on the image semantic coding feature map to obtain a set of image local semantic feature matrices; S312, using the image content text description semantic coding feature vector as a query vector, and using each image local semantic feature matrix in the set of image local semantic feature matrices as a key matrix, inputting the query vector and the key matrix into a prompt learning network to obtain a set of image-image text description local query prompt semantic coding vectors.

[0049] More specifically, the step S312 further includes: calculating the cosine similarity between the query vector and each key vector in the key matrix to obtain a set of image-image text description local semantic similarities; inputting the set of image-image text description local semantic similarities into a softmax function for normalization processing to obtain a set of normalized image-image text description local semantic similarities; using the set of normalized image-image text description local semantic similarities as a weight distribution, calculating the position-weighted sum of each key vector in the key matrix to obtain the image-image text description local query prompt semantic encoding vector.

[0050] That is, first, the image semantic encoding feature map is decomposed into local features along the channel dimension to enhance feature resolution and focus more on cross-modal interaction of detail features. Then, the image content text description semantic encoding feature vector is used as the query vector, and each image local semantic feature matrix after decomposition of the image semantic encoding feature map along the channel dimension is used as the key matrix. The two are sent to the prompt learning network for attention query encoding, and the attention weight is calculated through fine-grained semantic similarity measurement. The feature parts associated with the image content text description semantics are selected from each image local semantic feature matrix, thereby generating a set of image-image text description local query prompt semantic encoding vectors to enhance the understanding of the correlation and interaction relationship between modalities.

[0051] Figure 6 FIG. 4 is a flowchart of sub-step S32 of the multi-modal large model training data collection method according to an embodiment of the present application. Figure 6 As shown, the step S32 includes the steps of: S321, inputting each image-image text description local query hint semantic coding vector in the set of the image content text description semantic coding feature vector and the image-image text description local query hint semantic coding vector into a cross-modal mask weaving network based on hint information to obtain a set of image local semantic feature mask weight matrices based on hint information; S322, based on the set of image local semantic feature mask weight matrices based on hint information, performing feature modulation aggregation on the set of image local semantic feature matrices to obtain the image-image text description cross-modal hint-guided joint coding feature map.

[0052] More specifically, the step S321 further includes: calculating the vector product between the image content text description semantic encoding feature vector and the transposed vector of the image-image text description local query hint semantic encoding vector to obtain the image-image text description cross-modal query attention matrix; dividing the image-image text description cross-modal query attention matrix by the square root of the feature scale of the image-image text description local query hint semantic encoding vector, and inputting the softmax function for normalization processing to obtain the image local semantic feature mask weight matrix based on hint information. That is, the generated image-image text description local query hint semantic encoding vectors and the image content text description semantic encoding feature vectors are respectively input into the cross-modal mask weaving network based on hint information for further hint learning, and the cross-modal mask weaving network processes the two based on the self-attention mechanism, which can effectively highlight the important interaction parts between modalities, while suppressing irrelevant information and reducing the influence of noise, thereby creating a mask that emphasizes the most relevant part of modal interaction, that is, the image local semantic feature mask weight matrix based on hint information.

[0053] More specifically, the step S322 further includes: calculating the set of image-image text description cross-modal local granularity significant interaction matrices obtained by multiplying the set of image local semantic feature mask weight matrices based on the prompt information and the set of image local semantic feature matrices by position points; and performing feature cascade aggregation on the set of image-image text description cross-modal local granularity significant interaction matrices to obtain the image-image text description cross-modal prompt-guided joint encoding feature map. That is, using each image local semantic feature mask weight matrix based on the prompt information as a weight, each image local semantic feature matrix is ​​weighted modulated to emphasize the feature part of the image feature that is significantly associated with the text data, thereby achieving guided fusion of cross-modal data semantic information. Finally, each weighted modulated image local semantic feature matrix is ​​subjected to information aggregation along the channel dimension to restore the original feature map structure and generate an image-image text description cross-modal prompt-guided joint encoding feature map. Through this continuous prompt-assisted learning process, the semantic association between cross-modal data can be better captured, and the quality of generated image samples can be further improved.

[0054] Accordingly, the step S3 includes: processing the image semantic coding feature map and the image content text description semantic coding feature vector using the following cross-modal joint coding formula to obtain an image-image text description cross-modal prompt guidance joint coding feature map, wherein the cross-modal joint coding formula is:

[0055]

[0056]

[0057]

[0058]

[0059] in, represents the semantic encoding feature vector of the text description of the image content, represents the image semantic encoding feature map, represents the eigendecomposition, , , and Respectively represent the first, second, and and The local semantic feature matrix of the image, that is, the feature matrix of the image semantic coding feature map along the channel dimension, is the number of channels of the image semantic encoding feature map, , 、 and Respectively represent the The first, second, and and row vectors, namely the key vectors, is the number of rows of the local semantic feature matrix of the image, Indicates prompt learning network, represents vector multiplication, represents the transpose of a vector, represents the magnitude of a vector, and Respectively represent and The local semantic similarity between image-image text description, represents the exponential function with base e, Indicates Normalized image-image text description local semantic similarity, It means point multiplication by position. Indicates image-image text description local query hint semantic encoding vector, The feature scale of the image-image text description local query hint semantic encoding vector, represents the normalized exponential function, , , and Respectively represent the first, second, and and A local semantic feature mask weight matrix of the image based on the hint information, Indicates cascade, Represents the image-image-text description cross-modal cue-guided joint encoding feature map.

[0060] In the above-mentioned multimodal large model training data acquisition method, the step S4 performs image generation based on the image-image text description cross-modal prompt-guided joint coding feature map to obtain the generated image sample. In a specific example of the present application, the step S4 further includes: inputting the image-image text description cross-modal prompt-guided joint coding feature map into a sample enhancer based on a diffusion model to obtain the generated image sample. That is, a sample enhancer is constructed using a diffusion model with excellent performance in the field of image generation, and a high-quality image sample is generated by performing a diffusion process on the image-image text description cross-modal prompt-guided joint coding feature map. Specifically, the diffusion model converts the image data into noise data by gradually adding noise, and then gradually recovers the details and structure of the image from the noise by learning the distribution of the noise data, and restores a clear image, thereby generating an enhanced sample that is highly consistent with the original image data in both visual and semantics.

[0061] Here, in the case where the image semantic coding feature map and the image content text description semantic coding feature vector represent the image semantic coding features and the text semantic coding features of the image content text description respectively, when performing cross-modal joint encoding assisted by continuous prompts, the cross-modal joint coding feature representation of the image-image text description cross-modal prompt-guided joint coding feature map will also cause the image-image text description cross-modal prompt-guided joint coding feature map to have a lack of interactive feature instance decision-making relative to the distribution field selective state space of the heteromodal data feature prompts, thereby affecting the cross-modal semantic logical dependency and reducing the accuracy of the diffusion generation results based on the cross-modal joint distribution.

[0062] Preferably, inputting the image-image text description cross-modal prompt-guided joint encoding feature map into a diffusion model-based sample enhancer to obtain a generated image sample comprises:

[0063] Expanding the image-image text description cross-modal prompt-guided joint encoding feature map into an image-image text description cross-modal prompt-guided joint encoding feature vector;

[0064] Calculate the distance between each pair of eigenvalues ​​of the image-image text description cross-modal prompt-guided joint encoding feature vector, such as the L2 distance, and take the square root of the distance to obtain the image-image text description cross-modal prompt-guided joint encoding distance representation matrix, that is,

[0065]

[0066] in, represents the image-image text description cross-modal prompt-guided joint encoding feature vector, and represents the image-image text description cross-modal hint-guided joint encoding feature vector and Eigenvalues, represents the image-image text description cross-modal hint-guided joint encoding feature vector and The distance between eigenvalues, Representation of image-image text description cross-modal cue-guided joint encoding distance representation matrix The eigenvalue of the position;

[0067] The image-image text description cross-modal prompt-guided joint encoding autocorrelation matrix of the image-image text description cross-modal prompt-guided joint encoding feature vector is obtained, that is, ,in, represents the image-image text description cross-modal prompt-guided joint encoding feature vector, represents the transpose of a vector, represents matrix multiplication, Represents the image-image text description cross-modal cue-guided joint encoding autocorrelation matrix;

[0068] The image-image text description cross-modal prompt-guided joint coding feature vector is matrix-multiplied with the image-image text description cross-modal prompt-guided joint coding distance representation matrix to obtain the image-image text description cross-modal prompt-guided joint coding first-level mapping vector, that is, ,in, represents the image-image text description cross-modal prompt-guided joint encoding feature vector, Represents the image-image text description cross-modal cue-guided joint encoding distance representation matrix, represents matrix multiplication, Represents the first-level mapping vector of image-image text description cross-modal cue-guided joint encoding;

[0069] The first-level mapping vector of the image-image text description cross-modal prompt-guided joint encoding is matrix-multiplied by the matrix product of the image-image text description cross-modal prompt-guided joint encoding distance representation matrix and the image-image text description cross-modal prompt-guided joint encoding autocorrelation matrix to obtain a multi-level mapping vector of the image-image text description cross-modal prompt-guided joint encoding ,in, Represents the image-image text description cross-modal cue-guided joint encoding of the first-level mapping vector, Represents the image-image text description cross-modal cue-guided joint encoding distance representation matrix, represents matrix multiplication, represents the image-image text description cross-modal cue-guided joint encoding autocorrelation matrix, Represents image-image text description cross-modal cue-guided joint encoding multi-level mapping vector;

[0070] The image-image text description cross-modal prompt-guided joint coding multi-level mapping vector is interpolated with the image-image text description cross-modal prompt-guided joint coding associated eigenvector (if the eigenvalue is insufficient, interpolation or zero padding) composed of the eigenvalue of the image-image text description cross-modal prompt-guided joint coding auto-association matrix to obtain an optimized image-image text description cross-modal prompt-guided joint coding feature vector;

[0071] The optimized image-image text description cross-modal cue-guided joint encoding feature vector is passed through a diffusion model-based sample enhancer to obtain a generated image sample.

[0072] Therefore, through the linear target mapping representation of the similarity distance representation matrix of the image-image text description cross-modal prompt-guided joint encoding feature vector after the image-image text description cross-modal prompt-guided joint encoding feature map is expanded, the self-correlation complete similarity instantiation of the image-image text description cross-modal prompt-guided joint encoding feature map is instantiated with a secondary target mapping representation based on a multi-level distribution hierarchy, and the association mismatch negative influence factor is compensated by the association fusion kernel bias to improve the degree of instance determination of the eigenvalue of the image-image text description cross-modal prompt-guided joint encoding feature map under similarity constraints, that is, the degree of significance of the eigenvalue as an instance for classification regression determination, thereby improving the image quality of the generated image sample obtained by inputting the image-image text description cross-modal prompt-guided joint encoding feature map into the sample enhancer based on the diffusion model.

[0073] In the above multi-modal large model training data acquisition method, the step S5 determines whether the generated image sample is a qualified enhanced sample based on the semantic offset between the generated image sample and the first modality data. Figure 7 FIG. 5 is a flowchart of sub-step S5 of the multi-modal large model training data collection method according to an embodiment of the present application. Figure 7 As shown, the step S5 includes the steps of: S51, performing image feature extraction based on a deep separable convolutional neural network model on the generated image sample to obtain a semantic coding feature map of the generated image sample; S52, calculating the cosine similarity between the semantic coding feature map of the generated image sample and the image semantic coding feature map as a cross-modal generated semantic offset coefficient; S53, in response to the cross-modal generated semantic offset coefficient being less than or equal to a preset threshold, determining that the generated image sample is a qualified enhanced text.

[0074] Specifically, in step S51, the generated image sample is subjected to image feature extraction based on a deep separable convolutional neural network model to obtain a semantically encoded feature map of the generated image sample. It should be understood that in order to verify whether the generated image sample meets the quality standard, the present application further performs a comparative analysis on the generated image sample and the original first modal data to ensure that the generated image sample is semantically consistent with the original image. Specifically, the generated image sample is subjected to image feature extraction using a deep separable convolutional neural network model to utilize the efficient feature extraction capability of the deep separable convolutional neural network to mine the key semantic information in the image sample and obtain a semantically encoded feature map of the generated image sample. In addition, by using the same network model for image encoding, the consistency of the feature extraction process can be ensured, which helps to improve the accuracy of subsequent feature comparison and analysis.

[0075] Specifically, the step S52 calculates the cosine similarity between the generated image sample semantic coding feature map and the image semantic coding feature map as a cross-modal semantic offset coefficient. That is, the calculation method of cosine similarity is used as a metric to calculate the semantic offset between the generated image sample semantic coding feature map and the image semantic coding feature map to evaluate the semantic consistency between the two. The closer the value of the cosine similarity is to 1, the more semantically consistent the generated image sample is with the original image data, and vice versa, it indicates that its semantic consistency is poor.

[0076] Specifically, in step S53, in response to the cross-modal generated semantic shift coefficient being less than or equal to a preset threshold, the generated image sample is determined to be a qualified enhanced text. That is, by comparing the cross-modal generated semantic shift coefficient with the preset threshold, it is determined whether the generated image sample meets the preset semantic consistency standard. If the cross-modal generated semantic shift coefficient is higher than the threshold, it is considered that the generated image sample meets the quality standard and can be used for subsequent training or application. If it is lower than the threshold, it is necessary to adjust the network parameters in the image sample generation process to improve the quality of the image sample. In this way, high-quality image samples can be effectively screened out for further model training and verification.

[0077] In summary, the multimodal large model training data acquisition method based on the embodiment of the present application is explained, which uses deep learning-based data processing technology to perform semantic feature extraction and cross-modal joint encoding of semantically aligned image data and image content text description data to capture the semantic association between modalities and achieve cross-modal semantic information fusion. On this basis, image samples are further generated, and then, by measuring the semantic offset between the generated image samples and the original image data, it is possible to intelligently identify whether the generated image samples are qualified enhanced samples. In this way, the multimodal training data set can be effectively enriched, the quality of the data can be ensured, and the problem of cross-modal semantic consistency in the process of multimodal data enhancement can be solved, thereby improving the training efficiency of the multimodal large model.

[0078] Furthermore, a multimodal large model training data acquisition system is also provided.

[0079] Figure 8 FIG. 1 is a block diagram of a multi-modal large model training data acquisition system according to an embodiment of the present application. Figure 8 As shown, according to an embodiment of the present application, a multimodal large model training data acquisition system 100 includes: a modal data acquisition module 110, which is used to acquire first modal data and second modal data marked as aligned, wherein the first modal data is an image and the second modal data is a text description of the image content; a semantic feature extraction module 120, which is used to perform semantic feature extraction on the first modal data and the second modal data respectively to obtain an image semantic coding feature map and an image content text description semantic coding feature vector; a cross-modal joint encoding module 130, which is used to perform cross-modal joint encoding on the image semantic coding feature map and the image content text description semantic coding feature vector to obtain an image-image text description cross-modal prompt-guided joint coding feature map; an image generation module 140, which is used to generate an image based on the image-image text description cross-modal prompt-guided joint coding feature map to obtain a generated image sample; a sample quality assessment module 150, which is used to determine whether the generated image sample is a qualified enhanced sample based on the semantic offset between the generated image sample and the first modal data.

[0080] Here, those skilled in the art can understand that the specific operations of each module in the above multi-modal large model training data acquisition system have been described in the above reference. Figures 1 to 7 The description of the multimodal large model training data acquisition method has been introduced in detail, and therefore, its repeated description will be omitted.

[0081] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above embodiments are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0082] In the above embodiments, the description of each embodiment has its own emphasis. For the parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiment described above is only schematic. For example, the unit division is only a logical function division, and there may be other division methods in actual implementation. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0083] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference to a figure in a claim should not be considered as limiting the claim to which it relates.

[0084] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units stated in the system claims can also be implemented by one unit through software or hardware.

[0085] Finally, it should be noted that the above description has been given for the purpose of illustration and description. In addition, the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A multimodal large model training data collection method, characterized in that: include: Acquire first modality data and second modality data marked as aligned, wherein the first modality data is an image, and the second modality data is a text description of image content; Performing semantic feature extraction on the first modal data and the second modal data respectively to obtain an image semantic coding feature map and an image content text description semantic coding feature vector; Performing cross-modal joint encoding on the image semantic encoding feature map and the image content text description semantic encoding feature vector to obtain an image-image text description cross-modal prompt guidance joint encoding feature map; Based on the image-image text description cross-modal prompt, the joint encoding feature map is guided to generate an image to obtain a generated image sample; Determining whether the generated image sample is a qualified enhanced sample based on a semantic offset between the generated image sample and the first modality data; Among them, the image semantic coding feature map and the image content text description semantic coding feature vector are cross-modally jointly encoded, including: fine-grained local query encoding of the image semantic coding feature map and the image content text description semantic coding feature vector to obtain a set of image-image text description local query prompt semantic coding vectors; using the set of image-image text description local query prompt semantic coding vectors as prompt information, fine-grained feature modulation of the image semantic coding feature map to obtain the image-image text description cross-modal prompt guided joint coding feature map.

2. The multimodal large model training data acquisition method according to claim 1, characterized in that: Semantic feature extraction is performed on the first modal data and the second modal data respectively to obtain an image semantic coding feature map and an image content text description semantic coding feature vector, including: Performing image feature extraction based on a deep separable convolutional neural network model on the first modal data to obtain the image semantic encoding feature map; The second modality data is semantically encoded to obtain a semantic encoding feature vector of the text description of the image content.

3. The multimodal large model training data acquisition method according to claim 2, characterized in that: Fine-grained local query encoding is performed on the image semantic encoding feature map and the image content text description semantic encoding feature vector to obtain a set of image-image text description local query prompt semantic encoding vectors, including: Performing local feature decomposition along the channel dimension on the image semantic coding feature map to obtain a set of image local semantic feature matrices; The semantic encoding feature vector of the text description of the image content is used as a query vector, and each image local semantic feature matrix in the set of image local semantic feature matrices is used as a key matrix, and the query vector and the key matrix are input into a prompt learning network to obtain a set of image-image text description local query prompt semantic encoding vectors.

4. The multimodal large model training data acquisition method according to claim 3 is characterized in that: The image content text description semantic encoding feature vector is used as a query vector, and each image local semantic feature matrix in the set of image local semantic feature matrices is used as a key matrix, and the query vector and the key matrix are input into a prompt learning network to obtain a set of image-image text description local query prompt semantic encoding vectors, including: Calculating the cosine similarity between the query vector and each key vector in the key matrix to obtain a set of image-image text description local semantic similarities; Inputting the set of local semantic similarities of the image-image text description into a softmax function for normalization processing to obtain a set of normalized local semantic similarities of the image-image text description; The set of normalized image-image text description local semantic similarities is used as weight distribution, and the position-weighted sum of each key vector in the key matrix is ​​calculated to obtain the image-image text description local query prompt semantic encoding vector.

5. The multimodal large model training data acquisition method according to claim 4, characterized in that: Taking the set of the image-image text description local query hint semantic coding vectors as hint information, fine-grained feature modulation is performed on the image semantic coding feature map to obtain the image-image text description cross-modal hint guidance joint coding feature map, including: Inputting each image-image text description local query prompt semantic coding vector in the set of the image content text description semantic coding feature vector and the image-image text description local query prompt semantic coding vector into a cross-modal mask weaving network based on prompt information to obtain a set of image local semantic feature mask weight matrices based on prompt information; Based on the set of image local semantic feature mask weight matrices based on the prompt information, feature modulation aggregation is performed on the set of image local semantic feature matrices to obtain the image-image text description cross-modal prompt-guided joint encoding feature map.

6. The multimodal large model training data acquisition method according to claim 5, characterized in that: Inputting each image-image text description local query hint semantic encoding vector in the set of the image content text description semantic encoding feature vector and the image-image text description local query hint semantic encoding vector into a cross-modal mask weaving network based on hint information to obtain a set of image local semantic feature mask weight matrices based on hint information, including: Calculating the vector product between the image content text description semantic encoding feature vector and the transposed vector of the image-image text description local query hint semantic encoding vector to obtain an image-image text description cross-modal query attention matrix; After dividing the image-image text description cross-modal query attention matrix by the square root of the feature scale of the image-image text description local query prompt semantic encoding vector, the matrix is ​​input into the softmax function for normalization to obtain the image local semantic feature mask weight matrix based on the prompt information.

7. The multimodal large model training data acquisition method according to claim 6, characterized in that: Based on the set of image local semantic feature mask weight matrices based on the prompt information, feature modulation aggregation is performed on the set of image local semantic feature matrices to obtain the image-image text description cross-modal prompt-guided joint encoding feature map, including: Calculating the point-by-point multiplication between the set of image local semantic feature mask weight matrices based on the prompt information and the set of image local semantic feature matrices to obtain a set of image-image text description cross-modal local granularity significant interaction matrices; The set of the image-image text description cross-modal local granularity significant interaction matrices is subjected to feature cascade aggregation to obtain the image-image text description cross-modal cue-guided joint encoding feature map.

8. The multimodal large model training data acquisition method according to claim 7, characterized in that: Based on the image-image text description cross-modal prompt guiding the joint encoding feature map to generate an image to obtain a generated image sample, including: The image-image text description cross-modal prompt-guided joint encoding feature map is input into a diffusion model-based sample enhancer to obtain the generated image sample.

9. The multimodal large model training data acquisition method according to claim 8, characterized in that: Determining whether the generated image sample is a qualified enhanced sample based on a semantic offset between the generated image sample and the first modality data includes: Performing image feature extraction based on a deep separable convolutional neural network model on the generated image sample to obtain a semantic encoding feature map of the generated image sample; Calculating the cosine similarity between the generated image sample semantic encoding feature map and the image semantic encoding feature map as a cross-modal generated semantic offset coefficient; In response to the cross-modal generated semantic shift coefficient being less than or equal to a preset threshold, the generated image sample is determined to be a qualified enhanced text.

10. A multimodal large model training data acquisition system, characterized in that: include: A modality data acquisition module, configured to acquire first modality data and second modality data marked as aligned, wherein the first modality data is an image and the second modality data is a text description of the image content; A semantic feature extraction module, used to extract semantic features from the first modal data and the second modal data respectively to obtain an image semantic coding feature map and an image content text description semantic coding feature vector; A cross-modal joint encoding module, used for cross-modal joint encoding of the image semantic encoding feature map and the image content text description semantic encoding feature vector to obtain an image-image text description cross-modal prompt guidance joint encoding feature map; An image generation module, configured to generate an image based on the image-image text description cross-modal prompt-guided joint encoding feature map to obtain a generated image sample; A sample quality assessment module is used to determine whether the generated image sample is a qualified enhanced sample based on the semantic offset between the generated image sample and the first modality data.

Images