Data processing method and device based on multimodal large model
By performing feature extraction and residual vector quantization of image data, combined with visual feature reconstruction of large language models and decoders, the problem of multimodal data representation inconsistency is solved, and the effectiveness and image data of cross-modal tasks are improved integrity.
Patent Information
- Application Number
- CN202411075300.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-07
AI Technical Summary
Multimodal large models have challenges in representation inconsistencies when processing image data and text data, limiting their potential and effectiveness in cross-modal tasks.
Feature extraction and residual vector quantization of image data through ViT model, so that it is unified with the serialized encoded text data dimensions, then spliced with text data, autoregressive prediction is used using a large language model, and then using a decoder for Visual features are reconstructed and decoded into pixel space through the U-Net model, filling in pixel details.
The problem of inconsistent representation of multimodal data is solved, the integrity and prediction quality of image data are ensured, and the performance of the model in cross-modal tasks is improved.
Smart Images

Figure CN118606684B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device based on a multimodal large model. Background Art
[0002] Multimodal large models are an important research direction in the field of artificial intelligence. They aim to integrate information from multiple modalities such as text, images, audio and video into a unified model framework to achieve more powerful understanding and generation capabilities, and have shown great potential in cross-modal tasks. However, due to the inconsistency in the representation of multimodal data, multimodal large models face many challenges when processing data of different modalities. For example, there are significant differences in structure and semantic expression between image data and text data: image data usually contains rich visual information, such as color, shape, and texture, while text data focuses on semantic expression, such as concepts, relationships, and events. At present, the commonly used data processing methods for multimodal large models often adopt an unbalanced treatment of data of different modalities, which greatly limits the potential and effectiveness of the model in cross-modal tasks.
[0003] Therefore, there is an urgent need for a method that can effectively solve the inconsistency problem of multimodal data. Summary of the invention
[0004] The present invention provides a data processing method and device based on a multimodal large model, which are used to solve the problem of inconsistent representation of multimodal data in image data and text data.
[0005] In a first aspect, a data processing method based on a multimodal large model is provided. The method comprises:
[0006] Acquire multimodal data, and extract image data and text data from the multimodal data;
[0007] Serialize and encode the text data according to a preset dictionary to obtain first text data;
[0008] Using the ViT model to extract features from the image data, obtaining first image data, performing residual vector quantization on the first image data, and obtaining second image data that is unified with the first text data;
[0009] splicing the second image data and the first text data to obtain preprocessed data, performing autoregressive prediction on the preprocessed data using a large language model, and obtaining next feature data of the preprocessed data;
[0010] Extracting first image feature data from the next feature data, and reconstructing visual features of the first image feature data using a decoder to obtain visual feature reconstructed data;
[0011] The U-Net model is used to decode the visual feature reconstruction data into the pixel space to obtain the second image feature data after filling the pixel details.
[0012] In a possible design, performing residual vector quantization on the first image data to obtain the second image data includes:
[0013] Performing a nearest neighbor search on the first image data according to a preset code book to obtain initial quantized data, wherein the preset code book includes a coding sequence number and an embedded representation;
[0014] Quantize depth according to preset The residual vector calculation of the initial quantized data is performed in a loop iteration to obtain the first image data at a quantization depth of 1 to The quantization result sequence when
[0015] The quantization result sequence is summed up to obtain the second image data.
[0016] Optionally, according to a preset quantization depth The formula for calculating the residual vector for loop iteration with the initial quantized data is as follows:
[0017] ,
[0018] in, is the quantization result of the quantized data when the quantization depth is d, For nearest neighbor search, is the residual when the quantized depth is d-1, is a preset code book;
[0019] ,
[0020] in, is the residual when the quantized depth is d, for The embedding in the codebook indicates that when the residual depth is 0, The initial quantitative data.
[0021] Optionally, when a non-differentiable problem occurs during the nearest neighbor search process, a stop gradient propagation method is used to stop updating the gradient update during the nearest neighbor search process.
[0022] In one possible design, the training objective of the decoder is defined as follows:
[0023] ,
[0024] in is the i-th feature data in the first image feature data, is the i-th feature data in the real visual embedding in the multimodal data, Calculates the cosine similarity function.
[0025] In a possible design scheme, the U-Net model is used to decode the visual feature reconstruction data into the pixel space to obtain the second image feature data after filling the pixel details, including:
[0026] Gaussian noise is added during the training process of the U-Net model to conditionally denoise the visual feature reconstruction data and obtain the second image feature data after filling in pixel details.
[0027] Optionally, the optimization function of the U-Net model during training is as follows:
[0028] ,
[0029] in, is the expected function, is Gaussian noise, Reconstruct the latent state of the data during the decoding diffusion process for visual features, For time, Reconstructing data for visual features.
[0030] In a possible design scheme, the ViT model is used to extract features from image data to obtain first image data, including:
[0031] The image data is divided into non-overlapping image blocks according to the image data size and the preset image block size, and the ViT model is used to perform serialization feature extraction on the non-overlapping image blocks to obtain first image data containing features of different image blocks.
[0032] In a possible design, the second image data and the first text data are spliced to obtain preprocessed data, including:
[0033] Insert an [IMG] at the beginning of the second image data as the start tag of the image data, and insert a [ / IMG] at the end as the end tag of the image data;
[0034] The concatenation form of the second image data and the first text data includes two forms: [image, text] and [text, image].
[0035] In a second aspect, a data processing device based on a multimodal large model is provided. The device comprises:
[0036] An acquisition module, used for acquiring multimodal data and extracting image data and text data from the multimodal data;
[0037] A text preprocessing module, used for serializing and encoding text data according to a preset dictionary to obtain first text data;
[0038] An image preprocessing module, used to extract features from image data using a ViT model, obtain first image data, perform residual vector quantization on the first image data, and obtain second image data that is unified with the first text data;
[0039] A prediction module, used for splicing the second image data and the first text data to obtain preprocessed data, performing autoregressive prediction on the preprocessed data using a large language model, and obtaining next feature data of the preprocessed data;
[0040] A visual feature reconstruction module is used to extract first image feature data from the next feature data, and use a decoder to perform visual feature reconstruction on the first image feature data to obtain visual feature reconstruction data;
[0041] The decoding module is used to decode the visual feature reconstruction data into the pixel space using the U-Net model to obtain the second image feature data after filling the pixel details.
[0042] The present invention provides a data processing method and device based on a multimodal large model. The method includes: acquiring multimodal data, extracting image data and text data from the multimodal data, performing feature extraction and residual vector quantization on the image data using a ViT model, acquiring second image data that is unified with the first text data after serialization encoding, then splicing the second image data and the first text data, performing autoregressive prediction using a large language model, acquiring next feature data, then extracting first image feature data from the next feature data, performing visual feature reconstruction on the first image feature data, and then decoding the visual feature reconstruction data into a pixel space to acquire second image feature data after filling in pixel details. That is to say, by performing feature extraction and residual vector quantization on the image data, the dimensions of the second image data and the first text data are unified, thus solving the problem of inconsistent representation of multimodal data. Then, a large language model is used for autoregressive prediction to obtain the next feature data, extract the first image feature data from the next feature data, perform visual feature reconstruction on the first image feature data, and then decode the visual feature reconstruction data into the pixel space to obtain the second image feature data after filling in the pixel details. This solves the problem of missing image data after the image data and text data are unified, thus ensuring the integrity of the final predicted image data.
[0043] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.
[0045] Figure 1 Flow chart of a data processing method based on a multimodal large model according to an embodiment of the present invention;
[0046] Figure 2 A flowchart of residual vector quantization according to an embodiment of the present invention;
[0047] Figure 3 This is a schematic diagram of the structure of the large-scale language model autoregressive prediction according to an embodiment of the present invention;
[0048] Figure 4 It is a structural schematic diagram of a data processing method based on a multimodal large model according to an embodiment of the present invention;
[0049] Figure 5 It is a schematic diagram of the structure of a data processing device based on a multimodal large model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0050] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.
[0051] It should be noted that the description of the embodiments of the present invention is only for more clearly illustrating the technical solutions of the embodiments of the present invention, and does not constitute a limitation on the technical solutions provided by the embodiments of the present invention.
[0052] The embodiment of the present invention provides a data processing method based on a multimodal large model. Figure 1 Flow chart of a data processing method based on a multimodal large model according to an embodiment of the present invention. Figure 1 , the method comprising:
[0053] S101. Acquire multimodal data, and extract image data and text data from the multimodal data.
[0054] In this embodiment, the multimodal data includes image data and text data. The image data and text data are extracted from the multimodal data respectively to facilitate unified encoding.
[0055] S102. Serialize and encode the text data according to a preset dictionary to obtain first text data.
[0056] Among them, the text serialization encoding process includes word segmentation, vector conversion, padding and embedding generation. First, the word segmentation uses a word segmenter to decompose the original text data into smaller components, and extracts words, subwords or characters in the text data. These smaller components are called tokens. Then, the tokens are converted into their corresponding ID vectors according to the preset dictionary. The preset dictionary contains all possible tokens and their corresponding ID vectors. If the length of the converted ID vector does not meet the requirements of the model, the padding marker (usually 1) is used to pad the ID vector to the required length to ensure that the dimensions of all text input data are consistent for batch processing. Finally, the ID vector is converted into a high-dimensional embedding vector by looking up the embedding matrix, mapping tokens to high-dimensional vectors to capture the semantic relationship between tokens. Among them, the embedding matrix is a learned matrix with the number of rows being the size of the preset dictionary and the number of columns being the dimension of the embedding vector to be generated.
[0057] For example, the original text data is "you are gorgeous!", and the tokens obtained after word segmentation are ["you", "are", "gorge", "ous", "!"], and then the ID vector converted according to the preset dictionary can be "[[CLS], 462, 7832, 54637, 4536, 2342, [SEP]]", and the ID vector after filling can be "[[CLS], 462, 7832, 54637, 4536, 2342, [SEP],1,1,1,1…]", and finally, through the embedding matrix search, a continuous high-dimensional embedding vector, that is, the first text data, can be obtained.
[0058] In this embodiment, by serializing and encoding text data through a preset dictionary, the text data can be effectively converted into a format that can be processed and calculated by a multimodal model, providing a better basis for subsequent understanding and generation of multimodal data.
[0059] S103. Use the ViT model to extract features from the image data to obtain first image data, perform residual vector quantization on the first image data, and obtain second image data that is unified with the first text data.
[0060] Among them, the ViT model is based on the Transformer model. By dividing the image data into image blocks of fixed size and processing the image block data as a sequence, it can capture the global information of the image data more effectively. The ViT model mainly includes four key components: image segmentation, linear projection, position encoding, and Transformer encoder. First, the input image data is divided into multiple image blocks, and each image block is projected into an embedding space of fixed dimension through a linear layer. Then, position encoding is added to store the position information of the image block data. Finally, the embedded sequence information is input into the standard Transformer encoder for encoding processing to obtain feature encoding data.
[0061] In this embodiment, feature extraction and residual vector quantization are used to preprocess image data. While capturing the complex patterns and structures in the image, the features of the image data are effectively compressed and optimized to obtain second image data that is unified with the first text data, thereby solving the problem of inconsistent representation between image data and text data.
[0062] Optionally, the ViT model is used to extract features from the image data to obtain first image data, including:
[0063] The image data is divided into non-overlapping image blocks according to the image data size and the preset image block size, and the ViT model is used to perform serialization feature extraction on the non-overlapping image blocks to obtain first image data containing features of different image blocks.
[0064] Furthermore, given an image data , can be divided into non-overlapping image blocks, where H is the height of the image data, W is the width of the image data, C is the number of channels of the image data, and P is the size of the image block to be segmented. For example, if the size of the image data is 224×224×3 and the size of the image block to be segmented is 16, it can be segmented into 14×14 non-overlapping image blocks.
[0065] In this embodiment, by dividing the image data into non-overlapping image blocks, the local features in each image block can be effectively captured, helping the model to better understand the details and structure of the image.
[0066] Figure 2 Flow chart of residual vector quantization of an embodiment of the present invention, as shown in Figure 2 shown.
[0067] Optionally, performing residual vector quantization on the first image data to obtain second image data includes:
[0068] Performing a nearest neighbor search on the first image data according to a preset code book to obtain initial quantized data, wherein the preset code book includes a coding sequence number and an embedded representation;
[0069] Quantize depth according to preset The residual vector calculation of the initial quantized data is performed in a loop iteration to obtain the first image data at a quantization depth of 1 to The quantization result sequence when
[0070] The quantization result sequence is summed up to obtain the second image data.
[0071] The preset codebook is a set of predefined representative embedding representations, each of which has a unique coding sequence number. The preset codebook includes the coding sequence number and the embedding representation. For example, the preset codebook set C can be expressed as , k is the coding number, is the embedding representation corresponding to the encoding number, and K is the number of embedding representations in the codebook.
[0072] Nearest neighbor search is a method for finding the data point that is most similar or closest to a given query point in a data set. In the present invention, it is a method for finding the embedding representation that is most similar to the image data in a preset code book. By searching for the embedding representation that is closest to the image data in the preset code book, the image data is quantized to achieve effective image encoding and compression. The formula for nearest neighbor search is as follows:
[0073] ,in The image data needs to be quantified.
[0074] In addition, when performing the nearest neighbor search, the embedded representation in the preset codebook is also gradient updated so that the preset codebook can better cover and represent the image data.
[0075] In this embodiment, the nearest neighbor search is performed on the first image data using a preset code book, which reduces the computational complexity in the image data encoding process and achieves optimal encoding of the image data. Then, through the iterative residual vector calculation and the summation calculation of the quantization result sequence, the accuracy of the image data quantization is gradually improved, and the feature loss of the image data in the quantization process is reduced.
[0076] Optionally, according to a preset quantization depth The formula for calculating the residual vector for loop iteration with the initial quantized data is as follows:
[0077] ,
[0078] in, is the quantization result of the quantized data when the quantization depth is d, For nearest neighbor search, is the residual when the quantized depth is d-1, is a preset code book;
[0079] ,
[0080] in, is the residual when the quantized depth is d, for The embedding in the codebook indicates that when the residual depth is 0, The initial quantitative data.
[0081] Furthermore, according to the above formula, the first image data is quantized from 1 to The quantization result sequence is:
[0082] ,
[0083] The second image data obtained by summing the quantization result sequence can be expressed as:
[0084] .
[0085] In this embodiment, the residual vector is calculated through cyclic iteration to continuously correct the result of the previous residual quantization, reduce the quantization error, and improve the quality of the second image data. Finally, the quantization result sequence is summed up and the residual vectors in these quantization result sequences are fused into a globally optimal residual vector, thereby retaining the image details in the second image data and further improving the image quality.
[0086] Optionally, when a non-differentiable problem occurs during the nearest neighbor search process, a stop gradient propagation method is used to stop updating the gradient update during the nearest neighbor search process.
[0087] The principle of the nearest neighbor search is to find the embedding representation that is most similar to the image data in the preset code book. Since this process is a discrete process based on distance. For a given image data, the nearest neighbor is determined based on the distance comparison results of all possible embedding representations. Once the vector representation of the image data changes slightly, it may cause a mutation in the selection of the nearest neighbor, that is, a sudden jump from one embedding representation to another. This mutation means that the nearest neighbor search function is discontinuous at some points and is therefore non-differentiable at these points. When the data dimension is high, this distance calculation and the selection of the nearest neighbor embedding representation will be more complicated, and the impact of the non-differentiable problem will be more serious when performing the training optimization of the nearest neighbor search. Therefore, the stop gradient propagation method is chosen to calculate the gradient only for the image data that can be differentiated, and not to calculate the gradient of the image data that has the non-differentiable problem. The nearest neighbor search is optimized and updated, which simplifies the gradient calculation in the nearest neighbor search process.
[0088] In this embodiment, the stop gradient propagation method is used to solve the non-differentiable problem that occurs in the nearest neighbor search process, thereby ensuring the stability of the nearest neighbor search and improving the computational efficiency and memory utilization.
[0089] S104. Concatenate the second image data and the first text data to obtain preprocessed data, use a large language model to perform autoregressive prediction on the preprocessed data, and obtain next feature data of the preprocessed data.
[0090] Figure 3 Schematic diagram of the structure of the large language model autoregressive prediction of the embodiment of the present invention. Figure 3 shown.
[0091] Among them, the autoregressive prediction of the large language model is mainly used to process time series information. Based on the existing sequence information and time series information, the next feature data is predicted, the context information and feature structure of the input multimodal data are fully understood, and the complete next feature data is constructed. The probability of each next feature data sequence generated by the autoregressive prediction of the large language model in the present invention can be expressed as:
[0092] ,
[0093] in, is the next feature data sequence to be predicted, S is the number of features in the next feature data sequence, is the i-th feature in the next feature data sequence to be predicted.
[0094] In this embodiment, the text data and image data after dimension unification are spliced and input into a large language model for autoregressive prediction, which can ensure the generation of more coherent and natural feature data and enhance the coherence and logic of the generated results.
[0095] Optionally, the second image data and the first text data are spliced to obtain preprocessed data, including:
[0096] Insert an [IMG] at the beginning of the second image data as the start tag of the image data, and insert a [ / IMG] at the end as the end tag of the image data;
[0097] The concatenation form of the second image data and the first text data includes two forms: [image, text] and [text, image].
[0098] In this embodiment, inserting a start tag and an end tag at the beginning and end of the image data respectively helps the model to more accurately identify and process the image data and avoid confusion with the text data. In addition, the two splicing forms help the model to better understand the relationship between the image and the text, thereby improving the effect of multimodal learning.
[0099] S105. Extract the first image feature data from the next feature data, and use the decoder to reconstruct the visual features of the first image feature data to obtain visual feature reconstructed data.
[0100] The main function of the decoder is to convert the low-dimensional first image feature data back to the dimensional space of the original image data. The decoder used in the present invention includes but is not limited to a fully connected decoder, a recurrent neural network decoder and a Transformer decoder.
[0101] In this embodiment, the decoder is used to reconstruct the visual features of the first image feature data, so as to enhance the predicted first image feature data and improve the quality of the first image feature data.
[0102] Optionally, the training objective of the decoder is defined as follows:
[0103] ,
[0104] in is the i-th feature data in the first image feature data, is the i-th feature data in the real visual embedding in the multimodal data, Calculates the cosine similarity function.
[0105] In this embodiment, the decoder is optimized using cosine similarity during the training process, and the decoder is continuously adjusted to learn the direction information of the first image feature data, which helps to capture more robust important information in the first image feature data.
[0106] S106. Use the U-Net model to decode the visual feature reconstruction data into the pixel space to obtain the second image feature data after filling the pixel details.
[0107] Among them, the U-Net model is a convolutional neural network architecture widely used in the field of images, which consists of a contraction path for downsampling and an expansion path for upsampling. The structure is "U" shaped, with a contraction path on the left and an expansion path on the right. The contraction path is responsible for feature extraction, including a series of convolutional layers and maximum pooling layers, aiming to capture context and spatial information from the input image data. The expansion path is responsible for feature reconstruction, including a series of convolutional layers and upsampling layers, aiming to generate a high-resolution output image from the feature image data. In addition, there is a jump connection mechanism between the contraction path and the expansion path, which can fuse image feature information of different scales, so that the U-Net model can focus on the global structure and local details of the image at the same time. In the present invention, the visual feature reconstruction data is input into the U-Net model, and the convolutional layer and the maximum pooling layer in the contraction path extract features from the input image in turn, and then the convolutional layer and the upsampling layer in the expansion path continue to reconstruct the features, and finally the feature map in the contraction path is connected to the feature map in the expansion path by the jump connection mechanism to obtain the second image feature data.
[0108] In this embodiment, the U-Net model is used to decode the visual feature reconstruction data into the pixel space, so as to better restore the image details, enrich the pixel details of the visual feature reconstruction data, and enhance the quality and pixel details of the second image feature data.
[0109] Optionally, the visual feature reconstruction data is decoded into a pixel space using a U-Net model to obtain second image feature data after filling in pixel details, including:
[0110] Gaussian noise is added during the training process of the U-Net model to conditionally denoise the visual feature reconstruction data and obtain the second image feature data after filling in pixel details.
[0111] Gaussian noise is also called normal distribution noise, and its characteristic is that it obeys normal distribution. Conditional denoising is performed on the visual feature reconstruction data, that is, during the training process of the U-Net model, randomly generated random values with normal distribution are added to the visual feature reconstruction data, and image data with Gaussian noise added is obtained, thereby forcing the model to learn robust features in the image feature data.
[0112] In this embodiment, by adding Gaussian noise during the training process of the U-Net model, conditional denoising is performed, which effectively enhances the robustness and detail recovery ability of the U-Net model.
[0113] Optionally, the optimization function of the U-Net model during training is as follows:
[0114] ,
[0115] in, is the expected function, is Gaussian noise, Reconstruct the latent state of the data during the decoding diffusion process for visual features, For time, Reconstructing data for visual features.
[0116] Figure 4 Schematic diagram of the structure of the data processing method based on the multimodal large model according to an embodiment of the present invention. Figure 4 shown.
[0117] First, the image data is subjected to feature extraction and residual vector quantization to obtain the second image data. At the same time, the text is serialized and encoded to obtain the first text data. The dimensions of the second image data and the first text data are unified. Then, the second image data and the first text data are spliced to form preprocessed data, which are input into a large language model for autoregressive prediction to obtain the next feature data. In order to restore the image data compression information in the predicted multimodal data, the first image feature data is extracted from the next feature data, and the visual features of the first image feature data are sequentially reconstructed and decoded into the pixel space to obtain the complete second image feature data. The unification of multimodal data is achieved, and a balanced treatment is adopted for image data and text data. At the same time, the loss of image information caused by residual vector quantization is reduced as much as possible, ensuring the quality of the image feature data in the next multimodal data generated by the prediction.
[0118] Based on the data processing method based on a multimodal large model provided in this embodiment, multimodal data can be obtained, image data and text data can be extracted from the multimodal data, and the ViT model can be used to perform feature extraction and residual vector quantization on the image data to obtain second image data that is unified with the first text data after serialization encoding, and then the second image data and the first text data are spliced, and autoregressive prediction is performed using a large language model to obtain the next feature data, and then the first image feature data is extracted from the next feature data, and visual features are reconstructed for the first image feature data, and the visual feature reconstruction data is decoded into the pixel space to obtain the second image feature data after filling in pixel details. That is to say, by performing feature extraction and residual vector quantization on the image data, the dimensions of the second image data and the first text data are unified, thus solving the problem of inconsistent representation of multimodal data. Then, a large language model is used for autoregressive prediction to obtain the next feature data, extract the first image feature data from the next feature data, perform visual feature reconstruction on the first image feature data, and then decode the visual feature reconstruction data into the pixel space to obtain the second image feature data after filling in the pixel details. This solves the problem of missing image data after the image data and text data are unified, thus ensuring the integrity of the final predicted image data.
[0119] Combination of the above Figure 1-Figure 4 The data processing method based on the multimodal large model provided in the embodiment of the present application is described in detail. Figure 5 A data processing device based on a multimodal large model for executing the data processing method based on a multimodal large model provided in an embodiment of the present application is described in detail.
[0120] Figure 5 FIG. 1 is a schematic diagram of the structure of a data processing device based on a multimodal large model according to an embodiment of the present invention. Figure 5 , the device comprises:
[0121] An acquisition module 501 is used to acquire multimodal data and extract image data and text data from the multimodal data;
[0122] The text preprocessing module 502 is used to serialize and encode the text data according to a preset dictionary to obtain first text data;
[0123] An image preprocessing module 503 is used to extract features from image data using a ViT model to obtain first image data, perform residual vector quantization on the first image data, and obtain second image data that is unified with the first text data;
[0124] A prediction module 504 is used to concatenate the second image data and the first text data to obtain preprocessed data, and to perform autoregressive prediction on the preprocessed data using a large language model to obtain next feature data of the preprocessed data;
[0125] The visual feature reconstruction module 505 is used to extract the first image feature data from the next feature data, and use the decoder to perform visual feature reconstruction on the first image feature data to obtain visual feature reconstruction data;
[0126] The decoding module 506 is used to decode the visual feature reconstruction data into the pixel space using the U-Net model to obtain the second image feature data after filling the pixel details.
[0127] The technical effects of the data processing device based on the multimodal large model refer to the technical effects of the data processing method based on the multimodal large model mentioned above, and will not be repeated here.
[0128] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0129] The units or modules involved in the embodiments of the present invention may be implemented by software or hardware. The units or modules described may also be arranged in a processor, for example, may be described as: a processor includes an acquisition module 501, a text preprocessing module 502, an image preprocessing module 503, a prediction module 504, a visual feature reconstruction module 505 and a decoding module 506, wherein the names of these modules do not constitute a limitation on the modules themselves in some cases.
[0130] An embodiment of the present invention provides a computer-readable storage medium, which may be a computer-readable storage medium included in the data processing device based on the multimodal large model described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into an electronic device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the quality assessment model training method described in the present invention.
[0131] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.
Claims
1. A data processing method based on a multimodal large model, characterized in that: The method comprises: Acquire multimodal data, and extract image data and text data from the multimodal data; Serialize and encode the text data according to a preset dictionary to obtain first text data; Using the ViT model to extract features from the image data to obtain first image data, performing residual vector quantization on the first image data to obtain second image data that is unified with the first text data; splicing the second image data and the first text data to obtain preprocessed data, performing autoregressive prediction on the preprocessed data using a large language model, and obtaining next feature data of the preprocessed data; Extracting first image feature data from the next feature data, and reconstructing visual features of the first image feature data using a decoder to obtain visual feature reconstructed data; The visual feature reconstruction data is decoded into a pixel space using a U-Net model to obtain second image feature data after filling in pixel details.
2. The data processing method based on multimodal large model according to claim 1 is characterized in that: The performing residual vector quantization on the first image data to obtain second image data unified with the first text data includes: Performing a nearest neighbor search on the first image data according to a preset code book to obtain initial quantized data, wherein the preset code book includes a coding sequence number and an embedded representation; Quantize depth according to preset The residual vector calculation of the initial quantized data is performed in a loop iteration to obtain the first image data at a quantization depth of 1 to The quantization result sequence when A sum calculation is performed on the quantization result sequence to obtain the second image data.
3. The data processing method based on multimodal large model according to claim 2 is characterized in that: The preset quantization depth The formula for calculating the residual vector by loop iteration with the initial quantized data is as follows: , in, is the quantization result of the quantized data when the quantization depth is d, For nearest neighbor search, To quantize the residual when the depth is d-1, is the preset code book; , in, is the residual when the quantized depth is d, for The embedding in the codebook indicates that when the residual depth is 0, is the initial quantitative data.
4. The data processing method based on multimodal large model according to claim 2 is characterized in that: When a non-differentiable problem occurs in the nearest neighbor search process, a stop gradient propagation method is used to stop updating the gradient update in the nearest neighbor search process.
5. The data processing method based on multimodal large model according to claim 1 is characterized in that: The training objective of the decoder is defined as follows: , in is the i-th feature data in the first image feature data, is the i-th feature data in the real visual embedding in the multimodal data, Calculates the cosine similarity function.
6. The data processing method based on multimodal large model according to claim 1 is characterized in that: The method of decoding the visual feature reconstruction data into a pixel space using a U-Net model to obtain second image feature data after filling pixel details includes: Gaussian noise is added during the training process of the U-Net model, and conditional denoising is performed on the visual feature reconstruction data to obtain the second image feature data after the pixel details are filled.
7. The data processing method based on multimodal large model according to claim 6 is characterized in that: The optimization function of the U-Net model during training is as follows: , in, is the expected function, is Gaussian noise, reconstructing the potential state of the data in the decoding diffusion process for the visual features, For time, Data is reconstructed for the visual features.
8. The data processing method based on multimodal large model according to claim 1 is characterized in that: The step of extracting features from the image data using the ViT model to obtain first image data includes: The image data is divided into non-overlapping image blocks according to the image data size and the preset image block size, and the ViT model is used to perform serialized feature extraction on the non-overlapping image blocks to obtain the first image data containing features of different image blocks.
9. The data processing method based on multimodal large model according to claim 1 is characterized in that: The step of splicing the second image data and the first text data to obtain pre-processed data includes: Insert an [IMG] at the beginning of the second image data as a start tag of the image data, and insert a [ / IMG] at the end as an end tag of the image data; The concatenation form of the second image data and the first text data includes two forms: [image, text] and [text, image].
10. A data processing device based on a multimodal large model, characterized in that: The device comprises: An acquisition module, used to acquire multimodal data and extract image data and text data from the multimodal data; A text preprocessing module, used for serializing and encoding the text data according to a preset dictionary to obtain first text data; An image preprocessing module, used to extract features from the image data using a ViT model to obtain first image data, perform residual vector quantization on the first image data, and obtain second image data that is unified with the first text data; A prediction module, used for splicing the second image data and the first text data to obtain preprocessed data, performing autoregressive prediction on the preprocessed data using a large language model, and obtaining next feature data of the preprocessed data; A visual feature reconstruction module, used for extracting first image feature data from the next feature data, and performing visual feature reconstruction on the first image feature data using a decoder to obtain visual feature reconstruction data; A decoding module is used to decode the visual feature reconstruction data into a pixel space using a U-Net model to obtain second image feature data after filling in pixel details.
