Multi-modal data feature representation optimization method based on comparative learning and two-stage mask

Abstract

The invention discloses a multi-modal data feature representation optimization method based on comparative learning and two-stage masks. The method comprises the following steps of: firstly, aiming at an input single image-text pair, generating two groups of heterogeneous image-text data with mode missing as two inputs of a model by applying two random mask strategies with different mask areas and proportions; wherein the first group of data applies a high-proportion mask to the image and applies a low-proportion mask to the text; the second group applies a low-scale mask to the image and a high-scale mask to the text. Then, the two groups of data respectively pass through an image encoder and a text encoder which share parameters, and two different multi-modal fusion feature vectors are generated through a cross-modal fusion encoder; according to the method, masked modal information is recovered through a decoder, and the reconstruction loss of the difference between a recovery result and original data is calculated. Meanwhile, two multi-modal fusion feature vectors generated twice are subjected to comparative learning, and the comparative loss of feature representation distances under different enhanced views for approaching the same image-text pair is calculated. And finally, performing weighted summation on the reconstruction loss and the comparison loss to form an overall loss function, and optimizing model parameters through back propagation. When the trained encoder is used for a downstream multi-modal classification task, the classification precision and generalization ability of the model can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of multi-modal machine learning and deep learning, and particularly relates to a method for optimizing multi-modal data feature representation, in particular a method for recovering missing modal information and improving downstream task performance by combining contrastive learning and a two-stage masking strategy. BACKGROUND

[0002] With the rapid development of information technology, we are in an era surrounded by multi-modal data such as images, texts, speeches, and videos. Multi-modal data has a more comprehensive representation ability than single-modal data because it can provide information from different dimensions, and has shown great application potential in medical analysis, autonomous driving, sentiment analysis, and other fields.

[0003] One of the core challenges of multi-modal learning is how to effectively fuse information from different modalities to generate a unified and high-quality feature representation. Existing multi-modal representation learning methods, such as fusion representation, shared representation, and alignment representation, have made significant progress. However, in practical applications, data acquisition processes often result in partial or even complete missing of some modal data due to sensor failure, network problems, or incomplete information provided by users. This modality missing problem can severely degrade the quality of the fused feature representation, and thus limit the accuracy and reliability of downstream tasks such as classification and retrieval.

[0004] To address the modality missing problem, early research work mainly focuses on using a single complete modality to recover another completely missing modality. Existing methods mostly focus on feature representation and fusion, and pay less attention to the impact of partial modality missing and the corresponding recovery strategies. In addition, these methods often ignore the complementary role of intra-modal context information for the recovery task, resulting in insufficient details and fidelity of the recovered information. For example, traditional multi-modal completion models usually only focus on the recovery of a single modality, ignoring the important associated information that other co-existing modalities can provide.

[0005] Therefore, there is an urgent need for a new method that can not only handle the complex scenario of partial modality missing, but also fully utilize the intra-modal and inter-modal context information to generate more robust and accurate multi-modal feature representations to cope with the challenge of incomplete data in the real world. SUMMARY

[0006] In order to overcome the above-mentioned prior art, the purpose of the present application is to provide a multi-modal data feature representation optimization method based on contrast learning and double-stage mask. This method aims to solve the problem that the existing model cannot fully utilize the context information when processing partial modal missing data, thereby improving the feature quality of the recovered data and significantly improving the accuracy and generalization ability of the downstream classification task.

[0007] In order to achieve the above-mentioned purpose, the technical solution adopted by the present application is:

[0008] A multi-modal data feature representation optimization method based on contrast learning and double-stage mask, comprising the following steps:

[0009] Step 1, twice different proportion mask operation is performed on the input image text pair:

[0010] For the input single image-text pair, two groups of heterogeneous image-text data with modal missing are generated by applying two different mask regions and random mask strategies with different proportions; the first group of data applies a high proportion of random pixel block mask to the image, while the text data remains intact; the second group of data applies a low proportion of random pixel block mask to the same image, and randomly replaces or masks the text, which serves as the two heterogeneous inputs of the model;

[0011] Step 2, encoding and cross-modal fusion are performed on the processed image text pair:

[0012] The above two groups of heterogeneous image-text data are sent into the image encoder and text encoder with shared parameters respectively to generate the image and text feature vectors of each group; then, the feature vectors of each group are spliced and input into the cross-modal fusion encoder, and deep fusion is performed through the cross-attention mechanism, finally obtaining two different multi-modal fusion feature vectors M1 and M2;

[0013] Step 3, mask reconstruction is performed on the image and text:

[0014] The multi-modal fusion feature vectors M1 and M2 are sent into the corresponding decoder respectively to recover the masked modal information in step 1; based on the difference between the recovery result and the original data, the reconstruction loss is calculated, wherein the image recovery adopts mean square error loss and the text recovery adopts cross-entropy loss;

[0015] Step 4, contrast learning is performed using the global feature after modal fusion:

[0016] The two generated multi-modal fusion feature vectors M1 and M2 are subjected to contrast learning; by calculating a contrast loss function based on the improved CLIP, the distance between the feature representations of the same image-text pair under different enhanced views is narrowed, while the distance between the features of other negative sample pairs in the batch is pushed away;

[0017] Step 5, multi-loss joint optimization;

[0018] The reconstruction loss and the contrast loss obtained by the above calculation are weighted and summed to form a unified overall loss function, and all parameters in the model (including the encoder, the decoder and the fusion module) are optimized and trained end-to-end through the back propagation algorithm;

[0019] Step 6, downstream task fine-tuning and verification;

[0020] After training, the images, texts and cross-modal fusion encoders obtained by training are used for downstream tasks; the specific method is to add a classification head at the output end of the fusion encoder, and train the classification head using the labeled data set of a specific task, and finally verify the effectiveness of the multi-modal feature representation generated by the present application through indicators such as classification accuracy and F1 value.

[0021] The two different proportions of the input image text pair are masked, specifically including:

[0022] 1) When inputting for the first time, high proportion of image mask and low proportion of text mask are used to mask the input image and text pair respectively to generate Mask img1 and Mask text1 . This is to force the model to understand the scene content by relying on the text modality and try to restore the image content. In this stage, the corresponding text can be applied with a lower proportion of mask as a strong contextual clue for restoring the image.

[0023] 2) When inputting for the second time, low proportion of image mask and high proportion of text mask are used to mask the same group of image and text to generate Mask img2 and Mask text2 , for example, 30% and 70% mask proportion. This input provides relatively complete visual information and slightly incomplete text information. It aims to force the model to understand the scene content by relying on the image modality and try to restore the text content.

[0024] This artificially constructed asymmetric information ensures that the model focuses on learning a one-way cross-modal dependency relationship at each stage, avoiding the tendency of the model to take a "shortcut" or ignore the other modality when the single modality information is sufficient. More importantly, the deep and effective feature vector representation of the same image-text pair learned from the two perspectives, although derived from different input states and inference paths, should ideally point to the core meaning in the same cross-modal joint semantic space.

[0025] The processed image-text pair is encoded and cross-modal fused, specifically including:

[0026] The first set of image-text data is sent into the image encoder and the text encoder respectively, generating the first set of image feature vectors and text feature vectors; then, the set of feature vectors is input into the cross-modal fusion encoder to obtain the multi-modal fusion feature vector M1; the second set of image-text data is sent into the image encoder and the text encoder shared with the parameters in step 2 respectively, generating the second set of image feature vectors and text feature vectors; then, the set of feature vectors is input into the cross-modal fusion encoder to obtain the multi-modal fusion feature vector M2.

[0027] 2) Use Vision Transformer as the encoder. First, the input image is divided into a series of non-overlapping image blocks, and each block is linearly projected into a vector. Then, position encoding is added to these vectors to preserve the spatial structure information of the image. Finally, this vector sequence is input into a multi-layer Transformer encoder to capture the global dependency between image blocks through self-attention mechanism, and finally output the encoded image feature vector:

[0028]

[0029] represents the feature vector encoded by the masked image at the i-th stage.

[0030] 2) Use BERT or similar Transformer encoder. The input text is first tokenized, and then each word is mapped into a word embedding vector. Similar to images, these vectors are also added with position encoding to preserve the word order information. The processed vector sequence is sent into a multi-layer Transformer encoder to capture the contextual semantic relationship through self-attention mechanism, and output the encoded text feature vector:

[0031]

[0032] represents the feature vector encoded by the masked text at the i-th stage.

[0033] 3) The encoded image feature y encoded and the text feature x encoded are concatenated in the sequence dimension to form a longer sequence. This fused sequence is sent into an independent, deeper or wider cross-modal Transformer encoder. Through the cross-attention layers inside the encoder, the features of image and text can interact and align with each other, so as to learn the cross-modal deep fusion representation:

[0034]

[0035] M iThe feature vector after cross-modal fusion for the i-th stage image-text pair.

[0036] The image and text are masked and reconstructed, and specifically, the method comprises the following steps:

[0037] The fused feature vectors M1 and M2 are sent to the corresponding decoder. The image decoder adopts an asymmetric structure and only needs to process the tokens of the masked part to reconstruct the original image pixels, and the loss function is the mean square error. The text decoder is responsible for predicting the masked text words, and the loss function is the cross-entropy loss.

[0038] 3) The fused features M1 and M2 are sent to a relatively light image decoder. The decoder receives all tokens (including the features of the visible part and the special mask token [MASK] representing the masked area), and predicts the original pixel values of the masked area through a multi-layer Transformer decoder. The loss function adopts the mean square error, which directly calculates the L2 distance between the reconstructed pixels and the original pixels, which is effective for restoring continuous image signals:

[0039]

[0040] L img represents the image reconstruction loss, and N represents the total number of masked image blocks, represents the predicted pixel value of the i-th image block of the model, y i represents the true pixel value of the i-th image block.

[0041] 4) The fused features M1 and M2 are sent to the text decoder. The text decoder is also composed of multiple layers of Transformer, and its task is to predict the original ID of the words that are masked or replaced when input. This is a classification problem, so the loss function adopts the cross-entropy loss, which calculates the difference between the probability distribution of the words predicted by the model and the true words:

[0042]

[0043] L text represents the text reconstruction loss, and N represents the total number of masked words, w t represents the original true word at the t-th position, q(w t ) represents the distribution of the true word, and p(w t ) represents the probability predicted by the model.

[0044] 3) The image reconstruction loss and the text reconstruction loss are combined together by weighted summation to guide the model to learn the ability to restore different modal information.

[0045] The global features after modal fusion are used for contrastive learning, and specifically, the method comprises the following steps:

[0046] 1) From the feature vectors M1 and M2 obtained after two fusions, extract the specific tokens representing the global semantics, usually the output vectors corresponding to the [CLS] token and These two vectors are considered as two different representations of the same image-text pair in the shared feature space.

[0047] 2) In a training batch, collect all the and vectors generated by all image-text pairs. L2-normalize all these vectors, the formula is:

[0048]

[0049] represents the normalized result of the [CLS] token corresponding to the i-th stage image-text pair.

[0050] 3) Then calculate their cosine similarity between each other, forming a similarity matrix. In this matrix, the similarity between and from the same original image-text pair is considered as the similarity of the "positive sample pair", while the similarity with the feature vectors of any other image-text pair is considered as the similarity of the "negative sample pair".

[0051] 4) This method uses InfoNCE loss, which is the core contrastive loss function used in models such as CLIP. Its goal is to maximize the similarity of positive sample pairs, i.e. let their cosine similarity approach 1, while minimizing the similarity with all negative sample pairs in the batch. This is achieved by projecting the feature vectors of the two different modalities of images and texts into the same feature vector space to compare and match similarities. The form of the loss function is as follows:

[0052]

[0053] L contrast represents the contrastive learning loss, sim is the cosine similarity, τ is an adjustable temperature hyperparameter, used to control the smoothness of the similarity distribution, K is the number of all negative samples in the batch.

[0054] The multi-loss joint optimization specifically includes:

[0055] The reconstruction loss and the contrastive loss are weighted and summed to form the overall loss function, which is a combined loss function used to drive the final model training, and the parameters of the encoder, decoder and fusion encoder are optimized and trained end-to-end through the backpropagation algorithm. Specifically, it includes:

[0056] 1) Reconstruction loss of the image after high-ratio masking in the first stage L img1 .

[0057] 2) Reconstruction loss of the text after low-ratio masking in the first stage L text1 .

[0058] 3) Reconstruction loss of the image after high-ratio masking in the second stage L img2 .

[0059] 4) Reconstruction loss of the text after low-ratio masking in the second stage L text2 .

[0060] 5) Contrastive loss between global features generated by two inputs L contrast .

[0061] Weighted fusion of the five types of loss functions:

[0062] L total = α1L img1 + α2L text1 + α3L img2 + α4L text2 + α5L contrast (8) L total represents the final joint loss, where α1, α2, α3, α4, α5 are balanced learnable parameters. By calculating L total optimize all parameters of the model, including the image encoder, the text encoder, the fusion encoder, the image decoder, and the text decoder. This ensures that the model learns to reconstruct while effectively aligning cross-modal features.

[0063] The downstream task fine-tuning and verification specifically includes:

[0064] 1) The pre-training stage is unsupervised and does not rely on any task labels. In order to verify its effectiveness, the learned feature representation needs to be applied to specific, supervised downstream tasks, and multi-modal classification is one of the most direct verification methods.

[0065] 2) After pre-training is complete, discard the decoder part. Connect the output end of the trained cross-modal fusion encoder to a simple classification head. Then, use specific task datasets with class labels, such as MM-IMDB (movie posters and type classification) and UPMC-FOOD101 (food pictures and category classification), to fine-tune the newly added classification head and optionally the entire encoder.

[0066] 3) To comprehensively evaluate the performance of the model in real complex environments, the verification method is to apply simulated real-world interference residual damage processing to the image data in the test set. This process aims to test the robustness of the model feature representation, and the specific methods include: applying Gaussian blur, motion blur and other filters to simulate the blur caused by camera defocus or fast motion; adding Gaussian noise to the image to simulate sensor or transmission signal interference noise processing; and through algorithm superposition of fog, rain effect or adjustment of brightness and contrast, visual effect interference processing is carried out to test the adaptability of the model under variable conditions. In verification, these images processed by one or more residual damage are paired with the original text data to form a new test set, and the classification accuracy, F1 value and other key performance indicators of the model are re-evaluated on this basis. Finally, by comparing the performance difference of the model on residual damage data and original clear data, its robustness can be quantified: the smaller the performance decline, the less sensitive the multi-modal feature representation learned by the method is to pixel-level interference, and the stronger the robustness.

[0067] The beneficial effects of the present application are:

[0068] The present application can more effectively handle the scenario of partial missing of multi-modal data through the innovative two-stage mask and contrast learning strategy. This method not only learns to recover the missing content, but also forces the model to capture deeper consistency relationships between modalities through contrast learning, thereby generating higher quality and more robust feature representations. Experiments on standard data sets such as MM-IMDB and UPMC-FOOD101 prove that the present application can significantly improve the classification accuracy and generalization ability of the model under data missing conditions, providing an effective solution to the problem of incomplete multi-modal data in the real world. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 Pre-training process schematic diagram;

[0070] Figure 2 Fine-tuning process schematic diagram;

[0071] Figure 3 Image mask process schematic diagram;

[0072] Figure 4 Contrast loss calculation process schematic diagram;

[0073] Figure 5 CLIP-based multi-modal loss function structure schematic diagram;

[0074] Figure 6 Overall method flowchart. DETAILED DESCRIPTION

[0075] This example is implemented on a computer equipped with an NVIDIA 2080Ti graphics card, 32GB of memory, using Python 3.8 and the PyTorch framework. The experimental data set uses MM-IMDB and UPMC-FOOD101.

[0076] Step one, this step is the core of data augmentation, aiming to provide positive sample pairs with the same semantics but different perspectives for subsequent contrastive learning. For each image-text pair (image y, text x) in a batch, perform two different mask operations to generate two sets of heterogeneous inputs:

[0077] 1) In the first stage, apply a high proportion of random block mask (e.g. 70%) to the image y to generate Mask image1 ; At the same time, apply a low proportion of random word mask or replacement (e.g. 30%) to the text x to generate Mask text1 . This setting aims to force the model to rely mainly on relatively complete text information to understand the scene and infer and reconstruct the heavily occluded image content accordingly.

[0078] 2) In the second stage, apply a low proportion of random block mask (e.g. 30%) to the same image y to generate Mask image2 ; At the same time, apply a high proportion of random word mask or replacement (e.g. 70%) to the same text x to generate Mask text2 This time, the model has relatively complete visual information and needs to rely on it to restore the severely missing text content.

[0079] This artificially constructed, asymmetric information input ensures that the model focuses on learning one-way cross-modal dependency at different stages, effectively avoiding the model's tendency to "shortcut learning" and ignore the other modality when single modality information is sufficient. More importantly, the feature vectors M1 and M2 generated by the two inputs, although derived from different inference paths, should ideally point to the same core meaning in the joint semantic space, providing ideal positive sample pairs for subsequent contrastive learning.

[0080] Step two, the two-stage model adopts an encoder with shared parameters, using Vision Transformer as the image encoder. The input masked image is first segmented into multiple image patches, then linearly embedded and positionally encoded. The encoder only processes the image patches that are not masked, outputting image feature vectors. Use BERT or similar Transformer structure as the text encoder. After the input text is word-embedded and positionally encoded, the text feature vector is generated by the encoder. After the image and text feature vectors are spliced, they are sent to the cross-modal fusion encoder. This encoder deeply fuses the information of the two modalities through a multi-layer cross-attention mechanism, and outputs multi-modal fusion feature vectors M1 and M2 respectively.

[0081] Step three, the multi-modal fusion feature vectors M1 and M2 are sent to the corresponding decoder respectively, and this step aims to train the model to recover the missing information through the self-supervised task.

[0082] 1) Image reconstruction: This process draws on the asymmetric design of the Masked Autoencoder (MAE). The fusion feature M1 or M2 is sent to a relatively lightweight image decoder. The decoder receives all the masked text, including the features of the visible part and the special mask token [MASK] representing the masked area, and predicts the original pixel values of the masked area through a multi-layer

[0083] Transformer decoder. The loss function uses Mean Squared Error (MSE) to directly calculate the L2 distance between the reconstructed pixels and the original pixels, the formula is:

[0084]

[0085] L img represents the image reconstruction loss, N represents the total number of masked image blocks, represents the predicted pixel value of the i-th image block by the model, y i represents the true pixel value of the i-th image block.

[0086] 2) Text reconstruction: The fusion feature M1 or M2 is also sent to a text decoder. The task of this decoder is to predict the original ID of the words that were masked or replaced when input. This is a multi-classification problem, so the loss function uses Cross-Entropy Loss to calculate the difference between the model's predicted word probability distribution and the true word:

[0087]

[0088] L text represents the text reconstruction loss, N represents the total number of masked words, w t represents the original true word at the t-th position, q(wt ) represents the distribution of real words, p(w t ) represents the probability predicted by the model.

[0089] Step four, this step aims to pull the representations of the same sample different views (M1 and M2) closer, and push the representations of different samples further apart.

[0090] 1) Extract a feature vector from M1 and M2 respectively, which can represent the whole, that is, the output vector of the [CLS] position and L2 normalize the two feature vectors. Calculate a cosine similarity matrix of all vector pairs in a batch. Use InfoNCE loss (a contrastive loss based on CLIP improvement) to calculate the contrastive loss L contrast . The goal is to maximize the similarity between the feature vectors generated by the same source image-text pair and , while minimizing their similarity with the feature vectors generated by other image-text pairs in the batch (negative samples).

[0091] 2) Use InfoNCE loss to optimize the feature space. The goal is to maximize the cosine similarity of positive sample pairs, while minimizing their similarity with all negative sample pairs in the batch. The loss function is as follows:

[0092]

[0093] L contrast represents the contrastive learning loss, sim is the cosine similarity, τ is an adjustable temperature hyperparameter, which controls the smoothness of the similarity distribution, and K is the number of all negative samples in the batch.

[0094] Step five, the final loss function used to drive model training is a total loss function that combines reconstruction and contrastive learning tasks, which is composed of five parts weighted by summing the above losses. The final total loss function is:

[0095] L total = α1L img1 + α2L text1 + α3L img2 + α4L text2 + α5L contrast (4)L total represents the final joint loss, where α1, α2, α3, α4, α5 are balance learnable parameters. By calculating L total optimize the gradient of all trainable parameters of the model, and use optimizers such as AdamW for backpropagation and parameter update, achieving the training of the entire model.

[0096] Step six, after the model is pre-trained, its encoder part has learned robust multi-modal feature representation capability. In order to verify its effectiveness, it is applied to the multi-modal classification task.

[0097] 1) After pre-training, the decoder part is discarded. A simple classification head (such as one or more fully connected layers plus a Softmax activation function) is connected at the output of the trained cross-modal fusion encoder. Then, the newly added classification head and optionally the entire encoder are fine-tuned using the specific task dataset MM-IMDB and UPMC-FOOD101 with class labels.

[0098] 2) In order to comprehensively evaluate the performance of the model in real complex environments, the image data in the test set is subjected to residual processing that simulates real-world interference. Specific methods include: applying Gaussian blur, motion blur to simulate out-of-focus or motion scenes; adding Gaussian noise to simulate sensor interference; superimposing fog, rain effects or adjusting brightness and contrast through algorithms to simulate variable visual conditions. These images after one or more residual processing are paired with the original text to form a new test set, and the classification accuracy, F1 value and other key performance indicators of the model are re-evaluated on this basis. By comparing the performance decline of the model on residual data and original clear data, its robustness can be quantified: the smaller the performance decline, the less sensitive the multi-modal features learned by the method to pixel-level interference, the stronger the robustness.

Claims

1. A multi-modal data feature representation optimization method based on contrastive learning and two-stage mask, characterized in that, Comprising the following steps: Step 1, twice different proportion mask operation is performed on the input image text pair: For the input single image-text pair, two groups of heterogeneous image-text data with missing modalities are generated by applying two kinds of mask regions and random mask strategies with different proportions; the first group of data applies a high proportion of random pixel block mask to the image, while the text data remains complete; the second group of data applies a low proportion of random pixel block mask to the same image, and randomly replaces or masks the text, which serves as the two heterogeneous inputs of the model; Step 2, encoding and cross-modal fusion are performed on the processed image text pair: The above two groups of heterogeneous image-text data are sent into the image encoder and text encoder sharing parameters respectively to generate image and text feature vectors of each group; then, the feature vectors of each group are spliced and input into the cross-modal fusion encoder, and deep fusion is performed through the cross attention mechanism, and finally two different multi-modal fusion feature vectors M1 and M2 are obtained; Step 3, mask reconstruction is performed on the image and text; The multi-modal fusion feature vectors M1 and M2 are sent into the corresponding decoder respectively to recover the masked modal information in step 1; based on the difference between the recovery result and the original data, the reconstruction loss is calculated, wherein the image recovery adopts mean square error loss, and the text recovery adopts cross entropy loss; Step 4, using the global feature after modal fusion for contrastive learning; The two generated multi-modal fusion feature vectors M1 and M2 are subjected to contrastive learning; by calculating a contrastive loss function improved based on CLIP, the distance between the feature representations of the same image-text pair under different enhanced views is narrowed, while the distance between the features of other negative sample pairs in the batch is pushed away; Step 5, multi-loss joint optimization; The reconstruction loss and the contrastive loss calculated above are weighted and summed to form a unified total loss function, and all parameters in the model (including the encoder, decoder and fusion module) are optimized and trained end-to-end through the back propagation algorithm; Step 6, downstream task fine-tuning and verification; After training, the image, text and cross-modal fusion encoder obtained by training are used for downstream tasks; the specific method is to append a classification head to the output end of the fusion encoder, and train the classification head using a specific task labeled data set, and finally verify the effectiveness of the multi-modal feature representation generated by the present application through classification accuracy, F1 value and other indicators.

2. The method of claim 1, wherein, The twice different proportion mask operation on the input image text pair specifically comprises: 1) At the first time of input, using high proportion of image mask and low proportion of text mask to respectively mask the input image and text pair to generate Mask img1 and Mask text1 ; 2) At the second input, apply corresponding mask processing to the same set of images and text using a low proportion of image mask and a high proportion of text mask to generate Mask img2 and Mask text2 .

3. The method of claim 1, wherein, The encoding and cross-modal fusion of the processed image text pair specifically comprises: The first group of image-text data is respectively sent into an image encoder and a text encoder to generate a first group of image feature vectors and text feature vectors; then, the group of feature vectors is input into a cross-modal fusion encoder to obtain a multi-modal fusion feature vector M1; the second group of image-text data is respectively sent into an image encoder and a text encoder sharing parameters with the image encoder and the text encoder in step 2 to generate a second group of image feature vectors and text feature vectors; then, the group of feature vectors is input into the cross-modal fusion encoder to obtain a multi-modal fusion feature vector M2; 1) Using a Vision Transformer as the encoder; first, the input image is divided into a series of non-overlapping image blocks, each block is linearly projected into a vector; then, position encoding is added to these vectors to preserve the spatial structure information of the image; finally, this vector sequence is input into a multi-layer Transformer encoder to capture the global dependency between image blocks through a self-attention mechanism, and finally output the encoded image feature vector: represents the encoded feature vector of the masked image at the i-th stage; 2) Using BERT or similar Transformer encoder; the input text is first segmented, and then each word is mapped into a word embedding vector; similar to images, these vectors are also added with position encoding to preserve the order information; the processed vector sequence is sent into a multi-layer Transformer encoder to capture the context semantic relationship through a self-attention mechanism, and output the encoded text feature vector: represents the feature vector encoded by the masked text at the i-th stage; 3) the encoded image features y encoded and the text features x encoded Concatenate along the sequence dimension to form a longer sequence; this fused sequence is fed into a separate, deeper or wider cross-modal Transformer encoder; this encoder enables the image and text features to interact, align with each other, and thus learn a cross-modal deep fused representation through its internal cross-attention layers: M i is the feature vector of the i-th stage image-text pair after cross-modal fusion.

4. The method of claim 1, wherein, The image and text are masked and reconstructed, specifically including: The fused feature vectors M1 and M2 are sent into corresponding decoders; the image decoder adopts an asymmetric structure and only needs to process the tokens of the masked part to reconstruct the original image pixels, and its loss function is mean square error; the text decoder is responsible for predicting the masked text words, and its loss function is cross-entropy loss, specifically including: 1) The fusion features M1 and M2 are sent into a relatively light image decoder; the decoder receives all tokens (including visible features and special mask tokens [MASK] representing the masked area), and predicts the original pixel values of the masked area through a multi-layer Transformer decoder; the loss function adopts mean square error, which directly calculates the L2 distance between the reconstructed pixels and the original pixels, which is effective for restoring continuous image signals: L img denotes the image reconstruction loss, N denotes the total number of masked image patches, denotes the predicted pixel value of the i-th image patch of the model, y i denotes the true pixel value of the i-th image patch; 2) The fusion features M1 and M2 are sent into a text decoder; the text decoder is also composed of multiple layers of Transformers, and its task is to predict the original ID of the word that is masked or replaced during input; This is a classification problem, so the loss function adopts cross-entropy loss to calculate the difference between the word probability distribution predicted by the model and the true word: L text represents the text reconstruction loss, N represents the total number of masked words, w t represents the original true word at the t-th position, q(w t ) represents the distribution of the true word, p(w t ) represents the probability predicted by the model; 3) The image reconstruction loss and the text reconstruction loss are combined together through weighted summation to guide the model to learn the ability to restore different modal information.

5. The method of claim 1, wherein, The global features after modal fusion are used for contrast learning, specifically including: 1) From the feature vectors M1 and M2 obtained after two fusions, respectively extract the specific token representing the global semantics, usually the output vector corresponding to the [CLS] token and These two vectors are considered as two different representations of the same image-text pair in a shared feature space; 2) In one training batch, collect all the image-text pairs produced and vectors; L2-normalize all these vectors with the formula: denotes the normalized result of the [CLS] token corresponding to the text-image pair of the i-th stage; 3) Then calculate their cosine similarity between each other, forming a similarity matrix; in this matrix, the similarity between the feature vectors from the same original image-text pair is considered as the similarity of "positive sample pair", while the similarity between the feature vectors from any other image-text pair is considered as the similarity of "negative sample pair"; and ​ 4) The method uses InfoNCE loss, which is the core contrast loss function used in models such as CLIP; its goal is to maximize the similarity of positive sample pairs, i.e. to make their cosine similarity close to 1, while minimizing the similarity with all negative sample pairs in the batch; this similarity comparison and matching is achieved by projecting the feature vectors of two different modalities, images and texts, into the same feature vector space; the form of the loss function is as follows: L contrast represents the contrastive learning loss, sim is the cosine similarity, τ is a tunable temperature hyperparameter that controls the smoothness of the similarity distribution, and K is the number of all negative samples in the batch.

6. The method of claim 1, wherein, The multi-loss joint optimization specifically includes: The reconstruction loss and the contrast loss are weighted and summed to form the overall loss function, which is a combined loss function used to drive model training, and the parameters of the encoder, decoder, and fusion encoder are optimized and trained end-to-end through the backpropagation algorithm; specifically, it includes: 1) Reconstruction loss L of the first stage image after high proportion mask img1 ; 2) Reconstruction loss L of the text after the first stage with low proportion mask text1 ; 3) Reconstruction loss L of the image after the second stage with high proportion mask img2 ; 4) Reconstruction loss L of the text after the second stage with low ratio masking text2 ; 5) Contrastive loss L between global features produced by two inputs contrast ; Weighted fusion of five types of loss functions: L total = a1L img1 + a2L text1 + a3L img2 + a4L text2 + a5L contrast (8) L total represents the final joint loss, where a1, a2, a3, a4, a5 are balanced learnable parameters, which are computed by calculating L total optimizing the gradients of all parameters of the model, including the parameters of the image encoder, the text encoder, the fusion encoder, the image decoder, the text decoder; this ensures that the model is learning to reconstruct while also effectively aligning the cross-modal features.

7. The method of claim 1, wherein, The downstream task fine-tuning and verification specifically includes: 1) The pre-training phase is unsupervised and does not rely on any task labels; in order to verify its effectiveness, the learned feature representation needs to be applied to specific, supervised downstream tasks, and multi-modal classification is one of the most direct verification methods; 2) After pre-training is complete, discard the decoder part; connect the output end of the trained cross-modal fusion encoder to a simple classification head; then, use specific task datasets with class labels, such as MM-IMDB (movie posters and type classification) and UPMC-FOOD101 (food pictures and category classification), to fine-tune the newly added classification head and optionally the entire encoder; 3) In order to comprehensively evaluate the performance of the model in real complex environments, the verification method is to apply simulated real-world interference residual loss processing to the image data in the test set; this process aims to test the robustness of the model's feature representation, and the specific methods include: applying Gaussian blur, motion blur, and other filters to simulate the blur caused by camera defocus or rapid motion; adding Gaussian noise to the image to simulate sensor or transmission signal interference noise; and through algorithmic superposition of fog, rain effects, or adjustment of brightness and contrast, visual effect interference processing is performed to test the adaptability of the model under varying conditions; During verification, these images processed with one or more residual losses are paired with the original text data to form a new test set, and the classification accuracy, F1 value, and other key performance indicators of the model are re-evaluated on this basis; finally, by comparing the performance difference of the model on residual loss data and original clear data, the robustness can be quantified: the smaller the performance decline, the less sensitive the multi-modal feature representation learned by the method is to pixel-level interference, and the stronger the robustness.

Images