Multi-modal learning method for modal information imbalance scene

Abstract

The invention discloses a modal information imbalance scene-oriented multi-modal learning method, and relates to the technical field of multi-modal information learning, and the method comprises a pre-training stage and a downstream task stage. In the pre-training stage, image information, text information and audio information are collected to construct a data set, a uniform coding vector is obtained through a mapping module, joint representation vectorization is carried out to form a ternary representation vector, and after random masking, the ternary representation vector is processed by a Transform codec to train a model M; in the downstream task stage, the multi-modal information is transmitted into a model M for training after being subjected to a mapping module and joint representation, and multi-modal fusion information is generated; according to the model M based on the Transform codec, the problem that the performance of a traditional model is reduced when the mode is unbalanced can be effectively solved, the mode information is more balanced, the multi-mode information sensing capacity is improved, the downstream task learning effect is improved by combining downstream task data training, and the method has good landing performance, high efficiency and universality.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information learning, and more specifically, to a multimodal learning method for scenarios with unbalanced modal information. Background Art

[0002] In multimodal information learning, traditional models often experience a significant drop in training performance when faced with unbalanced modal information. Data from different modalities may differ greatly in quantity, quality, and feature distribution, making it difficult for the model to fully utilize the information from each modality, affecting the effect of multimodal fusion and the performance of downstream tasks. Summary of the invention

[0003] In view of the deficiencies in the prior art, the object of the present invention is to provide a multimodal learning method for scenarios with unbalanced modal information.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A multimodal learning method for scenarios with imbalanced modal information, including a pre-training stage and a downstream task stage;

[0006] The pre-training stage includes data collection and data set construction, inputting into the mapping module Mapping_Module to obtain the encoding vector, joint representation vectorization operation, random mask and Transformer module processing, and training to generate a multimodal fusion model M;

[0007] The Transformer module consists of a Transformer encoding module and a Transformer decoding module;

[0008] The downstream task stage includes multimodal information input and feature mapping, joint representation vectorization operation to generate ternary representation vectors, and multimodal fusion model M training.

[0009] Preferably, the data collection and data set construction include:

[0010] The data collection includes images, texts, and audio information;

[0011] The collected information is preprocessed, feature extracted, labeled and divided to complete the construction of the data set, and the resulting data set includes image data set I1, text data set T1, and audio data set V1.

[0012] Preferably, the inputting into the mapping module Mapping_Module to obtain the encoding vector includes:

[0013] The mapping module Mapping_Module can receive data of different modes and output a coding vector in a unified form;

[0014] Determine the structure of the mapping module Mapping_Module, which is composed of specific feature extractors for different modalities;

[0015] For image data, a pre-trained convolutional neural network was selected with the last classification layer removed as the image feature extractor;

[0016] For text data, select a pre-trained language model as a text feature extractor;

[0017] For audio data, a neural network model based on Mel-frequency cepstral coefficients is selected as the audio feature extractor;

[0018] The image dataset I1, text dataset T1, and audio dataset V1 are respectively passed into the mapping module Mapping_Module to obtain the corresponding encoding vectors I_patches, T_patches, and V_patches;

[0019] Input the image data set I1 batch by batch into the image feature extraction part of the mapping module Mapping_Module to obtain the image coding vector I_patches;

[0020] Input the text dataset T1 batch by batch into the text feature extraction part of the mapping module Mapping_Module to obtain the text encoding vector T_patches;

[0021] The audio data set V1 is input batch by batch into the audio feature extraction part of the mapping module Mapping_Module to obtain the audio encoding vector V_patches.

[0022] Preferably, the joint representation vectorization operation includes:

[0023] The three encoding vectors I_patches, T_patches, and V_patches are jointly represented and vectorized according to the position information, context relevance, and pixel similarity;

[0024] The three encoding vectors are combined by simple concatenation to form n groups of ternary representation vectors (I, T, V);

[0025] Directly concatenate n groups of ternary representation vectors together to form a dimensional vector F;

[0026] If the dimension of the concatenated dimensional vector F is too high, it will increase the computational complexity and lead to the risk of overfitting. Then, the dimensional vector F is reduced in dimension to obtain the reduced dimensional vector F.

[0027] In order to make the features of different dimensions comparable, the dimension vector F after dimensionality reduction is normalized to obtain the normalized dimension vector F, which is the final dimension vector F.

[0028] Preferably, the random mask and Transformer module processing includes:

[0029] For n groups of ternary representation vectors, randomly select a part of them for masking at a masking rate x; combine the randomly masked ternary representation vectors together to form a new vector F_m;

[0030] Pass F_m into the Transformer encoding module to obtain the encoding token F_mask_tokens;

[0031] Determine the hyperparameters of the Transformer encoding module, including the number of layers, hidden layer dimensions, and number of attention heads;

[0032] Construct a Transformer encoder based on the hyperparameters. Each encoder layer consists of a multi-head self-attention mechanism and a feed-forward neural network.

[0033] In each encoder layer, the input vector is first processed by a multi-head self-attention mechanism to capture the dependencies between different positions; then the output of the attention mechanism is further processed by a feedforward neural network to extract higher-level features; after processing by multiple encoder layers, the encoded vector F_mask_tokens is obtained;

[0034] The randomly masked ternary representation vector is restored through the Transformer decoding module to form a dimensional vector F_re;

[0035] Determine the hyperparameters of the Transformer decoding module, including the number of layers, hidden layer dimensions, and number of attention heads;

[0036] Construct a Transformer decoder based on the hyperparameters. Each decoder layer consists of a masked multi-head self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network.

[0037] In each decoder layer, the decoded part is first processed through the multi-head self-attention mechanism to capture its internal dependencies; then the encoded token is processed through the encoder-decoder attention mechanism to obtain the information of the masked part; finally, the output of the attention mechanism is further processed by the feedforward neural network; after iterative processing of multiple decoder layers, the randomly masked ternary representation vector is gradually restored to form a dimensional vector F_re.

[0038] Preferably, the training to generate the multimodal fusion model M includes defining a loss function, optimizing model parameters, evaluating model performance, repeating the training process, and generating the multimodal fusion model M;

[0039] The defined loss function uses loss functions such as mean square error and cross entropy to measure the difference between the output of the model and the true value;

[0040] The optimization model parameters use a stochastic gradient descent optimization algorithm to update the parameters of the model to minimize the loss function;

[0041] The evaluation model performance, during the training process, regularly uses the validation set to evaluate the performance of the model;

[0042] The repeated training process continuously adjusts hyperparameters, improves model structure and training strategy, and repeats the training process until the performance of the model on the validation set reaches expectations;

[0043] The multimodal fusion model M is generated. When the performance of the model on the validation set reaches a certain requirement, the model can be considered to have been trained. The trained model is saved as the multimodal fusion model M capable of handling modal imbalance.

[0044] Preferably, the multimodal information input and feature mapping include:

[0045] Collect multimodal information consisting of images, text, and audio for downstream tasks;

[0046] The collected multimodal information is input into the mapping module Mapping_Module respectively, and after extracting the features, unified encoding vectors I_patches, T_patches and V_patches are obtained.

[0047] Preferably, the joint representation vectorization operation generates a ternary representation vector, including:

[0048] The three encoding vectors are jointly represented based on location information, context relevance and pixel similarity;

[0049] The three encoding vectors are jointly represented and vectorized to form m groups of ternary representation vectors (I, T, V).

[0050] Preferably, the multimodal fusion model M training includes:

[0051] The generated m groups of triple representation vectors are passed into the multimodal fusion model M for training;

[0052] After multiple rounds of training, the multimodal fusion model M can fuse the input ternary representation vectors into a unified representation and generate multimodal fused information.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention can achieve the task of processing multimodal imbalance through the multimodal fusion model M proposed based on the Transformer codec, solve the problem that the training performance of the traditional model is greatly reduced when the modal information is unbalanced, so that the information of each modality is obtained more balanced, and the trained model has better multimodal information perception ability;

[0055] 2. The multimodal feature training proposed in the present invention can be combined with downstream task data to perform multimodal feature training on the basis of pre-training, and the obtained multimodal information can obtain more accurate multimodal results in downstream tasks;

[0056] 3. The fused multimodal information obtained by the technical solution of the present invention can automatically adjust the weight of the multimodal fusion model according to the scale proportion of different modes, solving the pain point of poor model performance caused by imbalanced conventional multimodal data; in addition, it provides complete and comprehensive multimodal information for downstream tasks of such multimodal tasks, which can effectively improve the learning effect of downstream tasks;

[0057] 4. The technical solution of the present invention has good feasibility, high efficiency and versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A structural diagram of a multimodal learning method for scenarios with imbalanced modal information is proposed for the present invention. DETAILED DESCRIPTION

[0059] Reference Figure 1 .

[0060] The embodiment further illustrates a multimodal learning method for a scenario with imbalanced modal information proposed by the present invention.

[0061] A multimodal learning method for scenarios with imbalanced modal information, including a pre-training stage and a downstream task stage;

[0062] The pre-training stage includes data collection and data set construction, inputting into the mapping module Mapping_Module to obtain the encoding vector, joint representation vectorization operation, random mask and Transformer module processing, and training to generate a multimodal fusion model M;

[0063] The Transformer module consists of a Transformer encoding module and a Transformer decoding module;

[0064] The downstream task stage includes multimodal information input and feature mapping, joint representation vectorization operation to generate ternary representation vectors, and multimodal fusion model M training.

[0065] The data collection and data set construction include:

[0066] The data collection includes images, texts, and audio information;

[0067] The collected information is preprocessed, feature extracted, annotated and divided to complete the construction of the data set, and the data sets obtained include image data set I1, text data set T1, and audio data set V1;

[0068] The image dataset is constructed;

[0069] Preprocess the acquired images, including image resizing, image enhancement, and image denoising, to improve image quality and data consistency;

[0070] Annotate the preprocessed images, including image classification annotation and target detection annotation;

[0071] The processed image data is divided into a training set, a validation set and a test set to obtain an image data set I1;

[0072] The text dataset is constructed;

[0073] Preprocess the collected text, including text cleaning, word segmentation, stop word removal, and stem extraction or word form restoration, to improve the usability of the text;

[0074] Annotate the preprocessed text, including text classification annotation and sentiment analysis annotation;

[0075] Divide the processed text data into a training set, a validation set, and a test set to obtain a text data set T1;

[0076] The audio data set is constructed;

[0077] Pre-process the collected audio, including audio noise reduction, audio equalization, and audio segmentation;

[0078] Extract features from the preprocessed audio, including time domain features, frequency domain features, and time-frequency domain features;

[0079] Annotate the audio after feature extraction, including audio classification annotation and speech recognition annotation;

[0080] The processed audio data is divided into a training set, a validation set, and a test set to obtain the audio data set V1.

[0081] The input to the mapping module Mapping_Module obtains the encoding vector, including:

[0082] The mapping module Mapping_Module can receive data of different modes and output a coding vector in a unified form;

[0083] Determine the structure of the mapping module Mapping_Module, which is composed of specific feature extractors for different modalities;

[0084] For image data, a pre-trained convolutional neural network was selected with the last classification layer removed as the image feature extractor;

[0085] For text data, select a pre-trained language model as a text feature extractor;

[0086] For audio data, a neural network model based on Mel-frequency cepstral coefficients is selected as the audio feature extractor;

[0087] The image dataset I1, text dataset T1, and audio dataset V1 are respectively passed into the mapping module Mapping_Module to obtain the corresponding encoding vectors I_patches, T_patches, and V_patches;

[0088] Input the image data set I1 batch by batch into the image feature extraction part of the mapping module Mapping_Module to obtain the image coding vector I_patches;

[0089] Input the text dataset T1 batch by batch into the text feature extraction part of the mapping module Mapping_Module to obtain the text encoding vector T_patches;

[0090] The audio data set V1 is input batch by batch into the audio feature extraction part of the mapping module Mapping_Module to obtain the audio encoding vector V_patches.

[0091] The joint representation vectorization operation includes:

[0092] The three encoding vectors I_patches, T_patches, and V_patches are jointly represented and vectorized according to the position information, context relevance, and pixel similarity;

[0093] Consider location information;

[0094] For the image coding vector, different weights can be assigned according to the positional relationship of pixels in the image; for example, pixels in the center area of ​​the image may be more important than pixels in the edge area and can be given higher weights;

[0095] For text encoding vectors, different weights can be given to words according to their positions in the sentence; for example, words at the beginning and end of a sentence may be more important than words in the middle and can be given higher weights;

[0096] Different weights can be given to the audio coding vector according to the position in the time series; for example, the segments at the beginning and end of the audio may be more important than the segments in the middle and can be given a higher weight;

[0097] Consider contextual relevance;

[0098] For text encoding vectors, use contextual information in the language model; for example, capture the semantic relationship between words through the attention mechanism;

[0099] For image encoding vectors, consider the feature correlation between adjacent pixels; for example, use convolution operations to capture the feature correlation of local areas;

[0100] For audio encoding vectors, consider the connection between audio features in previous and next time periods; for example, use recurrent neural networks or convolutional neural networks to capture the time series features of audio;

[0101] Consider pixel similarity;

[0102] Only for the image coding vector, the pixels in the image are grouped based on the similarity of features such as color and texture, and then a joint representation of the image is constructed based on the features of the pixel groups; for example, a clustering algorithm is used to group similar pixels into one category, and then the features of the cluster center are used to represent the features of the pixel group;

[0103] The three encoding vectors are combined by simple concatenation to form n groups of ternary representation vectors (I, T, V);

[0104] For the first set of triple representation vectors;

[0105] Select a specific vector from the image coding vector I_patches, denoted as I_1;

[0106] Select a corresponding vector from the text encoding vector T_patches, denoted as T_1;

[0107] Select a corresponding vector from the audio coding vector V_patches, denoted as V_1;

[0108] To simply concatenate these three vectors, you can use the array concatenation operation or specific vector concatenation function in the programming language; for example, in Python, you can use the concatenate function of the numpy library to concatenate the three vectors along a specific dimension to form a new vector (I_1, T_1, V_1), which is the first set of ternary representation vectors;

[0109] For the second set of triple representation vectors;

[0110] In a similar way, select another vector from I_patches, T_patches, and V_patches, denoted as I_2, T_2, and V_2 respectively;

[0111] Perform concatenation operation to form vector (I_2, T_2, V_2), which is the second set of ternary representation vectors;

[0112] Repeat the above steps until n groups of ternary representation vectors are generated;

[0113] Continuously select different vectors from the three encoding vectors for splicing, and generate the third group, the fourth group, and finally the nth group of ternary representation vectors (I, T, V);

[0114] Directly concatenate n groups of ternary representation vectors together to form a dimensional vector F;

[0115] Assuming that the dimension of each ternary representation vector is d, and there are n data samples in total, the dimension of the concatenated dimensional vector F is n×d; for example, for the first set of ternary representation vectors (I 1 ,T 1 ,V 1 ), expand it into a long vector [I 1 ; T 1 ; V 1 ], where “;” represents the vector concatenation operation; then the same operation is performed on the second set of ternary representation vectors to obtain [I 2 ; T 2 ; V 2 ]; Similarly, concatenate n groups of ternary representation vectors together to obtain the dimension vector F = [I 1 ; T 1 ; V 1 ;I 2 ; T 2 ; V 2;...;I n ; T n ; V n ];

[0116] If the dimension of the concatenated dimensional vector F is too high, it will increase the computational complexity and lead to the risk of overfitting. Then, the vector F is subjected to dimensionality reduction processing to obtain the dimensional vector F after dimensionality reduction.

[0117] Use principal component analysis (PCA), linear discriminant analysis (LDA) and other dimensionality reduction methods to project high-dimensional vectors into a low-dimensional space while retaining as much important information as possible; for example, use principal component analysis to reduce the dimensionality of vector F; first calculate the covariance matrix of vector F, then calculate the eigenvalues ​​and eigenvectors of the covariance matrix, select the eigenvectors corresponding to the largest eigenvalues ​​as the projection matrix, project vector F into the low-dimensional space, and obtain the reduced-dimensional vector F;

[0118] In order to make the features of different dimensions comparable, the dimension vector F after dimensionality reduction is normalized to obtain the normalized dimension vector F, which is the final dimension vector F;

[0119] Common normalization methods include minimum-maximum normalization and Z-score normalization. For example, using Z-score normalization, the mean of the vector F is subtracted from each element of the vector, and then divided by the standard deviation of the vector to obtain the normalized dimension vector F.

[0120] The random mask is processed with the Transformer module, including:

[0121] For n groups of ternary representation vectors, randomly select a part of them for masking at a masking rate x; combine the randomly masked ternary representation vectors together to form a new vector F_m;

[0122] Pass F_m into the Transformer encoding module to obtain the encoding token F_mask_tokens;

[0123] Determine the hyperparameters of the Transformer encoding module, including the number of layers, hidden layer dimensions, and number of attention heads;

[0124] Construct a Transformer encoder based on the hyperparameters. Each encoder layer consists of a multi-head self-attention mechanism and a feed-forward neural network.

[0125] In each encoder layer, the input vector is first processed by a multi-head self-attention mechanism to capture the dependencies between different positions; then the output of the attention mechanism is further processed by a feedforward neural network to extract higher-level features; after processing by multiple encoder layers, the encoded vector F_mask_tokens is obtained;

[0126] The randomly masked ternary representation vector is restored through the Transformer decoding module to form a dimensional vector F_re;

[0127] Determine the hyperparameters of the Transformer decoding module, including the number of layers, hidden layer dimensions, and number of attention heads;

[0128] Construct a Transformer decoder based on the hyperparameters. Each decoder layer consists of a masked multi-head self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network.

[0129] In each decoder layer, the decoded part is first processed through the multi-head self-attention mechanism to capture its internal dependencies; then the encoded token is processed through the encoder-decoder attention mechanism to obtain the information of the masked part; finally, the output of the attention mechanism is further processed by the feedforward neural network; after iterative processing of multiple decoder layers, the randomly masked ternary representation vector is gradually restored to form a dimensional vector F_re.

[0130] The training and generating of the multimodal fusion model M includes defining a loss function, optimizing model parameters, evaluating model performance, repeating the training process, and generating the multimodal fusion model M;

[0131] The defined loss function uses loss functions such as mean square error and cross entropy to measure the difference between the output of the model and the true value;

[0132] The optimization model parameters use a stochastic gradient descent optimization algorithm to update the parameters of the model to minimize the loss function;

[0133] The evaluation model performance, during the training process, regularly uses the validation set to evaluate the performance of the model;

[0134] The repeated training process continuously adjusts hyperparameters, improves model structure and training strategy, and repeats the training process until the performance of the model on the validation set reaches expectations;

[0135] The multimodal fusion model M is generated. When the performance of the model on the validation set reaches a certain requirement, the model can be considered to have been trained. The trained model is saved as the multimodal fusion model M capable of handling modal imbalance.

[0136] The multimodal information input and feature mapping include:

[0137] Collect multimodal information consisting of images, text, and audio for downstream tasks;

[0138] The collected multimodal information is input into the mapping module Mapping_Module respectively, and after extracting the features, unified encoding vectors I_patches, T_patches and V_patches are obtained.

[0139] The joint representation vectorization operation generates a ternary representation vector, including:

[0140] The three encoding vectors are jointly represented based on location information, context relevance and pixel similarity;

[0141] The three encoding vectors are jointly represented and vectorized to form m groups of ternary representation vectors (I, T, V).

[0142] The multimodal fusion model M training includes:

[0143] The generated m groups of triple representation vectors are passed into the multimodal fusion model M for training;

[0144] After multiple rounds of training, the multimodal fusion model M can fuse the input ternary representation vectors into a unified representation and generate multimodal fused information.

[0145] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multimodal learning method for scenarios with unbalanced modal information, characterized in that: Includes pre-training stage and downstream task stage; The pre-training stage includes data collection and data set construction, inputting into the mapping module Mapping_Module to obtain the encoding vector, joint representation vectorization operation, random mask and Transformer module processing, and training to generate a multimodal fusion model M; The Transformer module consists of a Transformer encoding module and a Transformer decoding module; The downstream task stage includes multimodal information input and feature mapping, joint representation vectorization operation to generate ternary representation vectors, and multimodal fusion model M training.

2. According to claim 1, a multimodal learning method for modal information imbalance scenarios is characterized in that: The data collection and data set construction include: The data collection includes images, texts, and audio information; The collected information is preprocessed, feature extracted, labeled and divided to complete the construction of the data set, and the resulting data set includes image data set I1, text data set T1, and audio data set V1.

3. According to claim 2, a multimodal learning method for modal information imbalance scenarios is characterized in that: The input to the mapping module Mapping_Module obtains the encoding vector, including: The mapping module Mapping_Module can receive data of different modes and output a coding vector in a unified form; Determine the structure of the mapping module Mapping_Module, which is composed of specific feature extractors for different modalities; For image data, a pre-trained convolutional neural network was selected with the last classification layer removed as the image feature extractor; For text data, select a pre-trained language model as a text feature extractor; For audio data, a neural network model based on Mel-frequency cepstral coefficients is selected as the audio feature extractor; The image dataset I1, text dataset T1, and audio dataset V1 are respectively passed into the mapping module Mapping_Module to obtain the corresponding encoding vectors I_patches, T_patches, and V_patches; Input the image data set I1 batch by batch into the image feature extraction part of the mapping module Mapping_Module to obtain the image coding vector I_patches; Input the text dataset T1 batch by batch into the text feature extraction part of the mapping module Mapping_Module to obtain the text encoding vector T_patches; The audio data set V1 is input batch by batch into the audio feature extraction part of the mapping module Mapping_Module to obtain the audio encoding vector V_patches.

4. The multimodal learning method for modal information imbalance scenarios according to claim 3, characterized in that: The joint representation vectorization operation includes: The three encoding vectors I_patches, T_patches, and V_patches are jointly represented and vectorized according to the position information, context relevance, and pixel similarity; The three encoding vectors are combined by simple concatenation to form n groups of ternary representation vectors (I, T, V); Directly concatenate n groups of ternary representation vectors together to form a dimensional vector F; If the dimension of the concatenated dimensional vector F is too high, it will increase the computational complexity and lead to the risk of overfitting. Then, the dimensional vector F is reduced in dimension to obtain the reduced dimensional vector F. In order to make the features of different dimensions comparable, the dimension vector F after dimensionality reduction is normalized to obtain the normalized dimension vector F, which is the final dimension vector F.

5. The multimodal learning method for modal information imbalance scenarios according to claim 4, characterized in that: The random mask is processed with the Transformer module, including: For n groups of ternary representation vectors, randomly select a part of them for masking at a masking rate x; combine the randomly masked ternary representation vectors together to form a new vector F_m; Pass F_m into the Transformer encoding module to obtain the encoding token F_mask_tokens; Determine the hyperparameters of the Transformer encoding module, including the number of layers, hidden layer dimensions, and number of attention heads; Construct a Transformer encoder based on the hyperparameters. Each encoder layer consists of a multi-head self-attention mechanism and a feed-forward neural network. In each encoder layer, the input vector is first processed by a multi-head self-attention mechanism to capture the dependencies between different positions; then the output of the attention mechanism is further processed by a feedforward neural network to extract higher-level features; after processing by multiple encoder layers, the encoded vector F_mask_tokens is obtained; The randomly masked ternary representation vector is restored through the Transformer decoding module to form a dimensional vector F_re; Determine the hyperparameters of the Transformer decoding module, including the number of layers, hidden layer dimensions, and number of attention heads; Construct a Transformer decoder based on the hyperparameters. Each decoder layer consists of a masked multi-head self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network. In each decoder layer, the decoded part is first processed through the multi-head self-attention mechanism to capture its internal dependencies; then the encoded token is processed through the encoder-decoder attention mechanism to obtain the information of the masked part; finally, the output of the attention mechanism is further processed by the feedforward neural network; after iterative processing of multiple decoder layers, the randomly masked ternary representation vector is gradually restored to form a dimensional vector F_re.

6. The multimodal learning method for modal information imbalance scenarios according to claim 5, characterized in that: The training and generating of the multimodal fusion model M includes defining a loss function, optimizing model parameters, evaluating model performance, repeating the training process, and generating the multimodal fusion model M; The defined loss function uses loss functions such as mean square error and cross entropy to measure the difference between the output of the model and the true value; The optimization model parameters use a stochastic gradient descent optimization algorithm to update the parameters of the model to minimize the loss function; The evaluation model performance, during the training process, regularly uses the validation set to evaluate the performance of the model; The repeated training process continuously adjusts hyperparameters, improves model structure and training strategy, and repeats the training process until the performance of the model on the validation set reaches expectations; The multimodal fusion model M is generated. When the performance of the model on the validation set reaches a certain requirement, the model can be considered to have been trained. The trained model is saved as the multimodal fusion model M capable of handling modal imbalance.

7. The multimodal learning method for modal information imbalance scenarios according to claim 6, characterized in that: The multimodal information input and feature mapping include: Collect multimodal information consisting of images, text, and audio for downstream tasks; The collected multimodal information is input into the mapping module Mapping_Module respectively, and after extracting the features, unified encoding vectors I_patches, T_patches and V_patches are obtained.

8. The multimodal learning method for modal information imbalance scenarios according to claim 7, characterized in that: The joint representation vectorization operation generates a ternary representation vector, including: The three encoding vectors are jointly represented based on location information, context relevance and pixel similarity; The three encoding vectors are jointly represented and vectorized to form m groups of ternary representation vectors (I, T, V).

9. The multimodal learning method for modal information imbalance scenarios according to claim 8, characterized in that: The multimodal fusion model M training includes: The generated m groups of triple representation vectors are passed into the multimodal fusion model M for training; After multiple rounds of training, the multimodal fusion model M can fuse the input ternary representation vectors into a unified representation and generate multimodal fused information.

Images