Multimodal data classification method, terminal device, and storage medium

Abstract

This invention discloses a multimodal data classification method, terminal device, and storage medium. For cases with missing modalities, it obtains missing modality augmentation information from multiple sources using different methods, further mitigating the problem of missing modalities at the data level. Addressing the high cost of retraining large models, this invention employs a gating mechanism to fuse missing modality augmentation information from multiple sources at the model level, thereby further mitigating the impact of missing modalities and improving the final classification accuracy of the model.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and in particular to a multimodal data classification method, terminal device, and storage medium. Background Technology

[0002] Multimodal models are widely used in classification, cross-modal retrieval, cross-modal generation, and other fields. Training and using multimodal models require a large amount of multimodal data, which refers to data collected in different modalities, including images, audio, text, and video. In practical applications, due to equipment failure, incomplete data collection, or other reasons, some modal data may be unavailable or incomplete, leading to missing modal data. Missing modal data is common during model training and use. For example, in image recognition, factors such as lighting and angle may prevent the acquisition of certain image modal data; in speech recognition, interference from background noise may prevent the acquisition of certain audio modal data. Missing modal data reduces the completeness and accuracy of multimodal data.

[0003] Intermodal interactions in multimodal models are often implemented using Transformer networks based on attention mechanisms. However, research shows that Transformer-based multimodal models are poorly robust to missing modality data; in cases of large amounts of missing modality data, their performance is even worse than using only single-modality data. For a multimodal model that performs poorly with missing modalities, existing techniques aim to improve its performance by modifying the model's structure or proposing new training methods. However, as the size of most pre-trained Transformer models continues to increase, retraining on the dataset is very expensive. Given the limited computational resources in most real-world applications, retraining is often not even feasible. Furthermore, simply using traditional data augmentation methods offers limited relief for missing modalities. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal data classification method, terminal device and storage medium to improve the accuracy of multimodal data classification, in order to address the shortcomings of the existing technology.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a multimodal data classification method, comprising:

[0006] Third characteristic

[0007] S1. Input the complete data into the first encoder to obtain the first feature, add noise to the first feature to obtain the noisy feature; input the text data / image data into the second encoder to obtain the second feature; concatenate the noisy feature and the second feature, and input the concatenated feature into the denoising network to obtain the generated feature; repeat this process to obtain the trained denoising network; wherein, the complete data includes text data and image data;

[0008] Text / image data is input into the Clip model, and the most similar data is obtained by calculating cosine similarity. The most similar data is then input into the third encoder to obtain the retrieval features.

[0009] Text / image data is input into the fourth encoder to obtain the original single-modal features; the data modality input to the fourth encoder is different from the data modality input to the second and third encoders.

[0010] S2. Concatenate the generated features, retrieved features, and original single-modal features. Use the concatenated features as input to the multimodal fusion module and the output of the multimodal fusion module as input to the pooling layer to obtain the classification result.

[0011] S3. Repeat steps S1 and S2 until the classification result meets the set requirements, and the trained multimodal fusion module is obtained.

[0012] Compared to traditional methods for addressing missing modalities that rely solely on single data augmentations, this invention utilizes various methods to obtain missing modality augmentation information from multiple sources, further mitigating the problem at the data level. To address the high cost of retraining large models, this invention employs a gating mechanism to fuse missing modality augmentation information from multiple sources at the model level, thereby further mitigating the impact of missing modalities and ultimately improving the model's classification accuracy.

[0013] In this invention, the data mode input by the fourth encoder is different from the data mode input by the second encoder and the third encoder. For example, if the data input by the fourth encoder is text data, then the data input by the second encoder and the third encoder is image data; if the data input by the fourth encoder is image data, then the data input by the second encoder and the third encoder is text data.

[0014] In step S2, before concatenating the generated features, retrieved features, and original single-modality features, the following steps are also included:

[0015] The generated features and retrieved features are weighted and fused to obtain weighted fused features;

[0016] In S2, the concatenation of the generated feature, the retrieved feature, and the single-modal original feature is replaced by the concatenation of the weighted fusion feature and the third feature.

[0017] During model training, the multimodal fusion module has a large number of parameters, making model training very costly. Therefore, these parameters need to be frozen during model fine-tuning. The gating unit allows for the automatic allocation of weights among the missing modal augmentation information from multiple sources while the parameters of the multimodal fusion module are frozen. This enables the model to automatically select better augmentation features during training, thereby achieving better classification results.

[0018] The gating unit includes a pooling layer and a multilayer perceptron. After the generated feature and the retrieved feature are input into the pooling layer, the output of the pooling layer is used as the input of the multilayer perceptron, and the output of the multilayer perceptron is used as the input of the activation function. The activation function outputs the weights of the generated feature and the weights of the retrieved feature. The generated feature is multiplied by its weight, and the retrieved feature is multiplied by its weight. The two multiplication results are then fused to obtain the weighted fused feature.

[0019] The advantage of the gated unit structure lies in its simple structure and its ability to accommodate more modal missing information and augmentation information from more sources.

[0020] The method of the present invention further includes:

[0021] S4. Input the multimodal data into the second encoder and Clip model, and use the trained denoising network and multimodal fusion module to obtain the final classification result.

[0022] During the t-th iteration, the features after adding noise Represented as: Where, α t It is a hyperparameter for controlling noise. The first feature is used. The form of the noise addition process is consistent with the original diffusion model. We rewrite it into a form suitable for multimodal feature estimation in the latent space, i.e., in the formula... in This represents the characteristics of the first modality, allowing it to be extended to more modalities.

[0023] The denoising network comprises multiple cascaded Transformer modules, with the last Transformer module connected to a linear layer. Some of these Transformer modules undergo long-term connections. Specifically, the output of the first Transformer module is concatenated with the output of the (N-1)th Transformer module, then subjected to a linear operation, and the result is used as the input to the Nth Transformer module. Similarly, the output of the second Transformer module is concatenated with the output of the (N-2)th Transformer module, then subjected to a linear operation, and the result is used as the input to the (N-1)th Transformer module, and so on, where N is the number of Transformer modules. Traditional denoising networks are typically in the form of U-Net. This invention uses Transformer modules to replace the original convolutional layers in U-Net and incorporates long-term connections. This allows for better interaction of multimodal information using attention mechanisms, ultimately generating better missing modality generation information.

[0024] As an inventive concept, the present invention also provides a terminal device, comprising:

[0025] One or more processors;

[0026] A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement the steps of the method described above.

[0027] As an inventive concept, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] 1. This invention addresses the issue of missing modal data by obtaining missing modal enhancement information from multiple sources through different methods, which can further alleviate the problem of missing modalities at the data level;

[0030] 2. To address the high cost of retraining large models, this invention employs a gating mechanism to fuse missing modality enhancement information from multiple sources at the model level, thereby further mitigating the impact of missing modalities and improving the final classification accuracy of the model. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the overall framework structure of an embodiment of the present invention.

[0032] Figure 2 This is a structural diagram of the gate control unit according to an embodiment of the present invention;

[0033] Figure 3 This is a diagram of the denoising network structure according to an embodiment of the present invention;

[0034] Figure 4 The following is a comparison of the effects of three different missing modalities in the embodiments of the present invention: (a) missing text modality, (b) missing image modality, and (c) randomly missing two modalities. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Example 1

[0037] Embodiment 1 of this invention provides a multimodal data classification method applicable to the problem of missing modalities of M different modalities. For simplicity without loss of generality, we consider a multimodal dataset consisting of m=2 modalities m1 and m2 (e.g., features of images and text). Given a multimodal dataset D... c , It is a modally complete dataset. It is the i-th sample pair, y i This is the label of this sample pair, y i ∈{0,1,2,...,C}, |C| is the number of categories in the classification task, in D c Based on this, we generated two new datasets. and in The meaning is to replace the data of the m1 modality in the original i-th sample pair with uniform data (we hope that this uniform data does not carry special semantic information as much as possible, so the features extracted from the missing modality data will be initialized as a matrix of all 1s). and Similarly, data from another modality was replaced.

[0038] We first look at the modality complete dataset D. c A base model was trained as a baseline model. To preserve the format of the multimodal inputs when testing for missing modalities, we randomly selected ηD. cand (or The training and testing data consist of a complete dataset, and the proportion of missing modalities is controlled by adjusting the randomly selected proportion η. Our goal is to achieve, as far as possible, the model's performance as possible when using complete modal data, even with missing modalities.

[0039] The overall framework diagram of the model is as follows: Figure 1 As shown, we will first look at the modality complete dataset D. c A baseline model was trained on this model, shown on the left side of the diagram. Subsequent experimental validation was also performed on this model. This model consists of a text feature extractor, an image feature extractor, a multimodal fusion module, and a final classifier. The right side of the diagram shows the model's feature enhancement module, which includes a missing modality retrieval model and a missing modality generation model. We will discuss these two models in detail later. During the model's inference phase, for data with missing modalities, feature enhancement is performed simultaneously through both the missing modality retrieval model and the missing modality generation model. The missing modality retrieval model retrieves features of the missing modality from the dataset based on existing modality features. The missing modality generation model generates features of the missing modality based on existing modality features as conditional information. The two enhanced features are gating-enabled through a gate mechanism, allowing the model to autonomously select the better enhanced features.

[0040] We designed a latent space multimodal joint diffusion model based on the unidiffuser structure, which uses information from one modality as guidance to generate information from another modality. First, we concatenate all modal features to obtain the overall features. Taking two modalities as an example, let's consider x... m1 and x m2 Features are combined into a single feature x m The subsequent forward noise addition and reverse noise removal processes of the diffusion model are both based on this overall characteristic. The diffusion model interferes with the data by gradually injecting noise, and this noise addition process can be formalized by the following formula:

[0041]

[0042] Where α t It is a hyperparameter for controlling noise. This means that when t is large enough, it will eventually... It is almost entirely converted into Gaussian noise. The reverse process of the diffusion model is to convert Gaussian noise into... Reduced to This process uses a latent space network with parameter θ for modeling, and uses the network after t steps of noise addition. Using conditional information y as input, predict the noise added in step t. Thus making The features can be reduced to the features of step t-1.

[0043]

[0044] Finally, by repeating the restoration t times, it can be achieved Restore to At this point, we can restore the... Features are split into various modalities based on the index before splicing, while retaining the features of the original missing modalities.

[0045] Considering the excellent performance of intermodal interactions in Transformer-based backbone networks, we use a Transformer as the skeleton to simulate this noisy prediction network. Taking image and text modalities as an example, the overall features concatenated from image and text features are noise-added and used together with time step t as input to the noisy prediction network. The conditional information y only contains single-modal features of text or image, and this conditional information remains unchanged throughout the training process. The interactions between different modal information and the interaction with conditional information are implemented using a multi-head attention mechanism.

[0046] During the model training phase, we added noise using the process shown in Equation 1, and parameterized the noise prediction network using the above model. The parameters are then optimized using the loss function described below.

[0047]

[0048] During the inference phase of the model, existing modal features are used as conditional information, while features of text and images are generated simultaneously, and features of missing modalities are retained as inputs for subsequent gating devices.

[0049] We use the Clip model, based on contrastive learning, as the missing modality retrieval model. It leverages existing data from one modality to retrieve missing modality data from all available data from another modality, serving as one source of augmented data. Contrastive learning is a deep learning training strategy that learns the differences and similarities between samples by comparing similar and dissimilar pairs. For example, it learns the similarity between pairs of text and images and the differences between different pairs of text and images, thus obtaining better representations. Clip pre-trains image and text encoders using contrastive learning to predict which images pair with which texts in the dataset. This allows us to calculate the similarity between the two modalities using the following formula, thereby retrieving the index of the most similar missing modality data from the existing modality data. The formula below illustrates the process of retrieving m2 modality data using m1 modality data.

[0050]

[0051] in This indicates all other available m2 modal data after the operation of simulating the missing modality.

[0052] We obtained augmented data from two sources using a missing modality generation model and a missing modality retrieval model. We believe these two augmented features have different importance, so we designed a gating unit to automatically assign importance weights to the two features. This gating unit mainly consists of a pooling layer, an MLP, and a sigmoid function. The gating unit takes the two features from the generation and retrieval models as inputs and performs a weighted fusion of the gating unit's output with the two features through a multiplication operation. The closer the gating unit's output is to 1, the higher the importance of the corresponding feature; the closer the output is to 0, the lower the importance of the corresponding feature. This allows the entire model to automatically select better features during training. Figure 2 It is the detailed structure of the gating mechanism.

[0053] Figure 3 This is a structural diagram of the denoising network. Essentially, the denoising network is a noise prediction network that predicts the noise added at time t, and then performs denoising to obtain the multimodal joint features at time t-1. The backbone of the denoising network is a Transformer, where each Transformer block includes a multi-head attention mechanism, two normalization operations, and a multilayer perceptron (MLP). This invention uses long connections to simulate the structure of U-Net, allowing later Transformer blocks closer to the output to obtain information closer to the original. The inputs to the denoising network are the encoding at time t, the control information features at time t, and the multimodal joint features after denoising at time t. Its output is the predicted amount of noise during the denoising process from time t-1 to time t.

[0054] We tested the performance of the baseline model with missing unimodalities (text or image) and with randomly missing modalities, as well as the performance of adding our feature enhancement model. We also conducted some ablation experiments to verify the effect of using the generative model alone or the retrieval model alone to enhance the data.

[0055] We selected two datasets with both image and text modalities for our experiments: the MVSA-Single* dataset and the MM-IMDB dataset.

[0056] MVSA-Single* is a sentiment classification dataset with both image and text modalities. The original dataset is MVSA-multiple, which collects image-text pairs from Twitter tweets and provides manual annotations. The dataset we use is further manually annotated based on MVSA-multiple data to ensure a match between the image and text annotations for each tweet. This dataset presents a three-class classification problem. Experiment 1 uses the image and text modal information from this dataset to predict the sentiment category of tweets. We evaluate the results using accuracy and F1-mac metrics.

[0057] MM-IMDB is a movie genre classification dataset with both image and text modalities. Since a movie may have multiple different genres, this task is a multi-label classification. Experiment 2 uses the image and text modal information from this dataset to predict the movie's classification category. We use the F1-mac metric for evaluation on this dataset.

[0058] We first tested the feature enhancement model proposed in this embodiment on the MVSA-Single* dataset.

[0059] Model and Training: In the backbone model, we trained a multimodal sentiment classification model using the MVSA-Single* dataset. This model includes a text feature extractor, an image feature extractor, and a multimodal fusion module. We chose BERT-Base and ResNet-152 as the text and image feature extractors, respectively. The multimodal feature fusion module utilizes self-attention to fuse information from both modalities, and its structure is the same as BERT-Base. The backbone model was trained on the complete dataset for 30 epochs as a baseline, and the text and image feature extractors extracted features from the text and images. These features serve as input to the missing modality generation model. In the feature enhancement model, we trained a latent space joint diffusion model based on the features extracted from the backbone model. We concatenated the features of the two modalities for noise addition, using one modality's features as control information. A u-vit structure was used as the noise prediction network to predict the noise added at each step. The diffusion model was trained for 300 epochs using the ddpm algorithm. During use, existing modal features were input as control information, and features of the missing modality were sampled from Gaussian noise to serve as enhancement features for the missing modality generation model. For the missing modality retrieval model, we used clips to extract features from images and text separately. When text was missing, we calculated the cosine similarity between the corresponding sample's image features and all available text features, selecting the text with the highest similarity as the enhancement feature for the missing modality retrieval model. The two enhancement features were automatically weighted using a gating unit.

[0060] Input Construction: We define the missing rate *n* as the proportion of incomplete data in each modality to the entire dataset. In visual and language tasks, there are three possible modal missing scenarios: text missing, image missing, and both missing. For the first two scenarios, a missing text (missing image) rate *n* represents the presence of *n* image-only (text-only) data and (1-n) complete data. For the case of random missing in both modalities, we set the odds of missing text and images to be constant, and a sample pair cannot be missing both image and text simultaneously. The final input contains *n / 2* pure text data, *n / 2* pure image data, and (1-n) complete data. Furthermore, we tested training the model using only one modality as a comparison, where the model input is only one modality, and only features from one modality are used during multimodal feature fusion.

[0061] Main results: Figure 4In the table, we present quantitative results for three different missing modalities. The dashed lines represent the baseline without any augmentation methods, while the solid lines represent the method of this embodiment. The missing rate ranges from 0% to 100%. As can be seen from the table, missing modal data has a significant impact on this multimodal classification model. The evaluation metric decreases significantly as the proportion of missing data increases. When the missing data is very large, the model's performance is even worse than when trained using only single-modal data. The method of this embodiment shows significant improvements in accuracy and F1-mac in all missing modalities, especially when a large amount of modal data is missing, where the improvement is greater than when only a small amount of data is missing. This effectively mitigates the adverse effects of missing modal data on the model.

[0062] We conducted a series of ablation experiments on the MMIMDB dataset to demonstrate the effectiveness of our method. In Experiment 1, we first tested the impact of two different augmentation features on the model. We evaluated the model's performance using only the two augmentation features in two modalities: missing text, missing images, and random missing text. Table 1 shows that in the cases of missing images and random missing text, both augmentation methods showed improvement compared to the original method (miss_aware_prompt), proving the effectiveness of our method. The overall performance was even better after fusing the two methods using a gating mechanism. However, the model's performance decreased in the case of missing text. According to Ma's research, the text modality plays a dominant role in classification in the MMIMDB dataset; therefore, text bias can significantly affect the model's classification results. In this experiment, we directly used the weights of the unidiffuser as the weights of the generative model without fine-tuning them on this dataset, resulting in poor performance of the generated features. This also reflects the limitation of this invention, namely its heavy reliance on the quality of the augmented features generated by the feature augmentation model.

[0063] Table 1: Comparison of different augmentation methods on the MMIMDB dataset

[0064]

[0065] We also conducted some analysis on the effectiveness of the gating mechanism. We analyzed and statistically analyzed the output results of the Sigmoid function in the trained gating unit on the MM-IMDB test set, and the results are shown in Table 2.

[0066] Table 2: Ablation studies on gating mechanisms

[0067]

[0068] The test set contains 16,120 image-text pairs. The results show that the model selected all the retrieved text as text enhancement features. Since the quality of the generated text is very poor, this proves that the gating mechanism performs well in selecting superior features. In terms of image enhancement, the difference in the selection of the two enhancement features is not significant, but the generated images are selected more often overall.

[0069] This invention focuses on the missing modality problem faced by multimodal classification models in practical applications. To mitigate the impact of missing modalities on classification results, we propose a method that fuses retrieval feature enhancement based on contrastive learning and generative feature enhancement based on a diffusion model. This method is applicable to transformer-based multimodal fusion models to alleviate the performance degradation caused by missing modalities. Under the aforementioned experimental settings, we tested the performance of our enhancement model under various scenarios, including missing single modalities and randomly missing modalities, and demonstrated the effectiveness of the method in this invention through ablation experiments.

[0070] Example 2

[0071] Embodiment 2 of the present invention provides a terminal device corresponding to Embodiment 1 above. The terminal device can be a processing device for a client, such as a mobile phone, a laptop, a tablet computer, a desktop computer, etc., to execute the method of the above embodiments.

[0072] The terminal device in this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in Embodiment 1 described above.

[0073] In some implementations, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0074] In other implementations, the processor can be any type of general-purpose processor, such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation here.

[0075] Example 3

[0076] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to Embodiment 1 above, on which a computer program / instructions are stored. When the computer program / instructions are executed by a processor, they implement the steps of the method of Embodiment 1 above.

[0077] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0078] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0079] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0082] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A multimodal data classification method, characterized in that, include: S1. Input the complete data into the first encoder to obtain the first feature, add noise to the first feature to obtain the noisy feature; input the text data / image data into the second encoder to obtain the second feature; concatenate the noisy feature and the second feature, and input the concatenated feature into the denoising network to obtain the generated feature of the missing modality; Repeat this process to obtain the trained denoising network; wherein, the complete data includes text data and image data; Text / image data is input into the Clip model, and the most similar data is obtained by calculating cosine similarity. The most similar data is then input into the third encoder to obtain the retrieval features of the missing modality. The text / image data is input into the fourth encoder to obtain the original single-modal features of the text / image data; the data modality input by the fourth encoder is different from the data modality input by the second and third encoders. S2. Concatenate the generated features, retrieved features, and original single-modal features. Use the concatenated features as input to the multimodal fusion module and the output of the multimodal fusion module as input to the first pooling layer to obtain the classification result. S3. Repeat steps S1 and S2 until the classification result meets the set requirements, and the trained multimodal fusion module is obtained.

2. The multimodal data classification method according to claim 1, characterized in that, In step S2, before concatenating the generated features, retrieved features, and original single-modality features, the following steps are also included: The generated features and retrieved features are weighted and fused using a gating unit to obtain weighted fused features. In S2, the concatenation of the generated features, the retrieved features, and the original single-modal features is replaced by the concatenation of the weighted fusion features and the original single-modal features.

3. The multimodal data classification method according to claim 2, characterized in that, The gating unit includes a second pooling layer and a multilayer perceptron; after the generated features and retrieved features are input to the second pooling layer, the output of the second pooling layer is used as the input of the multilayer perceptron, the output of the multilayer perceptron is used as the input of the activation function, and the activation function outputs the weights of the generated features and the weights of the retrieved features. The generated feature is multiplied by its weight, the retrieved feature is multiplied by its weight, and the two multiplication results are combined to obtain the weighted fused feature.

4. The multimodal data classification method according to claim 1 or 2, characterized in that, Also includes: S4. Input the multimodal data into the second encoder and Clip model, and use the trained denoising network and multimodal fusion module to obtain the final classification result.

5. The multimodal data classification method according to claim 1, characterized in that, The first feature is subjected to t-step noise addition, and the feature after t-step noise addition is... Represented as: ;in, It is a hyperparameter for controlling noise. , This is the first characteristic.

6. The multimodal data classification method according to claim 1, characterized in that, The denoising network comprises multiple cascaded Transformer modules, with the last Transformer module connected to a linear layer. The output of the first Transformer module is concatenated with the output of the (N-1)th Transformer module, and then subjected to a linear operation. The result is used as the input of the Nth Transformer module. The output of the second Transformer module is concatenated with the output of the (N-2)th Transformer module, and then subjected to a linear operation. The result is used as the input of the (N-1)th Transformer module, and so on. N is the number of Transformer modules.

7. A terminal device, characterized in that, include: One or more processors; A memory having stored one or more programs thereon, which, when executed by one or more processors, cause the one or more processors to perform the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.

Images