Fine-grained multimodal meta-learning recognition method for modality loss

Abstract

This paper discloses a fine-grained multimodal meta-learning recognition method for modality missingness to address the modality missingness problem. The method includes: sampling on a complete dataset and constructing multiple meta-tasks for missing modalities for training; within each task, extracting local and global features from the multimodal data and fusing the features using an attention mechanism; feeding the fused features into a completion module to supplement the missing modal information, and finally combining the features for classification. By using meta-learning on a complete dataset, the model demonstrates excellent generalization performance across different missing modalities, enabling effective information completion and fine-grained multimodal data classification.

Description

Technical Field

[0001] The present invention belongs to the field of fine-grained multimodal recognition under modality loss, and specifically relates to a fine-grained multimodal meta-learning recognition method for modality loss. Background Art

[0002] Fine-grained recognition is a fundamental research topic in computer vision and pattern recognition, aiming to identify subcategories within the same general category (e.g., different types of flowers, dogs, or cars). The key challenge in fine-grained recognition lies in understanding fine-grained differences to fully distinguish objects with highly similar overall appearance or descriptions but different fine-grained features. With the increasing availability of multimedia data, multimodal fine-grained recognition has made tremendous progress in recent years. These methods focus on improving fine-grained recognition accuracy by combining multimodal data to establish joint representations. However, multimodal fine-grained data often lacks modality, i.e., the multimodal data is incomplete. For example, for a rare bird, we may have image and text modal data, but it is difficult to obtain the bird's song (i.e., audio modal data), let alone video modality. Therefore, the task of fine-grained multimodal recognition in the absence of modality is more realistic and more challenging.

[0003] To address the poor performance of neural network models in fine-grained recognition tasks, researchers have proposed using an attention mechanism to highlight important component information in the data, to further extract fine-grained information, and thus improve recognition accuracy. At the same time, researchers have also proposed building a reconstruction network to reconstruct the information of missing modalities, thereby alleviating the phenomenon of decreased recognition accuracy caused by modality loss. In terms of fusing multimodal data, some research works map multimodal data into the same subspace by mapping, reducing the differences between modal features, and fusing multimodal features. The idea of ​​meta-learning is to let the model learn to learn, so that it can quickly learn new tasks based on the acquisition of existing knowledge, and can achieve good performance on multiple tasks.

[0004] Existing methods fail to consider how to classify fine-grained multimodal data in the absence of modalities. Existing methods for data reconstruction suffer from mediocre reconstruction quality and little improvement in recognition accuracy. Furthermore, multimodal data recognition methods fail to consider fine-grained information and fail to fully integrate data features, resulting in low recognition accuracy for fine-grained multimodal data. Summary of the Invention

[0005] The purpose of the present invention is to provide a fine-grained multimodal meta-learning recognition method for modality loss.

[0006] The technical solution for achieving the purpose of the present invention is as follows: In a first aspect, the present invention provides a fine-grained multimodal meta-learning recognition method for modality loss, comprising the following steps:

[0007] Step 1: Sampling on a dataset with complete fine-grained multimodal data, constructing multiple meta-tasks with missing modal data, and training the model on each task;

[0008] Step 2: Within each task, extract local features and global features from fine-grained multimodal data, and use the attention mechanism to fuse the local features and global features respectively;

[0009] Step 3: Send the fused local features and global features to the corresponding completion modules respectively to complete the features of the missing modality. Finally, the completed local and global features are spliced ​​together for classification.

[0010] Step 4: After the meta-task training is completed, the obtained model is trained on a dataset with missing fine-grained multimodal data to obtain the final model and classification results.

[0011] In a second aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0012] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0013] Compared with the existing technology, the present invention has the following significant advantages: (1) The present invention uses meta-learning to sample a dataset with complete fine-grained multimodal data, and constructs multiple meta-task training models with missing modal data, so that the model can complete the information of the missing modalities, and at the same time obtain good generalization performance, and has a good classification effect on fine-grained multimodal data under any modality missing situation. (2) The present invention uses the attention mechanism to fuse local features and global features respectively, so that different modal features can fully interact and highlight the discriminative features, and splice the local features and global features for recognition, thereby improving the accuracy of the model in recognizing fine-grained multimodal data. (3) The present invention designs a completion module for modality missing, completes the missing modal information, and completes the overall features so that the fused features are as little affected as possible by the modality missing, reduces the bias of the fused features, and improves the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of the fine-grained multimodal meta-learning recognition method for modality loss of the present invention.

[0015] Figure 2 This is a model diagram of the fine-grained multimodal meta-learning recognition method for modality loss in the present invention. DETAILED DESCRIPTION

[0016] Combine Figure 1 、 Figure 2 , a fine-grained multimodal meta-learning recognition method for modality loss, including the following steps:

[0017] Step 1: Sample a dataset with complete fine-grained multimodal data, construct multiple meta-tasks with missing modal data, and train the model on each task.

[0018] This paper first uses a complete dataset of fine-grained multimodal data Upsampling to obtain multiple meta-tasks with missing modal data The model is trained on each task. Taking the i-th task as an example, there is a training set and test set The initial model parameters are θ, and the loss of the model on the training set is calculated and the model parameters are updated by gradient descent to obtain the trained model parameters.

[0019]

[0020] in Represents the gradient of the model on the training set of the ith meta-task, and α is the learning rate within the meta-task. After obtaining the updated model parameters, according to The gradient on the test set of the i-th task updates the initial parameters θ of the model, and the objective function can be written as:

[0021]

[0022] in For a task that includes many meta-tasks Take the i-th meta-task, The output result of the model trained on the i-th task on the i-th test set.

[0023] The objective function optimizes model parameters by minimizing the model's loss on the test set for each task. By training on multiple meta-tasks with missing modal information, the model acquires the ability to complete missing modal information and achieves good generalization performance. Finally, the model is trained on a dataset with fine-grained multimodal data missing to produce the final model and classification results.

[0024] Step 2: In each task, local features and global features are extracted from fine-grained multimodal data, and the attention mechanism is used to fuse the local features and global features respectively.

[0025] In each task, we first extract local and global features from fine-grained multimodal data. Taking images, text, audio, and video as examples, we first extract the features of the image modality through a convolutional neural network with a spatial attention mechanism. To X img Perform global average pooling to obtain global features x img , and obtain the global features x of audio and video modalities respectively by similar operations aud , x video , for text data, long short-term memory network is used to extract features x txt Taking the fusion of image modality and audio modality features as an example, the attention mechanism is used to calculate the dot product of the image modality and text modality features to obtain the relationship coefficient. The relationship coefficient is multiplied by the audio modality feature and added to the image modality feature to obtain the fusion feature x of the image modality and audio modality. img&aud , formulas such as:

[0026]

[0027] in Represents the relationship coefficient between the image modality and the audio modality features. In the same way, the fusion features of the image modality features and other modality features can be obtained. All fusion features are added together to obtain the fusion feature x of the image modality features and all other modality features. img_fuse :

[0028] x img_fuse =(x img&aud +x img&txt +x img&video ) / 3

[0029] where x img&txt is the fusion feature of image modality and text modality, x img&video is the fusion feature of the image modality and the video modality. For the remaining modalities, the fusion feature can also be obtained in a similar way. Finally, all features are summed up to obtain the global fusion feature:

[0030] x agg =x img_fuse +x aud_fuse +x txt_fuse +x video_fuse

[0031] where x aud_fuse 、x txt_fuse 、x video_fuse They are the fusion features of audio, text, video modal features and all other modal features. For the fusion of local features, taking the image modality as an example, we first extract the image feature X through the deep model. img , which contains 144 cases, and then uses the Gumbel top-k method to selectimg Select k key cases from the 144 cases. Specifically, for the i-th case Learn a linear model to get the score s for the corresponding case i i :

[0032]

[0033] where w T To transpose the linear model parameter matrix, b is the bias parameter of the linear model, and ReLU is the activation function; then the case scores are reparameterized according to the Gumbel-max technique:

[0034]

[0035] where |X img | represents the number of cases, λ is the temperature parameter, g i is a sample obtained from the Gumbel distribution Gumbel (0,1) = -log (-log (q)), q ~ Uniform (0,1). u i Represents the importance of case i, and selects the top k important cases to form the local features of the image modality

[0036]

[0037] For the remaining modes, local features are obtained in a similar way. For the fusion of local features of images and local features of audio, the operation is similar to formula x, and feature fusion is performed on each case:

[0038]

[0039] in The relationship between the a-th case after the image modality is filtered and the b-th case after the audio modality is filtered is calculated, and exp is the result with base e. is the normalized relationship coefficient between the image and audio modal features. After obtaining the fused features of the image and audio modalities, operations similar to global features can be performed to obtain the final fused features of the local features.

[0040] In step 3, the fused local features and global features are sent to the corresponding completion modules respectively to complete the features of the missing modalities. Finally, the completed local and global features are spliced ​​together for classification.

[0041] The local features and global features fused in step 2 are sent to the corresponding completion modules respectively. The completion modules are divided into completion modules for each modality and completion modules for overall features. Taking the global feature as an example, for the completion module of each modality, the global feature x after fusion of each modality is sent to the completion module of each modality. agg The features of each modality are decoupled through the decoupling network ψ(·). The decoupling network consists of two layers of perceptrons and ReLU activation functions. Take the image modality as an example:

[0042]

[0043] x img_fuse_re It is the decoupled image feature. The model goal is to make the l2 distance between the decoupled feature and the original feature of the image modality as small as possible:

[0044] min(x img_fuse_re -x img_fuse ) 2

[0045] For the overall feature completion module, the model goal is to make the l2 distance between the features of each modal fusion and the global fusion features as small as possible:

[0046] min(x img_fuse -x agg ) 2

[0047] This operation can make the fusion features of each modality as unaffected as possible by the lack of modality and reduce the bias of the fusion features. agg The completed global features are added to the decoupled missing mode features, and the completed global features are concatenated with the local features for classification.

[0048] Step 4: After the meta-task training is completed, the obtained model is trained on a dataset with missing fine-grained multimodal data to obtain the final model and classification results.

[0049] The trained model is then retrained on a dataset with missing fine-grained multimodal data to obtain the final model and classification results. The model has been trained on multiple tasks with missing modal data, acquiring the ability to complete missing modal information and fuse modal features. It achieves good classification results for fine-grained multimodal data in any missing modality. The model is trained using the classification cross entropy loss function and the reconstruction mean squared error loss function.

[0050] The effects of the present invention will be described below in conjunction with the application of the present invention in bird identification and food category identification.

[0051] Comparison method:

[0052] CRA proposes a cascaded residual autoencoder to complete the features of missing modalities for classification.

[0053] ViLT uses the Transformer structure to fuse and classify multimodal features.

[0054] PKU FG-XMedia is a multimodal bird dataset containing 200 bird species. It includes four modalities: image, audio, video, and text. The audio data consists of bird calls, and the text data consists of text describing the birds. We divided the dataset into a training set and a test set, using 2994 examples for training and 2930 for testing, according to the official dataset specifications. We tested the classification accuracy of the proposed method under different missingness rates. The results are as follows:

[0055] Table 1 Classification accuracy of different methods on the PKU FG-XMedia dataset under different data missing rates

[0056]

[0057] UPMC Food-101 is a multimodal classification dataset for food categories. It has two modalities: food images and food text descriptions. We divided the dataset into a training set and a test set according to the official classification of the dataset. The training set contains 16,895 samples, and the test set contains 11,507 samples. We tested the classification accuracy of the proposed method under different missing rates. The results are as follows:

[0058] Table 2 Classification accuracy of different methods on the UPMC Food-101 dataset under different data missing rates

[0059]

[0060]

[0061] From the results, the method proposed in the present invention has higher classification accuracy on missing multimodal data than previous methods, and its accuracy decreases more slowly with the increase of modality missing rate, showing better classification performance.

Claims

1. A fine-grained multimodal meta-learning recognition method for modality loss, characterized by: The following steps are involved: Step 1: Sampling on a dataset with complete fine-grained multimodal data, constructing multiple meta-tasks with missing modal data, and training the model on each task; Step 2: Within each task, extract local features and global features from fine-grained multimodal data, and use the attention mechanism to fuse the local features and global features respectively; Specifically: In each meta-task, local features and global features are first extracted from fine-grained multimodal data, and the features of the image modality are extracted through a convolutional neural network with a spatial attention mechanism. To X img Perform global average pooling to obtain global features x img , similarly, we get the global features x of audio and video modalities aud , x video , for text data, long short-term memory network is used to extract features x txt ; To fuse the image modality and audio modality features, we use the attention mechanism to calculate the dot product of the image modality and text modality features to obtain the relationship coefficient. We multiply the relationship coefficient by the audio modality feature and add the image modality feature to obtain the fusion feature x of the image modality and audio modality. img&aud , the formula is as follows: in Represents the relationship coefficient between the image modality and the audio modality features; in the same way, the fusion features of the image modality features and other modality features are obtained, and all fusion features are added together to obtain the fusion feature x of the image modality features and all other modality features img_fuse : x img_fuse =(x img&aud +x img&txt +x img&video ) / 3 where x img&txt is the fusion feature of image modality and text modality, x img&video is the fusion feature of the image modality and the video modality; for the remaining modalities, the fusion feature is obtained in a similar way, and finally all features are summed up to obtain the global fusion feature: x agg =x img_fuse +x aud_fuse +x txt_fuse +x video_fuse where x aud_fuse 、x txt_fuse 、x video_fuse They are the fusion features of audio, text, video modal features and all other modal features; for the fusion of local features, the image feature X is extracted through the deep model img , which contains 144 cases, using the Gumbel top-k method from X img Select k key cases from the 144 cases; for the i-th case Learn a linear model to get the score s for the corresponding case i i : in is the transposition of the linear model parameter matrix, b is the bias parameter of the linear model, and ReLU is the activation function; the case scores are reparameterized according to the Gumbel-max technique: where |X img | represents the number of cases, λ is the temperature parameter, g i is a sample obtained from the Gumbel distribution Gumbel(0,1)=-log(-log(q)),q~Uniform(0,1);u i Represents the importance of case i, and selects the top k important cases to form the local features of the image modality For the remaining modes, local features are obtained in a similar way. For the fusion of local features of images and local features of audio, feature fusion is performed on each case: in Calculate the relationship between the a-th case after the image modality is filtered and the b-th case after the audio modality is filtered. exp is the result with base e. It is the relationship coefficient between the normalized image modality and audio modality features. After obtaining the fusion features of the image and audio modalities, the final fusion features of the local features are obtained. Step 3: Send the fused local features and global features to the corresponding completion modules respectively to complete the features of the missing modality. Finally, the completed local and global features are spliced ​​together for classification. Step 4: After the meta-task training is completed, the obtained model is trained on a dataset with missing fine-grained multimodal data to obtain the final model and classification results.

2. The fine-grained multimodal meta-learning recognition method for modality loss according to claim 1 is characterized in that: Step 1: In a dataset with complete fine-grained multimodal data Upsampling to obtain multiple meta-tasks with missing modal data The model is trained on each task; for the i-th task, there is a training set and test set The initial model parameters are θ, and the loss of the model on the training set is calculated and the model parameters are updated by gradient descent to obtain the trained model parameters. in Represents the gradient of the model on the training set of the ith meta-task, α is the learning rate within the meta-task; after obtaining the updated model parameters, according to The gradient on the test set of the i-th task updates the initial parameters θ of the model, and the objective function is written as: in For a task that includes many meta-tasks Take the i-th meta-task, The output result of the model trained on the i-th task on the i-th test set.

3. The fine-grained multimodal meta-learning recognition method for modality loss according to claim 1 is characterized in that: Step 3: Send the fused local features and global features to the corresponding completion modules respectively. The completion modules are divided into completion modules for each modality and completion modules for overall features. For global features, the completion module of each modality sends the global features x after fusion of each modality to the corresponding completion module. agg The features of each modality are decoupled through the decoupling network ψ(·). The decoupling network consists of two layers of perceptrons and ReLU activation functions. For the image modality: x img_fuse_re It is the decoupled image feature. The model goal is to make the l2 distance between the decoupled feature and the original feature of the image modality as small as possible: min(x img_fuse_re -x img_fuse ) 2 For the overall feature completion module, the model goal is to make the l2 distance between the features of each modal fusion and the global fusion features as small as possible: min(x img_fuse -x agg ) 2 Finally, the global feature x agg The completed global features are added to the decoupled missing mode features, and the completed global features are concatenated with the local features for classification.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 3 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images