An adaptive fusion multi-modal data set recognition method based on a pre-trained model

Abstract

The application discloses a kind of based on pre-training model's adaptive fusion multi-modal dataset identification method, it is related to the multi-modal learning in depth learning field and the technical field of class incremental learning.The application proposes the parameter efficient fine-tuning method based on expert mixing and an adaptive weighted fusion module can dynamically adapt to incremental scene, avoid happening catastrophic forgetting.Meanwhile, using text enhancement technology and newly proposed loss function, the model can achieve better classification performance in the class incremental process.In addition, the application does not need to store old data in the class incremental learning process, so it will not involve privacy issues, and the parameter amount of the model is smaller, compared with several classical class incremental learning methods implemented by the application, the application obtains better classification performance on the smart classroom scene dataset.

Description

Technical Field

[0001] This invention relates to the fields of multimodal learning and incremental learning in the field of deep learning. Background Technology

[0002] Ideally, artificial intelligence should be able to mimic human learning and cognitive processes. Human perception typically relies on the coordinated use of multiple senses, such as vision and hearing, to recognize new things. In the process of human learning, humans need to continuously absorb new knowledge. This has given rise to the new research field of multimodal incremental learning.

[0003] Multimodal incremental learning integrates information from different modalities, enabling models to adapt and update in constantly changing environments. It efficiently utilizes existing data resources and newly acquired data to adapt to new information and changes. The challenge in multimodal incremental learning lies in how to integrate and utilize the complementary information contained in different modalities and mitigate catastrophic forgetting in the model.

[0004] Current research in the field of class incremental learning largely focuses on class incremental classification tasks for single-modality images. With the rapid development of pre-trained visual language models, the paper "Boosting continual learning of vision-language models via mixture-of-experts adapters" applies the CLIP model pre-trained on a large-scale image-text pair dataset to class incremental learning using a parameter-efficient fine-tuning method to address the catastrophic forgetting problem, achieving remarkable results. However, these methods cannot handle multimodal datasets like the Smart Classroom dataset (containing text, image, and audio modalities), and therefore cannot utilize the gain information from different modalities.

[0005] The paper “Audioclip: Extending clip to image, text and audio” integrates an audio encoder into a visual language model. However, AudioCLIP has limited generalization ability on specialized data and suffers a significant performance drop when performing incremental fine-tuning. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the prior art by proposing an adaptive fusion multimodal incremental learning method based on a pre-trained model. This method enables efficient fine-tuning of the parameters of the multimodal pre-trained model during the multimodal incremental learning process, thereby overcoming catastrophic forgetting and better utilizing multimodal information to complete the classification task.

[0007] The technical solution of this invention is as follows:

[0008] An adaptive fusion multimodal dataset recognition method based on a pre-trained model includes the following steps:

[0009] Step 1: Collect multimodal sample data simultaneously in different scenarios, classify the sample data, and then preprocess the collected sample data so that each sample data includes three modalities: image, audio and text, with the text being the category name;

[0010] Step 2: Perform task partitioning for incremental learning of classes. Divide the preprocessed dataset into multiple tasks, including a base task and several incremental tasks, with the classes in each task being non-overlapping.

[0011] Step 3: Use a set of text prompt templates to enhance the text data, and obtain multiple enhanced set representations for each category label;

[0012] Step 4: Use an image feature extractor Audio Feature Extractor Text Feature Extractor Feature extraction was performed on the image data, audio data, and enhanced text data respectively. The output is then projected. The output is then averaged to obtain image features, audio features, and text features respectively.

[0013] Step 5: Use an adaptive weighted fusion module to fuse image features and audio features to obtain fused features;

[0014] Step 6: Calculate the cosine similarity between the fused features and the text features, and obtain the recognition result based on the cosine similarity.

[0015] Furthermore, the sample data was categorized into the following 33 categories: analysis, erasing the blackboard, writing on the blackboard, discussion, drinking water, eating, packing a schoolbag, raising a hand, listening to a lecture, listening to music, packing a computer, reading, relaxing, reviewing, scratching one's head, setting up equipment, sleeping, speaking, standing, student demonstration, taking a schoolbag, taking a bottle, taking off clothes, taking a photo, conversing, teaching, unknown, using a computer, using a tablet, using a mobile phone, walking, writing, and yawning.

[0016] Furthermore, in step 4, the audio feature extractor It includes: L basic units connected in series. The input data of each unit passes through a first normalization layer and a multi-head self-attention module in sequence. The outputs of the normalization module and the multi-head self-attention module are fused to obtain the first fused data. The first fused data passes through a second normalization layer and a multilayer perceptron in sequence. The first fused data and the output data of the multilayer perceptron are fused to obtain the output of the basic unit.

[0017] The image feature extractor and the text feature extractor have the same structure, comprising L sequentially connected basic units. The input data of each unit sequentially passes through a first normalization layer and a multi-head self-attention module, and first fused data is obtained after fusing the outputs of the layer normalization module and the multi-head self-attention module; the first fused data is divided into three paths, wherein one path is not processed, the second path sequentially passes through a second normalization layer and a multi-layer perceptron, and the third path passes through a MoE structure, then the three paths are fused to serve as the output of the basic unit; the MoE structure comprises router layers and low-rank adaptation adapter layers, and the number of routers is the same as the number of low-rank adaptation adapters; for task t, the router corresponding to the task allocates the task to the corresponding low-rank adaptation adapter, and the low-rank adaptation adapter outputs the result after processing.

[0018] Further, the specific method of the MoE structure is:

[0019]

[0020] wherein, y t represents the output of the MoE structure, represents the router in task t for the i-th low-rank adaptation adapter ε i allocated gating weight, x t represents the feature vector processed by task t, N e is the total number of routers;

[0021] the calculation formula of the gating weight is as follows:

[0022]

[0023] wherein, the Softmax(·) function is used for normalization operation, the Topk(·) function is used for selecting the top k most relevant low-rank adaptation adapters, c t represents the <[BOS_never_used_51bce0c785ca2f68081bfa7d91973934]> token in x t .

[0024] Further, the specific method of the adaptive weighted fusion module in step 5 is:

[0025] a masking threshold th is preset in the adaptive weighted fusion module to mask noise interference in weak modalities; the correlation r between the image feature and the audio feature is calculated and compared with th; if r<th, only is used, otherwise and Weighted concatenation is performed, and then processed through a feature fusion layer;

[0026] The formula for calculating correlation is as follows:

[0027]

[0028] for The corresponding average value, where i represents the i-th feature.

[0029] Furthermore, the total loss function for training steps 2 to 6 includes continuously weighted cross-modal prototype contrast loss and multimodal fusion mutual information loss function;

[0030] Continuously weighted cross-modal prototype contrastive loss function The formula is as follows:

[0031]

[0032] The sim(·,·) function is used to calculate the cosine similarity between two vectors. This represents the cross-entropy loss, where N represents the total number of features;

[0033] Multimodal fusion mutual information loss function The goal is to maximize the fusion features Image features and audio features The mutual information between them is expressed by the following formula:

[0034]

[0035]

[0036] The total loss function is composed of:

[0037]

[0038] Where α∈[0,1] are hyperparameters, and P(.) represents the probability distribution function.

[0039] This invention explores the application of multimodal base models in the field of class incremental learning, providing new ideas for research in this area. In the context of class incremental scenarios, for multi-source heterogeneous input data, this invention proposes an efficient parameter fine-tuning method based on Mixture of Experts (MoE) and an adaptive weighted fusion module that can dynamically adapt to the incremental scenario and avoid catastrophic forgetting. Simultaneously, by utilizing text enhancement techniques and a newly proposed loss function, the model achieves better classification performance during class incremental learning. Furthermore, this invention does not require storing old data during class incremental learning, thus avoiding privacy issues, and the model has a smaller number of parameters. Compared with several classic class incremental learning methods, this invention achieves superior classification performance on a smart classroom scenario dataset. Attached Figure Description

[0040] Figure 1 This is an overall flowchart of the adaptive fusion multimodal incremental learning method based on a pre-trained model in an embodiment of the present invention.

[0041] Figure 2 This is a general framework diagram of the adaptive fusion multimodal incremental learning method based on a pre-trained model in an embodiment of the present invention.

[0042] Figure 3 This is a structural diagram of the image and text feature extractors in the multimodal incremental feature extractor in an embodiment of the present invention.

[0043] Figure 4 This is a graph showing the comparison of multimodal incremental task classification performance in embodiments of the present invention. Detailed Implementation

[0044] To address the limitations of the multimodal pre-trained model AudioCLIP in generalizing to specialized data and the catastrophic forgetting that occurs during incremental learning, this invention proposes an adaptive fusion multimodal incremental learning method based on a pre-trained model.

[0045] like Figure 1 As shown, embodiments of the present invention include the following steps:

[0046] Step 1: Process the smart classroom scenario dataset, obtain data containing three modalities—image, audio, and text—as a multimodal dataset for training and testing, and perform data preprocessing;

[0047] The dataset in this embodiment consists of data from a smart classroom scenario dataset. By randomly sampling the image set in the smart classroom scenario dataset, each original image is cropped according to its bounding box to become a separate image sample. The audio data corresponding to each new image is the audio associated with its original image, and the label on the bounding box is the label corresponding to this new image. Further processing is then performed to form training and testing sets.

[0048] The preprocessing used included: resizing the image to 224×224 pixels and recalculating the pixel values ​​of the RGB channels according to the following formula: Normalization is performed, where x is the original input data, μ is the mean of all training samples, and σ is the standard deviation of all training samples. ′ The data is obtained after normalization; Mel-Frequency Cepstral Coefficients (MFCC) features are directly extracted from the audio file.

[0049] Step 2: In the incremental scenario, divide the dataset obtained in Step 1 into tasks;

[0050] The dataset obtained in step one is divided into T = 5 tasks, including 1 basic task (task 0) and 4 incremental tasks. The model learns C on the basic task. b =13 categories, learning C on each incremental task i There are 5 categories, and the categories learned for each task are mutually exclusive. After the model completes training and testing on the basic task, it sequentially accesses each incremental task for training and testing. During the training phase of task t, the model can only access the data in task t; while during the testing phase of task t, the model needs to be tested on all seen categories, that is, the test set contains test samples of all categories from task 0 to t.

[0051] Step 3: Increase the diversity of text branches;

[0052] like Figure 2 As shown in the text branch at the bottom left, a set of text prompt templates is used for each of the 33 category labels in the dataset. Each category label is expanded to include more detailed and descriptive N text sentences, thus providing the model with more contextual information, making it easier for the model to understand and distinguish different categories, and improving the model's generalization ability on unseen data.

[0053] Step 4: Construct a parameter-efficient multimodal incremental feature extractor;

[0054] like Figure 2As shown on the left, the multimodal incremental classification model contains three feature extractors. These are used to extract input data from three modalities: image, audio, and text. The audio feature extractor is one such example. Employs a pre-trained audio encoder, which is kept frozen throughout the training phase; and an image feature extractor. and text feature extractor MoE structures are added to the pre-trained image encoder and text encoder respectively. During the training phase, the rest of the two feature extractors are kept frozen except for the newly added MoE structures.

[0055] MoE is composed of a group of experts It consists of task-specific routers, where the number of experts is equal to the total number of tasks, i.e., N. e =T. For task t, there is a router specific to that task. It integrates the expert's output through gating weights. The calculation formula is as follows:

[0056]

[0057] in, In task t For the i-th expert ε i Each assigned gating weight, x t Let y represent the feature vector processed by task t. t This represents the output of the MoE structure. Gating weights. The calculation formula is as follows:

[0058]

[0059] The Softmax(·) function is used for normalization, and the Topk(·) function is used to select the top k most relevant experts. t x represents t The [CLS] tag in the text.

[0060] Here, "expert" refers to a low-rank adaptation (LoRA) with fewer parameters. It achieves efficient fine-tuning of parameters in large-scale pre-trained models by decomposing the parameters into the product of two low-rank matrices. During training, different expert adapters learn knowledge from different tasks. When learning a new task, it iterates through each sample in the training set for that task. For each sample, it iterates through each expert. If the gating value corresponding to a sample at a certain expert is equal to 0, then the weight update of that expert is prevented during backpropagation, thus preventing interference between knowledge from different categories.

[0061] The multimodal incremental feature extractor constructed here is as follows: Figure 3 As shown, it utilizes the shared knowledge learned from old tasks by accessing frozen experts, and learns task-specific knowledge for new tasks by optimizing trainable experts.

[0062] Step 5: Construct an adaptive weighted fusion module to perform feature fusion on the image and audio features output by the multimodal incremental feature extractor;

[0063] Three parallel feature extractors are used to respectively extract features of multimodal input data (v t , a t , l t ), to obtain Through a frozen projection layer , the features of three different modalities can be projected into a common embedding space. In order to overcome the shortcomings of a single modality and fully utilize the advantages of both image and audio modalities, the image features are processed herein and audio features are processed through a trainable adaptive weighted fusion module to obtain fused features:

[0064]

[0065] The structure of the adaptive weighted fusion module is specifically introduced below. In the smart classroom scenario dataset, the data quality of the image modality is better than that of the audio modality, so the image modality is referred to as a strong modality and the audio modality as a weak modality. This is to fully utilize the effective complementary information in both image and audio modalities and prevent the model from being interfered by low data quality of a single modality.

[0066] First, a masking threshold th is preset in to mask noise interference in the weak modality. By calculating the correlation r between and , and comparing it with th. If r < th, only is used; otherwise, and are weighted and spliced, and processed through a feature fusion layer. The calculation formula of correlation is as follows:

[0067]

[0068] Second, The feature fusion layer in this model is built upon an MLP (Multi-Level Processing) architecture and incorporates learnable weight parameters and an attention mechanism. Image and audio features are weighted by assigning a learnable weight parameter to each modality's features. These two learnable weight parameters can be dynamically adjusted based on the data quality of that modality or its performance during training. Their initial value is 1, indicating equal emphasis on image and audio features initially. Setting these learnable weight parameters helps the model adaptively adjust the influence of different modalities, better balance their contributions, capture complementary information between modalities, reduce noise and redundancy, and enhance the effectiveness of feature fusion. Furthermore, an attention mechanism is used to determine which modality's features should be prioritized during feature fusion, enabling the model to better focus on important information and ignore irrelevant or secondary information, thereby improving the model's adaptability.

[0069] Step Six: Obtain the model's prediction results;

[0070] By calculating fusion features With text prototype The probability prediction distribution is obtained by using the cosine similarity between the pairs of values, as shown in the following formula:

[0071]

[0072] The Softmax(·) function is used to convert the similarity score into a probability distribution, and the sim(·,·) function is used to calculate the cosine similarity between two vectors. It is a vector where each element represents the probability that a multimodal input sample belongs to a certain class. Model selection. The category corresponding to the largest element is used as the final prediction result for classification decision.

[0073] Step 7: Construct a loss function for comparative training of multimodal incremental classification models;

[0074] During the training phase, the model is optimized through backpropagation using the constructed total loss function. The total loss function combines two loss functions: continuously weighted cross-modal prototype contrastive loss and multimodal fusion mutual information loss.

[0075] Continuously weighted cross-modal prototype contrast loss Cross-entropy loss Based on this, a continuous similarity measure w is used. ij As a weight, cross-modal alignment is performed, achieving better modal alignment results. The formula is as follows:

[0076]

[0077] Multimodal fusion mutual information loss The goal is to maximize the fusion features Image features and audio features The mutual information between them allows the fused features to contain more information from the image and audio modalities, more fully capturing the detailed information of different modalities and reducing the loss of effective information. The formula is as follows:

[0078]

[0079] The total loss function is composed of:

[0080]

[0081] Where α∈[0,1] are hyperparameters.

[0082] Step 8: After training for a task is completed, use the trained multimodal incremental classification model to classify on the test set and evaluate its classification accuracy and forgetting rate; repeat this process until the last task is completed.

[0083] The results of the experimental test are as follows Figure 4 As shown, the model trained using the pre-trained model-based adaptive fusion multimodal incremental learning method of this invention achieves optimal classification performance on the smart classroom scenario dataset compared to direct generalization and partial fine-tuning using a pre-trained model, and the two classic continuous learning methods LwF and iCaRL based on AudioCLIP. This invention fills the gap in incremental learning research in the field of multi-source heterogeneous data input and classification.

Claims

1. An adaptive fusion multimodal dataset recognition method based on a pre-trained model, comprising the following steps: Step 1: Collect multimodal sample data simultaneously in different scenarios, classify the sample data, and then preprocess the collected sample data so that each sample data includes three modalities: image, audio and text, with the text being the category name; Step 2: Perform task partitioning for incremental learning of classes. Divide the preprocessed dataset into multiple tasks, including a base task and several incremental tasks, with the classes in each task being non-overlapping. Step 3: Use a set of text prompt templates to enhance the text data, and obtain multiple enhanced set representations for each category label; Step 4: Use an image feature extractor Audio Feature Extractor Text feature extractor Feature extraction was performed on the image data, audio data, and enhanced text data respectively. The output is then projected. The output is then averaged to obtain image features, audio features, and text features respectively. Step 5: Use an adaptive weighted fusion module to fuse image features and audio features to obtain fused features; The specific method of the adaptive weighted fusion module is as follows: A masking threshold is preset in the adaptive weighted fusion module. To shield against noise interference in weak modes; by calculating image features and audio features Correlation between ,and Compare; if Only use Otherwise, and Weighted concatenation is performed, and then processed through a feature fusion layer; The formula for calculating correlation is as follows: ； , for , The corresponding average value, where i represents the i-th feature; Step 6: Calculate the cosine similarity between the fused features and the text features, and obtain the recognition result based on the cosine similarity.

2. The adaptive fusion multimodal dataset recognition method based on a pre-trained model as described in claim 1, characterized in that, The audio feature extractor in step 4 It includes: L basic units connected in series. The input data of each unit passes through a first normalization layer and a multi-head self-attention module in sequence. The outputs of the normalization module and the multi-head self-attention module are fused to obtain the first fused data. The first fused data passes through a second normalization layer and a multilayer perceptron in sequence. The first fused data and the output data of the multilayer perceptron are fused to obtain the output of the basic unit. The image feature extractor and text feature extractor The structure is identical, comprising L basic units connected in series. The input data of each unit passes sequentially through a first normalization layer and a multi-head self-attention module. The outputs of the normalization module and the multi-head self-attention module are fused to obtain the first fused data. The first fused data is divided into three paths: one path is not processed, the second path passes sequentially through a second normalization layer and a multilayer perceptron, and the third path passes through a MoE structure. The three paths are then fused to serve as the output of the basic unit. The MoE structure includes a router layer and a low-rank adaptation adapter layer, with the number of routers and low-rank adaptation adapters being the same. For task t, a router with specific characteristics is used to assign the task, which is then assigned to the corresponding low-rank adaptation adapter for processing and output.

3. The adaptive fusion multimodal dataset recognition method based on a pre-trained model as described in claim 1, characterized in that, The specific method for constructing the MoE structure is as follows: ； in, This represents the output of the MoE structure. Indicates in the task Middle router For the first Low-rank adaptation adapter Assigned gating weights, Indicates task The feature vector being processed This represents the total number of routers. Gating weights The calculation formula is as follows: ； in, The function is used to perform normalization operations. The function is used to select before The most relevant low-rank adaptation adapter express The [CLS] tag in the text.

4. The adaptive fusion multimodal dataset recognition method based on a pre-trained model as described in claim 1, characterized in that, The total loss function for training steps 2 through 6 includes continuously weighted cross-modal prototype contrast loss and multimodal fusion mutual information loss function; Continuously weighted cross-modal prototype contrastive loss function The formula is as follows: ； ； The function is used to calculate the cosine similarity between two vectors. This represents the cross-entropy loss, where N represents the total number of features; Multimodal fusion mutual information loss function The goal is to maximize the fusion features Image features and audio features The mutual information between them is expressed by the following formula: ； ； ； The total loss function is composed of: ； in, For hyperparameters, This represents the probability distribution function.

5. The adaptive fusion multimodal dataset recognition method based on a pre-trained model as described in claim 1, characterized in that, The sample data were categorized into the following 33 categories: analysis, erasing the blackboard, writing on the blackboard, discussion, drinking water, eating, packing a schoolbag, raising a hand, listening to a lecture, listening to music, packing a computer, reading, relaxing, reviewing, scratching one's head, setting up equipment, sleeping, speaking, standing, student demonstration, taking a schoolbag, taking a bottle, taking off clothes, taking a photo, conversing, teaching, unknown, using a computer, using a tablet, using a mobile phone, walking, writing, and yawning.

Images