Self-adaptive fusion multi-modal data set identification method based on pre-training model
By using the adaptive fusion multimodal class incremental learning method in multimodal class incremental learning, combined with MoE structure and adaptive weighted fusion module, the class incremental learning problem in multimodal data set is solved, and the class incremental learning problem is achieved in The efficient classification performance and small parameter quantity on multimodal data sets make up for the gaps in the field of multi-source heterogeneous data input and classification in the prior art.
Patent Information
- Application Number
- CN202411012805.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The prior art is difficult to effectively handle class incremental learning tasks in multimodal data sets (including text, image and audio data), especially in the case of avoiding catastrophic forgetting, and cannot fully utilize multimodal information for classification tasks.
Adaptive fusion multimodal class incremental learning method based on pre-trained models is adopted to enhance text through text prompt templates, combine image and audio feature extractors to use MoE structures for feature extraction, and use adaptive weighted fusion module and specific losses. Functions are efficiently fine-tuned to parameters, dynamically adapt to incremental scenarios, and avoid forgetting.
In the process of class incremental learning, better classification performance and smaller model parameters are achieved, catastrophic forgetting is avoided, and better classification performance is achieved on the smart classroom scene dataset.
Smart Images

Figure CN119004168A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multimodal learning and class incremental learning in the field of deep learning. Background Art
[0002] Ideally, artificial intelligence should be able to imitate the learning and cognitive process of humans, and human perception usually relies on the synergy of multiple senses, such as vision, hearing and other multimodal ways to recognize new things. In the process of human learning, humans need to constantly absorb new knowledge. This has led to the emergence of a new research field of multimodal incremental learning.
[0003] Multimodal incremental learning can integrate information from different modalities, allowing the model to adapt and update in a constantly changing environment, and efficiently utilize existing data resources and newly acquired data to adapt to new information and changes. In multimodal incremental learning, the difficulty of research lies in how to integrate and utilize the complementary information contained in different modal data and alleviate the catastrophic forgetting of the model.
[0004] At present, most of the research in the field of incremental learning focuses on incremental classification tasks of a single image modality. With the rapid development of pre-trained visual language models, the paper "Boosting continual learning of vision-language models via mixture-of-experts adapters" applies the CLIP model pre-trained on a large-scale image-text pair dataset to incremental learning through a parameter-efficient fine-tuning method to solve the problem of catastrophic forgetting, and has achieved very good results. However, these methods cannot handle multi-modal datasets such as the Smart Classroom dataset (which contains data in three modalities: text, image, and audio), and cannot utilize gain information from different modalities.
[0005] In the paper “Audioclip: Extending clip to image, text and audio”, an audio encoder is integrated into the visual language model. However, AudioCLIP has limited generalization ability on specialized data and suffers from significant performance degradation when incrementally fine-tuned. Summary of the invention
[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose an adaptive fusion multimodal incremental learning method based on a pre-trained model, so as to achieve efficient fine-tuning of the parameters of the multimodal pre-trained model in the multimodal incremental learning process, while overcoming catastrophic forgetting and better utilizing multimodal information to complete classification tasks.
[0007] The technical solution of the present invention is as follows:
[0008] A method for adaptive fusion multimodal data set recognition based on a pre-trained model comprises the following steps:
[0009] Step 1: Collect multimodal sample data in different scenarios at the same time, classify the sample data, and then preprocess the collected sample data so that each sample data includes three modalities: image, audio, and text, with text as the category name;
[0010] Step 2: Perform task division for incremental learning. Divide the preprocessed dataset into multiple tasks, including a basic task and several incremental tasks, and the categories in each task are disjoint.
[0011] Step 3: Use a set of text hint templates to perform text enhancement on the text data to obtain multiple enhanced set representations for each category label;
[0012] Step 4: Use image feature extractor Audio Feature Extractor Text Feature Extractor Feature extraction is performed on image data, audio data, and enhanced text data respectively. The output is then projected. The output of is then averaged to obtain image features, audio features, and text features respectively;
[0013] Step 5: Adopt an adaptive weighted fusion module to fuse the image features and the audio features to obtain fused features;
[0014] Step 6: Calculate the cosine similarity between the fusion feature and the text feature, and obtain the recognition result based on the cosine similarity.
[0015] Furthermore, the sample data were classified into the following 33 categories: analyzing, erasing the blackboard, writing on the blackboard, discussing, drinking water, eating, packing schoolbags, raising hands, listening to lectures, listening to music, packing up computers, reading, relaxing, reviewing, scratching heads, setting up equipment, sleeping, speaking, standing, student demonstrations, taking schoolbags, taking bottles, taking off clothes, taking pictures, talking, teaching, unknown, using computers, using tablets, using mobile phones, walking, writing, and yawning.
[0016] Furthermore, the audio feature extractor in step 4 The method comprises: L basic units connected in series in sequence, the input data of each unit is sequentially passed through a first normalization layer, a multi-head self-attention module, and the outputs of the layer normalization module and the multi-head self-attention module are fused to obtain first fused data; the first fused data is sequentially passed through a second normalization layer and a multi-layer perceptron, and the first fused data and the output data of the multi-layer perceptron are fused as the output of the basic unit;
[0017] The image feature extractor and the text feature extractor have the same structure, including L basic units connected in series in sequence. The input data of each unit passes through a first normalization layer, a multi-head self-attention module, a layer normalization module in sequence, and the outputs of the multi-head self-attention modules are fused to obtain first fused data; the first fused data is divided into three paths, one path is not processed, the second path passes through a second normalization layer and a multi-layer perceptron in sequence, the third path passes through a MoE structure, and then the three paths are fused as the output of this basic unit; the MoE structure includes a router layer and a low-rank adaptation adapter layer, and the number of routers is the same as the number of low-rank adaptation adapters; for a task t, the router with the feature assigns the task to the corresponding low-rank adaptation adapter, and the task is processed by the low-rank adaptation adapter and then output.
[0018] Furthermore, the specific method of the MoE structure is as follows:
[0019]
[0020] where y t represents the output of the MoE structure, represents the gating weight assigned by the router to the i-th low-rank adaptation adapter ε i in task t, x t represents the feature vector processed by task t, and N e is the total number of routers;
[0021] The gating weight is calculated as follows:
[0022]
[0023] where the Softmax(·) function is used for the normalization operation, the Topk(·) function is used to select the top k most relevant low-rank adaptation adapters, and c t represents the [CLS] token in x t .
[0024] Furthermore, the specific method of the adaptive weighted fusion module in step 5 is as follows:
[0025] A shielding threshold th is preset in the adaptive weighted fusion module to shield the noise interference in the weak modality; by calculating the correlation r between the image feature and the audio feature and comparing it with th; if r < th, only use otherwise, and Perform weighted concatenation and process through feature fusion layer;
[0026] The calculation formula of correlation is as follows:
[0027]
[0028] for The corresponding average value, i represents the i-th feature.
[0029] Furthermore, the total loss function when training steps 2 to 6 includes a continuous weighted cross-modal prototype contrast loss and a multimodal fusion mutual information loss function;
[0030] Continuously Weighted Cross-Modal Prototype Contrastive Loss Function The formula is as follows:
[0031]
[0032] The sim(·,·) function is used to calculate the cosine similarity between two vectors. represents the cross entropy loss, and N represents the total number of features;
[0033] Multimodal fusion mutual information loss function The goal is to maximize the fusion feature With image features and audio features The mutual information between them is given by the following formula:
[0034]
[0035]
[0036] The composition of the total loss function is:
[0037]
[0038] Among them, α∈[0,1] is a hyperparameter, and P(.) represents the probability distribution function.
[0039] The present invention explores the application of multimodal basic models in the field of class incremental learning, and provides new ideas for the research in the field of multimodal class incremental learning. Under the setting of class incremental scenarios, for multi-source heterogeneous input data, the present invention proposes an efficient parameter fine-tuning method based on expert mixture (MoE, Mixture of Experts) and an adaptive weighted fusion module that can dynamically adapt to incremental scenarios to avoid catastrophic forgetting. At the same time, text enhancement technology and the newly proposed loss function are used to enable the model to achieve better classification performance in the class incremental process. In addition, the present invention does not need to store old data in the class incremental learning process, so there is no privacy issue, and the number of parameters of the model is small. Compared with several classic class incremental learning methods, the present invention obtains better classification performance on the smart classroom scene data set. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is an overall flow chart of the adaptive fusion multimodal incremental learning method based on the pre-trained model in an embodiment of the present invention.
[0041] Figure 2 It is an overall framework diagram of the adaptive fusion multimodal incremental learning method based on the pre-training model in an embodiment of the present invention.
[0042] Figure 3 It is a structural diagram of the image and text feature extractors in the multimodal incremental feature extractor in an embodiment of the present invention.
[0043] Figure 4 It is a comparison curve chart of multimodal incremental task classification effects in an embodiment of the present invention. DETAILED DESCRIPTION
[0044] In order to solve the problems that the multimodal pre-trained model AudioCLIP has limited generalization ability on dedicated data and catastrophic forgetting occurs in the incremental process, the present invention proposes an adaptive fusion multimodal incremental learning method based on the pre-trained model.
[0045] like Figure 1 As shown, the embodiment of the present invention includes the following steps:
[0046] Step 1: Process the smart classroom scenario dataset, obtain data containing three modalities of image, audio and text as a multimodal dataset for training and testing, and perform data preprocessing;
[0047] The data set in this embodiment is composed of data from the smart classroom scene data set. By randomly sampling the picture set in the smart classroom scene data set, each original picture is cropped according to the annotation box as a separate picture sample. The audio data corresponding to each new picture is the audio associated with the original picture to which it belongs, and the label carried by the annotation box is the label corresponding to this new picture. Further sorting is performed to form a training set and a test set.
[0048] The preprocessing used includes: transforming the image size to 224×224, and calculating the pixel values of the three RGB channels according to the formula: Normalization is performed, where x is the original input data, μ is the mean of all training samples, σ is the standard deviation of all training samples, and x ′ The data is obtained after normalization processing; the Mel-Frequency Cepstral Coefficients (MFCC) features are directly extracted from the audio file.
[0049] Step 2: In the class increment scenario, divide the data set obtained in step 1 into tasks;
[0050] Perform task division and divide the data set obtained in step 1 into T = 5 tasks, including 1 basic task (task 0) and 4 incremental tasks. The model learns C on the basic task b = 13 categories, C is learned on each incremental task i = 5 categories, and the categories learned by each task are mutually exclusive. After the model completes the basic task training and testing, it sequentially accesses each incremental task for training and testing. During the training phase of task t, the model can only access the data in task t; and during the testing phase of task t, the model needs to be tested on all categories it has seen, that is, the test set contains test samples of all categories in tasks 0 to t.
[0051] Step 3: Increase the diversity of text branches;
[0052] like Figure 2 As shown in the text branch on the lower left, a set of text prompt templates are used for each of the 33 category labels in the dataset. Each category label is expanded to contain more details and describe N text sentences, thereby providing more contextual information for the model, making it easier for the model to understand and distinguish different categories, and improving the model's generalization ability on unseen data.
[0053] Step 4: Build a parameter-efficient multimodal incremental feature extractor;
[0054] like Figure 2As shown in the left part, the multimodal incremental classification model contains three feature extractors They are used to extract input data from three modalities: image, audio, and text. Use a pre-trained audio encoder and keep it frozen during the training phase; image feature extractor and text feature extractor The MoE structure is added to the pre-trained image encoder and text encoder respectively. During the training phase, the rest of the two feature extractors except the newly added MoE structure are kept frozen.
[0055] The MoE is composed of a group of experts and task-specific routers, where the number of experts is equal to the total number of tasks, i.e., N e = T. For task t, there is a task-specific router It integrates the output of experts through gated weights. The calculation formula is as follows:
[0056]
[0057] in, In task t is the i-th expert ε i The assigned gating weights, x t Represents the feature vector processed by task t, y t Represents the output of the MoE structure. Gating weight The calculation formula is as follows:
[0058]
[0059] Among them, the Softmax(·) function is used for normalization operation, the Topk(·) function is used to select the top k most relevant experts, and c t Represents x t The [CLS] tag in .
[0060] The expert here refers to the low-rank adaptation adapter (LoRA) with fewer parameters. It achieves efficient fine-tuning of the parameters of large-scale pre-trained models by decomposing the parameters into the product of two low-rank matrices. During the training process, different expert adapters will learn knowledge in different tasks. When learning a new task, iterate over each sample in the task training set, and for each sample, iterate over each expert. If the gate value corresponding to the sample at a certain expert is equal to 0, the weight update of this expert is prevented during the back-propagation process, thereby preventing interference between knowledge of different categories.
[0061] The multimodal incremental feature extractor constructed here is as follows Figure 3 As shown, it utilizes the shared knowledge learned in old tasks by accessing frozen experts and learns the specific knowledge of new tasks by optimizing trainable experts.
[0062] Step Five: Construct an adaptive weighted fusion module to perform feature fusion on the image and audio features output by the multi-modal incremental feature extractor;
[0063] Use three parallel feature extractors to separately extract the features of the multi-modal input data (v t , a t , l t ), obtaining After passing through a frozen projection layer , the features of the three different modalities can be projected into a common embedding space. To overcome the deficiencies of a single modality and make full use of the advantages of both the image and audio modalities, the image features and the audio features are passed through a trainable adaptive weighted fusion module to obtain the fused features:
[0064]
[0065] The following specifically introduces the composition of the adaptive weighted fusion module In the smart classroom scenario dataset, the data quality of the image modality is better than that of the audio modality. Therefore, the image modality is called the strong modality and the audio modality is called the weak modality. To be able to make full use of the effective complementary information in both the image and audio modalities and prevent the low data quality of a single modality from interfering with the model.
[0066] First, a shielding threshold th is preset in to shield the noise interference in the weak modality. By calculating the correlation r between and and comparing it with th. If r < th, only is used. Otherwise, and are weighted and concatenated and processed through a feature fusion layer. The formula for calculating the correlation is as follows:
[0067]
[0068] Secondly, The feature fusion layer in is built based on MLP, and introduces learnable weight parameters and attention mechanism. By setting a learnable weight parameter for the features of each modality, image features and audio features are weighted. These two learnable weight parameters can be dynamically adjusted according to the data quality of the modality or the performance during the training process. Their initial values are both 1, indicating that image and audio features are given equal importance in the initial state. The setting of learnable weight parameters helps the model to adaptively adjust the influence of different modalities, better balance the contributions of different modalities, capture complementary information between modalities, reduce noise and redundancy, and enhance the effectiveness of feature fusion. In addition, by using the attention mechanism, it is decided which modality's features should be paid more attention to during feature fusion, so that the model can better focus on important information and ignore irrelevant or minor information, thereby improving the model's adaptability.
[0069] Step 6: Get the prediction results of the model;
[0070] By calculating the fusion features With text prototype The cosine similarity between them is used to obtain the probability prediction distribution, and the formula is as follows:
[0071]
[0072] Among them, the Softmax(·) function is used to convert the similarity score into a probability distribution, and the sim(·,·) function is used to calculate the cosine similarity between two vectors. is a vector where each element represents the probability that a multimodal input sample belongs to a certain category. Model selection The category corresponding to the largest element in is taken as the final prediction result for classification decision.
[0073] Step 7: Construct a loss function for comparative training of multimodal incremental classification models;
[0074] During the training phase, the model is optimized by backpropagation using the constructed total loss function, which combines the two loss functions: continuous weighted cross-modal prototype contrast loss and multimodal fusion mutual information loss.
[0075] Continuously Weighted Cross-Modal Prototype Contrastive Loss In cross entropy loss Based on the use of a continuous similarity measure w ij As a weight, cross-modal alignment is performed to achieve better modality alignment. The formula is as follows:
[0076]
[0077] Multimodal fusion mutual information loss The goal is to maximize the fusion feature With image features and audio features The mutual information between them can make the fusion feature contain more information from the image modality and the audio modality, more fully capture the details of different modalities, and reduce the loss of effective information. The formula is as follows:
[0078]
[0079] The composition of the total loss function is:
[0080]
[0081] Among them, α∈[0,1] is a hyperparameter.
[0082] Step 8: After the training of a task is completed, use the trained multimodal incremental classification model to classify the test set and evaluate its classification accuracy and forgetting rate; and so on until the last task is completed.
[0083] The results of the experimental test are as follows Figure 4 As shown in the figure, the model trained by the adaptive fusion multimodal incremental learning method based on the pre-trained model in the present invention achieves the best classification performance on the smart classroom scene dataset compared with direct generalization and partial fine-tuning of the pre-trained model and the two classic continuous learning methods of LwF and iCaRL based on AudioCLIP. This invention fills the gap in the research of incremental learning in the field of multi-source heterogeneous data input and classification.
Claims
1. A method for adaptive fusion multimodal data set recognition based on a pre-trained model, comprising the following steps: Step 1: Collect multimodal sample data in different scenarios at the same time, classify the sample data, and then preprocess the collected sample data so that each sample data includes three modalities: image, audio, and text, with text as the category name; Step 2: Perform task division for incremental learning. Divide the preprocessed dataset into multiple tasks, including a basic task and several incremental tasks, and the categories in each task are disjoint. Step 3: Use a set of text hint templates to perform text enhancement on the text data to obtain multiple enhanced set representations for each category label; Step 4: Use image feature extractor Audio Feature Extractor Text Feature Extractor Feature extraction is performed on image data, audio data, and enhanced text data respectively. The output is then projected. The output of is then averaged to obtain image features, audio features, and text features respectively; Step 5: Adopt an adaptive weighted fusion module to fuse the image features and the audio features to obtain fused features; Step 6: Calculate the cosine similarity between the fusion feature and the text feature, and obtain the recognition result based on the cosine similarity.
2. The method for adaptive fusion multimodal data set recognition based on a pre-training model according to claim 1, characterized in that: The audio feature extractor in step 4 The method comprises: L basic units connected in series in sequence, the input data of each unit is sequentially passed through a first normalization layer, a multi-head self-attention module, and the outputs of the layer normalization module and the multi-head self-attention module are fused to obtain first fused data; the first fused data is sequentially passed through a second normalization layer and a multi-layer perceptron, and the first fused data and the output data of the multi-layer perceptron are fused as the output of the basic unit; The image feature extractor and text feature extractor The structure is the same, including L basic units connected in series in sequence, the input data of each unit passes through the first normalization layer, the multi-head self-attention module in sequence, and the outputs of the layer normalization module and the multi-head self-attention module are fused to obtain the first fused data; the first fused data is divided into three paths, one path is not processed, the second path passes through the second normalization layer and the multi-layer perceptron in sequence, and the third path passes through the MoE structure, and then the three paths are fused as the output of the basic unit; the MoE structure includes a router layer and a low-rank adaptive adapter layer, and the number of the routers is the same as the number of the low-rank adaptive adapters; for the router with characteristics of task t, the task is assigned to the corresponding low-rank adaptive adapter, which is processed by the low-rank adaptive adapter and output.
3. The method for adaptive fusion multimodal data set recognition based on a pre-training model according to claim 1, characterized in that: The specific method of MoE structure is: Among them, y t represents the output of the MoE structure, Indicates that the router in task t is the i-th low-rank adaptation adapter ε i The assigned gating weight, x t Represents the feature vector processed by task t, N e is the total number of routers; Gating weight The calculation formula is as follows: Among them, the Softmax(·) function is used for normalization operation, the Topk(·) function is used to select the top k most relevant low-rank adaptation adapters, and c t Represents x t The [CLS] tag in .
4. The method for adaptive fusion multimodal data set recognition based on a pre-training model according to claim 1, characterized in that: The specific method of the adaptive weighted fusion module in step 5 is: A shielding threshold th is preset in the adaptive weighted fusion module to shield the noise interference in the weak modality; by calculating the correlation r between the image feature and the audio feature , compare it with th; if r < th, only use Otherwise, and are weighted and stitched, and processed through the feature fusion layer; The formula for calculating the correlation is as follows: for The corresponding average value, i represents the i-th feature.
5. The method for adaptive fusion multimodal data set recognition based on a pre-training model according to claim 1, characterized in that: The total loss function for training steps 2 to 6 includes the continuous weighted cross-modal prototype contrast loss and the multimodal fusion mutual information loss function; Continuously Weighted Cross-Modal Prototype Contrastive Loss Function The formula is as follows: The sim(·,·) function is used to calculate the cosine similarity between two vectors. represents the cross entropy loss, and N represents the total number of features; Multimodal fusion mutual information loss function The goal is to maximize the fusion feature With image features and audio features The mutual information between them is as follows: The composition of the total loss function is: Among them, α∈[0,1] is a hyperparameter, and P(.) represents the probability distribution function.
6. The method for adaptive fusion multimodal data set recognition based on a pre-training model according to claim 1, characterized in that: The sample data were classified into the following 33 categories: analyzing, erasing the blackboard, writing on the blackboard, discussing, drinking water, eating, packing up schoolbags, raising hands, listening to lectures, listening to music, packing up computers, reading, relaxing, reviewing, scratching heads, setting up equipment, sleeping, speaking, standing, student demonstrations, taking schoolbags, taking bottles, taking off clothes, taking pictures, talking, teaching, unknown, using computers, using tablets, using mobile phones, walking, writing, and yawning.
Citation Information
Patent Citations
Recommendation tag generation method and device for multi-modal data model, equipment and medium
CN117290596A
Multi-modal joint learning method, device and equipment, readable storage medium and product
CN118261224A
Systems and methods for learning emotion representations from verbal and nonverbal communication
WO2024148018A2
Cited By
Continuous evolution learning method based on consistent hybrid expert prompt generator
CN119250233A
Metacosmic police service processing system based on multiple modes
CN119272943A
Intelligent construction site safety monitoring method and system fused with multi-modal large model
CN119399702A
Rail transit construction safety identification model dynamic adaptation method and system
CN120635839A
