Fine adjustment method of emotion recognition large model for multi-mode few-label sample

Abstract

The invention discloses an emotion recognition large model fine tuning method for a multi-mode few-label sample, and relates to the technical field of behavior detection. The invention provides an emotion recognition large model fine tuning method for a multi-modal few-label sample, and the method comprises the steps: collecting multi-modal image data and audio data, and carrying out the data enhancement of the multi-modal image data and audio data; extracting features of the image data and the audio data after data enhancement by using a pre-training model, and mapping the features to a hidden space to respectively obtain an image domain projection embedding representation and an audio domain projection embedding representation; constructing a total loss function based on the uncertainty of the feature consistency loss and the prediction consistency loss; and repeating the steps until the model converges. Therefore, by the adoption of the method, the multi-modal data is used for emotion recognition, and richer features can be obtained; the efficiency can be improved by adopting a method for finely adjusting the pre-training model; by training the dynamic adjustment weighting coefficient, the adaptability and robustness of the model can be improved, and the generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of behavior detection technology, and in particular to a method for fine-tuning a large emotion recognition model for multi-modal few-label samples. Background Art

[0002] Emotion recognition plays an increasingly important role in modern education. Through emotion recognition, educators can understand students' emotional state, adjust teaching strategies in a timely manner, improve learning experience, and enhance teaching effectiveness. At the same time, emotion recognition can also help identify learning barriers, provide personalized tutoring, and promote students' all-round development.

[0003] Emotion recognition usually relies on multimodal data, including images (such as facial expressions) and audio (such as voice intonation). Single-modal data often cannot fully capture the emotional state of students, while multimodal data can provide richer and more accurate information. For example, facial expressions can reflect students' immediate emotions, while voice intonation can reflect the depth and persistence of emotions. By fusing multimodal data, students' emotional states can be more accurately identified and analyzed.

[0004] Although multimodal data has obvious advantages in emotion recognition, there are many challenges in obtaining a large amount of labeled multimodal data. Labeling multimodal data requires a lot of manpower and time, and in real scenarios, labeled data is often very limited, far less than unlabeled data. This lack of labeling poses a severe challenge to the training and performance of the model.

[0005] Therefore, there is an urgent need for a method to fine-tune large emotion recognition models for multimodal, few-label samples to solve the problem of using unlabeled data for student emotion recognition in practical applications and provide a reference for the construction of intelligent education systems. Summary of the invention

[0006] The purpose of the present invention is to provide a large-scale emotion recognition model fine-tuning method for multimodal few-label samples, which can achieve effective knowledge transfer, detect student behavior using a small amount of unlabeled local samples, and solve the problem of insufficient labeled samples on the school side.

[0007] To achieve the above object, the present invention provides a method for fine-tuning a large model for emotion recognition of multimodal few-label samples, comprising the following steps:

[0008] S1, collect multimodal image data and audio data, and perform data enhancement respectively;

[0009] S2. Using the pre-trained model, extract the features of the image data and audio data after data enhancement, and map the features to the latent space to obtain the image domain projection embedding representation and the audio domain projection embedding representation respectively;

[0010] S3. Construct a total loss function based on the uncertainty of feature consistency loss and prediction consistency loss;

[0011] S4. Repeat steps S1 to S3 until the model converges.

[0012] Preferably, in step S1, data enhancement of the image includes flipping, rotating, cropping, scaling, translating, color change, noise injection, blurring, shearing, affine transformation, geometric transformation, random erasing, and contrast enhancement; data enhancement of the audio includes time stretching and compression, pitch change, adding background noise, random shearing and splicing, reverberation, spectrum transformation, time shift, dynamic range compression, increasing and decreasing volume, filtering, and adding noise.

[0013] Preferably, in step S2, an image encoder G is used I , audio encoder G A Feature extraction is performed on the image enhancement data and audio enhancement data respectively, and the image projector R I , Audio Projector R A The extracted image features and audio features are mapped to the latent space to obtain image representation and audio representation respectively.

[0014] Preferably, step S3 includes selecting a positive sample pair and a negative sample pair, and calculating the cosine similarity of the positive sample pair and the negative sample pair, respectively, to obtain the feature consistency loss, which is specifically as follows:

[0015]

[0016] in,

[0017]

[0018] Where, L IA,i is the feature consistency loss, is the similarity of the positive sample pair in the latent space, S neg is the similarity of negative sample pairs in the hidden space, i≠j means that the negative sample pairs come from different groups, are the first image representation and the second image representation respectively, are the first audio representation and the second audio representation respectively, and δ is a predefined constant.

[0019] Preferably, the method comprises inputting the image representation into the image domain classifier F I , obtain the first image prediction output and the second image prediction output, and input the audio representation into the audio domain classifier F A , the first audio prediction output and the second audio prediction output are obtained, thereby obtaining the prediction consistency loss, which is as follows:

[0020]

[0021] L pc =(L pci +L pca ) / 2

[0022] In the formula, N is the number of unlabeled samples, are the first image prediction output and the second image prediction output, respectively. The first audio prediction output and the second audio prediction output respectively.

[0023] Preferably, the total loss function is as follows:

[0024]

[0025] In the formula, α IA , α PC denote the uncertainty of feature consistency loss and prediction consistency loss respectively.

[0026] Therefore, the present invention adopts the above-mentioned large model fine-tuning method for emotion recognition for multimodal few-label samples, which has the following technical effects:

[0027] (1) By combining the two modalities of image and audio data, the multi-dimensional information of students' emotions is captured, making emotion recognition more comprehensive and accurate; by calculating the consistency loss of image and audio representations, it ensures that different modal data remain consistent in a unified latent space, thereby improving the accuracy of the model when processing multimodal data.

[0028] (2) Using a model pre-trained on a large-scale dataset as the basis, it has rich feature representations. At the same time, fine-tuning through a small amount of labeled data can fully utilize the knowledge of the pre-trained model and significantly improve the effect of emotion recognition. By using a large amount of unlabeled data, through data enhancement and consistency loss methods, the model can effectively learn on unlabeled data, further improving the performance and generalization ability of the model.

[0029] (3) The pre-trained model has been fully trained on a large-scale dataset, and only minor adjustments need to be made to the model during fine-tuning, which significantly reduces the training time. The use of pre-trained models and the uncertainty dynamic weighting method effectively reduces the demand for computing resources. In addition, the pre-trained model provides a good initial weight, making the fine-tuning process more efficient.

[0030] (4) According to the uncertainty of feature consistency loss and prediction consistency loss, the weighting coefficient is dynamically adjusted so that the model can balance the weights of different loss tasks and improve the adaptability and robustness of the model under different data distributions; by maintaining consistency under different enhanced views, the model has stronger robustness in the face of data noise and changes, thereby improving the generalization ability of the model.

[0031] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The present invention is a flowchart of an embodiment of a method for fine-tuning a large model of emotion recognition for multi-modal and few-label samples. DETAILED DESCRIPTION

[0033] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0034] like Figure 1 As shown, the present invention provides a method for fine-tuning a large emotion recognition model for multimodal few-label samples, comprising the following steps:

[0035] S1. Since single-modal data may not provide sufficient information, and the combination of multimodal data can more accurately reflect the students' true emotional state, multimodal image data and audio data are collected as the original data for feature extraction, and data enhancement is performed on the multimodal image data and audio data respectively, as follows:

[0036] Data enhancement of images includes flipping, rotating, cropping, scaling, translating, color changing, noise injection, blurring, shearing, affine transformation, geometric transformation, random erasing, and contrast enhancement;

[0037] Audio data enhancement includes time stretching and compression, pitch shifting, adding background noise, random cutting and splicing, reverberation, spectral transformation, time shifting, dynamic range compression, increasing and decreasing volume, filtering, and adding noise.

[0038] S2. Use the pre-trained model to extract features from the enhanced data of the two modalities respectively, and map the features to the latent space to obtain their respective representations.

[0039] Among them, pre-trained models refer to models that have been pre-trained on large-scale datasets. These models have learned general feature representations and can be used for various downstream tasks.

[0040] First, the image data Perform two data augmentations and Perform two data augmentations on the audio data and The two augmented data of the image and After the image encoder G I Get image features and The two enhanced audio data and After the image encoder G A Get image features and

[0041] Then, the features are mapped to the latent space to obtain the embedded representation of each feature, as follows:

[0042] The image features and Input to image projector R I , get the image domain projection embedding (image representation) and

[0043] The audio features and Input to audio projector R A Get audio domain projected embedding (audio representation) and

[0044] S3. Based on the uncertainty of feature consistency loss and prediction consistency loss, the total loss function is constructed as follows:

[0045] (1) Calculate the feature contrast loss. The specific steps are as follows:

[0046] Select positive sample pairs, that is, the same set of image representation and audio representation, and calculate the cosine similarity between them, aiming to make the positive samples close to each other in the hidden space. For negative sample pairs, select different sets of image enhancements and audio enhancements, and calculate the cosine similarity between these negative sample pairs to ensure that the negative samples are far away from each other in the hidden space. The loss function calculation formula is as follows:

[0047]

[0048] In the formula, Represents the similarity of positive sample pairs in the latent space, where the positive sample pairs are image representations and audio representations from the same group; i≠j represents the similarity between negative sample pairs, where i≠j means that the negative sample pairs come from different groups, that is, the i-th group samples and the j-th group samples come from different groups; δ is a predefined constant used to maintain the distance between negative sample pairs.

[0049] (2) Calculate the prediction consistency loss. The specific process is as follows:

[0050] Embedding image domain projection and Input into the image domain classifier respectively to obtain two image domain prediction outputs and For the prediction of labeled samples, the cross entropy loss function with the label is calculated. For the two predicted outputs of unlabeled samples, the prediction consistency loss between the two is calculated. The specific formula is as follows:

[0051]

[0052] Where N is the number of unlabeled samples.

[0053] Projecting the audio domain into an embedded representation and Input into the audio domain classifier respectively to obtain two audio domain prediction outputs and For the prediction of labeled samples, the cross entropy loss function with the label is calculated. For the two predicted outputs of unlabeled samples, the prediction consistency loss between the two is calculated. The specific formula is as follows:

[0054]

[0055] The calculation formula for prediction consistency loss is:

[0056] L pc =(L pci +L pca ) / 2

[0057] (3) Calculate the total loss function as follows:

[0058]

[0059] Among them, α IA and α PC are the standard deviations, representing the uncertainty of feature consistency loss and prediction consistency loss, respectively, and their values ​​are updated by back-propagation in iterations.

[0060] S4. Repeat steps S1 to S3 until the model converges.

[0061] The above method is used to identify students' emotions, and the specific applications are as follows:

[0062] Data collection and preparation: Install cameras and microphones in the classroom to capture students’ facial expressions and voice data in real time. Select a portion of the data for manual annotation, including the students’ emotional state (such as happiness, sadness, anger, etc.). Perform enhancement processing such as rotation, cropping, flipping, and color transformation on image data, and perform enhancement processing such as noise addition, pitch shifting, and time domain transformation on audio data. Generate two enhanced views for each modality to increase data diversity and model robustness.

[0063] Feature extraction and representation mapping: Use image models and audio models pre-trained on large-scale datasets, input the enhanced data into the pre-trained model for feature extraction, and map the features to the latent space to obtain representations of different modalities.

[0064] Compute image-audio consistency loss: Compute the consistency loss between image and audio representations to ensure the consistency of multimodal data in the latent space.

[0065] Calculate prediction consistency loss: Use the classifier to predict the image and audio representations, calculate the consistency loss between the prediction results, and ensure the consistency of the augmented data when predicting.

[0066] Uncertainty dynamic weighting: According to the uncertainty of feature consistency loss and prediction consistency loss, the weighting coefficient is dynamically adjusted to calculate the total loss function, taking the uncertainty into account to improve the overall performance of the model.

[0067] Model training and fine-tuning: Use the optimizer to update the model parameters and repeat the above steps until the model converges. Through multiple rounds of iterations, the model is continuously optimized to enable more accurate emotion recognition.

[0068] System deployment and application: Deploy the trained model to the computing devices in the classroom, process the data collected by the camera and microphone in real time, analyze the students' emotional state in real time through the model, and feedback the results to the teacher. Teachers can adjust their teaching strategies in a timely manner based on the emotion recognition results fed back by the system. For example, when the system recognizes that most students are confused or impatient, teachers can adjust their explanation methods or interact in a timely manner. The system can record the emotional change trends of each student and provide teachers with personalized teaching suggestions. For example, for students who have long shown negative emotions, the system can suggest that teachers provide one-on-one tutoring or psychological counseling.

[0069] Implementing multimodal large model fine-tuning technology for student emotion recognition in schools can bring the following benefits:

[0070] Improve teaching quality: Real-time identification of students' emotional state helps teachers adjust teaching strategies in a timely manner and improve classroom teaching effectiveness.

[0071] Personalized counseling: Through emotional recognition, we can identify students with emotional problems, provide personalized counseling, and promote their all-round development.

[0072] Enhance teacher-student interaction: The emotional information fed back by the system can enhance the interaction between teachers and students and create a positive classroom atmosphere.

[0073] Data-driven teaching: Use emotion recognition data to provide decision support for education administrators and promote data-driven teaching reforms.

[0074] Therefore, the present invention adopts the above-mentioned large model fine-tuning method for emotion recognition for multimodal few-label samples, which can improve the accuracy of emotion recognition and provide effective technical support for the construction of intelligent education systems.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for fine-tuning a large model for emotion recognition with multimodal and few-label samples, characterized in that: The following steps are involved: S1, collect multimodal image data and audio data, and perform data enhancement respectively; S2. Using the pre-trained model, extract the features of the image data and audio data after data enhancement, and map the features to the latent space to obtain the image domain projection embedding representation and the audio domain projection embedding representation respectively; S3. Construct a total loss function based on the uncertainty of feature consistency loss and prediction consistency loss; S4. Repeat steps S1 to S3 until the model converges.

2. According to claim 1, a large model fine-tuning method for emotion recognition for multimodal few-label samples is characterized in that: In step S1, data enhancement of the image includes flipping, rotating, cropping, scaling, translating, color change, noise injection, blurring, shearing, affine transformation, geometric transformation, random erasing, and contrast enhancement; data enhancement of the audio includes time stretching and compression, pitch change, adding background noise, random shearing and splicing, reverberation, spectrum transformation, time shift, dynamic range compression, increasing and decreasing volume, filtering, and adding noise.

3. The method for fine-tuning a large emotion recognition model for multimodal, few-label samples according to claim 1, characterized in that: In step S2, the image encoder G is used I , audio encoder G A Feature extraction is performed on the image enhancement data and audio enhancement data respectively, and the image projector R I , Audio Projector R A The extracted image features and audio features are mapped to the latent space to obtain image representation and audio representation respectively.

4. The method for fine-tuning a large emotion recognition model for multimodal, few-label samples according to claim 1, characterized in that: Step S3 includes selecting a positive sample pair and a negative sample pair, and calculating the cosine similarity of the positive sample pair and the negative sample pair respectively to obtain the feature consistency loss, which is as follows: in, Where, L IA,i is the feature consistency loss, is the similarity of the positive sample pair in the latent space, S neg is the similarity of negative sample pairs in the hidden space, i≠j means that the negative sample pairs come from different groups, are the first image representation and the second image representation respectively, are the first audio representation and the second audio representation respectively, and δ is a predefined constant.

5. The method for fine-tuning a large emotion recognition model for multimodal, few-label samples according to claim 3, characterized in that: This involves inputting the image representation into the image domain classifier F I , obtain the first image prediction output and the second image prediction output, and input the audio representation into the audio domain classifier F A , obtain the first audio prediction output and the second audio prediction output, thereby obtaining the prediction consistency loss, which is as follows: L pc =(L pci +L pca ) / 2 In the formula, N is the number of unlabeled samples, are the first image prediction output and the second image prediction output, respectively. The first audio prediction output and the second audio prediction output respectively.

6. The method for fine-tuning a large emotion recognition model for multimodal, few-label samples according to claim 1, characterized in that: The total loss function is as follows: In the formula, α IA , α PC denote the uncertainty of feature consistency loss and prediction consistency loss respectively.

Images