Student emotion recognition method based on multi-modal comparison pre-training model
Through multimodal comparison pre-trained model combining image and audio data, the problem of insufficient single-modal emotion recognition information and difficulty in fusion of cross-modal data is solved, and high-accurate emotion recognition is achieved, which is suitable for real-time emotion monitoring in the field of education , reducing the cost of data labeling.
Patent Information
- Application Number
- CN202410949481.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing emotion recognition technologies rely on single modal data, making it difficult to fully capture students' emotional states. Due to the difficulty of fusion of cross-modal data and insufficient labeling data, emotional recognition accuracy is not high.
Using a pre-trained model based on multimodal contrast, through the combination of image and audio data, the multimodal contrast learning method is used to perform pre-training without labeled data, and the image domain, audio domain and image-audio consistency losses are calculated to generate The total loss function is trained to reduce dependence on the annotated data, and fine-tune it with pre-trained models.
It significantly improves the accuracy and robustness of emotion recognition, reduces the cost and time of data annotation, is suitable for real-time emotion monitoring in the field of education, and improves teaching effectiveness and resource allocation efficiency.
Smart Images

Figure CN118965253A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning, and in particular to a student emotion recognition method based on a multimodal contrast pre-training model. Background Art
[0002] In the field of education, students’ emotional state has an important impact on learning outcomes and classroom atmosphere. Being able to monitor and analyze students’ emotional state in real time can help teachers adjust teaching strategies, improve the quality of classroom interaction, and intervene in possible problems in a timely manner, thereby improving the overall teaching effect and students’ learning experience.
[0003] Most existing emotion recognition technologies rely on single-modal data, such as sentiment analysis based only on images (facial expressions) or audio (voice features). This approach often fails to fully capture students' emotional states, because emotional expressions are usually multimodal, including facial expressions, voice intonation, body movements, etc.
[0004] Image and audio data have different spatial and temporal characteristics. How to effectively integrate the information of these two modalities in the same model is a major challenge. Existing methods often fail to fully utilize the complementarity of cross-modal information, resulting in low accuracy of emotion recognition.
[0005] Emotion recognition models usually require a large amount of labeled data for training, but the labeling process of emotion data is complex, time-consuming and labor-intensive. Especially in educational scenarios, it is even more difficult to obtain high-quality emotion labeled data, which limits the promotion of emotion recognition models in practical applications.
[0006] Therefore, a student emotion recognition method based on a multimodal contrastive pre-training model is proposed to solve the above problems. This method combines image and audio data and uses a multimodal contrastive learning pre-training method to achieve effective pre-training without labeled data, solving the limitations of single-modal emotion recognition, the difficulty of cross-modal data fusion, and the problem of insufficient labeled data, thereby significantly improving the accuracy and robustness of emotion recognition, and has broad application prospects. Summary of the invention
[0007] The purpose of the present invention is to provide a student emotion recognition method based on a multimodal contrast pre-training model. By fusing image and audio data and utilizing a multimodal contrast learning pre-training method, effective pre-training can be achieved without labeled data, thereby solving the problems of insufficient single-modal emotion recognition information, difficulty in cross-modal data fusion, and dependence on labeled data, thereby significantly improving the accuracy of emotion recognition.
[0008] To achieve the above object, the present invention provides a student emotion recognition method based on a multimodal contrast pre-training model, comprising the following steps:
[0009] Step S1, performing data enhancement on multimodal data, where the multimodal data includes images and audio;
[0010] Step S2: Use an image encoder and an audio encoder to extract features from the original data and enhanced data of the image and audio respectively, and then map the features to a latent space to obtain respective embedded representations;
[0011] Step S3, calculating image domain contrast loss;
[0012] Step S4, calculating the audio domain contrast loss;
[0013] Step S5, calculating the image-audio consistency loss;
[0014] Step S6, calculating the weighted sum of the three losses in step S3, step S4, and step S5 to obtain a total loss function, and performing training based on the total loss function to obtain a pre-trained model;
[0015] Step S7: Use the pre-trained model and a small amount of labeled data for fine-tuning to ultimately achieve emotion recognition of students.
[0016] Preferably, the step S1 of enhancing the unlabeled input data includes:
[0017] (1) Data enhancement of images includes flipping, rotating, cropping, scaling, translating, color change, noise injection, blurring, shearing, affine transformation, geometric transformation, random erasing, and contrast enhancement;
[0018] (2) Audio data enhancement includes time stretching and compression, pitch change, adding background noise, random cutting and splicing, reverberation, spectrum transformation, time shift, dynamic range compression, increasing and decreasing volume, filtering, and adding noise.
[0019] Preferably, in step S2, an image encoder and an audio encoder are used to extract features from the original data and enhanced data of the image and audio respectively, and the specific steps are as follows:
[0020] For image data and audio data The sample after data augmentation is and Image data and image augmentation data After the image encoder G I Get image features and Audio Data and audio enhancement data After the audio encoder G A Get audio features and
[0021] Preferably, in step S2, the features are mapped to a latent space to obtain respective embedded representations, and the specific steps are as follows:
[0022] The image features and Input to image projector R I Get image domain projection embedding and
[0023] The audio features and Input to audio projector R A Get audio domain projected embedding and
[0024] Preferably, the image domain contrast loss is calculated in step S3, and the specific steps are as follows:
[0025] Calculate the cosine similarity between positive sample pairs so that the positive sample pairs are close to each other in the latent space and the negative sample pairs are far away from each other in the latent space; where the positive sample pair is the embedding representation of the same original image sample and its enhanced sample, and the negative sample pair is the embedding representation of the current image sample and other different image samples;
[0026] The calculation formula is as follows:
[0027]
[0028] Among them, τ is the temperature parameter used to adjust the distribution of similarity values; I i≠j is an indicator function, which takes the value 1 when i=j and 0 otherwise. pred is a collection of unlabeled image data.
[0029] Preferably, the audio domain contrast loss is calculated in step S4, and the specific steps are as follows:
[0030] Calculate the cosine similarity between positive sample pairs so that the positive sample pairs are close to each other in the latent space and the negative sample pairs are far away from each other in the latent space; wherein the positive sample pair is the embedding representation of the same original audio sample and its enhanced sample, and the negative sample pair is the embedding representation of the current audio sample and other different audio samples;
[0031] The calculation formula is as follows:
[0032]
[0033] Among them, τ is the temperature parameter used to adjust the distribution of similarity values; I i≠jis an indicator function, which takes the value 1 when i=j and 0 otherwise; D pred is a collection of unlabeled image data.
[0034] Preferably, the image-audio consistency loss is calculated in step S5, and the specific steps are as follows:
[0035] Select positive sample pairs for the same set of original image representations and audio representations, and calculate the cosine similarity between the positive sample pairs so that the positive samples are close to each other in the hidden space; select negative sample pairs as the representation of the current image sample and other different audio samples, or the representation of the current audio sample and other different image samples, as well as the image enhancement representation and the audio enhancement representation; calculate the cosine similarity between the negative sample pairs so that the negative samples are far away from each other in the hidden space;
[0036] The loss function calculation formula is as follows:
[0037]
[0038] in, represents the similarity of the same set of original image representations and audio representations in the latent space, represents the similarity between negative sample pairs; δ is a predefined constant used to maintain the distance between negative sample pairs.
[0039] Preferably, the total loss function in step S6 is calculated as follows:
[0040] L total =α(L I,i +L A,i )+(1-α)L IA,i
[0041] Among them, α is the weight coefficient.
[0042] Therefore, the present invention adopts the above-mentioned student emotion recognition method based on the multimodal contrast pre-training model, which has the following beneficial effects:
[0043] (1) The present invention reduces the reliance on a large amount of labeled data by automatically learning features, solving the problem of time-consuming, costly, and inconsistent data annotation in traditional methods. By using data enhancement technology, multiple enhanced views are generated, and feature extraction is performed on the time windows of these views. The model can capture more useful information from unlabeled data.
[0044] (2) The present invention introduces temporal consistency loss and contrast loss to ensure that the model can not only maintain the continuity of features of adjacent time steps, but also enhance the feature differences between different time windows. This design enables the model to better capture the temporal dynamic characteristics of video data, thereby improving the accuracy and robustness of behavior detection.
[0045] (3) The present invention is highly practical. Student behavior detection in the field of education requires processing a large amount of unlabeled video data. The self-supervised learning method significantly reduces the labeling cost and time. Real-time monitoring of students' attention, participation, and emotional state enables teachers to adjust teaching strategies in a timely manner, provide personalized support, and improve teaching effectiveness. In addition, this method can also help educational institutions optimize resource allocation and improve teaching quality.
[0046] (4) The present invention not only solves the bottleneck problem of data labeling, but also improves the practicality and accuracy of the model through efficient feature extraction and behavior detection, and is expected to play an important role in future educational practice.
[0047] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 The present invention is a flowchart of an embodiment of a method for student emotion recognition based on a multimodal contrast pre-training model. DETAILED DESCRIPTION
[0049] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.
[0050] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0051] Example
[0052] See also Figure 1 The present invention provides a student emotion recognition method based on a multimodal contrast pre-training model, comprising the following steps:
[0053] Step S1: perform data enhancement on multimodal data, where the multimodal data includes images and audio.
[0054] The unlabeled input data enhancement in the above step S1 includes:
[0055] (1) Data augmentation of images includes flipping, rotation, cropping, scaling, translation, color change, noise injection, blurring, shearing, affine transformation, geometric transformation, random erasing, contrast enhancement, etc. These methods can be used alone or in combination to increase the diversity of data.
[0056] (2) Audio data enhancement includes time stretching and compression, pitch change, adding background noise, random cutting and splicing, reverberation, spectrum transformation, time shift, dynamic range compression, increasing and decreasing volume, filtering, adding noise, etc. These methods can be used alone or in combination to increase the diversity of the data.
[0057] Step S2: Use an image encoder and an audio encoder to extract features from the original data and enhanced data of the image and audio respectively, and then map the features into a latent space to obtain respective embedded representations.
[0058] In the above step S2, the image encoder and the audio encoder are used to extract features from the original data and enhanced data of the image and audio respectively. The specific steps are as follows:
[0059] For image data and audio data The sample after data augmentation is and Image data and image augmentation data After the image encoder G I Get image features and Audio Data and audio enhancement data After the audio encoder G A Get audio features and
[0060] In the above step S2, the features are mapped to the latent space to obtain their respective embedded representations. The specific steps are as follows:
[0061] The image features and Input to image projector R I Get image domain projection embedding and
[0062] The audio features and Input to audio projector R A Get audio domain projected embedding and
[0063] Step S3: Calculate image domain contrast loss.
[0064] In the above step S3, the image domain contrast loss is calculated, and the specific steps are as follows:
[0065] Calculate the cosine similarity between positive sample pairs so that the positive sample pairs are close to each other in the latent space and the negative sample pairs are far away from each other in the latent space; where the positive sample pair is the embedding representation of the same original image sample and its enhanced sample, and the negative sample pair is the embedding representation of the current image sample and other different image samples;
[0066] The calculation formula is as follows:
[0067]
[0068] Among them, τ is the temperature parameter used to adjust the distribution of similarity values; I i≠j is an indicator function, which takes the value 1 when i=j and 0 otherwise. pred is a collection of unlabeled image data.
[0069] Step S4: Calculate the audio domain contrast loss.
[0070] In the above step S4, the audio domain contrast loss is calculated, and the specific steps are as follows:
[0071] Calculate the cosine similarity between positive sample pairs so that the positive sample pairs are close to each other in the latent space and the negative sample pairs are far away from each other in the latent space; wherein the positive sample pair is the embedding representation of the same original audio sample and its enhanced sample, and the negative sample pair is the embedding representation of the current audio sample and other different audio samples;
[0072] The calculation formula is as follows:
[0073]
[0074] Among them, τ is the temperature parameter used to adjust the distribution of similarity values; I i≠j is an indicator function, which takes the value 1 when i=j and 0 otherwise; D pred is a collection of unlabeled image data.
[0075] Step S5: Calculate the image-audio consistency loss.
[0076] In the above step S5, the image-audio consistency loss is calculated, and the specific steps are as follows:
[0077] Select positive sample pairs for the same set of original image representations and audio representations, and calculate the cosine similarity between the positive sample pairs so that the positive samples are close to each other in the hidden space; select negative sample pairs as the representation of the current image sample and other different audio samples, or the representation of the current audio sample and other different image samples, as well as the image enhancement representation and the audio enhancement representation; calculate the cosine similarity between the negative sample pairs so that the negative samples are far away from each other in the hidden space;
[0078] The loss function calculation formula is as follows:
[0079]
[0080] in, represents the similarity of the same set of original image representations and audio representations in the latent space, represents the similarity between negative sample pairs; δ is a predefined constant used to maintain the distance between negative sample pairs.
[0081] Step S6, calculate the weighted sum of the three losses in step S3, step S4, and step S5 to obtain a total loss function, perform training based on the total loss function and obtain a pre-trained model.
[0082] The total loss function in the above step S6 is calculated as follows:
[0083] L total =α(L I,i +L A,i )+(1-α)L IA,i
[0084] Among them, α is the weight coefficient.
[0085] Step S7: Use the pre-trained model and a small amount of labeled data for fine-tuning to ultimately achieve emotion recognition of students.
[0086] This embodiment provides a student emotion recognition method based on a multimodal contrast pre-training model, and the specific application steps are as follows:
[0087] (1) Each classroom is equipped with a high-definition camera and a high-sensitivity microphone to capture students’ facial expressions and voice features, respectively. The camera is installed at the front of the classroom and can clearly capture the facial expressions of all students. The microphones are placed at different locations in the classroom to ensure that the students’ voices can be clearly recorded.
[0088] (2) The classroom emotion monitoring system collects students’ facial images and voice data in real time and performs preliminary preprocessing, such as denoising and image enhancement. The collected data is fed into the multimodal emotion analysis module of the classroom emotion monitoring system.
[0089] (3) Using a large amount of unlabeled image and audio data, the classroom emotion monitoring system learns the common feature representation of images and audio through a multimodal contrastive learning pre-training model. The multimodal contrastive learning pre-training method is used to ensure that the image and audio representations of the same period are close to each other in the latent space.
[0090] (4) In the classroom, the classroom emotion monitoring system processes and analyzes the collected image and audio data in real time. The classroom emotion monitoring system extracts the feature representation of images and audio through the pre-trained model and performs emotion recognition. The classroom emotion monitoring system displays the students' emotional state (such as happiness, confusion, boredom, etc.) to the teacher in the form of charts or real-time feedback.
[0091] (5) Teachers can understand students' emotional state in real time through the emotion analysis results provided by the classroom emotion monitoring system. Based on the feedback from the classroom emotion monitoring system, teachers can adjust their teaching strategies, such as adding interactive sessions, changing the teaching rhythm, or paying more attention to students who are in low spirits. Teachers can understand students' emotional state in real time and adjust their teaching methods in a timely manner to improve the quality of classroom interaction and student participation. The classroom emotion monitoring system can also record and analyze historical emotion data to provide teachers with long-term emotion change trends to help optimize teaching plans.
[0092] Therefore, the present invention adopts the above-mentioned student emotion recognition method based on the multimodal contrast pre-training model, which reduces the cost and difficulty of data annotation and makes the system easier to promote and apply. This intelligent emotion monitoring system has demonstrated its strong application prospects in the field of education, which can not only improve classroom teaching effectiveness, but also provide important data support for educational research.
[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A student emotion recognition method based on a multimodal contrast pre-training model, characterized in that: The following steps are involved: Step S1, performing data enhancement on multimodal data, where the multimodal data includes images and audio; Step S2: Use an image encoder and an audio encoder to extract features from the original data and enhanced data of the image and audio respectively, and then map the features to a latent space to obtain respective embedded representations; Step S3, calculating image domain contrast loss; Step S4, calculating audio domain contrast loss; Step S5, calculating the image-audio consistency loss; Step S6, calculating the weighted sum of the three losses in step S3, step S4, and step S5 to obtain a total loss function, and performing training based on the total loss function to obtain a pre-trained model; Step S7: Use the pre-trained model and a small amount of labeled data for fine-tuning to ultimately achieve emotion recognition of students.
2. According to claim 1, a student emotion recognition method based on a multimodal contrast pre-training model is characterized in that: The step S1 of enhancing the unlabeled input data includes: (1) Data enhancement of images includes flipping, rotating, cropping, scaling, translating, color change, noise injection, blurring, shearing, affine transformation, geometric transformation, random erasing, and contrast enhancement; (2) Audio data enhancement includes time stretching and compression, pitch change, adding background noise, random cutting and splicing, reverberation, spectrum transformation, time shift, dynamic range compression, increasing and decreasing volume, filtering, and adding noise.
3. The method for student emotion recognition based on a multimodal contrast pre-training model according to claim 2 is characterized in that: In step S2, an image encoder and an audio encoder are used to extract features from the original data and enhanced data of the image and audio respectively. The specific steps are as follows: For image data and audio data The sample after data augmentation is and Image data and image augmentation data After the image encoder G I Get image features and Audio Data and audio enhancement data After the audio encoder G A Get audio features and 4. The method for student emotion recognition based on a multimodal contrast pre-training model according to claim 3 is characterized in that: In step S2, the features are mapped to the latent space to obtain respective embedded representations. The specific steps are as follows: The image features and Input to image projector R I Get image domain projection embedding and The audio features and Input to audio projector R A Get audio domain projected embedding and 5. The method for student emotion recognition based on a multimodal contrast pre-training model according to claim 4 is characterized in that: In step S3, the image domain contrast loss is calculated, and the specific steps are as follows: Calculate the cosine similarity between positive sample pairs so that the positive sample pairs are close to each other in the latent space and the negative sample pairs are far away from each other in the latent space; where the positive sample pair is the embedding representation of the same original image sample and its enhanced sample, and the negative sample pair is the embedding representation of the current image sample and other different image samples; The calculation formula is as follows: Among them, τ is the temperature parameter used to adjust the distribution of similarity values; I i≠j is an indicator function, which takes the value 1 when i=j and 0 otherwise. pred is a collection of unlabeled image data.
6. The method for student emotion recognition based on a multimodal contrast pre-training model according to claim 5, characterized in that: In step S4, the audio domain contrast loss is calculated, and the specific steps are as follows: Calculate the cosine similarity between positive sample pairs so that the positive sample pairs are close to each other in the latent space and the negative sample pairs are far away from each other in the latent space; wherein the positive sample pair is the embedding representation of the same original audio sample and its enhanced sample, and the negative sample pair is the embedding representation of the current audio sample and other different audio samples; The calculation formula is as follows: Among them, τ is the temperature parameter used to adjust the distribution of similarity values; I i≠j is an indicator function, which takes the value 1 when i=j and 0 otherwise; D pred is a collection of unlabeled image data.
7. The method for student emotion recognition based on a multimodal contrast pre-training model according to claim 5, characterized in that: In step S5, the image-audio consistency loss is calculated, and the specific steps are as follows: Select positive sample pairs for the same set of original image representations and audio representations, and calculate the cosine similarity between the positive sample pairs so that the positive samples are close to each other in the hidden space; select negative sample pairs as the representation of the current image sample and other different audio samples, or the representation of the current audio sample and other different image samples, as well as the image enhancement representation and the audio enhancement representation; calculate the cosine similarity between the negative sample pairs so that the negative samples are far away from each other in the hidden space; The loss function calculation formula is as follows: in, represents the similarity of the same set of original image representations and audio representations in the latent space, represents the similarity between negative sample pairs; δ is a predefined constant used to maintain the distance between negative sample pairs.
8. The method for student emotion recognition based on a multimodal contrast pre-training model according to claim 5, characterized in that: The total loss function in step S6 is calculated as follows: L total =α(L I,i +L A,i )+(1-α)L IA,i Among them, α is the weight coefficient.
Citation Information
Cited By
Electroencephalogram and synchronous physiological signal emotion recognition method based on cross-modal comparative learning and multi-scale representation
CN119848608A
A method for emotion recognition based on EEG and synchronized physiological signals based on cross-modal contrastive learning and multi-scale representation
CN119848608B
Cross-modal learning preference optimization enhanced three-dimensional face generation method and system
CN120510260A
