Multi-modal classroom emotion recognition method and system based on modal adaptive learning

Abstract

The invention belongs to but not limited to the technical field of emotion recognition, and discloses a multi-modal classroom emotion recognition method and system based on modal adaptive learning, and the method comprises the steps: building a multi-modal data collection system, collecting the multi-modal data, such as texts, audios and videos, in a classroom teaching process in real time, and carrying out the real-time collection of the multi-modal data; and preprocessing and feature extraction are carried out on data of different modes. For emotion expression feature differences of different classroom scenes, a multi-modal data fusion method based on adaptive modal scoring (AMS) is provided, and a time sequence modeling and emotion classification model is constructed. The multi-modal fusion and time sequence modeling mechanism simulates the multi-dimensional perception of the teacher on the emotional states of the students in the class, and enhances the interpretability of the emotion recognition system. According to the system, weighted fusion is carried out on data features from texts, audios and videos through fusion of multi-modal data, and complete emotion feature representation is formed.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of emotion recognition technology, and in particular relates to a multimodal classroom emotion recognition method and system based on modal adaptive learning. Background Art

[0002] With the continuous development of educational technology, classroom emotional computing has gradually become an important part of the intelligent education system. The emotional state of students in the classroom directly affects their learning motivation, concentration and overall academic performance. Studies have shown that positive emotions (such as interest, happiness, etc.) can improve students' cognitive flexibility, learning motivation and classroom participation, thereby promoting learning outcomes; while negative emotions (such as anxiety, depression, etc.) inhibit students' cognitive functions, reduce information processing efficiency, and even lead to avoidance of learning tasks. Therefore, accurately identifying students' emotional states in classroom situations can not only help teachers optimize teaching strategies, but also improve the personalization and pertinence of teaching to better meet students' learning needs.

[0003] Most existing emotion recognition technologies rely on single-modality (such as facial expressions or voice signals) data for emotion analysis. Although certain effects have been achieved in some situations, single-modality recognition methods usually lack robustness and have low recognition accuracy due to the diversity and restraint of emotional expression in classroom environments. To this end, researchers have gradually turned to multimodal emotion recognition methods, which capture students' emotional characteristics from multiple angles by fusing data from multiple modalities (such as text, voice, and video), thereby improving the accuracy and adaptability of emotion recognition. However, existing multimodal emotion recognition methods generally adopt a static modality fusion strategy, that is, a fixed combination of modal weights is used to process data in different situations. This static fusion method is difficult to flexibly respond to changes in classroom scenes, resulting in the system being difficult to capture the dynamic changes of emotional characteristics in a classroom environment where emotional expression is complex and changeable, and both recognition accuracy and adaptability are limited.

[0004] To address this problem, adaptive modality selection technology has gradually attracted attention. This technology enables emotion recognition systems to achieve higher flexibility and accuracy in different situations by dynamically adjusting the weights of different modalities. For example, in a quiet conversation scene, the voice modality can convey rich emotional information through intonation and rhythm, so the voice modality has a higher weight; while in a noisy classroom scene, the visual modality (such as facial expressions and body language) is more critical. However, most of the existing adaptive modality selection methods ignore the changes in classroom situations, and cannot effectively identify and adapt to the differences in emotional characteristics in different classroom scenes, resulting in limited recognition effects.

[0005] Technical problems existing in existing technologies

[0006] Based on the above analysis, the existing multimodal emotion recognition methods have the following technical problems that need to be solved urgently:

[0007] Limitations of static modal fusion: Most existing multimodal emotion recognition methods adopt static fusion strategies, which cannot flexibly adjust the weights of different modalities according to the dynamic changes of classroom scenes, resulting in a decrease in the recognition accuracy of the system when the situation changes.

[0008] Lack of contextual awareness: Existing methods lack scene perception mechanisms, making it difficult to effectively identify and adapt to differences in emotional characteristics in different classroom scenarios, resulting in insufficient robustness of emotion recognition in complex classroom environments.

[0009] In view of the above analysis, the technical problems that urgently need to be solved in the existing technology are: most of the existing multimodal emotion recognition methods adopt static fusion strategies, which are unable to flexibly adjust the weights of different modalities according to changes in classroom scenes, resulting in a decrease in recognition accuracy in complex classroom situations.

[0010] Therefore, there is an urgent need for an adaptive modality selection mechanism based on scene perception, which can dynamically adjust the modality combination weights according to the real-time changes in the classroom context, so as to accurately capture the emotional state of students in different scenarios, thereby improving the accuracy and robustness of the emotion recognition system. Summary of the invention

[0011] In response to the problems existing in the prior art, the present invention provides a multimodal classroom emotion recognition method and system based on modal adaptive learning, which effectively improves the accuracy and adaptability of emotion recognition through modal combination optimization and adaptive modal scoring (AMS) mechanism in different scenarios, and is particularly suitable for complex and dynamic classroom environments.

[0012] The present invention is implemented as follows: a multimodal classroom emotion recognition method based on modality adaptive learning, comprising:

[0013] S1. Multimodal data acquisition and preprocessing;

[0014] S2. Feature extraction;

[0015] S3. Modal combination optimization in different scenarios;

[0016] S4. Adaptive modality scoring mechanism (AMS);

[0017] S5. Multimodal fusion and time series modeling.

[0018] Furthermore, S1 specifically includes: using a multimodal data acquisition system to collect students' multimodal data such as text, audio and video in real time during classroom teaching. These data cover a variety of emotional clues such as students' language, voice, facial expressions and body movements in the classroom, and can comprehensively reflect students' emotional expressions in different classroom situations. In order to ensure the consistency and quality of each modal data, the present invention pre-processes the collected data, including data cleaning, format standardization and time synchronization processing, eliminating noise and redundant information, thereby providing high-quality input for subsequent feature extraction.

[0019] Furthermore, S2 specifically includes: for data of different modalities, advanced deep learning models are used for feature extraction. The semantic and emotional features of text data are extracted through the BERT model, the emotional features of speech signals are extracted through MFCC for audio data, and the emotional clues of facial expressions and body postures are extracted through 3D convolutional neural networks (3D-CNN) for video data. The extraction of these features provides a basis for subsequent emotion recognition.

[0020] Furthermore, S3 specifically includes: for different scenarios in classroom teaching (such as lectures, discussions, questions and answers, etc.), real-time identification of the contextual features of the current scenario, and dynamic adjustment of the combined weights of each modality to adapt to the differences in students' emotional expressions in different scenarios. Specifically, students' emotional expressions will be different in different classroom scenarios: in the "lecture" scenario, the emotional information of visual modalities such as facial expressions and postures is more prominent; while in the "discussion" or "question and answer" scenarios, voice features can better reflect the students' emotional state.

[0021] The scene perception mechanism is used to identify the current classroom situation and adjust the weight of each mode according to the characteristics of the situation. The specific weight distribution is calculated by the following formula:

[0022]

[0023] in, represents the weight of mode m at time step t.

[0024] It represents the initial weight of mode m at time step t. The initial weight is preset according to the scene characteristics (such as lectures, discussions, etc.).

[0025] s m is the importance coefficient of modality m in the current scene, indicating the relative weights of different modalities in different scenes. In the “teaching” scene, the visual modality s m The value is higher, while in the “discussion” scenario, the s m Higher value.

[0026] Through the weight distribution formula, the weight of each modality in different situations is dynamically adjusted to ensure that the most effective emotional information is captured in different classroom scenes. The effect of this formula is reflected in different situations as follows: when the scene perception mechanism detects the "teaching" scene, the system weights the visual modality (such as facial expression, posture) Increase; when the scene is converted to "discussion" or "question and answer", the weight is dynamically adjusted to enhance the influence of speech modality in emotion recognition.

[0027] In addition, to ensure the real-time performance of scene recognition, the scene perception mechanism uses the following scene judgment formula to achieve rapid recognition:

[0028]

[0029] Among them, c t is the classroom scene category at the current time step t, and its value is "lecture", "discussion" or "question and answer".

[0030] C represents the scene category set.

[0031] f(c,x t ) is the scene classification function, according to the input emotional feature x t Classification is performed to ensure efficient and accurate scene judgment.

[0032] Furthermore, S4 specifically includes: In order to further optimize the fusion of multimodal data, an adaptive modality scoring mechanism (AMS) is adopted. This mechanism dynamically adjusts the weight combination of each modality through real-time feedback (such as the accuracy of emotion recognition and system response time). The update formula of the AMS mechanism is: AMS(s t ,a t )←AMS(s t ,a t )+α[R(s t ,a t )+γ·maxAMS(s t+1 ,a t )-AMS(s t ,a t )]

[0033] This formula is used to calculate the t Update modal combination a t The meaning of each symbol in the formula is as follows:

[0034] AMS(s t ,a t ): means in state s t Select action a in (current classroom scene) tThe adaptive score of (modal combination). This score reflects the performance of the modal combination in the current scenario. The higher the score, the more suitable the modal combination is for the current scenario.

[0035] α: Learning rate, controls the step size of AMS value update. Usually set to 0≤α≤1. The larger the value, the more sensitive the system is to new feedback and the larger the update step size. When the value is smaller, the system update is more stable and the response is slower.

[0036] R(s t ,a t ): Reward function, used to evaluate the t Next select modal combination a t The reward function is defined based on the accuracy of emotion recognition and the system response time, and the formula is as follows:

[0037] γ: Discount factor, which indicates the importance the system places on future scenarios, and is usually set to 0≤γ≤1. The larger the discount factor, the more the system focuses on long-term benefits; the smaller the discount factor, the more the system focuses on current benefits.

[0038] maxAMS(s t+1 ,a t ): indicates the state s at the next moment t+1 The maximum AMS value obtained by selecting any modal combination a under . This reflects the system's prediction of the optimal modal combination in future scenarios and guides the system to make a better choice in the current scenario.

[0039] R(s t ,a t )=ω1·Accuracy(s t ,a t )+ω2·ResponseTime(s t ,a t )

[0040] Accurracy(s t ,a t ): This modal combination is in scene s t The higher the accuracy, the greater the reward.

[0041] ResponseTime(s t ,a t ): The response time of emotion recognition. The shorter the response time, the greater the reward. If the response time is too long, it can be set to a negative value.

[0042] ω1 and ω2 are weight coefficients used to balance the effects of accuracy and response time on the reward.

[0043] Furthermore, S5 specifically includes: using the self-attention mechanism to perform weighted fusion on the extracted multimodal features to ensure that the features of different modalities can be effectively combined to enhance the accuracy and robustness of emotion recognition. The fused features are modeled in time series through the gated recurrent unit (GRU) to capture the temporal dynamic changes of the emotional state, and finally the students' emotional state is classified as positive, neutral or negative through the Softmax classifier.

[0044] Another object of the present invention is to provide a multimodal emotion recognition system based on modal adaptive learning for implementing the multimodal classroom emotion recognition method based on modal adaptive learning, comprising:

[0045] Data preprocessing module, used for multimodal data acquisition and preprocessing;

[0046] A feature extraction module, used for feature extraction;

[0047] Modal combination optimization module, used to perform modal combination optimization in different scenarios;

[0048] A modal scoring module, used for performing adaptive modal scoring;

[0049] The multimodal fusion and time series modeling module is used for multimodal fusion and time series modeling.

[0050] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multimodal emotion recognition method based on modal adaptive learning.

[0051] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the multimodal emotion recognition method based on modal adaptive learning.

[0052] Another object of the present invention is to provide an information data processing terminal, which includes the multimodal emotion recognition system based on modal adaptive learning.

[0053] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0054] First, the present invention discloses a multimodal classroom emotion recognition method based on scene perception and modal adaptive learning. This method overcomes the limitations of existing mainstream models in view of the diversity and dynamism of classroom situations. Existing state-of-the-art models such as CORECT and COGMEN have deficiencies in adaptability and dynamic adjustment in classroom emotion recognition. Specifically, CORECT is based on Relational Temporal Graph Neural Networks (RT-GCN) and cross-modal interaction mechanisms, and is able to model global and local contextual information in multimodal data, which is suitable for emotion recognition in dialogue situations. However, in a classroom environment with diverse emotional expressions and frequent scene changes, CORECT's static modal fusion strategy is difficult to adapt to situational changes in real time, resulting in a decrease in recognition accuracy. On the other hand, COGMEN draws on human cognitive mechanisms to capture the details of complex emotional expressions through mechanisms such as attention transfer and cognitive load, but is also limited by static fusion strategies and is difficult to cope with frequent switching of classroom situations.

[0055] In response to these problems, the present invention proposes a scene perception mechanism and an adaptive modality scoring (AMS) mechanism, which enables the system to identify the current classroom scene (such as "lecture", "discussion" or "question and answer") in real time and dynamically adjust the modality combination. For example, in the "lecture" scene, the visual modality (such as facial expressions and postures) has a higher weight, while in the "discussion" or "question and answer" scene, the weight of the voice modality is increased to ensure that the system captures the most critical emotional information in different situations. Experimental data show that on the IEMOCAP dataset, SMAL has an accuracy of 84.13% and an F1 score of 84.17%, which is significantly better than CORECT (accuracy 83.03%, F1 score 83%) and COGMEN (accuracy 82.82%, F1 score 82.76%), and shows higher robustness and adaptability in complex emotional expression scenes.

[0056] Adaptive Modal Scoring (AMS) mechanism ensures the real-time and dynamic adjustment of emotion recognition. The AMS mechanism of the present invention dynamically adjusts the weight of each modality through real-time feedback (such as recognition accuracy and response time), so that the system can flexibly adjust the modal combination weight according to situation changes to ensure the best recognition effect. Unlike static fusion models such as CORECT and COGMEN, SMAL can respond to situation changes in real time through the AMS mechanism, significantly improving the recognition accuracy in classroom situation switching. On the CMU-MOSEI dataset, SMAL has an accuracy of 89.2% and an F1 score of 88.6%, far exceeding ATT (accuracy of 74.8%, F1 score of 72.7%) and CORECT (accuracy of 88.7%, F1 score of 87.66%), indicating that the AMS mechanism can effectively balance recognition accuracy and response speed in changing classroom situations.

[0057] The advantages of multimodal emotion recognition in classroom scenarios can effectively improve classroom teaching effects. The SMAL model of the present invention can accurately identify the emotional state in classroom situations through the combination of scene perception and AMS mechanism, and provide real-time and accurate emotional data support for classroom teaching. By capturing the emotional state of students, teachers can better understand the learning status of students, thereby optimizing teaching strategies and enhancing classroom interaction effects. On the TEDC data set, SMAL has an accuracy rate of 93.65% and an F1 score of 92.85%, which is significantly better than CORECT (accuracy rate of 89.81%, F1 score of 89.95%). It has higher practical value in classroom application scenarios and provides effective support for improving classroom teaching effects.

[0058] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0059] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0060] The multimodal emotion recognition method based on scene perception and modal adaptive learning proposed in the present invention provides key technical support for emotion monitoring, classroom intervention, personalized learning, etc. in intelligent education systems. Through this method, the system can perceive and analyze students' emotional changes in the classroom in real time, and provide teachers with teaching adjustment suggestions, thereby improving classroom interaction effects and students' learning experience. The present invention effectively solves the high cost problem in artificial emotion monitoring, greatly improves the accuracy and real-time performance of emotion recognition, and provides a solid technical foundation for intelligent education. The emotion recognition function of this system can not only be used for real-time classroom monitoring, but can also be widely used in intelligent learning recommendations, personalized learning services and other fields.

[0061] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0062] The multimodal emotion recognition method based on scene perception proposed in the present invention can recognize students' emotional states in real time and accurately in complex classroom scenes, and simulate teachers' feedback processing on students' emotions. Compared with the existing emotion recognition methods based on single modality or simple modal combinations, the present invention innovatively proposes an AMS mechanism, which significantly improves the robustness and flexibility of emotion recognition by adaptively adjusting the weights and processing frequencies of different modalities. In addition, asynchronous processing and multimodal fusion strategies effectively solve the problem of time series information loss of multimodal data, ensuring the integrity and accuracy of emotional features. The technology of the present invention has good scalability and generalization capabilities, and provides a new technical framework for classroom emotion recognition.

[0063] (3) The technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0064] In the field of classroom emotion recognition, there is always the problem of how to accurately capture students' emotional states in a changing classroom environment. Traditional static modal fusion methods are difficult to cope with the complexity of situations and the diversity of emotional signals. The present invention successfully solves this long-standing technical problem by introducing scene perception and AMS mechanism. Experimental results show that the SMAL model can maintain high-precision emotion recognition in a variety of classroom situations, significantly improving the adaptability of the emotion recognition system in practical applications.

[0065] (4) The technical solution of the present invention overcomes technical prejudice:

[0066] Single modality recognition in the prior art is often subject to the weight allocation of a specific modality, lacks comprehensive consideration of multi-modality, and is prone to bias. The AMS mechanism of the present invention dynamically adjusts the modality weight based on real-time feedback, overcomes the technical bias brought by a single modality, and enables the system to automatically select the optimal modality combination in different situations, thereby being more objective and accurate in the recognition process, ensuring the efficiency and reliability of the emotion recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a schematic diagram of a multimodal emotion recognition method based on modality adaptive learning provided by an embodiment of the present invention;

[0068] Figure 2 is a flow chart of a multimodal emotion recognition method based on modality adaptive learning provided by an embodiment of the present invention;

[0069] Figure 3 It is a structural diagram of a multimodal emotion recognition system based on modal adaptive learning provided by an embodiment of the present invention.

[0070] Figure 4 This is the training process performance of the SMAL model on the CMU-MOSEI dataset.

[0071] Figure 5 This is the performance of the SAML model’s training process on the IEMOCAP dataset. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0073] The multimodal classroom emotion recognition method based on modality adaptive learning of the present invention realizes accurate recognition of complex emotions through five steps: acquiring multimodal data, extracting features, optimizing modality combination, adaptive scoring and fusion modeling. The following is a detailed working principle:

[0074] First, in step S1 (multimodal data acquisition and preprocessing), the system acquires data of different modalities through a variety of sensing devices, such as speech, facial expressions, body language, heart rate and other emotion-related signals. To ensure data quality, the system preprocesses the raw data, including denoising, standardization and segmentation, so that subsequent analysis can be performed under the same standard. After preprocessing, the data of different modalities are converted into a unified format that can be directly used for feature extraction. This process ensures the integrity and consistency of the data of each modality.

[0075] In step S2 (feature extraction), the system extracts features from the data of each modality. Data features of different modalities are different, such as the pitch, rhythm and speed in voice data, and expression changes in facial images. The system uses deep learning models to extract these modality features, such as using convolutional neural networks (CNN) to extract image features of the visual modality and using long short-term memory networks (LSTM) to extract temporal features of the voice modality. These features represent the emotional information contained in different modalities and provide a basis for subsequent emotion recognition.

[0076] In step S3 (modal combination optimization in different scenarios), the system optimizes the multimodal combination based on the requirements of different application scenarios. For example, in a noisy environment, the emotional information of the speech modality is distorted, so the system will reduce the weight of the speech modality and give priority to the visual modality or other reliable modalities. By analyzing the scene characteristics, the system selects the modal combination that best suits the current environment to ensure the accuracy of emotion recognition. This optimization step improves the adaptability and recognition accuracy of the system in different situations.

[0077] The core of step S4 (adaptive modality scoring mechanism) is to assign weights to different modalities to enhance the influence of high-confidence modalities in the final result. The system assigns scores to modalities based on the quality and similarity of emotional feature extraction of each modality, and adaptively adjusts the weights of each modality. For example, when the accuracy of expression recognition in the visual modality is high, the system will increase the weight of the modality; when the background noise in the audio modality is large, the system will reduce the weight of the modality accordingly. This adaptive scoring mechanism enables the system to dynamically adjust the modality priority to more accurately reflect the user's emotional state.

[0078] In step S5 (multimodal fusion and time series modeling), the system fuses the multimodal features that have been optimized by scoring and performs time series modeling. Multimodal fusion uses a weighted fusion strategy to combine the features of different modalities into a representative emotion vector. At the same time, the system uses time series modeling (such as LSTM or Transformer networks) to capture the dynamic changes of emotional states over time and identify the fluctuation trends of emotions. Through fusion and time series modeling, the system can accurately identify the changes in emotions within a specific time period and ultimately output the user's emotion recognition results.

[0079] The multimodal emotion recognition method of the present invention not only improves the accuracy of emotion recognition, but also enables the system to flexibly adapt to different application scenarios and environments through the acquisition of multimodal data, feature extraction, adaptive weight allocation and time series modeling. This method is applicable to a variety of scenarios that require emotion analysis, such as intelligent customer service systems, educational interactive platforms and mental health testing, and provides an efficient, accurate and adaptable solution for the field of emotion computing.

[0080] The multimodal classroom emotion recognition method based on modal adaptive learning of the present invention realizes efficient emotion recognition of different modal data through multiple steps. First, in step S1, data including multiple modalities such as speech, text, and images are collected through a multimodal data acquisition and preprocessing module. In order to ensure data consistency and adaptability, the preprocessing process includes data denoising, alignment, and normalization to ensure that the data features of each modality can meet the subsequent feature extraction requirements. This step lays the data foundation for the emotion recognition system and ensures the quality and effectiveness of the input data.

[0081] Next, enter step S2 for feature extraction. For different modalities, the system uses specific feature extraction algorithms, such as using convolutional neural networks (CNN) to extract visual emotional features in image modalities, using natural language processing (NLP) methods to extract semantic emotional features of text, or extracting emotional features of speech through time-frequency analysis methods. After extraction, the features of each modality will be encoded into a high-dimensional feature vector to retain the emotional information of each modality to the greatest extent. This sub-modal feature extraction method can improve the pertinence of feature extraction and make the data features of each modality more representative.

[0082] In step S3, the system selects and optimizes the modal combinations in different emotional scenarios through the modal combination optimization module. Different modal combinations are required to recognize emotions in different scenarios. For example, in some scenarios, the emotional information of voice and image is more important, while in other scenarios, text information is dominant. Through the machine learning algorithm, the correlation between each modal feature and emotion classification is analyzed, and the system automatically adjusts the modal combination weight to adapt to the needs of different scenarios and optimize the accuracy of emotion recognition.

[0083] Finally, steps S4 and S5 complete the adaptive modality scoring and multimodal fusion. The adaptive modality scoring mechanism dynamically scores the importance of different modalities and adjusts the weight of each modality according to the intensity of different emotional expressions; then, through multimodal fusion and time series modeling, multimodal features are integrated and the temporal dependency of emotions is established, further improving the accuracy of emotion recognition. Time series modeling can also capture the dynamic characteristics of emotional changes, allowing the emotion recognition system to more accurately identify the continuous changes in emotions, thereby improving the system's recognition effect on complex emotional expressions.

[0084] like Figure 1 , Figure 2 As shown, the present invention specifically comprises the following steps:

[0085] 1. Multimodal data acquisition and preprocessing

[0086] Through the multimodal data acquisition system, multimodal data such as text, audio and video of students in the classroom teaching process are collected in real time. These data cover a variety of emotional clues such as language, voice, facial expressions and body movements of students in the classroom, and can comprehensively reflect the emotional expression of students in different classroom situations. In order to ensure the consistency and quality of each modal data, the present invention pre-processes the collected data, including data cleaning, format standardization and time synchronization processing, eliminating noise and redundant information, so as to provide high-quality input for subsequent feature extraction.

[0087] 2. Feature extraction

[0088] Advanced deep learning models are used to extract features for data of different modalities. The BERT model is used to extract semantic and emotional features from text data, MFCC is used to extract emotional features of speech signals from audio data, and 3D-CNN is used to extract emotional clues of facial expressions and body postures from video data. The extraction of these features provides a basis for subsequent emotion recognition.

[0089] 3. Modal combination optimization in different scenarios

[0090] A core innovation of the present invention is that, for different scenarios in classroom teaching (such as lectures, discussions, questions and answers, etc.), the contextual features of the current scenario are identified in real time, and the combined weights of each modality are dynamically adjusted to adapt to the differences in students' emotional expressions in different scenarios. Specifically, students' emotional expressions will be different in different classroom scenarios: in the "lecture" scenario, the emotional information of visual modalities such as facial expressions and postures is more prominent; while in the "discussion" or "question and answer" scenarios, voice features can better reflect the emotional state of students.

[0091] To achieve this optimization, the present invention introduces a scene perception mechanism to identify the current classroom context and adjust the weight of each modality according to the characteristics of the context. The specific weight distribution is calculated by the following formula:

[0092]

[0093] in, represents the weight of mode m at time step t.

[0094] It represents the initial weight of mode m at time step t. The initial weight is preset according to the scene characteristics (such as lectures, discussions, etc.).

[0095] s m is the importance coefficient of modality m in the current scene, indicating the relative weights of different modalities in different scenes. For example, in the “lecture” scene, the importance coefficient of the visual modality s m The value is higher, while in the “discussion” scenario, the s m Higher value.

[0096] Through this weight distribution formula, the present invention can dynamically adjust the weight of each modality in different situations to ensure that the most effective emotional information is captured in different classroom scenes. The effect of this formula in different situations is as follows: when the scene perception mechanism detects the "teaching" scene, the system weights the visual modality (such as facial expression, posture) Increase; when the scene is converted to "discussion" or "question and answer", the weight is dynamically adjusted to enhance the influence of speech modality in emotion recognition.

[0097] In addition, to ensure the real-time performance of scene recognition, the scene perception mechanism of the present invention realizes rapid recognition through the following scene judgment formula:

[0098]

[0099] Among them, c t is the classroom scene category at the current time step t, and its value is "lecture", "discussion" or "question and answer".

[0100] C represents the scene category set.

[0101] f(c,x t ) is the scene classification function, according to the input emotional feature x t Classification is performed to ensure efficient and accurate scene judgment.

[0102] Through the above two steps of optimization, namely real-time scene recognition and dynamic weight adjustment, the present invention can flexibly respond to various situations in the classroom environment, dynamically allocate modal combination weights, and improve the accuracy and adaptability of emotion recognition. This scene-aware modal combination optimization mechanism enables the system to effectively capture key emotional information in complex classroom situations, ensuring the accuracy and robustness of the system.

[0103] 4. Adaptive Modal Scoring (AMS)

[0104] In order to further optimize the fusion of multimodal data, this paper proposes an adaptive modality scoring mechanism (AMS). This mechanism uses real-time feedback (such as the accuracy of emotion recognition and system response) to

[0105] time) to dynamically adjust the weight combination of each modality. The update formula of the AMS mechanism is:

[0106] AMS(s t ,a t )←AMS(s t ,a t )+α[R(s t ,a t )+γ·maxAMS(s t+1 ,a t )-AMS(s t ,a t )]

[0107] This formula is used to calculate the t Update modal combination a t The meaning of each symbol in the formula is as follows:

[0108] AMS(s t ,a t ): means in state s t Select action a in (current classroom scene) t The adaptive score of (modal combination). This score reflects the performance of the modal combination in the current scenario. The higher the score, the more suitable the modal combination is for the current scenario.

[0109] α: Learning rate, controls the step size of AMS value update. Usually set to 0≤α≤1. The larger the value, the more sensitive the system is to new feedback and the larger the update step size. When the value is smaller, the system update is more stable and the response is slower.

[0110] R(s t ,a t ): Reward function, used to evaluate the t Next select modal combination a tThe reward function is defined based on the accuracy of emotion recognition and the system response time, and the formula is as follows:

[0111] γ: Discount factor, which indicates the importance the system places on future scenarios, and is usually set to 0≤γ≤1. The larger the discount factor, the more the system focuses on long-term benefits; the smaller the discount factor, the more the system focuses on current benefits.

[0112] maxAMS(s t+1 ,a t ): indicates the state s at the next moment t+1 The maximum AMS value obtained by selecting any modal combination a under . This reflects the system's prediction of the optimal modal combination in future scenarios and guides the system to make a better choice in the current scenario.

[0113] R(s t ,a t )=ω1·Accuracy(s t ,a t )+ω2·ResponseTime(s t ,a t )

[0114] Accurracy(s t ,a t ): This modal combination is in scene s t The higher the accuracy, the greater the reward.

[0115] ResponseTime(s t ,a t ): The response time of emotion recognition. The shorter the response time, the greater the reward. If the response time is too long, it can be set to a negative value.

[0116] ω1 and ω2 are weight coefficients used to balance the effects of accuracy and response time on the reward.

[0117] 5. Multimodal Fusion and Time Series Modeling

[0118] The present invention adopts the self-attention mechanism to perform weighted fusion on the extracted multimodal features, ensuring that the features of different modalities can be effectively combined to enhance the accuracy and robustness of emotion recognition. The fused features are modeled in time series through the gated recurrent unit (GRU) to capture the temporal dynamic changes of the emotional state, and finally the emotional state of the students is classified as positive, neutral or negative through the Softmax classifier.

[0119] An application embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of a multimodal emotion recognition method based on modal adaptive learning.

[0120] An application embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of a multimodal emotion recognition method based on modal adaptive learning.

[0121] An application embodiment of the present invention provides an information data processing terminal, which includes a multimodal emotion recognition system based on modality adaptive learning.

[0122] The following are two specific embodiments of a multimodal classroom emotion recognition method based on modality adaptive learning:

[0123] Example 1: Emotion Recognition System in Video Conference

[0124] In remote video conferences, emotion recognition systems can help understand the emotional state of participants, thereby enhancing communication. The system obtains data from different modalities through cameras, microphones, and text information streams.

[0125] 1. Multimodal data acquisition and preprocessing: The system obtains the facial expressions of participants from the camera, collects voice through the microphone, and extracts text from the conference chat records. Image data is denoised, aligned, and normalized; voice data is denoised and voice segmented; and text data is filtered for emotional keywords and word vectors are processed to ensure data clarity and consistency.

[0126] 2. Feature extraction: The system uses a convolutional neural network (CNN) to extract expression features from images to capture changes in facial micro-expressions. Speech features are converted into time-frequency domain features through spectrograms to extract emotional tone features such as pitch and speed. The text uses a pre-trained BERT model to extract semantic features and generate an emotional vector.

[0127] 3. Modality combination optimization: In meetings, the emotional expression of voice and images is more important than text. The system automatically optimizes the modality combination in the scene, giving priority to facial expressions and voice information, but when the quality of the voice modality decreases (such as low volume or fuzzy audio), the system will adaptively increase the weight of the text modality to ensure recognition accuracy.

[0128] 4. Adaptive modality scoring and multimodal fusion: The system scores the quality of each modality, such as image clarity, speech signal-to-noise ratio, etc., and dynamically adjusts the fusion weight of the modality. Through adaptive fusion and long short-term memory network (LSTM) for time series modeling, the system can identify changes in emotions over time, and output emotional change charts and status prompts for each participant in the meeting, thereby helping the meeting host to better understand the emotional dynamics of the participants.

[0129] Example 2: Sentiment Analysis of Social Media Video Content

[0130] This embodiment is applied to sentiment analysis of short video content on social media to understand the emotions and emotional tendencies of video publishers, and is suitable for scenarios such as content recommendation or user sentiment analysis.

[0131] 1. Multimodal data acquisition and preprocessing: The system obtains video image frame sequences, background music, and video subtitles from short video content. The image frames are denoised and standardized to ensure the consistent quality of consecutive frames; the music part is processed by audio segmentation and noise reduction, and the video subtitles are processed by natural language processing (NLP) technology to remove irrelevant information and stop words.

[0132] 2. Feature extraction: The image frames are processed through a convolutional neural network (CNN) to extract visual emotional features, such as facial expressions and body language. The music is converted into spectral features, and the emotional tone of the music (such as cheerfulness and depression) is extracted through an audio emotion recognition model. The subtitle text is processed through a natural language processing model, such as BERT, to extract the emotional features of the text semantics and generate word vectors.

[0133] 3. Modality combination optimization: For social media videos, the system mainly relies on visual and music modes to judge emotional expression. If the expression is not obvious or the music is missing in the image frame, the system will automatically increase the weight of the subtitle text, so that emotion recognition can adapt to different video content styles.

[0134] 4. Adaptive modality scoring and multimodal fusion: The system scores the image frame clarity, audio quality, and subtitle semantic matching through modality scoring, and automatically adjusts the fusion weights of different modalities. The Transformer model is then used for multimodal fusion and time series modeling to generate a video sentiment analysis report. The report contains the overall sentiment tendency of each video (such as positive, negative, neutral), as well as a sentiment change trend chart, to provide support for video content recommendation and user sentiment analysis.

[0135] The present invention discloses a multimodal classroom emotion recognition system based on scene perception and modal adaptive learning. Through the implementation of hardware, software or a combination of hardware and software, it can realize efficient and accurate emotion analysis in a complex classroom environment. The system uses multimodal data (text, audio, video expressions, gestures, etc.) to obtain students' emotional states and provide data support for the education evaluation system in real time. The following is the technical effect evidence obtained by combining data and charts in the actual project test process.

[0136] 1. Implementation

[0137] The embodiments of the present invention may be implemented by hardware, software, or a combination of hardware and software, specifically including the following implementations:

[0138] Hardware implementation: The core functions of emotion recognition are implemented through semiconductor components such as very large scale integrated circuits (VLSI), field programmable gate arrays (FPGA) or logic chips to improve the system's processing efficiency and real-time response capabilities. This type of hardware implementation is suitable for scenarios that require high computing speed and can process and analyze multimodal data in an instant.

[0139] Software implementation: The core algorithms of emotion recognition (including scene perception, adaptive modality scoring (AMS) mechanism, multimodal fusion and emotion classification) can be implemented with software code and stored in the computer's memory (such as hard disk or solid-state drive), and the program is run through an appropriate instruction execution system (such as a microprocessor or dedicated hardware). The software implementation is flexible and suitable for a variety of computing platforms, and the algorithm performance can be optimized through software updates.

[0140] Implementation of hardware and software combination: In the implementation of hardware and software combination, some modules (such as data preprocessing module) are implemented by hardware to improve real-time performance; other parts (such as emotion recognition algorithm) are completed by software to enhance the flexibility and scalability of the system. For example, FPGA can be used to process audio and video data in real time, while emotion classification can be implemented by software modules running on microprocessors.

[0141] 2. Evidence of technical effects

[0142] The technical solution of the present invention demonstrates its remarkable technical effect through experimental data and results of actual projects. The following are specific evidence data:

[0143] Experimental data and model performance: Experimental results on the CMU-MOSEI dataset show that the verification accuracy of the SMAL model reached 89.2%, while the accuracy of the mainstream models MUIT and ATT were 71.2% and 74.8% respectively. In addition, the F1 score of the SMAL model reached 88.6%, which is significantly higher than 69.9% of MUIT and 72.7% of ATT. These data demonstrate the superior performance of the SMAL model in emotion recognition, and its accuracy and F1 score are significantly better than other comparison models.

[0144] Asynchronous processing of multimodal data and adaptive modal combination optimization: The present invention introduces an adaptive modal scoring (AMS) mechanism, which adjusts the weights of different modalities through dynamic feedback, thereby achieving flexible modal combination optimization in changing classroom situations. The introduction of the AMS mechanism effectively improves the accuracy and robustness of emotion recognition, and keeps the system's emotion recognition accuracy at a high level in different situations. In addition, the AMS mechanism can still ensure that each modal feature is integrated within a suitable time window in complex situations, making the emotion recognition system highly adaptable and real-time.

[0145] Real-time performance and response speed: Through the asynchronous processing mechanism, the system can adaptively adjust the processing frequency according to the characteristics of each modality. For example, video data is processed slowly, and audio data is processed more frequently, thus ensuring that the data of each modality can be integrated in time and the timing information is retained. The experimental results show that the system has low latency and fast response speed in emotion recognition in changing classroom situations, which can meet the needs of real-time emotion monitoring.

[0146] 3. System applicability and scalability

[0147] The emotion recognition system of the present invention has good applicability and is suitable for real-time monitoring of students' emotional states in classroom situations. It is also suitable for fields such as education evaluation, emotion analysis, and personalized learning recommendations. The system supports exporting recognition results as CSV format files for subsequent analysis and processing. In addition, the system is designed with good scalability and can be connected to other educational intelligent systems or data platforms, with great commercial application prospects.

[0148] 4. Innovative technical effects of the present invention

[0149] The present invention achieves the following innovative technical effects in emotion recognition: significantly improves the accuracy and adaptability of emotion recognition: experimental results show that the SMAL model performs well in multimodal emotion recognition tasks, with verification accuracy and F1 scores significantly higher than mainstream models. Especially in complex classroom situations, through the scene perception mechanism and AMS dynamic weight adjustment mechanism, the system can flexibly adapt to situational changes and ensure recognition accuracy.

[0150] Enhance the real-time and robustness of the system: Through asynchronous processing mechanism and adaptive modal combination optimization, the present invention successfully overcomes the limitations of static modal fusion in traditional emotion recognition methods, enabling the system to always maintain high recognition performance and response speed in different classroom situations, meeting the needs of real-time monitoring. Improve the application value and scalability of the system: The emotion recognition system is not only suitable for classroom situations, but can also be extended to other fields of emotion monitoring and educational evaluation. The system's structural design and data export function make it more practical and commercially viable in educational intelligent applications.

[0151] Table 1 is the performance comparison of SMAL and mainstream models.

[0152]

[0153] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A multimodal classroom emotion recognition method based on modality adaptive learning, characterized in that: include: S1. Multimodal data acquisition and preprocessing: Acquire emotion-related multimodal data, including speech, facial expressions, body language, and heart rate data, through multiple sensing devices, and perform denoising, standardization, and segmentation on the acquired data to form a unified format for feature extraction; S2. Feature extraction: Apply deep learning models to extract features for each modality data, including using convolutional neural networks (CNN) to extract visual modality image features and long short-term memory networks (LSTM) to extract speech modality temporal features, to obtain multi-modal emotional feature information; S3. Modal combination optimization in different scenarios: According to the current environmental characteristics, the combination of different modalities is optimized, and the weight distribution of modalities is adjusted according to the data interference in specific scenarios, so as to give priority to the modal combination suitable for the current environment and improve the accuracy of emotion recognition; S4. Adaptive modality scoring mechanism: Based on the extraction quality and credibility of each modality feature, each modality is scored and its weight is adaptively adjusted. For example, when the expression recognition accuracy is high, the visual modality weight is increased, and when the audio noise is large, the audio modality weight is reduced, so as to dynamically optimize the modality priority. S5. Multimodal fusion and time series modeling: Weighted fusion of the multimodal features after scoring optimization is performed, the emotional features of different modalities are integrated into an emotional vector, and the dynamic changes of the emotional state are analyzed using the time series modeling method, so as to output the user's emotion recognition results.

2. The multimodal classroom emotion recognition method based on modality adaptive learning as claimed in claim 1, characterized in that: S1 specifically includes: through a multimodal data acquisition system, real-time collection of students' multimodal data such as text, audio and video during classroom teaching; these data cover students' language, voice, facial expressions and body movements in the classroom and other emotional clues, which can comprehensively reflect students' emotional expression in different classroom situations; to ensure the consistency and quality of each modal data, the present invention pre-processes the collected data, including data cleaning, format standardization and time synchronization processing, to eliminate noise and redundant information, thereby providing high-quality input for subsequent feature extraction.

3. The multimodal classroom emotion recognition method based on modality adaptive learning as claimed in claim 1, characterized in that: S2 specifically includes: for data of different modalities, advanced deep learning models are used for feature extraction; semantic and emotional features of text data are extracted through the BERT model, emotional features of speech signals of audio data are extracted through MFCC, and emotional clues of facial expressions and body postures of video data are extracted through 3D convolutional neural networks.

4. The multimodal classroom emotion recognition method based on modality adaptive learning as claimed in claim 1, characterized in that: S3 specifically includes: for different scenarios in classroom teaching, real-time identification of the situational features of the current scenario and dynamic adjustment of the combined weights of each modality to adapt to the differences in students' emotional expressions in different scenarios; students' emotional expressions will be different in different classroom scenarios: in the "lecture" scenario, the emotional information of visual modalities such as facial expressions and postures is more prominent; in the "discussion" or "question and answer" scenario, voice features can better reflect the students' emotional state; The scene perception mechanism is used to identify the current classroom situation and adjust the weight of each mode according to the characteristics of the situation; the specific weight distribution is calculated by the following formula: in, represents the weight of mode m at time step t; represents the initial weight of mode m at time step t, and the initial weight is preset according to the scene characteristics; s m is the importance coefficient of modality m in the current scene, indicating the relative weights of different modalities in different scenes; in the "teaching" scene, s of the visual modality m The value is higher, while in the "discussion" scenario, the s m Higher value; Through the weight distribution formula, the weight of each modality in different situations is dynamically adjusted to ensure that the most effective emotional information is captured in different classroom scenes; the effect of this formula is reflected in different situations as follows: when the scene perception mechanism detects the "teaching" scene, the system will weight the visual modality Added; When the scene is converted to "discussion" or "question and answer", the weight is dynamically adjusted to enhance the influence of speech modality in emotion recognition; In addition, to ensure the real-time performance of scene recognition, the scene perception mechanism uses the following scene judgment formula to achieve rapid recognition: Among them, c t is the classroom scene category at the current time step t, and its value is "lecture", "discussion" or "question and answer"; C represents the scene category set; f(c,x t ) is the scene classification function, according to the input emotional feature x t Classification is performed to ensure efficient and accurate scene judgment.

5. The multimodal classroom emotion recognition method based on modality adaptive learning as claimed in claim 1, characterized in that: S4 specifically includes: in order to further optimize the fusion of multimodal data, an adaptive modality scoring mechanism is adopted; this mechanism dynamically adjusts the weight combination of each modality through real-time feedback; the update formula of the AMS mechanism is: AMS(s t ,a t )←AMS(s t ,a t )+α[R(s t ,a t )+γ·maxAMS(s t+1 ,a t )-AMS(s t ,a t )] This formula is used to calculate the t Update modal combination a t The meanings of the symbols in the formula are as follows: AMS(s t ,a t ): means in state s t Next select action a t The adaptive score of the modality combination reflects the performance of the modality combination in the current scenario. The higher the score, the more suitable the modality combination is for the current scenario. α: learning rate, which controls the step size of AMS value update; usually set to 0≤α≤1. A larger value means that the system is more sensitive to new feedback and the update step size is larger; a smaller value means that the system update is more stable and the response is slower; R(s t ,a t ): Reward function, used to evaluate the t Next select modal combination a t The reward function is defined based on the accuracy of emotion recognition and the system response time, and the formula is as follows: R(s t ,a t )=ω1·Accuracy(s t ,a t )+ω2·ResponseTime(s t ,a t ) Accurracy(s t ,a t ): This modal combination is in scene s t The higher the accuracy, the greater the reward; ResponseTime(s t ,a t ): The response time of emotion recognition. The shorter the response time, the greater the reward. If the response time is too long, it can be set to a negative value. ω1 and ω2 are weight coefficients used to balance the impact of accuracy and response time on rewards; γ: discount factor, which indicates the importance the system places on future scenarios. It is usually set to 0≤γ≤1. The larger the discount factor, the more the system focuses on long-term benefits. The smaller the discount factor, the more the system focuses on current benefits. maxAMS(s t+1 ,a t ): indicates the state s at the next moment t+1 The maximum AMS value obtained by selecting any modal combination a under ; this item reflects the system's prediction of the optimal modal combination in future scenarios and guides the system to make a better choice in the current scenario.

6. The multimodal classroom emotion recognition method based on modality adaptive learning as claimed in claim 1, characterized in that: S5 specifically includes: using the self-attention mechanism to perform weighted fusion on the extracted multimodal features to ensure that the features of different modalities can be effectively combined to enhance the accuracy and robustness of emotion recognition; the fused features are modeled through the gated recurrent unit to capture the temporal dynamic changes of the emotional state, and finally the students' emotional state is divided into positive, neutral or negative through the Softmax classifier.

7. A multimodal emotion recognition system based on modal adaptive learning that implements the multimodal emotion recognition method based on modal adaptive learning as claimed in any one of claims 1 to 6, characterized in that: include: Data preprocessing module, used for multimodal data acquisition and preprocessing; A feature extraction module, used for feature extraction; Modal combination optimization module, used to perform modal combination optimization in different scenarios; A modal scoring module, used for performing adaptive modal scoring; The multimodal fusion and time series modeling module is used for multimodal fusion and time series modeling.

8. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multimodal emotion recognition method based on modal adaptive learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the multimodal emotion recognition method based on modal adaptive learning as described in any one of claims 1 to 6.

10. An information data processing terminal, comprising the multimodal emotion recognition system based on modal adaptive learning as claimed in claim 7.

Images