A classroom multi-modal emotion state monitoring and early warning method and system based on space-time alignment

Abstract

The present application relates to the field of multi-modal data processing and intelligent perception technology, and provides a kind of multi-modal emotional state monitoring and early warning method and system based on space-time alignment in classroom.The present application method includes: (1) classroom multi-modal data acquisition and space-time alignment; (2) double-flow multi-modal feature extraction; (3) bidirectional cross-modal feature fusion; (4) time series modeling and emotional state determination; (5) emotional trend assessment and early warning; (6) result generation and report output.The present application method and system can stably realize multi-modal emotional state monitoring and early warning in intensive classroom environment, and provide objective, continuous emotional state analysis basis for classroom teaching management.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing and intelligent sensing technology, specifically to a method and system for monitoring and early warning of multimodal emotional states in the classroom based on video and audio data. Background Technology

[0002] In classroom teaching settings, students' emotional states are closely related to their level of engagement and participation in class. Objective and continuous monitoring of students' emotional states, and issuing early warnings when abnormal fluctuations occur, can provide data support for classroom management and learning process regulation. However, in actual teaching environments, classrooms are densely populated with significant individual differences, making it difficult to perceive students' emotional states accurately and in real time manually. Therefore, there is an urgent need to leverage information technology to achieve automated monitoring and analysis.

[0003] In existing technologies, automatic identification methods for students' emotional states are mainly based on the analysis of single-modal data, such as facial expression recognition methods based on facial images or emotion analysis methods based on speech signals. However, these single-modal methods have significant limitations in complex classroom environments. On the one hand, students' facial expressions are easily affected by individual habits, posture changes, and environmental factors. Different students exhibit significant differences in their facial expressions in the same situation, and relying solely on visual information can easily lead to biased judgments of emotional states. On the other hand, changes in tone, speed, and pitch in speech signals are not only related to emotional states but are also affected by factors such as speaking habits, language content, and environmental noise. Analyzing solely based on the audio modality also makes it difficult to guarantee the stability and accuracy of recognition.

[0004] To improve the robustness of emotion state recognition, some existing technologies attempt to introduce multimodal fusion strategies involving video and audio data. However, current multimodal methods generally assume that different modalities are strictly synchronized on the time axis and fuse them through simple feature concatenation or weighting. This approach is effective in experimental settings, but in real classroom scenarios, due to factors such as differences in the location of acquisition devices, signal transmission delays, and inconsistent sampling frequencies, video and audio data often exhibit nonlinear time offsets. This makes frame-level synchronization-based fusion methods difficult to apply, thus affecting the reliability of multimodal analysis results.

[0005] Furthermore, existing technologies mostly focus on classifying or judging emotional states at a single point in time, lacking continuous modeling and trend analysis of emotional states changing over time. This makes it difficult to depict the dynamic evolution of students' emotional states and to provide timely and effective early warning information when abnormal fluctuations occur. This analysis method, primarily based on instantaneous judgment, cannot meet the actual needs of continuous monitoring and process analysis in classroom teaching.

[0006] In summary, existing student classroom emotional state monitoring technologies still have shortcomings in terms of spatiotemporal alignment of multimodal data, stable association of individual identities, effective fusion of multimodal features, and analysis of emotional state change trends. There is an urgent need for a multimodal emotional state monitoring and early warning method and system that can operate stably in real classroom environments and has high robustness. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for monitoring and early warning of multimodal emotional states in the classroom based on spatiotemporal alignment. This addresses technical problems in existing technologies, such as the difficulty in stably aligning multimodal data in a real classroom environment, the difficulty in accurately associating individual student identities, the insufficient robustness of multimodal emotional state analysis, and the lack of continuous monitoring and early warning mechanisms for changes in emotional states. By performing spatiotemporal alignment and identity association on video and audio data in a classroom setting, and by jointly modeling and temporally analyzing multimodal features, continuous monitoring and trend early warning of changes in students' emotional states can be achieved, thereby providing objective data support for classroom teaching management.

[0008] To achieve the above objectives, this invention provides a method for monitoring and early warning of multimodal emotional states in the classroom based on spatiotemporal alignment, the method comprising the following steps:

[0009] (1) Classroom multimodal data acquisition and spatiotemporal alignment; In the classroom teaching scenario, students' video data V0 and audio data A0 are collected synchronously, and processed by intermittent time windows to form video data sequence V1 and audio data sequence A1 respectively; Based on the visual region of interest and the audio acquisition direction, further processing is performed to obtain video data sequence V2 and audio data sequence A2, and the temporal and spatial dimensions are aligned to generate multimodal data sequences corresponding to specific students;

[0010] (2) Dual-stream multimodal feature extraction; Based on the multimodal data sequence, the video data sequence V2 and the audio data sequence A2 are subjected to independent feature analysis processing respectively. Visual feature extraction processing is used to generate a visual feature sequence representing changes in the student's facial state, and auditory feature extraction processing is used to generate an auditory feature sequence representing changes in speech state, thereby forming a dual-stream feature representation;

[0011] (3) Bidirectional cross-modal feature fusion: Joint modeling of visual feature sequences and auditory feature sequences at the feature level, and through bidirectional cross-modal feature fusion processing, visual features and auditory features guide each other and perform semantic alignment and enhancement, generating a multimodal feature representation containing visual enhancement features and auditory enhancement features;

[0012] (4) Temporal modeling and emotion state determination: Based on multimodal feature representation, temporal correlation analysis is performed on the multimodal feature sequence in the classroom teaching process, and the emotion state representation results corresponding to different time nodes are output;

[0013] (5) Emotion trend assessment and early warning; Based on the emotional state representation results at continuous time points, assess the intensity and fluctuation characteristics of emotional state changes over time, construct the emotional state change trend, and generate corresponding early warning information when the intensity or fluctuation characteristics meet the preset conditions.

[0014] (6) Results generation and report output: The emotional state representation results and early warning information are integrated and structured to generate an emotional state analysis report that describes the trend of changes in students' emotional state and early warning situations during classroom teaching.

[0015] This invention also provides a classroom multimodal emotional state monitoring and early warning system based on spatiotemporal alignment, the system comprising:

[0016] The multimodal data acquisition and alignment module is used to simultaneously acquire students' video data V0 and audio data A0 in classroom teaching scenarios. It processes the data through intermittent time windows to form video data sequence V1 and audio data sequence A1. Based on the visual region of interest and the audio acquisition direction, it further processes the data to obtain video data sequence V2 and audio data sequence A2. It then performs alignment processing in the time and spatial dimensions to generate a multimodal data sequence that corresponds one-to-one with a specific student.

[0017] The dual-stream feature extraction module is used to perform feature analysis processing on the video data sequence V2 and the audio data sequence A2 in the multimodal data sequence, respectively, converting the original video data into a visual feature sequence representing changes in the student's facial state, and converting the original audio data into an auditory feature sequence representing changes in speech state.

[0018] The bidirectional cross-modal feature fusion module is used to perform joint modeling of visual feature sequences and auditory feature sequences at the feature level. Through bidirectional cross-modal feature fusion processing, visual features and auditory features can introduce each other's information to generate a multimodal feature representation that includes visual enhancement features and auditory enhancement features.

[0019] The temporal modeling and emotion state determination module performs temporal correlation analysis on the time series features in the classroom teaching process based on multimodal feature representation, and outputs the emotion state representation results corresponding to different time nodes;

[0020] The emotion trend assessment and early warning module is used to calculate the intensity and magnitude of changes in emotion state based on the emotion state representation results, and generate corresponding early warning information according to preset early warning judgment conditions.

[0021] The results generation and report output module is used to integrate and structure the emotional state representation results and early warning information to generate an emotional state analysis report that describes the trend of changes in students' emotional states and early warning situations during classroom teaching.

[0022] Compared with existing technologies, this invention introduces a multimodal data spatiotemporal alignment mechanism based on physical space constraints in classroom teaching scenarios, achieving a stable correspondence between video and audio data in both temporal and spatial dimensions, thereby improving the accuracy of the association between multimodal data and individual students. Simultaneously, through dual-stream feature extraction and bidirectional cross-modal feature fusion, it jointly models multi-source information in complex classroom environments, enhancing the robustness of emotion state analysis under conditions of occlusion, noise, and acquisition delay. Furthermore, by performing temporal modeling and trend assessment of the emotion state change process, it achieves continuous monitoring and tiered early warning of students' emotion states, avoiding misjudgments caused by judging solely based on instantaneous states, and providing reliable technical support for the objective analysis and process monitoring of classroom emotion states. Attached Figure Description

[0023] Figure 1 is a schematic diagram of the classroom multimodal emotional state monitoring and early warning method according to an embodiment of the present invention.

[0024] Figure 2 is a schematic diagram of the structure of the visual feature extraction module in an embodiment of the present invention.

[0025] Figure 3 is a schematic diagram of the auditory feature extraction module in an embodiment of the present invention.

[0026] Figure 4 is a schematic diagram of the structure of the multimodal feature fusion module in an embodiment of the present invention.

[0027] Figure 5 is a schematic diagram of the student's real-time emotional status panel in an embodiment of the present invention.

[0028] Figure 6 is a schematic diagram illustrating the trend analysis of students' emotional state over time in an embodiment of the present invention.

[0029] Figure 7 is a schematic diagram of result generation and report output in an embodiment of the present invention. Detailed Implementation

[0030] The following detailed description, in conjunction with the accompanying drawings, provides a detailed explanation of the specific implementation methods and systems for classroom multimodal emotional state monitoring and early warning based on spatiotemporal alignment as described in this invention.

[0031] It should be understood that the following embodiments are only used to explain the technical solutions of the present invention and do not constitute a limitation on the scope of protection of the present invention. Equivalent substitutions or modifications made by those skilled in the art to the embodiments without departing from the technical concept of the present invention should all fall within the scope of protection of the present invention.

[0032] In this embodiment, a classroom teaching scenario is used as the application environment. By performing multimodal acquisition, spatiotemporal alignment, feature analysis, and time-series modeling of student video and audio data, continuous monitoring and trend early warning of changes in students' emotional states are achieved. The overall process is as follows: Figure 1 As shown.

[0033] Step (1) Classroom multimodal data acquisition and spatiotemporal alignment

[0034] In this embodiment, the collection and spatiotemporal alignment of classroom multimodal data are performed first to provide a stable and reliable data foundation for subsequent multimodal feature analysis.

[0035] (1-1) Multimodal data acquisition. In classroom teaching scenarios, video data V0 and audio data A0 of students are simultaneously acquired by multimodal acquisition terminals deployed in the classroom. Video data V0 is used to record the facial and upper body image information of students in the classroom, and audio data A0 is used to record the students' speech and related sound information in the classroom environment.

[0036] To adapt to real-world classroom scenarios with multiple students present simultaneously, a region of interest (ROI) is pre-defined within the video capture frame, corresponding to each student's seat position based on the classroom seating arrangement. This pre-defines the spatial scope of subsequent visual data. The ROI remains spatially fixed during the capture process, thus avoiding the computational overhead associated with complex target tracking algorithms.

[0037] In this embodiment, the system can periodically collect classroom multimodal data using a time window sampling method, forming a video data sequence V1 and an audio data sequence A1. The length of the video data, the length of the audio data, and the sampling frequency for each collection can be set according to the specific teaching scenario and equipment conditions.

[0038] (1-2) Visual data processing. For video data sequence V1, fixed regions of interest corresponding to each student's seat position are pre-defined in the video capture frame. The video frame is then cropped based on the regions of interest to obtain video data sequence V2 corresponding to each student's seat position.

[0039] The cropped image data is normalized in size and arranged according to the seating order of students in the classroom, forming a visual data sequence with a clear spatial order. This processing method ensures that each image data point can be stably mapped to a specific student during subsequent visual feature analysis.

[0040] (1-3) Auditory data processing. In this embodiment, for audio data sequence A1, a virtual pickup beam pointing to the physical coordinates of each seat is constructed using beamforming and sound source localization technology of a microphone array. This technology is equivalent to forming multiple directional audio in space, spatially filtering and separating the independent speech stream of a specific seat from the mixed background sound field, and filtering out silence and ambient noise through speech activity detection to obtain audio data sequence A2 corresponding to each seat.

[0041] (1-4) Spatiotemporal alignment and identity association. Video data sequence V2 and audio data sequence A2 from the same seat coordinate (i.e. the same student) are bound to each other by identity. Based on the unified timestamp at the time of collection, video data sequence V2 and audio data sequence A2 are aligned to generate a high-quality multimodal data sequence that corresponds one-to-one with a specific student.

[0042] Step (2) Dual-stream multimodal feature extraction

[0043] After completing the collection and spatiotemporal alignment of classroom multimodal data, visual and auditory features are extracted from the multimodal data sequences corresponding to specific individual students, so as to convert the video data sequence V2 and the audio data sequence A2 into feature representations that are easy to integrate and analyze later.

[0044] (2-1) Visual Feature Extraction. In one embodiment, the visual feature extraction module is used to extract visual feature information that can characterize changes in facial state from student image data, and its structure is shown in the figure below. Figure 2 As shown.

[0045] Specifically, the input video data sequence V2 is preprocessed, including size normalization, pixel value normalization, and data format conversion, to meet the input requirements of the visual feature extraction module. Subsequently, the preprocessed image data is input into the visual feature extraction network EmoNeXt. The EmoNeXt module consists of a Spatial Transformation Network (STN), ConvNeXt, a Squeeze-and-Excitation (SE) module, and self-attention regularization. Through multi-layer feature mapping and nonlinear transformation, it extracts local region features and overall structural features from the image step by step.

[0046] In one embodiment, the visual feature extraction process uses an STN spatial transformation network to perform spatial correction on the input image to reduce the impact of factors such as changes in student head posture and shooting angle shift on the stability of the feature extraction results. Next, the input is fed into a hierarchical structure with four stages. Each stage contains multiple ConvNeXt blocks. ConvNeXt consists of multiple core components, including convolutional layers, normalization layers, activation functions, deep convolutional layers, fully connected layers, and Dropath layers. At the same time, a channel weight adjustment or feature enhancement mechanism (SE) module is inserted after each ConvNeXt stage to adaptively adjust the importance of different feature channels, thereby highlighting the expressive ability of key facial region features. Finally, the features are output.

[0047] Through the above processing, visual feature sequences corresponding to each student and sampling time window are obtained, which are used for subsequent multimodal feature fusion and temporal analysis.

[0048] (2-2) Auditory Feature Extraction. In one embodiment, the auditory feature extraction module is used to perform feature analysis processing on the audio data sequence A2 obtained in step (1) to extract auditory feature information that can reflect changes in the student's speech state. Its structure is shown in the figure below. Figure 3 As shown.

[0049] Specifically, the acquired audio data is first preprocessed, including speech activity detection, frame segmentation, and frequency domain transformation, to filter out background noise and silent segments, obtaining audio feature representations containing valid speech information. This preprocessing reduces the impact of classroom environmental noise and invalid speech on subsequent feature analysis.

[0050] In one embodiment, the preprocessed audio features are input into a frame-level feature extraction network. This module includes a one-dimensional convolutional layer, batch normalization, and a ReLU activation function to extract frame-level auditory feature representations that reflect changes in speech intensity, rhythmic features, and temporal structure. The temporal feature mapping structure may include one-dimensional convolution operations and nonlinear transformations to capture local variation features of the speech signal in the time dimension.

[0051] To further enhance the stability of auditory features in cases where speech segments are sparse or speech durations are short, in one embodiment, frame-level auditory features can be aggregated. By concatenating the input attention pooling features, mean pooling features, and standard deviation pooling features, the feature information from different time frames is weighted or statistically summarized, and a statistically stable auditory feature vector representation is generated through a fully connected layer.

[0052] Through the above processing, auditory feature sequences corresponding to each student and sampling time window are obtained, which are used for subsequent multimodal feature fusion and temporal modeling.

[0053] Step (3) Bidirectional cross-modal feature fusion

[0054] After completing the dual-stream multimodal feature extraction, the obtained visual and auditory feature sequences are subjected to cross-modal feature fusion processing to achieve joint modeling of different modal features at the feature level. The structural diagram of the bidirectional cross-modal feature fusion module is shown below. Figure 4 As shown.

[0055] (3-1) Feature Dimension Mapping and Alignment Preparation. Since visual feature sequences and auditory feature sequences differ in feature dimension, representation space and numerical distribution, the features of the two modalities are first processed by dimension mapping before cross-modal fusion.

[0056] In one embodiment, linear or convolutional feature mapping is performed on the visual feature sequence and the auditory feature sequence respectively. The features of different modalities are uniformly mapped to the same feature representation space through a one-dimensional convolutional layer. Then, position encoding is added to the visual feature sequence and the auditory feature sequence to make them comparable in the channel dimension and the time dimension, thereby providing a foundation for subsequent cross-modal association modeling.

[0057] (3-2) Two-way cross-modal feature interaction. After the feature dimension unification is completed, two-way cross-modal feature interaction processing is performed so that the visual feature sequence and the auditory feature sequence can introduce each other's information during the fusion process.

[0058] In one embodiment, by constructing a bidirectional cross-modal association path, visual features can be combined with auditory feature information within the corresponding time range during the fusion process, and auditory features can be combined with visual feature information within the corresponding time range during the fusion process. Through this bidirectional interaction, different modal features can complement and enhance each other at the feature level, thereby reducing the impact of incomplete or noise-affected single-modal information on the stability of the fusion result.

[0059] Bidirectional cross-modal feature interaction does not rely on strict frame-level synchronization. Instead, it models the correlation between different modal features in the feature representation space, thus adapting to the actual situation of multimodal data acquisition delays or time misalignments in the classroom environment. This feature interaction mechanism uses Query-Key-Value operations to achieve "semantic soft alignment" between features. The model further calculates the nonlinear correlation between modalities, allowing the model to calculate the nonlinear correlation between "facial calmness" and "vocal anxiety," thereby identifying the deep state of "latent anxiety" and avoiding being deceived by students' surface expressions. Specifically, it includes two symmetrical cross-modal branches, namely the auditory and visual CrossTransformer branches, with the structure as follows: Figure 4 As shown.

[0060] (3-2-1) Auditory Query Visual Branch. The auditory query visual branch (A→V) consists of: this branch based on auditory features... As a query vector, using visual features The model uses multi-head attention as both a key and a value. This computational process facilitates cross-modal information interaction. Even if there's a physical time lag between an audio clip of a sigh and the visual action of a student looking down, the attention mechanism can automatically lock onto and align with the visual frame where the action occurred by calculating maximum semantic similarity, aggregating it into the current auditory representation. Furthermore, when the speech signal becomes blurred due to environmental noise, the model utilizes clear facial expressions (such as frowning) to enhance the emotional confidence of the speech features; or when the tone of voice is flat, it uses visual cues to uncover potential emotional fluctuations.

[0061] Query, key-value calculation:

[0062]

[0063] in, The weight matrix is ​​a learnable matrix. It is the scaling factor.

[0064] The formula for calculating attention is as follows:

[0065]

[0066] The intermediate features generated by this operation This represents the auditory features enhanced by visual information, that is, how visual information affects speech features at that auditory moment.

[0067] (3-2-2) Visual query auditory branch. The visual query auditory branch (V→A) is symmetrical, with visual features... Auditory features as a query vector The keys and values ​​are used as keys and values. The computation process is also carried out through a multi-head attention mechanism.

[0068] The formulas for query, key, and value calculations are as follows:

[0069]

[0070] The formula for calculating attention is as follows:

[0071]

[0072] The intermediate features generated by this operation This represents the visual features enhanced by the speech information.

[0073] (3-3) Multimodal feature generation. After completing the bidirectional cross-modal feature interaction, the enhanced visual and auditory features are fused and output.

[0074] In one embodiment, feature integration and normalization can be introduced based on the features obtained from cross-modal interactions to stabilize the feature distribution and improve the expressive power of the fused features. Using multimodal feature representations as input for subsequent temporal modeling and emotion state determination provides a unified feature foundation for the continuous analysis of multimodal emotion states in the classroom.

[0075] Step (4) Temporal modeling and emotion state determination

[0076] After completing the multimodal feature fusion, the obtained multimodal feature representations are subjected to temporal modeling and state determination processing to characterize the changes in students' emotional states over time during classroom teaching, such as... Figure 5 As shown.

[0077] (4-1) Temporal modeling of multimodal features. In one embodiment, based on the multimodal feature representation obtained in step (3), a temporal feature analysis model is constructed to perform temporal correlation analysis on the multimodal feature sequence within a continuous time window during classroom teaching.

[0078] Specifically, multimodal feature representations are arranged into a feature sequence according to time order. A temporal modeling structure is used to model the correlation between different time points, enabling the determination of the emotional state at the current time point to incorporate contextual information from preceding and following time windows, thereby reducing the impact of instantaneous abnormal fluctuations on the analysis results. Through the above temporal modeling process, a temporal feature representation containing temporal contextual information is obtained, which reflects the evolution of students' emotional states during the classroom process. The specific process is as follows:

[0079] Will and The input is fed into a standalone self-attention Transformer encoder, which can understand the current emotional state within the context of the entire time window (5 seconds or longer), thereby distinguishing between "brief frown" and "persistent confusion," and strengthening long-term temporal contextual dependencies. The calculation formula for the self-attention encoding process is as follows:

[0080]

[0081] After multiple stacking steps, the feature vector of the last time step in the sequence is finally extracted. and This serves as the final modal representation for that time window.

[0082] (4-2) Output of emotional state representation. In one embodiment, the temporal feature representation is processed by state mapping to output the emotional state representation results corresponding to each time node.

[0083] The emotional state representation results are used to describe the characteristics of students' emotional states at corresponding time points. These representations can take the form of state vectors or state labels, reflecting the relative differences between different emotional states. The emotional state representation results do not rely on feature judgments at a single time point, but rather comprehensively consider feature information across multiple time windows, thereby improving the stability of emotional state determination. The specific process is as follows:

[0084] The high-level semantic vectors output from the two branches mentioned above and Channel-level stitching is performed to achieve fusion, and the final fused feature is generated through a projection layer. The calculation formula is as follows:

[0085]

[0086] To enhance the robustness of the features, a calculation formula for the projection block, which includes Dropout and residual structure, is introduced as follows:

[0087]

[0088] The final output fused feature is calculated using the following formula:

[0089]

[0090] The classification output is performed using a fully connected layer, and the calculation formula is as follows:

[0091]

[0092] Ultimately, what was obtained It is the non-normalized probability score of various emotions (such as focus, anxiety, confusion, etc.) within the current time window, which is input into the Softmax layer for classification.

[0093] (4-3) Identity association based on physical order. In one embodiment, based on the characteristic that the multimodal data sequence in step (1) follows the physical order of classroom seats during the generation process, the emotional state representation results output in step (4-2) are processed for identity association.

[0094] Specifically, based on the seating order index followed during the construction of the multimodal feature sequence, the emotional state representation results at the corresponding time nodes are associated with specific student individuals one by one, and time identifiers are added to the emotional state representation results to form emotional state time series data corresponding to each student.

[0095] By utilizing the stable sequence relationships inherent in the physical layout of the classroom, a rapid and stable mapping between emotional state assessment results and student identity can be achieved, avoiding computational overhead caused by additional identity recognition or target tracking operations.

[0096] Step (5) Sentiment Trend Assessment and Early Warning

[0097] After completing the temporal modeling and determination of emotional states, the trend of emotional state changes for each student during classroom teaching is assessed, and corresponding early warning information is generated when abnormal changes are detected. The trend of emotional state changes is illustrated in the figure below. Figure 6 As shown.

[0098] (5-1) Calculation of emotional change characteristics. In one embodiment, based on the emotional state representation results output in step (4), the emotional state changes of each student at consecutive time points are quantitatively analyzed.

[0099] Specifically, for each time point, an intensity index is extracted from the emotional state representation results to describe the degree of the emotional state. Based on the intensity indices of adjacent time points, a change index reflecting the speed or magnitude of emotional change is calculated. The intensity index is used to characterize the relative level of the emotional state at the current time point, and the change index is used to characterize the fluctuation of the emotional state over time.

[0100] Through the above calculations, a sequence of emotional state change characteristics over time is formed, which can be used to reflect the dynamic evolution trend of students' emotional state in the classroom teaching process.

[0101] (5-1-1) Emotional Intensity Value Definition. Emotional intensity value. For the current moment The sum of probabilities of dominant negative emotions (such as anxiety, confusion, fatigue). If the current emotion is positive (such as focus) or neutral, the emotion intensity value will be set to 0 or a low-weight value, calculated as follows:

[0102]

[0103] (anxiety), (Puzzled), (Fatigue) refers to a point in time. When the dominant emotion is positive or neutral, the model outputs the probability of the dominant negative emotion. Set to 0 or a lower value.

[0104] (5-1-2) Fluctuation Amplitude Value Definition. Fluctuation amplitude value The absolute difference between the current emotional intensity value and the emotional intensity value at the previous moment is used to quantify the intensity of emotional change. The calculation formula is as follows:

[0105]

[0106] It is the emotional intensity value at the current moment. It represents the emotional intensity value from the previous moment. Specifically, as follows... Figure 7 As shown.

[0107] (5-2) Emotional trend analysis and early warning determination. In one embodiment, the characteristic sequence of emotional state changes is compared with preset early warning determination conditions to determine whether there are any emotional state changes that require attention.

[0108] When the intensity or change indicators of emotional state meet the corresponding early warning criteria, a corresponding early warning message is generated. This message indicates the degree of abnormality in the change of emotional state, and its warning level, criteria, and parameter settings can be configured according to specific application scenarios. Continuous analysis of emotional state change trends avoids misjudgments caused by relying solely on a single time point, thereby improving the stability and reliability of emotional state monitoring results.

[0109] Based on the emotional intensity value and the fluctuation amplitude value, this embodiment sets two levels of judgment thresholds for graded early warning. and fluctuation range value Students' emotional states are categorized into the following three states:

[0110] Table 1. Criteria for Determining Early Warning Levels

[0111]

[0112] The calculated emotional intensity Fluctuation range The warning level (Level) triplet is stored in the historical archive, and a visualization engine is used to draw an individual's emotional electrocardiogram. The chart visually presents the student's emotional fluctuations as shown in Table 2.

[0113] Table 2 Student Emotion Assessment Form

[0114]

[0115] Step (6) Result Generation and Report Output

[0116] After completing the emotional trend assessment and early warning determination, the emotional state analysis results formed during the classroom teaching process are summarized and output to form a results report that can be viewed and analyzed later.

[0117] (6-1) Results Data Integration. In one embodiment, based on the emotional state representation results obtained in step (4) and the early warning information generated in step (5), the emotional state representation results and early warning information of each student in the classroom teaching process are summarized and organized.

[0118] Specifically, the emotional state representations, emotional change characteristics, and early warning information at different time points are structured to form a dataset reflecting the process of students' emotional state changes. This dataset can then be organized by individual student, chronological order, or classroom stage for easier analysis and presentation.

[0119] (6-2) Generation of Emotional State Analysis Report. In one embodiment, the result data set is analyzed and formatted to generate an emotional state analysis report describing the changes in students' emotional states during classroom teaching.

[0120] Emotional state analysis reports can be presented in text, chart, or a combination of both. They are used to demonstrate the overall distribution characteristics of students' emotional states, trends over time, and corresponding warnings. Figure 7 As shown, the above methods enable an intuitive representation of the monitoring results of multimodal emotional states in the classroom, providing technical support for the process analysis of classroom emotional states.

[0121] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A method for monitoring and early warning of multimodal emotional states in the classroom based on spatiotemporal alignment, characterized in that... The method includes the following steps: (1) Classroom multimodal data acquisition and spatiotemporal alignment; In the classroom teaching scenario, students' video data V0 and audio data A0 are collected synchronously, and processed by intermittent time windows to form video data sequence V1 and audio data sequence A1 respectively; Based on the visual region of interest and the audio acquisition direction, further processing is performed to obtain video data sequence V2 and audio data sequence A2, and the temporal and spatial dimensions are aligned to generate multimodal data sequences corresponding to specific students; (2) Dual-stream multimodal feature extraction; Based on the multimodal data sequence, the video data sequence V2 and the audio data sequence A2 are subjected to independent feature analysis processing respectively. Visual feature extraction processing is used to generate a visual feature sequence representing changes in the student's facial state, and auditory feature extraction processing is used to generate an auditory feature sequence representing changes in speech state, thereby forming a dual-stream feature representation; (3) Bidirectional cross-modal feature fusion: Joint modeling of visual feature sequences and auditory feature sequences at the feature level, and through bidirectional cross-modal feature fusion processing, visual features and auditory features guide each other and perform semantic alignment and enhancement, generating a multimodal feature representation containing visual enhancement features and auditory enhancement features; (4) Temporal modeling and emotion state determination: Based on multimodal feature representation, temporal correlation analysis is performed on the multimodal feature sequence in the classroom teaching process, and the emotion state representation results corresponding to different time nodes are output; (5) Sentiment trend assessment and early warning; Based on the emotional state representation results at continuous time points, the intensity and fluctuation characteristics of emotional state changes over time are evaluated, and the trend of emotional state change is constructed. When the intensity or fluctuation characteristics meet the preset conditions, corresponding early warning information is generated. (6) Results generation and report output; The results of emotional state representation and early warning information are integrated and structured to generate an emotional state analysis report that describes the trend of changes in students' emotional state and early warning situations during classroom teaching.

2. The classroom multimodal emotional state monitoring and early warning method based on spatiotemporal alignment according to claim 1, characterized in that... The specific process of classroom multimodal data acquisition and spatiotemporal alignment described in step (1) includes: (1-1) Multimodal data synchronous acquisition: Through the acquisition terminal deployed in the classroom environment, the video data V0 and audio data A0 of students in the classroom scene are acquired synchronously, and the video data V0 and audio data A0 are sampled and processed by intermittent time window method, forming video data sequence V1 and audio data sequence A1 respectively; (1-2) Based on the physical space visual data attribution limitation, for video data sequence V1, a fixed visual interest region corresponding to each student's seat position is pre-defined in the video capture screen. The video screen is cropped according to the visual interest region to obtain video data sequence V2 corresponding to each student's seat position. (1-3) Based on the spatial directivity of audio data attribution, for audio data sequence A1, the beamforming and sound source localization technology of microphone array is used to construct virtual pickup beams pointing to the physical coordinates of each seat. This technology is equivalent to forming multiple directional audio in space, spatially filtering and separating the independent speech stream of a specific seat from the mixed background sound field, filtering out silence and environmental noise through speech activity detection, and obtaining audio data sequence A2 corresponding to each seat; (1-4) Audiovisual spatiotemporal alignment and identity association: Video data sequence V2 and audio data sequence A2 from the same seat coordinates, i.e. the same student, are bound to each other. Based on the unified timestamp at the time of collection, the video data sequence V2 and audio data sequence A2 are aligned to generate a high-quality multimodal data sequence that corresponds one-to-one with a specific student.

3. The classroom multimodal emotional state monitoring and early warning method based on spatiotemporal alignment according to claim 1, characterized in that... The specific process of dual-stream multimodal feature extraction in step (2) includes: (2-1) Visual feature extraction: The video data sequence V2 that has been spatiotemporally aligned is subjected to feature analysis and processing. The student's facial region is encoded by a visual feature extraction network, and the original video data is converted into a visual feature sequence to characterize the changes in the student's facial state. (2-2) Auditory feature extraction: The spatiotemporally aligned audio data sequence A2 is subjected to feature analysis and processing. The speech signal is encoded by the auditory feature extraction network, and the original audio data is converted into an auditory feature sequence to characterize the changes in speech state.

4. The classroom multimodal emotional state monitoring and early warning method based on spatiotemporal alignment according to claim 1, characterized in that... The specific process of bidirectional cross-modal feature fusion in step (3) includes: (3-1) Unified processing of feature dimensions: In response to the differences in feature dimensions and representation space between the visual feature sequence and the auditory feature sequence in step (2), feature mapping processing is performed on the visual feature sequence and the auditory feature sequence respectively, so that the feature sequences of different modalities are mapped to a unified feature representation space. (3-2) Bidirectional cross-modal association modeling: In a unified feature representation space, based on the interrelationship between visual features and auditory features, bidirectional cross-modal feature fusion processing is performed, so that visual features introduce auditory feature information during the fusion process, and auditory features introduce visual feature information during the fusion process, so as to achieve mutual enhancement of different modal features at the feature level. (3-3) Multimodal feature generation: The visual enhancement features and auditory enhancement features after bidirectional cross-modal feature fusion are combined to generate a multimodal feature representation that contains both visual and auditory information, which is used for subsequent temporal modeling and emotion state determination.

5. The classroom multimodal emotional state monitoring and early warning method based on spatiotemporal alignment according to claim 1, characterized in that... The specific process of temporal modeling and emotion state determination in step (4) includes: (4-1) Temporal modeling of multimodal features: Based on the multimodal feature representation in step (3), temporal correlation analysis is performed on the multimodal feature sequence in the classroom teaching process, the dependency relationship between different time nodes is modeled, and a temporal feature representation containing temporal context information is generated; (4-2) Output of emotional state representation: The temporal feature representation is processed by state mapping, and the emotional state representation results corresponding to each time node are output to reflect the changes in students' emotional state during classroom teaching. (4-3) Identity association based on physical order: Based on the characteristics of the physical order of classroom seats in the multimodal data sequence in step (1) during the generation process, the emotional state representation results are associated with the corresponding students by index mapping, and time identifiers are added to the emotional state representation results to form an individual emotional state time series.

6. The method for monitoring and early warning of multimodal emotional states in the classroom based on spatiotemporal alignment according to claim 1, characterized in that... The specific process of emotion trend assessment and early warning described in step (5) includes: (5-1) Calculation of emotional change characteristics: Based on the emotional state representation results in step (4), determine the emotional state intensity index corresponding to each time node, and calculate the emotional state change amplitude index based on the emotional state intensity index of adjacent time nodes, which is used to characterize the dynamic characteristics of emotional state changes over time. (5-2) Graded early warning judgment: The intensity index and change range index of emotional state are compared with the preset early warning judgment conditions. Based on the comparison results, the emotional state is divided into different early warning levels, and corresponding early warning information is generated when the corresponding early warning level conditions are met.

7. The method for monitoring and early warning of multimodal emotional states in the classroom based on spatiotemporal alignment according to claim 1, characterized in that... The specific process of generating and reporting the results in step (6) includes: (6-1) Results data integration: Based on the emotional state representation results in step (4) and the early warning information in step (5), the emotional state representation results and early warning information formed in the classroom teaching process are summarized and structured to generate a set of results data reflecting changes in students' emotional state and early warning situations. (6-2) Emotional state report generation: The result data set is analyzed and formatted to generate an emotional state analysis report presented in natural language or visualization form, which is used to describe the overall distribution characteristics, changing trends and corresponding early warning situations of students' emotional states during classroom teaching.

8. A classroom multimodal emotional state monitoring and early warning system based on spatiotemporal alignment, characterized in that, The system includes: The multimodal data acquisition and alignment module is used to simultaneously acquire students' video data V0 and audio data A0 in classroom teaching scenarios. It processes the data through intermittent time windows to form video data sequence V1 and audio data sequence A1. Based on the visual region of interest and the audio acquisition direction, it further processes the data to obtain video data sequence V2 and audio data sequence A2. It then performs alignment processing in the time and spatial dimensions to generate a multimodal data sequence that corresponds one-to-one with a specific student. The dual-stream feature extraction module is used to perform feature analysis processing on the video data sequence V2 and the audio data sequence A2 in the multimodal data sequence, respectively, converting the original video data into a visual feature sequence representing changes in the student's facial state, and converting the original audio data into an auditory feature sequence representing changes in speech state. The bidirectional cross-modal feature fusion module is used to perform joint modeling of visual feature sequences and auditory feature sequences at the feature level. Through bidirectional cross-modal feature fusion processing, visual features and auditory features can introduce each other's information to generate a multimodal feature representation that includes visual enhancement features and auditory enhancement features. The temporal modeling and emotion state determination module performs temporal correlation analysis on the time series features in the classroom teaching process based on multimodal feature representation, and outputs the emotion state representation results corresponding to different time nodes; The emotion trend assessment and early warning module is used to calculate the intensity and magnitude of changes in emotion state based on the emotion state representation results, and generate corresponding early warning information according to preset early warning judgment conditions. The results generation and report output module is used to integrate and structure the emotional state representation results and early warning information to generate an emotional state analysis report that describes the trend of changes in students' emotional states and early warning situations during classroom teaching.

Images