A video semantic emotion recognition system based on educational scenarios

Abstract

The present invention discloses a video semantic emotion recognition system based on an educational scenario, and the present invention relates to the technical field of educational systems. The video semantic emotion recognition system based on an educational scenario includes: a data preprocessing and enhancement module, a face and posture processing module, a feature extraction and classification module, a bimodal fusion and recognition module, a state and time domain analysis module, and a class quality evaluation module; through comprehensive preprocessing and enhancement, the quality of the video frame is improved, laying a good foundation for subsequent analysis. The face and posture processing module can cope with complex situations such as head turning and occlusion to ensure data integrity. The bimodal fusion features, combined with deep neural network time series modeling, can more accurately identify emotions. The state and time domain analysis module innovatively establishes a database containing time domain information, which can not only analyze students' activity and concentration, but also analyze changes in emotional state in combination with time domain factors.

Description

Technical Field

[0001] The present invention relates to the technical field of educational systems, and in particular to a video semantic emotion recognition system based on educational scenarios. Background Art

[0002] Semantic sentiment recognition in educational settings uses technologies such as natural language processing, computer vision, and machine learning to analyze the sentiment of multimodal data, including text, speech, and video, generated during the teaching process. This technology aims to identify semantic and emotional information, including the emotional states, attitudes, and satisfaction levels of participants, such as students and teachers. This technology is crucial for improving teaching quality, optimizing teaching strategies, and promoting student mental health.

[0003] Publication No. CN118551282A discloses a student emotion recognition method for online teaching. This method addresses the scarcity of existing student emotion databases and the limited availability of multimodal emotion recognition methods. It establishes a student emotion database and multimodal emotion analysis method. It first focuses on specific online learning scenarios and establishes a corresponding database. It then leverages text and image data sources to construct a comprehensive emotion recognition model. Specifically, it utilizes a dual-branch feature fusion of video (student class video clips) and text (student classroom comments) to recognize student classroom expressions. Training and optimization are performed using a previously established database to achieve in-depth analysis and understanding of student emotions. The model leverages multimodal information from video and text to better grasp student emotions, meeting the actual requirements of human emotion analysis and increasing the reliability of the model's emotion classification.

[0004] However, as shown in the above technology, the evaluation of teaching quality in existing classroom teaching, especially the recognition of students' emotions, requires a comprehensive evaluation of students' postures, facial expressions and other aspects, but it will be affected by many factors. For example, it is difficult to accurately recognize students' faces due to occlusion, turning of the head or light, and thus it is difficult to obtain accurate data for emotional judgment; and the existing technology only simply performs current emotional judgment, and whether the teacher can grasp the rhythm of the class is also an important part of teaching quality. Students may have different performances at different times. Only by analyzing all of them can the teaching quality be evaluated more fairly. Summary of the Invention

[0005] In response to the deficiencies of the existing technology, the present invention provides a video semantic emotion recognition system based on educational scenarios, which solves the problems still existing in the above-mentioned existing technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a video semantic emotion recognition system based on educational scenarios, comprising:

[0007] Data preprocessing and enhancement module: denoises classroom video frames, balances lighting, handles occlusions, and performs rotation operations to enhance data;

[0008] Face and posture processing module: uses algorithms to detect faces, track and locate key points, detect head posture, and further process incomplete facial data caused by head turning or occlusion;

[0009] Feature extraction and classification module: extracts features from the face and recognizes and classifies expressions;

[0010] Bimodal fusion and recognition module: This module integrates students’ facial expression features and head posture features, and uses deep neural networks for time series modeling and recognition;

[0011] Status and time domain analysis module: Analyzes students' activity and concentration, and combines time domain factors to analyze the rate of change of students' emotional state in different time periods within a day and each class;

[0012] Class quality evaluation module: Comprehensively evaluates class quality based on student activity, concentration, rate of change of students' emotional state and teacher's teaching voice quality.

[0013] Preferably, the operations of the data preprocessing and enhancement module specifically include:

[0014] Denoising: Use Gaussian filtering or median filtering algorithms to denoise classroom video frames to reduce the impact of classroom ambient noise on subsequent processing;

[0015] Lighting balancing: Using histogram equalization or adaptive histogram equalization algorithm to enhance the contrast of classroom video frames and make the classroom lighting in the image uniform;

[0016] Histogram equalization principle: by calculating the histogram of the video frame, the pixel values ​​are redistributed to make the pixel values ​​evenly distributed;

[0017] Occlusion processing: Use image restoration algorithms to repair the occluded parts of students in classroom videos and restore the occluded facial or body areas;

[0018] Data enhancement: Rotate, scale, crop, and flip classroom video frames to increase data diversity and adapt to student recognition at different angles and distances.

[0019] Preferably, the steps of performing facial tracking and key point positioning by the face and posture processing module include:

[0020] Use the MTCNN or RetinaFace algorithm to detect student faces in classroom videos and output the bounding box of the student's face. Then use the 68-point or 98-point facial landmark detection model in the Dlib library to locate facial landmarks and accurately capture facial features including eyes, mouth, and eyebrows.

[0021] The bounding box is represented as (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the detection box, w and h are the width and height of the detection box respectively; the key point is represented as (x i ,y i ), where i is the index of the key point;

[0022] Optical flow face tracking:

[0023] After detecting the student's face and locating the key points, the optical flow equation is used to calculate the motion vector of the facial key points between consecutive video frames; let I(x, y, t) represent the pixel intensity at the image position (x, y) at time t; for the facial feature point P(x i ,y i , t), in the next frame t+1, the position becomes P(x i +dx i ,y i +dy i , t+1), where (dx i ,dy i ) is the motion vector of the feature point P, and I(x i ,y i , t)≈I(x i +dx i ,y i +dy i , t+1);

[0024] The optical flow constraint equation is used to ensure the brightness consistency of facial feature points between consecutive frames. Let u=(u i , v i ) is the feature point P i The optical flow vector, where u i and v i Represent the velocity components in the x and y directions respectively; ∇I i =(Ix i , Iy i ) represents the feature point P i The gradient of the image at i Indicates P i The rate of change of image intensity over time; the optical flow constraint equation ∇I i ·u i +İt i =0 means that at the facial feature point P iAt this point, the dot product of the image gradient and the optical flow vector should be equal to the negative of the rate of change of the image intensity over time;

[0025] By calculating the motion vectors of pixel points in the video frame, continuously track the motion trajectory of the student's face; set a tracking window for each key point and set a tracking threshold to determine whether the tracking is effective; the tracking threshold includes a displacement threshold and a speed threshold, and set the displacement threshold to D t , displacement , if d is greater than the set displacement threshold D t , then it is judged that the tracking fails; set the speed threshold to V t , the time interval from the i-th frame to the i + 1-th frame is Δt, then the speed of the key point , if v is greater than the set speed threshold V t , then it is judged that the tracking is abnormal or may fail.

[0026] Preferably, occlusion and head-turning detection:

[0027] Occlusion detection: During the tracking process, regularly check the visibility of key points within the face bounding box; set a key point visibility threshold, when the number of key points is below the threshold, when the key points are occluded or disappear, that is, trigger occlusion detection, and judge whether the key points are visible by calculating the gradient or color difference of the pixels around the key points; when occlusion is detected, pause the optical flow method tracking, and instead use the face re-detection algorithm to re-search for the face within a few frames after occlusion; once the face is re-detected, immediately resume the optical flow method tracking and update the key point information;

[0028] Key point visibility threshold: Let the total number of key points be N, the number of visible key points be n^, and the visibility threshold be set to T^. When n^ < T^, trigger occlusion detection;

[0029] Key point visibility judgment: For each key point, calculate the gradient or color difference of the surrounding pixels:

[0030] Gradient calculation:

[0031] ;

[0032] ;

[0033] where I(x, y) is the pixel value of the image at the position (x, y), and k is the scaling factor of the Sobel operator;

[0034] Gradient magnitude calculation: The gradient magnitude is the length of the gradient vector, reflecting the overall change intensity of the image at a certain point;

[0035] ;

[0036] Threshold comparison: set a gradient modulus threshold T G , if G <T G , then it is considered that a key point is not visible in the current frame:

[0037] ;

[0038] Color difference calculation:

[0039] diff R =│R key -R surr │;

[0040] diff G =│G key -G surr │;

[0041] diff B =│B key -B surr │;

[0042] Among them, (R key , G key , B key ) is the RGB value of the key point, (R surr , G surr , B surr ) are the RGB values ​​of the surrounding pixels;

[0043] Use Euclidean distance to combine the color differences of the three RGB channels into one value:

[0044] ;

[0045] Or use weighted sum to combine the color differences of the three RGB channels into one value:

[0046] diff 总 =α⋅diff R +β⋅diff G +γ⋅diff B ;

[0047] Among them, α, β, and γ are weight coefficients, which can be adjusted as needed;

[0048] Threshold comparison: set a color difference threshold T diff , if diff 总 <T diff , then the key point is considered invisible in the current frame, that is:

[0049] ;

[0050] Head-turn detection: By analyzing the angular changes of facial key points, we can determine whether the student has turned their head. We set a threshold for the angular change of key points. If the angular change of a key point exceeds the preset threshold, it is considered a head turn. The angle change between the two key point vectors is calculated using the vector angle formula. When a head turn is detected, the optical flow tracking method is also suspended, and the head pose estimation algorithm is used to try to predict the possible position of the student's head. Based on this, the facial re-detection algorithm is used to search for faces near the predicted position. Once the face is re-detected, tracking is resumed and the key point information is updated.

[0051] Assume that the facial key point set is H = (H1, H2, ..., H m ), where H j =(x j ,y j ) represents the coordinates of the jth key point, and j is any natural number from 1 to m. To detect head turning, two key points are selected to form a vector, and the angle change between the two vectors is calculated;

[0052] Assume that the vectors of the two key points are and , where (x1, y1) and (x2, y2) are the keypoint coordinates in the initial frame, and (x1′, y1′) and (x2′, y2′) are the keypoint coordinates in the current frame;

[0053] Define an angle change threshold θ′. If the angle θ between the two vectors exceeds the threshold θ′, it is considered a head turn.

[0054] The angle θ between vectors v1 and v2 is calculated using the following formula:

[0055] ;

[0056] Among them, the dot product: ;

[0057] vector modulus;

[0058] ;

[0059] .

[0060] Preferably, the operation of extracting facial features by the feature extraction and classification module specifically includes:

[0061] S1 uses a multi-angle face detector: select a detector that supports multi-angle face detection, set the detected face bounding box list B = (B1, B2, ..., B s ), where B z =(x z ,y z , w z, h z ) represents the bounding box coordinates of the z-th face, where z is any natural number from 1 to s;

[0062] S2 preliminary alignment

[0063] Calculate the rotation angle: Based on the detected face bounding box, calculate the rotation angle of the face; by analyzing the aspect ratio of the bounding box And the preset frontal face aspect ratio r0 to estimate the rotation angle Q:

[0064] ;

[0065] Where p is the adjustment coefficient used to calibrate the calculation of the rotation angle;

[0066] Image rotation: Input image I and rotation angle Q, perform rotation correction on the face image so that the face image is roughly facing the front, and output the rotated image I′;

[0067] S3 keypoint detection

[0068] Use key point detection algorithm: On the face image after rotation correction, use the key point detection algorithm to detect the key points of the face, and output the key point set A = (A1, A2, ..., A E ), where A e =(x e ,y e ) represents the coordinates of the e-th key point, and e is any natural number from 1 to E;

[0069] S4 Fine Alignment

[0070] Calculate the transformation matrix: Based on the filtered key points, use affine transformation or similarity transformation to calculate the transformation matrix from the detected key points F to the standard key point set F0. The standard key point set is a set of key point templates of the frontal face.

[0071] Apply transformation: Apply the calculated transformation matrix to the face image to achieve fine alignment;

[0072] S5 post-processing and optimization

[0073] Interpolation and smoothing: After alignment, the image is interpolated or smoothed to remove distortion or noise generated during the transformation process;

[0074] Cropping and scaling: Based on the aligned image, crop the face area and scale it as needed for subsequent processing or analysis.

[0075] Preferably, the step of performing facial expression classification by the feature extraction and classification module specifically includes:

[0076] A1. Database Construction: Collect and annotate a large number of facial expression images, covering common expressions of happiness, sadness, surprise, anger, fear, and disgust; let X be the original image dataset, where X f represents the fth image, Y is the corresponding expression label set, Y f Represents X f expression labels; use the deep learning model to train these images and extract deep features; let F(X f ) is from X f The extracted feature vector has a dimension of g; the features and the corresponding expression labels are stored in the database, that is, (F(X f ), Y f ) for each image; perform face detection and alignment, let O be the transformation matrix, X aligned is the aligned image, then: X aligned =O(X);

[0077] A2. Real-time image acquisition and preprocessing: Use a camera or image input device to collect students' facial expression images in real time. real The image is collected in real time; the collected image is preprocessed, including grayscale, denoising, face detection and alignment, to obtain X real, aligned ;

[0078] A3 image comparison analysis:

[0079] A3.1 Feature vector matching: The depth feature vector F(X real， aligned ) or the original image is compared with the feature vectors in the database; the similarity between the feature vectors is calculated using the cosine similarity or Euclidean distance measurement method, the feature vectors with the highest similarity are selected, and the corresponding expression labels are checked;

[0080] A3.2 Expression Classification: Based on the matched feature vectors and their expression labels, the final expression category is determined using majority voting, weighted averaging, or other strategies.

[0081] Preferably, the working steps of the dual-modal fusion and recognition module specifically include:

[0082] Feature fusion: The facial expression features and head posture features of students are spliced ​​or weighted fused to form comprehensive features; the weighted fusion formula is:

[0083] ;

[0084] in, and ω are weight coefficients, J face Represents facial expression features, J poserepresents the head posture feature, J represents the comprehensive feature after fusion;

[0085] Deep recognition: Use LSTM, GRU or Transformer deep neural networks to perform time series modeling and deep recognition of comprehensive features to identify the changing trends of students' emotional states; deep neural networks extract time series features through multi-layer networks and output student emotion recognition results.

[0086] Preferably, the steps of performing status analysis by the status and time domain analysis module specifically include:

[0087] Data preparation: Collect students’ facial expressions and posture data from classroom videos and divide them into small segments with a time interval of Δl;

[0088] Activity calculation: define the positive emotion set E pos ={happy, excited, ...};

[0089] For each time segment l, count the number of times positive emotions appear n pos (l);

[0090] Calculate the activity index R:

[0091] ;

[0092] Where L is the total number of time segments, W pos (l) is the average duration of positive emotions;

[0093] Concentration calculation: defining the concentration emotion set E focus ={serious, thinking, ...}, calculate the concentration index Z:

[0094] ;

[0095] Among them, n focus (l) is the number of times the focused emotion appears, W focus (l) is the average duration of focused emotion.

[0096] Preferably, the step of analyzing the change of emotional state by combining the state and time domain analysis module with the time domain factor specifically includes:

[0097] Analysis of different time periods within a day: Divide a day into M time periods, and count the frequency of each emotion category in each time period and draw a histogram;

[0098] Analysis of different time periods of each class: Divide each class into multiple stages: beginning, middle, and end, and calculate the rate of change of emotional state in each stage, that is, the rate of increase or decrease in the frequency of each emotional category in the next stage relative to the previous stage.

[0099] Preferably, the work content of the class quality evaluation module specifically includes:

[0100] Recognize the teacher's teaching voice in the classroom video, use big language technology to identify semantics, and obtain the evaluation value PJ through manual evaluation;

[0101] Combining information on student activity, concentration, the rate of change of emotional state combined with time domain factors, and evaluation values, a weighted average or fuzzy comprehensive evaluation method is used to comprehensively evaluate the quality of the class. The comprehensive evaluation formula is:

[0102] ;

[0103] in, is the weight coefficient.

[0104] The present invention provides a video semantic emotion recognition system based on educational scenarios. Compared with the existing technology, it has the following advantages:

[0105] 1. This educational video semantic emotion recognition system improves video frame quality through comprehensive preprocessing and enhancement, laying a solid foundation for subsequent analysis. The facial and posture processing module can handle complex situations such as head turning and occlusion to ensure data integrity. Bimodal fusion features, combined with deep neural network time series modeling, enable more accurate emotion recognition. The state and time domain analysis module innovatively establishes a database containing time domain information, which not only analyzes student activity and concentration, but also analyzes changes in emotional state by combining time domain factors. The class quality evaluation module integrates multi-dimensional information to make evaluation more comprehensive and objective, providing a more scientific and effective means for teaching quality assessment and student status analysis.

[0106] 2. This educational video semantic emotion recognition system uses optical flow facial tracking to achieve stable and accurate tracking of students' faces by calculating motion vectors, combined with tracking windows and threshold settings. This allows for continuous tracking even with slight student movement, effectively addressing the tracking instability issue encountered in existing technologies. The occlusion detection mechanism periodically checks keypoint visibility, using gradient or color differences to determine keypoint visibility. Upon detecting occlusion, it pauses optical flow tracking and restarts the face search, effectively addressing tracking failures caused by occlusion and improving tracking robustness. Head turn detection analyzes the angular changes of facial keypoints and calculates the angle using a vector angle formula. Upon detecting a head turn, it similarly pauses tracking, predicts the head position, and restarts the face search, ensuring continuous tracking even in the presence of head turns. These measures not only enhance the stability and accuracy of student facial tracking but also enable effective acquisition of facial information even in complex scenarios, such as when students are obscured or their heads are turned, providing more reliable data support for subsequent applications such as emotion recognition.

[0107] 3. This video semantic emotion recognition system for educational scenarios uses a multi-angle face detector to accurately detect faces from different angles. Preliminary alignment corrects facial orientation by calculating rotation angles. Keypoint detection precisely locates key facial regions. Fine alignment further eliminates bias. Post-processing and optimization enhance image quality, laying a solid foundation for subsequent analysis. For facial expression classification, a database is constructed by collecting and annotating a large number of expression images and training them using a deep learning model to extract highly discriminative deep features, providing a rich and reliable reference for classification. Real-time image acquisition and preprocessing ensure clear and standardized facial images. For image comparison and analysis, feature vector matching utilizes appropriate metrics to accurately identify similar feature vectors, and expression classification employs a rational strategy to determine the final category. Overall, this module accurately extracts facial features and classifies expressions, providing strong support for applications such as emotion analysis and teaching evaluation in educational settings, helping to improve teaching quality and effectiveness.

[0108] 4. This educational video semantic emotion recognition system collects classroom video data and divides it into time segments to accurately count the number of occurrences and average duration of positive and focused emotions, calculate activity and concentration indexes, and objectively quantify students' classroom status. Combined with temporal factor analysis, the system divides a day into multiple time periods and plots emotion frequencies in a histogram, visualizing the distribution of emotions throughout the day. By dividing each class into different phases and calculating the rate of change in emotion, it provides a clear understanding of emotional dynamics throughout the class. This solution not only comprehensively and meticulously analyzes students' emotional states but also deeply incorporates temporal factors, providing educators with a richer and more accurate basis for understanding student classroom performance and optimizing teaching strategies, thereby improving teaching quality. By integrating multiple aspects of information, such as student activity and concentration, and employing methods such as weighted averaging, it can more accurately and objectively reflect the actual classroom situation, providing a strong basis for teaching improvements and helping to enhance teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 is a system module block diagram of the present invention;

[0110] Figure 2 It is a schematic diagram of the overall steps of the present invention. DETAILED DESCRIPTION

[0111] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0112] See Figure 1-Figure 2 , the present invention provides the following four technical solutions:

[0113] The first embodiment: a video semantic emotion recognition system based on an educational scenario, comprising:

[0114] Data preprocessing and enhancement module: denoises classroom video frames, balances lighting, handles occlusions, and performs rotation operations to enhance data;

[0115] Face and posture processing module: uses algorithms to detect faces, track and locate key points, detect head posture, and further process incomplete facial data caused by head turning or occlusion;

[0116] Feature extraction and classification module: extracts features from the face and recognizes and classifies expressions;

[0117] Bimodal fusion and recognition module: This module integrates students’ facial expression features and head posture features, and uses deep neural networks for time series modeling and recognition;

[0118] Status and time domain analysis module: Analyzes students' activity and concentration, and combines time domain factors to analyze the rate of change of students' emotional state in different time periods within a day and each class;

[0119] Class quality evaluation module: Comprehensively evaluates class quality based on student activity, concentration, rate of change of students' emotional state and teacher's teaching voice quality.

[0120] Comprehensive preprocessing and enhancement improves video frame quality, laying a solid foundation for subsequent analysis. The facial and posture processing module handles complex situations like head turning and occlusion, ensuring data integrity. Bimodal fusion features, combined with deep neural network temporal modeling, enable more accurate emotion recognition. The state and temporal domain analysis module innovatively establishes a database containing temporal information, enabling analysis not only of student activity and concentration but also of changes in emotional state by integrating temporal factors. The class quality evaluation module integrates multi-dimensional information for more comprehensive and objective evaluation, providing a more scientific and effective approach for teaching quality assessment and student status analysis.

[0121] The second embodiment differs from the first embodiment mainly in that the operations of the data preprocessing and enhancement module specifically include:

[0122] Denoising: Use Gaussian filtering or median filtering algorithms to denoise classroom video frames to reduce the impact of classroom ambient noise on subsequent processing;

[0123] Lighting balancing: Using histogram equalization or adaptive histogram equalization algorithm to enhance the contrast of classroom video frames and make the classroom lighting in the image uniform;

[0124] Histogram equalization principle: by calculating the histogram of the video frame, the pixel values ​​are redistributed to make the pixel values ​​evenly distributed;

[0125] Occlusion processing: Use image restoration algorithms (such as texture synthesis-based algorithms) to repair the occluded parts of students in classroom videos and restore the occluded facial or body areas;

[0126] Data enhancement: Rotate, scale, crop, and flip classroom video frames to increase data diversity and adapt to student recognition at different angles and distances.

[0127] In terms of data preprocessing and enhancement, Gaussian filtering, median filtering denoising, histogram equalization and other illumination equalization processing are used, combined with image restoration algorithms to handle occlusion, and data is enhanced through operations such as rotation, which effectively improves the quality of video frames and provides a reliable foundation for subsequent analysis.

[0128] In this embodiment, the steps of performing facial tracking and key point positioning by the face and posture processing module include:

[0129] Use efficient algorithms such as MTCNN or RetinaFace to detect student faces in classroom videos, output the bounding box of the student's face, and then use the 68-point or 98-point facial landmark detection model in the Dlib library to locate facial landmarks and accurately capture facial features such as eyes, mouth, and eyebrows.

[0130] The bounding box is represented as (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the detection box, w and h are the width and height of the detection box respectively; the key point is represented as (x i ,y i ), where i is the index of the key point;

[0131] Optical flow face tracking:

[0132] After detecting the student's face and locating the key points, the optical flow equation is used to calculate the motion vector of the facial key points between consecutive video frames; let I(x, y, t) represent the pixel intensity at the image position (x, y) at time t; for the facial feature point P(x i ,y i , t), in the next frame t+1, the position becomes P(x i +dx i ,y i +dy i , t+1), where (dx i ,dy i ) is the motion vector of the feature point P, and I(x i ,y i , t)≈I(x i +dxi ,y i +dy i , t+1);

[0133] The optical flow constraint equation is used to ensure the brightness consistency of facial feature points between consecutive frames. Let u=(u i , v i ) is the feature point P i The optical flow vector, where u i and v i Represent the velocity components in the x and y directions respectively; ∇I i =(Ix i , Iy i ) represents the feature point P i The gradient of the image at i Indicates P i The rate of change of image intensity over time; the optical flow constraint equation ∇I i ·u i +İt i =0 means that at the facial feature point P i At , the dot product of the image gradient and the optical flow vector should be equal to the inverse of the rate of change of the image intensity over time;

[0134] By calculating the motion vector of the pixel points in the video frame, the movement trajectory of the student's face is continuously tracked; a tracking window is set for each key point, and a tracking threshold is set to determine whether the tracking is effective; the tracking threshold includes a displacement threshold and a speed threshold, and the displacement threshold is set to D t , displacement , if d is greater than the set displacement threshold D t , it is judged as tracking failure; let the speed threshold be V t , the time interval from the i-th frame to the i+1-th frame is Δt, then the speed of the key point is , if v is greater than the set speed threshold V t , it is judged that the tracking is abnormal or may fail.

[0135] In the face and pose processing module, MTCNN or RetinaFace is used to efficiently detect faces and output bounding boxes. The Dlib library is used to precisely locate key points, accurately capturing facial features. Optical flow face tracking calculates motion vectors, combined with tracking windows and threshold settings, to achieve stable and accurate tracking of students' faces. This allows for continuous tracking even with slight movements, effectively resolving the issue of unstable tracking found in existing technologies.

[0136] In this embodiment, occlusion and head turning detection:

[0137] Occlusion Detection: During the tracking process, regularly check the visibility of key points within the facial bounding box; set a threshold for key point visibility. When the number of key points is below this threshold, i.e., when key points (such as eyes, mouth, etc.) are occluded or disappear, occlusion detection is triggered. Determine the visibility of key points by calculating the gradient or color differences of the pixels around the key points; when occlusion is detected, pause the optical flow tracking and instead use a face re-detection algorithm (such as MTCNN or RetinaFace) to re-search for the face within a few frames after occlusion; once the face is re-detected, immediately resume the optical flow tracking and update the key point information;

[0138] Threshold for Key Point Visibility: Let the total number of key points be N, the number of visible key points be n^, and the visibility threshold be set as T^. When n^ < T^, occlusion detection is triggered;

[0139] Example: If N = 68 (such as using 68-point facial landmarks of Dlib), Tvisible can be set to 30, meaning that when the number of visible key points is less than 30, the face is considered occluded;

[0140] Judgment of Key Point Visibility: For each key point, calculate the gradient or color differences of the surrounding pixels:

[0141] Gradient Calculation:

[0142] ;

[0143] ;

[0144] where I(x, y) is the pixel value of the image at position (x, y), and k is the scaling factor of the Sobel operator;

[0145] Gradient Magnitude Calculation: The gradient magnitude is the length of the gradient vector, which can reflect the overall change intensity of the image at a certain point;

[0146] ;

[0147] Threshold Comparison: Set a gradient magnitude threshold T G , if G < T G , then a certain key point is considered invisible in the current frame, i.e.:

[0148] ;

[0149] Color Difference Calculation:

[0150] diff R =│R key -R surr │;

[0151] diff G =│Gkey -G surr │;

[0152] diff B =│B key -B surr │;

[0153] Among them, (R key , G key , B key ) is the RGB value of the key point, (R surr , G surr , B surr ) is the RGB value of the surrounding pixels (which can be averaged);

[0154] Use Euclidean distance to combine the color differences of the three RGB channels into one value:

[0155] ;

[0156] Or use weighted sum to combine the color differences of the three RGB channels into one value:

[0157] diff 总 =α⋅diff R +β⋅diff G +γ⋅diff B ;

[0158] Among them, α, β, and γ are weight coefficients, which can be adjusted as needed;

[0159] Threshold comparison: set a color difference threshold T diff , if diff 总 <T diff , then the key point is considered invisible in the current frame, that is:

[0160] ;

[0161] Head-turn detection: By analyzing the angular changes of facial key points, we can determine whether the student has turned their head. We set a threshold for the angular change of key points. If the angular change of a key point exceeds the preset threshold, it is considered a head turn. The angle change between the two key point vectors is calculated using the vector angle formula. When a head turn is detected, the optical flow tracking method is also suspended, and the head pose estimation algorithm is used to try to predict the possible position of the student's head. Based on this, the facial re-detection algorithm is used to search for faces near the predicted position. Once the face is re-detected, tracking is resumed and the key point information is updated.

[0162] Assume that the facial key point set is H = (H1, H2, ..., H m ), where H j =(x j,y j ) represents the coordinates of the jth key point, and j is any natural number from 1 to m. To detect head turning, two key points (such as the left eye and the right eye, or the nose tip and the left ear) are selected to form a vector, and the angle change between the two vectors is calculated;

[0163] Assume that the vectors of the two key points are and , where (x1, y1) and (x2, y2) are the keypoint coordinates in the initial frame, and (x1′, y1′) and (x2′, y2′) are the keypoint coordinates in the current frame;

[0164] Define an angle change threshold θ′. If the angle θ between the two vectors exceeds the threshold, it is considered a head turn.

[0165] The angle θ between vectors v1 and v2 is calculated using the following formula:

[0166] ;

[0167] Among them, the dot product: ;

[0168] vector modulus;

[0169] ;

[0170] .

[0171] The occlusion detection mechanism periodically checks the visibility of key points, using gradient or color differences to determine whether a key point is visible. When occlusion is detected, it pauses optical flow tracking and re-searches the face, effectively resolving the problem of tracking failure caused by occlusion and improving the robustness of tracking. Head turn detection analyzes the angular changes of facial key points and calculates the angle using a vector angle formula. When a head turn is detected, it also pauses tracking and predicts the head position, then re-searches the face to ensure continuous tracking even in head-turning situations. These measures not only enhance the stability and accuracy of student facial tracking, but also effectively obtain facial information in complex scenarios, such as when students are occluded or turn their heads, providing more reliable data support for subsequent applications such as emotion recognition.

[0172] The third embodiment differs from the first embodiment mainly in that the operation of extracting facial features by the feature extraction and classification module specifically includes:

[0173] S1 uses a multi-angle face detector: select a detector that can detect multi-angle faces, set the list of detected face bounding boxes B = (B1, B2, ..., B s ), where B c =(x z ,yz , w z , h z ) represents the bounding box coordinates of the zth face (upper left corner ( x z ,y z ) and width w z , height h z ), where B z =(x z ,y z , w z , h z ) represents the bounding box coordinates of the z-th face, where z is any natural number from 1 to s;

[0174] S2 preliminary alignment (rotation correction)

[0175] Calculate the rotation angle: Based on the detected face bounding box, calculate the rotation angle of the face; by analyzing the aspect ratio of the bounding box And the preset frontal face aspect ratio r0 to estimate the rotation angle Q:

[0176] ;

[0177] Where p is the adjustment coefficient used to calibrate the calculation of the rotation angle;

[0178] Image rotation: Input image I and rotation angle Q, perform rotation correction on the face image so that the face image is roughly facing the front, and output the rotated image I′;

[0179] S3 keypoint detection

[0180] Use key point detection algorithm: On the face image after rotation correction, use key point detection algorithm (such as Dlib's 68-point detection, FAN, LAB, etc.) to detect the key points of the face (such as the corners of the eyes, the corners of the mouth, the tip of the nose, etc.), and output the key point set A = (A1, A2, ..., A E ), where A e =(x e ,y e ) represents the coordinates of the e-th key point, and e is any natural number from 1 to E;

[0181] S4 fine alignment (affine transformation or similarity transformation)

[0182] Calculate the transformation matrix: Based on the filtered key points, use affine transformation or similarity transformation to calculate the transformation matrix from the detected key points F to the standard key point set F0. The standard key point set is a set of key point templates of the frontal face.

[0183] Apply transformation: Apply the calculated transformation matrix to the face image to achieve fine alignment;

[0184] S5 post-processing and optimization

[0185] Interpolation and smoothing: After alignment, the image is interpolated or smoothed to remove distortion or noise generated during the transformation process;

[0186] Cropping and scaling: Based on the aligned image, crop the face area and scale it as needed for subsequent processing or analysis.

[0187] The multi-angle face detector can accurately detect faces at different angles, providing an accurate basis for subsequent processing. Preliminary alignment corrects the face by calculating the rotation angle so that it faces roughly the front, improving the effect of subsequent processing. Key point detection uses advanced algorithms to accurately obtain facial key points. Fine alignment uses affine or similarity transformations to further eliminate deviations and make facial features more standardized. In the post-processing and optimization stage, interpolation and smoothing are used to eliminate distortion noise, and cropping and scaling are used to ensure the appropriate facial area. The overall solution is comprehensive and detailed, effectively improving the accuracy and stability of facial feature extraction, providing more reliable and high-quality feature data for subsequent tasks such as emotion recognition, and facilitating more accurate analysis of student status.

[0188] In this embodiment, the steps of facial expression classification by the feature extraction and classification module specifically include:

[0189] A1. Database Construction: Collect and annotate a large number of facial expression images, covering common expressions of happiness, sadness, surprise, anger, fear, and disgust; let X be the original image dataset, where X f represents the fth image, Y is the corresponding expression label set, Y f Represents X f expression labels; use deep learning models (such as CNN, ResNet, etc.) to train these images and extract deep features; let F(X f ) is from X f The extracted feature vector has a dimension of g; the features and the corresponding expression labels are stored in the database, that is, (F(X f ), Y f ) are correct; these feature vectors should have high discrimination and robustness, which can usually be measured by indicators such as model accuracy and recall rate; perform face detection and alignment on each image, let O be the transformation matrix, X aligned is the aligned image, then: X aligned =O(X);

[0190] A2. Real-time image acquisition and preprocessing: Use a camera or image input device to collect students' facial expression images in real time. realThe image is collected in real time; the collected image is preprocessed, including grayscale, denoising, face detection and alignment, to obtain X real, aligned ;

[0191] A3 image comparison analysis:

[0192] A3.1 Feature vector matching: The depth feature vector F(X real， aligned ) (if extraction is performed) or the original image (if direct image comparison is performed) is compared with the feature vectors in the database; the similarity between the feature vectors is calculated using metrics such as cosine similarity and Euclidean distance, and the feature vectors with the highest similarity are selected and the corresponding expression labels are checked;

[0193] A3.2 Expression Classification: Based on the matched feature vectors and their expression labels, the final expression category is determined using majority voting, weighted averaging, or other strategies.

[0194] By collecting and annotating a large number of facial expression images and training them using deep learning models, we extract highly discriminative and robust deep features, providing a solid foundation for subsequent classification. Real-time image acquisition and preprocessing ensure clear, standardized facial images, improving classification accuracy. During image comparison and analysis, feature vector matching utilizes multiple metrics to calculate similarity and accurately identify similar feature vectors. Expression classification utilizes strategies such as majority voting, comprehensively considering matching results to achieve more scientific and rational classification. This overall solution effectively improves the accuracy and reliability of facial expression classification, enabling real-time and accurate recognition of student expressions.

[0195] In this embodiment, the working steps of the dual-modal fusion and recognition module specifically include:

[0196] Feature fusion: The facial expression features and head posture features of students are spliced ​​or weighted to form a comprehensive feature to more comprehensively describe the emotional state of the students. The weighted fusion formula is:

[0197] ;

[0198] in, and ω are weight coefficients, J face Represents facial expression features, J pose represents the head posture feature, J represents the comprehensive feature after fusion;

[0199] Deep recognition: Use LSTM, GRU or Transformer deep neural networks to perform time series modeling and deep recognition of comprehensive features to identify the changing trends of students' emotional states; deep neural networks extract time series features through multi-layer networks and output student emotion recognition results.

[0200] Feature fusion combines or weights facial expression and head posture features to form comprehensive features, providing a more comprehensive and accurate description of students' emotional states and avoiding the limitations of single features. Deep recognition utilizes deep neural networks such as LSTM, GRU, or Transformer to perform temporal modeling and deep recognition of comprehensive features, effectively capturing changing trends in emotional states and outputting accurate emotion recognition results. This combination not only improves the accuracy of emotion recognition but also dynamically reflects changes in students' emotions, providing richer and more reliable information for sentiment analysis and teaching evaluation in educational scenarios, thereby facilitating teaching improvements.

[0201] The fourth embodiment differs from the first embodiment mainly in that the steps of performing state analysis by the state and time domain analysis module specifically include:

[0202] Data preparation: Collect students’ facial expressions and posture data from classroom videos and divide them into small segments with a time interval of Δl;

[0203] Activity calculation: define the positive emotion set E pos ={happy, excited, ...};

[0204] For each time segment l, count the number of times positive emotions appear n pos (l);

[0205] Calculate the activity index R:

[0206] ;

[0207] Where L is the total number of time segments, W pos (l) is the average duration of positive emotions;

[0208] Concentration calculation: defining the concentration emotion set E focus ={serious, thinking, ...}, calculate the concentration index Z:

[0209] ;

[0210] Among them, n focus (l) is the number of times the focused emotion appears, W focus (l) is the average duration of focused emotion.

[0211] In this embodiment, the steps of analyzing the change of emotional state by combining the state and time domain analysis module with the time domain factor specifically include:

[0212] Analysis of different time periods within a day: Divide a day into M time periods, and count the frequency of each emotion category in each time period and draw a histogram;

[0213] Analysis of different time periods of each class: Divide each class into multiple stages: beginning, middle, and end, and calculate the rate of change of emotional state in each stage, that is, the rate of increase or decrease in the frequency of each emotional category in the next stage relative to the previous stage.

[0214] By collecting classroom video data and segmenting it into time segments, we accurately count the occurrences and average duration of positive and focused emotions, calculate activity and concentration indices, and objectively quantify students' classroom status. When combined with temporal factor analysis, we divide the day into multiple time periods and plot emotion frequencies in histograms, visualizing the distribution of emotions throughout the day. By dividing each class into different phases and calculating the rate of change in emotion, we can gain a clearer understanding of emotional dynamics throughout the class. This solution not only comprehensively and meticulously analyzes students' emotional states but also deeply incorporates temporal factors, providing educators with a richer and more accurate basis for understanding student classroom performance and optimizing teaching strategies, ultimately improving teaching quality.

[0215] In this embodiment, the work content of the class quality evaluation module specifically includes:

[0216] Recognize the teacher's teaching voice in the classroom video, use big language technology to identify semantics, and obtain the evaluation value PJ through manual evaluation;

[0217] Combining information on student activity, concentration, the rate of change of emotional state combined with time domain factors, and evaluation values, a weighted average or fuzzy comprehensive evaluation method is used to comprehensively evaluate the quality of the class. The comprehensive evaluation formula is:

[0218] ;

[0219] in, is the weight coefficient.

[0220] By integrating various information such as students' activity level and concentration, and adopting methods such as weighted average, we can more accurately and objectively reflect the actual situation in the classroom, provide a strong basis for teaching improvement, and help improve teaching quality.

[0221] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0222] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0223] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A video semantic emotion recognition system based on educational scenarios, characterized by: include: Data preprocessing and enhancement module: denoises classroom video frames, balances lighting, handles occlusions, and performs rotation operations to enhance data; The face and posture processing module uses the MTCNN or RetinaFace algorithm to detect student faces in classroom videos, outputting bounding boxes of the students' faces. It then uses the 68-point or 98-point facial landmark detection model in the Dlib library to locate facial landmarks, accurately capturing facial features such as the eyes, mouth, and eyebrows. After detecting the student's face and locating the landmarks, it uses optical flow equations to calculate the motion vectors of the facial landmarks between consecutive video frames. By calculating the motion vectors of the pixels in the video frames, it continuously tracks the movement of the student's face. Set a tracking window for each key point and set a tracking threshold to determine whether the tracking is effective; detect head pose and further process incomplete facial data caused by head turning or occlusion; occlusion detection: During the tracking process, regularly check the visibility of key points within the facial bounding box; Set a key point visibility threshold. When the number of key points falls below the threshold, or a key point is occluded or disappears, occlusion detection is triggered. The key point's visibility is determined by calculating the gradient or color difference of the pixels around the key point. When occlusion is detected, optical flow tracking is paused, and a face re-detection algorithm is used to search for the face within a few frames after the occlusion. Once the face is re-detected, optical flow tracking is resumed and key point information is updated. Head turn detection: By analyzing the angular changes of facial key points, we can determine whether the student has turned their head. We set a threshold for the angular change of key points. If the angular change of a key point exceeds the preset threshold, it is considered a head turn. The angle change between the two key point vectors is calculated using the vector angle formula. When a head turn is detected, the optical flow tracking method is also suspended, and the head pose estimation algorithm is used to try to predict the possible position of the student's head. The face re-detection algorithm is used to search for faces near the predicted position. Once the face is re-detected, tracking is resumed and the key point information is updated. Feature extraction and classification module: extracts features from the face and recognizes and classifies expressions; calculates the rotation angle of the face based on the detected face bounding box; Perform rotation correction on the facial image so that it faces the front. Use a key point detection algorithm to detect the key points of the face on the rotation-corrected facial image. Calculate the transformation matrix based on the key points and apply it to the fine alignment of the facial image. Bimodal fusion and recognition module: This module integrates students' facial expression features and head posture features, uses deep neural networks for time series modeling and recognition, and identifies the changing trends of students' emotional states. This module is then used to analyze the changes in students' emotional states using time-domain molecular analysis. Status and time domain analysis module: Analyzes students' activity and concentration, and combines time domain factors to analyze the rate of change of students' emotional state in different time periods within a day and each class; Class quality evaluation module: Comprehensively evaluates class quality based on student activity, concentration, rate of change of students' emotional state and teacher's teaching voice quality.

2. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The operations of the data preprocessing and enhancement module specifically include: Denoising process: Use Gaussian filtering or median filtering algorithm to denoise the classroom video frames, reducing the impact of classroom environmental noise on subsequent processing; Illumination equalization process: Adopt histogram equalization or adaptive histogram equalization algorithm to enhance the contrast of classroom video frames, making the classroom illumination in the image uniform; Principle of histogram equalization: By calculating the histogram of the video frame, redistribute the pixel values to make the pixel value distribution uniform; Occlusion handling: Use image inpainting algorithm to repair the occluded parts of students in the classroom video, restoring the occluded facial or body areas; Data augmentation: Rotate, scale, crop, and flip the classroom video frames to increase data diversity to adapt to student recognition at different angles and distances.

3. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The bounding box is represented as (x, y, w, h), where (x, y) are the coordinates of the upper left corner of the detection box, and w and h are the width and height of the detection box respectively; The key point is represented by (x i ,y i ), where i is the index of the key point; Optical flow-based face tracking: Let I(x, y, t) represent the pixel intensity at the position (x, y) of the image at time t; For facial feature points P(x i ,y i , t), in the next frame t+1, the position becomes P(x i +dx i ,y i +dy i , t+1), where (dx i ,dy i ) is the motion vector of the feature point P, and I(x i ,y i , t)≈I(x i +dx i ,y i +dy i , t+1); The optical flow constraint equation is used to ensure the brightness consistency of facial feature points between consecutive frames. Let u=(u i , v i ) is the feature point P i The optical flow vector, where u i and v i Represent the velocity components in the x and y directions respectively; ∇I i =(Ix i , Iy i ) represents the feature point P i The gradient of the image at i Indicates P i The rate of change of image intensity over time; the optical flow constraint equation ∇I i ·u i +İt i =0 means that at the facial feature point P i At , the dot product of the image gradient and the optical flow vector should be equal to the inverse of the rate of change of the image intensity over time; Tracking threshold includes displacement threshold and speed threshold. Set the displacement threshold to D t , displacement , if d is greater than the set displacement threshold D t , it is judged as tracking failure; let the speed threshold be V t , the time interval from the i-th frame to the i+1-th frame is Δt, then the speed of the key point is , if v is greater than the set speed threshold V t , it is judged that the tracking is abnormal or may fail.

4. The video semantic emotion recognition system based on educational scenarios according to claim 3 is characterized by: Visibility threshold of key points: Let the total number of key points be N, the number of visible key points be n^, and the visibility threshold be set as T^. When n^ < T^, trigger occlusion detection; Visibility judgment of key points: For each key point, calculate the gradient or color difference of the surrounding pixels: Gradient calculation: ； ； Where I(x, y) is the pixel value of the image at the position (x, y), and k is the scaling factor of the Sobel operator; Gradient magnitude calculation: The gradient magnitude is the length of the gradient vector, reflecting the overall change intensity of the image at a certain point; ； Threshold comparison: set a gradient modulus threshold T G , if G <T G , then it is considered that a key point is not visible in the current frame: ； Color difference calculation: diff R =│R key -R surr │; diff G =│G key -G surr │; diff B =│B key -B surr │; Among them, (R key , G key , B key ) is the RGB value of the key point, (R surr , G surr , B surr ) are the RGB values ​​of the surrounding pixels; Use the Euclidean distance to synthesize the color differences of the RGB three channels into one value: ； Or use the weighted sum method to synthesize the color differences of the RGB three channels into one value: diff 总 =α⋅diff R +β⋅diff G +γ⋅diff B ; Where α, β, γ are weight coefficients, which are adjusted as needed; Threshold comparison: set a color difference threshold T diff , if diff 总 <T diff , then the key point is considered invisible in the current frame, that is: ； Assume that the facial key point set is H = (H1, H2, ..., H m ), where H j =(x j ,y j ) represents the coordinates of the jth key point, and j is any natural number from 1 to m. To detect head turning, two key points are selected to form a vector, and the angle change between the two vectors is calculated; Assume that the vectors of the two key points are and , where (x1, y1) and (x2, y2) are the keypoint coordinates in the initial frame, and (x1′, y1′) and (x2′, y2′) are the keypoint coordinates in the current frame; Define the angle change threshold θ′. If the angle θ between two vectors exceeds the threshold θ′, it is regarded as turning the head; The angle θ between the vectors v1 and v2 is calculated by the following formula: ； Among them, the dot product: ; Vector magnitude; ； 。 5. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The operations of the feature extraction and classification module for extracting facial features specifically include: S1 uses a multi-angle face detector: select a detector that supports multi-angle face detection, set the detected face bounding box list B = (B1, B2, ..., B s ), where B z =(x z ,y z , w z , h z ) represents the bounding box coordinates of the z-th face, where z is any natural number from 1 to s; S2 Preliminary alignment Calculate the rotation angle: by analyzing the aspect ratio of the bounding box And the preset frontal face aspect ratio r0 to estimate the rotation angle Q: ； Where p is an adjustment coefficient used to calibrate the calculation of the rotation angle; Rotate the image: Input the image I and the rotation angle Q, and output the rotated image I′; S3 Key point detection Use key point detection algorithm: Output key point set A = (A1, A2, ..., A E ), where A e =(x e ,y e ) represents the coordinates of the e-th key point, and e is any natural number from 1 to E; S4 Fine alignment Calculate the transformation matrix: Based on the selected key points, use affine transformation or similarity transformation to calculate the transformation matrix from the detected key points F to the standard key point set F0. The standard key point set is the key point template set of the frontal face; Apply the transformation: Apply the calculated transformation matrix to the face image to achieve fine alignment; S5 Post-processing and optimization Interpolation and smoothing: After alignment, perform interpolation or smoothing on the image to eliminate the distortion or noise generated during the transformation process; Cropping and scaling: According to the aligned image, crop the face area and scale it as needed for subsequent processing or analysis.

6. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The steps of the feature extraction and classification module for facial expression classification specifically include: A1. Database Construction: Collect and annotate a large number of facial expression images, covering common expressions of happiness, sadness, surprise, anger, fear, and disgust; let X be the original image dataset, where X f represents the fth image, Y is the corresponding expression label set, Y f Represents X f expression labels; use the deep learning model to train these images and extract deep features; let F(X f ) is from X f The extracted feature vector has a dimension of g; the features and the corresponding expression labels are stored in the database, that is, (F(X f ), Y f ) for each image; perform face detection and alignment, let O be the transformation matrix, X aligned is the aligned image, then: X aligned =O(X); A2. Real-time image acquisition and preprocessing: Use a camera or image input device to collect students' facial expression images in real time. real The image is collected in real time; the collected image is preprocessed, including grayscale, denoising, face detection and alignment, to obtain X real, aligned ; A3 Image contrast analysis: A3.1 Feature vector matching: The depth feature vector F(X real， aligned ) or the original image is compared with the feature vectors in the database; the similarity between the feature vectors is calculated using the cosine similarity or Euclidean distance measurement method, the feature vectors with the highest similarity are selected, and the corresponding expression labels are checked; A3.2 Expression Classification: Based on the matched feature vectors and their expression labels, the final expression category is determined using majority voting, weighted averaging, or other strategies.

7. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The working steps of the dual-modal fusion and recognition module specifically include: Feature fusion: The facial expression features and head posture features of students are spliced ​​or weighted fused to form comprehensive features; the weighted fusion formula is: ； in, and ω are weight coefficients, J face Represents facial expression features, J pose represents the head posture feature, J represents the comprehensive feature after fusion; Deep recognition: Use LSTM, GRU or Transformer deep neural networks to perform time series modeling and deep recognition of comprehensive features to identify the changing trends of students' emotional states; deep neural networks extract time series features through multi-layer networks and output student emotion recognition results.

8. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The steps of performing state analysis by the state and time domain analysis module specifically include: Data preparation: Collect students’ facial expressions and posture data from classroom videos and divide them into small segments with a time interval of Δl; Activity calculation: define the positive emotion set E pos ={happy, excited, ...}; For each time segment l, count the number of times positive emotions appear n pos (l); Calculate the activity index R: ； Where L is the total number of time segments, W pos (l) is the average duration of positive emotions; Concentration calculation: defining the concentration emotion set E focus ={serious, thinking, ...}, calculate the concentration index Z: ； Among them, n focus (l) is the number of times the focused emotion appears, W focus (l) is the average duration of focused emotion.

9. The video semantic emotion recognition system based on educational scenarios according to claim 8, characterized in that: The steps of analyzing the change of emotional state by combining the state and time domain analysis module with the time domain factor specifically include: Analysis of different time periods within a day: Divide a day into M time periods, and count the frequency of each emotion category in each time period and draw a histogram; Analysis of different time periods of each class: Divide each class into multiple stages: beginning, middle, and end, and calculate the change rate S of the emotional state in each stage, that is, the increase or decrease rate of the frequency of each emotional category in the next stage relative to the previous stage.

10. The video semantic emotion recognition system based on educational scenarios according to claim 9, characterized in that: The work content of the class quality evaluation module specifically includes: Recognize the teacher's teaching voice in the classroom video, use big language technology to identify semantics, and obtain the evaluation value PJ through manual evaluation; Combining information on student activity, concentration, the rate of change of emotional state combined with time domain factors, and evaluation values, a weighted average or fuzzy comprehensive evaluation method is used to comprehensively evaluate the quality of the class. The comprehensive evaluation formula is: ； in, is the weight coefficient.

Images