Education scene-based video semantic emotion recognition system

Abstract

The invention discloses a video semantic emotion recognition system based on an education scene, and relates to the technical field of education systems. The video semantic emotion recognition system based on the education scene comprises a data preprocessing and enhancing module, a face and posture processing module, a feature extraction and classification module, a bimodal fusion and recognition module, a state and time domain analysis module and a class quality evaluation module. Through comprehensive preprocessing and enhancement, the video frame quality is improved, and a good foundation is laid for subsequent analysis. The face and posture processing module can cope with complex conditions such as head turning and shielding, and data integrity is guaranteed. And bimodal fusion features are combined with deep neural network time sequence modeling, so that emotion can be recognized more accurately. The state and time domain analysis module creatively establishes a database containing time domain information, not only can analyze activeness and concentration degree of students, but also can analyze emotional state changes in combination with time domain factors.

Description

Technical Field

[0001] The present invention relates to the technical field of educational systems, and in particular to a video semantic emotion recognition system based on educational scenarios. Background Art

[0002] Semantic emotion recognition in educational scenarios refers to the use of natural language processing, computer vision, and machine learning technologies to perform sentiment analysis on multimodal data such as text, voice, and video generated in the process of education and teaching, in order to identify semantic emotional information such as the emotional state, attitude, and satisfaction of participants such as students and teachers. This technology is of great significance in improving teaching quality, optimizing teaching strategies, and promoting students' mental health.

[0003] Publication No. CN118551282A discloses a student emotion recognition method for online teaching. Aiming at the problems of scarcity of existing student emotion database and few multimodal emotion recognition methods, a student emotion database and a multimodal emotion analysis method are established. First, the method focuses on specific online learning scenarios and establishes corresponding databases. Then, the method uses text and image data sources to build a comprehensive emotion recognition model. Specifically, the method uses the dual-branch feature fusion of video (students’ class video clips) and text (students’ classroom comments) to realize the recognition of students’ classroom expressions, and uses the previously established database for training and optimization to realize in-depth analysis and understanding of students’ emotions. The model better grasps students’ emotions by using multimodal information of video and text, which meets the actual human emotion analysis requirements and also increases the reliability of the model’s emotion classification.

[0004] However, as shown in the above technology, the evaluation of teaching quality in existing classroom teaching, especially the recognition of students' emotions, requires a comprehensive evaluation of students' postures, facial expressions and other aspects, but it will be affected by many factors. For example, it is difficult to accurately recognize students' faces due to occlusion, turning of heads or light, and thus it is difficult to obtain accurate data for emotional judgment; and the existing technology only simply performs current emotional judgment, and whether the teacher can grasp the rhythm of the class is also an important part of teaching quality. Students may have different performances at different times. Only by analyzing them can the teaching quality be evaluated more fairly. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a video semantic emotion recognition system based on educational scenarios, which solves the problems that still exist in the above-mentioned prior art.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a video semantic emotion recognition system based on educational scenarios, comprising: Data preprocessing and enhancement module: remove noise, balance illumination, and handle occlusion of classroom video frames, and perform rotation operations to enhance data; Face and posture processing module: uses algorithms to detect faces, track, locate key points, detect head postures, and further process incomplete facial data caused by head turning or occlusion; Feature extraction and classification module: extract features from the face and recognize and classify expressions; further process posture features; Bimodal fusion and recognition module: Fusion of facial and posture features, using deep neural networks for temporal modeling and recognition; Status and time domain analysis module: Analyze students' activity and concentration, identify their emotional status, and establish an emotion recognition database with time domain information to compare features. Combined with time domain factors, analyze the changes in students' emotional status in different time periods within a day and in each class. Class quality evaluation module: Comprehensively evaluate the class quality based on emotional state, time domain factors and teacher’s teaching voice quality.

[0007] Preferably, the operations of the data preprocessing and enhancement module specifically include: Denoising: Use Gaussian filtering or median filtering algorithms to denoise classroom video frames to reduce the impact of classroom environmental noise on subsequent processing; Lighting balance processing: Use histogram equalization or adaptive histogram equalization algorithm to enhance the contrast of classroom video frames and make the classroom lighting in the image uniform; Histogram equalization principle: by calculating the histogram of the video frame, the pixel values ​​are redistributed to make the pixel values ​​evenly distributed; Occlusion processing: Use image restoration algorithms to repair the occluded parts of students in classroom videos and restore the occluded facial or body areas; Data enhancement: Rotate, scale, crop, and flip classroom video frames to increase data diversity to accommodate student recognition at different angles and distances.

[0008] Preferably, the steps of performing facial tracking and key point positioning by the facial and posture processing module include: Use MTCNN or RetinaFace algorithm to detect students’ faces in classroom videos, output the bounding box of students’ faces, and then use the 68-point or 98-point facial key point detection model in the Dlib library to locate facial key points and accurately capture facial features including eyes, mouth, and eyebrows; The bounding box is represented as (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the detection box, w and h are the width and height of the detection box respectively; the key point is represented as (x i ,y i ), where i is the index of the key point; Optical flow face tracking: After detecting the student's face and locating the key points, the optical flow equation is used to calculate the motion vector of the facial key points between consecutive video frames; let I(x, y, t) represent the pixel intensity at the image position (x, y) at time t; for the facial feature point P(x i ,y i , t), in the next frame t+1, the position becomes P(x i +dx i ,y i +dy i , t+1), where (dx i ,dy i ) is the motion vector of feature point P, and I(x i ,y i , t)≈I(x i +dx i ,y i +dy i , t+1); The optical flow constraint equation is used to ensure the brightness consistency of facial feature points between consecutive frames. Let u = (u i , v i ) is the feature point P i The optical flow vector of i and v i Represent the velocity components in the x and y directions respectively; ∇I i =(Ix i , Iy i ) represents the feature point P i The gradient of the image at i Indicates P i The rate of change of image intensity over time; the optical flow constraint equation ∇I i ·u i +İt i =0 means that at the facial feature point P i At , the dot product of the image gradient and the optical flow vector should be equal to the inverse of the rate of change of the image intensity over time; By calculating the motion vector of the pixel points in the video frame, the motion trajectory of the student's face is continuously tracked; a tracking window is set for each key point, and a tracking threshold is set to determine whether the tracking is effective; the tracking threshold includes a displacement threshold and a speed threshold, and the displacement threshold is set to D t , displacement , if d is greater than the set displacement threshold D t , it is judged as tracking failure; let the speed threshold be V t , the time interval from the i-th frame to the i+1-th frame is Δt, then the speed of the key point is , if v is greater than the set speed threshold Vt , it is determined as tracking anomaly or possible failure.

[0009] Preferably, occlusion and head-turning detection: Occlusion detection: During tracking, regularly check the visibility of key points within the facial bounding box; set a key point visibility threshold. When the number of key points is below the threshold or the key points are occluded or disappear, occlusion detection is triggered. Calculate the gradient or color difference of the pixels around the key points to determine whether the key points are visible; when occlusion is detected, pause the optical flow tracking and instead use the face re-detection algorithm to re-search for the face within several frames after occlusion; once the face is re-detected, immediately resume the optical flow tracking and update the key point information; Key point visibility threshold: Let the total number of key points be N, the number of visible key points be n^, and the visibility threshold be set as T^. When n^ < T^, occlusion detection is triggered; Key point visibility judgment: For each key point, calculate the gradient or color difference of the surrounding pixels: Gradient calculation: ; ; where I(x, y) is the pixel value of the image at the position (x, y), and k is the scaling factor of the Sobel operator; Gradient magnitude calculation: The gradient magnitude is the length of the gradient vector, reflecting the overall change intensity of the image at a certain point; ; Threshold comparison: Set a gradient magnitude threshold T G , if G < T G , it is considered that a certain key point is invisible in the current frame, that is: ; Color difference calculation: diff R = │R key - R surr │; diff G = │G key - G surr │; diff B = │B key - B surr │; where, (R key , G key , B key ) is the RGB value of the key point, (R surr , G surr , B surr) is the RGB value of the surrounding pixels; Use Euclidean distance to combine the color differences of the three RGB channels into one value: ; Or use the weighted sum method to combine the color differences of the three RGB channels into one value: diff 总 =α⋅diff R +β⋅diff G +γ⋅diff B ; Among them, α, β, and γ are weight coefficients, which can be adjusted as needed; Threshold comparison: set a color difference threshold T diff , if diff 总 <T diff , then the key point is considered invisible in the current frame, that is: ; Head turn detection: By analyzing the angle change of facial key points, determine whether the student has turned his head; set the angle change threshold of the key points. If the angle change of the key points exceeds the preset threshold, it is regarded as a head turn, and the angle change between the two key point vectors is calculated using the vector angle formula; when a head turn is detected, the optical flow tracking is also suspended, and the possible position of the student's head is predicted through the head posture estimation algorithm; on this basis, the face re-detection algorithm is used to search for the face near the predicted position; once the face is re-detected, the tracking is resumed and the key point information is updated; Assume that the set of facial key points is H = (H1, H2, ..., H m ), where H j =(x j ,y j ) represents the coordinates of the jth key point, and j is any natural number from 1 to m. In order to detect head turning, two key points are selected to form a vector, and the angle change between the two vectors is calculated; Assume that the vectors of the two key points are and , where (x1, y1) and (x2, y2) are the key point coordinates in the initial frame, and (x1′, y1′) and (x2′, y2′) are the key point coordinates in the current frame; Define an angle change threshold θ′. If the angle θ between the two vectors exceeds the threshold θ′, it is considered as a head turn. The angle θ between vectors v1 and v2 is calculated by the following formula: ; Among them, the dot product: ; vector norm; ; .

[0010] Preferably, the operation of extracting facial features by the feature extraction and classification module specifically includes: S1 uses a multi-angle face detector: select a detector that supports multi-angle face detection, set the detected face bounding box list B = (B1, B2, ..., B s ), where B z =(x z ,y z , w z ,h z ) represents the bounding box coordinates of the zth face, where z is any natural number from 1 to s; S2 preliminary alignment Calculate the rotation angle: Based on the detected face bounding box, calculate the rotation angle of the face; by analyzing the aspect ratio of the bounding box And the preset frontal face aspect ratio r0 to estimate the rotation angle Q: ; Where p is the adjustment coefficient used to calibrate the calculation of the rotation angle; Rotate the image: Input image I and rotation angle Q, perform rotation correction on the face image so that the face image faces roughly the front, and output the rotated image I′; S3 key point detection Use key point detection algorithm: On the face image after rotation correction, use the key point detection algorithm to detect the key points of the face, and output the key point set A = (A1, A2, ..., A E ), where A e =(x e ,y e ) represents the coordinates of the e-th key point, and e is any natural number from 1 to E; S4 Fine Alignment Calculate the transformation matrix: Based on the filtered key points, use affine transformation or similarity transformation to calculate the transformation matrix from the detected key point F to the standard key point set F0. The standard key point set is a set of key point templates for the frontal face. Apply transformation: Apply the calculated transformation matrix to the face image to achieve fine alignment; S5 post-processing and optimization Interpolation and smoothing: After alignment, the image is interpolated or smoothed to eliminate distortion or noise generated during the transformation process; Cropping and scaling: Based on the aligned images, crop the face area and scale it as needed for subsequent processing or analysis.

[0011] Preferably, the step of performing facial expression classification by the feature extraction and classification module specifically includes: A1. Database construction: Collect and annotate a large number of facial expression images, covering common expressions of happiness, sadness, surprise, anger, fear, and disgust; let X be the original image dataset, where X f represents the fth image, Y is the corresponding expression label set, Y f Represents X f expression labels; use deep learning models to train these images and extract deep features; let F(X f ) is from X f The extracted feature vector has a dimension of g; the features are stored in the database together with the corresponding expression labels, that is, (F(X f ), Y f ) For each image, perform face detection and alignment. Let O be the transformation matrix, X aligned is the aligned image, then: X aligned =O(X); A2. Real-time image acquisition and preprocessing: Use a camera or image input device to collect students' facial expression images in real time. real is the image collected in real time; the collected image is preprocessed, including grayscale, denoising, face detection and alignment, to obtain X real, aligned ; A3 image comparison analysis: A3.1 Feature vector matching: The depth feature vector F(X real， aligned ) or the original image is compared with the feature vectors in the database; the similarity between the feature vectors is calculated using the cosine similarity or Euclidean distance measurement method, several feature vectors with the highest similarity are selected, and the corresponding expression labels are checked; A3.2 Expression classification: Based on the matched feature vectors and their expression labels, the final expression category is determined by majority voting, weighted average or other strategies.

[0012] Preferably, the working steps of the dual-modal fusion and recognition module specifically include: Feature fusion: The facial expression features and head posture features of students are spliced ​​or weighted to form comprehensive features; the weighted fusion formula is: ; in, and ω are weight coefficients, J face Represents facial expression features, Jpose represents the head posture feature, and J represents the comprehensive feature after fusion; Deep recognition: Use LSTM, GRU or Transformer deep neural networks to perform time series modeling and deep recognition of comprehensive features to identify the changing trends of students' emotional states; deep neural networks extract time series features through multi-layer networks and output student emotion recognition results.

[0013] Preferably, the steps of performing state analysis by the state and time domain analysis module specifically include: Data preparation: Collect students’ facial expression and posture data from classroom videos and divide them into small segments with a time interval of Δl; Activity calculation: define the positive emotion set E pos ={happy, excited, ...}; For each time segment l, count the number of times positive emotions appear n pos (l); Calculate the activity index R: ; Where L is the total number of time segments, W pos (l) is the average duration of positive emotions; Concentration calculation: Defining the concentration emotion set E focus ={serious, thinking, ...}, calculate the concentration index Z: ; Among them, n focus (l) is the number of occurrences of focused emotion, W focus (l) is the average duration of focused emotion.

[0014] Preferably, the step of analyzing the change of emotional state by combining the state and time domain analysis module with the time domain factor specifically includes: Analysis of different time periods within a day: Divide a day into M time periods, count the frequency of each emotion category in each time period, and draw a histogram; Analysis of different time periods in each class: Divide each class into multiple stages: beginning, middle, and end, and calculate the rate of change of emotional state in each stage, that is, the rate of increase or decrease in the frequency of each emotional category in the next stage relative to the previous stage.

[0015] Preferably, the work content of the class quality evaluation module specifically includes: Recognize the teacher's teaching voice in the classroom video, use big language technology to recognize semantics, and obtain the evaluation value PJ through manual evaluation; Combining information on students' activity, concentration, the change rate of emotional state combined with time domain factors, and evaluation values, a weighted average or fuzzy comprehensive evaluation method is used to comprehensively evaluate the quality of the class; the comprehensive evaluation formula is: ; in, is the weight coefficient.

[0016] The present invention provides a video semantic emotion recognition system based on educational scenarios. Compared with the prior art, it has the following beneficial effects: 1. This video semantic emotion recognition system based on educational scenarios improves the quality of video frames through comprehensive preprocessing and enhancement, laying a good foundation for subsequent analysis. The face and posture processing module can cope with complex situations such as head turning and occlusion to ensure data integrity. The dual-modal fusion features, combined with deep neural network time series modeling, can more accurately identify emotions. The state and time domain analysis module innovatively establishes a database containing time domain information, which can not only analyze students' activity and concentration, but also analyze changes in emotional states in combination with time domain factors. The class quality evaluation module integrates multi-dimensional information to make the evaluation more comprehensive and objective, providing a more scientific and effective means for teaching quality evaluation and student status analysis.

[0017] 2. The video semantic emotion recognition system based on educational scenarios uses optical flow facial tracking to achieve stable and accurate tracking of students’ faces by calculating motion vectors, combined with tracking windows and threshold settings. It can continue to track students even if they move slightly, effectively solving the problem of unstable tracking in existing technologies. The occlusion detection mechanism periodically checks the visibility of key points and uses gradient or color differences to determine whether key points are visible. When occlusion is detected, it pauses the optical flow tracking and re-searches the face, effectively solving the problem of tracking failure caused by occlusion and improving the robustness of tracking. The head turn detection analyzes the angle changes of facial key points and uses the vector angle formula to calculate the angle. When a head turn is detected, it also pauses tracking and predicts the head position, and then re-searches the face to ensure continuous tracking even when the head is turned. These measures not only enhance the stability and accuracy of student facial tracking, but also effectively obtain facial information in complex scenarios, such as when students are occluded or turn their heads, providing more reliable data support for subsequent applications such as emotion recognition.

[0018] 3. In the video semantic emotion recognition system based on educational scenarios, in terms of facial feature extraction, the multi-angle face detector ensures accurate detection of faces at different angles, the initial alignment corrects the face direction by calculating the rotation angle, the key point detection accurately locates the key facial areas, the fine alignment further eliminates the deviation, and the post-processing and optimization improve the image quality, laying a good foundation for subsequent analysis. In the facial expression classification link, the database construction extracts high-discrimination deep features by collecting and annotating a large number of expression images and training them with deep learning models, providing rich and reliable references for classification. Real-time image acquisition and preprocessing ensure the acquisition of clear and standardized face images. In the image comparison analysis, feature vector matching uses appropriate measurement methods to accurately find similar feature vectors, and expression classification uses reasonable strategies to determine the final category. Overall, this module can accurately extract facial features and accurately classify expressions, providing strong support for applications such as emotion analysis and teaching evaluation in educational scenarios, and helping to improve teaching quality and effectiveness.

[0019] 4. This video semantic emotion recognition system based on educational scenarios collects classroom video data and divides it into time segments, accurately counts the number of occurrences and average duration of positive emotions and focused emotions, calculates the activity and concentration index, and can objectively quantify students' classroom status. When combined with time domain factor analysis, a day is divided into multiple time periods to count the emotion frequency and draw a histogram, which can intuitively present the distribution pattern of emotions within a day; each class is divided into different stages to calculate the emotion change rate, which can clearly understand the emotional dynamics during the classroom process. This solution not only comprehensively and meticulously analyzes the students' emotional state, but also deeply combines time domain factors to provide educators with richer and more accurate basis for understanding students' classroom performance and optimizing teaching strategies, helping to improve teaching quality; by integrating multiple aspects of information such as students' activity and concentration, and using methods such as weighted average, it can more accurately and objectively reflect the actual situation in the classroom, provide a strong basis for teaching improvement, and help improve teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a system module block diagram of the present invention; Figure 2 It is a schematic diagram of the overall steps of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1-Figure 2 , the present invention provides the following four technical solutions: The first implementation mode: a video semantic emotion recognition system based on an educational scenario, comprising: Data preprocessing and enhancement module: remove noise, balance illumination, and handle occlusion of classroom video frames, and perform rotation operations to enhance data; Face and posture processing module: uses algorithms to detect faces, track, locate key points, detect head postures, and further process incomplete facial data caused by head turning or occlusion; Feature extraction and classification module: extract features from the face and recognize and classify expressions; further process posture features; Bimodal fusion and recognition module: Fusion of facial and posture features, using deep neural networks for temporal modeling and recognition; Status and time domain analysis module: Analyze students' activity and concentration, identify their emotional status, and establish an emotion recognition database with time domain information to compare features. Combined with time domain factors, analyze the changes in students' emotional status in different time periods within a day and in each class. Class quality evaluation module: Comprehensively evaluate the class quality based on emotional state, time domain factors and teacher’s teaching voice quality.

[0023] Through comprehensive preprocessing and enhancement, the quality of video frames is improved, laying a good foundation for subsequent analysis. The face and posture processing module can cope with complex situations such as head turning and occlusion to ensure data integrity. The dual-modal fusion feature, combined with deep neural network time series modeling, can more accurately identify emotions. The state and time domain analysis module innovatively establishes a database containing time domain information, which can not only analyze students' activity and concentration, but also analyze changes in emotional states in combination with time domain factors. The class quality evaluation module integrates multi-dimensional information to make the evaluation more comprehensive and objective, providing a more scientific and effective means for teaching quality evaluation and student status analysis.

[0024] The second implementation mode is mainly different from the first implementation mode in that the operation contents of the data preprocessing and enhancement module specifically include: Denoising: Use Gaussian filtering or median filtering algorithms to denoise classroom video frames to reduce the impact of classroom environmental noise on subsequent processing; Lighting balance processing: Use histogram equalization or adaptive histogram equalization algorithm to enhance the contrast of classroom video frames and make the classroom lighting in the image uniform; Histogram equalization principle: by calculating the histogram of the video frame, the pixel values ​​are redistributed to make the pixel values ​​evenly distributed; Occlusion processing: Use image restoration algorithms (such as texture synthesis-based algorithms) to repair the occluded parts of students in classroom videos and restore the occluded facial or body areas; Data enhancement: Rotate, scale, crop, and flip classroom video frames to increase data diversity to accommodate student recognition at different angles and distances.

[0025] In terms of data preprocessing and enhancement, Gaussian filtering, median filtering denoising, histogram equalization and other illumination equalization processing are used, combined with image restoration algorithms to handle occlusion, and data is enhanced through operations such as rotation, which effectively improves the quality of video frames and provides a reliable foundation for subsequent analysis.

[0026] In this embodiment, the steps of performing facial tracking and key point positioning by the face and posture processing module include: Use efficient algorithms such as MTCNN or RetinaFace to detect student faces in classroom videos, output the bounding box of the student's face, and then use the 68-point or 98-point facial key point detection model in the Dlib library to locate facial key points and accurately capture the facial features of the eyes, mouth, and eyebrows.

[0027] The bounding box is represented as (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the detection box, w and h are the width and height of the detection box respectively; the key point is represented as (x i ,y i ), where i is the index of the key point; Optical flow face tracking: After detecting the student's face and locating the key points, the optical flow equation is used to calculate the motion vector of the facial key points between consecutive video frames; let I(x, y, t) represent the pixel intensity at the image position (x, y) at time t; for the facial feature point P(x i ,y i , t), in the next frame t+1, the position becomes P(x i +dx i ,y i +dy i , t+1), where (dx i ,dy i ) is the motion vector of feature point P, and I(x i ,y i , t)≈I(x i +dx i ,y i +dy i , t+1); The optical flow constraint equation is used to ensure the brightness consistency of facial feature points between consecutive frames. Let u = (u i , v i ) is the feature point P i The optical flow vector of i and v iRepresent the velocity components in the x and y directions respectively; ∇I i =(Ix i , Iy i ) represents the feature point P i The gradient of the image at i Indicates P i The rate of change of image intensity over time; the optical flow constraint equation ∇I i ·u i +İt i =0 means that at the facial feature point P i At , the dot product of the image gradient and the optical flow vector should be equal to the inverse of the rate of change of the image intensity over time; By calculating the motion vector of the pixel points in the video frame, the motion trajectory of the student's face is continuously tracked; a tracking window is set for each key point, and a tracking threshold is set to determine whether the tracking is effective; the tracking threshold includes a displacement threshold and a speed threshold, and the displacement threshold is set to D t , displacement , if d is greater than the set displacement threshold D t , it is judged as tracking failure; let the speed threshold be V t , the time interval from the i-th frame to the i+1-th frame is Δt, then the speed of the key point is , if v is greater than the set speed threshold V t , it is judged that the tracking is abnormal or may fail.

[0028] In the face and posture processing module, MTCNN or RetinaFace is used to efficiently detect faces and output bounding boxes, and the Dlib library is used to accurately locate key points, which can accurately capture facial features. Optical flow face tracking calculates motion vectors, combines tracking windows and threshold settings, and achieves stable and accurate tracking of students' faces. It can continue to track even if the students move slightly, effectively solving the problem of unstable tracking in existing technologies.

[0029] In this embodiment, occlusion and head turning detection: Occlusion detection: During the tracking process, the visibility of key points within the facial bounding box is checked regularly. A key point visibility threshold is set. When the number of key points is lower than the threshold, the key points (such as eyes, mouth, etc.) are occluded or disappear, which triggers occlusion detection. The key points are judged to be visible by calculating the gradient or color difference of the pixels around the key points. When occlusion is detected, the optical flow tracking is suspended, and the face re-detection algorithm (such as MTCNN or RetinaFace) is used to re-search the face within a few frames after the occlusion. Once the face is re-detected, the optical flow tracking is resumed immediately and the key point information is updated. Key point visibility threshold: Let the total number of key points be N, the number of visible key points be n^, and the visibility threshold be set as T^. When n^ < T^, occlusion detection is triggered; Example: If N = 68 (such as using 68-point facial landmarks of Dlib), Tvisible can be set to 30, that is, when the number of visible key points is less than 30, it is considered that the face is occluded; Key point visibility judgment: For each key point, calculate the gradient or color difference of the surrounding pixels: Gradient calculation: ; ; where I(x, y) is the pixel value of the image at the position (x, y), and k is the scaling factor of the Sobel operator; Gradient magnitude calculation: The gradient magnitude is the length of the gradient vector, which can reflect the overall change intensity of the image at a certain point; ; Threshold comparison: Set a gradient magnitude threshold T G , if G < T G , then it is considered that a certain key point is invisible in the current frame, that is: ; Color difference calculation: diff R = │R key - R surr │; diff G = │G key - G surr │; diff B = │B key - B surr │; where, (R key , G key , B key ) are the RGB values of the key point, and (R surr , G surr , B surr ) are the RGB values of the surrounding pixels (the average value can be taken); Use the Euclidean distance to combine the color differences of the RGB three channels into one value: ; Or use the weighted sum method to combine the color differences of the RGB three channels into one value: diff 总 = α ⋅ diff R + β ⋅ diff G+γ⋅diff B ; Among them, α, β, and γ are weight coefficients, which can be adjusted as needed; Threshold comparison: set a color difference threshold T diff , if diff 总 <T diff , then the key point is considered invisible in the current frame, that is: ; Head turn detection: By analyzing the angle change of facial key points, determine whether the student has turned his head; set the angle change threshold of the key points. If the angle change of the key points exceeds the preset threshold, it is regarded as a head turn, and the angle change between the two key point vectors is calculated using the vector angle formula; when a head turn is detected, the optical flow tracking is also suspended, and the possible position of the student's head is predicted through the head posture estimation algorithm; on this basis, the face re-detection algorithm is used to search for the face near the predicted position; once the face is re-detected, the tracking is resumed and the key point information is updated; Assume that the set of facial key points is H = (H1, H2, ..., H m ), where H j =(x j ,y j ) represents the coordinates of the jth key point, and j is any natural number from 1 to m. In order to detect head turning, two key points (such as the left eye and the right eye, or the nose tip and the left ear) are selected to form a vector, and the angle change between the two vectors is calculated; Assume that the vectors of the two key points are and , where (x1, y1) and (x2, y2) are the key point coordinates in the initial frame, and (x1′, y1′) and (x2′, y2′) are the key point coordinates in the current frame; Define the angle change threshold θ′. If the angle θ between the two vectors exceeds the threshold, it is considered as a head turn. The angle θ between vectors v1 and v2 is calculated by the following formula: ; Among them, the dot product: ; vector norm; ; .

[0030] The occlusion detection mechanism periodically checks the visibility of key points and uses gradient or color differences to determine whether key points are visible. When occlusion is detected, it pauses optical flow tracking and re-searches the face, effectively solving the problem of tracking failure caused by occlusion and improving the robustness of tracking. Head turn detection analyzes the angle changes of facial key points and calculates the angle using the vector angle formula. When a head turn is detected, it also pauses tracking and predicts the head position, and then re-searches the face to ensure continuous tracking even when the head turns. These measures not only enhance the stability and accuracy of student facial tracking, but also effectively obtain facial information in complex scenarios, such as when students are occluded or turn their heads, providing more reliable data support for subsequent applications such as emotion recognition.

[0031] The third implementation mode is mainly different from the first implementation mode in that the operation of extracting facial features by the feature extraction and classification module specifically includes: S1 uses a multi-angle face detector: select a detector that can detect multi-angle faces, set the list of detected face bounding boxes B = (B1, B2, ..., B s ), where B c =(x z ,y z , w z ,h z ) represents the bounding box coordinates of the zth face (upper left corner ( x z ,y z ) and width w z , height h z ), where B z =(x z ,y z , w z ,h z ) represents the bounding box coordinates of the zth face, where z is any natural number from 1 to s; S2 preliminary alignment (rotation correction) Calculate the rotation angle: Based on the detected face bounding box, calculate the rotation angle of the face; by analyzing the aspect ratio of the bounding box And the preset frontal face aspect ratio r0 to estimate the rotation angle Q: ; Where p is the adjustment coefficient used to calibrate the calculation of the rotation angle; Rotate the image: Input image I and rotation angle Q, perform rotation correction on the face image so that the face image faces roughly the front, and output the rotated image I′; S3 key point detection Use key point detection algorithm: On the face image after rotation correction, use key point detection algorithm (such as Dlib's 68-point detection, FAN, LAB, etc.) to detect the key points of the face (such as the corners of the eyes, the corners of the mouth, the tip of the nose, etc.), and output the key point set A = (A1, A2, ..., A E ), where A e =(x e ,y e ) represents the coordinates of the e-th key point, and e is any natural number from 1 to E; S4 fine alignment (affine transformation or similarity transformation) Calculate the transformation matrix: Based on the filtered key points, use affine transformation or similarity transformation to calculate the transformation matrix from the detected key point F to the standard key point set F0. The standard key point set is a set of key point templates for the frontal face. Apply transformation: Apply the calculated transformation matrix to the face image to achieve fine alignment; S5 post-processing and optimization Interpolation and smoothing: After alignment, the image is interpolated or smoothed to eliminate distortion or noise generated during the transformation process; Cropping and scaling: Based on the aligned images, crop the face area and scale it as needed for subsequent processing or analysis.

[0032] The multi-angle face detector can accurately detect faces at different angles, providing an accurate basis for subsequent processing. Preliminary alignment corrects the face by calculating the rotation angle so that it is roughly facing the front, improving the effect of subsequent processing. Key point detection uses advanced algorithms to accurately obtain facial key points. Fine alignment uses affine or similarity transformations to further eliminate deviations and make facial features more standard. In the post-processing and optimization stage, interpolation and smoothing eliminate distortion noise, and cropping and scaling ensure that the face area is appropriate. The overall solution is comprehensive and detailed, which effectively improves the accuracy and stability of facial feature extraction, provides more reliable and high-quality feature data for subsequent tasks such as emotion recognition, and helps to analyze students' status more accurately.

[0033] In this embodiment, the steps of facial expression classification by the feature extraction and classification module specifically include: A1. Database construction: Collect and annotate a large number of facial expression images, covering common expressions of happiness, sadness, surprise, anger, fear, and disgust; let X be the original image dataset, where X f represents the fth image, Y is the corresponding expression label set, Y f Represents X f expression labels; use deep learning models (such as CNN, ResNet, etc.) to train these images and extract deep features; let F(X f ) is from Xf The extracted feature vector has a dimension of g; the features are stored in the database together with the corresponding expression labels, that is, (F(X f ), Y f ) are correct; these feature vectors should have high discrimination and robustness, which can usually be measured by indicators such as model accuracy and recall rate; perform face detection and alignment on each image, let O be the transformation matrix, X aligned is the aligned image, then: X aligned =O(X); A2. Real-time image acquisition and preprocessing: Use a camera or image input device to collect students' facial expression images in real time. real is the image collected in real time; the collected image is preprocessed, including grayscale, denoising, face detection and alignment, to obtain X real, aligned ; A3 image comparison analysis: A3.1 Feature vector matching: The depth feature vector F(X real， aligned ) (if extraction is performed) or the original image (if images are compared directly) is compared with the feature vectors in the database; the similarity between feature vectors is calculated using metrics such as cosine similarity and Euclidean distance, and the feature vectors with the highest similarity are selected, and the corresponding expression labels are checked; A3.2 Expression classification: Based on the matched feature vectors and their expression labels, the final expression category is determined by majority voting, weighted average or other strategies.

[0034] By collecting and annotating a large number of expression images and using deep learning models for training, we extract deep features with high discrimination and robustness, providing a solid foundation for subsequent classification. Real-time image acquisition and preprocessing ensure the acquisition of clear and standardized face images, improving classification accuracy. In image comparison and analysis, feature vector matching uses a variety of metrics to calculate similarity and accurately find similar feature vectors; expression classification uses strategies such as majority voting, comprehensively considers matching results, and makes classification more scientific and reasonable. The overall solution effectively improves the accuracy and reliability of facial expression classification, and can accurately identify student expressions in real time.

[0035] In this embodiment, the working steps of the dual-modal fusion and recognition module specifically include: Feature fusion: The facial expression features and head posture features of students are spliced ​​or weighted to form comprehensive features to more comprehensively describe the emotional state of students; the weighted fusion formula is: ; in, and ω are weight coefficients, J face Represents facial expression features, Jpose represents the head posture feature, and J represents the comprehensive feature after fusion; Deep recognition: Use LSTM, GRU or Transformer deep neural networks to perform time series modeling and deep recognition of comprehensive features to identify the changing trends of students' emotional states; deep neural networks extract time series features through multi-layer networks and output student emotion recognition results.

[0036] Feature fusion forms comprehensive features by splicing or weighted fusion of facial expression and head posture features, which can more comprehensively and accurately describe the emotional state of students and avoid the limitations of single features. Deep recognition uses deep neural networks such as LSTM, GRU or Transformer to perform time series modeling and deep recognition of comprehensive features, which can effectively capture the changing trend of emotional states and output accurate emotion recognition results. This combination not only improves the accuracy of emotion recognition, but also dynamically reflects the changes in students' emotions, providing richer and more reliable information for emotion analysis and teaching evaluation in educational scenarios, and helping to improve teaching.

[0037] The fourth implementation mode is mainly different from the first implementation mode in that the steps of performing state analysis by the state and time domain analysis module specifically include: Data preparation: Collect students’ facial expression and posture data from classroom videos and divide them into small segments with a time interval of Δl; Activity calculation: define the positive emotion set E pos ={happy, excited, ...}; For each time segment l, count the number of times positive emotions appear n pos (l); Calculate the activity index R: ; Where L is the total number of time segments, W pos (l) is the average duration of positive emotions; Concentration calculation: Defining the concentration emotion set E focus ={serious, thinking, ...}, calculate the concentration index Z: ; Among them, n focus (l) is the number of occurrences of focused emotion, W focus (l) is the average duration of focused emotion.

[0038] In this embodiment, the steps of analyzing the change of emotional state by combining the state and time domain analysis module with the time domain factor specifically include: Analysis of different time periods within a day: Divide a day into M time periods, count the frequency of each emotion category in each time period, and draw a histogram; Analysis of different time periods in each class: Divide each class into multiple stages: beginning, middle, and end, and calculate the rate of change of emotional state in each stage, that is, the rate of increase or decrease in the frequency of each emotional category in the next stage relative to the previous stage.

[0039] By collecting classroom video data and dividing it into time segments, accurately counting the number of occurrences and average duration of positive emotions and focused emotions, and calculating the activity and concentration index, the student's classroom status can be objectively quantified. When combined with time domain factor analysis, a day is divided into multiple time periods to count the emotion frequency and draw a histogram, which can intuitively present the distribution pattern of emotions within a day; dividing each class into different stages to calculate the emotion change rate can clearly understand the emotional dynamics during the classroom process. This solution not only comprehensively and meticulously analyzes the emotional state of students, but also deeply combines time domain factors to provide educators with richer and more accurate basis for understanding students' classroom performance and optimizing teaching strategies, helping to improve teaching quality.

[0040] In this embodiment, the working contents of the class quality evaluation module specifically include: Recognize the teacher's teaching voice in the classroom video, use big language technology to recognize semantics, and obtain the evaluation value PJ through manual evaluation; Combining information on students' activity, concentration, the change rate of emotional state combined with time domain factors, and evaluation values, a weighted average or fuzzy comprehensive evaluation method is used to comprehensively evaluate the quality of the class; the comprehensive evaluation formula is: ; in, is the weight coefficient.

[0041] By integrating information on students' activity level, concentration level, and other aspects, and adopting methods such as weighted average, we can more accurately and objectively reflect the actual situation in the classroom, provide a strong basis for teaching improvement, and help improve teaching quality.

[0042] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0043] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0044] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A video semantic emotion recognition system based on educational scenarios, characterized in that: Including: Data preprocessing and enhancement module: Denoise, perform illumination equalization, occlusion processing on classroom video frames, and perform rotation operations to enhance data; Facial and pose processing module: Use algorithms to detect faces, track, locate key points, detect head poses, and further process incomplete facial data caused by head turning or occlusion; Feature extraction and classification module: Extract features from faces and recognize and classify expressions; Further process pose features; Bimodal fusion and recognition module: Fuse facial and pose features, and perform temporal modeling and recognition using a deep neural network; State and time-domain analysis module: Analyze students' activity levels and concentration, recognize emotional states, and establish an emotional recognition database containing time-domain information for feature comparison, and combine time-domain factors to analyze the changes in students' emotional states within a day and at different time periods of each class; Class quality evaluation module: Comprehensively evaluate class quality based on emotional states, time-domain factors, and teachers' teaching voice quality.

2. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The specific operation content of the data preprocessing and enhancement module includes: Denosing processing: Use Gaussian filtering or median filtering algorithms to denoise classroom video frames, reducing the impact of classroom environment noise on subsequent processing; Illumination equalization processing: Adopt histogram equalization or adaptive histogram equalization algorithms to enhance the contrast of classroom video frames and make the classroom illumination in the image uniform; Principle of histogram equalization: By calculating the histogram of video frames, redistribute pixel values to make the pixel value distribution uniform; Occlusion processing: Use image inpainting algorithms to repair the occluded parts of students in classroom videos and restore the occluded facial or body areas; Data enhancement: Perform rotation, scaling, cropping, and flipping operations on classroom video frames to increase data diversity to adapt to student recognition at different angles and distances.

3. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized in that: The steps for the facial and pose processing module to perform facial tracking and key point localization include: Use the MTCNN or RetinaFace algorithm to detect students' faces in classroom videos, output the bounding boxes of students' faces, and then use the 68-point or 98-point face key point detection model in the Dlib library to perform facial key point localization to accurately capture facial features including eyes, mouth, and eyebrows; The bounding box is represented as (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the detection box, w and h are the width and height of the detection box respectively; the key point is represented as (x i ,y i ), where i is the index of the key point; Optical flow method for facial tracking: After detecting the student's face and locating the key points, the optical flow equation is used to calculate the motion vector of the facial key points between consecutive video frames; let I(x, y, t) represent the pixel intensity at the image position (x, y) at time t; for the facial feature point P(x i ,y i , t), in the next frame t+1, the position becomes P(x i +dx i ,y i +dy i , t+1), where (dx i ,dy i ) is the motion vector of feature point P, and I(x i ,y i , t)≈I(x i +dx i ,y i +dy i , t+1); The optical flow constraint equation is used to ensure the brightness consistency of facial feature points between consecutive frames. Let u = (u i , v i ) is the feature point P i The optical flow vector, where u i and v i Represent the velocity components in the x and y directions respectively; ∇I i =(Ix i , Iy i ) represents the feature point P i The gradient of the image at i Indicates P i The rate of change of image intensity over time; the optical flow constraint equation ∇I i ·u i +İt i = 0 means that at the facial feature point P i At , the dot product of the image gradient and the optical flow vector should be equal to the inverse of the rate of change of the image intensity over time; By calculating the motion vector of the pixel points in the video frame, the motion trajectory of the student's face is continuously tracked; a tracking window is set for each key point, and a tracking threshold is set to determine whether the tracking is effective; the tracking threshold includes a displacement threshold and a speed threshold, and the displacement threshold is set to D t , displacement , if d is greater than the set displacement threshold D t , it is judged as tracking failure; let the speed threshold be V t , the time interval from the i-th frame to the i+1-th frame is Δt, then the speed of the key point is , if v is greater than the set speed threshold V t , it is judged that the tracking is abnormal or may fail.

4. The video semantic emotion recognition system based on educational scenarios according to claim 3 is characterized by: Occlusion and head turning detection: Occlusion detection: During tracking, regularly check the visibility of key points within the facial bounding box; Set the key point visibility threshold. When the number of key points is lower than the threshold, that is, when the key points are occluded or disappear, occlusion detection is triggered. Determine the visibility of key points by calculating the gradient or color difference of the pixels around the key points; When occlusion is detected, pause the optical flow method tracking and instead use the face re-detection algorithm to re-search for faces within several frames after occlusion; Once the face is re-detected, immediately resume the optical flow method tracking and update the key point information; Key point visibility threshold: Let the total number of key points be N, the number of visible key points be \(\hat{n}\), and the visibility threshold be set as \(\hat{T}\). When \(\hat{n}<\hat{T}\), occlusion detection is triggered; Key point visibility judgment: For each key point, calculate the gradient or color difference of the surrounding pixels: Gradient calculation: ； ； Where I(x, y) is the pixel value of the image at position (x, y), and k is the scaling factor of the Sobel operator; Gradient modulus calculation: Gradient modulus is the length of the gradient vector, reflecting the overall change intensity of the image at a certain point; ； Threshold comparison: Set a gradient modulus threshold T G , if G <T G , then it is considered that a key point is not visible in the current frame: ； Color difference calculation: diff R =│R key -R surr │; diff G =│G key -G surr │; diff B =│B key -B surr │; Among them, (R key , G key , B key ) is the RGB value of the key point, (R surr , G surr , B surr ) is the RGB value of the surrounding pixels; Use Euclidean distance to combine the color differences of the three RGB channels into one value: ； Or use the weighted sum method to combine the color differences of the three RGB channels into one value: diff 总 =α⋅diff R +β⋅diff G +γ⋅diff B ; Among them, α, β, and γ are weight coefficients, which can be adjusted as needed; Threshold comparison: set a color difference threshold T diff , if diff 总 <T diff , then the key point is considered invisible in the current frame, that is: ； Head turn detection: By analyzing the angle change of facial key points, determine whether the student has turned his head; set the angle change threshold of the key points. If the angle change of the key points exceeds the preset threshold, it is regarded as a head turn, and the angle change between the two key point vectors is calculated using the vector angle formula; when a head turn is detected, the optical flow tracking is also suspended, and the possible position of the student's head is predicted through the head posture estimation algorithm; on this basis, the face re-detection algorithm is used to search for the face near the predicted position; once the face is re-detected, the tracking is resumed and the key point information is updated; Assume that the set of facial key points is H = (H1, H2, ..., H m ), where H j =(x j ,y j ) represents the coordinates of the jth key point, and j is any natural number from 1 to m. In order to detect head turning, two key points are selected to form a vector, and the angle change between the two vectors is calculated; Assume that the vectors of the two key points are and , where (x1, y1) and (x2, y2) are the key point coordinates in the initial frame, and (x1′, y1′) and (x2′, y2′) are the key point coordinates in the current frame; Define an angle change threshold θ′. If the angle θ between the two vectors exceeds the threshold θ′, it is considered as a head turn. The angle θ between vectors v1 and v2 is calculated by the following formula: ； Among them, the dot product: ; vector norm; ； 。 5. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized in that: The operation of extracting facial features by the feature extraction and classification module specifically includes: S1 uses a multi-angle face detector: select a detector that supports multi-angle face detection, set the detected face bounding box list B = (B1, B2, ..., B s ), where B z =(x z ,y z , w z ,h z ) represents the bounding box coordinates of the zth face, where z is any natural number from 1 to s; S2 preliminary alignment Calculate the rotation angle: Based on the detected face bounding box, calculate the rotation angle of the face; by analyzing the aspect ratio of the bounding box And the preset frontal face aspect ratio r0 to estimate the rotation angle Q: ； Where p is the adjustment coefficient used to calibrate the calculation of the rotation angle; Rotate the image: Input image I and rotation angle Q, perform rotation correction on the face image so that the face image faces the front, and output the rotated image I′; S3 key point detection Use key point detection algorithm: On the face image after rotation correction, use the key point detection algorithm to detect the key points of the face, and output the key point set A = (A1, A2, ..., A E ), where A e =(x e ,y e ) represents the coordinates of the e-th key point, and e is any natural number from 1 to E; S4 Fine Alignment Calculate the transformation matrix: Based on the filtered key points, use affine transformation or similarity transformation to calculate the transformation matrix from the detected key point F to the standard key point set F0. The standard key point set is a set of key point templates for the frontal face. Apply transformation: Apply the calculated transformation matrix to the face image to achieve fine alignment; S5 post-processing and optimization Interpolation and smoothing: After alignment, the image is interpolated or smoothed to eliminate distortion or noise generated during the transformation process; Cropping and scaling: Based on the aligned images, crop the face area and scale it as needed for subsequent processing or analysis.

6. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The steps of facial expression classification by the feature extraction and classification module specifically include: A1. Database construction: Collect and annotate a large number of facial expression images, covering common expressions of happiness, sadness, surprise, anger, fear, and disgust; let X be the original image dataset, where X f represents the fth image, Y is the corresponding expression label set, Y f Represents X f expression labels; use deep learning models to train these images and extract deep features; let F(X f ) is from X f The extracted feature vector has a dimension of g; the features are stored in the database together with the corresponding expression labels, that is, (F(X f ), Y f ) For each image, perform face detection and alignment. Let O be the transformation matrix, X aligned is the aligned image, then: X aligned =O(X); A2. Real-time image acquisition and preprocessing: Use a camera or image input device to collect students' facial expression images in real time. real is the image collected in real time; the collected image is preprocessed, including grayscale, denoising, face detection and alignment, to obtain X real, aligned ; A3 image comparison analysis: A3.1 Feature vector matching: The depth feature vector F(X real， aligned ) or the original image is compared with the feature vectors in the database; the similarity between the feature vectors is calculated using the cosine similarity or Euclidean distance measurement method, several feature vectors with the highest similarity are selected, and the corresponding expression labels are checked; A3.2 Expression classification: Based on the matched feature vectors and their expression labels, the final expression category is determined by majority voting, weighted average or other strategies.

7. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The working steps of the dual-modal fusion and recognition module specifically include: Feature fusion: The facial expression features and head posture features of students are spliced ​​or weighted to form comprehensive features; the weighted fusion formula is: ； in, and ω are weight coefficients, J face Represents facial expression features, J pose represents the head posture feature, and J represents the comprehensive feature after fusion; Deep recognition: Use LSTM, GRU or Transformer deep neural networks to perform time series modeling and deep recognition of comprehensive features to identify the changing trends of students' emotional states; deep neural networks extract time series features through multi-layer networks and output student emotion recognition results.

8. The video semantic emotion recognition system based on educational scenarios according to claim 1 is characterized by: The steps of performing state analysis by the state and time domain analysis module specifically include: Data preparation: Collect students’ facial expression and posture data from classroom videos and divide them into small segments with a time interval of Δl; Activity calculation: define the positive emotion set E pos ={happy, excited, ...}; For each time segment l, count the number of times positive emotions appear n pos (l); Calculate the activity index R: ； Where L is the total number of time segments, W pos (l) is the average duration of positive emotions; Concentration calculation: Defining the concentration emotion set E focus ={serious, thinking, ...}, calculate the concentration index Z: ； Among them, n focus (l) is the number of occurrences of focused emotion, W focus (l) is the average duration of focused emotion.

9. The video semantic emotion recognition system based on educational scenarios according to claim 8, characterized in that: The step of analyzing the change of emotional state by combining the state and time domain analysis module with the time domain factor specifically includes: Analysis of different time periods within a day: Divide a day into M time periods, count the frequency of each emotion category in each time period, and draw a histogram; Analysis of different time periods of each class: Divide each class into multiple stages: beginning, middle, and end, and calculate the rate of change S of the emotional state in each stage, that is, the rate of increase or decrease in the frequency of each emotional category in the next stage relative to the previous stage.

10. The video semantic emotion recognition system based on educational scenarios according to claim 9, characterized in that: The work content of the class quality evaluation module specifically includes: Recognize the teacher's teaching voice in the classroom video, use big language technology to recognize semantics, and obtain the evaluation value PJ through manual evaluation; Combining information on students' activity, concentration, the change rate of emotional state combined with time domain factors, and evaluation values, a weighted average or fuzzy comprehensive evaluation method is used to comprehensively evaluate the quality of the class; the comprehensive evaluation formula is: ； in, is the weight coefficient.

Images