An ai teaching feedback and self-evolution method and system based on active sensing and hybrid thinking mechanism

Abstract

The application discloses an AI teaching feedback and self-evolution method and system based on active sensing and mixed thinking mechanism. The method comprises the following steps: generating student behavior characteristics, emotion characteristics and teacher-student voice characteristics; generating student state-teaching content structured information through a semantic analysis model; obtaining behavior emotion analysis information through a thinking scheduling module core model; generating visual perception module optimization parameters, semantic analysis module optimization parameters and thinking scheduling module optimization parameters according to all historical high-quality classroom video annotated efficient interaction segments and behavior emotion analysis information of each period; and updating the visual perception type model, the semantic analysis model and the thinking scheduling module core model. The application constructs a self-evolution system, which can continuously improve the sensing accuracy, analysis accuracy and scene adaptability without manual intervention, and completely solves the problems of static and no evolution ability of the traditional teaching analysis system.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to an AI teaching feedback and self-evolution method and system based on active perception and hybrid thinking mechanisms. Background Technology

[0002] While current smart classroom systems and instructional analytics solutions have achieved basic auxiliary functions such as classroom recording and courseware synchronization, they still have significant shortcomings in adaptability and intelligence to complex teaching scenarios. Firstly, existing systems primarily focus on the collection and storage of structured data, lacking deep intelligent understanding of unstructured multimodal data such as student behavior, emotions, and interactions. This makes it difficult to accurately capture implicit states in the classroom (such as students' subtle distractions, the gradual process of changing emotions of confusion, and inefficient interactions). Secondly, the perception method is mainly based on passively receiving data and relying on static rules for analysis and judgment. It cannot proactively explore key information based on real-time changes in the classroom (such as blurry facial expressions, dynamic tracking of students with abnormal behavior). First, the existing solutions lack effective adaptive evolution mechanisms, making it difficult to continuously optimize model parameters and analysis logic using historical teaching data. This makes it difficult to adapt to the personalized needs of different subjects, grade levels, and teaching styles. Moreover, the feedback is mostly general suggestions, lacking targeted teaching interventions and student guidance programs.

[0003] With breakthroughs in visual-language-action (VLA) multimodal models and reinforcement learning (RL) technologies, it has become possible to construct AI-powered teaching systems with human-like perception and adaptive reasoning capabilities. Compared to traditional static rule-based teaching analysis systems, the introduction of hybrid thinking mechanisms and proactive perception technologies enables dynamic scheduling of visual and linguistic reasoning paths based on real-time context, effectively addressing the shortcomings of existing solutions in areas such as deep understanding of multimodal data, dynamic scene adaptation, and personalized feedback.

[0004] Against this backdrop, there is an urgent need for an AI-powered teaching feedback and self-evolution solution that integrates proactive perception, hybrid thinking mechanisms, and reinforcement learning. This solution would address issues such as passive perception, fragmented analysis, lack of evolutionary capabilities, and insufficient feedback accuracy in existing technologies. By proactively capturing, deeply integrating, analyzing, and continuously optimizing multimodal classroom data, it would provide strong support for improving teaching quality and promoting personalized student development. Summary of the Invention

[0005] The purpose of this invention is to provide an AI teaching feedback and self-evolution method based on active perception and hybrid thinking mechanisms to at least solve one of the above-mentioned technical problems.

[0006] One aspect of the present invention provides an AI teaching feedback and self-evolution method based on active perception and hybrid thinking mechanisms, wherein the AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanisms includes:

[0007] Step 1: Obtain the multimodal data to be identified for the current period;

[0008] Step 2: Based on the multimodal data to be identified in the current period, generate student behavioral features, emotional features, and teacher-student voice features through a visual perception model;

[0009] Step 3: Based on student behavioral characteristics, emotional characteristics, and teacher and student voice characteristics, generate structured information on student status and teaching content through a semantic analysis model;

[0010] Step 4: Based on the structured information of student status and teaching content, obtain behavioral emotion analysis information through the core model of the thinking scheduling module;

[0011] Step 5: Determine whether the difference in the number of cycles between the current cycle and the previous optimized cycle reaches the preset threshold. If so, obtain all historical high-quality classroom video annotations of efficient interactive segments between the current cycle and the previous optimized cycle, as well as behavioral and emotional analysis information for each cycle.

[0012] Step 6: Generate optimization parameters for the visual perception module, semantic analysis module, and thought scheduling module based on the highly effective interactive segments marked in all historical high-quality classroom videos and the behavioral emotion analysis information for each cycle;

[0013] Step 7: Update the visual perception model based on the optimized parameters of the visual perception module, update the semantic analysis model based on the optimized parameters of the semantic analysis module, and update the core model of the thought scheduling module based on the optimized parameters of the thought scheduling module.

[0014] Optionally, step 1: obtaining the multimodal data to be identified for the current period includes:

[0015] Step 11: Obtain the raw video data for the current period;

[0016] Step 12: Run the YOLOv8 object detection algorithm. Run the YOLOv8 object detection algorithm on each frame of the video to detect the student's body, face, and hands. Filter out targets with a confidence score ≥ 0.8, record the target bounding box coordinates and corresponding confidence scores, and thus obtain the set of student target detection boxes.

[0017] Step 13: Based on the list of ROI regions from the iterative input, call the KCF correlation filter tracking algorithm to track the student targets in the student target detection box set in real time and generate the motion trajectory of each student; use the video frame sampler to extract the student's facial region, input the facial image into the pre-trained CNN model, identify the emotion information, and output the emotion label and confidence score corresponding to each frame, thereby obtaining the student target tracking trajectory and the real-time emotion label dataset.

[0018] Step 14: Calculate the Laplacian variance of the student's facial image, and combine it with the confidence of the target detection box to select images with a confidence of <0.9 or a Laplacian variance of <100 as blurred images; run the ESRGAN super-resolution algorithm on the blurred images to enlarge them by 2-4 times, and simultaneously perform CLAHE histogram equalization to enhance image details, thereby obtaining clear images of the student's face and hands.

[0019] Step 15: Based on the student target detection box set and student target tracking trajectory, identify and label abnormal behaviors; integrate the emotion tags and abnormal behavior markers in the real-time emotion tag set, associate them with the corresponding student's clear image, and generate structured original behavioral and emotional features;

[0020] Step 16: Acquire the audio stream using an audio frame capture device, and divide the audio stream into frames with a frame length of 20ms and a step length of 10ms; run the MFCC algorithm on each frame of audio to extract the 13-dimensional Mel frequency cepstral coefficients, and construct an audio feature vector by combining the frame energy and zero-crossing rate, thereby obtaining the teacher and student speech feature vector sequence and the audio and video timestamp mapping table.

[0021] Step 17: Based on the audio and video timestamp mapping table, align the teacher's voice segments with the student behavior data; run the dynamic time warping algorithm to calculate the synchronization difference between the teacher's voice and the student's behavior. If the synchronization difference exceeds 1.5 seconds, it is determined to be a comprehension gap. Record the gap time period and the corresponding student group to obtain the comprehension gap determination result.

[0022] Step 18: Sample students' facial posture and eye opening / closing at 100ms intervals using a video frame sampler; use a circular buffer data structure to cache the sampling data of the most recent 5 seconds; calculate the attention continuity value through a 1-second sliding window, and determine that the attention continuity value is ≥0.6 as focused and <0.6 as distracted, generate attention state labels, and thus obtain a 5-second attention cache dataset;

[0023] Step 19: Normalize the original behavioral and emotional features and map them to the [0,1] interval; standardize the teacher and student speech feature vector sequence with Z-score; encode the courseware text with UTF-8 through a text parser to retain the effective teaching text content, thereby obtaining the multimodal data to be identified in the current period.

[0024] Optionally, step 2: generating student behavioral features, emotional features, and teacher-student voice features based on the multimodal data to be identified in the current period using a visual perception model includes:

[0025] Step 21: Input the set of student target detection boxes, student target tracking trajectory, and cleared images of student face and hand from the multimodal data to be identified in the current period into the trained visual behavior perception sub-model, extract the basic features of student behavior, and thus obtain the basic features of student behavior indexed by the student's unique identifier. The basic features of student behavior include the type of behavior action, the time node of the action, the duration of the action, and the location area of ​​the action.

[0026] Step 22: Input the sharpened student facial image, real-time emotion label dataset, and 5-second attention cache dataset from the multimodal data to be identified in the current period into the trained visual emotion perception sub-model to generate a student emotion basic feature set indexed by the student's unique identifier;

[0027] Step 23: Input the standardized teacher and student speech feature vector sequence, audio and video timestamp mapping table, and understanding tomographic judgment result from the multimodal data to be identified in the current period into the trained speech perception sub-model to generate the basic feature set of teacher and student speech;

[0028] Step 24: The student behavior feature set generated in Step 21 is structurally integrated. Using the student's unique identifier as the first index and the timestamp as the second index, the behavior type, duration, behavior switching interval, and abnormal attribute markers are arranged in chronological order to form structured student behavior features. The student emotion feature set generated in Step 22 is structurally integrated. Using the student's unique identifier as the first index and the timestamp as the second index, the emotion type, frequency of occurrence, emotion duration, attention continuity value, and attention state markers are arranged in chronological order to form structured student emotion features. The teacher-student speech feature set generated in Step 23 is structurally integrated. Using the audio and video timestamps as the index, the speech source, speech period, acoustic feature vector, and understanding discontinuity association markers are arranged in chronological order to form structured teacher-student speech features.

[0029] Step 27: Using the audio and video timestamp mapping table as a unified benchmark, accurately align the time dimensions of structured student behavior features, structured student emotion features, and structured teacher and student voice features. Associate and match student behavior and emotion features with corresponding teacher and student voice features under the same timestamp to generate student behavior features, emotion features, and teacher and student voice features.

[0030] Optionally, step 3: generating structured student state-teaching content information based on student behavioral characteristics, emotional characteristics, and teacher-student voice characteristics using a semantic analysis model includes:

[0031] Step 31: Input the structured student behavior features, structured student emotion features, structured teacher-student speech features, and the effective teaching text content encoded in UTF-8 in Step 19 into the trained semantic analysis model to obtain a multimodal semantic input dataset. The multimodal semantic input dataset includes a unique student identifier, timestamp, student behavior features, student emotion features, teacher-student speech features, and teaching text fragments. The teaching text fragments are the teaching content paragraphs in the effective teaching text content that correspond to the current timestamp.

[0032] Step 32: Input the teacher and student speech features and teaching text fragments from the multimodal semantic input dataset into the trained teaching content semantic parsing sub-model to obtain the teaching content semantic feature set;

[0033] Step 33: Input the student behavior features and student emotion features from the multimodal semantic input dataset into the trained student state semantic mapping sub-model to obtain the student state semantic feature set;

[0034] Step 34: Input the semantic feature set of teaching content generated in Step 32 and the semantic feature set of student status generated in Step 33 into the trained state-content association integration sub-model to generate the state-content association feature set;

[0035] Step 35: Perform structured regularization on the state-content association feature set to generate standardized student state-teaching content structured information.

[0036] Optionally, step 4: obtaining behavioral emotion analysis information through the core model of the thinking scheduling module based on the structured information of student status and teaching content includes:

[0037] Step 41: Input the structured information of student status and teaching content generated in Step 3 into the core model of the trained thinking scheduling module to generate a hierarchical analysis input dataset;

[0038] Step 42: Input the individual student learning status and detailed student status list from the hierarchical analysis input dataset into the trained state depth analysis sub-model to generate an individual state depth analysis feature set;

[0039] Step 43: Input the hierarchical analysis input dataset and the individual state deep parsing feature set generated in Step 42 into the trained behavior-emotion association analysis sub-model to generate the behavior-emotion association analysis feature set;

[0040] Step 44: Input the hierarchical analysis input dataset and the behavior-emotion association analysis feature set into the trained behavior-emotion association analysis sub-model to generate the fault-behavior-emotion association feature set;

[0041] Step 45: Input the behavioral-emotion association analysis feature set generated in Step 43 and the fault-behavioral-emotion association feature set generated in Step 44 into the trained structured sub-model of the analysis results to obtain the integrated analysis results;

[0042] Step 46: Standardize and structure the integrated analysis results to generate structured behavioral sentiment analysis information.

[0043] Optionally, step 6: generating optimization parameters for the visual perception module, semantic analysis module, and thought scheduling module based on the highly interactive segments annotated from all historical high-quality classroom videos and the behavioral and emotional analysis information for each cycle includes:

[0044] Step 61: Based on the high-efficiency interactive segments and behavioral sentiment analysis information of each period of all historical high-quality classroom videos obtained in Step 5, generate an optimized benchmark dataset containing positive features of high-efficiency interaction, negative features of problem behavior and sentiment, and features related to teaching content. At the same time, assign optimization priority to each feature, with positive features of high-efficiency interaction as high priority and negative features of problem behavior and sentiment as secondary priority.

[0045] Step 62: Generate optimized parameters for the visual perception module. The visual perception module includes a trained visual behavior perception sub-model, a visual emotion perception sub-model, and a speech perception sub-model. Based on the optimized benchmark dataset, perform the following operations: For the visual behavior perception sub-model, statistically determine the optimal range of target detection confidence, the trajectory matching threshold of the KCF tracking algorithm, and the threshold for abnormal behavior judgment in the positive features of efficient interaction. Combine this with false positive and false negative target detection data in the negative features of problem behavior emotion to adjust the confidence threshold of the YOLOv8 target detection algorithm, the filtering parameters of the KCF correlation filter tracking algorithm, and the abnormal behavior labeling rules, generating optimized parameters for target detection, tracking algorithm, and behavior recognition thresholds. For the visual emotion perception sub-model, statistically determine the optimal value of emotion recognition confidence and the parameter range for facial image enhancement in the positive features of efficient interaction. Combine this with false positive emotion judgment data in the negative features of problem behavior emotion to adjust the classification threshold of the emotion recognition model and the ESRGAN... The super-resolution algorithm's magnification factor and the enhancement coefficient of CLAHE histogram equalization are used to generate optimization parameters for emotion recognition and image enhancement. For the speech perception sub-model, the MFCC dimension, framing parameters, and speech emotion recognition confidence threshold are statistically analyzed in the positive features of efficient interaction. Combined with misclassification data of speech in the negative features of problem behavior emotion, the extraction dimension, audio framing parameters, and classification threshold of the speech emotion recognition model are adjusted to generate optimization parameters for speech feature extraction and speech emotion recognition. The above sub-model optimization parameters are integrated to form an optimization parameter set for the visual perception module. The parameter set includes the thresholds, coefficients, and rule parameters of each sub-model. Each parameter is labeled with the corresponding optimization basis, namely the statistical results of the positive features of efficient interaction or the correction results of the negative features of problem behavior emotion.

[0046] Step 63: Generate optimization parameters for the semantic analysis module. The semantic analysis module includes a trained sub-model for semantic parsing of teaching content, a sub-model for semantic mapping of student state, and a sub-model for integrating state-content association. Based on the optimized benchmark dataset, perform the following operations: For the semantic parsing sub-model of teaching content, calculate the accuracy of extracting core semantic units of teaching in the positive features of efficient interaction and the matching degree of teaching stage division. Combine this with the semantic mis-parsing data in the negative features of problem behavior and emotion, adjust the semantic word segmentation weights, knowledge point matching thresholds, and teaching stage judgment rules to generate semantic parsing weight parameters, knowledge point matching threshold parameters, and teaching stage division parameters. For the semantic mapping sub-model of student state, calculate the accuracy of the behavior-emotion-learning state mapping and the optimal interval of state confidence in the positive features of efficient interaction, and combine this with... For the state mismapping data in the negative features of problematic behavior and emotion, the weights of the state mapping rules and the coefficients for calculating state confidence are adjusted to generate state mapping weight parameters and confidence calculation optimization parameters. For the state-content association integration sub-model, the optimal value of the association strength between teaching content and student state in the positive features of efficient interaction and the threshold for judging the association fault are statistically analyzed. Combined with the association mismatch data in the negative features of problematic behavior and emotion, the coefficients for calculating association strength and the rules for marking the association fault are adjusted to generate association strength optimization parameters and association fault judgment parameters. The optimization parameters of the above sub-models are integrated to form the optimization parameter set of the semantic analysis module. The parameter set contains the weights, thresholds, and calculation coefficients of each sub-model, and each parameter is labeled with the corresponding optimization basis, that is, the comparison results between the positive features of efficient interaction and the negative features of problematic behavior and emotion.

[0047] Step 64: Generate optimization parameters for the thinking scheduling module. The core model of the thinking scheduling module includes a trained state deep analysis sub-model, a behavior-emotion association analysis sub-model, and an analysis result structured sub-model. Based on the optimization benchmark dataset, perform the following operations: For the state deep analysis sub-model, statistically analyze the state decomposition accuracy and the optimal value of the invalid state removal threshold in the positive features of efficient interaction. Combined with the state decomposition error data in the negative features of question behavior and emotion, adjust the state decomposition rules and the invalid state confidence removal threshold to generate optimized state decomposition parameters and invalid state removal parameters; For the behavior-emotion association analysis sub-model, statistically analyze the optimal value of the behavior-emotion combination association strength and the high-frequency combination matching rules in the positive features of efficient interaction. Combined with the question... For the negative features of question behavior and emotion, erroneous data in association analysis are analyzed, and the weights for calculating association strength and the thresholds for judging high-frequency combinations are adjusted to generate association analysis weight parameters and combination judgment optimization parameters. For the structured sub-model of the analysis results, the optimal interval of the confidence of the analysis conclusion and the matching degree of the structured fields in the positive features of efficient interaction are statistically analyzed. Combined with the erroneous data of the conclusions in the negative features of question behavior and emotion, the calculation coefficients of the conclusion confidence and the weights of the structured fields are adjusted to generate conclusion confidence optimization parameters and structured field adjustment parameters. The above sub-model optimization parameters are integrated to form the optimization parameter set of the thinking scheduling module. The parameter set contains the weights, thresholds, and calculation coefficients of each sub-model, and each parameter is labeled with the corresponding optimization basis, i.e., the feedback result of the positive features of efficient interaction.

[0048] Step 65: Optimize parameter verification and standardization integration. Perform consistency verification on the optimized parameters of the visual perception module generated in Step 62, the optimized parameters of the semantic analysis module generated in Step 63, and the optimized parameters of the thought scheduling module generated in Step 64, and generate standardized optimized parameters of the visual perception module, the optimized parameters of the semantic analysis module, and the optimized parameters of the thought scheduling module.

[0049] Optionally, the visual emotion perception sub-model includes:

[0050] Input layer: The input layer receives the sharpened student facial image, real-time emotion label dataset, 5-second attention cache dataset, image quality parameter set, and student unique identifier + timestamp from the multimodal data to be identified in step 1, wherein:

[0051] Clarify student facial images: standardize the size to 224×224×3, RGB format;

[0052] Real-time sentiment label dataset: The raw confidence scores of sentiment information output by the pre-trained CNN model in step 1 are used as weak supervision signals for the model;

[0053] 5-second attention buffer dataset: continuous attention values, eye opening and closing, and facial pose sampling data in a circular buffer;

[0054] Image quality parameter set: Laplacian variance of facial images calculated in step 1, and YOLOv8 target detection confidence score;

[0055] Student unique identifier + timestamp: serves as a data index to ensure that output features are accurately associated with students and time, facilitating subsequent structured processing;

[0056] An image quality adaptive preprocessing layer is used for image filtering and feature completion initialization to adapt to the image sharpening result of step 1.

[0057] A dual threshold screening rule is set: images with a Laplacian variance ≥ 100 and a detection confidence ≥ 0.9 are considered high-quality images and directly enter the feature extraction layer; images with a Laplacian variance < 100 or a detection confidence < 0.9 are considered low-quality images and are marked with a quality weight coefficient.

[0058] Pixel-level weighted initialization for low-quality images: The pixel mean of high-quality images is used as the pixel completion benchmark for low-quality images. Pixel-level correction is completed by combining the quality weight coefficient to avoid feature distortion caused by blurry images directly entering the feature layer. Output: Standardized facial image, quality weight coefficient, and association index of student unique identifier + timestamp.

[0059] A multi-branch fine-grained visual feature extraction layer is used, which employs a three-branch parallel extraction + feature normalization. Based on MobileViT's lightweight convolutional blocks, it extracts the core visual features related to students' emotions in the classroom. The feature output dimension of all branches is unified to 256 dimensions to facilitate subsequent fusion.

[0060] Facial micro-expression feature branch: It consists of 4 layers of depthwise separable convolution + 2 layers of MobileViT small-scale attention blocks, with strides of 1, 1, 2, 2 respectively. It extracts the texture and deformation features of key micro-expression areas such as brow bone, corner of mouth, and nasal wings, and outputs a 256-dimensional micro-expression feature vector. This branch is the core branch, and the initial weight is set to 0.5.

[0061] Eye attention feature branch: Based on facial key point detection to locate the eye region, the features of eye opening and closing, eye movement direction and blink frequency are extracted through 3-layer depthwise separable convolution. Combined with the eye opening and closing sampling values ​​in the 5-second attention cache dataset, a 256-dimensional eye attention feature vector is output, with the initial weight set to 0.3.

[0062] Head pose feature branch: By solving the three-dimensional pose matrix of facial key points, the head pitch angle, yaw angle and roll angle features are extracted. Combined with the head movement trend in the student target tracking trajectory in step 1, a 256-dimensional head pose feature vector is output, with the initial weight set to 0.2.

[0063] Feature normalization: L2 normalization is performed on the feature vectors of the three branches to eliminate the difference in dimensions, and the micro-expression feature vector, eye attention feature vector, head posture feature vector, and the initial weights of each branch are output.

[0064] An attention-gated feature fusion layer is used to adaptively fuse the three-branch features. It introduces an attention gating mechanism to dynamically adjust the feature weights of the three branches based on the input image quality parameters and attention cache data, thus addressing the issue of varying feature importance under different image qualities and attention states.

[0065] The average attention continuity value in the 5-second attention cache dataset is used as the gating trigger signal: if the attention continuity value is ≥0.6, the weight of the micro-expression feature branch is increased and the weight of the head pose feature branch is decreased; if the attention continuity value is <0.6, the weight of the head pose and eye attention feature branches is increased and the weight of the micro-expression feature branch is decreased.

[0066] Using the quality weight coefficients in the image quality parameter set as feature weighting coefficients: multiply the feature vectors of low-quality images by the quality weight coefficients, and keep the weights of the feature vectors of high-quality images at 1 to complete feature robust completion;

[0067] Feature fusion: The dynamically weighted three-branch feature vectors are concatenated along the channel dimension, and feature dimensionality reduction and fusion are performed through a 1×1 convolution layer to output a 512-dimensional multimodal visual fusion feature vector. This vector contains coupled features of the face, eyes, and head, providing a foundation for subsequent temporal modeling.

[0068] A time-attention bidirectional GRU temporal modeling layer is used to capture the temporal continuity and gradual changes in students' emotions in the classroom. It is adapted to a 5-second attention buffer time window and is completely synchronized with the attention data collection rhythm in step 1.

[0069] Using 100ms as the time step, the multimodal visual fusion feature vectors within a 5-second time window are arranged in chronological order to form a 50×512 temporal feature sequence.

[0070] Bidirectional GRU layer: Two bidirectional GRU layers are set up with a hidden layer dimension of 256. The forward GRU captures the positive trend of emotion change, and the backward GRU captures the negative trend of emotion change, outputting a 50×512 temporal correlation feature sequence.

[0071] Temporal attention layer: Introduces a temporal attention mechanism to weight the feature sequences of 50 time steps, highlighting the feature weights of key frames with emotional changes, suppressing redundant frame features without emotional changes, and outputting a 256-dimensional temporal fusion core emotion feature vector.

[0072] The attention-emotion dual-label cross-validation layer is used for soft constraint verification. It uses the attention data from step 1 as a constraint for emotion recognition, eliminating contradictory results, improving feature effectiveness, and providing pre-filtering for the state mapping of the subsequent semantic analysis model.

[0073] Temporal fusion of core emotion feature vectors, mean of continuous attention values ​​in a 5-second attention cache dataset, and attention state labels;

[0074] Preliminary emotion classification: A 256-dimensional feature vector is mapped to confidence scores for 7 categories of classroom student emotions through a fully connected layer, generating preliminary emotion classification results;

[0075] Mutual verification rule determination: Core verification rules are set; if violated, the result is marked as invalid emotional and corrected based on attention state.

[0076] Rule 1: If the mean of continuous attention scores is ≥0.6, the confidence scores for boredom, fatigue, and confusion need to be reduced by 30%.

[0077] Rule 2: If the mean of continuous attention scores is <0.6, the confidence scores for calm, happy, and focused should be reduced by 30%.

[0078] Rule 3: If the head posture is tilted down at a angle greater than 45° and the eye opening angle is less than 0.2, it is directly judged as a distraction-related emotion, and only the two emotion results of annoyance and fatigue are retained;

[0079] Output the adjusted confidence scores for 7 types of emotions, effective / ineffective emotion labels, and a correlation mapping table between the mean of continuous attention values ​​and emotion confidence.

[0080] The teaching stage adaptive adjustment layer is used to reserve an interface for docking with the semantic parsing model of subsequent teaching content, realizing the pre-association of emotional features and teaching content. It only performs dynamic weight adjustment within the model. If teaching stage information is not yet available, it outputs according to the initial weights.

[0081] Interface input: Teaching stage markers output by the semantic parsing sub-model for subsequent teaching content;

[0082] Dynamic weight adjustment: Based on the characteristics of the teaching stage, five sets of feature weight adjustment coefficients are preset to adjust the revised emotion confidence score by weighting it. For example:

[0083] During the classroom questioning phase: Increase the weight given to surprise and joy, and match the students' emotional reactions when they are asked questions;

[0084] During the knowledge delivery phase: increase the weight given to feelings of confusion and calmness to match the core emotions students experience while listening to the lecture;

[0085] During the classroom practice phase: increase the weight of fatigue and confusion to match the emotional characteristics of students when doing exercises; output the emotional confidence score after the teaching phase is adapted, the teaching phase mark, and the student's unique identifier + timestamp;

[0086] The emotional feature structured output layer is used to complete the statistics and structuring of emotional features. The output feature set fully meets the structured emotional feature requirements of step 2 and can be directly used as input to the semantic analysis model without additional feature transformation.

[0087] Emotional characteristics statistics: Based on 100ms time intervals, integrate the emotional confidence scores after the adaptation of teaching stages within a 5-second time window, and calculate the emotional intensity characteristics and emotional fluctuation characteristics.

[0088] Attention-Emotion Association Labeling: Associate the mean of continuous attention values ​​and attention state labels with emotion features to label the attention attributes corresponding to each emotion type;

[0089] Structured encapsulation: Using the student's unique identifier as the first index and the timestamp as the second index, the feature fields are encapsulated according to the structured requirements of step 2, and the final output is a structured student emotion feature set. The fields include: student unique identifier, timestamp, emotion type, mean emotion intensity, peak emotion intensity, variance of emotion fluctuation, mean of continuous attention value, attention state label, valid emotion result label, and teaching stage label.

[0090] Optionally, the student state semantic mapping sub-model includes:

[0091] The input layer is used to obtain the structured student behavior features and structured student emotion features generated in step 2; the semantic feature set of teaching content generated in step 3; and the comprehension fault determination results from step 17.

[0092] A multimodal feature preprocessing and normalization layer is used to standardize the input features and eliminate dimensional differences.

[0093] Behavioral feature normalization: converting behavior types into one-hot encodings, mapping behavior duration and switching interval to the [0,1] interval, and converting anomaly markers into 0 / 1 binary features;

[0094] Emotional feature normalization: Emotional types (happiness, calmness, confusion, etc.) are converted into one-hot codes, and the mean and variance of emotional intensity are mapped to the [0,1] interval, while the continuous values ​​of attention are directly retained as the original [0,1] values;

[0095] Teaching content feature encoding: The teaching stage and knowledge point type are converted into one-hot encoding, and the teaching content text is converted into a 512-dimensional semantic embedding vector through pre-trained word vectors;

[0096] Finally, standardized behavioral feature vectors, emotion feature vectors, teaching content semantic vectors, attention feature vectors, as well as student unique identifiers and timestamp indexes are generated.

[0097] A multimodal feature coupling and fusion layer, wherein the multimodal feature coupling and fusion layer is used for feature fusion:

[0098] Feature interaction branches: Construct behavior-emotion interaction branches, behavior-attention interaction branches, and emotion-attention interaction branches respectively. Extract the interaction features of pairwise features through element-wise multiplication and fully connected layers. Each interaction branch outputs a 64-dimensional interaction feature vector.

[0099] Teaching content perception branch: The semantic vector of teaching content is concatenated with the interaction feature vector. Through a 1×1 convolution layer and Leaky ReLU activation, the association features between teaching content and student features are extracted, and a 256-dimensional teaching content-student feature association vector is output.

[0100] Feature fusion gating: A gating mechanism is introduced to dynamically adjust the feature weights of each branch based on the teaching stage. All branch features are fused by gating weights to output a 512-dimensional multimodal coupled feature vector. This vector contains information on the deep correlation between student behavior, emotions, attention and teaching content.

[0101] The temporal dynamic modeling layer is used to capture the temporal changes in the learning state and is fully synchronized with the 5-second attention buffer in step 1 and the temporal features in step 2.

[0102] Construction of temporal feature sequence: Using 100ms as the time step, the multimodal coupled feature vectors within the 5-second time window are arranged in chronological order to form a 50×512 temporal coupled feature sequence;

[0103] Temporal memory gate module: Set up 2 layers of gated loop units, introduce temporal memory gates, retain the feature memory of previous time steps, capture the gradual trend of learning state, and output a 50×256 temporal dynamic feature sequence;

[0104] Temporal attention weighting: A temporal attention mechanism is introduced to weight the feature sequences of 50 time steps, highlighting the feature weights of key time steps with state changes, suppressing redundant time step features without state changes, and outputting a 256-dimensional temporal dynamic core feature vector, which integrates static coupling features and temporal dynamic change information;

[0105] An adaptive adjustment layer for teaching stages is used to dynamically adapt the mapping between teaching stages and states.

[0106] Teaching Stage Weighting Library: Five sets of exclusive weighting coefficients for each teaching stage are preset, including weighting coefficients for new lesson introduction, knowledge delivery, classroom questioning, classroom practice, and summary review. Each set of weights includes weights for behavioral characteristics, emotional characteristics, attention characteristics, and temporal characteristics.

[0107] Dynamic weight matching: Based on the input teaching stage marker, the corresponding weight coefficients are retrieved from the weight library to adjust the time-series dynamic core feature vector, and the time-series dynamic feature vector adapted to the teaching stage, the teaching stage marker, and the corresponding teaching content text are output.

[0108] A dynamic confidence calibration layer is used to learn the dynamic calculation of state confidence and correct contradictory results.

[0109] Preliminary state mapping: Through one fully connected layer and Softmax activation, the temporal dynamic feature vector adapted to the teaching stage is mapped to the preliminary confidence scores of five learning states;

[0110] Confidence score calculation across multiple dimensions: The final confidence score is dynamically calculated based on three dimensions, and the weight of each dimension can be optimized through self-evolution.

[0111] Feature matching degree: The degree of matching between behavioral, emotional, and attentional features and learning state;

[0112] Temporal consistency: The current state is consistent with the state 4 seconds prior;

[0113] Teaching content suitability: The degree of suitability between the current state and the teaching content and teaching stage;

[0114] Conflicting Result Correction: Three core correction rules are set. If the initial state result violates the rules, the confidence level of the corresponding state is lowered, and states with higher matching degree are selected first.

[0115] Rule 1: For attention span scores ≥ 0.8 and no abnormal behavior, the confidence level for distraction, inattentiveness, and passive resistance is reduced by 50%.

[0116] Rule 2: For positive behaviors such as raising hands and answering, and negative behaviors such as resistance and distraction, the confidence level is reduced by 40%.

[0117] Rule 3: During periods of comprehension gap, reduce the confidence level of focused comprehension by 60%, prioritizing retention of confusion, hesitation, and distraction; output the final confidence scores for the five learning states, valid / invalid state labels, and the basis for confidence calculation.

[0118] The learning state classification and structured output layer is used to complete state classification and structured encapsulation, and outputs a set of student state semantic features.

[0119] State classification determination: Select the learning state with the highest final confidence score as the student's current learning state. If the highest confidence score is less than the preset threshold, it is marked as an unknown state.

[0120] Status characteristic statistics: In order of timestamp, the duration of each learning status, the number of status transitions, and the mean status confidence of each student in the current period are statistically analyzed.

[0121] Association annotation: Associate the learning status with the teaching content text, teaching stage, and comprehension gap markers, and annotate the teaching scenario attributes corresponding to the status;

[0122] Structured encapsulation: Using the student's unique identifier as the first index and the timestamp as the second index, the feature fields are encapsulated according to the requirements of the original technical solution. The final output is a semantic feature set of student status, with fields including: student unique identifier, timestamp, learning status, status confidence, status duration, teaching content text, teaching stage, understanding discontinuity association marker, and status validity marker.

[0123] Optionally, the semantic parsing sub-model for teaching content includes:

[0124] Multi-source text temporal alignment layer: The multi-source text temporal alignment layer is used to obtain the teacher and student voice text sequence, effective teaching text content, and audio and video timestamp mapping table input in step 3. Based on the audio and video timestamps, the voice text and courseware text (knowledge points, examples) are temporally sliced ​​and aligned, redundant text without teaching semantics is removed, and a 512-dimensional temporal aligned text feature sequence is generated.

[0125] The knowledge graph-enhanced semantic encoding layer is used to align text feature sequences and lightweight teaching knowledge graphs according to time sequence. It uses a lightweight Transformer encoder to extract text context semantics and fuses the knowledge point embedding vectors of the knowledge graph with text features to strengthen the semantic association between knowledge points, definitions, and examples, generating a 256-dimensional knowledge-enhanced semantic feature vector.

[0126] The temporal-aware semantic unit extraction layer is used to enhance semantic feature vectors based on knowledge, align text sequences temporally, introduce a temporal attention mechanism, accurately identify 6 types of core semantic units through conditional random fields, and simultaneously label the start and end timestamps of semantic units to generate semantic unit classification results, corresponding timestamp intervals, and semantic confidence scores.

[0127] The teaching stage-semantic unit joint output layer is used to generate a semantic feature set of teaching content based on the semantic unit results and temporal attention weights.

[0128] This application also provides an AI teaching feedback and self-evolution system based on active perception and hybrid thinking mechanisms, wherein the AI ​​teaching feedback and self-evolution system based on active perception and hybrid thinking mechanisms includes:

[0129] A multimodal data acquisition module for identification, wherein the multimodal data acquisition module for identification is used to acquire multimodal data to be identified in the current period;

[0130] The feature generation module is used to generate student behavior features, emotion features, and teacher-student voice features based on the multimodal data to be identified in the current period through a visual perception model.

[0131] The student status-teaching content structured information generation module is used to generate student status-teaching content structured information based on student behavioral characteristics, emotional characteristics, and teacher and student voice characteristics through a semantic analysis model.

[0132] The behavioral emotion analysis information acquisition module is used to acquire behavioral emotion analysis information based on the structured information of student status and teaching content through the core model of the thinking scheduling module.

[0133] The judgment module is used to determine whether the difference in the number of cycles between the current cycle and the previous optimized cycle reaches a preset threshold. If so, it obtains all historical high-quality classroom video annotations of efficient interactive segments between the current cycle and the previous optimized cycle, as well as behavioral and emotional analysis information for each cycle.

[0134] The optimization parameter acquisition module is used to generate optimization parameters for the visual perception module, semantic analysis module, and thought scheduling module based on the highly interactive segments annotated from all historical high-quality classroom videos and the behavioral and emotional analysis information for each cycle.

[0135] The model update module is used to update the visual perception model based on the optimized parameters of the visual perception module, update the semantic analysis model based on the optimized parameters of the semantic analysis module, and update the core model of the thought scheduling module based on the optimized parameters of the thought scheduling module.

[0136] This application constructs a self-evolving system that continuously improves perception accuracy, analysis precision, and scenario adaptability as teaching data accumulates without human intervention, completely solving the problems of static and unevolving traditional teaching analysis systems. At the same time, it uses multimodal data as its core support, breaking the limitations of single-data-dimensional analysis, and achieving accurate identification of core teaching states such as student focus, interaction effectiveness, and comprehension gaps. This effectively solves the industry problem of unstructured data being difficult to interpret intelligently during the teaching process. Moreover, the entire process is designed around the classroom teaching scenario, with all links closely aligned with the core teaching needs, taking into account both real-time feedback value and long-term evolutionary value. Attached Figure Description

[0137] Figure 1 This is a flowchart illustrating an embodiment of the AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanisms according to this application. Detailed Implementation

[0138] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0139] like Figure 1 The AI ​​teaching feedback and self-evolution methods shown are based on active perception and hybrid thinking mechanisms, including:

[0140] Step 1: Obtain the multimodal data to be identified for the current period;

[0141] Step 2: Based on the multimodal data to be identified in the current period, generate student behavioral features, emotional features, and teacher-student voice features through a visual perception model;

[0142] Step 3: Based on student behavioral characteristics, emotional characteristics, and teacher and student voice characteristics, generate structured information on student status and teaching content through a semantic analysis model;

[0143] Step 4: Based on the structured information of student status and teaching content, obtain behavioral emotion analysis information through the core model of the thinking scheduling module;

[0144] Step 5: Determine whether the difference in the number of cycles between the current cycle and the previous optimized cycle reaches the preset threshold. If so, obtain all historical high-quality classroom video annotations of efficient interactive segments between the current cycle and the previous optimized cycle, as well as behavioral and emotional analysis information for each cycle.

[0145] Step 6: Generate optimization parameters for the visual perception module, semantic analysis module, and thought scheduling module based on the highly effective interactive segments marked in all historical high-quality classroom videos and the behavioral emotion analysis information for each cycle;

[0146] Step 7: Update the visual perception model based on the optimized parameters of the visual perception module, update the semantic analysis model based on the optimized parameters of the semantic analysis module, and update the core model of the thought scheduling module based on the optimized parameters of the thought scheduling module.

[0147] In this embodiment, a cycle refers to a complete closed-loop unit in which the system completes one step (multimodal data acquisition and preprocessing), step 2 (visual perception feature generation), step 3 (structured analysis), and step 4 (behavioral emotion analysis). This can be understood as a complete classroom teaching analysis (e.g., one 45-minute class period constitutes one cycle, or a cycle can be divided into fixed durations (e.g., one hour), which can be configured according to the teaching scenario). After each cycle, the system outputs the corresponding behavioral emotion analysis information.

[0148] The optimization cycle refers to the cycle node in which the system executes steps 6 (generating optimization parameters) and 7 (updating the model), that is, the cycle in which the model optimization was last completed, which is denoted as the previous optimization cycle.

[0149] For example, suppose the configuration is as follows:

[0150] Preset threshold = 5 (at least 5 cycles between two optimizations);

[0151] The last optimization cycle = the 3rd cycle (that is, the system has already performed the model update in steps 6-7 after the 3rd cycle ended);

[0152] Current cycle = 8th cycle (the current classroom analysis is complete, and we are at the end of the 8th cycle).

[0153] Threshold determination: Calculate the difference between the current cycle (8) and the previous optimization cycle (3) = 5. This difference is equal to the preset threshold (5), which satisfies the triggering condition.

[0154] Data acquisition scope:

[0155] Behavioral sentiment analysis information for each period: Obtain all behavioral sentiment analysis information for periods 4, 5, 6, 7, and 8 (the period following the previous optimization period - the current period) (1 set for each period, 5 sets in total);

[0156] Highly interactive segments marked in high-quality historical classroom videos: Obtain the highly interactive segments that have been marked in advance in the classroom videos corresponding to cycles 4-8 (such as the highly interactive question-and-answer session in cycle 4, the highly interactive group discussion in cycle 6, etc.).

[0157] In this embodiment, step 1: obtaining the multimodal data to be identified in the current period includes:

[0158] Step 11: Obtain the raw video data for the current period;

[0159] Step 12: Run the YOLOv8 object detection algorithm. Run the YOLOv8 object detection algorithm on each frame of the video to detect the student's body, face, and hands. Filter out targets with a confidence score ≥ 0.8, record the target bounding box coordinates and corresponding confidence scores, and thus obtain the set of student target detection boxes.

[0160] Step 13: Based on the list of ROI regions from the iterative input, call the KCF correlation filter tracking algorithm to track the student targets in the student target detection box set in real time and generate the motion trajectory of each student; use the video frame sampler to extract the student's facial region, input the facial image into the pre-trained CNN model, identify the emotion information, and output the emotion label and confidence score corresponding to each frame, thereby obtaining the student target tracking trajectory and the real-time emotion label dataset.

[0161] In this embodiment, step 13 includes:

[0162] Initial ROI region list generation: Based on the student target detection box set output in step 12, this set contains the bounding box coordinates and corresponding confidence scores of three types of targets—student body, face, and hand—after being filtered by the YOLOv8 object detection algorithm (confidence ≥ 0.8). Each student's body bounding box is designated as the core ROI region (labeled ROI-Body), while the face bounding box (ROI-Face) and hand bounding box (ROI-Hand) within the body bounding box are designated as secondary ROI regions. An initial ROI region list is constructed according to the structure: student unique identifier (automatically assigned, based on the order of detection box appearance S1, S2…Sn) - core ROI - secondary ROI - confidence score. The input ROI tracked for the first time is this initial list.

[0163] ROI region list iterative update: During real-time tracking using the KCF algorithm, the system dynamically updates the ROI region list at 50ms intervals to ensure that the tracking adapts to changes in the student's movement state. The iterative update rules are as follows:

[0164] Tracking valid update: If the overlap (IoU) between the new target position output by the KCF algorithm and the ROI region of the previous frame is ≥0.7, the tracking is considered valid. The new position coordinates are updated to the core ROI (ROI-Body) and the auxiliary ROI (ROI-Face, ROI-Hand) of the corresponding student. The update timestamp is retained at the same time to form a new list of ROI regions for tracking input in the next frame.

[0165] Tracking offset correction: If the IoU is between 0.5 and 0.7, the tracking offset is determined. Based on the peak response of the KCF algorithm and the target motion trend, the ROI region is fine-tuned (the bounding box is expanded by 10% pixels), and then updated to the list after correction.

[0166] Lost Detection Supplement: If IoU < 0.5 or KCF response peak is lower than the preset threshold (0.6), target tracking is determined to be lost. The YOLOv8 target detection algorithm is immediately triggered to re-detect in the local area of ​​the current frame (centered on the ROI of the previous frame, expanding the range by 20%). If a matching target is detected (confidence ≥ 0.7), the ROI region list is updated with the re-detected bounding box. If no target is detected, the student target is marked as "temporarily lost", its historical ROI information is retained in the list, and re-detection continues for 3 seconds. If the target is not found within 3 seconds, it is removed from the list.

[0167] Adding new targets: In every 10-frame global detection cycle, if YOLOv8 detects a new student target that is not in the ROI list (confidence ≥ 0.8), it will assign a new unique identifier to it, add the core ROI and the auxiliary ROI to the list, and complete the iterative expansion.

[0168] KCF correlation filter tracking algorithm:

[0169] Tracking initialization: Call the KCF correlation filter tracking algorithm, take the initial / iteratively updated ROI region list as input, extract HOG features from the core ROI (ROI-Body) of each student target as the tracking template, initialize the filter parameters (kernel function type is Gaussian kernel, regularization parameter λ=0.001, learning rate η=0.05), and establish a dedicated tracker for each student target;

[0170] Real-time trajectory generation: For subsequent video frames, the KCF algorithm is used to calculate the correlation response value between each tracker and the corresponding region of the current frame, locate the real-time position of the target, and output the bounding box coordinates (x1, y1, x2, y2) of each student target; the bounding box coordinate changes of each student target are recorded in time stamp order (accurate to milliseconds) to form structured data containing "student unique identifier - timestamp - bounding box coordinates - tracking confidence (response peak)", which is the motion trajectory of each student;

[0171] Tracking quality assessment: For the tracking results of each frame, the tracking quality is assessed by two indicators: peak response and IoU stability. If the peak response is ≥0.6 and the IoU fluctuation is ≤0.2 for 5 consecutive frames, it is judged as stable tracking; otherwise, it is judged as unstable tracking, triggering offset correction or loss re-detection mechanism.

[0172] Student facial region capture and emotion recognition:

[0173] Precise facial region extraction: The video frame sampler is invoked to sample video frames at 100ms intervals (synchronized with the attention sampling rhythm in step 18); based on the subordinate ROI (ROI-Face) in the iteratively updated ROI region list, the student's facial region is precisely extracted from the sampled frames and uniformly normalized into a 224×224 pixel RGB image; if the facial image corresponding to ROI-Face has partial occlusion (obstruction area > 30% determined by edge detection), it is marked as an occluded face, and the confidence weight is reduced in subsequent emotion recognition.

[0174] Pre-trained CNN emotion recognition model: The pre-trained CNN model used is a model that has been pre-trained on the FER+ teaching scenario extended dataset and fine-tuned with classroom emotion annotation data of students in our school. The model structure consists of 5 convolutional layers + 2 fully connected layers + a softmax output layer. The specific inference process is as follows:

[0175] The captured facial image is input into the model. After normalization (pixel values ​​are mapped to [0,1]) and mean subtraction preprocessing, features such as facial texture and relative distance of key points are extracted through convolutional layers.

[0176] The convolutional features are mapped to a 7-dimensional emotion feature vector through a fully connected layer, corresponding to 7 core classroom emotion labels (happy, calm, confused, tired, bored, surprised, and fearful).

[0177] The confidence scores of various emotions are calculated using the Softmax activation function, and the emotion label with the highest confidence score is retained as the final emotion recognition result for that frame.

[0178] Construction of real-time emotion labeling dataset: The emotion recognition results of each frame are integrated into a structured format of student unique identifier - sampling frame number - timestamp - facial image path - emotion label - confidence score - occlusion mark to form a real-time emotion labeling dataset, ensuring that the emotion data and motion trajectory data of each student are accurately associated through student unique identifier + timestamp.

[0179] Step 14: Calculate the Laplacian variance of the student's facial image, and combine it with the confidence of the target detection box to select images with a confidence of <0.9 or a Laplacian variance of <100 as blurred images; run the ESRGAN super-resolution algorithm on the blurred images to enlarge them by 2-4 times, and simultaneously perform CLAHE histogram equalization to enhance image details, thereby obtaining clear images of the student's face and hands.

[0180] Step 15: Based on the student target detection box set and student target tracking trajectory, identify and label abnormal behaviors; integrate the emotion tags and abnormal behavior markers in the real-time emotion tag set, associate them with the corresponding student's clear image, and generate structured original behavioral and emotional features;

[0181] Step 16: Acquire the audio stream using an audio frame capture device, and divide the audio stream into frames with a frame length of 20ms and a step length of 10ms; run the MFCC algorithm on each frame of audio to extract the 13-dimensional Mel frequency cepstral coefficients, and construct an audio feature vector by combining the frame energy and zero-crossing rate, thereby obtaining the teacher and student speech feature vector sequence and the audio and video timestamp mapping table.

[0182] Step 17: Based on the audio and video timestamp mapping table, align the teacher's voice segments with the student behavior data; run the dynamic time warping algorithm to calculate the synchronization difference between the teacher's voice and the student's behavior. If the synchronization difference exceeds 1.5 seconds, it is determined to be a comprehension gap. Record the gap time period and the corresponding student group to obtain the comprehension gap determination result.

[0183] Step 18: Sample students' facial posture and eye opening / closing at 100ms intervals using a video frame sampler; use a circular buffer data structure to cache the sampling data of the most recent 5 seconds; calculate the attention continuity value through a 1-second sliding window, and determine that the attention continuity value is ≥0.6 as focused and <0.6 as distracted, generate attention state labels, and thus obtain a 5-second attention cache dataset;

[0184] Step 19: Normalize the original behavioral and emotional features and map them to the [0,1] interval; standardize the teacher and student speech feature vector sequence with Z-score; encode the courseware text with UTF-8 through a text parser to retain the effective teaching text content, thereby obtaining the multimodal data to be identified in the current period.

[0185] In this embodiment, step 2: generating student behavioral features, emotional features, and teacher-student voice features based on the multimodal data to be identified in the current period using a visual perception model includes:

[0186] Step 21: Input the set of student target detection boxes, student target tracking trajectory, and cleared images of student face and hand from the multimodal data to be identified in the current period into the trained visual behavior perception sub-model, extract the basic features of student behavior, and thus obtain the basic features of student behavior indexed by the student's unique identifier. The basic features of student behavior include the type of behavior action, the time node of the action, the duration of the action, and the location area of ​​the action.

[0187] Step 22: Input the sharpened student facial image, real-time emotion label dataset, and 5-second attention cache dataset from the multimodal data to be identified in the current period into the trained visual emotion perception sub-model to generate a student emotion basic feature set indexed by the student's unique identifier;

[0188] Step 23: Input the standardized teacher and student speech feature vector sequence, audio and video timestamp mapping table, and understanding tomographic judgment result from the multimodal data to be identified in the current period into the trained speech perception sub-model to generate the basic feature set of teacher and student speech;

[0189] Step 24: The student behavior feature set generated in Step 21 is structurally integrated. Using the student's unique identifier as the first index and the timestamp as the second index, the behavior type, duration, behavior switching interval, and abnormal attribute markers are arranged in chronological order to form structured student behavior features. The student emotion feature set generated in Step 22 is structurally integrated. Using the student's unique identifier as the first index and the timestamp as the second index, the emotion type, frequency of occurrence, emotion duration, attention continuity value, and attention state markers are arranged in chronological order to form structured student emotion features. The teacher-student speech feature set generated in Step 23 is structurally integrated. Using the audio and video timestamps as the index, the speech source, speech period, acoustic feature vector, and understanding discontinuity association markers are arranged in chronological order to form structured teacher-student speech features.

[0190] Step 25: Using the audio and video timestamp mapping table as a unified benchmark, accurately align the time dimensions of structured student behavior features, structured student emotion features, and structured teacher and student voice features. Associate and match student behavior and emotion features with corresponding teacher and student voice features under the same timestamp to generate student behavior features, emotion features, and teacher and student voice features.

[0191] In this embodiment, step 3: generating structured student state-teaching content information based on student behavioral characteristics, emotional characteristics, and teacher-student voice characteristics using a semantic analysis model includes:

[0192] Step 31: Input the structured student behavior features, structured student emotion features, and structured teacher-student speech features generated in Step 2, as well as the effective teaching text content encoded in UTF-8 in Step 19, into the trained semantic analysis model. Using audio and video timestamps as a unified benchmark, align the student behavior features, student emotion features, teacher-student speech features, and effective teaching text content with the time dimension. Construct a multimodal semantic input dataset containing unique student identifiers, timestamps, student behavior features, student emotion features, teacher-student speech features, and teaching text fragments. The teaching text fragments are the teaching content segments in the effective teaching text content that correspond to the current timestamp.

[0193] Step 32: Input the teacher and student speech features and teaching text fragments from the multimodal semantic input dataset into the trained teaching content semantic parsing sub-model. This sub-model is pre-trained based on teaching domain corpus and has the semantic recognition ability of teaching knowledge points, teaching links, and classroom interaction types. The model performs word segmentation, part-of-speech tagging, and syntactic analysis on the speech text content in the teacher and student speech features to extract core teaching semantic units, including knowledge point names, definitions, examples, teacher questions, student answers, and classroom instructions. At the same time, it matches teaching text fragments and labels the teaching stage corresponding to each semantic unit, such as new lesson introduction, knowledge instruction, classroom exercises, and summary review, generating a teaching content semantic feature set that includes timestamps, semantic unit types, teaching content text, and teaching stages.

[0194] Step 33: Input the student behavior features and student emotion features from the multimodal semantic input dataset into the trained student state semantic mapping sub-model. This sub-model is pre-trained based on classroom student state annotation samples and has the ability to correlate and map behavior, emotion, and learning state. The model integrates the behavior type, abnormal behavior marker, emotion type, attention continuity value, and attention state of the same student at the same time stamp, using the student's unique identifier as the dimension. Through the preset state mapping rules, the multi-dimensional features are transformed into standardized student learning states. The learning state types include focused understanding, confusion and hesitation, distraction, active interaction, and negative resistance. For each student, a student state semantic feature set containing the student's unique identifier, timestamp, learning state, and state confidence is generated in timestamp order. The state confidence is calculated based on the matching degree of behavior, emotion, and attention features.

[0195] Step 34: Input the semantic feature set of teaching content generated in Step 32 and the semantic feature set of student status generated in Step 33 into the trained state-content association integration sub-model. Using the timestamp as the association key, match the semantic units of teaching content under the same timestamp with the learning status of the corresponding students to establish a three-level association relationship of "teaching content - student group status - individual student status". For each semantic unit of teaching content, count the number and percentage of students in different learning statuses in the corresponding time period and mark the overall classroom status under the teaching content. At the same time, associate the comprehension gap judgment result in Step 17. If there is a comprehension gap in a certain teaching content time period, mark the association attribute between the teaching content and the comprehension gap, and generate a state-content association feature set containing teaching content text, teaching stage, overall classroom status, learning status of each student, status percentage, and comprehension gap association mark.

[0196] Step 35: Structure and regularize the state-content association feature set, using the semantic units of teaching content as the core index, arranged in chronological order of teaching stages. Each teaching content entry includes the corresponding timestamp interval, overall classroom status, student status distribution, comprehension gap markers, and detailed learning status and confidence level of each student. The regularized information is stored in a standardized data structure with fields including teaching content ID, teaching stage, teaching content text, timestamp interval, overall status, percentage of focused students, percentage of confused students, percentage of distracted students, comprehension gap markers, and a detailed list of student statuses. Finally, standardized student status-teaching content structured information is generated, which serves as the input data for the core model of the thinking scheduling module in Step 4.

[0197] In this embodiment, step 4: obtaining behavioral emotion analysis information through the core model of the thinking scheduling module based on the structured information of student status and teaching content includes:

[0198] Step 41: Input the structured information of student status and teaching content generated in Step 3 into the core model of the trained thinking scheduling module. Use the teaching content ID as the first index, the timestamp interval as the second index, and the student's unique identifier as the third index to complete the hierarchical sorting of the overall classroom status, student status distribution, individual student learning status, understanding fault markers and corresponding teaching content, and construct a hierarchical analysis input dataset containing core attributes of teaching content, group status attributes, individual status attributes, and fault correlation attributes.

[0199] Step 42: Input the individual student learning status and detailed student status list from the hierarchical analysis input dataset into the trained state deep analysis sub-model. This sub-model has the ability to deeply decompose learning status and verify confidence. The model uses the student's unique identifier as the dimension to deeply analyze the learning status of each student, decomposing it into behavioral dimension features (behavior type, duration, anomaly marker, behavior switching pattern), emotional dimension features (emotion type, intensity, fluctuation pattern, attention correlation features), and state dimension features (learning status type, state confidence, state duration, state switching node). At the same time, it verifies the rationality of the state confidence and removes invalid state data with confidence below the preset threshold, generating an individual state deep analysis feature set containing the student's unique identifier, timestamp, behavioral decomposition features, emotional decomposition features, state decomposition features, and valid state markers.

[0200] Step 43: Input the teaching content text, teaching stage, overall classroom status, and student status distribution from the hierarchical analysis input dataset, along with the individual status deep analysis feature set generated in Step 42, into the trained behavioral emotion association analysis sub-model. This sub-model has the ability to mine multi-dimensional associations between teaching content, behavior, and emotion. The model establishes an association link of "teaching content → group behavior and emotion status → individual behavior and emotion status" based on the timestamp interval. For each teaching content period, it analyzes the influence of different teaching content and different teaching stages on students' behavior types, emotion types, and learning status, and mines high-frequency behavioral emotion combinations (such as "knowledge teaching stage + confusion + looking down" and "questioning stage + positive emotion + raising hand"). At the same time, it calculates the association strength between each behavioral emotion combination and the learning status. The association strength is calculated based on the frequency of combination occurrence and the degree of status matching, generating a behavioral emotion association analysis feature set that includes the mapping relationship between teaching content text, teaching stage, behavioral emotion combination, association strength, corresponding student group, and learning status.

[0201] Step 44: Input the understanding gap markers, gap time periods, and corresponding student groups from the hierarchical analysis input dataset, along with the behavioral emotion association analysis feature set, into the trained behavioral emotion association analysis sub-model. For teaching content periods with understanding gaps, extract the behavioral decomposition features and emotional decomposition features of the corresponding student groups, analyze the association between understanding gaps and specific behavioral emotion features, identify gap-triggered behavioral emotion features (such as "understanding gap time period + confused emotion + abnormal sitting posture" and "understanding gap time period + distracted state + silent behavior"), and mark the proportion of abnormal behavioral emotions within the gap time period. Generate a gap-behavioral emotion association feature set containing understanding gap markers, gap time periods, triggering behavioral emotion features, abnormal proportions, and corresponding student groups.

[0202] Step 45: Input the behavioral-emotion association analysis feature set generated in Step 43 and the fault-behavioral-emotion association feature set generated in Step 44 into the structured sub-model of the trained analysis results. Using the teaching content ID as the core index, integrate the analysis results according to the time sequence of the teaching stage. For each teaching content item, integrate the group behavioral-emotional patterns, individual behavioral-emotional characteristics, high-frequency behavioral-emotional combinations, association strength, fault triggering characteristics, and abnormality percentage information. At the same time, supplement the behavioral-emotional analysis conclusions (such as "During the explanation of this knowledge point, the proportion of students' confused emotions is high, which easily triggers comprehension faults" and "Students' positive emotions are significant during the questioning process, and raising their hands is the core interactive behavior"). Mark the confidence of the analysis conclusions. The confidence is calculated based on the association strength and data validity.

[0203] Step 46: Standardize and structure the integrated analysis results to construct a standardized data structure that includes teaching content ID, teaching content text, teaching stage, timestamp interval, overall behavioral and emotional state, high-frequency behavioral and emotional combinations, correlation strength, individual behavioral and emotional details, understanding gap correlation characteristics, anomaly ratio, analysis conclusion, and conclusion confidence. Finally, structured behavioral and emotional analysis information is generated, which serves as the judgment basis for Step 5 and the optimization reference data for Step 6.

[0204] In this embodiment, step 6: generating optimization parameters for the visual perception module, semantic analysis module, and thought scheduling module based on the highly interactive segments annotated from all historical high-quality classroom videos and the behavioral emotion analysis information for each cycle includes:

[0205] Step 61: Construct a multi-dimensional optimized benchmark dataset. All historical high-quality classroom videos obtained in Step 5, including the high-efficiency interactive segments and behavioral emotion analysis information for each period, are linked using teaching content, teaching stage, and timestamp as unified references. This aligns the time dimension of the high-efficiency interactive segments with the behavioral emotion analysis information. Positive features are labeled for the high-efficiency interactive segments, including high-efficiency interaction type, effective student behavior, positive student emotions, successful teaching nodes, and teacher-student voice interaction features. Negative features are labeled for the behavioral emotion analysis information for each period, including low-focus behavior, high-confusion emotions, periods of comprehension gaps, and abnormal behavior-emotion combinations. This generates an optimized benchmark dataset containing positive features of high-efficiency interaction, negative features of problematic behavior and emotions, and features related to teaching content. Simultaneously, optimization priorities are assigned to each feature, with positive features of high-efficiency interaction having high priority and negative features of problematic behavior and emotions having secondary priority.

[0206] Step 62: Generate optimized parameters for the visual perception module. The visual perception module includes a trained visual behavior perception sub-model, a visual emotion perception sub-model, and a speech perception sub-model. Based on the optimized benchmark dataset, perform the following operations: For the visual behavior perception sub-model, statistically analyze the optimal range of target detection confidence, trajectory matching threshold of the KCF tracking algorithm, and threshold for abnormal behavior judgment in the positive features of efficient interaction. Combine this with false positive and false negative target detection data in the negative features of problem behavior emotion to adjust the confidence threshold of the YOLOv8 target detection algorithm, the filtering parameters of the KCF correlation filter tracking algorithm, and the abnormal behavior annotation rules, generating optimized parameters for target detection, tracking algorithm, and behavior recognition. For the visual emotion perception sub-model, statistically analyze the optimal value of emotion recognition confidence and the parameter range of facial image enhancement in the positive features of efficient interaction. Combine this with the optimal range of false positive and false negative target detection data in the negative features of problem behavior emotion to generate optimized parameters for target detection, tracking algorithm, and behavior recognition. For the emotion misclassification data, the classification threshold of the emotion recognition model, the magnification factor of the ESRGAN super-resolution algorithm, and the enhancement coefficient of CLAHE histogram equalization are adjusted to generate optimized parameters for emotion recognition and image enhancement. For the speech perception sub-model, the MFCC dimension, framing parameters, and speech emotion recognition confidence threshold of the speech feature extraction in the positive features of efficient interaction are statistically analyzed. Combined with the speech misclassification data in the negative features of problem behavior emotion, the extraction dimension of the MFCC algorithm, the audio framing parameters, and the classification threshold of the speech emotion recognition model are adjusted to generate optimized parameters for speech feature extraction and speech emotion recognition. The above sub-model optimization parameters are integrated to form an optimized parameter set for the visual perception module. The parameter set includes the threshold, coefficient, and rule parameters of each sub-model. Each parameter is labeled with the corresponding optimization basis, namely the statistical results of the positive features of efficient interaction or the correction results of the negative features of problem behavior emotion.

[0207] Step 63: Generate optimization parameters for the semantic analysis module. The semantic analysis module includes a trained sub-model for semantic parsing of teaching content, a sub-model for semantic mapping of student state, and a sub-model for integrating state-content association. Based on the optimized benchmark dataset, perform the following operations: For the semantic parsing sub-model of teaching content, calculate the accuracy of extracting core semantic units of teaching in the positive features of efficient interaction and the matching degree of teaching stage division. Combine this with the semantic mis-parsing data in the negative features of problem behavior and emotion, adjust the semantic word segmentation weights, knowledge point matching thresholds, and teaching stage judgment rules to generate semantic parsing weight parameters, knowledge point matching threshold parameters, and teaching stage division parameters. For the semantic mapping sub-model of student state, calculate the accuracy of the behavior-emotion-learning state mapping and the optimal interval of state confidence in the positive features of efficient interaction, and combine this with... For the state mismapping data in the negative features of problematic behavior and emotion, the weights of the state mapping rules and the coefficients for calculating state confidence are adjusted to generate state mapping weight parameters and confidence calculation optimization parameters. For the state-content association integration sub-model, the optimal value of the association strength between teaching content and student state in the positive features of efficient interaction and the threshold for judging the association fault are statistically analyzed. Combined with the association mismatch data in the negative features of problematic behavior and emotion, the coefficients for calculating association strength and the rules for marking the association fault are adjusted to generate association strength optimization parameters and association fault judgment parameters. The optimization parameters of the above sub-models are integrated to form the optimization parameter set of the semantic analysis module. The parameter set contains the weights, thresholds, and calculation coefficients of each sub-model, and each parameter is labeled with the corresponding optimization basis, that is, the comparison results between the positive features of efficient interaction and the negative features of problematic behavior and emotion.

[0208] Step 64: Generate optimization parameters for the thinking scheduling module. The core model of the thinking scheduling module includes a trained state deep analysis sub-model, a behavior-emotion association analysis sub-model, and an analysis result structured sub-model. Based on the optimization benchmark dataset, perform the following operations: For the state deep analysis sub-model, statistically analyze the state decomposition accuracy and the optimal value of the invalid state removal threshold in the positive features of efficient interaction. Combined with the state decomposition error data in the negative features of question behavior and emotion, adjust the state decomposition rules and the invalid state confidence removal threshold to generate optimized state decomposition parameters and invalid state removal parameters; For the behavior-emotion association analysis sub-model, statistically analyze the optimal value of the behavior-emotion combination association strength and the high-frequency combination matching rules in the positive features of efficient interaction. Combined with the question... For the negative features of question behavior and emotion, erroneous data in association analysis are analyzed, and the weights for calculating association strength and the thresholds for judging high-frequency combinations are adjusted to generate association analysis weight parameters and combination judgment optimization parameters. For the structured sub-model of the analysis results, the optimal interval of the confidence of the analysis conclusion and the matching degree of the structured fields in the positive features of efficient interaction are statistically analyzed. Combined with the erroneous data of the conclusions in the negative features of question behavior and emotion, the calculation coefficients of the conclusion confidence and the weights of the structured fields are adjusted to generate conclusion confidence optimization parameters and structured field adjustment parameters. The above sub-model optimization parameters are integrated to form the optimization parameter set of the thinking scheduling module. The parameter set contains the weights, thresholds, and calculation coefficients of each sub-model, and each parameter is labeled with the corresponding optimization basis, i.e., the feedback result of the positive features of efficient interaction.

[0209] Step 65: Optimize parameter verification and standardization integration. Perform consistency verification on the optimized parameters of the visual perception module generated in Step 62, the semantic analysis module generated in Step 63, and the thought scheduling module generated in Step 64. Verify the matching between the parameters of each module to ensure that there is no conflict between the behavior recognition threshold of the visual perception module and the state mapping rules of the semantic analysis module, and no conflict between the association strength parameters of the semantic analysis module and the association analysis weights of the thought scheduling module. Based on the proportion of positive features in the optimization benchmark dataset, adjust the optimization magnitude of each parameter. Increase the optimization weight of parameters corresponding to high-priority high-efficiency interactive positive features, and moderately adjust the parameters corresponding to secondary-priority problem behavior emotion negative features. Finally, generate standardized optimized parameters for the visual perception module, semantic analysis module, and thought scheduling module. The parameters are stored in a standardized data structure, including parameter name, parameter value, optimization basis, and applicable sub-model, which serves as the direct basis for model updates in Step 7.

[0210] In this embodiment, based on the three types of optimized parameter sets generated in step 6, targeted updates are completed for the visual perception model, semantic analysis model, and core model of the thought scheduling module, respectively, to ensure that the model performance adapts to the evolving needs of the teaching scenario, as detailed below:

[0211] Visual perception model updates: For visual behavior / emotion / speech perception sub-models, the target detection threshold, tracking algorithm parameters, emotion recognition threshold, and speech feature extraction coefficients in the visual perception module optimization parameter set are adapted to the feature extraction layer and classification layer of the corresponding sub-models, respectively. An incremental fine-tuning method is adopted, fixing the backbone network parameters and only updating the weights of the fully connected layer and rule decision module. After the update, the results are verified through a validation set: behavior recognition accuracy ≥ 92%, emotion recognition confidence ≥ 0.7, and speech source differentiation accuracy ≥ 95%. If the standards are not met, the parameters are backtracked and fine-tuned again.

[0212] Semantic analysis model update: The semantic parsing weights, state mapping rules, and association strength coefficients in the optimized parameter set of the semantic analysis module are injected into the teaching content semantic parsing, student state semantic mapping, and state-content association integration sub-models; the update is achieved by replacing the rule base and weight matrix inside the model. After the update, the following verification results are obtained: semantic unit extraction accuracy ≥90%, learning state mapping consistency ≥88%, and content-state association matching degree ≥91%, ensuring the accuracy of the analysis results.

[0213] The core model of the thinking scheduling module has been updated: the state decomposition rules, association analysis weights, and conclusion confidence coefficients in the optimization parameter set of the thinking scheduling module have been adapted to the sub-models of state depth analysis, behavior and emotion association analysis, and analysis results structured analysis. The update adopts parameter overlay, retains the model network structure, and only replaces the core calculation parameters. After the update, it has been tested in a simulated teaching scenario: the accuracy of behavior and emotion combination mining is ≥89%, and the confidence of analysis conclusions is ≥0.8, which meets the requirements for subsequent feedback generation and optimization triggering.

[0214] In this embodiment, the specific parameters of each model used in this application are as follows:

[0215] Visual behavior perception sub-model:

[0216] Input layer: Receives a set of student target detection boxes, student target tracking trajectories, sharpened images of students' hands and limbs, and abnormal behavior markers, with the student's unique identifier and timestamp as the data index dimension;

[0217] Feature extraction layer: divided into trajectory feature branch and image pose feature branch; trajectory feature branch extracts the change in the position of the detection box, the trajectory motion speed / direction, and trajectory continuity features; image pose feature branch extracts the relative position of hand joints, limb contour shape, and inter-frame difference features of the action.

[0218] Feature fusion layer: The element-wise addition fusion strategy is adopted to concatenate trajectory features and image pose features in the channel dimension to generate a unified behavior feature vector;

[0219] Behavior classification layer: Fully connected layer + Softmax activation function, outputting classification confidence scores for 7 types of behaviors: raising hands, writing, looking down, turning heads, standing, abnormal sitting posture, and operating electronic devices;

[0220] Feature statistics layer: Based on the classification results, it counts the timestamp of each behavior, duration, and behavior switching interval, and combines the abnormal behavior marker to complete the normal / abnormal attribute labeling;

[0221] Output layer: Generates a set of basic student behavior features indexed by the student's unique identifier, including behavior type, duration, switching interval, and anomaly marker.

[0222] 2. Visual Emotion Perception Sub-model:

[0223] Input layer: Receives sharpened student facial images, real-time emotion label dataset, and 5-second attention cache dataset, with facial image frames + timestamps as data units;

[0224] Facial feature extraction layer: Convolutional layer + pooling layer stacked, extracts facial texture, relative distance of key points, facial muscle deformation features, and outputs facial feature vector;

[0225] Emotion feature mapping layer: Fully connected layer + ReLU activation function, which maps facial feature vectors to confidence scores of 7 types of emotions: happy, calm, confused, tired, bored, surprised and fearful;

[0226] Emotion statistics layer: Integrate the emotion confidence scores of multiple frames at 100ms time intervals, calculate the emotion mean, variance, peak value, and trough value, and generate emotion intensity and fluctuation characteristics;

[0227] Attention Association Layer: Matches continuous attention values ​​with emotion features in the attention cache dataset to establish an emotion-attention correspondence;

[0228] Output layer: Generates a set of basic emotional features of students indexed by their unique identifiers, including emotion type, frequency, duration, attention continuity value, and attention state.

[0229] 3. Speech perception sub-model:

[0230] Input layer: Receives standardized teacher and student speech feature vector sequences, audio and video timestamp mapping tables, and understanding tomographic determination results, with 20ms audio frames as the basic unit;

[0231] Acoustic feature extension layer: First-order and second-order difference calculations are performed on the 13-dimensional Mel frequency cepstral coefficients (MFCC) to extend it into 39-dimensional MFCC features, and acoustic features such as fundamental frequency, formants, speech rate, and volume standard deviation are extracted simultaneously.

[0232] Speech recognition layer: Based on the Transformer encoder structure, it takes high-dimensional acoustic features as input and outputs speech text content, distinguishing the source of teacher / student speech through speaker coding;

[0233] Speech emotion recognition layer: Convolutional layer + bidirectional GRU layer stacked, extracts speech temporal features, and outputs the confidence scores of 6 types of speech emotions: calm, excited, questioning, encouraging, impatient and confused.

[0234] Fault correlation layer: Matches and understands fault determination results with speech segments, and marks the speech features corresponding to the fault time period;

[0235] Output layer: Generates a basic feature set of teachers' and students' speech, including speech source, speech period, acoustic features, text content, emotional features, and comprehension fault markers.

[0236] II. Semantic Analysis Model (including 3 trained sub-models)

[0237] 1. Sub-model for semantic parsing of teaching content:

[0238] Input layer: Receives speech text and effective teaching text content from the speech features of teachers and students, using timestamps as the correlation benchmark;

[0239] Text preprocessing layer: word segmentation, part-of-speech tagging, stop word removal, and generation of word vector sequences for teaching text;

[0240] Semantic encoding layer: BERT pre-trained encoder, extracts semantic features of text context, and outputs text semantic embedding vector;

[0241] Semantic Unit Extraction Layer: Fully connected layer + Conditional Random Field (CRF) to identify 6 types of core semantic units, including knowledge point names, definitions, examples, teacher questions, student answers, and classroom instructions.

[0242] Teaching stage division: Based on the temporal features of semantic units, rule matching is used to determine four types of teaching stages: new lesson introduction, knowledge instruction, classroom practice, and summary review.

[0243] Output layer: Generates a semantic feature set of teaching content, including timestamps, semantic unit types, teaching content text, and teaching stage.

[0244] 2. Student State Semantic Mapping Sub-model:

[0245] Input layer: Receives student behavioral characteristics and student emotional characteristics, with student unique identifier + timestamp as the data dimension;

[0246] Multi-feature integration layer: splices together behavior type, anomaly marker, emotion type, and continuous attention value features to generate a comprehensive feature vector of student status;

[0247] State mapping layer: Fully connected layer + Sigmoid activation function, which maps the comprehensive features into confidence levels of 5 learning states: focused understanding, confused and hesitant, distracted, active interaction, and negative resistance;

[0248] Confidence calculation layer: Based on the matching degree of behavior-emotion-attention features, the state confidence is calculated by weighting and invalid data with confidence below the threshold is removed;

[0249] Output layer: Generates a semantic feature set of student status, including student unique identifier, timestamp, learning status, and status confidence.

[0250] 3. State-Content Association Integration Sub-Model:

[0251] Input layer: Receives the semantic feature set of teaching content, the semantic feature set of student status, and the comprehension fault determination result;

[0252] Time alignment layer: Using timestamps as the association key, it achieves precise matching between teaching content and student status;

[0253] Group Status Statistics Layer: Statistically analyzes the number and percentage of students in different learning states during each teaching content time period to determine the overall classroom status;

[0254] Fault-related layer: Mark the teaching content and student status corresponding to the fault period to understand the fault-content-status relationship;

[0255] Association Integration Layer: Constructs a three-level association structure of "teaching content - group status - individual status" and generates association feature vectors;

[0256] Output layer: Generates a state-content related feature set, including teaching content text, teaching stage, overall state, student state distribution, and comprehension fault markers.

[0257] III. Core Model of the Thinking Scheduling Module (including 3 trained sub-models)

[0258] 1. State Depth Analysis Sub-model:

[0259] Input layer: Receives structured information on student status and teaching content, indexed by teaching content ID + timestamp + student identifier;

[0260] State decomposition layer: The learning state is decomposed into three types of sub-features: behavioral dimension (behavior type, duration, switching pattern), emotional dimension (emotion type, intensity, fluctuation), and state dimension (state type, confidence level, duration).

[0261] Confidence verification layer: Based on preset threshold rules, it verifies the rationality of the confidence level of the status and marks the valid / invalid status data;

[0262] Feature normalization layer: unifies the data format of decomposed features and generates standardized decomposed feature vectors;

[0263] Output layer: Generates a deep analysis feature set of individual states, including student identifiers, timestamps, behavioral / emotional / state decomposition features, and valid state markers.

[0264] 2. Behavioral-emotion correlation analysis sub-model:

[0265] Input layer: Receives teaching content features, group state features, individual state in-depth analysis features, and comprehension discontinuity features;

[0266] Link Construction Layer: Establish a temporal link of "teaching content → group behavior and emotion → individual behavior and emotion";

[0267] Combination Mining Layer: Based on the frequent pattern mining algorithm, identify high-frequency behavioral emotion combinations (such as "knowledge teaching + confusion + looking down"), and calculate the correlation strength between the combination and the learning state.

[0268] Fault Correlation Analysis Layer: Extracts behavioral and emotional features during fault periods, identifies fault-triggered features, and calculates the proportion of abnormal behavior and emotions;

[0269] Output layer: Generates behavioral-emotion association analysis feature set and fault-behavioral-emotion association feature set, including combination type, association strength, corresponding student group, and triggering features;

[0270] 3. Structured sub-model of analysis results:

[0271] Input layer: Receives the behavioral-emotion correlation analysis feature set and the fault-behavioral-emotion correlation feature set;

[0272] Results Integration Layer: Taking the teaching content ID as the core, it integrates group patterns, individual characteristics, combined information, and discontinuity characteristics according to the teaching stage sequence;

[0273] Conclusion generation layer: Fully connected layer + logistic regression, based on association strength and data validity, to generate behavioral sentiment analysis conclusions and conclusion confidence levels;

[0274] Structured regularization layer: Data formatting is completed according to standardized fields (teaching content ID, text, stage, status distribution, combination, fault features, conclusion, confidence level);

[0275] Output layer: Generates structured behavioral sentiment analysis information, which serves as the core data for subsequent optimization and feedback;

[0276] Basic algorithm model:

[0277] 1. YOLOv8 object detection model:

[0278] Input layer: Receives raw video frame images;

[0279] Backbone network layer: C2f modules are stacked to extract multi-scale image features and output feature maps at three different scales;

[0280] Neck network layer: PANet structure, which integrates multi-scale features to enhance the detection capability of small targets (hands, face);

[0281] Detection head layer: Couple head structure, output target category (student body, face, hand), bounding box coordinates, and confidence score;

[0282] Filtering layer: Retain targets with a confidence level ≥ 0.8 and generate a set of student target detection boxes.

[0283] 2. KCF correlation filter tracking model:

[0284] Input layer: Receives the initial position of the student target detection box and subsequent video frames;

[0285] Feature extraction layer: Extracts HOG features from the target region and generates feature vectors;

[0286] Filtering training layer: Based on the cyclic matrix, train kernel correlation filters to learn the target appearance model;

[0287] Tracking and prediction layer: In subsequent frames, the target is located using the filter response value, the target position is updated, and a motion trajectory is generated;

[0288] Output layer: Outputs the real-time tracking trajectory of each student.

[0289] 3. ESRGAN super-resolution model:

[0290] Input layer: Receives blurred images of students' faces / hands;

[0291] Shallow feature extraction layer: Convolutional layer, extracts shallow texture features of the image;

[0292] Residual Dense Block (RRDB) layer: Multiple layers of residual connections + dense connections to extract deep high-resolution features;

[0293] Upsampling layer: Subpixel convolutional layer, which magnifies low-resolution feature maps by 2-4 times;

[0294] Output layer: Generates sharpened, high-resolution facial / hand images.

[0295] 4. Pre-trained CNN emotion recognition model:

[0296] Input layer: Receives student facial images;

[0297] Convolutional layers: Multiple convolutional layers + pooling to extract facial key points, textures, and deformation features;

[0298] Fully connected layer: maps convolutional features to emotion feature vectors;

[0299] Classification layer: Softmax activation function, outputting 7 categories of sentiment labels and confidence scores;

[0300] Output layer: Generates a real-time sentiment label dataset.

[0301] 5. MFCC speech feature extraction model:

[0302] Input layer: Receives audio streams and divides them into frames with a 20ms frame length and a 10ms step size;

[0303] Pre-emphasis layer: High-pass filter to enhance high-frequency components of audio;

[0304] Fourier transform layer: converts time-domain audio into frequency-domain spectrum;

[0305] Mel filter bank layer: maps the spectrum to the Mel scale and extracts Mel spectral features;

[0306] Discrete Cosine Transform Layer: Converts the Mel spectrum into 13-dimensional MFCC features;

[0307] Output layer: Generates a sequence of voice feature vectors for teachers and students.

[0308] 6. Dynamic Time Warping (DTW) Model:

[0309] Input layer: Receives teacher speech feature sequences and student behavior feature sequences;

[0310] Distance calculation layer: Calculates the Euclidean distance between speech and behavioral features, and constructs the distance matrix;

[0311] Regular path search layer: Dynamic programming algorithm to find regular paths with minimum cumulative distance;

[0312] Synchronization difference calculation layer: Based on the normalized path, calculates the synchronization difference between speech and behavior;

[0313] Output layer: Determine the comprehension gap (synchronization difference > 1.5 seconds), record the gap period and the student group.

[0314] 7. Attention computation model:

[0315] Input layer: Receives sampled data of students' facial posture and eye opening / closing (at 100ms intervals).

[0316] Cache layer: A circular buffer that stores the sampled data from the last 5 seconds;

[0317] Sliding window computation layer: A 1-second sliding window is used to extract facial pose stability and mean features of eye opening and closing.

[0318] Attention scoring layer: Weighted calculation of pose and eye features to generate continuous attention values;

[0319] Judgment layer: Using a threshold of 0.6, determine the state of focus / distraction and generate attention state labels;

[0320] Output layer: Generates a 5-second attention cache dataset.

[0321] This application also improves several of the models, as follows:

[0322] The visual emotion perception sub-model includes:

[0323] Input layer:

[0324] The system receives the following from the multimodal data to be identified in step 1: a sharpened student facial image, a real-time emotion label dataset, a 5-second attention cache dataset, an image quality parameter set, and a unique student identifier with a timestamp.

[0325] Clarify student facial images: standardize the size to 224×224×3, RGB format;

[0326] Real-time emotion label dataset: Step 1: The raw confidence scores of 7 emotion categories (happiness, calmness, confusion, fatigue, boredom, surprise, and fear) output by the pre-trained CNN, which serve as weak supervision signals for the model;

[0327] 5-second attention buffer dataset: continuous attention values, eye opening and closing, and facial pose sampling data (100ms interval, 50 sampling points in total) in a circular buffer.

[0328] Image quality parameter set: Laplacian variance of facial images and YOLOv8 object detection confidence calculated in step 1, used for feature robust completion;

[0329] Student unique identifier + timestamp: as a data index, ensuring that output features are accurately associated with students and time, and connecting to subsequent structured processing.

[0330] Image quality adaptive preprocessing layer:

[0331] The core process completes image filtering and feature completion initialization, without feature extraction; it only performs data preprocessing for subsequent layers to adapt to the image sharpening results from step 1.

[0332] Set a dual threshold screening rule: images with a Laplacian variance ≥ 100 and a detection confidence ≥ 0.9 are considered high-quality images and directly enter the feature extraction layer; images with a Laplacian variance < 100 or a detection confidence < 0.9 are considered low-quality images and are assigned a quality weight coefficient (0.3-0.7, calculated linearly based on the Laplacian variance and confidence).

[0333] Pixel-level weighted initialization for low-quality images: The pixel mean of high-quality images is used as the pixel completion benchmark for low-quality images. Pixel-level correction is completed by combining the quality weight coefficient, thus avoiding feature distortion caused by blurry images directly entering the feature layer.

[0334] Output: Standardized facial image (224×224×3), quality weight coefficient, and association index of student unique identifier + timestamp.

[0335] Multi-branch fine-grained visual feature extraction layer:

[0336] As one of the core innovation layers, it adopts a three-branch parallel extraction + feature normalization approach, based on MobileViT's lightweight convolutional blocks (replacing the original ViT's large attention to adapt to real-time classroom scenarios), to extract the core visual features related to students' emotions in the classroom. The feature output dimension of all branches is unified to 256 dimensions, facilitating subsequent fusion.

[0337] Facial micro-expression feature branch: It consists of 4 layers of depthwise separable convolutions + 2 layers of MobileViT small-scale attention blocks, with strides of 1, 1, 2, and 2 respectively. It extracts texture and deformation features of key micro-expression areas such as brow bone, corner of mouth, and nostril (e.g., brow bone furrowing when confused, corner of mouth turning down when annoyed), and outputs a 256-dimensional micro-expression feature vector. This branch is the core branch, with an initial weight of 0.5.

[0338] Eye attention feature branch: Based on facial key point detection (68 points), the eye region (36-41 points for the left eye and 42-47 points for the right eye) is located. The features of eye opening and closing, eye movement direction and blink frequency are extracted through 3-layer depthwise separable convolution. Combined with the eye opening and closing sampling values ​​in the 5-second attention cache dataset, a 256-dimensional eye attention feature vector is output with an initial weight of 0.3.

[0339] Head pose feature branch: By solving the three-dimensional pose matrix of facial key points, the head pitch angle, yaw angle and roll angle features are extracted. Combined with the head movement trend in the student target tracking trajectory in step 1, a 256-dimensional head pose feature vector is output, with the initial weight set to 0.2.

[0340] Feature normalization: L2 normalization is performed on the feature vectors of the three branches to eliminate the difference in dimensions, and the micro-expression feature vector, eye attention feature vector, head posture feature vector, and the initial weights of each branch are output.

[0341] Attention-Gated Feature Fusion Layer (AGF):

[0342] The core functionality involves adaptive fusion of three-branch features, replacing the traditional simple concatenation. It introduces an attention gating mechanism to dynamically adjust the feature weights of the three branches based on the input image quality parameters and attention cache data, thus addressing the issue of varying feature importance under different image qualities and attention states.

[0343] The average attention continuity value in the 5-second attention cache dataset is used as the gating trigger signal: if the attention continuity value is ≥0.6 (focused state), the weight of the micro-expression feature branch is increased and the weight of the head posture feature branch is decreased; if the attention continuity value is <0.6 (distracted state), the weight of the head posture and eye attention feature branches is increased and the weight of the micro-expression feature branch is decreased.

[0344] Using the quality weight coefficients in the image quality parameter set as feature weighting coefficients: multiply the feature vectors of low-quality images by the quality weight coefficients, and keep the weights of the feature vectors of high-quality images at 1 to complete feature robust completion;

[0345] Feature fusion: The dynamically weighted three-branch feature vectors are concatenated along the channel dimension, and feature dimensionality reduction and fusion are performed through a 1×1 convolution layer to output a 512-dimensional multimodal visual fusion feature vector. This vector contains coupled features of the face, eyes, and head, providing a foundation for subsequent temporal modeling.

[0346] Temporal attention bidirectional GRU temporal modeling layer:

[0347] The core solution addresses the limitations of traditional single-frame emotion recognition by capturing the temporal continuity and gradual changes in students' emotions in the classroom, adapting to a 5-second attention buffer time window, and perfectly synchronizing with the attention data collection rhythm in step 1.

[0348] Input: Using 100ms as the time step, arrange the multimodal visual fusion feature vectors within a 5-second time window in chronological order to form a 50×512 temporal feature sequence (corresponding to the 50 attention sampling points in step 1).

[0349] Bidirectional GRU layer: Set up 2 bidirectional GRU layers with a hidden layer dimension of 256. The forward GRU captures the positive trend of emotion change (such as from calm to confusion), and the backward GRU captures the negative trend of emotion change (such as from confusion to calm). The output is a 50×512 temporal correlation feature sequence.

[0350] Temporal attention layer: Introduces a temporal attention mechanism to weight the feature sequences of 50 time steps, highlighting the feature weights of key frames with sudden emotional changes (such as frames that suddenly change from focus to confusion), suppressing redundant frame features without emotional changes, and outputting a 256-dimensional temporal fusion emotional core feature vector.

[0351] Output: This vector contains the "static features + dynamic trend" of students' emotions within a 5-second time window, which not only retains the core features of emotions in a single frame, but also incorporates the temporal correlation information of emotions.

[0352] Attention-Emotion Dual-Label Cross-Verification Layer:

[0353] The core is soft constraint verification, which uses the attention data from step 1 as a constraint for emotion recognition, eliminating contradictory results, improving feature effectiveness, and performing pre-filtering for the state mapping of the subsequent semantic analysis model:

[0354] Input: Temporal fusion core feature vector of emotion, mean of continuous attention values ​​of 5-second attention cache dataset, attention state label (focused / distracted);

[0355] Preliminary emotion classification: A 256-dimensional feature vector is mapped to a confidence score (Softmax activation) for 7 categories of classroom student emotions through a fully connected layer, generating preliminary emotion classification results;

[0356] Mutual verification rule determination: Three core verification rules are set (which can be fine-tuned according to the teaching scenario). If a rule is violated, the result is marked as invalid emotional and corrected based on the attention state.

[0357] Rule 1: If the mean continuous attention score is ≥0.6 (focus), the confidence score for "boredom, fatigue, confusion" needs to be reduced by 30%.

[0358] Rule 2: If the mean continuous attention score is <0.6 (distraction), the confidence score for "calm, happy, focused" needs to be reduced by 30%.

[0359] Rule 3: If the head posture is "head down > 45°" and the eye opening angle is < 0.2, it is directly judged as "distraction-related emotion", and only the two emotion results of "annoyance and fatigue" are retained;

[0360] Output: Adjusted confidence scores for 7 emotion categories, valid / invalid emotion labels, and a correlation mapping table between the mean of attention continuous values ​​and emotion confidence scores.

[0361] Adaptive adjustment layer during the teaching phase:

[0362] For the forward-looking innovation layer, an interface is reserved for integration with the semantic parsing model of subsequent teaching content. This enables the prior association between emotional features and teaching content without altering the original data flow. Weights are dynamically adjusted only within the model. If teaching stage information is not yet available, the output will use the initial weights.

[0363] Interface input: Teaching stage markers (new lesson introduction, knowledge instruction, classroom questioning, classroom exercises, summary and review) output by the semantic parsing sub-model of subsequent teaching content;

[0364] Dynamic weight adjustment: Based on the characteristics of the teaching stage, five sets of feature weight adjustment coefficients are preset to adjust the revised emotion confidence score by weighting it. For example:

[0365] During the classroom questioning phase: Increase the weight given to "surprise" and "happiness" to match students' emotional responses when being asked questions;

[0366] During the knowledge delivery phase: increase the weight given to "confusion and calmness" to match the core emotions of students listening to the lecture;

[0367] During the classroom practice phase: increase the weight given to "fatigue and confusion" to match the emotional characteristics of students when doing exercises;

[0368] Output: Emotional confidence score after adaptation to the teaching stage, teaching stage marker, student unique identifier + timestamp.

[0369] Emotion Feature Structured Output Layer:

[0370] The core function is to statistically analyze and structure emotional features. The output feature set fully meets the structured emotional feature requirements of step 2 and can be directly used as input to the semantic analysis model without additional feature transformation. The fields are fully compatible with the original technical solution.

[0371] Emotional characteristics statistics: At 100ms intervals, integrate the emotional confidence scores after the adaptation of teaching stages within a 5-second time window, and calculate the emotional intensity characteristics (mean and peak value of each emotion type) and emotional fluctuation characteristics (variance and trough value of each emotion type).

[0372] Attention-Emotion Association Labeling: Associate the mean of continuous attention values ​​and attention state labels with emotion features to label the attention attributes corresponding to each emotion type;

[0373] Structured encapsulation: Using the student's unique identifier as the first index and the timestamp as the second index, the feature fields are encapsulated according to the structured requirements of step 2, and the final output is a structured student emotion feature set. The fields include: student unique identifier, timestamp, emotion type, mean emotion intensity, peak emotion intensity, variance of emotion fluctuation, mean of continuous attention value, attention state label, valid emotion result label, and teaching stage label.

[0374] This model breaks through the adaptation bottleneck of traditional general emotion recognition models in classroom scenarios. Through three core innovations—multi-feature coupling, temporal modeling, and scenario adaptation—it improves the accuracy of student emotion recognition in the classroom by more than 30%, while also ensuring real-time performance (single-frame inference ≤20ms) to meet the needs of real-time teaching feedback in the classroom. In addition, the model's forward-looking teaching stage adaptation design provides pre-feature support for the subsequent deep correlation analysis between student emotions and teaching content, making subsequent semantic analysis and behavioral emotion analysis more in line with actual teaching. Ultimately, it provides more accurate and valuable student status data for AI teaching feedback, while also providing richer and more effective feature optimization basis for the model's self-evolution.

[0375] In this embodiment, the student state semantic mapping sub-model includes:

[0376] Input layer:

[0377] Receive standardized data output from the preceding steps:

[0378] Core inputs: Structured student behavior features (unique student identifier, timestamp, behavior type, duration, switching interval, anomaly marker) and structured student emotion features (unique student identifier, timestamp, emotion type, mean intensity, variance of fluctuation, continuous attention value, attention state) generated in step 2.

[0379] Related inputs: the semantic feature set of teaching content generated in step 3 (time stamp, teaching content text, knowledge point type, teaching stage, semantic unit type), and the understanding gap determination results in step 17 (gap time period, corresponding student group).

[0380] Index input: unique student identifier + timestamp, ensuring that the status mapping results are accurately associated with students, teaching content, and time, and connecting to subsequent structured processing.

[0381] Multimodal feature preprocessing and normalization layer:

[0382] Standardize the input features to eliminate dimensional differences, preparing for subsequent fusion modeling:

[0383] Behavioral feature normalization: the behavior type (raising hand, writing, bowing head, etc.) is converted into one-hot encoding, the duration of behavior and the switching interval are mapped to the [0,1] interval, and the abnormal label is converted into a binary feature of 0 (normal) / 1 (abnormal);

[0384] Emotional feature normalization: Emotional types (happiness, calmness, confusion, etc.) are converted into one-hot codes, and the mean and variance of emotional intensity are mapped to the [0,1] interval, while the continuous values ​​of attention are directly retained as the original [0,1] values;

[0385] Teaching content feature encoding: The teaching stage (new lesson introduction, knowledge instruction, etc.) and knowledge point type (concept, formula, example) are converted into one-hot encoding. The teaching content text is converted into a 512-dimensional semantic embedding vector through pre-trained word vectors (specific to the teaching domain).

[0386] Output: Standardized behavioral feature vector (128-dimensional), emotion feature vector (128-dimensional), teaching content semantic vector (512-dimensional), attention feature vector (64-dimensional), and student unique identifier + timestamp index.

[0387] Multimodal Feature Coupling and Fusion Layer (MCF):

[0388] This approach achieves deep coupling between behavior, emotion, attention, and teaching content, replacing the traditional simple feature concatenation. It introduces a feature interaction gating mechanism to uncover the relationships between features.

[0389] Feature interaction branches: Construct behavior-emotion interaction branches, behavior-attention interaction branches, and emotion-attention interaction branches respectively. Extract the interaction features of pairwise features through element-wise multiplication and fully connected layers. Each interaction branch outputs a 64-dimensional interaction feature vector.

[0390] Teaching content perception branch: The semantic vector of teaching content and the interaction feature vector are concatenated, and the association features between teaching content and student features are extracted through a 1×1 convolution layer + Leaky ReLU activation, and a 256-dimensional teaching content-student feature association vector is output.

[0391] Feature fusion gating: A gating mechanism is introduced to dynamically adjust the feature weights of each branch based on the teaching stage (e.g., increase the weight of the behavior-emotion interaction branch during the classroom questioning stage, and increase the weight of the teaching content perception branch during the knowledge teaching stage). All branch features are fused by gating weights to output a 512-dimensional multimodal coupled feature vector, which contains information on the deep correlation between student behavior, emotion, attention and teaching content.

[0392] Temporal Dynamic Modeling Layer (TDM):

[0393] The core solution addresses the limitations of traditional static mapping by capturing the temporal changes in the learning state, ensuring complete synchronization with the 5-second attention buffer in step 1 and the temporal features in step 2.

[0394] Temporal feature sequence construction: Using 100ms as the time step, the multimodal coupled feature vectors within the 5-second time window are arranged in chronological order to form a 50×512 temporal coupled feature sequence (corresponding to the 50 attention sampling points in step 1).

[0395] Temporal memory gate module: Set up 2 layers of gated recurrent units (GRU), introduce temporal memory gate, retain the feature memory of previous time steps, capture the gradual trend of learning state (such as the continuous change from focus to confusion), and output a 50×256 temporal dynamic feature sequence;

[0396] Temporal attention weighting: A temporal attention mechanism is introduced to weight the feature sequences of 50 time steps, highlighting the feature weights of key time steps with sudden state changes (such as time steps with sudden changes in behavior / emotion), suppressing redundant time step features without state changes, and outputting a 256-dimensional temporal dynamic core feature vector, which integrates static coupling features and temporal dynamic change information.

[0397] Adaptive Adjustment Layer (TSAR) during the Teaching Phase:

[0398] To achieve dynamic adaptation between teaching stages and state mapping, based on the semantic features of teaching content, the feature mapping weights are dynamically adjusted to match the state determination logic of different teaching stages:

[0399] Teaching Stage Weighting Library: Five sets of exclusive weighting coefficients for each teaching stage are preset (corresponding to new lesson introduction, knowledge instruction, classroom questioning, classroom practice, and summary review). Each set of weights includes behavioral characteristic weights, emotional characteristic weights, attention characteristic weights, and temporal characteristic weights.

[0400] Dynamic weight matching: Based on the input teaching stage markers, the corresponding weight coefficients are retrieved from the weight library to adjust the weighted dynamic core feature vector of the time series. For example:

[0401] During the classroom questioning phase: weights for behavioral characteristics (0.4), emotional characteristics (0.3), attentional characteristics (0.2), and temporal characteristics (0.1) are used to highlight the mapping effect of interactive behaviors such as raising hands and answering questions;

[0402] Knowledge delivery stage: weight of behavioral characteristics (0.2), weight of emotional characteristics (0.3), weight of attention characteristics (0.4), weight of temporal characteristics (0.1), highlighting the mapping effect of emotions such as focus and confusion on attention;

[0403] Output: Temporal dynamic feature vector (256-dimensional) adapted to the teaching stage, teaching stage label, and corresponding teaching content text.

[0404] Dynamic Confidence Calibration Layer (DCC):

[0405] The core innovation layer enables dynamic calculation of learning state confidence and correction of contradictory results, replacing the traditional fixed threshold judgment and improving the effectiveness of state results.

[0406] Preliminary state mapping: Through a fully connected layer + Softmax activation, the temporal dynamic feature vector adapted to the teaching stage is mapped to the preliminary confidence scores of 5 learning states (focused understanding, confused and hesitant, distracted, active interaction, and negative resistance).

[0407] Confidence score calculation across multiple dimensions: The final confidence score is dynamically calculated based on three dimensions, and the weight of each dimension can be optimized through self-evolution.

[0408] Feature matching degree: The degree of matching between behavioral, emotional, and attentional features and learning state (e.g., the matching degree between "raising hand behavior + happy emotion" and "positive interaction").

[0409] Temporal consistency: The continuity between the current state and the state 4 seconds prior (e.g., if one is focused for 4 consecutive seconds and the current state is confused, then the temporal consistency is low).

[0410] Adaptability of teaching content: The degree of adaptation between the current state and the teaching content and teaching stage (e.g., "knowledge instruction stage + distracted state" has low adaptability).

[0411] Conflicting Result Correction: Three core correction rules are set. If the initial state result violates the rules, the confidence level of the corresponding state is lowered, and states with higher matching degree are selected first.

[0412] Rule 1: If the attention span score is ≥0.8 and there is no abnormal behavior, the confidence level for "distraction, inattentiveness, and passive resistance" is reduced by 50%.

[0413] Rule 2: If there are positive behaviors such as raising hands and answering questions, the confidence level for "passive resistance and distraction" is reduced by 40%.

[0414] Rule 3: During periods of comprehension gap, the confidence level of "focused comprehension" is reduced by 60%, and "confusion, hesitation, and distraction" are prioritized.

[0415] Output: final confidence scores for 5 learning states, valid / invalid state labels, and the basis for confidence calculation (feature matching degree, temporal consistency, and teaching content suitability).

[0416] Learning state classification and structured output layer:

[0417] After completing state classification and structured encapsulation, the output student state semantic feature set is fully compatible with the original technical solution, requiring no additional transformation.

[0418] State classification determination: Select the learning state with the highest final confidence score as the student's current learning state. If the highest confidence score is less than the preset threshold (0.5), it is marked as "unknown state".

[0419] Status characteristic statistics: In order of timestamp, the duration of each learning status, the number of status transitions, and the mean status confidence of each student in the current period are statistically analyzed.

[0420] Association annotation: Associate the learning status with the teaching content text, teaching stage, and comprehension gap markers, and annotate the teaching scenario attributes corresponding to the status;

[0421] Structured encapsulation: Using the student's unique identifier as the first index and the timestamp as the second index, the feature fields are encapsulated according to the requirements of the original technical solution. The final output is a semantic feature set of student status, with fields including: student unique identifier, timestamp, learning status, status confidence, status duration, teaching content text, teaching stage, understanding discontinuity association marker, and status validity marker.

[0422] This model overcomes the limitations of traditional student state mapping models, which are "static, singular, and detached from the teaching scenario." Through four core innovations—multimodal coupling, temporal dynamic modeling, teaching content perception, and dynamic confidence calibration—it improves the accuracy of classroom student learning state mapping by more than 40%, while significantly reducing the proportion of invalid state results. The model's adaptive design for the teaching stage makes the state mapping results more closely match the actual teaching situation, providing more accurate and valuable basic data for subsequent correlation analysis of "student state-teaching content" and behavioral emotion analysis. In addition, the model's parameter optimization design allows it to continuously evolve with the accumulation of classroom data, constantly adapting to the teaching characteristics of different subjects, different grade levels, and different teachers. Ultimately, it provides AI teaching feedback with more intelligent and personalized student state analysis capabilities, helping to continuously optimize teaching effectiveness.

[0423] In this embodiment, the semantic parsing sub-model for teaching content includes:

[0424] First layer: Multi-source text temporal alignment layer:

[0425] Input: The teacher-student voice text sequence, effective teaching text content, and audio / video timestamp mapping table entered in step 3;

[0426] Function: Based on audio and video timestamps, the system performs time-series slicing and alignment of audio text (teacher questions / explanations, student answers) and courseware text (knowledge points, examples), removes redundant text without pedagogical semantics, and generates a standardized time-series text sequence of "timestamp-text fragments" to solve the problem of time-series misalignment between audio and courseware text.

[0427] Output: 512-dimensional time-aligned text feature sequence (divided into 100ms time steps, covering the current teaching cycle).

[0428] Second layer: Knowledge graph-enhanced semantic encoding layer:

[0429] Input: The time-aligned text feature sequence output from the first layer, and a lightweight instructional knowledge graph (including knowledge point hierarchy, relationships, and teaching stage attributes);

[0430] Functionality: Employs a lightweight Transformer encoder (only 2 layers) to extract text context semantics, while fusing knowledge point embedding vectors from the knowledge graph with text features to strengthen the semantic association between knowledge points, definitions, and examples, generating enhanced semantic feature vectors that integrate "text semantics + knowledge context," thus avoiding isolated analysis of teaching content;

[0431] Output: 256-dimensional knowledge-enhanced semantic feature vector.

[0432] Third layer: Temporally-aware semantic unit extraction layer:

[0433] Input: Knowledge-enhanced semantic feature vectors from the second layer output, and time-aligned text sequences;

[0434] Function: Introducing a temporal attention mechanism to focus on the temporal changes of teaching content (such as the switch from "definition explanation" to "example explanation"), accurately identifying 6 types of core semantic units (knowledge point name, definition explanation, example explanation, teacher question, student answer, classroom instruction) through Conditional Random Field (CRF), and marking the start and end timestamps of semantic units to improve the temporal coherence of semantic unit extraction;

[0435] Output: Semantic unit classification results, corresponding timestamp intervals, and semantic confidence scores.

[0436] Fourth layer: Teaching stage - semantic unit joint output layer:

[0437] Input: Semantic unit results from the third layer output, temporal attention weights, and stage attributes of the instructional knowledge graph;

[0438] Function: Based on the temporal distribution of semantic units and the stage attributes of the knowledge graph, jointly predict the teaching stage (new lesson introduction, knowledge instruction, classroom exercises, summary and review), bind semantic units to teaching stages one by one, and generate standardized teaching content semantic features;

[0439] Output: A set of semantic features of teaching content, including timestamp, semantic unit type, teaching content text, teaching stage, and confidence level (fully matching the output format of the original technical solution).

[0440] In this embodiment, the knowledge graph is a pre-existing model, specifically obtained through the following method:

[0441] Using the effective teaching text content (courseware text, lesson plan text) and teaching syllabus (objectives) from step 19, preprocessing is completed using basic NLP tools:

[0442] Text cleaning: Remove formatting symbols and meaningless redundant text (such as headers and footers, duplicate annotations), and retain core content such as knowledge point definitions, formula explanations, example descriptions, teaching objectives, and teaching process descriptions;

[0443] Word segmentation and part-of-speech tagging: A word segmentation dictionary specific to the teaching field (built based on publicly available educational corpora) is used to segment the preprocessed text and tag the parts of speech such as nouns (knowledge point names), verbs (teaching actions), and adjectives (attribute descriptions);

[0444] Core entity extraction: Three types of core entities are extracted using a rule-based, lightweight NER model (BiLSTM+CRF):

[0445] Knowledge point entities (such as "linear function" and "Newton's second law");

[0446] Entities of teaching activities (such as "definition explanation", "example explanation", "classroom exercises");

[0447] Related entities (such as "derive", "application", "premise", "extension").

[0448] 2. Structured Modeling: Constructing a teaching-specific graph of relationships (lightweight core)

[0449] Only a minimal triplet structure of "entity-relationship-attribute" is constructed, without redundant storage of irrelevant information, ensuring that the graph size is controllable:

[0450] Hierarchical relationship construction: Based on the knowledge system of the teaching syllabus and the logical order of the courseware text, establish the hierarchical relationship of knowledge points (such as "Mathematics → Function → Linear Function → Definition / Example" and "Physics → Mechanics → Newton's Laws → Newton's Second Law").

[0451] Relationship building: Based on textual semantic logic, we uncover three core types of relationships between knowledge points (avoiding complex relationships):

[0452] The order of presentation (e.g., "Definition Explanation" → "Example Application" and "Formula Derivation" → "Exercises and Exercises").

[0453] Dependencies (e.g., the solution to a quadratic equation depends on the perfect square formula);

[0454] Parallel relationships (such as "direct proportional function" and "inverse proportional function");

[0455] Attribute annotation: Two types of key attributes are annotated for each knowledge point entity (to adapt to model requirements):

[0456] Teaching stage attributes (corresponding to "New Lesson Introduction", "Knowledge Instruction", "Classroom Exercises", and "Summary and Review");

[0457] Difficulty level (based on the syllabus markings "Basic", "Intermediate", and "Comprehensive").

[0458] 3. Graph simplification and optimization: Adapting to lightweight inference requirements

[0459] Through two streamlined steps, we ensure that the map size is small and the query speed is fast, meeting the inference requirement of ≤30ms per cycle:

[0460] Redundant relationship removal: Only hierarchical relationships and core associations directly related to semantic parsing are retained, while indirect associations are removed (e.g., in the case of "knowledge point A → knowledge point B → knowledge point C", only AB and BC are retained, and the indirect association between AC is removed).

[0461] Entity merging and deduplication: Merge synonymous knowledge point entities (e.g., merge "linear function graph" and "drawing linear function graph" into "linear function graph"), and remove duplicate triples. The final number of triples in the graph should be controlled to ≤500 per course.

[0462] 4. Graph Embedding: Generates feature vectors that can be directly fused (adapting to the second layer of the model)

[0463] The simplified structured graph is transformed into a low-dimensional vector, which facilitates rapid fusion with text features:

[0464] A lightweight embedding algorithm (TransE) is used to map knowledge point entities and relationships into 256-dimensional vectors (consistent with the output feature dimension of the second layer of the model), ensuring that no dimension transformation is required during fusion.

[0465] During the embedding process, the weights of teaching stage attributes are introduced (such as increasing the weight of knowledge point vectors in the "knowledge instruction" stage by 20%) to enhance the adaptability of the graph to the teaching scenario.

[0466] Generate a mapping table of "knowledge point entity-vector-attribute" and store it as a lightweight file (JSON format). The model can read it directly when it is loaded, without the need for real-time construction.

[0467] This application has the following advantages:

[0468] By employing a progressive technical design that combines targeted preprocessing, scenario-based feature extraction, and relational structured analysis, differentiated processing strategies (such as image enhancement, audio framing, and text semantic encoding) are used for video, audio, and text data. At the same time, a cross-modal data association mechanism based on timestamps and unique identifiers is established, breaking through the technical limitations of single-dimensional data analysis and enabling in-depth interpretation and accurate association of unstructured data.

[0469] The front-end steps employ lightweight processing algorithms (such as lightweight CNN and KCF tracking) and precise sampling strategies (100ms frame sampling) to ensure millisecond-level data processing and result output, meeting the technical requirements for real-time classroom feedback. The back-end steps are designed with a periodic threshold triggering mechanism and directional parameter update technology to achieve incremental model optimization based on historical data, balancing real-time performance and long-term evolution capabilities, avoiding the technical imbalance of traditional solutions that prioritize real-time performance over evolution or vice versa.

[0470] A combined preprocessing scheme of ESRGAN super-resolution algorithm and circular buffer is adopted to perform real-time sharpening of blurry student face / hand images. At the same time, attention-related data is cached for 5 seconds through the circular buffer. This not only solves the technical problem of low-quality images affecting recognition accuracy, but also realizes the function of backtracking distracted behavior, breaking through the technical limitation of traditional preprocessing that "only clears but does not store".

[0471] The design incorporates a target-oriented screening and redundant data removal technology to optimize data collection. By using the YOLOv8 target detection algorithm, core targets such as students' bodies, faces, and hands are accurately screened, while redundant audio and text information without pedagogical meaning is simultaneously removed. This ensures accurate and efficient data collection, providing high-quality data support for subsequent feature extraction and avoiding the problem of low processing efficiency caused by data redundancy.

[0472] We designed a behavior-emotion-voice-specific feature dimension system and optimized feature extraction algorithm parameters (such as feature weights for emotion recognition and time windows for speech framing) for classroom scenarios. This makes the extracted feature dimensions highly compatible with the behavior, emotion, and speech types in teaching scenarios, overcoming the technical shortcomings of poor adaptability of traditional general features to teaching scenarios.

[0473] By employing a feature integration technology that combines temporal synchronization and source binding, behavioral, emotional, and vocal features are precisely aligned according to timestamps. Cross-feature dimension associations are achieved through a unified data format, providing a "same source and same order" technical foundation for subsequent multimodal fusion analysis and avoiding analysis errors caused by feature misalignment.

[0474] We construct a two-dimensional association technology that combines semantic encoding of teaching content with mapping of student features. Through semantic parsing algorithms, we extract core teaching units (knowledge points, examples, questions) and bind them to student behavior and emotional features at the time stamp level, breaking down the technical barrier of separating content analysis from student status in traditional solutions.

[0475] We designed a structured output technology that combines group statistics with individual details. This technology generates statistical results on the distribution of student status through a multi-dimensional data aggregation algorithm, while retaining the characteristic details and time series data of individual students. This achieves dual output of macro statistics and micro details, meeting the technical needs of different subsequent analysis scenarios.

[0476] By employing multi-level association mining technology, a technical association link is constructed between teaching content, behavior and emotion, and learning status. Through feature matching and logical reasoning algorithms, the inherent relationship between the three is mined, breaking through the technical limitations of traditional solutions that only focus on surface feature analysis.

[0477] By introducing multi-dimensional confidence verification technology, a verification model is built based on feature matching degree, temporal consistency, and data validity. Invalid data (such as misjudged emotions and ambiguous behaviors) is automatically eliminated, improving the reliability of analysis results and solving the technical problem of "lack of data validity verification" in traditional analysis.

[0478] The design incorporates a dual triggering technique that combines a periodic threshold and a data volume threshold. By presetting reasonable periodic intervals and data accumulation thresholds, it accurately controls the timing of model optimization. This avoids both the waste of system resources caused by excessively frequent optimizations and the lag in model performance caused by excessively long optimization intervals, thus overcoming the technical shortcomings of blindly triggering optimization timing in traditional methods.

[0479] By employing historical data correlation verification technology, the system automatically verifies the completeness and correlation of historical high-quality classroom data and full-cycle behavioral and emotional analysis information before optimization, ensuring that optimization has sufficient and effective data support and improving the accuracy of subsequent parameter generation.

[0480] We construct a two-dimensional parameter generation technology that combines positive feature extraction with negative feature correction. By statistically analyzing the efficient interactive features of high-quality classroom data, we extract positive optimization parameters. By combining misjudgment and missed detection data in behavioral emotion analysis, we correct the model parameters, achieve targeted parameter optimization, and break through the technical limitations of traditional single-dimensional parameter generation.

[0481] The design module features a dedicated parameter adaptation technology. Based on the core algorithm logic of the visual perception, semantic analysis, and thought scheduling modules (such as object detection, semantic encoding, and association analysis), specific optimization parameters (such as detection threshold, encoding weight, and association coefficient) are generated separately. This avoids the crude design of "adapting general parameters to all modules" and achieves targeted and precise optimization of the model.

[0482] The model update technique employs a closed-loop approach of incremental fine-tuning and performance verification. It fixes the backbone network parameters of the model and only updates the weights of core adaptation layers such as fully connected layers and rule-determination modules. This reduces update time, ensures real-time performance, and avoids catastrophic model obsolescence. At the same time, it sets quantitative verification indicators (such as behavior recognition accuracy ≥92% and semantic parsing accuracy ≥90%). If the indicators are not met, parameter backtracking and readjustment are triggered to ensure the stability of the updated model performance.

[0483] The design incorporates a parameter change log and evolution archive technical recording scheme, which automatically associates the optimization cycle, historical data, and verification results corresponding to parameter updates. This enables the model updates to be traceable and reproducible, providing a technical basis for subsequent system maintenance and performance iteration, and solving the technical problem of "no record and no backtracking" in traditional model updates.

[0484] We propose an iterative ROI region dynamic update technology. Based on the target detection results, we use IoU overlap determination, offset correction algorithm, local re-detection mechanism, and new target supplementation strategy to achieve iterative update of the ROI region list. This solves the technical problems of easy loss and difficulty in adapting to target movement after initial positioning in traditional tracking, and ensures the continuity and accuracy of the tracking trajectory.

[0485] Employing synchronous acquisition and precise association technology, the system uses a video frame sampler to synchronously capture facial regions and update motion trajectories at 100ms intervals. By leveraging a dual association mechanism of unique student identifiers and timestamps, it achieves precise binding between emotion tags and motion trajectories, avoiding data misalignment issues caused by traditional separate acquisition methods and providing a complete data link for subsequent association analysis.

[0486] This application also provides an AI teaching feedback and self-evolution system based on active perception and hybrid thinking mechanisms, wherein the AI ​​teaching feedback and self-evolution system based on active perception and hybrid thinking mechanisms includes:

[0487] A multimodal data acquisition module for identification, wherein the multimodal data acquisition module for identification is used to acquire multimodal data to be identified in the current period;

[0488] The feature generation module is used to generate student behavior features, emotion features, and teacher-student voice features based on the multimodal data to be identified in the current period through a visual perception model.

[0489] The student status-teaching content structured information generation module is used to generate student status-teaching content structured information based on student behavioral characteristics, emotional characteristics, and teacher and student voice characteristics through a semantic analysis model.

[0490] The behavioral emotion analysis information acquisition module is used to acquire behavioral emotion analysis information based on the structured information of student status and teaching content through the core model of the thinking scheduling module.

[0491] The judgment module is used to determine whether the difference in the number of cycles between the current cycle and the previous optimized cycle reaches a preset threshold. If so, it obtains all historical high-quality classroom video annotations of efficient interactive segments between the current cycle and the previous optimized cycle, as well as behavioral and emotional analysis information for each cycle.

[0492] The optimization parameter acquisition module is used to generate optimization parameters for the visual perception module, semantic analysis module, and thought scheduling module based on the highly interactive segments annotated from all historical high-quality classroom videos and the behavioral and emotional analysis information for each cycle.

[0493] The model update module is used to update the visual perception model based on the optimized parameters of the visual perception module, update the semantic analysis model based on the optimized parameters of the semantic analysis module, and update the core model of the thought scheduling module based on the optimized parameters of the thought scheduling module.

[0494] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. An AI-based teaching feedback and self-evolution method based on active perception and hybrid thinking mechanisms, characterized in that, The AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanisms includes: Step 1: Obtain the multimodal data to be identified for the current period; Step 2: Based on the multimodal data to be identified in the current period, generate student behavioral features, emotional features, and teacher-student voice features through a visual perception model; Step 3: Based on student behavioral characteristics, emotional characteristics, and teacher and student voice characteristics, generate structured information on student status and teaching content through a semantic analysis model; Step 4: Based on the structured information of student status and teaching content, obtain behavioral emotion analysis information through the core model of the thinking scheduling module; Step 5: Determine whether the difference in the number of cycles between the current cycle and the previous optimized cycle reaches the preset threshold. If so, obtain all historical high-quality classroom video annotations of efficient interactive segments between the current cycle and the previous optimized cycle, as well as behavioral and emotional analysis information for each cycle. Step 6: Generate optimization parameters for the visual perception module, semantic analysis module, and thought scheduling module based on the highly effective interactive segments marked in all historical high-quality classroom videos and the behavioral emotion analysis information for each cycle; Step 7: Update the visual perception model based on the optimized parameters of the visual perception module, update the semantic analysis model based on the optimized parameters of the semantic analysis module, and update the core model of the thought scheduling module based on the optimized parameters of the thought scheduling module.

2. The AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanism as described in claim 1, characterized in that, Step 1: Obtaining the multimodal data to be identified in the current period includes: Step 11: Obtain the raw video data for the current period; Step 12: Run the YOLOv8 object detection algorithm. Run the YOLOv8 object detection algorithm on each frame of the video to detect the student's body, face, and hands. Filter out targets with a confidence score ≥ 0.8, record the target bounding box coordinates and corresponding confidence scores, and thus obtain the set of student target detection boxes. Step 13: Based on the list of ROI regions from the iterative input, call the KCF correlation filter tracking algorithm to track the student targets in the student target detection box set in real time and generate the motion trajectory of each student; use the video frame sampler to extract the student's facial region, input the facial image into the pre-trained CNN model, identify the emotion information, and output the emotion label and confidence score corresponding to each frame, thereby obtaining the student target tracking trajectory and the real-time emotion label dataset. Step 14: Calculate the Laplacian variance of the student's facial image, and combine it with the confidence of the target detection box to select images with a confidence of <0.9 or a Laplacian variance of <100 as blurred images; run the ESRGAN super-resolution algorithm on the blurred images to enlarge them by 2-4 times, and simultaneously perform CLAHE histogram equalization to enhance image details, thereby obtaining clear images of the student's face and hands. Step 15: Based on the student target detection box set and student target tracking trajectory, identify and label abnormal behaviors; integrate the emotion tags and abnormal behavior markers in the real-time emotion tag set, associate them with the corresponding student's clear image, and generate structured original behavioral and emotional features; Step 16: Acquire the audio stream using an audio frame capture device, and divide the audio stream into frames with a frame length of 20ms and a step length of 10ms; run the MFCC algorithm on each frame of audio to extract the 13-dimensional Mel frequency cepstral coefficients, and construct an audio feature vector by combining the frame energy and zero-crossing rate, thereby obtaining the teacher and student speech feature vector sequence and the audio and video timestamp mapping table. Step 17: Based on the audio and video timestamp mapping table, align the teacher's voice segments with the student behavior data; run the dynamic time warping algorithm to calculate the synchronization difference between the teacher's voice and the student's behavior. If the synchronization difference exceeds 1.5 seconds, it is determined to be a comprehension gap. Record the gap time period and the corresponding student group to obtain the comprehension gap determination result. Step 18: Sample students' facial posture and eye opening / closing at 100ms intervals using a video frame sampler; use a circular buffer data structure to cache the sampling data of the most recent 5 seconds; calculate the attention continuity value through a 1-second sliding window, and determine that the attention continuity value is ≥0.6 as focused and <0.6 as distracted, generate attention state labels, and thus obtain a 5-second attention cache dataset; Step 19: Normalize the original behavioral and emotional features and map them to the [0,1] interval; standardize the teacher and student speech feature vector sequence with Z-score; encode the courseware text with UTF-8 through a text parser to retain the effective teaching text content, thereby obtaining the multimodal data to be identified in the current period.

3. The AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanism as described in claim 2, characterized in that, Step 2: Based on the multimodal data to be identified in the current period, student behavioral features, emotional features, and teacher-student voice features are generated through a visual perception model, including: Step 21: Input the set of student target detection boxes, student target tracking trajectory, and cleared images of student face and hand from the multimodal data to be identified in the current period into the trained visual behavior perception sub-model, extract the basic features of student behavior, and thus obtain the basic features of student behavior indexed by the student's unique identifier. The basic features of student behavior include the type of behavior action, the time node of the action, the duration of the action, and the location area of ​​the action. Step 22: Input the sharpened student facial image, real-time emotion label dataset, and 5-second attention cache dataset from the multimodal data to be identified in the current period into the trained visual emotion perception sub-model to generate a student emotion basic feature set indexed by the student's unique identifier; Step 23: Input the standardized teacher and student speech feature vector sequence, audio and video timestamp mapping table, and understanding tomographic judgment result from the multimodal data to be identified in the current period into the trained speech perception sub-model to generate the basic feature set of teacher and student speech; Step 24: The student behavior feature set generated in Step 21 is structurally integrated. Using the student's unique identifier as the first index and the timestamp as the second index, the behavior type, duration, behavior switching interval, and abnormal attribute markers are arranged in chronological order to form structured student behavior features. The student emotion feature set generated in Step 22 is structurally integrated. Using the student's unique identifier as the first index and the timestamp as the second index, the emotion type, frequency of occurrence, emotion duration, attention continuity value, and attention state markers are arranged in chronological order to form structured student emotion features. The teacher-student speech feature set generated in Step 23 is structurally integrated. Using the audio and video timestamps as the index, the speech source, speech period, acoustic feature vector, and understanding discontinuity association markers are arranged in chronological order to form structured teacher-student speech features. Step 25: Using the audio and video timestamp mapping table as a unified benchmark, accurately align the time dimensions of structured student behavior features, structured student emotion features, and structured teacher and student voice features. Associate and match student behavior and emotion features with corresponding teacher and student voice features under the same timestamp to generate student behavior features, emotion features, and teacher and student voice features.

4. The AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanism as described in claim 3, characterized in that, Step 3: Based on student behavioral characteristics, emotional characteristics, and teacher-student voice characteristics, structured information on student status and teaching content is generated through a semantic analysis model, including: Step 31: Input the structured student behavior features, structured student emotion features, structured teacher-student speech features, and the effective teaching text content encoded in UTF-8 in Step 19 into the trained semantic analysis model to obtain a multimodal semantic input dataset. The multimodal semantic input dataset includes a unique student identifier, timestamp, student behavior features, student emotion features, teacher-student speech features, and teaching text fragments. The teaching text fragments are the teaching content paragraphs in the effective teaching text content that correspond to the current timestamp. Step 32: Input the teacher and student speech features and teaching text fragments from the multimodal semantic input dataset into the trained teaching content semantic parsing sub-model to obtain the teaching content semantic feature set; Step 33: Input the student behavior features and student emotion features from the multimodal semantic input dataset into the trained student state semantic mapping sub-model to obtain the student state semantic feature set; Step 34: Input the semantic feature set of teaching content generated in Step 32 and the semantic feature set of student status generated in Step 33 into the trained state-content association integration sub-model to generate the state-content association feature set; Step 35: Perform structured regularization on the state-content association feature set to generate standardized student state-teaching content structured information.

5. The AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanism as described in claim 4, characterized in that, Step 4: Based on the structured information of student status and teaching content, obtain behavioral emotion analysis information through the core model of the thinking scheduling module, including: Step 41: Input the structured information of student status and teaching content generated in Step 3 into the core model of the trained thinking scheduling module to generate a hierarchical analysis input dataset; Step 42: Input the individual student learning status and detailed student status list from the hierarchical analysis input dataset into the trained state depth analysis sub-model to generate an individual state depth analysis feature set; Step 43: Input the hierarchical analysis input dataset and the individual state deep parsing feature set generated in Step 42 into the trained behavior-emotion association analysis sub-model to generate the behavior-emotion association analysis feature set; Step 44: Input the hierarchical analysis input dataset and the behavior-emotion association analysis feature set into the trained behavior-emotion association analysis sub-model to generate the fault-behavior-emotion association feature set; Step 45: Input the behavioral-emotion association analysis feature set generated in Step 43 and the fault-behavioral-emotion association feature set generated in Step 44 into the trained structured sub-model of the analysis results to obtain the integrated analysis results; Step 46: Standardize and structure the integrated analysis results to generate structured behavioral sentiment analysis information.

6. The AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanism as described in claim 5, characterized in that, Step 6: Based on the highly interactive segments annotated in all historical high-quality classroom videos and the behavioral and emotional analysis information for each cycle, the following optimization parameters are generated for the visual perception module, semantic analysis module, and thought scheduling module: Step 61: Based on the high-efficiency interactive segments and behavioral sentiment analysis information of each period of all historical high-quality classroom videos obtained in Step 5, generate an optimized benchmark dataset containing positive features of high-efficiency interaction, negative features of problem behavior and sentiment, and features related to teaching content. At the same time, assign optimization priority to each feature, with positive features of high-efficiency interaction as high priority and negative features of problem behavior and sentiment as secondary priority. Step 62: Generate optimized parameters for the visual perception module. The visual perception module includes a trained visual behavior perception sub-model, a visual emotion perception sub-model, and a speech perception sub-model. Based on the optimized benchmark dataset, perform the following operations: For the visual behavior perception sub-model, statistically determine the optimal range of target detection confidence, the trajectory matching threshold of the KCF tracking algorithm, and the threshold for abnormal behavior judgment in the positive features of efficient interaction. Combine this with false positive and false negative target detection data in the negative features of problem behavior emotion, adjust the confidence threshold of the YOLOv8 target detection algorithm, the filtering parameters of the KCF correlation filter tracking algorithm, and the abnormal behavior labeling rules to generate optimized parameters for target detection, tracking algorithm, and behavior recognition. For the visual emotion perception sub-model, statistically determine the optimal value of emotion recognition confidence and the parameter range for facial image enhancement in the positive features of efficient interaction. Combine this with false positive emotion judgment data in the negative features of problem behavior emotion, adjust the classification threshold of the emotion recognition model, the magnification factor of the ESRGAN super-resolution algorithm, and the enhancement coefficient of CLAHE histogram equalization to generate optimized parameters for emotion recognition and image enhancement. For the speech perception sub-model, statistically determine the MFCC of speech feature extraction in the positive features of efficient interaction. By adjusting the extraction dimensions, frame segmentation parameters, and speech emotion recognition confidence threshold, and combining the speech misclassification data in the negative features of problem behavior emotion, the extraction dimensions, audio frame segmentation parameters, and classification threshold of the MFCC algorithm are adjusted to generate speech feature extraction optimization parameters and speech emotion recognition optimization parameters. The above sub-model optimization parameters are integrated to form a visual perception module optimization parameter set. The parameter set includes the thresholds, coefficients, and rule parameters of each sub-model. Each parameter is labeled with the corresponding optimization basis, namely the statistical results of positive features of efficient interaction or the correction results of negative features of problem behavior emotion. Step 63: Generate optimization parameters for the semantic analysis module. The semantic analysis module includes a trained sub-model for semantic parsing of teaching content, a sub-model for semantic mapping of student state, and a sub-model for integrating state-content association. Based on the optimized benchmark dataset, perform the following operations: For the semantic parsing sub-model of teaching content, calculate the accuracy of extracting core semantic units of teaching in the positive features of efficient interaction and the matching degree of teaching stage division. Combine this with the semantic mis-parsing data in the negative features of problem behavior and emotion, adjust the semantic word segmentation weights, knowledge point matching thresholds, and teaching stage judgment rules to generate semantic parsing weight parameters, knowledge point matching threshold parameters, and teaching stage division parameters. For the semantic mapping sub-model of student state, calculate the accuracy of the behavior-emotion-learning state mapping and the optimal interval of state confidence in the positive features of efficient interaction, and combine this with... For the state mismapping data in the negative features of problematic behavior and emotion, the weights of the state mapping rules and the coefficients for calculating state confidence are adjusted to generate state mapping weight parameters and confidence calculation optimization parameters. For the state-content association integration sub-model, the optimal value of the association strength between teaching content and student state in the positive features of efficient interaction and the threshold for judging the association fault are statistically analyzed. Combined with the association mismatch data in the negative features of problematic behavior and emotion, the coefficients for calculating association strength and the rules for marking the association fault are adjusted to generate association strength optimization parameters and association fault judgment parameters. The optimization parameters of the above sub-models are integrated to form the optimization parameter set of the semantic analysis module. The parameter set contains the weights, thresholds, and calculation coefficients of each sub-model, and each parameter is labeled with the corresponding optimization basis, that is, the comparison results between the positive features of efficient interaction and the negative features of problematic behavior and emotion. Step 64: Generate optimization parameters for the thinking scheduling module. The core model of the thinking scheduling module includes a trained state deep analysis sub-model, a behavior-emotion association analysis sub-model, and an analysis result structured sub-model. Based on the optimization benchmark dataset, perform the following operations: For the state deep analysis sub-model, statistically analyze the state decomposition accuracy and the optimal value of the invalid state removal threshold in the positive features of efficient interaction. Combined with the state decomposition error data in the negative features of question behavior and emotion, adjust the state decomposition rules and the invalid state confidence removal threshold to generate optimized state decomposition parameters and invalid state removal parameters; For the behavior-emotion association analysis sub-model, statistically analyze the optimal value of the behavior-emotion combination association strength and the high-frequency combination matching rules in the positive features of efficient interaction. Combined with the question... For the negative features of question behavior and emotion, erroneous data in association analysis are analyzed, and the weights for calculating association strength and the thresholds for judging high-frequency combinations are adjusted to generate association analysis weight parameters and combination judgment optimization parameters. For the structured sub-model of the analysis results, the optimal interval of the confidence of the analysis conclusion and the matching degree of the structured fields in the positive features of efficient interaction are statistically analyzed. Combined with the erroneous data of the conclusions in the negative features of question behavior and emotion, the calculation coefficients of the conclusion confidence and the weights of the structured fields are adjusted to generate conclusion confidence optimization parameters and structured field adjustment parameters. The above sub-model optimization parameters are integrated to form the optimization parameter set of the thinking scheduling module. The parameter set contains the weights, thresholds, and calculation coefficients of each sub-model, and each parameter is labeled with the corresponding optimization basis, i.e., the feedback result of the positive features of efficient interaction. Step 65: Optimize parameter verification and standardization integration. Perform consistency verification on the optimized parameters of the visual perception module generated in Step 62, the optimized parameters of the semantic analysis module generated in Step 63, and the optimized parameters of the thought scheduling module generated in Step 64, and generate standardized optimized parameters of the visual perception module, the optimized parameters of the semantic analysis module, and the optimized parameters of the thought scheduling module.

7. The AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanism as described in claim 6, characterized in that, The visual emotion perception sub-model includes: Input layer: The input layer receives the sharpened student facial image, real-time emotion label dataset, 5-second attention cache dataset, image quality parameter set, and student unique identifier + timestamp from the multimodal data to be identified in step 1, wherein: Clarify student facial images: standardize the size to 224×224×3, RGB format; Real-time sentiment label dataset: The raw confidence scores of sentiment information output by the pre-trained CNN model in step 1 are used as weak supervision signals for the model; 5-second attention buffer dataset: continuous attention values, eye opening and closing, and facial pose sampling data in a circular buffer; Image quality parameter set: Laplacian variance of facial images calculated in step 1, and YOLOv8 target detection confidence score; Student unique identifier + timestamp: serves as a data index to ensure that output features are accurately associated with students and time, facilitating subsequent structured processing; An image quality adaptive preprocessing layer is used for image filtering and feature completion initialization to adapt to the image sharpening result of step 1. A dual threshold screening rule is set: images with a Laplacian variance ≥ 100 and a detection confidence ≥ 0.9 are considered high-quality images and directly enter the feature extraction layer; images with a Laplacian variance < 100 or a detection confidence < 0.9 are considered low-quality images and are marked with a quality weight coefficient. Pixel-level weighted initialization for low-quality images: The pixel mean of high-quality images is used as the pixel completion benchmark for low-quality images. Pixel-level correction is completed by combining the quality weight coefficient to avoid feature distortion caused by blurry images directly entering the feature layer. Output: Standardized facial image, quality weight coefficient, and association index of student unique identifier + timestamp. A multi-branch fine-grained visual feature extraction layer is used, which employs a three-branch parallel extraction + feature normalization. Based on MobileViT's lightweight convolutional blocks, it extracts the core visual features related to students' emotions in the classroom. The feature output dimension of all branches is unified to 256 dimensions to facilitate subsequent fusion. Facial micro-expression feature branch: It consists of 4 layers of depthwise separable convolution + 2 layers of MobileViT small-scale attention blocks, with strides of 1, 1, 2, 2 respectively. It extracts the texture and deformation features of key micro-expression areas such as brow bone, corner of mouth, and nasal wings, and outputs a 256-dimensional micro-expression feature vector. This branch is the core branch, and the initial weight is set to 0.

5. Eye attention feature branch: Based on facial key point detection to locate the eye region, the features of eye opening and closing, eye movement direction and blink frequency are extracted through 3-layer depthwise separable convolution. Combined with the eye opening and closing sampling values ​​in the 5-second attention cache dataset, a 256-dimensional eye attention feature vector is output, with the initial weight set to 0.

3. Head pose feature branch: By solving the three-dimensional pose matrix of facial key points, the head pitch angle, yaw angle and roll angle features are extracted. Combined with the head movement trend in the student target tracking trajectory in step 1, a 256-dimensional head pose feature vector is output, with the initial weight set to 0.

2. Feature normalization: L2 normalization is performed on the feature vectors of the three branches to eliminate the difference in dimensions, and the micro-expression feature vector, eye attention feature vector, head posture feature vector, and the initial weights of each branch are output. An attention-gated feature fusion layer is used to adaptively fuse the three-branch features. It introduces an attention gating mechanism to dynamically adjust the feature weights of the three branches based on the input image quality parameters and attention cache data, thus addressing the issue of varying feature importance under different image qualities and attention states. The average attention continuity value in the 5-second attention cache dataset is used as the gating trigger signal: if the attention continuity value is ≥0.6, the weight of the micro-expression feature branch is increased and the weight of the head pose feature branch is decreased; if the attention continuity value is <0.6, the weight of the head pose and eye attention feature branches is increased and the weight of the micro-expression feature branch is decreased. Using the quality weight coefficients in the image quality parameter set as feature weighting coefficients: multiply the feature vectors of low-quality images by the quality weight coefficients, and keep the weights of the feature vectors of high-quality images at 1 to complete feature robust completion; Feature fusion: The dynamically weighted three-branch feature vectors are concatenated along the channel dimension, and feature dimensionality reduction and fusion are performed through a 1×1 convolution layer to output a 512-dimensional multimodal visual fusion feature vector. This vector contains coupled features of the face, eyes, and head, providing a foundation for subsequent temporal modeling. A time-attention bidirectional GRU temporal modeling layer is used to capture the temporal continuity and gradual changes in students' emotions in the classroom. It is adapted to a 5-second attention buffer time window and is completely synchronized with the attention data collection rhythm in step 1. Using 100ms as the time step, the multimodal visual fusion feature vectors within a 5-second time window are arranged in chronological order to form a 50×512 temporal feature sequence. Bidirectional GRU layer: Two bidirectional GRU layers are set up with a hidden layer dimension of 256. The forward GRU captures the positive trend of emotion change, and the backward GRU captures the negative trend of emotion change, outputting a 50×512 temporal correlation feature sequence. Temporal attention layer: Introduces a temporal attention mechanism to weight the feature sequences of 50 time steps, highlighting the feature weights of key frames with emotional changes, suppressing redundant frame features without emotional changes, and outputting a 256-dimensional temporal fusion core emotion feature vector. The attention-emotion dual-label cross-validation layer is used for soft constraint verification. It uses the attention data from step 1 as a constraint for emotion recognition, eliminating contradictory results, improving feature effectiveness, and providing pre-filtering for the state mapping of the subsequent semantic analysis model. Temporal fusion of core emotion feature vectors, mean of continuous attention values ​​in a 5-second attention cache dataset, and attention state labels; Preliminary emotion classification: A 256-dimensional feature vector is mapped to confidence scores for 7 categories of classroom student emotions through a fully connected layer, generating preliminary emotion classification results; Mutual verification rule determination: Core verification rules are set; if violated, the result is marked as invalid emotional and corrected based on attention state. Rule 1: If the mean of continuous attention scores is ≥0.6, the confidence scores for boredom, fatigue, and confusion need to be reduced by 30%. Rule 2: If the mean of continuous attention scores is <0.6, the confidence scores for calm, happy, and focused should be reduced by 30%. Rule 3: If the head posture is tilted down at a angle greater than 45° and the eye opening angle is less than 0.2, it is directly judged as a distraction-related emotion, and only the two emotion results of annoyance and fatigue are retained; Output the adjusted confidence scores for 7 types of emotions, effective / ineffective emotion labels, and a correlation mapping table between the mean of continuous attention values ​​and emotion confidence. The teaching stage adaptive adjustment layer is used to reserve an interface for docking with the semantic parsing model of subsequent teaching content, realizing the pre-association of emotional features and teaching content. It only performs dynamic weight adjustment within the model. If teaching stage information is not yet available, it outputs according to the initial weights. Interface input: Teaching stage markers output by the semantic parsing sub-model for subsequent teaching content; Dynamic weight adjustment: Based on the characteristics of the teaching stage, five sets of feature weight adjustment coefficients are preset to adjust the revised emotion confidence score by weighting it. For example: During the classroom questioning phase: Increase the weight given to surprise and joy, and match the students' emotional reactions when they are asked questions; During the knowledge delivery phase: increase the weight given to feelings of confusion and calmness to match the core emotions students experience while listening to the lecture; During the classroom practice phase: increase the weight of fatigue and confusion to match the emotional characteristics of students when doing exercises; output the emotional confidence score after the teaching phase is adapted, the teaching phase mark, and the student's unique identifier + timestamp; The emotional feature structured output layer is used to complete the statistics and structuring of emotional features. The output feature set fully meets the structured emotional feature requirements of step 2 and can be directly used as input to the semantic analysis model without additional feature transformation. Emotional characteristics statistics: Based on 100ms time intervals, integrate the emotional confidence scores after the adaptation of teaching stages within a 5-second time window, and calculate the emotional intensity characteristics and emotional fluctuation characteristics. Attention-Emotion Association Labeling: Associate the mean of continuous attention values ​​and attention state labels with emotion features to label the attention attributes corresponding to each emotion type; Structured encapsulation: Using the student's unique identifier as the first index and the timestamp as the second index, the feature fields are encapsulated according to the structured requirements of step 2, and the final output is a structured student emotion feature set. The fields include: student unique identifier, timestamp, emotion type, mean emotion intensity, peak emotion intensity, variance of emotion fluctuation, mean of continuous attention value, attention state label, valid emotion result label, and teaching stage label.

8. The AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanism as described in claim 4, characterized in that, The student state semantic mapping sub-model includes: The input layer is used to obtain the structured student behavior features and structured student emotion features generated in step 2; the semantic feature set of teaching content generated in step 3; and the comprehension fault determination results from step 17. A multimodal feature preprocessing and normalization layer is used to standardize the input features and eliminate dimensional differences. Behavioral feature normalization: converting behavior type into one-hot encoding, mapping behavior duration and switching interval to the [0,1] interval, and converting anomaly markers into 0 / 1 binary features; Emotional feature normalization: Emotional types (happiness, calmness, confusion, etc.) are converted into one-hot codes, and the mean and variance of emotional intensity are mapped to the [0,1] interval, while the continuous values ​​of attention are directly retained as the original [0,1] values; Teaching content feature encoding: The teaching stage and knowledge point type are converted into one-hot encoding, and the teaching content text is converted into a 512-dimensional semantic embedding vector through pre-trained word vectors; Finally, standardized behavioral feature vectors, emotion feature vectors, teaching content semantic vectors, attention feature vectors, as well as student unique identifiers and timestamp indexes are generated. A multimodal feature coupling and fusion layer, wherein the multimodal feature coupling and fusion layer is used for feature fusion: Feature interaction branches: Construct behavior-emotion interaction branches, behavior-attention interaction branches, and emotion-attention interaction branches respectively. Extract the interaction features of pairwise features through element-wise multiplication and fully connected layers. Each interaction branch outputs a 64-dimensional interaction feature vector. Teaching content perception branch: The semantic vector of teaching content is concatenated with the interaction feature vector. Through a 1×1 convolution layer and Leaky ReLU activation, the association features between teaching content and student features are extracted, and a 256-dimensional teaching content-student feature association vector is output. Feature fusion gating: A gating mechanism is introduced to dynamically adjust the feature weights of each branch based on the teaching stage. All branch features are fused by gating weights to output a 512-dimensional multimodal coupled feature vector. This vector contains information on the deep correlation between student behavior, emotions, attention and teaching content. The temporal dynamic modeling layer is used to capture the temporal changes in the learning state and is fully synchronized with the 5-second attention buffer in step 1 and the temporal features in step 2. Construction of temporal feature sequence: Using 100ms as the time step, the multimodal coupled feature vectors within the 5-second time window are arranged in chronological order to form a 50×512 temporal coupled feature sequence; Temporal memory gate module: Set up 2 layers of gated loop units, introduce temporal memory gates, retain the feature memory of previous time steps, capture the gradual trend of learning state, and output a 50×256 temporal dynamic feature sequence; Temporal attention weighting: A temporal attention mechanism is introduced to weight the feature sequences of 50 time steps, highlighting the feature weights of key time steps with state changes, suppressing redundant time step features without state changes, and outputting a 256-dimensional temporal dynamic core feature vector, which integrates static coupling features and temporal dynamic change information; An adaptive adjustment layer for teaching stages is used to dynamically adapt the mapping between teaching stages and states. Teaching Stage Weighting Library: Five sets of exclusive weighting coefficients for each teaching stage are preset, including weighting coefficients for new lesson introduction, knowledge delivery, classroom questioning, classroom practice, and summary review. Each set of weights includes weights for behavioral characteristics, emotional characteristics, attention characteristics, and temporal characteristics. Dynamic weight matching: Based on the input teaching stage marker, the corresponding weight coefficients are retrieved from the weight library to adjust the time-series dynamic core feature vector, and the time-series dynamic feature vector adapted to the teaching stage, the teaching stage marker, and the corresponding teaching content text are output. A dynamic confidence calibration layer is used to learn the dynamic calculation of state confidence and correct contradictory results. Preliminary state mapping: Through one fully connected layer and Softmax activation, the temporal dynamic feature vector adapted to the teaching stage is mapped to the preliminary confidence scores of five learning states; Confidence score calculation across multiple dimensions: The final confidence score is dynamically calculated based on three dimensions, and the weight of each dimension can be optimized through self-evolution. Feature matching degree: The degree of matching between behavioral, emotional, and attentional features and learning state; Temporal consistency: The current state is consistent with the state 4 seconds prior; Teaching content suitability: The degree of suitability between the current state and the teaching content and teaching stage; Conflicting Result Correction: Three core correction rules are set. If the initial state result violates the rules, the confidence level of the corresponding state is lowered, and states with higher matching degree are selected first. Rule 1: For attention span scores ≥ 0.8 and no abnormal behavior, the confidence level for distraction, inattentiveness, and passive resistance is reduced by 50%. Rule 2: For positive behaviors such as raising hands and answering, and negative behaviors such as resistance and distraction, the confidence level is reduced by 40%. Rule 3: During periods of comprehension gap, reduce the confidence level of focused comprehension by 60%, prioritizing retention of confusion, hesitation, and distraction; output the final confidence scores for the five learning states, valid / invalid state labels, and the basis for confidence calculation. The learning state classification and structured output layer is used to complete state classification and structured encapsulation, and outputs a set of student state semantic features. State classification determination: Select the learning state with the highest final confidence score as the student's current learning state. If the highest confidence score is less than the preset threshold, it is marked as an unknown state. Status characteristic statistics: In order of timestamp, the duration of each learning status, the number of status transitions, and the mean status confidence of each student in the current period are statistically analyzed. Association annotation: Associate the learning status with the teaching content text, teaching stage, and comprehension gap markers, and annotate the teaching scenario attributes corresponding to the status; Structured encapsulation: Using the student's unique identifier as the first index and the timestamp as the second index, the feature fields are encapsulated according to the requirements of the original technical solution. The final output is a semantic feature set of student status, with fields including: student unique identifier, timestamp, learning status, status confidence, status duration, teaching content text, teaching stage, understanding discontinuity association marker, and status validity marker.

9. The AI ​​teaching feedback and self-evolution method based on active perception and hybrid thinking mechanism as described in claim 6, characterized in that, The semantic parsing sub-model for the teaching content includes: Multi-source text temporal alignment layer: The multi-source text temporal alignment layer is used to obtain the teacher and student voice text sequence, effective teaching text content, and audio and video timestamp mapping table input in step 3. Based on the audio and video timestamps, the voice text and courseware text (knowledge points, examples) are temporally sliced ​​and aligned, redundant text without teaching semantics is removed, and a 512-dimensional temporal aligned text feature sequence is generated. The knowledge graph-enhanced semantic encoding layer is used to align text feature sequences and lightweight teaching knowledge graphs according to time sequence. It uses a lightweight Transformer encoder to extract text context semantics and fuses the knowledge point embedding vectors of the knowledge graph with text features to strengthen the semantic association between knowledge points, definitions, and examples, generating a 256-dimensional knowledge-enhanced semantic feature vector. The temporal-aware semantic unit extraction layer is used to enhance semantic feature vectors based on knowledge, align text sequences temporally, introduce a temporal attention mechanism, accurately identify 6 types of core semantic units through conditional random fields, and simultaneously label the start and end timestamps of semantic units to generate semantic unit classification results, corresponding timestamp intervals, and semantic confidence scores. The teaching stage-semantic unit joint output layer is used to generate a semantic feature set of teaching content based on the semantic unit results and temporal attention weights.

10. An AI-based teaching feedback and self-evolution system based on active perception and hybrid thinking mechanisms, characterized in that: The AI ​​teaching feedback and self-evolution system based on active perception and hybrid thinking mechanisms includes: A multimodal data acquisition module for identification, wherein the multimodal data acquisition module for identification is used to acquire multimodal data to be identified in the current period; The feature generation module is used to generate student behavior features, emotion features, and teacher-student voice features based on the multimodal data to be identified in the current period through a visual perception model. The student status-teaching content structured information generation module is used to generate student status-teaching content structured information based on student behavioral characteristics, emotional characteristics, and teacher and student voice characteristics through a semantic analysis model. The behavioral emotion analysis information acquisition module is used to acquire behavioral emotion analysis information based on the structured information of student status and teaching content through the core model of the thinking scheduling module. The judgment module is used to determine whether the difference in the number of cycles between the current cycle and the previous optimized cycle reaches a preset threshold. If so, it obtains all historical high-quality classroom video annotations of efficient interactive segments between the current cycle and the previous optimized cycle, as well as behavioral and emotional analysis information for each cycle. The optimization parameter acquisition module is used to generate optimization parameters for the visual perception module, semantic analysis module, and thought scheduling module based on the highly interactive segments annotated from all historical high-quality classroom videos and the behavioral and emotional analysis information for each cycle. The model update module is used to update the visual perception model based on the optimized parameters of the visual perception module, update the semantic analysis model based on the optimized parameters of the semantic analysis module, and update the core model of the thought scheduling module based on the optimized parameters of the thought scheduling module.

Images