Micro-expression recognition method and system based on balanced adaptive grouped sampling

Abstract

The invention provides a micro-expression recognition method and system based on balanced adaptive grouping sampling, and relates to the technical field of micro-expression recognition, and the method comprises the steps: obtaining a data set; the similarity among a plurality of samples of the same student is calculated based on multi-scale structural similarity measurement, so that the identification difficulty of the samples is judged. According to the method, the structural similarity between the samples is calculated through multi-scale structural similarity measurement, so that the recognition difficulty of the samples is judged, grouping sampling is performed according to the recognition difficulty, and a micro-expression recognition model is trained based on a data set obtained through sampling. And meanwhile, the identification difficulty is adjusted according to a model output result, and sampling is performed again. Thus, re-sampling and model training are periodically performed, and the sampling weight of the sample is continuously updated according to the information fed back during model training, so that continuous reinforcement and feature optimization of the specific sample can be realized during training of the model, and the stability and generalization performance of the model are improved.

Description

Technical Field

[0001] This invention relates to the field of micro-expression recognition technology, and more specifically, to a micro-expression recognition method and system based on balanced adaptive group sampling. Background Technology

[0002] Intelligent recognition and feedback of students' emotions and attention states are typically achieved through micro-expression recognition technology and visual perception systems. These technologies help teachers understand students' level of focus and emotional changes in the classroom in real time, allowing them to adjust their teaching strategies accordingly.

[0003] In recent years, the rapid development of deep learning technology has significantly improved the performance of micro-expression recognition. Deep learning methods, through an end-to-end training framework, adaptively learn multi-scale, multi-channel spatial-temporal features, reducing reliance on manual feature engineering, optimizing task objectives, and enhancing the model's generalization ability in complex scenarios. Compared to traditional methods, these techniques exhibit stronger representational capabilities and can further improve performance through continuous training on large-scale data. Although deep models have significant advantages in representation learning, improper selection of training samples can easily lead to biased learning, overfitting to easy samples, or neglecting boundary samples, thus affecting recognition accuracy and generalization ability. In this context, sampling strategies become one of the key means to improve the performance of micro-expression recognition.

[0004] Existing micro-expression recognition methods generally ignore the imbalance in sample distribution and differences in learning difficulty during training. This causes models to focus more on easily identifiable samples while neglecting difficult samples with high recognition value. The main reason for this is that existing training sampling strategies mostly use random or fixed distribution methods, lacking the ability to dynamically adapt to the complexity of sample structure and the model's learning state. They cannot achieve continuous reinforcement and feature optimization of difficult samples during training, thus limiting the stability and generalization performance of micro-expression recognition models in real classroom environments. Summary of the Invention

[0005] The purpose of this invention is to provide a micro-expression recognition method and system based on balanced adaptive group sampling, so as to improve the above-mentioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows: Firstly, this application provides a micro-expression recognition method based on balanced adaptive group sampling, including: Obtain a training dataset, wherein a sample in the training dataset includes a student's micro-expression image and classroom status labels; The similarity between the first sample and multiple second samples is calculated based on the multi-scale structural similarity metric, and the recognition difficulty of the first sample is evaluated based on the multiple similarities corresponding to the first sample to obtain the recognition difficulty index of the first sample. The first sample and the second sample are different samples corresponding to the same student. Obtain the recognition difficulty index for each sample, and sample the training dataset based on multiple recognition difficulty indices to obtain a first training dataset; A preset micro-expression recognition model is trained based on the first training dataset to obtain first information and the trained micro-expression recognition model. The micro-expression recognition model is used to predict students' classroom status based on the micro-expression images. The first information includes the output information of the micro-expression recognition model during the training process. The recognition difficulty index of the sample is updated based on the first information, and the training dataset is sampled based on the updated recognition difficulty index to obtain a second training dataset. The micro-expression recognition model is then trained again based on the second training dataset.

[0006] Furthermore, obtaining the training dataset includes: Acquire the original image sequence, which includes an infrared image sequence and a visible light image sequence; Based on a preset face detection algorithm, the face regions of multiple students in each frame of the original image sequence are located, and multiple first images containing only the face regions are cropped from the images. Affine transformation alignment, scaling and rotation angle unification, pixel normalization and illumination compensation are sequentially performed on multiple first images to obtain multiple micro-expression images; The micro-expression images are labeled based on preset classroom status categories to obtain the classroom status labels corresponding to the micro-expression images; The first micro-expression image sequence and the second micro-expression image sequence are aligned at the frame level to obtain a micro-expression image frame sequence. The first micro-expression image sequence and the second micro-expression image sequence are image sequences composed of multiple frames of micro-expression images of the same student in the same time period, generated based on the infrared image sequence and the visible light image sequence, respectively. Multiple micro-expression image frame sequences and classroom status labels corresponding to each micro-expression image in the micro-expression image frame sequence are obtained to construct the training dataset, where one micro-expression image frame sequence corresponds to one student and one time period.

[0007] Furthermore, the step of assessing the recognition difficulty of the first sample based on multiple similarities corresponding to the first sample includes: Calculate the average similarity of the multiple similarities corresponding to the first sample to obtain the average similarity of the first sample; Calculate the average similarity for each sample to obtain multiple average similarity values, and normalize the multiple average similarity values ​​to obtain multiple recognition difficulty indicators, wherein the recognition difficulty indicators are the normalized average similarity values.

[0008] Furthermore, the sampling of the training dataset based on multiple recognition difficulty indicators includes: The multiple recognition difficulty indicators are sorted, and the samples are divided into samples with different recognition difficulty levels based on a preset recognition difficulty level threshold, so as to obtain the recognition difficulty level corresponding to each sample. The recognition difficulty level includes difficult, medium and easy. Sampling is performed in groups until all samples have been sampled at least once, resulting in multiple sampling groups. Each sampling group includes samples of different recognition difficulty levels in a preset first proportion, wherein the third sample has the largest proportion in the first proportion, and the third sample is the sample with the recognition difficulty level of "difficult". Once all the third samples have been sampled once, the third samples are sampled again.

[0009] Furthermore, the micro-expression recognition model includes: A feature extraction network based on a convolutional neural network is used to extract spatial features of the micro-expression images, the spatial features including local details and global structural information; A time-series dynamic modeling module based on a long short-term memory network is used to generate hidden state vectors based on the spatial features and to generate time series features based on multiple hidden state vectors. A fully connected layer is used to map the hidden state vector of the time series features to the category dimension; A classifier is used to receive the output of the fully connected layer and convert the output into a confidence level for each classroom state category, and take the classroom state category with the highest confidence level as the prediction result.

[0010] Furthermore, training the preset micro-expression recognition model based on the first training dataset includes: The loss function of the micro-expression recognition model is selected according to the task. When the task is an emotion and attention state classification task, the loss function is the cross-entropy loss function; when the task is a feature discrimination task between samples, the loss function is the triplet loss function; when the task is a time series modeling task, the loss function is the temporal consistency loss function. The training of the micro-expression recognition model is optimized using the cross-entropy loss function, the triplet loss function, and the temporal consistency loss function.

[0011] Furthermore, updating the recognition difficulty index of the sample based on the first information includes: The prediction result of the fourth sample based on the micro-expression recognition model is compared with the classroom status label corresponding to the fourth sample, and the recognition difficulty index of the sample is updated according to the comparison result to obtain the updated recognition difficulty index of the sample.

[0012] Furthermore, the sampling of the training dataset based on the updated recognition difficulty index includes: The recognition difficulty index of the updated sample and the recognition difficulty index of the sample before the update are weighted and fused to obtain the recognition difficulty index of the sample after weighting and fusion. The training dataset is sampled based on the recognition difficulty index obtained by weighted fusion of each sample to obtain the second training dataset.

[0013] Secondly, this application also provides a micro-expression recognition system based on balanced adaptive group sampling, comprising: The acquisition module is used to acquire a training dataset, wherein a sample of the training dataset includes a student's micro-expression image and classroom status labels; The first processing module is used to calculate the similarity between the first sample and multiple second samples based on multi-scale structural similarity measurement, and to evaluate the recognition difficulty of the first sample based on the multiple similarities corresponding to the first sample, so as to obtain the recognition difficulty index of the first sample. The first sample and the second sample are different samples corresponding to the same student. The second processing module is used to obtain the recognition difficulty index of each sample, and to sample the training dataset based on multiple recognition difficulty indices to obtain the first training dataset. The third processing module is used to train a preset micro-expression recognition model based on the first training dataset to obtain first information and the trained micro-expression recognition model. The micro-expression recognition model is used to predict the student's classroom status based on the micro-expression image. The first information includes the output information of the micro-expression recognition model during the training process. The fourth processing module is used to update the recognition difficulty index of the sample based on the first information, and to sample the training dataset based on the updated recognition difficulty index to obtain a second training dataset, and to retrain the micro-expression recognition model based on the second training dataset.

[0014] The beneficial effects of this invention are as follows: This invention calculates the structural similarity between samples using a multi-scale structural similarity metric to determine the recognition difficulty of each sample. Based on this difficulty, samples are grouped and sampled, and a micro-expression recognition model is trained using the resulting dataset. Simultaneously, the recognition difficulty is adjusted based on the model's output, and sampling is repeated. This periodic resampling and model training, along with continuous updates to sample weights based on training feedback, ensures that the model continuously strengthens and optimizes features for specific samples during training, improving its stability and generalization performance.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a micro-expression recognition method based on balanced adaptive group sampling as described in an embodiment of the present invention. Figure 2 This is another flowchart illustrating a micro-expression recognition method based on balanced adaptive group sampling as described in an embodiment of the present invention; Figure 3 This is a model architecture diagram of a micro-expression recognition model based on a balanced adaptive group sampling micro-expression recognition method as described in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0020] Example 1 It's important to note that, to enhance student engagement in the classroom, visual perception systems can help teachers monitor students' classroom status in real time. Micro-expression recognition technology has significant application value in classroom teaching. It can capture subtle emotional changes and fluctuations in students' attention, helping teachers more accurately understand student participation and adjust teaching strategies accordingly. As a specific recognition task, micro-expression recognition aims to extract emotional features from subtle and instantaneous facial expression changes, revealing students' true emotional reactions and psychological states in an unconscious state, providing more objective and real-time emotional and attention analysis support for classroom teaching.

[0021] Micro-expression recognition is a key task in visual perception and emotion computing, aiming to recover an individual's true emotional state from extremely short-lived, low-amplitude facial muscle movements. Unlike typical macro-expressions, micro-expressions are very short-lived, minimal in amplitude, and easily masked by head posture, lighting changes, and occlusion. Therefore, they place higher demands on the spatial resolution, temporal response capability, and noise robustness of the perception system. In classroom teaching scenarios, micro-expressions can reflect students' instantaneous emotions, comprehension difficulties, and attention fluctuations, providing crucial clues for achieving intelligent classroom feedback and personalized teaching support.

[0022] In recent years, the rapid development of deep learning technology has significantly improved the performance of micro-expression recognition. Convolutional Neural Networks (CNNs) excel at extracting spatial features of the face, while 3D convolutional or temporal networks (such as 3D-CNN, CNN+RNN / LSTM) can effectively model the temporal evolution of expressions. Through attention mechanisms and local feature enhancement modules, models can focus more on discriminative facial regions (such as the area around the eyes and corners of the mouth). These deep learning methods, through end-to-end training frameworks, adaptively learn multi-scale, multi-channel spatial-temporal features, reducing reliance on manual feature engineering, optimizing task objectives, and enhancing the model's generalization ability in complex scenarios. Compared with traditional methods, these techniques demonstrate stronger representational capabilities and can further improve performance through continuous training on large-scale data. Although deep models have significant advantages in representation learning, the quality of training data and sample distribution still profoundly affect the final recognition results. Micro-expression datasets often face problems such as class imbalance, sample scarcity, and uneven distribution of easy and difficult samples. Samples for certain emotion categories are scarce, easy samples account for a high proportion and are highly repetitive, while the "difficult samples" that can truly improve the model's discriminative ability are relatively scarce. If training samples are not selected properly, the model is prone to biased learning, overfitting to easy samples or ignoring boundary samples, thus affecting recognition accuracy and generalization ability. In this case, sampling strategies become one of the key means to improve the performance of micro-expression recognition. Effective sampling methods reconstruct or reweight training samples at the data level, thereby balancing class distribution, amplifying the influence of difficult samples, and guiding the model to focus on samples with stronger discriminative power. Common sampling methods include undersampling / oversampling based on class weights, difficult sample mining, and online sample resampling. Studies have shown that single static resampling often fails to balance multiple classes and multiple difficulty samples simultaneously, while dynamic and adaptive sampling methods based on model feedback are more effective in guiding the learning process. For example, dynamically adjusting the sampling probability based on the model's current prediction confidence or loss can achieve a "from easy to difficult" or "learning by difficulty" training strategy, thereby improving the model's sensitivity to subtle differences. In addition, by grouping samples by difficulty or semantics and using scientific sampling strategies within / between groups, cross-class discrimination boundaries can be strengthened while maintaining intra-class diversity, which is particularly important for the recognition of subtle and easily confused micro-expressions.

[0023] Identifying emotional features from fleeting and subtle micro-expressions is more challenging than identifying emotional states from natural facial expressions. In recent years, research has increasingly focused on leveraging deep learning models to enhance feature representation capabilities and spatiotemporal dynamic modeling accuracy, aiming to capture more subtle and discriminative facial expression changes. However, these methods generally overlook the imbalance in sample distribution and differences in learning difficulty during training, causing models to tend to focus on easily identifiable samples while neglecting challenging samples with high discriminative value. The main reason for this is that existing training sampling strategies often employ random or fixed distributions, lacking the ability to dynamically adapt to the complexity of sample structure and the model's learning state. This prevents continuous reinforcement and feature optimization of challenging samples during training, thus limiting the stability and generalization performance of micro-expression recognition models in real-world classroom environments.

[0024] To address the aforementioned technical problems, this embodiment provides a micro-expression recognition method based on balanced adaptive group sampling. It utilizes a balanced adaptive group sampling mechanism that combines structure awareness and dynamic feedback to enhance the micro-expression recognition model's ability to discriminate subtle facial features and learn from difficult samples. The key technology is the proposed model architecture incorporating the balanced adaptive group sampling mechanism. This architecture first constructs an initial difficulty prior based on image structural similarity, then dynamically updates the difficulty score by combining the model's prediction confidence, adaptively reconstructing the training batch. Specifically, the system first calculates the structural similarity between each sample and samples with the same identity using Multi-Scale Structural Similarity Measure (MS-SSIM) to obtain an initial recognition difficulty index. The samples are then divided into three categories: "difficult / moderate / easy," and training groups are constructed according to a preset ratio to increase the sampling weight of difficult samples. The training groups are then input into the model to train it. During training, the system collects the prediction confidence and correctness information of the model for each sample in real time. According to the set fusion strategy, the structural prior and model feedback are weighted and fused into a dynamic difficulty score. The system periodically regroups and adjusts the sampling probabilities within and between groups to ensure that the training focuses on high-value samples that have not yet been mastered while retaining sufficient sample diversity to maintain gradient stability. The training groups are continuously optimized, thereby continuously optimizing the model.

[0025] Specifically, refer to Figure 1 The micro-expression recognition method based on balanced adaptive group sampling in this embodiment includes steps S1, S2, S3, S4 and S5; S1. Obtain the training dataset, wherein a sample in the training dataset includes a student's micro-expression image and classroom status label; Specifically, step S1 includes: Acquire the original image sequence, which includes an infrared image sequence and a visible light image sequence; Based on a preset face detection algorithm, the face regions of multiple students in each frame of the original image sequence are located, and multiple first images containing only the face regions are cropped from the images. Affine transformation alignment, scaling and rotation angle unification, pixel normalization and illumination compensation are sequentially performed on multiple first images to obtain multiple micro-expression images; The micro-expression images are labeled based on preset classroom status categories to obtain the classroom status labels corresponding to the micro-expression images; The first micro-expression image sequence and the second micro-expression image sequence are aligned at the frame level to obtain a micro-expression image frame sequence. The first micro-expression image sequence and the second micro-expression image sequence are image sequences composed of multiple frames of micro-expression images of the same student in the same time period, generated based on the infrared image sequence and the visible light image sequence, respectively. Multiple micro-expression image frame sequences and classroom status labels corresponding to each micro-expression image in the micro-expression image frame sequence are obtained to construct the training dataset, where one micro-expression image frame sequence corresponds to one student and one time period.

[0026] It is understandable that, in this embodiment, in combination with Figure 2 and Figure 3 In a classroom setting, raw video data of students in visible light or visible light-infrared dual-modality mode is acquired in real time using video capture devices. This data is used to capture information such as students' micro-expression changes, subtle facial muscle movements, head posture, and gaze direction. Specifically, for raw video data of a certain modality, the preprocessing stage first uses a deep learning-based face detection algorithm to automatically locate the face of each student in the raw video frame and crop it to obtain multiple raw images containing only the student's face region. These face region images undergo affine transformation alignment, scale and rotation angle unification, pixel normalization, and illumination compensation to improve the robustness of subsequent models in extracting subtle expression features. Then, the above processing is performed on multiple frames of the raw video data to obtain multiple micro-expression images of students in each frame. Based on the timestamp and student number corresponding to the micro-expression images, multiple micro-expression images of the same student number at different times within a time period are combined into a micro-expression image sequence. Furthermore, the visible light micro-expression image sequence at the same time is paired with the infrared micro-expression image sequence to ensure that the dual-modal data is completely synchronized in time.

[0027] Furthermore, a corresponding classroom status label is added to each frame of micro-expression images. Each captured facial image corresponds to three distinct data identifiers: student ID, timestamp, and classroom status label. The student ID uniquely identifies the student corresponding to the image, the timestamp records the specific time the image frame was captured, and the classroom status label is manually annotated, indicating the student's behavioral state in that frame, such as focused, distracted, or looking down. These three types of information constitute structured data and are essential attributes for each image frame during dataset construction and training. This completes the construction of structured samples. Then, multiple samples corresponding to each student are integrated based on the timestamp and student ID to construct a training dataset for emotion and attention recognition.

[0028] Preferably, classroom status labels are added to micro-expression images manually. The entire process from video to image is divided into two main stages: fully automated image processing and manual status labeling. Fully automated image processing first utilizes the YOLOv8 + DeepSORT combination and OpenCV tools to achieve fully automated processing. The script automatically reads the classroom video and analyzes it frame by frame. YOLOv8 accurately detects student targets in each frame (filtering irrelevant backgrounds such as desks, chairs, and blackboards) and locates the student area. Then, the DeepSORT multi-object tracking algorithm assigns a unique ID to each student and stably tracks their position across frames (it can handle scenarios such as slight movement and brief occlusion, ensuring that the IDs are not confused). Subsequently, the system automatically creates independent folders named "student_XXX" (XXX is the unique ID). Each folder is divided into "origin" (storing complete student images) and "face" (storing cropped face images). Images of the same student from different frames are named according to the video frame order (e.g., 0001.jpg, 0002.jpg) and then saved in categories. The face images are generated by further cropping from the student images using OpenCV, focusing on facial expressions to assist in subsequent judgment. After automated processing, the manual status labeling stage begins. Since student states such as focus, inattentiveness, raising hands, and looking down fall under the category of subjective behavior, manual operation based on the standardized dataset is required. Labelers can view images arranged frame-by-frame in each student's folder (prioritizing face images to improve judgment efficiency). They can label each frame with the corresponding state by creating a label file (e.g., labels.txt, recording by "image number + state," e.g., "0001 Focused," "0002 Inattentive") or using tools like LabelImg to generate standardized files. This ultimately forms a complete dataset of "student unique ID - time-series image - state label."

[0029] S2. Calculate the similarity between the first sample and multiple second samples based on multi-scale structural similarity measurement, and evaluate the recognition difficulty of the first sample based on the multiple similarities corresponding to the first sample to obtain the recognition difficulty index of the first sample. The first sample and the second sample are different samples corresponding to the same student. Specifically, the structural similarity measure of multi-scale structural similarity (MS-SSIM) is used to calculate the structural similarity between each sample and samples of the same class, that is, the structural similarity between a sample corresponding to a student and other samples corresponding to that student, to obtain the similarity score. Based on the similarity score, the initial difficulty prior of the samples is obtained. Samples are divided into three categories—difficult, moderate, and easy—according to the recognition difficulty index, and the sampling weight of difficult samples is increased according to a preset ratio to enhance the model's focus on high-value samples in the early stages of training. Preferably, a separate processing script is used to calculate the image similarity results and save them as a txt file. This file is then used in the first round of reconstructing the sampling order of the samples; the results are read in, and the sampling order is adjusted based on the results.

[0030] Understandably, in this embodiment, the multimodal student micro-expression image sequences obtained after data preprocessing are first reconstructed using a Balanced Adaptive Group Sampling (BAGS) mechanism before training. An initial recognition difficulty index for each sample is calculated based on a multi-scale structural similarity metric, and the samples are divided into three categories—difficult, moderate, and easy—combined with sample category information to determine the initial sampling order and weights in the training batch. For a micro-expression image sequence in a sample corresponding to a particular student, its MS-SSIM similarity is calculated one by one with other micro-expression image sequences belonging to that student, and the average of all similarity results is used to obtain the recognition difficulty index for that sample.

[0031] It is worth noting that, due to the lack of reference data in the first round of sampling, it is impossible to determine the recognition difficulty of an image or whether the image is a difficult sample. Therefore, this embodiment uses the results of multi-scale structural similarity measurement as a reference for sampling. Multi-scale structural similarity measurement is an indicator for evaluating image similarity, and the sampling order is reconstructed based on the results of this indicator. After the first round of model training, the true prediction result can be obtained. Subsequent times, the difficulty assessment results of the first round can be continuously corrected based on the true prediction results. Then, before the next round of training, the sampling order is reconstructed based on the corrected results. The sampling mechanism design of this embodiment transforms traditional random sampling optimization into balanced adaptive group sampling, realizing the reconstruction of samples and avoiding the phenomenon of imbalanced sample distribution and differences in learning difficulty during the training process, as well as the problem that the model is more likely to focus on easily identifiable samples while ignoring difficult samples with high identification value.

[0032] S3. Obtain the recognition difficulty index of each sample, and sample the training dataset based on multiple recognition difficulty indices to obtain the first training dataset; Specifically, step S3 includes: The multiple recognition difficulty indicators are sorted, and the samples are divided into samples with different recognition difficulty levels based on a preset recognition difficulty level threshold, so as to obtain the recognition difficulty level corresponding to each sample. The recognition difficulty level includes difficult, medium and easy.

[0033] Sampling is performed in groups until all samples have been sampled at least once, resulting in multiple sampling groups. Each sampling group includes samples of different recognition difficulty levels in a preset first proportion, wherein the third sample has the largest proportion in the first proportion, and the third sample is the sample with the recognition difficulty level of "difficult". Once all the third samples have been sampled once, the third samples are sampled again.

[0034] Understandably, in this embodiment, by processing all micro-expression samples of each student using the method described in step S2, a recognition difficulty index corresponding to each sample is obtained, thereby generating a complete sample difficulty table. Subsequently, the samples of each student are sorted according to their recognition difficulty index from smallest to largest, where a smaller index value indicates more subtle micro-expression changes and higher recognition difficulty. The recognition difficulty index of each sample is normalized to the 0-1 range, and preliminary divisions are made based on this range: samples in the 0-0.33 range are difficult samples, samples in the 0.34-0.66 range are moderately difficult samples, and samples in the 0.67-1 range are easy samples. Through this division, the system can make a preliminary judgment on the sample difficulty before training, providing a basis for subsequent balanced adaptive group sampling and training batch construction.

[0035] Furthermore, after generating the sample difficulty table, sampling is performed in groups. Each group contains difficult, moderately difficult, and easy samples, with difficult samples making up the majority and moderately and easily difficult samples making up the minority, to ensure that the model prioritizes high-value samples during training. Preferably, the sampling logic is as follows: for difficult samples, a repeated sampling strategy with replacement is adopted, that is, when all difficult samples have been sampled, sampling can continue from the beginning; while moderately difficult and easy samples are sampled only in a preset small proportion in each group, and each group maintains the same sampling proportion. In this way, the system generates the final sampling order of the training batch and feeds it into the model for learning, enabling the model to continuously strengthen its ability to distinguish samples with subtle facial expression changes during training, while maintaining sample diversity and training stability.

[0036] Once all samples have been sampled at least once, the sampling for that batch is complete and the samples are fed into the model.

[0037] S4. Train a preset micro-expression recognition model based on the first training dataset to obtain first information and the trained micro-expression recognition model. The micro-expression recognition model is used to predict the student's classroom status based on the micro-expression image. The first information includes the output information of the micro-expression recognition model during the training process. Specifically, in this embodiment, a sentiment and attention recognition model, i.e., a micro-expression recognition model, is constructed based on a deep learning framework using a balanced adaptive group sampling (BAGS) mechanism. The model includes a feature extraction network, a temporal dynamic modeling module, and an attention enhancement unit. After initializing the network parameters, the training batches sampled in step 3 are used, with the sentiment and attention state labels corresponding to the images as supervision labels, for training. This allows the network to learn the feature expressions and temporal change patterns of students in different modalities. After training, the hyperparameters are fine-tuned based on the performance evaluation results, and the final model parameters are fixed.

[0038] Furthermore, in this embodiment, as Figure 3 As shown, the feature extraction network employs a two-dimensional convolutional neural network (CNN) structure with ResNet-50 as the backbone network to extract spatial hierarchical features from students' micro-expression images. The temporal dynamic modeling module uses a long short-term memory (LSTM) network structure to capture the dynamic information of students' expressions changing over time. The attention enhancement unit guides the network to focus on key areas such as the eyes, eyebrows, and corners of the mouth through a joint mechanism of channel attention and spatial attention, thereby improving the model's ability to discriminate subtle differences in expressions. Preprocessed samples are input into the model, and the corresponding emotion and attention state labels are used as supervision labels for training.

[0039] Understandably, the training batches reconstructed through a balanced adaptive grouping sampling mechanism are input into a two-dimensional convolutional neural network to extract fine-grained spatial features of students' micro-expressions, capturing local micro-changes in facial muscles and overall facial structure information. Simultaneously, keypoint information is combined to assist the attention mechanism in focusing on highly sensitive areas such as the eyes, eyebrows, corners of the mouth, and nostrils. Subsequently, the extracted spatial feature sequences are input into a Long Short-Term Memory (LSTM) network to model the short-term dynamic features of micro-expressions over time, thereby reflecting students' attention fluctuations and subtle emotional changes in the classroom.

[0040] After processing by a Long Short-Term Memory (LSTM) network, the input micro-expression spatial feature sequence is transformed into a dynamic feature representation reflecting changes in facial expressions over time. Specifically, the LSTM generates a hidden state vector for the spatial features of each frame, capturing short-term temporal dependencies and patterns in micro-expression changes, thus forming time-series features. These dynamic features can integrate global and local facial information, providing a representational basis for subsequent student attention state and emotion classification. After this round of model training is completed, the features are fed into a classifier, which provides a confidence score, which is the prediction result.

[0041] Specifically, after the model is trained, the time-series features output by the Long Short-Term Memory (LSTM) network are fed into a fully connected layer and a softmax classifier. The fully connected layer maps the vectors to the class dimension. The softmax classifier transforms the output into a probability (confidence value) for each class. The class with the highest probability is then selected as the prediction result.

[0042] Furthermore, the emotion and attention recognition model in this embodiment employs a supervised learning strategy to iteratively update model parameters, with the loss function selected based on the task. For each task, an appropriate loss function is used; for example, cross-entropy loss is used for emotion and attention state classification tasks; triplet loss is used for inter-sample feature discrimination tasks; and temporal consistency loss is used for time series modeling tasks. The entire process is optimized through the combined use of cross-entropy loss, triplet loss, and temporal consistency loss.

[0043] Understandably, in this system, the emotion and attention recognition model selects the corresponding loss function based on the characteristics of different tasks to optimize model performance.

[0044] For emotion and attention state classification tasks, the model needs to predict discrete category labels, such as a student's focused, distracted, or looking-down state, as well as different emotion categories. Therefore, the cross-entropy loss function is used to measure the difference between the model's output category probability distribution and the true label, thereby guiding the model to adjust parameters and improve classification accuracy.

[0045] For the task of distinguishing features between samples, the model focuses on the relative distance between feature vectors of different samples, hoping that features of samples of the same class are close together and features of samples of different classes are separated. Therefore, the triple loss function is used to encourage the model to learn the discriminative feature space and enhance the ability to recognize micro-expressions that are difficult to distinguish.

[0046] For time series modeling tasks, the model needs to capture the dynamic patterns of micro-expressions changing over time. Therefore, the temporal consistency loss function is used to ensure that the sequence features change smoothly and reasonably over time, and to enhance the sensitivity to short-term micro-expression changes.

[0047] By jointly optimizing cross-entropy loss, triplet loss, and temporal consistency loss, the model can simultaneously improve classification accuracy, feature discrimination, and temporal dynamic modeling capabilities, thereby more accurately reflecting students' attention fluctuations and emotional changes in the classroom.

[0048] For example, in a task involving emotion and attention state classification, during model training, each input image is first processed by a network to extract features, and then a fully connected layer and a softmax classifier are used to output the class probability. For each sample, the error between the predicted probability and the true label is calculated using the cross-entropy loss function, resulting in a scalar loss value. To stabilize training, multiple samples are typically grouped into batches, and the loss of each sample within a batch is averaged to obtain the batch loss value. Subsequently, the gradient is calculated using this batch loss through backpropagation, and the optimizer (such as Adam or SGD) updates the network parameters based on the gradient. After training, the average loss of the training set is recorded as a reference metric for model convergence and performance.

[0049] Furthermore, it can be understood that the training process of the emotion and attention recognition model in this embodiment is specifically as follows: Using labeled student emotion and attention state data, namely micro-expression images and their corresponding labels (student ID, timestamp, and classroom status label), the model is trained with end-to-end global optimization. The Balanced Adaptive Group Sampling (BAGS) mechanism is used to dynamically adjust the sample order, so that the model learns difficult samples first during training and gradually improves its ability to recognize complex samples.

[0050] Training batches are input into a 2D convolutional neural network (CNN) to extract spatial features from facial images, including local details and global structural information. Simultaneously, facial keypoint information is incorporated to assist the network in focusing on high-information regions such as the eyes, eyebrows, and corners of the mouth. The extracted spatial feature sequences are then input into a Long Short-Term Memory (LSTM) network for temporal modeling, used to capture subtle dynamic changes in students' facial muscles, instantaneous expression transitions, and attention fluctuation patterns, reflecting students' subtle emotional reactions and changes in concentration during class. Network parameters are iteratively updated during training through supervised learning to optimize micro-expression recognition and emotion discrimination capabilities. Through the synergistic effect of spatial feature extraction and temporal modeling, information from multiple frames is fused to enhance temporal consistency.

[0051] By optimizing network parameters through backpropagation, the predicted results are made to continuously approach the true labels, thereby improving the overall recognition accuracy and robustness of the model.

[0052] S5. Update the recognition difficulty index of the sample based on the first information, and sample the training dataset based on the updated recognition difficulty index to obtain a second training dataset. Train the micro-expression recognition model again based on the second training dataset.

[0053] Specifically, step S5 includes: The classroom state of the fourth sample predicted by the micro-expression recognition model is compared with the classroom state label corresponding to the fourth sample, and the recognition difficulty index of the sample is updated according to the comparison result to obtain the updated recognition difficulty index of the sample.

[0054] The recognition difficulty index of the updated sample and the recognition difficulty index of the sample before the update are weighted and fused to obtain the recognition difficulty index of the sample after weighting and fusion. The training dataset is sampled based on the recognition difficulty index obtained by weighted fusion of each sample to obtain the second training dataset.

[0055] Understandably, during training, the model's output prediction results and loss information are recorded. After the training of the current batch ends, the recorded prediction results are used to update the sample recognition difficulty index and the training batch order, while the recorded loss information is used for model optimization. The combination of the effects of prediction results and loss information enables the model to continuously strengthen its learning of difficult samples in multiple training batches, while maintaining sample diversity and gradient stability.

[0056] Furthermore, it can be understood that in this embodiment, after the model training is completed, for a student, a corresponding confidence distribution result will be obtained. The confidence distribution result includes the predicted category and predicted confidence of multiple samples corresponding to the student. Further, firstly, the predicted category of the sample with the highest confidence is extracted and compared with the true category of the sample. The predicted category is the emotion and attention category predicted in the model output result, and the true category is the emotion and attention state label assigned to the sample during the data preprocessing stage. If the predicted category and the true category match successfully, it indicates that the sample is relatively easy to identify; if the match fails, it indicates that the sample is relatively difficult to identify. Preferably, for samples that match successfully, their confidence result is amplified, and the result is constrained to the 0-1 range using a maximum-minimum normalization method, so as to reduce their sampling priority in subsequent sorting, thereby prioritizing the training of samples with higher recognition difficulty. The maximum and minimum values ​​are the maximum and minimum values ​​among the multiple predicted confidences corresponding to the student. For samples that fail to match, their confidence result is reduced, and the same normalization process is performed. The degree of amplification or reduction is determined based on the difference between the predicted confidences.

[0057] Understandably, the image similarity results evaluated using MS-SSIM differ from the confidence results obtained after model training. Therefore, it is necessary to continuously update the recognition difficulty results obtained from preprocessing. The recognition difficulty results obtained from preprocessing also have reference value, so the two need to be weighted and combined according to a preset ratio to comprehensively consider the recognition difficulty of the samples judged by both results.

[0058] Furthermore, the above process is performed sequentially on all samples to generate a new sample recognition difficulty table. Subsequently, the new evaluation table is weighted and fused with the historical evaluation table from the previous round, with the new evaluation table as the dominant proportion, thus forming an updated sample weight table. This weight table serves as the sampling basis for the next round of Balanced Adaptive Group Sampling (BAGS) mechanism, enabling dynamic adaptive adjustment of sample difficulty and sampling order. This allows the model to continuously focus on high-value micro-expression samples and subtle attentional fluctuation features during training across multiple training batches, improving recognition performance and training stability.

[0059] Understandably, in the Balanced Adaptive Group Sampling (BAGS) mechanism, although each training round can generate new sample recognition difficulty results based on the latest prediction results, directly using the new weights for sampling may lead to instability in the training process. This is because the prediction results of a single round are affected by factors such as batch sample distribution, random network initialization, and gradient fluctuations, causing some samples to be accidentally classified as easy or difficult to recognize, resulting in drastic changes in sampling weights. This may cause the model to over-adjust some samples or ignore truly high-value samples during training. To alleviate this problem, this embodiment performs a weighted fusion of the newly generated prediction results with the prediction results of the previous round (the first round uses the preprocessed recognition difficulty results, and the subsequent rounds use the weighted prediction results), updating the results with the new evaluation table as the dominant proportion. This smooths the sample difficulty assessment, preserving the sample difficulty information accumulated in historical training while reflecting the model's latest learning state, thus achieving a balance between the stability and dynamic adaptability of the sampling strategy. This weighted fusion ensures that the model continuously focuses on long-term difficult-to-identify samples during training, while adjusting the sampling priority based on the latest prediction results, which helps improve the recognition performance and training stability of micro-expression and subtle attention fluctuation features.

[0060] Furthermore, the adaptive nature is manifested in the process of gradually transitioning from a situation where the old weights are dominant and the new weights are auxiliary in the early stages to one where the old weights are auxiliary and the new weights are dominant.

[0061] It is worth noting that in this embodiment, during training, the model's prediction confidence and loss value for each sample are acquired in real time, and the sample difficulty score is adaptively updated. The structural prior and model feedback are integrated into a comprehensive dynamic difficulty index. The difficulty groups are periodically re-divided, and the sampling probabilities within and between groups are adjusted based on the grouping results to achieve adaptive difficulty training, improving the model's ability to discriminate samples with subtle facial expressions and attentional fluctuations. The adaptively sampled training data is input into the multimodal network, and a supervised learning strategy is used to iteratively update the model parameters. After each training cycle, a validation set evaluation is performed, and the learning rate and sampling ratio are adjusted based on the recognition accuracy and convergence stability.

[0062] In summary, the specific training process in this embodiment includes: Using labeled student emotion and attention state data, the model is trained through end-to-end global optimization.

[0063] The Balanced Adaptive Group Sampling (BAGS) mechanism is used to dynamically adjust the sample order, so that the model prioritizes learning difficult samples during training and gradually improves its ability to recognize complex samples.

[0064] By combining spatial feature extraction and temporal modeling, information from multiple frames is fused to enhance temporal consistency.

[0065] By optimizing network parameters through backpropagation, the predicted results are made to continuously approach the true labels, thereby improving the overall recognition accuracy and robustness of the model.

[0066] It is understood that the multimodal emotion and attention recognition network proposed in this embodiment, which integrates a balanced adaptive group sampling (BAGS) mechanism, is a novel and efficient deep learning training framework aimed at addressing issues such as large sample variability, subtle expression changes, and insufficient learning of easily confused samples in micro-expression recognition. This mechanism introduces a sample difficulty assessment method based on multi-scale structural similarity (MS-SSIM) to dynamically group and optimize the sampling of training data. This allows the model to prioritize learning samples with higher difficulty and subtle emotional changes during training, thereby enhancing the model's ability to recognize complex micro-expressions and attention fluctuations.

[0067] Furthermore, the trained micro-expression recognition model will be applied to real-time classroom teaching. Specifically: The micro-expression image frames of each student and the first information are acquired in real time and input into the trained micro-expression recognition model for prediction to obtain the real-time classroom status of each student.

[0068] Specifically, the classroom video acquisition device acquires visible light or visible light-infrared dual-modal video streams in real time, and obtains a micro-expression image sequence for each student through the same preprocessing as step S1.

[0069] The micro-expression image sequence is input into the trained emotion and attention recognition model to identify the subtle facial expression changes of students and output the emotion category and attention state prediction results corresponding to each frame of the image.

[0070] Furthermore, such as Figure 3 As shown, the final prediction results, after time-series smoothing and confidence averaging, are fed back to the teacher's end or classroom management system in real time through a visualization interface. The students' emotional tendencies and attention concentration are dynamically presented in the form of icons, curves, or heat maps, generating visual feedback, which is pushed to the teacher's display terminal to assist teachers in classroom teaching management and individual attention, adjust teaching strategies in a timely manner, realize intelligent recognition and feedback of classroom status, and verify its actual performance.

[0071] Specifically, the BAGS mechanism dynamically updates the sample difficulty score based on the sample prediction confidence and historical loss in each training cycle, and periodically reconstructs the training batch to achieve sample-balanced training with adaptive difficulty. This design effectively alleviates the problem of overfitting easy samples and neglecting difficult samples that easily occurs in traditional training processes, enabling the model to obtain more discriminative feature representations even with limited training data. Meanwhile, the CNN-LSTM joint network constructed in this embodiment combines multi-layer feature modeling capabilities in both spatial and temporal dimensions, enabling it to capture subtle facial muscle movements and dynamic emotional changes in students. Guided by the attention mechanism, it focuses on highly sensitive areas such as the eyes, eyebrows, and corners of the mouth, thereby significantly improving the accuracy and robustness of emotion and attention state recognition.

[0072] Furthermore, this embodiment utilizes a teacher-side visualization feedback module to display the recognition results in real-time using charts, curves, and heatmaps, enabling dynamic tracking and immediate feedback of student status. This mechanism effectively enhances teachers' ability to perceive students' learning status in the classroom environment, providing data-driven support for teaching decisions. Comprehensive experimental verification shows that this invention outperforms traditional methods in terms of recognition accuracy, response latency, and system stability. It can achieve high-precision recognition and intelligent feedback of students' emotional and attentional states in actual classroom scenarios, demonstrating significant application and promotion value.

[0073] Example 2 Unlike Embodiment 1, this embodiment provides a micro-expression recognition system based on balanced adaptive group sampling, including: The acquisition module is used to acquire a training dataset, wherein a sample of the training dataset includes a student's micro-expression image and classroom status labels; The first processing module is used to calculate the similarity between the first sample and multiple second samples based on multi-scale structural similarity measurement, and to evaluate the recognition difficulty of the first sample based on the multiple similarities corresponding to the first sample, so as to obtain the recognition difficulty index of the first sample. The first sample and the second sample are different samples corresponding to the same student. The second processing module is used to obtain the recognition difficulty index of each sample, and to sample the training dataset based on multiple recognition difficulty indices to obtain the first training dataset. The third processing module is used to train a preset micro-expression recognition model based on the first training dataset to obtain first information and the trained micro-expression recognition model. The micro-expression recognition model is used to predict the student's classroom status based on the micro-expression image. The first information includes the output information of the micro-expression recognition model during the training process. The fourth processing module is used to update the recognition difficulty index of the sample based on the first information, and to sample the training dataset based on the updated recognition difficulty index to obtain a second training dataset, and to retrain the micro-expression recognition model based on the second training dataset.

[0074] Specifically, in this embodiment, the system includes a student-side data collection device, an emotion and attention recognition server, and a teacher-side feedback terminal.

[0075] In use, the student's face is captured in real time by a camera and uploaded to the server. The server uses a balanced adaptive group sampling algorithm to preprocess the images and extract features to identify the student's emotions and attention status. After analysis, the identification results are displayed in real time on the teacher's feedback terminal, showing the distribution and fluctuation trend of the student's status, thus realizing intelligent feedback and intervention support in the teaching process.

[0076] It should be noted that the specific manner in which each module performs its operation in the apparatus described in the above embodiments has been described in detail in the embodiments of the method, and will not be elaborated here.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0078] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A micro-expression recognition method based on balanced adaptive packet sampling, characterized in that, The method comprises: obtaining a training data set, wherein one sample of the training data set comprises a micro-expression image of a student and a classroom state label; calculating similarities between a first sample and a plurality of second samples based on a multi-scale structural similarity measure, and evaluating the identification difficulty of the first sample based on a plurality of similarities corresponding to the first sample to obtain an identification difficulty indicator of the first sample, the first sample and the second sample being different samples corresponding to the same student; obtaining the identification difficulty indicator of each sample, and sampling the training data set based on a plurality of identification difficulty indicators to obtain a first training data set; training a preset micro-expression recognition model based on the first training data set to obtain first information and the trained micro-expression recognition model, the micro-expression recognition model being used to predict the classroom state of a student based on the micro-expression image, and the first information comprising output information of the micro-expression recognition model during the training process; updating the identification difficulty indicator of the sample based on the first information, and sampling the training data set based on the updated identification difficulty indicator to obtain a second training data set, and retraining the micro-expression recognition model based on the second training data set.

2. The micro-expression recognition method based on balanced adaptive packet sampling according to claim 1, characterized in that The method comprises: obtaining an original image sequence, the original image sequence comprising an infrared image sequence and a visible light image sequence; locating a plurality of face regions of students in each frame of image in the original image sequence based on a preset face detection algorithm, and cropping a plurality of first images containing only the face regions from the images; aligning, unifying scale and rotation angle, pixel normalizing and light compensation on a plurality of the first images in sequence to obtain a plurality of micro-expression images; annotating the micro-expression images based on a preset classroom state category to obtain the classroom state label corresponding to the micro-expression images; performing frame-level alignment on a first micro-expression image sequence and a second micro-expression image sequence to obtain a micro-expression image frame sequence, the first micro-expression image sequence and the second micro-expression image sequence being image sequences generated based on the infrared image sequence and the visible light image sequence respectively, and each comprising a plurality of micro-expression images of the same student in the same time period; obtaining a plurality of micro-expression image frame sequences and the classroom state label corresponding to each micro-expression image in the micro-expression image frame sequence, and constructing the training data set, one micro-expression image frame sequence corresponding to one student and one time period.

3. The micro-expression recognition method based on balanced adaptive packet sampling according to claim 1, characterized in that The method comprises: calculating a similarity average value of a plurality of similarities corresponding to the first sample to obtain the similarity average value of the first sample; calculating the similarity average value corresponding to each sample to obtain a plurality of similarity average values, and normalizing a plurality of similarity average values to obtain a plurality of identification difficulty indicators, the identification difficulty indicator being the normalized similarity average value.

4. The micro-expression recognition method based on balanced adaptive packet sampling according to claim 3, characterized in that , the sampling of the training data set based on the plurality of identification difficulty indicators comprises: sorting the plurality of identification difficulty indicators, and dividing the samples into samples of different identification difficulty levels based on a preset identification difficulty level threshold, to obtain the identification difficulty level corresponding to each sample, the identification difficulty level comprising difficult, medium, and easy; sampling in groups until all the samples are sampled at least once, to obtain a plurality of sampling groups, each sampling group comprising a preset first proportion of samples of different identification difficulty levels, wherein the proportion of the third sample in the first proportion is the largest, and the third sample is the sample of the identification difficulty level of difficult; when the third sample has been sampled once, the third sample is resampled.

5. The micro-expression recognition method based on balanced adaptive packet sampling according to claim 2, characterized in that The micro-expression recognition model comprises: a feature extraction network constructed based on a convolutional neural network, used to extract spatial features of the micro-expression image, the spatial features comprising local details and global structure information; a time sequence dynamic modeling module constructed based on a long short-term memory network, used to generate a hidden state vector based on the spatial features, and generate a time sequence feature based on a plurality of hidden state vectors; a fully connected layer, used to map the hidden state vector of the time sequence feature to a class dimension; a classifier, used to receive the output of the fully connected layer, and convert the output to a confidence of each classroom state class, and take the classroom state class with the largest confidence as a prediction result.

6. The micro-expression recognition method based on balanced adaptive packet sampling according to claim 5, characterized in that The training of the preset micro-expression recognition model based on the first training data set comprises: a loss function of the micro-expression recognition model is selected according to a task, wherein when the task is an emotion and attention state classification task, the loss function selects a cross-entropy loss function; when the task is a sample feature distinction task, the loss function selects a triplet loss function; when the task is a time sequence modeling task, the loss function selects a time sequence consistency loss function; the training of the micro-expression recognition model is optimized through the cross-entropy loss function, the triplet loss function, and the time sequence consistency loss function.

7. The micro-expression recognition method based on balanced adaptive packet sampling according to claim 5, characterized in that The updating of the identification difficulty indicator of the sample based on the first information comprises: comparing the prediction result of a fourth sample predicted by the micro-expression recognition model with the classroom state label corresponding to the fourth sample, and updating the identification difficulty indicator of the sample according to the comparison result, to obtain the updated identification difficulty indicator of the sample.

8. The micro-expression recognition method based on balanced adaptive packet sampling according to claim 7, characterized in that The sampling of the training data set based on the updated identification difficulty indicator comprises: weighting and fusing the identification difficulty indicator of the updated sample and the identification difficulty indicator of the sample before updating, to obtain the weighted and fused identification difficulty indicator of the sample; sampling the training data set based on the weighted and fused identification difficulty indicator of each sample, to obtain the second training data set. 9.A micro-expression recognition system based on balanced adaptive packet sampling, characterized in that It comprises: The acquisition module is configured to acquire a training data set including a plurality of samples, each sample including a sequence of micro-expression image frames of a student, and each student corresponding to a plurality of samples; The first processing module is configured to calculate similarities between a first sample and a plurality of second samples based on a multi-scale structural similarity metric, and evaluate a recognition difficulty of the first sample based on a plurality of similarities corresponding to the first sample to obtain a recognition difficulty indicator of the first sample, the first sample and the second samples being different samples corresponding to the same student; The second processing module is configured to acquire the recognition difficulty indicator of each sample, and sample the training data set based on a plurality of recognition difficulty indicators to obtain a first training data set; The third processing module is configured to train a preset micro-expression recognition model based on the first training data set to obtain first information and the trained micro-expression recognition model, the micro-expression recognition model being configured to predict a classroom state of a student based on a micro-expression image, and the first information including output information of the micro-expression recognition model in a training process; The fourth processing module is configured to update the recognition difficulty indicator of the sample based on the first information, sample the training data set based on the updated recognition difficulty indicator to obtain a second training data set, and retrain the micro-expression recognition model based on the second training data set.

Images