Classroom facial expression recognition method based on improved RT-DETR model
By improving the CAFI, MCFEConv, and RepC3_LWGA modules of the RT-DETR model, the problems of insufficient small-scale facial feature extraction and background interference in classroom facial expression recognition were solved, and high-precision expression recognition was achieved in complex classroom environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHANGCHUN NORMAL UNIV
- Filing Date
- 2025-12-09
- Publication Date
- 2026-05-01
AI Technical Summary
Existing RT-DETR models suffer from insufficient small-scale facial feature extraction and a lack of synergistic fusion between the fine-grained features required for expression classification and the global features of the detection task in classroom facial expression recognition tasks. This results in low recognition accuracy and makes it difficult to apply effectively in complex classroom environments.
The improved RT-DETR model enhances the ability to extract and filter facial expression features in the classroom by designing a CAFI module in the backbone network and an MCFEConv convolutional feature enhancement module and a RepC3_LWGA module in the head network. It combines local feature enhancement, spatial attention, adaptive positional encoding and Transformer global modeling to suppress background interference and enhance the robustness and discriminativeness of features.
It significantly improves the stability and accuracy of facial expression recognition in the classroom, and can effectively identify subtle changes in facial expressions in classroom scenarios with multiple students, long distances, and complex lighting conditions. It solves the problems of insufficient feature expression and low recognition accuracy of traditional models in complex backgrounds.
Smart Images
Figure CN121963268A_ABST
Abstract
Description
A Classroom Facial Expression Recognition Method Based on an Improved RT-DETR Model Technical Field
[0001] This invention belongs to the fields of image processing technology and education, and relates to a facial expression recognition method, specifically a classroom facial expression recognition method based on an improved RT-DETR model. Background Technology
[0002] With the deep implementation of smart education concepts, classroom teaching is shifting from a teacher-centered to a student-centered approach. Students' real-time learning status has become a core basis for optimizing teaching strategies and improving classroom quality. Facial expressions, as a direct external manifestation of human emotions and psychological states, can effectively reflect students' level of knowledge absorption. Therefore, facial expression recognition technology in classroom settings has become a crucial bridge connecting teaching and learning. However, traditional classroom status monitoring relies heavily on manual observation by teachers, which is not only limited by teachers' energy and subjective judgment, making it difficult to cover the real-time status of every student in a large-scale classroom, but also unable to achieve quantitative analysis and historical tracing of students' facial expression changes. There is an urgent need for an automated, high-precision facial expression recognition technology to fill this gap.
[0003] In recent years, facial expression recognition technology has moved from the traditional machine learning stage to a deep learning-dominated stage. Early methods manually extracted features such as local binary patterns and histograms of oriented gradients, combining them with classifiers such as support vector machines and random forests to determine expressions. However, these methods were limited by the representational power of manually extracted features and had poor robustness in complex environments. With the rise of deep learning technology, convolutional neural networks, with their powerful local feature extraction capabilities, significantly improved the accuracy of expression recognition. Subsequent architectures such as Vision Transformer further captured fine-grained features of expressions through global attention mechanisms, enabling breakthroughs in recognition accuracy in controlled environments such as laboratories. However, the classroom setting is not an ideal environment. Its dynamics and complexity pose unique challenges to the technology. For example, students may shift their posture due to looking down to take notes or turning their faces to communicate, or their faces may be obscured by masks, glasses, or books. In addition, classroom lighting changes drastically over time and from different locations. Traditional deep learning models are prone to feature extraction failures and sharp drops in recognition accuracy.
[0004] Addressing the core need for simultaneous detection and recognition of multiple faces in classrooms, integrated face detection and expression recognition technology has become a mainstream research direction. This approach requires accurately locating all student faces in a complex classroom background before classifying the expressions of each face. The collaborative performance of these two stages directly determines the overall recognition effect. RT-DETR, as an end-to-end object detection model based on the Transformer architecture, achieves a balance between real-time performance and detection accuracy due to its advantages of not requiring pre-training anchors and its strong ability to capture long-sequence features. Compared to traditional detectors such as the YOLO series and Faster R-CNN, it is better suited to scenarios with dynamically changing numbers of faces and large differences in target scale in the classroom. However, the original design of RT-DETR was for general object detection and was not optimized for the specific needs of classroom face and expression recognition. On the one hand, its feature extraction capability for small-scale faces is insufficient, leading to missed detections. On the other hand, the model only focuses on the detection task and does not collaboratively integrate the fine-grained features required for expression classification with the global features of the detection task. This results in the subsequent expression recognition module being unable to utilize the feature information from the detection stage, leading to information loss in the overall process. Summary of the Invention
[0005] To address the aforementioned problems of existing RT-DETR algorithms in handling classroom facial expression recognition tasks, this invention provides a classroom facial expression recognition method based on an improved RT-DETR model.
[0006] The objective of this invention is achieved through the following technical solution:
[0007] A classroom facial expression recognition method based on an improved RT-DETR model includes the following steps:
[0008] Step 1: Obtain a dataset of facial expression images from the classroom;
[0009] Step 2: Preprocess the dataset using a computer;
[0010] Step 3: Use the classroom facial expression recognition and detection model based on the improved RT-DETR model to train the dataset. The improved RT-DETR model designs the CAFI module in the backbone network to replace the AIFI module, the MCFEConv convolutional feature enhancement module in the head network and backbone network to replace the Conv convolution, and the RepC3_LWGA module in the head network to replace the traditional RepC3 module.
[0011] Step 4: Train the classroom facial expression recognition and detection model using the training set and validation set. After obtaining the optimal detection model, use the test set to evaluate the performance of the classroom facial expression recognition and detection model.
[0012] Step 5: Combine monitoring with the optimal detection model to perform real-time recognition and detection of students' facial expressions in the classroom.
[0013] Compared with the prior art, the present invention has the following advantages:
[0014] 1. This invention proposes an improved RT-DETR model for classroom facial expression recognition applications. A CAFI module is designed in the backbone network. Through a joint design of local feature enhancement, spatial attention guidance, adaptive positional encoding, and Transformer global modeling, more accurate extraction and filtering of classroom facial expression features is achieved. It can enhance local facial texture details, highlighting subtle expression changes in key areas such as eyebrows, eyes, and corners of the mouth. Simultaneously, it utilizes spatial attention to suppress interference from complex backgrounds such as blackboards, desks, and chairs, making the model more focused on student facial areas. In classroom scenarios with multiple students, long-distance shooting, and complex lighting, CAFI effectively improves the robustness and discriminativeness of feature representation. Furthermore, by introducing learnable two-dimensional positional encoding and combining it with the global dependency modeling capability of Transformer, the model can simultaneously capture local details and overall expression patterns, solving problems such as insufficient local texture representation, severe background noise interference, and missing positional semantics in traditional CNNs or simple Transformers. This significantly improves the stability and accuracy of classroom expression recognition tasks.
[0015] 2. An MCFEConv convolutional feature enhancement module was designed in the head network. It improves the feature representation ability of faces at different distances and sizes through multi-scale convolutional branches, enhances the response of key expression regions such as eyes, eyebrows, and corners of the mouth through channel attention, suppresses classroom background interference such as blackboards, walls, and desks through spatial attention, and maintains feature stability and improves training convergence speed through residual structure. This enables the network to obtain stronger expression detail modeling ability and higher feature robustness, solves the problem of insufficient feature representation caused by multiple people, small faces, and large scale changes in classroom scenes, solves the problem that conventional convolution is difficult to capture micro-expression details, solves the problem that complex background interference affects recognition accuracy, and solves the feature loss problem caused by gradient decay in deep networks.
[0016] 3. A RepC3_LWGA module was designed in the head network. By introducing a lightweight global attention sequence into the RepC3 structure, it achieves joint modeling of local convolutional features and global contextual features, possessing advantages such as high structural efficiency, low computational cost, and strong feature representation capabilities. The module can simultaneously preserve local details and global semantic relationships, enhance the response of facial expression regions, suppress classroom background interference, and improve the discriminativeness and robustness of features. This module can effectively solve the problem of insufficient feature representation caused by multiple students, multi-scale faces, and complex backgrounds in classroom scenarios, address the problem of incomplete expression information caused by the difficulty of traditional convolution in capturing long-range dependencies, and avoid the problem of excessive computational cost of self-attention, thereby significantly improving the stability and accuracy of classroom facial expression recognition. Attached Figure Description
[0017] Figure 1 is a flowchart of a classroom facial expression recognition method based on an improved RT-DETR model;
[0018] Figure 2 shows the detection model of the classroom facial expression recognition method based on the improved RT-DETR model;
[0019] Figure 3 is a schematic diagram of the CAFI module structure;
[0020] Figure 4 is a schematic diagram of the MCFEConv convolutional feature enhancement module structure;
[0021] Figure 5 is a schematic diagram of the RepC3_LWGA module structure;
[0022] Figure 6 is a schematic diagram of the LWGA Block module structure. Detailed Implementation
[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.
[0024] This invention provides a classroom facial expression recognition method based on an improved RT-DETR model, the method comprising the following steps:
[0025] Step 1: Obtain a dataset of facial expression images from the classroom.
[0026] Step 2: Preprocess the dataset using a computer.
[0027] Step 3: Use the classroom facial expression recognition and detection model based on the improved RT-DETR model to train the dataset. The improved RT-DETR model designs the CAFI module in the backbone network to replace the AIFI module, designs the MCFEConv convolutional feature enhancement module in the head network and backbone network to replace the Conv convolution, and designs the RepC3_LWGA module in the head network to replace the traditional RepC3 module.
[0028] In this step, as shown in Figure 3, the implementation process of the CAFI module is as follows: The input feature map is first imported as a four-dimensional tensor into the local location enhancement submodule. This submodule consists of a concatenation of depthwise separable convolutions and pointwise convolutions, which are applied in the spatial domain. Depthwise convolution models independent local receptive fields for each channel, and then... Convolution completes linear recombination between channels, explicitly enhancing the local texture structure and edge information of the input feature map. The feature map processed by this local enhancement submodule is added and fused with the original input feature map in a residual manner to form an intermediate feature representation that combines the original global semantics with local fine-grained expression cues. Subsequently, the intermediate features are fed into a lightweight spatial attention gating submodule. By applying global average pooling and global max pooling operations to the channel dimension respectively, two single-channel spatial response maps are generated. After concatenation in the channel dimension, they are input into convolution and activation operators to calculate a normalized spatial attention weight map. This spatial attention weight map weights the intermediate features position by position, explicitly suppressing the responses of classroom background areas such as blackboards, desks, and chairs, highlighting the concentrated areas of student faces and their key expression regions, thereby obtaining a salient feature map after background interference suppression. Building upon this, a two-dimensional sine and cosine positional encoding matching the spatial resolution of the feature map is further constructed. A learnable scaling factor is introduced to adaptively adjust the two-dimensional sine and cosine positional information. The positional encoding and the spatially attention-modulated feature sequence are input together into a Transformer encoding unit based on multi-head self-attention and a feedforward network, completing the modeling of global dependencies and the deep nonlinear mapping of high-dimensional features. The encoded sequence features are then restored to a four-dimensional feature map form consistent with the input through shape transformation, serving as the final output of the CAFI module. Through the cascaded system of local position enhancement, spatial attention guidance, and self-attention global modeling, multi-scale joint extraction and saliency screening of complex facial expression features in classroom scenarios are achieved. This effectively improves the discriminability and robustness of feature representation under conditions of multiple students, complex backgrounds, and subtle expression changes, providing stable and reliable feature support for high-level semantic tasks such as classroom facial expression recognition, and ensuring the accuracy and reliability of recognition results from the feature level. Its forward propagation mathematical expression can be expressed as:
[0029]
[0030] in, This represents the input feature map, with dimensions... , Represents the real number field; Indicates batch size, Indicates the number of channels in the feature map. Indicates the feature map height. Indicates the width of the feature map; This represents the intermediate feature map after local enhancement, by and Adding the residuals together, we get the same result. Dimension; This represents a local augmentation mapping, consisting of depthwise separable convolutions and pointwise convolutions. Represents the spatial attention weight graph, with dimensions... ; This represents the Sigmoid activation function, and its output is... ; This represents the convolution operator that generates the spatial attention map; This represents the global average pooling operator along the channel dimension; Represents the global max-pooling operator along the channel dimension; This indicates a channel-dimensional splicing operation; This represents element-wise multiplication; Represents a salient feature map, by Yes, the same as dimension X; Indicates a characteristic shape transformation operation; Represents a 3D feature sequence, dimension ; Indicates the total number of pixels in the space; This represents adaptive two-dimensional sine and cosine position coding, the same as... Dimension; This represents the learnable positional encoding scaling factor; This represents a two-dimensional sine and cosine position coding function; This represents the feature sequence after Transformer encoding, and is the same as... Dimension; This represents the Transformer encoding operator, which consists of a multi-head self-attention network and a feedforward network. Indicates the inverse shape transformation operation; This indicates the final output feature map of the CAFI module, which is the same as... Dimension.
[0031] In this step, the implementation process of the MCFEConv convolutional feature enhancement module is as follows: First, the input feature map of the future autonomous backbone network or the previous level feature fusion unit is denoted as... ,by A multi-branch convolutional structure is constructed for the input to simultaneously characterize facial regions at different scales and their facial expression details in a classroom scene. Specifically, the first branch uses a 1×1 convolution to reorganize the channel dimensions while maintaining spatial resolution, achieving fine-grained recoding of local texture information; the second branch uses a standard... The convolutional layer slides within the feature map space, extracting facial expression features adapted to a standard-scale student face region through multiplication and summation of the kernel weights and corresponding elements; the third branch employs dilation with a dilation rate. Convolution expands the effective receptive field without increasing parameter overhead, enhancing the ability to model large-scale contextual information such as distant students and the overall classroom layout. The multi-scale features output from the three branches are concatenated in the channel dimension and then processed... The convolutional branch fusion layer compresses the data back to the original channel dimension, resulting in an intermediate feature map that fuses multi-scale spatial receptive fields. Subsequently, to further highlight the channel-dimensional semantic information related to facial expression recognition, in the intermediate feature map... Introducing a channel attention mechanism, through the... Global average pooling is applied to obtain a global channel description vector. This vector is then transformed into a normalized channel weight vector through a compressed-dilated mapping constructed using two layers of pointwise convolution and a non-linear activation function. Finally, this normalized weight vector is applied in the form of channel-wise multiplication. The feature map of the completed channel recalibration is obtained. This suppresses redundant channel responses unrelated to facial expressions, enhancing the expressive power of key areas such as eyebrows, eyes, and mouth. Building upon this, a spatial attention mechanism is further introduced to suppress interference from complex classroom backgrounds: two single-channel response maps are obtained by applying average pooling and max pooling respectively along the channel dimension, and then concatenated along the channel dimension before being input into a larger receptive field. Convolutional layers and the sigmoid activation function generate spatial attention maps. and in an element-wise multiplication manner with Multiplying them yields the feature map with enhanced spatial saliency. This allows for the highlighting of high-response areas concentrated on student faces, while effectively suppressing interfering responses from background areas such as the blackboard, walls, and desks. Finally, After lightweight The convolutional and normalization layers perform a linear transformation and are then compared with the input feature map. Residual mapping fusion is performed to obtain the output features of the MCFEConv convolutional feature enhancement module. Through the joint design of multi-scale convolutional branch modeling, channel attention recalibration, and spatial attention-guided background suppression, efficient extraction and filtering of multi-scale facial expressions and subtle facial variations in classroom scenarios are achieved. This significantly improves the discriminativeness and robustness of feature representations, providing more discriminative and stable feature inputs for high-level semantic tasks such as classroom facial expression recognition. It ensures the accuracy and reliability of recognition results in complex teaching environments from a feature perspective. Its forward propagation mathematical expression can be represented as:
[0032]
[0033] Where X represents the input feature map, and the dimension is... (B is the batch size, C is the number of input channels, H is the feature map height, and W is the feature map width). Indicates the first one convolution; This represents the first normalization operator; Represents a non-linear activation function; This represents the output feature map of the first branch, with dimension [not specified]. ( (Number of branch channels); Indicates the second one convolution; The second normalization operator is represented; This represents the third normalization operator; This represents the output feature map of the third branch; This represents a feature concatenation operation along the channel dimension. Indicates the 4th convolution; This represents the fourth normalization operator; This represents the feature map after multi-branch splicing and fusion; Indicates global average pooling; Indicates a fully connected layer; This represents the Sigmoid activation function; Represents the channel attention weight vector; ⊙ represents element-wise multiplication; This represents the feature map after channel recalibration; This represents average pooling along the channel dimension; This represents max pooling along the channel dimension; express convolution; Represents the spatial attention weight map; This represents the feature map after spatial attention weighting; Indicates the 5th convolution; Y represents the 5th normalization operator; Y represents the final output feature map.
[0034] In this step, the implementation process of the RepC3_LWGA module is as follows: First, the input feature map of the future autonomous backhaul network or the feature fusion unit of the previous stage is denoted as... The input features are divided into two parallel branches by the feature splitting unit inside the RepC3 structure. For the main branch, through... Convolution compresses the number of channels, reducing the number of input channels. Mapped to the number of hidden channels This approach reduces the number of parameters and computational cost while constructing low-rank feature representations required for subsequent attention modeling. The compressed features are sequentially fed into a lightweight global attention sub-module sequence consisting of n LWGA Block modules cascaded together. Each LWGA Block module k adaptively weights and models features at different scales and semantic levels based on the current network stage parameters. By performing long-range dependency capture and important feature recalibration operations in the spatial and channel domains, it gradually enhances the discriminative response related to classroom faces and their expressions, suppresses background interference information unrelated to the target, and obtains the backbone enhanced features. Simultaneously, the shortcut branch also processes the input feature map... Apply Convolution is used to preserve the original structural information and local detail representations, providing a stable reference baseline for subsequent feature fusion. The enhanced features of the main branch, obtained by stacking LWGA Block modules, are concatenated with the features of the shortcut branches along the channel dimension to form an intermediate feature tensor that fuses multi-scale attention and structural information. Subsequently, a... Convolution performs channel compression and linear reconstruction on the concatenated feature maps, mapping the number of channels to the target output channels. The output characteristics of the RepC3_LWGA module are obtained. Through the modular design of the RepC3 and LWGA lightweight global attention sequences, while maintaining controllable overall parameter count and computational complexity, the network's ability to represent multi-scale facial targets and subtle facial expression changes in classroom scenarios is effectively improved. This enhances the discriminativeness and robustness of feature representations, providing more stable and sufficient feature inputs for subsequent detection and expression recognition subtasks. Its forward propagation mathematical expression can be represented as:
[0035]
[0036] in, This represents the input feature map of the RepC3_LWGA module, with dimensions... (B represents the batch size) (H represents the number of input channels, H is the feature map height, and W is the feature map width). Indicates the action on the main branches. Convolution is used to convert the number of channels from... Compressed to the number of hidden channels ; This represents the normalization operator on the main branch; Represents a non-linear activation function; This represents the intermediate features of the main branches after initial mapping, with dimensions... ,Depend on through After normalization, through Activated; Indicates the number of LWGA Block modules; This represents the lightweight global attention mapping operator corresponding to the k-th LWGA Block module; `stage` is a control parameter indicating the current network layer or scale information; `k` represents the index of the LWGA Block module, with a value ranging from... ; This represents the intermediate features of the main branch after the k-th LWGA Block module has been applied, with dimension [not specified]. ,Depend on Acting on get; This represents the final enhanced feature of the backbone branch after stacking n LWGA Block modules, with dimension [missing information]. ; Indicates the action applied to shortcut branches. convolution; This represents the normalization operator on the shortcut branch; Represents shortcut branch features, dimensions ,Depend on through After normalization, through The activation function is used to obtain; This represents a feature concatenation operation along the channel dimension; This represents the intermediate features after merging, and the dimension. ,Depend on splicing and get; Indicating the fusion stage Convolution is used to convert the number of channels from... Mapped to the number of output channels ; Represents the normalization operator for the fusion phase; This indicates the number of output channels of the RepC3_LWGA module; This represents the output feature map of the RepC3_LWGA module, with dimensions... ,Depend on through convolution, After normalization, through The activation function is obtained.
[0037] Furthermore, the LWGA Block (Lightweight Global Attention Block) module is used to model global dependencies in the feature map while keeping computational costs manageable. The specific implementation process is as follows: The input feature map is denoted as... First, through three sets of shared space dimensions Convolution respectively for Perform linear projection to generate query features Key features K and value features V are recombined and compressed in the channel dimension without changing the spatial resolution, providing a suitable feature representation for subsequent attention calculations. Then, to reduce the computational complexity of global attention, downsampling or pooling operations are applied to K and V in the spatial dimension to obtain a resolution of h× Compression bond features AND value characteristics This significantly reduces the number of positions involved in attention operations while maintaining the main semantic distribution. Based on this, Flattened along the spatial dimension into a sequence, a lightweight global attention weight matrix is constructed using matrix multiplication and scaled dot product. reuse logarithmic features We perform a weighted summation to obtain an intermediate response that incorporates global context information. .Will Reshape the input back to the same spatial dimensions to obtain the output O in the form of a feature map, and then... Output mapping units composed of convolution and nonlinear activation functions generate incremental responses. Finally, the residuals are compared with the original input feature map. The output characteristics of the LWGA Block are obtained by summing them. Through the aforementioned lightweight global attention modeling method based on spatial compression, the LWGA Block achieves explicit expression of long-range dependencies while maintaining a low number of parameters and computational cost. It can highlight key region responses related to the target face and its expressions, significantly improving its modeling capabilities for multiple students, multi-scale faces, and subtle expression changes in complex classroom scenarios. This is beneficial for maintaining high recognition accuracy and stability even in situations with occlusion, lighting changes, and background interference. Its forward propagation mathematical expression can be represented as:
[0038]
[0039] in, This represents the input feature map, with dimensions... ; Indicates the generation of the query convolution; Represents the Query feature map, dimensions ; Indicates the generation of the key convolution; Represents the key feature map, dimension ; Indicates the generation of Value convolution; Represents the Value feature map, dimension ; Indicates pooling operation; This represents the pooled key feature map, with dimensions... , The height of the feature map after pooling. The width of the feature map after pooling; This represents the pooled value feature map, with dimensions... ; Indicates a characteristic shape transformation operation; express The reshaped sequence form, dimensions ; This represents the total number of pixels in the original feature map space; express The reshaped sequence form, dimensions ; This represents the total number of pixels in the feature map space after pooling; express The reshaped sequence form, dimensions ; This represents the Softmax activation function; This represents the attention weight matrix, with dimensions... ; Indicates the number of channels The square root of; This represents the attention-weighted sequence features, with dimension [not specified]. ; Indicates the inverse shape transformation operation; express The reshaped feature map format, dimensions ; Indicates the output terminal convolution; Represents a non-linear activation function; Represents the residual features after feature enhancement, with dimensions... ; This represents the final output feature map, with dimensions... .
[0040] Step 4: Train the classroom facial expression recognition and detection model using the training and validation sets. After obtaining the optimal detection model, evaluate its performance using the test set. The specific steps are as follows:
[0041] Step 41: Input the training set and validation set into the classroom face expression recognition and detection model of the improved RT-DETR model, and set the number of training iterations. As the number of training iterations increases, the loss function curve of the detection model gradually converges. When the loss function curve converges and stabilizes, the detection model is trained to the optimal state, and its optimal model weight file is saved.
[0042] Step 42: Input the images to be detected from the test set into the trained optimal behavior recognition and detection model, and output the detected images, where the detected images include the type of each detected target, and the position of each target in the target detection image is marked;
[0043] Step 5: Combine the monitoring system with the optimal detection model to perform real-time classroom facial expression recognition and detection. The specific steps are as follows:
[0044] Step 51: Arrange high-definition surveillance cameras appropriately in the classroom environment to capture key information such as students' facial expressions in real time from multiple angles; Step 52: Transmit the video stream data collected in real time by the surveillance cameras to the computer processing terminal; Step 53: After receiving the video stream data, the computer terminal uses the optimal detection model obtained in Step 4 to extract and analyze features from each frame of the image. Once a student is detected to exhibit typical classroom facial expressions, the computer system will provide real-time feedback in a visual form on the computer display interface; Step 54: Adopt differentiated annotation methods based on different student behaviors to accurately identify and provide feedback on various expressions and behaviors.
[0045] Example:
[0046] This embodiment provides a classroom facial expression recognition method based on an improved RT-DETR model, as shown in Figure 1. The method includes the following steps:
[0047] Step 1: Obtain a dataset of facial expression images from the classroom.
[0048] We extracted seven basic facial expression classes from the publicly available FER-2013 facial expression dataset, containing 28,709 training samples and 7,178 test samples. Labels included anger, disgust, fear, happiness, sadness, surprise, and neutral. A subset of these expressions was selected for use.
[0049] Step 2: Preprocess the dataset using a computer. Divide the dataset into a validation set, a training set, and a test set, and finally process it into the RT-DETR network model recognition format.
[0050] In this step, the specific preprocessing steps are as follows: The classroom facial expression dataset is divided into a training set, a validation set, and a test set according to a set ratio of 8:1:1 using Python code. The training set will be used to train the model, the validation set will be used for evaluation during the training process, and the test set will be used to evaluate the performance of the model. The training set, validation set, and test set will be processed into the RT-DETR network model recognition format.
[0051] Step 3: Train the classroom facial expression recognition and detection model using the improved RT-DETR model on the dataset. As shown in Figure 2, the improved RT-DETR model replaces the AIFI module with the CAFI module in the backbone network, replaces the Conv convolution with the MCFEConv convolutional feature enhancement module in the head network and backbone network, and replaces the traditional RepC3 module with the RepC3_LWGA module in the head network.
[0052] This embodiment replaces the AIFI module with a CAFI module in the backbone network. Through a cascaded system of local location enhancement, spatial attention guidance, and self-attention global modeling, it achieves multi-scale joint extraction and saliency screening of complex facial expression features in classroom scenarios. This effectively improves the discriminativeness and robustness of feature representations under conditions of multiple students, complex backgrounds, and subtle facial expression changes, providing stable and reliable feature support for high-level semantic tasks such as classroom facial expression recognition, and ensuring the accuracy and reliability of recognition results from the feature level. As shown in Figure 3, the input feature map is first imported as a four-dimensional tensor into the local location enhancement submodule. This submodule consists of a cascade of depthwise separable convolutions and pointwise convolutions, applying [variables] to the spatial domain... Depthwise convolution models independent local receptive fields for each channel, and then... Convolution completes linear recombination between channels, explicitly enhancing the local texture structure and edge information of the input feature map. The feature map processed by this local enhancement unit is added and fused with the original input feature map in a residual manner to form an intermediate feature representation that combines the original global semantics with local fine-grained expression cues. Subsequently, the intermediate features are fed into a lightweight spatial attention gating submodule. By applying global average pooling and global max pooling operations to the channel dimensions respectively, two single-channel spatial response maps are generated. After concatenation in the channel dimensions, they are input into convolution and activation operators to calculate a normalized spatial attention weight map. This weight map weights the intermediate features position by position, explicitly suppressing the responses of classroom background areas such as blackboards, desks, and chairs, highlighting the responses of concentrated areas of student faces and their key expression regions, thereby obtaining a salient feature map after background interference suppression. Building upon this, the module further constructs a two-dimensional sine and cosine positional encoding that matches the spatial resolution of the feature map. A learnable scaling factor is introduced to adaptively adjust this absolute positional information. The positional encoding and the spatially attention-modulated feature sequence are then input into a Transformer encoding unit based on multi-head self-attention and a feedforward network, completing the modeling of global dependencies and the deep nonlinear mapping of high-dimensional features. The encoded sequence features are then reconstructed into a four-dimensional feature map consistent with the input through shape transformation, serving as the final output of the CAFI module.
[0053] In this embodiment, the MCFEConv convolutional feature enhancement module replaces the Conv convolution in the head network and backbone network. Through the joint design of multi-scale convolutional branch modeling, channel attention recalibration, and spatial attention-guided background suppression, it achieves efficient extraction and filtering of multi-scale faces and their subtle facial expression changes in classroom scenarios. This significantly improves the discriminativeness and robustness of feature representations, providing more discriminative and stable feature inputs for high-level semantic tasks such as classroom facial expression recognition. It ensures the accuracy and reliability of recognition results in complex teaching environments from the feature level. As shown in Figure 4, firstly, the input feature map of the future backbone network or the previous level feature fusion unit is used as input to construct a multi-branch convolutional structure, which is used to simultaneously characterize facial regions and their facial expression details at different scales in the classroom scenario. Specifically, the first branch employs a 1×1 convolution, reorganizing the channel dimension while maintaining spatial resolution to achieve fine-grained recoding of local texture information. The second branch uses a standard convolution, sliding within the feature map space and extracting expression features suitable for student face regions at a standard scale through multiplication and summation of the convolution kernel weights and corresponding elements. The third branch uses dilated convolution with a dilation rate to expand the effective receptive field without increasing additional parameter overhead, enhancing the modeling ability for large-scale contextual information such as distant students and the overall classroom layout. The multi-scale features output from the three branches are concatenated along the channel dimension and compressed back to the original channel dimension through a convolutional branch fusion layer to obtain an intermediate feature map with a fused multi-scale spatial receptive field. Subsequently, to further highlight the channel-dimensional semantic information related to facial expression discrimination, this embodiment introduces a channel attention mechanism on the fused feature map. By applying global average pooling to the feature map, a global channel description vector is obtained. This vector is then processed through two layers of pointwise convolution and a non-linear activation function for compression and expansion mapping, generating a normalized channel weight vector. This weight vector is then applied to the feature map via channel-wise multiplication, resulting in a recalibrated feature map. This suppresses redundant channel responses unrelated to facial expressions and enhances the expressive power of key regions such as eyebrows, eyes, and mouth. Building upon this, this embodiment further introduces a spatial attention mechanism to suppress interference from complex classroom backgrounds. By applying average pooling and max pooling operations to the channel dimensions, two single-channel response maps are obtained. These maps are then concatenated along the channel dimension and input into a convolutional layer with a large receptive field and a sigmoid activation function to generate a spatial attention map. This map is then multiplied with the input feature map via element-wise multiplication to obtain a spatially saliency-enhanced feature map. This highlights high-response regions concentrated on student faces while effectively suppressing interfering responses from background areas such as blackboards, walls, and desks. Finally, the features passed through lightweight convolution and normalization layers are linearly transformed and fused with the input feature map using residual mapping to obtain the output features of the MCFEConv convolutional feature enhancement module.
[0054] In this embodiment, a RepC3_LWGA module is designed in the head network to replace the traditional RepC3 module. Through the modular design of RepC3 and the lightweight global attention sequence of LWGA, the network's ability to represent multi-scale facial targets and subtle facial expression changes in classroom scenarios is effectively improved while maintaining controllable overall parameter count and computational complexity. This enhances the discriminativeness and robustness of feature representations, providing more stable and sufficient feature inputs for subsequent detection and expression recognition subtasks. As shown in Figure 5, firstly, the input feature map of the future backbone network or the previous stage feature fusion unit is used as the module input. The input features are divided into two parallel branches under the action of the feature splitting unit in the RepC3 structure. In the backbone branch, the number of channels is first compressed through convolution operations, mapping the input channels to lower-dimensional hidden channels. This reduces the number of parameters and computational cost while providing low-rank feature representations for subsequent lightweight attention modeling. Subsequently, the compressed backbone features are sequentially input into a lightweight global attention sub-module sequence composed of several LWGA Blocks connected in series. In this sequence, each LWGA Block adaptively weights features at different scales and semantic levels by combining the parameters of the current network stage. It continuously strengthens key responses related to classroom faces and their expressions by simultaneously performing long-range dependency modeling and feature recalibration in both spatial and channel dimensions, while suppressing background information irrelevant to the target, thus obtaining attention-enhanced backbone features. In parallel, the shortcut branch applies convolution operations to the input feature map to preserve the original structural information and local details, providing a stable reference baseline for subsequent feature fusion. Subsequently, the features enhanced by the LWGA Block sequence in the backbone branch are concatenated with the features output from the shortcut branch in the channel dimension, forming an intermediate feature tensor that simultaneously contains multi-scale attention information and original structural information. To generate the final output, convolution operations are used to compress the channels and linearly reassemble the concatenated feature map, mapping its channel count to a preset output channel count, thus obtaining the output features of the RepC3_LWGA module.
[0055] While maintaining a low number of parameters and minimal computational overhead, the LWGA Block achieves explicit modeling of long-range dependencies within feature maps, effectively highlighting key regions related to faces and expressions while suppressing interference from irrelevant background regions. In complex classroom scenarios, this module demonstrates stronger stability and discriminative power when handling multiple students, multi-scale faces, and subtle expression changes, helping to maintain high recognition accuracy and robustness under conditions of occlusion, lighting variations, and complex backgrounds. As shown in Figure 6, the specific implementation process is as follows: The input feature map is first linearly mapped using three sets of convolution operators with consistent spatial dimensions, generating query features, key features, and value features to characterize the correlation between features. This process does not change the spatial resolution of the feature map; through reorganization and compression along the channel direction, an adapted feature representation is constructed for subsequent attention calculations. Subsequently, to reduce the computational complexity of global attention, downsampling or pooling operations are performed on the key and value features in the spatial dimension, generating compressed key features and compressed horizontal features with lower resolution. While ensuring that key semantic information is not compromised, the number of spatial locations involved in attention calculations is significantly reduced, thereby significantly reducing the computational load. Building upon this, the query features, compressed key features, and compressed value features are sequentially flattened along the spatial dimension into a sequence. A lightweight global attention weight is constructed using the matching relationship between the query sequence and the key sequence. This attention weight is then used to weight and fuse the information in the value sequence, resulting in an intermediate response feature containing global contextual semantics. This intermediate response is restored to the same spatial size as the input features, yielding an enhanced response in the form of an output feature map. An incremental representation to supplement the input features is generated using an output mapping unit composed of convolution and a non-linear activation function. Finally, this incremental response is added to the original input feature map using a residual method to obtain the output feature of the LWGA Block.
[0056] Step 5: Combine monitoring with computer algorithms to perform real-time recognition and detection of students' facial expressions in the classroom.
[0057] High-definition surveillance cameras are strategically deployed in the classroom environment to provide full coverage of the student learning area without blind spots, simultaneously capturing key visual information such as student behavior, posture changes, and facial expressions from multiple angles. The camera's hardware parameters, including resolution, frame rate, and dynamic range, are pre-configured and optimized based on the classroom space scale, lighting conditions, and the detail capture requirements of the facial expression recognition task. This ensures sufficient clarity, stability, and continuity of the acquired video data, providing a reliable input source for subsequent accurate detection. The video stream data acquired by the cameras is transmitted in real-time to a pre-set computer processing terminal via wired or wireless transmission. During transmission, data bandwidth, latency, and stability must meet the real-time detection requirements. Upon receiving the video stream, the computer processing terminal immediately invokes the pre-trained optimal classroom facial expression detection model weights based on improved RT-DETR to perform rapid feature extraction, semantic analysis, and facial expression recognition processing on each input frame. When the model detects specific classroom facial expression features, the system will display the detection results in real-time on the terminal interface, providing immediate feedback and dynamic monitoring of classroom status information.
Claims
1. A classroom facial expression recognition method based on an improved RT-DETR model, characterized in that... The method includes the following steps: Step 1: Obtain a dataset of classroom facial expression images; Step 2: Preprocess the dataset using a computer; Step 3: Train the dataset using a classroom facial expression recognition and detection model based on an improved RT-DETR model. The improved RT-DETR model replaces the AIFI module with a CAFI module in the backbone network, replaces the Conv convolution with an MCFEConv convolutional feature enhancement module in the head network and backbone network, and replaces the traditional RepC3 module with a RepC3_LWGA module in the head network; Step 4: Train the classroom facial expression recognition and detection model using the training and validation sets. After obtaining the optimal detection model, evaluate its performance using a test set; Step 5: Combine monitoring with the optimal detection model to perform real-time recognition and detection of students' classroom facial expressions.
2. The classroom facial expression recognition method based on the improved RT-DETR model according to claim 1, characterized in that... The forward propagation mathematical expression for the CAFI module is as follows: in, Indicates the input feature map, This represents the intermediate feature map after local enhancement. This indicates a local location enhancement mapping. Represents the spatial attention weight map. This represents the Sigmoid activation function. This represents the convolution operator that generates the spatial attention map. This represents the global average pooling operator along the channel dimension. This represents the global max-pooling operator along the channel dimension. This indicates a channel dimension splicing operation. This represents element-wise multiplication. Represents a salient feature map. This indicates a characteristic shape transformation operation. Represents a 3D feature sequence. This represents adaptive two-dimensional sine and cosine position coding. This represents the learnable positional encoding scaling factor. This represents a two-dimensional sine and cosine position coding function. This represents the feature sequence after Transformer encoding. This represents the Transformer encoding operator. This indicates the inverse shape transformation operation. This represents the final output feature map of the CAFI module.
3. The classroom facial expression recognition method based on the improved RT-DETR model according to claim 1, characterized in that... The forward propagation mathematical expression of the MCFEConv convolutional feature enhancement module is as follows: in, Indicates the input feature map, Indicates the first one convolution, This represents the first normalization operator. ， This represents the output feature map of the first branch. This represents the output feature map of the second branch. Indicates the second one convolution, express convolution, This represents the third normalization operator. This represents the output feature map of the third branch. This indicates a feature concatenation operation along the channel dimension. Indicates the 4th convolution, This represents the 4th normalization operator. This represents the feature map after multi-branch splicing and fusion. Indicates global average pooling. Indicates a fully connected layer. This represents the Sigmoid activation function; This represents the channel attention weight vector, and ⊙ represents element-wise multiplication. This represents the feature map after channel recalibration. This represents average pooling along the channel dimension. This represents max pooling along the channel dimension. express convolution, Represents the spatial attention weight map. This represents the feature map after spatial attention weighting. Indicates the 5th convolution, This represents the 5th normalization operator. This represents the final output feature map.
4. The classroom facial expression recognition method based on the improved RT-DETR model according to claim 1, characterized in that... The forward propagation mathematical expression for the RepC3_LWGA module is as follows: in, This represents the input feature map of the RepC3_LWGA module. Indicates the action on the main branches. Convolution is used to convert the number of channels from... Compressed to the number of hidden channels ， This represents the normalization operator on the main branch. Represents a non-linear activation function. This represents the intermediate features of the main branches after the initial mapping. Indicates the number of LWGABlock modules. This represents the lightweight global attention mapping operator corresponding to the k-th LWGA Block module. `stage` is a control parameter indicating the current network layer or scale information, and `k` represents the LWGA Block module index, with a value ranging from... ， This represents the intermediate features of the main branch after the k-th LWGA Block module has been activated. This represents the final enhanced feature of the backbone branch after stacking n LWGA Block modules. Indicates the action applied to shortcut branches. convolution, This represents the normalization operator on the shortcut branch. Indicates the characteristics of shortcut branches. This represents a feature concatenation operation along the channel dimension. This represents the intermediate features after merging. Indicating the fusion stage Convolution is used to convert the number of channels from... Mapped to the number of output channels ， The normalization operator represents the normalization operator in the fusion phase. This indicates the number of output channels of the RepC3_LWGA module. This represents the output feature map of the RepC3_LWGA module.
5. The classroom facial expression recognition method based on the improved RT-DETR model according to claim 4, characterized in that... The forward propagation mathematical expression for the LWGA Block module is as follows: in, Indicates the input feature map, Indicates the generation of the query convolution, Represents the Query feature map. Indicates the generation of the key convolution, Represents the key feature map. Indicates the generation of Value convolution, Represents the Value feature map. This indicates a pooling operation. This represents the key feature map after pooling. This represents the value feature map after pooling. This indicates a characteristic shape transformation operation. express The reshaped sequence form, express The reshaped sequence form, express The reshaped sequence form, This represents the Softmax activation function. Represents the attention weight matrix. Indicates the number of channels The square root, This represents the sequence features after attention weighting. This indicates the inverse shape transformation operation. express The reshaped feature map format, Indicates the output terminal convolution, Represents a non-linear activation function. This represents the residual features after feature enhancement. This represents the final output feature map.
6. The classroom facial expression recognition method based on the improved RT-DETR model according to claim 1, characterized in that... The specific steps of step four are as follows: Step four one: Input the training set and validation set into the classroom face expression recognition and detection model of the improved RT-DETR model, and set the number of training times. As the number of training times increases, the loss function curve of the detection model gradually converges. When the loss function curve converges and stabilizes, the detection model is trained to the optimal state, and its optimal model weight file is saved; Step four two: Input the images to be detected in the test set into the trained optimal behavior recognition and detection model, and output the detected images, where the detected images include the type of each detected target, and the position of each target in the target detection image is marked.
7. The classroom facial expression recognition method based on the improved RT-DETR model according to claim 1, characterized in that... The specific steps of step five are as follows: Step 51: Arrange high-definition surveillance cameras reasonably in the classroom environment to capture key information such as students' facial expressions in real time from multiple angles; Step 52: Transmit the video stream data collected in real time by the surveillance cameras to the computer processing terminal; Step 53: After receiving the video stream data, the computer terminal uses the optimal detection model obtained in step four to extract and analyze features of each frame of the image. Once a student is detected to have a typical classroom facial expression, the computer system will provide real-time feedback in a visual form on the computer display interface; Step 54: Adopt differentiated annotation methods according to different student behaviors in order to accurately identify and provide feedback on various expressions and behaviors.