A method for detecting students' classroom behavior based on gaze target estimation

Abstract

The present invention belongs to the field of image processing and computer vision technology, and specifically relates to a student classroom behavior detection method based on gaze target estimation. The method comprises constructing and training a student classroom behavior detection model, obtaining real-time classroom video captured by a surveillance camera and inputting the trained student classroom behavior detection model into the model to obtain behavior detection results of all students in the classroom. The student classroom behavior detection model comprises a spatiotemporal feature pyramid network, a dynamic instance interaction head, a gaze target estimation module, a ROI mapping module, and a fusion module. The present invention is an end-to-end single-stage student classroom behavior detection method that can effectively improve the real-time performance of prediction. The method uses a dynamic instance interaction head and adopts a one-to-one label assignment method to realize crowded target detection in teaching scenarios and effectively reduce repeated predictions.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and in particular relates to a method for detecting student classroom behavior based on gaze target estimation. Background Art

[0002] With the gradual maturity of information technology, storage technology, and communication technology, artificial intelligence has further developed, and deep learning technology has also received increasing attention. Industries that have boomed with artificial intelligence, such as driverless cars and smart home systems, are constantly changing people's lifestyles and production methods. Machines are replacing humans, liberating productivity, and are widely used in various industries. Smart education has also become a focus of attention, with the goal of improving teaching effectiveness. Student posture information in the classroom is a key evaluation criterion for teaching quality. Traditional teaching quality assessment mainly relies on supervisory inspections and subjective scoring, which consumes a lot of manpower and material resources and is prone to errors and omissions. With the widespread use of video surveillance systems in schools, the demand for automated analysis technology for campus surveillance is increasing. Now, we can use video surveillance systems distributed in each classroom and deep learning technology to automatically and intelligently analyze the teaching situation in each classroom. Therefore, how to combine deep learning and computer vision technologies with video surveillance systems to perform real-time and reliable analysis, thereby reducing errors and improving work efficiency, is of great research significance.

[0003] Behavior detection is a key research area in intelligent object perception. It aims to leverage computer vision to process images or videos captured by an imaging processor and automatically extract information about human behavior. However, in classrooms, the crowded and heavily obscured environments make accurate detection difficult. Furthermore, student behaviors, such as playing with their phones or reading, are difficult to directly identify. Summary of the Invention

[0004] To address the above issues, the present invention provides a method for detecting student classroom behavior based on gaze target estimation, which includes constructing and training a student classroom behavior detection model, obtaining real-time classroom video captured by a surveillance camera and inputting the trained student classroom behavior detection model into the model to obtain behavior detection results for all students in the classroom; the student classroom behavior detection model includes a spatiotemporal feature pyramid network, a dynamic instance interaction head, a gaze target estimation module, a ROI mapping module, and a fusion module;

[0005] The training process of the student classroom behavior detection model includes the following steps:

[0006] S1. Obtain classroom video captured by a surveillance camera and input it into a spatiotemporal feature pyramid network to obtain video features. S2. Extract a keyframe image from the classroom video and extract the keyframe features of the keyframe image from the video features. Input the keyframe features into the dynamic instance interaction head and output the predicted box coordinates corresponding to each target in the keyframe image.

[0007] S3. Input the key frame features and all predicted box coordinates into the gaze target estimation module, and output the gaze target area corresponding to each target;

[0008] S4. The predicted frame coordinates of each target and the gaze target area are mapped by the ROI module to ROIAlign, and the posture information and gaze area information of each target are obtained;

[0009] S5. Fusing the video features with the posture information and gaze area information of each target to obtain the fused features corresponding to each target;

[0010] S6. Use the behavior detection network to process the fusion features corresponding to each target and obtain the behavior detection results of each target.

[0011] Furthermore, the process of obtaining key frame features in step S2 includes:

[0012] S11. The spatiotemporal feature pyramid network adopts the 3DResNet50-FPN architecture, which mainly includes a bottom-up path and a top-down path; wherein the bottom-up path includes the res2 convolution module, the res3 convolution module, the res4 convolution module, and the res5 convolution module;

[0013] S12. Extract multi-scale features from each frame of the classroom video using a spatiotemporal feature pyramid network, and combine the multi-scale features of all images into video features. The multi-scale features of each frame are composed of the output features of the res3 convolution module, the output features of the res4 convolution module, and the output features of the res5 convolution module obtained in the spatiotemporal feature pyramid network.

[0014] S13. Extract a key frame image from the classroom video, and extract the multi-scale features of the key frame image from the video features as the key frame features.

[0015] Furthermore, in the bottom-up path, all simple one-dimensional convolutions are replaced by graph convolution units; the graph convolution units include graph attention convolution and maximum pooling operations with a stride of 2.

[0016] Furthermore, the process of using the dynamic instance interaction head to perform detection in step S2 includes:

[0017] S21. The dynamic instance interaction head receives the keyframe features extracted by the spatiotemporal feature pyramid network and predicts the location and category of each object in the keyframe image.

[0018] S22. Use a one-to-one label assignment method to reduce duplicate predictions.

[0019] Furthermore, the one-to-one label allocation method includes:

[0020] S221. Set a first confidence threshold and compare it with the confidence of all predicted boxes;

[0021] S222. If the prediction box B i , i=1,2,…,M’s confidence is greater than the first confidence threshold, then the prediction box B i is an acceptable prediction; if the prediction box B i The confidence level of is not greater than the first confidence threshold, then the prediction box B i is the noise prediction; where M is the number of prediction boxes;

[0022] S223. Repeat step S222 until all prediction boxes are classified;

[0023] S224. Obtain the ground truth boxes of all targets to form a target ground truth set. Match any acceptable prediction with the target ground truth set. If the center point of the acceptable prediction falls within a ground truth box, the acceptable prediction is successfully matched with the ground truth box, and the matched ground truth box is removed from the target ground truth set. Repeat step S224 until all acceptable predictions are matched.

[0024] S225. Obtain the updated target true set after step S224, and match any noise prediction with the updated target true set.

[0025] Furthermore, the gaze target estimation module includes a target branch and a gaze prediction branch; the target branch includes ROI Align, target position encoding and a fully connected layer; the gaze prediction branch includes an encoding module and an MLP module.

[0026] Furthermore, the gaze target estimation module outputs the gaze target area corresponding to each target, including:

[0027] S31. Based on the key frame features, ROI Align is performed on any target to obtain the target features of the target, and the target position is encoded on the predicted box coordinates of the target to obtain a target position feature map;

[0028] S32. Connect the target feature and the target position feature map and pass them through a fully connected layer to calculate the attention map;

[0029] S33. Concatenate the key frame features, target features, and target position feature maps, and multiply the concatenated result with the attention map to obtain a scene feature map;

[0030] S34. The concatenation result of the scene feature map and the target feature is input into an encoding module for encoding, wherein the encoding module includes two cascaded convolutional layers;

[0031] S35. Input the feature map output by the encoding module into the MLP module for decoding operation, wherein the MLP module includes two MLP layers, one MLP layer is used to output the predicted probability of the gaze target in the scene, and the other MLP layer is used to output the predicted gaze heat map.

[0032] Beneficial effects of the present invention:

[0033] The present invention is an end-to-end single-stage student classroom behavior detection method. It only uses a spatiotemporal feature pyramid network to extract spatiotemporal features of videos and extract key frame features, which can effectively improve the real-time performance of predictions. It uses a dynamic instance interaction head and a one-to-one label assignment method to achieve crowded target detection in teaching scenarios and effectively reduce repeated predictions. It uses target gaze prediction to detect the target's gaze area, which can effectively guide the network to distinguish similar behaviors. It also uses spatiotemporal video features to accurately distinguish the start time of the behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is the logical framework diagram of the present invention;

[0035] Figure 2 Schematic diagram of the spatiotemporal feature pyramid in the present invention;

[0036] Figure 3 Schematic diagram of the dynamic interaction head in the present invention;

[0037] Figure 4 Schematic diagram of the gaze target estimation module in the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] Most existing behavior detection methods are two-stage methods, that is, they need to perform target detection in the first stage first, and then perform behavior detection in the second stage, which requires huge computational complexity. In order to effectively reduce the computational complexity and achieve real-time behavior detection, the present invention provides a student classroom behavior detection method based on gaze target estimation, such as Figure 1 As shown, the following steps are included:

[0040] S1. Obtain classroom videos captured by surveillance cameras and input them into the spatiotemporal feature pyramid network to obtain video features.

[0041] Specifically, the spatiotemporal feature pyramid network adopts the ResNet50-FPN architecture, such as Figure 2 As shown in the figure, it mainly includes bottom-up path, top-down path and lateral connections; the bottom-up path includes res2 convolution module, res3 convolution module, res4 convolution module and res5 convolution module.

[0042] Specifically, in a traditional pyramid structure, the bottom-up path is essentially a feedforward calculation of a downsampling convolutional neural network, where all convolution operations used are simple one-dimensional convolutions. However, in order to effectively obtain the temporal information between multiple frames in a video, the present invention replaces all simple one-dimensional convolutions in the bottom-up path with graph convolution units; these graph convolution units include a graph attention network (GAT) and a maximum pooling operation with a stride of 2.

[0043] Specifically, in the traditional pyramid structure, the top-down path is essentially to increase the resolution of feature maps with high-level semantic information; the feature map with a large receptive field at the top is upsampled, with a stride of 2, the same as the maximum pooling operation, and linear interpolation is used during upsampling. After upsampling, it is horizontally connected with the feature map of the same size output from the bottom-up path. The corresponding elements are added together during fusion. The specific formula can be expressed as:

[0044] F 1ow =interpolate(conv(F cur ))

[0045] Among them, F 1ow represents the upsampled features, F cur Represents the input features of the current layer, interpolate() represents the upsampling operation, and conv() is a 3×1×1 convolution, which is used to extract the timing information of the video.

[0046] S2. Extract a key frame image from the classroom video, extract the key frame features of the key frame image from the video features, input them into the dynamic instance interaction head, and output the predicted box coordinates corresponding to each target in the key frame image.

[0047] Specifically, for each frame in the classroom video, the output features of the res3 convolution module, the output features of the res4 convolution module, and the output features of the res5 convolution module obtained by the spatiotemporal feature pyramid network are extracted to form multi-scale features. The multi-scale features of all images are combined into video features. A key frame image is extracted from the classroom video, and the multi-scale features of this key frame image are extracted from the video features as key frame features.

[0048] Specifically, if Figure 2 As shown, the multi-scale features of a frame image include: the output feature of the res5 convolution module is the feature directly output by the res5 convolution module, the output feature of the res4 convolution module is the first splicing feature obtained by splicing the feature directly output by the res5 convolution module and the feature directly output by the res4 convolution module, and the output feature of the res3 convolution module is the second splicing feature obtained by splicing the feature directly output by the res3 convolution module and the first splicing feature.

[0049] Specifically, the input of the dynamic instance interaction head includes three parts: one is the key frame feature, the second is the learnable proposal box, and the third is the learnable proposal feature; the learnable proposal box is represented by a 4-D parameter from 0 to 1, indicating the standardized center coordinates, height and width, and storing the statistical ROI information of the entire dataset. N ROI areas are extracted from the image through the learnable proposal box to provide a rough ROI representation, and ROI Align is used to extract ROI features for each learnable proposal box; at the same time, the corresponding learnable proposal features are extracted to provide detailed information (such as appearance and posture).

[0050] Specifically, each ROI feature corresponds to a separate dynamic instance interaction head, and each dynamic instance interaction head will perform a one-to-one interaction between the ROI feature and its corresponding learnable proposal feature to achieve target positioning and classification, such as Figure 3 As shown, specifically including:

[0051] The shapes and sizes of the learnable proposal features and ROI features are (N, C) and (N, S×S, C), respectively. N=100, C=256, and S is the size of the RoI feature map. The learnable proposal features are linearly mapped to generate vectors, which are used as dynamic parameters and interact with the ROI features. This interaction takes the form of two 1×1 convolutional layers, where the generated dynamic parameters serve as the parameters of the two 1×1 convolutional layers, performing convolution operations on the ROI features. Due to the characteristics of 1×1 convolutions, insignificant channels in the ROI features are filtered out, highlighting the effective locations. Finally, the output of the convolutional network is linearly mapped to obtain the target features, which are used for subsequent target positioning and category prediction.

[0052] Specifically, the process of using the dynamic instance interaction head to perform detection in step S2 includes:

[0053] S21. The dynamic instance interaction head receives the keyframe features extracted by the spatiotemporal feature pyramid network and predicts the location and category of each object in the keyframe image.

[0054] S22. Use a one-to-one label assignment method to reduce duplicate predictions.

[0055] Specifically, when using a dynamic instance interaction head for target recognition, high-confidence prediction frames generally contain a large proportion of target objects, with only a small number of repeated predictions; while low-confidence prediction frames contain a large number of repeated predictions. Therefore, the present invention adopts a one-to-one label assignment method to reduce repeated predictions, including:

[0056] S221. Set a first confidence threshold and compare it with the confidence of all predicted boxes; in this embodiment, the first confidence threshold is 0.7;

[0057] S222. If the prediction box B i , i=1,2,…,M’s confidence is greater than the first confidence threshold, then the prediction box B i is an acceptable prediction; if the prediction box B i The confidence level of is not greater than the first confidence threshold, then the prediction box B i is the noise prediction; where M is the number of prediction boxes;

[0058] S223. Repeat step S222 until all prediction boxes are classified;

[0059] S224. Obtain the ground truth boxes of all targets to form a target ground truth set. Match any acceptable prediction with the target ground truth set. If the center point of the acceptable prediction falls within a ground truth box, the acceptable prediction is successfully matched with the ground truth box, and the matched ground truth box is removed from the target ground truth set. Repeat step S224 until all acceptable predictions are matched.

[0060] S225. Obtain the updated target true set after step S224, match any noise prediction with the updated target true set, and if the center point of the noise prediction falls within a certain true box, the noise prediction is successfully matched with the true box, and the matched true box is removed from the target true set; repeat step S225 until all acceptable predictions are matched or all true boxes are matched successfully.

[0061] Specifically, when the center point of an acceptable prediction falls in multiple real boxes, the IOU between the acceptable prediction and multiple real boxes are calculated respectively, and the real box with the largest IOU is selected.

[0062] S3. Input the key frame features and all predicted box coordinates into the gaze target estimation module, and output the gaze target area corresponding to each target.

[0063] Specifically, the gaze target estimation module includes a target branch and a gaze prediction branch.

[0064] Target branch: such as Figure 4 As shown in the figure, each target in the keyframe features is processed separately, and the target features of each target are obtained using ROI Align. The target position of each target's predicted box coordinates is encoded to obtain the target position feature map of each target. For each target, its target features are concatenated with its target position feature map, and then these two concatenated features are passed through a fully connected layer to calculate an attention map.

[0065] Specifically, binary images are more effective at encoding the location and relative depth of objects in a scene than position encoder-based approaches. The object position encoding described in the present invention involves converting a keyframe image into a binary image, where black pixels specify the bounding box of the object and white pixels represent the rest of the keyframe image. This image is then downsized using three consecutive max pooling operations to obtain a feature map of the object's position.

[0066] Look at the predicted branch: Figure 4 As shown in the figure, the keyframe features, target features, and target position feature map are concatenated and then multiplied with the attention map obtained by the target branch to output a scene feature map. Multiplying the scene feature map with the attention map obtained by the target enables the model to learn more scene features that are more likely to be attended to based on the target's attributes. The scene feature map is then concatenated with the target feature map. Finally, two convolutional layers are used in the encoding module to encode the concatenated features. The encoded features are then decoded by two MLP layers: one predicts the probability of the gazed target being within the scene, and the other predicts the gaze heatmap.

[0067] Loss Function: The gaze target estimation task requires high-quality gaze heatmaps to produce accurate results. This is done using the energy aggregation loss to guide the gaze target estimation process using the true target gaze region.

[0068] The energy of the target's gaze area is E b :

[0069]

[0070] Where W i,j Represents the energy of each pixel (i, j) in the predicted gaze heat map, x1 is the minimum horizontal coordinate of the real gaze target box, x2 is the maximum horizontal coordinate of the real gaze target box, y1 is the minimum vertical coordinate of the real gaze target box, y2 is the maximum vertical coordinate of the real gaze target box, and M is the number of all pixels in the real gaze target box.

[0071] The energy accumulation loss is as follows:

[0072]

[0073] Among them E b is the energy of the target gaze area, E I It is the total energy of this frame of image.

[0074] The total loss is the sum of energy collection loss and gaze loss:

[0075]

[0076] is the gaze loss, specifically the MSE loss between the predicted gaze heatmap and the true gaze heatmap; Energy accumulation loss.

[0077] S4. The predicted frame coordinates and gaze target area of ​​each target are mapped by ROIAlign module respectively to obtain the posture information and gaze area information of each target.

[0078] Specifically, the size of the feature map mapped to the predicted box coordinates and the gaze target position is calculated, and floating-point numbers are retained. This mainly includes: setting the feature map size after ROIAlign to 10×10, dividing the area in the feature map where the predicted box coordinates and gaze target position of a certain target are mapped into 100 sub-regions of equal size, selecting the maximum value in each sub-region as the pixel value of that sub-region, and thus obtaining 100 pixel values ​​to form a 10×10 feature map. Finally, the pose information and gaze area information of the target are obtained.

[0079] S5. Fusing the multi-scale features with the posture information and gaze area information of each target to obtain the fused features corresponding to each target;

[0080] S6. Use the behavior detection network to process the fusion features corresponding to each target and obtain the behavior detection results of each target.

[0081] The present invention hopes to realize an end-to-end student classroom behavior detection network at the lowest cost, so a simple linear layer is used to process all fused features, and cross entropy loss is used to train the behavior classification head.

[0082] The loss function is as follows:

[0083]

[0084] y is the true label value (positive value is 1, negative value is 0), is the predicted probability value ). It represents the difference between the true sample label and the predicted probability.

[0085] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting student classroom behavior based on gaze target estimation, characterized in that: Construct and train a student classroom behavior detection model. Real-time classroom videos captured by surveillance cameras are input into the trained student classroom behavior detection model to obtain behavior detection results for all students in the classroom. The student classroom behavior detection model includes a spatiotemporal feature pyramid network, a dynamic instance interaction head, a gaze target estimation module, a ROI mapping module, and a fusion module. The training process of the student classroom behavior detection model includes the following steps: S1. Obtain classroom videos captured by surveillance cameras and input them into the spatiotemporal feature pyramid network to obtain video features; S2. Extract a keyframe image from the classroom video, extract the keyframe features of the keyframe image from the video features, input them into the dynamic instance interaction head, and output the predicted box coordinates corresponding to each object in the keyframe image; S3. Input the key frame features and all predicted box coordinates into the gaze target estimation module, and output the gaze target area corresponding to each target; The gaze target estimation module includes a target branch and a gaze prediction branch; the target branch includes ROIAlign, target position encoding and a fully connected layer; the gaze prediction branch includes an encoding module and an MLP module; The gaze target estimation module outputs the gaze target area corresponding to each target, including: S31. Based on the key frame features, ROIAlign is performed on any target to obtain the target features of the target, and the target position is encoded on the predicted box coordinates of the target to obtain a target position feature map; S32. Connect the target feature and the target position feature map and pass them through a fully connected layer to calculate the attention map; S33. Concatenate the key frame features, target features, and target position feature maps, and multiply the concatenated result with the attention map to obtain a scene feature map; S34. The concatenation result of the scene feature map and the target feature is input into an encoding module for encoding, wherein the encoding module includes two cascaded convolutional layers; S35. The feature map output by the encoding module is input to the MLP module for decoding operation, wherein the MLP module includes two MLP layers, one MLP layer is used to output the probability of the predicted gaze target in the scene, and the other MLP layer is used to output the predicted gaze heat map; S4. The predicted frame coordinates of each target and the gaze target area are mapped by the ROI module to ROIAlign, and the posture information and gaze area information of each target are obtained; S5. Fusing the video features with the posture information and gaze area information of each target to obtain the fused features corresponding to each target; S6. Use the behavior detection network to process the fusion features corresponding to each target and obtain the behavior detection results of each target.

2. The method for detecting student classroom behavior based on gaze target estimation according to claim 1, characterized in that: The process of obtaining key frame features in step S2 includes: S11. The spatiotemporal feature pyramid network adopts the 3DResNet50-FPN architecture, which mainly includes a bottom-up path and a top-down path; wherein the bottom-up path includes the res2 convolution module, the res3 convolution module, the res4 convolution module, and the res5 convolution module; S12. Extract multi-scale features from each frame of the classroom video using a spatiotemporal feature pyramid network, and combine the multi-scale features of all images into video features; wherein the multi-scale features of each frame are composed of the output features of the res3 convolution module, the output features of the res4 convolution module, and the output features of the res5 convolution module obtained in the spatiotemporal feature pyramid network; S13. Extract a key frame image from the classroom video, and extract the multi-scale features of the key frame image from the video features as the key frame features.

3. The method for detecting student classroom behavior based on gaze target estimation according to claim 2, characterized in that: In the bottom-up path, all simple one-dimensional convolutions are replaced by graph convolution units; the graph convolution units include graph attention convolution and maximum pooling operations with a stride of 2.

4. The method for detecting student classroom behavior based on gaze target estimation according to claim 1, characterized in that: The process of using the dynamic instance interaction head to perform detection in step S2 includes: S21. The dynamic instance interaction head receives the keyframe features extracted by the spatiotemporal feature pyramid network and predicts the location and category of each object in the keyframe image. S22. Use a one-to-one label assignment method to reduce duplicate predictions.

5. The method for detecting student classroom behavior based on gaze target estimation according to claim 4, characterized in that: One-to-one label assignment methods include: S221. Set a first confidence threshold and compare it with the confidence of all predicted boxes; S222. If the prediction box B i , i=1,2,…,M’s confidence is greater than the first confidence threshold, then the prediction box B i is an acceptable prediction; if the prediction box B i The confidence level of is not greater than the first confidence threshold, then the prediction box B i is the noise prediction; where M is the number of prediction boxes; S223. Repeat step S222 until all prediction boxes are classified; S224. Obtain the ground truth boxes of all targets to form a target ground truth set. Match any acceptable prediction with the target ground truth set. If the center point of the acceptable prediction falls within a ground truth box, the acceptable prediction is successfully matched with the ground truth box, and the matched ground truth box is removed from the target ground truth set. Repeat step S224 until all acceptable predictions are matched. S225. Obtain the updated target true set after step S224, and match any noise prediction with the updated target true set.

Images