A classroom behavior detection method based on spatiotemporal characteristics

Abstract

The present invention belongs to the field of image processing and computer vision technology, and specifically relates to a classroom behavior detection method based on spatiotemporal features, comprising: selecting three key frames at intervals of K from a classroom video, gray-scaling them, and then splicing them according to the RGB three channels to form a three-channel spatiotemporal image containing motion information; using DarkNet‑19 as a feature extractor to obtain the features of the three-channel spatiotemporal image, adding a fully connected layer to output a preliminary proposed region; obtaining an STMap from the three-channel spatiotemporal image of the preliminary proposed region by stacking longitudinal scan lines frame by frame; initializing the STMap, and obtaining a motion information fluctuation feature map through the spatiotemporal feature extractor; using the target detection network YOLOv5 as the basic network, inputting the motion information fluctuation feature map into the network for detection, and performing post-processing to obtain the detection result. The present invention effectively reduces the computational complexity of the network and improves the accuracy of fine-grained student behavior detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and relates to a classroom behavior detection method based on spatiotemporal features. Background Art

[0002] In recent years, with the continuous development of computer vision and artificial intelligence technologies, my country has steadily promoted the construction of smart campuses, gradually building campuses characterized by smart teaching, smart management, and smart living. The student classroom is the most critical link in building a smart campus. The quality of the student classroom is determined by many factors, including instructional design, classroom practice, and teaching evaluation. Teaching evaluation provides feedback on instructional design and practice.

[0003] In traditional teaching evaluations, teachers typically assess the teacher's performance and student engagement from the back row. However, due to limited field of view, it's difficult to observe students' specific behavior, making this approach incomplete and subjective. With the development of smart campuses, most campuses have cameras installed. We can use advanced computer vision technology to automatically identify student behavior in the classroom, assisting in teaching evaluations.

[0004] Classroom behavior detection uses computer vision, machine learning, and other technologies to automatically detect and identify student behavior in the classroom. Traditional video behavior detection algorithms often utilize suggested regions or keyframes. While these methods reduce algorithm complexity, they often struggle with classroom videos that exhibit scale variations and occlusions.

[0005] Existing mainstream behavior detection methods are mainly divided into two-stream networks and networks based on three-dimensional convolution, and have achieved significant progress in recent years. However, the optical flow in two-stream networks only contains short-term temporal information, which is not ideal for modeling long-term videos and cannot effectively distinguish between student behavior categories with small inter-class differences. Although three-dimensional convolution has achieved significant improvements in extracting temporal information, it consumes a lot of computing resources, has a large model size, and has a slow detection speed, making it difficult to apply to classroom scenarios. Reducing the network size to enable behavior detection in teaching scenarios is of great research significance. Summary of the Invention

[0006] To solve the above technical problems, the present invention proposes a classroom behavior detection method based on spatiotemporal features, comprising the following steps:

[0007] S1. Select three key frames from the classroom video to be used for behavior detection, convert the key frames into grayscale, and then splice them according to the RGB three channels to obtain a three-channel spatiotemporal image containing motion information;

[0008] S2. Use the DarkNet-19 network as a feature extractor. Through multiple convolution and pooling operations of the DarkNet-19 network, features of different scales of the three-channel spatiotemporal image are extracted and irrelevant information is eliminated. Finally, the extracted features are compressed into a one-dimensional vector and passed to the fully connected layer. The preliminary proposed region is obtained through the softmax function.

[0009] S3, stacking the longitudinal scan lines frame by frame for the preliminary proposed region to form a two-dimensional matrix to obtain a spatiotemporal map STMap;

[0010] S4, initialize the spatiotemporal map STMap, decompose it using the improved DMD algorithm, and input the decomposition result into the input spatiotemporal feature extractor to obtain the motion information fluctuation feature map;

[0011] S5. Using the target detection network YOLOv5 as the basic network, the motion information fluctuation feature map is input into the network for detection, and post-processing is performed to output the behavior detection results of the target in the video.

[0012] The beneficial effects of the present invention are: the present invention adopts three-channel spatiotemporal images, fuses the temporal information of classroom videos into two-dimensional images, and obtains preliminary recommended areas through a small feature extractor DarkNet-19 and a fully connected layer, effectively reducing the amount of calculation. By generating STMap, the recommended area information of the three-channel spatiotemporal image can be effectively divided, thereby improving the feature extraction effect of the network. At the same time, the spatiotemporal feature extractor can obtain the temporal information in the STMap, thereby obtaining the temporal characteristics of the student's goals, which has a better identification effect on similar behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Flowchart of the classroom behavior detection method based on spatiotemporal features of the present invention;

[0014] Figure 2 A schematic diagram of generating a three-channel spatiotemporal image according to the present invention;

[0015] Figure 3 Generate a schematic diagram for the STMap of the present invention. DETAILED DESCRIPTION

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0017] The present invention proposes a classroom behavior detection method based on spatiotemporal characteristics, such as Figure 1 As shown, it is a logical framework diagram of this embodiment. The present invention mainly includes selecting three key frames at intervals of K in the classroom video, graying the key frames and then splicing them according to the RGB three channels to form a three-channel spatiotemporal image containing motion information; using a small DarkNet-19 as a feature extractor to obtain the features of the three-channel spatiotemporal image, adding a fully connected layer to output a preliminary proposed area; the three-channel spatiotemporal image of the preliminary proposed area is formed into a two-dimensional matrix by stacking longitudinal scan lines frame by frame, thereby obtaining an STMap; initializing the STMap and decomposing it using the improved DMD algorithm, and inputting the decomposition result into the spatiotemporal feature extractor to obtain a motion information fluctuation feature map; using the target detection network YOLOv5 as the basic network, inputting the motion information fluctuation feature map into the network for detection, and performing post-processing to output the behavior detection results of the students in the video.

[0018] S1. Select three key frames at intervals of K from the classroom video, convert the key frames into grayscale, and then splice them according to the RGB channels to form a three-channel spatiotemporal image containing motion information.

[0019] Figure 2 This is a schematic diagram of generating three-channel spatiotemporal images in this embodiment. Figure 2 As shown, every K (K = 0, 1, ..., n) frame interval in the video is set as a key frame, and every three key frames form a group. The key frames are grayscaled to form a single-channel grayscale image, and the three key frames are spliced ​​as R, G, and B channel images in chronological order to form a three-channel spatiotemporal image containing motion information. The three-channel spatiotemporal image contains the virtual shadow formed by the motion information of the three frames.

[0020] S2, using a small DarkNet-19 as a feature extractor to obtain the features of the three-channel spatiotemporal image, and adding a fully connected layer to output the preliminary proposed region;

[0021] The DarkNet-19 small feature extractor is used for feature extraction to reduce the network's computational load. The obtained feature information is input into the fully connected layer, and the softmax function is used to obtain preliminary region proposals for the three-channel spatiotemporal image, thereby reducing the computational burden caused by excessive pixels.

[0022] The preliminary proposed area is obtained through the softmax function, including:

[0023]

[0024] Among them, D represents the preliminary proposed area obtained by the softmax function, z iIt represents the one-dimensional vector compressed from features of different scales, and c represents the scale of the feature.

[0025] S3, the three-channel spatiotemporal image of the preliminary proposed area is stacked by longitudinal scanning lines frame by frame to form a two-dimensional matrix, thereby obtaining the STMap;

[0026] Figure 3 The STMap generation diagram for this embodiment is as follows: Figure 3 As shown, the vertical scan lines (l1, l2, l3) are stacked frame by frame according to the order of the three key frames in the R, G, and B channels to form an S n×3 A two-dimensional matrix of , where n represents the number of pixels per scan line, 3 represents the three key frames in the three channels, and each scan line represents the motion state of the target in the current key frame.

[0027] S4, initialize the STMap and decompose it using the improved DMD algorithm, and input the decomposition result into the input spatiotemporal feature extractor to obtain the motion information fluctuation feature map;

[0028] S41. Use the linear time-dependent operator A to reflect the change of the scan line pixels in the STMap. The change of the scan line over time is:

[0029] l x+1 =Al x

[0030] Among them, l x is the state of the current scan line, l x+1 is the state of the next scan line, assuming that the two have temporal correlation, A is a linear parameter, and the change of STMap can be expressed as S i+1 =AS i .

[0031] S42, STMap has a low-rank structure inside, and background pixels are highly correlated between adjacent columns. Then STMap can be represented by a combination of eigenvectors and eigenvalues ​​of the linear time-dependent operator A, as follows:

[0032] S=∑ i φ i b i λ i

[0033] Among them, φ i ,λ i are the eigenvector and eigenmatrix of A, b i are the coordinates of S based on the corresponding eigenvectors.

[0034] The matrix A is reconstructed using the DMD algorithm, and the parameters of the reconstruction result are input into the MLP network for training. The low-order rank of the matrix A is adaptively obtained, and the low-order rank fits the dynamic trajectory of the target in the original video sequence, that is:

[0035] ||S i+1 -AS i ||2→min

[0036] The STMap is decomposed into a low-rank background part and a sparse foreground part.

[0037] S43, the spatiotemporal feature extractor is based on the UNet model and uses a lightweight encoding module to replace the original encoder in the UNet model, reducing the semantic gap between the encoder and decoder. The low-rank background part and sparse foreground part of the STMap are input into the improved spatiotemporal feature extractor UNet model. The encoding module uses multi-layer convolution to downsample and extract features, and uses correlation calculation to achieve corresponding matching relationships between different features. The decoding module uses multiple upsampling operations based on the features that achieve corresponding matching relationships between different features to obtain predicted optical flows corresponding to different features. The output predicted optical flow is fused with the features of the corresponding encoding layer to obtain semantic information from different layers, thereby obtaining a motion information fluctuation feature map.

[0038] S5. Using the target detection network YOLOv5 as the basic network, the motion information fluctuation feature map is input into the network for detection, and post-processing is performed to output the behavior detection results of students in the video.

[0039] S51, inputting the obtained motion information fluctuation feature map into the convolutional neural network of the target detection network YOLO V5, generating a series of anchor boxes for locating and identifying the target on the motion information fluctuation feature map, calculating the confidence of each anchor box, setting a threshold, filtering out anchor boxes with confidence lower than the set threshold, and obtaining candidate boxes;

[0040] The threshold is set to 0.5. If an anchor frame with a confidence level lower than 0.5 is considered to contain target information, the anchor frame is discarded.

[0041] The confidence level includes:

[0042]

[0043] Among them, IOU represents confidence, area(r g ) represents the prediction box area, area(r n ) represents the ground-truth box area.

[0044] S52: Further screen the candidate boxes through the NMS algorithm, select the predicted bounding box with the highest confidence from all candidate boxes as the benchmark, then remove other bounding boxes whose confidence exceeds the predetermined threshold, select the bounding box with the second highest confidence from all candidate boxes as a benchmark, and remove all other bounding boxes whose confidence exceeds the predetermined threshold. Repeat the above operation until all predicted boxes are used as benchmarks to obtain the final detection box, judge the behavior information in the detection box, and obtain the target location, behavior category information and behavior start time.

[0045] The NMS threshold can be adjusted according to the actual scenario and is set to 0.35 as a reference.

[0046] The convolutional neural network of the target detection network YOLO V5 is trained, and the batch size of the network training is set to 16, the total number of iterations is 100 epochs, and the learning rate is 10 -3 , set the weight decay factor to 0.0005 and the momentum factor to 0.9.

[0047] During training, the network automatically calculates the spatiotemporal IoU ratio (IoU) between the link channel and the ground truth. This is the IoU ratio between the region and start time where the behavior occurred and the ground truth. The matching principle for link channels is as follows: for each ground truth in a segment, the link channel with the largest spatiotemporal IoU ratio is found. This link channel is matched with the ground truth and is considered a positive sample. Conversely, if a link channel does not match any ground truth, it matches the background and is considered a negative sample. For the remaining unmatched link channels, if the spatiotemporal IoU ratio of a ground truth is greater than the threshold of 0.5, then this link channel is also matched with this ground truth.

[0048] The loss function of the network includes regression loss and classification loss:

[0049]

[0050] Among them, N is the number of positive samples of the link channel, c is the category confidence prediction value, l is the prediction value, and g is the position parameter of the ground truth, α is the weight coefficient, set to 1, x represents the result of the network output, L conf() represents the classification loss, L loc() represents the regression loss.

[0051] In summary, the present invention selects three key frames at intervals of K from the video, grayscales the key frames, and then splices them according to the RGB channels to form a three-channel spatiotemporal image containing motion information. A small DarkNet-19 is used as a feature extractor to obtain the features of the three-channel spatiotemporal image, and a fully connected layer is added to output a preliminary proposed region. The three-channel spatiotemporal image of the preliminary proposed region is stacked frame by frame to form a two-dimensional matrix to obtain an STMap. The STMap is initialized and decomposed using an improved DMD algorithm. The decomposition result is input into the input spatiotemporal feature extractor to obtain a motion information fluctuation feature map. The object detection network YOLOv5 is used as the base network, and the motion information fluctuation feature map is input into the network for detection. After post-processing, the behavior detection results of the students in the video are output. This effectively reduces the computational complexity of the network and improves the accuracy of fine-grained behavior detection.

[0052] In the description of the present invention, it should be understood that the terms "coaxial", "bottom", "one end", "top", "middle", "the other end", "upper", "one side", "top", "inside", "outside", "front", "center", "both ends", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0053] In the present invention, unless otherwise clearly stipulated and limited, the terms "installation", "setting", "connection", "fixation", "rotation" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0054] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A classroom behavior detection method based on spatiotemporal features, characterized in that: include: S1. Select three key frames from the classroom video to be used for behavior detection, convert the key frames into grayscale, and then splice them according to the RGB three channels to obtain a three-channel spatiotemporal image containing motion information; S2. Use the DarkNet-19 network as a feature extractor. Through multiple convolution and pooling operations of the DarkNet-19 network, features of different scales of the three-channel spatiotemporal image are extracted and irrelevant information is eliminated. Finally, the extracted features are compressed into a one-dimensional vector and passed to the fully connected layer. The preliminary proposed region is obtained through the softmax function. S3, stacking the longitudinal scan lines frame by frame for the preliminary proposed region to form a two-dimensional matrix to obtain a spatiotemporal map STMap; S4, initialize the spatiotemporal map STMap, decompose it using the DMD algorithm, and input the decomposition result into the input spatiotemporal feature extractor to obtain the motion information fluctuation feature map; The spatiotemporal feature extractor comprises: The UNet model is used as the basic spatiotemporal feature extractor, and the original encoder in the UNet model is replaced by a lightweight encoding module to obtain an improved UNet model, which is then used as the final spatiotemporal feature extractor. The S4 specifically includes: S41: Use the linear time-dependent operator A to reflect the changes in the scan line pixels in the STMap, thereby extracting the timing information and obtaining the timing characteristics of the target; S42: According to the temporal characteristics of the target, the STMap is represented as a combination matrix of the eigenvectors and eigenvalues ​​of the linear time-dependent operator A. The DMD algorithm is used to find the low-order rank of the linear time-dependent operator A, and the low-order rank fits the dynamic trajectory of the target in the original video sequence, and the STMap is decomposed into a low-rank background part and a sparse foreground part; S43: The low-rank background part and sparse foreground part of the STMap are input into the improved spatiotemporal feature extractor UNet model. The encoding module uses multi-layer convolution to downsample and extract features, and uses correlation calculation to achieve corresponding matching relationships between different features. The decoding module uses multiple upsampling operations based on the features that achieve corresponding matching relationships between different features to obtain predicted optical flows corresponding to different features. The output predicted optical flow is fused with the features of the corresponding encoding layer to obtain semantic information from different layers, thereby obtaining a motion information fluctuation feature map. S5. Using the target detection network YOLOv5 as the basic network, the motion information fluctuation feature map is input into the network for detection, and post-processing is performed to output the behavior detection results of students in the video.

2. A classroom behavior detection method based on spatiotemporal features according to claim 1, characterized in that: The key frames include setting video frames at intervals of K as key frames, where K=0, 1, ..., n.

3. The classroom behavior detection method based on spatiotemporal features according to claim 1 is characterized in that: The preliminary proposed area is obtained through the softmax function, including: Among them, D represents the preliminary proposed area obtained by the softmax function, z i It represents the one-dimensional vector compressed from features of different scales, and c represents the scale of the feature.

4. The classroom behavior detection method based on spatiotemporal features according to claim 1 is characterized in that: The preliminary proposed region is stacked by vertical scan lines frame by frame to form a two-dimensional matrix to obtain the STMap, which includes: By stacking the vertical scan lines l1, l2, and l3 according to the order of the three key frames in the R, G, and B channels, each scan line represents the motion state of the target in the current key frame, forming an S n×3 A two-dimensional matrix of , where n represents the number of pixels per scan line and 3 represents 3 key frames in three channels.

5. The classroom behavior detection method based on spatiotemporal features according to claim 1 is characterized in that: The motion information fluctuation feature map is input into the network for detection and post-processing, and the behavior detection results of the target in the video are output, including: S51, inputting the obtained motion information fluctuation feature map into the convolutional neural network of the target detection network YOLO V5, generating a series of anchor boxes for locating and identifying the target on the motion information fluctuation feature map, calculating the confidence of each anchor box, setting a threshold, filtering out anchor boxes with confidence lower than the set threshold, and obtaining candidate boxes; S52: Further screen the candidate boxes through the NMS algorithm, select the predicted bounding box with the highest confidence from all candidate boxes as the benchmark, then remove other bounding boxes whose confidence exceeds the predetermined threshold, select the bounding box with the second highest confidence from all candidate boxes as a benchmark, and remove all other bounding boxes whose confidence exceeds the predetermined threshold. Repeat the above operation until all predicted boxes are used as benchmarks to obtain the final detection box, judge the behavior information in the detection box, and obtain the target location, behavior category information and behavior start time.

6. The classroom behavior detection method based on spatiotemporal features according to claim 5 is characterized in that: The confidence level includes: Among them, IOU represents confidence, area(r g ) represents the prediction box area, area(r n ) represents the ground-truth box area.

Images