A student classroom behavior recognition method, device, equipment and storage medium

Abstract

The application discloses a student classroom behavior recognition method and device, equipment and storage medium. The target detection processing is performed on the obtained classroom image to obtain a detection frame representing each student, and a key point representing a human body posture is detected from a region in each detection frame. A space-time graph convolutional neural network is used to process a key point sequence of the same student in continuous multiple frames of classroom images to determine the behavior category of the student. Through the space-time graph convolutional neural network, the spatial dimension features of the key points of a single frame and the time dimension features of the key points of the same student in the continuous multiple frames of classroom images are combined, so that the behavior difference between the target behavior and other interference behaviors can be effectively distinguished, and the scene generalization capability and the recognition accuracy are improved.

Description

A method, device, equipment, and storage medium for recognizing student classroom behavior. Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for recognizing student classroom behavior. Background Technology

[0002] Classroom teaching, as a fundamental form of instruction, has always held a central position in education. In researching classrooms, the various elements within the micro-classroom should always be the focus. Students, as the main participants in the classroom, demonstrate behaviors that reflect their emotions and learning states, providing crucial evidence for classroom teaching research.

[0003] With the development of information technology in education, the use of computers to process images and recognize student behavior plays an important role in classroom teaching research.

[0004] Current computer-aided student behavior recognition methods mostly employ optical flow or background modeling to identify motion regions, or combine target detection algorithms to locate the changing trends of the target in previous and subsequent frames, thereby recognizing student behaviors such as standing up and sitting down. However, the use of optical flow or background modeling methods is often greatly affected by changes in scene and lighting, and requires careful parameter settings, resulting in poor scene generalization, a high likelihood of false alarms, and low recognition accuracy. Summary of the Invention

[0005] This invention provides a method, device, equipment, and storage medium for recognizing student classroom behavior, which can improve the generalization ability of scenarios and improve the accuracy of behavior recognition.

[0006] In a first aspect, embodiments of the present invention provide a method for recognizing student classroom behavior, including:

[0007] Acquire classroom images of students during class;

[0008] The classroom image is subjected to target detection processing to obtain detection boxes representing each student;

[0009] Key points representing human posture are detected from the regions within each of the aforementioned detection frames;

[0010] A spatiotemporal graph convolutional neural network is used to process the key point sequence of the same student in multiple consecutive frames of classroom images to determine the student's behavior category.

[0011] Secondly, embodiments of the present invention also provide a student classroom behavior recognition device, comprising:

[0012] The image acquisition module is used to acquire classroom images of students during class.

[0013] The target detection module is used to perform target detection processing on the classroom image to obtain detection boxes representing each student;

[0014] The key point detection module is used to detect key points representing human posture from the regions within each of the detection frames.

[0015] The graph convolution processing module is used to process the key point sequence of the same student in multiple consecutive frames of classroom images using a spatiotemporal graph convolutional neural network to determine the student's behavior category.

[0016] Thirdly, embodiments of the present invention also provide a computer device, comprising:

[0017] One or more processors;

[0018] Storage device for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the student classroom behavior recognition method provided in the first aspect of the present invention.

[0020] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the student classroom behavior recognition method provided in the first aspect of the present invention.

[0021] The student classroom behavior recognition method provided in this invention performs target detection processing on acquired classroom images to obtain detection boxes representing each student. Key points characterizing human posture are detected from the regions within each detection box. A spatiotemporal graph convolutional neural network is then used to process the key point sequence of the same student in multiple consecutive classroom images to determine the student's behavior category. By combining the spatial dimension features of key points in a single frame with the temporal dimension features of key points of the same student in multiple consecutive classroom images using the spatiotemporal graph convolutional neural network, the behavioral differences between target behavior and other interfering behaviors can be effectively distinguished, thereby improving scene generalization ability and recognition accuracy. Attached Figure Description

[0022] Figure 1 is a flowchart of a student classroom behavior recognition method provided in Embodiment 1 of the present invention;

[0023] Figure 2A is a flowchart of a student classroom behavior recognition method provided in Embodiment 2 of the present invention;

[0024] Figure 2B is a schematic diagram of a graph convolutional neural network provided in an embodiment of the present invention;

[0025] Figure 3 is a structural schematic diagram of a student classroom behavior recognition device provided in an embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0028] Example 1

[0029] Figure 1 is a flowchart of a student classroom behavior recognition method provided in Embodiment 1 of the present invention. This embodiment is applicable to student classroom behavior recognition. The method can be executed by the student classroom behavior recognition device provided in this embodiment of the present invention. The device can be implemented by software and / or hardware, and is usually configured in a computer device. As shown in Figure 1, the method specifically includes the following steps:

[0030] S101. Obtain classroom images of students during class.

[0031] For example, classroom images of students during class are recorded using a camera. These images can be video stream images. In this embodiment of the invention, a real-time processing method can be used, i.e., real-time acquisition of classroom images and real-time recognition of student classroom behavior; or a pre-recording method can be used, i.e., pre-recording of classroom images of students during class and storing them locally or on a server, and retrieving these images from the local storage or server when student classroom behavior recognition is needed. This embodiment of the invention does not impose any limitations on this method.

[0032] S102. Perform target detection processing on the classroom image to obtain the detection box representing each student.

[0033] In this embodiment of the invention, an object detection model is used to perform object detection processing on each frame of the acquired classroom image to detect the target object (i.e., the student) and display it using a detection bounding box. In this embodiment of the invention, the object detection model used for object detection processing can be a common object detection model, such as the Fast-RCNN model or the YOLO (YouOnly Look Once) series of models, and this embodiment of the invention is not limited thereto.

[0034] S103. Detect key points representing human posture from the areas within each detection frame.

[0035] After obtaining the bounding boxes representing each student, a human keypoint detection model is used to perform human keypoint detection on the students within each bounding box, detecting the key points that represent human posture.

[0036] Human keypoint detection, also known as human pose estimation, is a relatively fundamental task in computer vision, serving as a prerequisite for human action recognition, behavior analysis, and human-computer interaction. Generally, human keypoint detection can be subdivided into single / multi-person keypoint detection and 2D / 3D keypoint detection. Furthermore, some algorithms perform keypoint tracking after detection, a process known as human pose tracking.

[0037] For example, in this embodiment of the invention, the human key point detection model can be a CPM (Convolutional Pose Machines) model, a DeeperCut model, an AlphaPose model, or a SimplePose model, and this embodiment of the invention does not limit it.

[0038] S104. Use a spatiotemporal graph convolutional neural network to process the key point sequence of the same student in multiple consecutive classroom images to determine the student's behavior category.

[0039] In this invention, a keypoint sequence is formed from keypoints of the same student in multiple consecutive classroom images. In this embodiment, a Spatial Temporal Graph Neural Network (ST-GNN) is used to process this sequence. The ST-GNN fully mines the keypoints of the same student across multiple frames, establishing relationships between them in both temporal and spatial dimensions, and determining the student's behavior category. This keypoint sequence serves as the skeleton spatiotemporal sequence of the ST-GNN. While single-frame keypoints describe the spatial distribution of student behavior, consecutive multi-frame keypoints describe the temporal changes in behavior. Therefore, the skeleton spatiotemporal sequence formed by the keypoint sequence of the same student in multiple consecutive classroom images contains rich temporal and spatial features of behavior, effectively distinguishing target behavior from other interfering behaviors, thereby improving scene generalization ability and recognition accuracy.

[0040] The student classroom behavior recognition method provided in this invention performs target detection processing on acquired classroom images to obtain detection boxes representing each student. Key points characterizing human posture are detected from the regions within each detection box. A spatiotemporal graph convolutional neural network is then used to process the key point sequence of the same student in multiple consecutive classroom images to determine the student's behavior category. By combining the spatial dimension features of key points in a single frame with the temporal dimension features of key point sequences of the same student in multiple consecutive classroom images using the spatiotemporal graph convolutional neural network, the behavioral differences between target behavior and other interfering behaviors can be effectively distinguished, thereby improving scene generalization ability and recognition accuracy.

[0041] Example 2

[0042] Figure 2A is a flowchart of a student classroom behavior recognition method provided in Embodiment 2 of the present invention. This embodiment refines the above-described Embodiment 1 and describes in detail the specific process of using a spatiotemporal graph convolutional neural network to process the key point sequence of the same student in multiple consecutive frames of classroom images to determine the student's behavior category. As shown in Figure 2A, the method includes:

[0043] S201. Obtain classroom images of students during class.

[0044] For example, classroom images of students during class are recorded using a camera. These images can be video stream images. In this embodiment of the invention, a real-time processing method can be used, i.e., real-time acquisition of classroom images and real-time recognition of student classroom behavior; or a pre-recording method can be used, i.e., pre-recording of classroom images of students during class and storing them locally or on a server, and retrieving these images from the local storage or server when student classroom behavior recognition is needed. This embodiment of the invention does not impose any limitations on this method.

[0045] S202. Perform target detection processing on the classroom image to obtain the detection box representing each student and the pose category of each student.

[0046] In this embodiment of the invention, a lightweight YOLO v4 / v5 object detection model is pre-trained using a large number of data samples labeled with standing and sitting posture categories.

[0047] In the model application stage, the trained target detection model is used to perform target detection processing on each frame of classroom image collected, and the target object (i.e., student) is detected and displayed using a detection box. At the same time, the target detection model outputs the pose category of the target object. In this embodiment of the invention, the pose category of the target object includes standing and sitting postures.

[0048] S203. Detect six key points from the area within the detection box, including the left eye, right eye, nose, left shoulder, right shoulder, and neck.

[0049] In this embodiment of the invention, a lightweight SimplePose keypoint detection model is trained in advance using labeled data samples of six keypoints, including the left eye, right eye, nose, left shoulder, right shoulder, and neck.

[0050] In the model application phase, the trained key point detection model is used to perform human key point detection on the area within the detection box detected in the aforementioned target detection steps, detecting 6 key points including the left eye, right eye, nose, left shoulder, right shoulder and neck.

[0051] S204. Perform target tracking processing on multiple classroom images to identify the same students in the multiple classroom images.

[0052] Target tracking involves predicting the size and position of a target in subsequent frames, given the initial frame size and position of the target in a video sequence. In this embodiment of the invention, each student in a classroom image is tracked to identify the same student across multiple frames (e.g., 20 frames) of classroom images, and the bounding boxes and keypoints for each student in the multiple classroom images are recorded.

[0053] In this embodiment of the invention, commonly used target tracking algorithms can be employed, such as optical flow, particle filtering, correlation filtering, or deep learning-based tracking methods. This embodiment of the invention does not limit the specific methods used.

[0054] In one embodiment of the present invention, considering that in a teaching scenario, students' displacement during class is mostly within a small range, an IoU-Tracker is used for target tracking. This involves calculating the IoU (Intersection over Union) between the tracking box and the detection box, and achieving efficient target tracking through IoU matching. The IoU is the ratio of the intersection to the union of the tracking box and the detection box. The specific tracking process is as follows:

[0055] In the aforementioned object detection step, for the first frame of the classroom image, in addition to outputting the detection boxes representing each student and the pose category of each student, an initial ID is also assigned to each detection box (i.e., each student). The initial ID is randomly assigned, and each student has a unique initial ID.

[0056] Calculate the intersection-union (IoU) ratio of the detection boxes in one frame of the classroom image with all detection boxes in the adjacent frame of the classroom image. Specifically, for the first frame of the classroom image, take one of the detection boxes as the target detection box, calculate the IoU ratio of the target detection box with each detection box (also called a tracking box) in the second frame of the classroom image, and then take the tracking box with the largest IoU ratio with the target detection box as the detection box of the same student, and assign the initial ID of the target detection box to the tracking box. Iterate through all detection boxes in the first frame of the classroom image and repeat the above process until all detection boxes in the first frame of the image have a corresponding tracking box in the second frame of the image.

[0057] For the second and third classroom images, repeat the above process, and continue this cycle until all classroom images show that the bounding boxes belonging to the same student have been assigned the same initial ID. This completes the target tracking for each student and identifies the same student across multiple classroom images.

[0058] In some embodiments of the present invention, in order to reduce the data processing pressure on the subsequent graph convolutional neural network and improve the recognition accuracy, the classroom images can be pre-screened to identify classroom image sequences suspected of having pose changes. The specific screening process is shown in steps S205-S209.

[0059] S205. The first determination result for whether a student's posture has changed is based on the posture category of the same student in multiple consecutive classroom images.

[0060] Specifically, as described above, in the object detection step, the object detection model outputs a detection box representing the target object (i.e., the student), and simultaneously outputs the pose category of the target object. In this embodiment of the invention, the pose category of the target object includes standing and sitting postures.

[0061] The first determination of whether a student's posture has changed is based on the posture category of the same student in multiple consecutive frames (e.g., 5 frames) of classroom images. Specifically, if the posture category of the same student in one or more frames of consecutive classroom images is different from the posture category in other frames (e.g., one frame shows a standing posture, and other frames show a sitting posture), then the first determination is that the student has changed posture. If the posture category of the same student in multiple consecutive classroom images is the same (e.g., the posture category in multiple frames is all sitting), then the first determination is that the student has not changed posture.

[0062] S206. When the first determination result is that the student's posture has changed, multiple frames of classroom images within a preset range are determined as the target images, with the classroom image where the posture change has occurred as the center.

[0063] In this embodiment of the invention, when the first determination result is that the student's posture has changed, multiple frames of classroom images within a preset range, centered on the classroom image where the posture change occurred, are determined as target images. For example, when the first determination result is that the student's posture has changed, i.e., a posture change occurs in a certain frame, a total of 40 classroom images (including the frame where the posture change occurred) centered on the classroom image where the posture change occurred are used as target images for subsequent graph convolution processing.

[0064] S207. When the first judgment result is that the student's posture has not changed, calculate the displacement of the student's key points in multiple consecutive classroom images.

[0065] To avoid the possibility of misjudging the pose category output in the target detection step, in this embodiment of the invention, when the first determination result is that the student has not changed pose, the displacement of the student's key points in multiple consecutive frames (e.g., 5 frames) of classroom images is calculated, that is, the displacement of the key points of the same student in the first and second consecutive classroom images.

[0066] S208. A second determination result based on the displacement of the student's key points to determine whether the student's posture has changed.

[0067] In some embodiments of the present invention, when the displacement of a student's key point is greater than a preset threshold, the second determination result is considered that the student has undergone a posture change.

[0068] To eliminate the problem of large discrepancies between the displacement of a single key point and the actual displacement caused by false detection or low positional accuracy of key points in the aforementioned key point detection steps, thus leading to misjudgment of the second determination result, in some embodiments of the present invention, the ratio of the average displacement of multiple key points to the width of the detection frame is calculated as the displacement degree, and the second determination result is determined based on the displacement degree. The specific process is as follows:

[0069] Calculate the displacement vectors of keypoints in the first and last frames of a series of consecutive classroom images. Specifically, for the same student, calculate the displacement vectors of the keypoints representing the left eye, right eye, nose, left shoulder, right shoulder, and neck in the first and last frames of a series of consecutive classroom images.

[0070] Calculate the ratio of the mean magnitude of each displacement vector to the width of the detection frame. Specifically, calculate the mean magnitude of the six displacement vectors, and then calculate the ratio of this mean to the width of the detection frame.

[0071] The calculated ratio is compared with a preset threshold. If the ratio is greater than the preset threshold, the second determination result is that the student's posture has changed. If the ratio is less than or equal to the preset threshold, the second determination result is that the student's posture has not changed.

[0072] S209. When the second determination result is that the student's posture has changed, determine multiple classroom images within a preset range as the target image, with the target frame classroom image where the posture change has occurred as the center.

[0073] In step S208 above, if the second determination result is that the student's posture has changed, multiple classroom images within a preset range are determined as the target images, centered on the target frame classroom image where the posture change occurred. For example, when the first determination result is that the student's posture has changed, that is, when a posture change occurs in a certain frame, a total of 40 classroom images (including the frame where the posture change occurred) centered on the classroom image where the posture change occurred are used as the target images for subsequent graph convolution processing.

[0074] If the second determination result is that the student did not change his posture, then the student is considered not to have changed his behavior.

[0075] S210. Use a spatiotemporal graph convolutional neural network to process the key point sequence of the same student in multiple consecutive target images to determine the student's behavior category.

[0076] In this embodiment of the invention, a spatiotemporal graph convolutional neural network is used to process the key point sequence of the same student in multiple consecutive target images. The pre-designed spatiotemporal graph convolutional neural network is used to fully mine the key point sequence of the same student in multiple classroom images, form the correlation between key points in the spatial and temporal dimensions, and determine the student's behavior category.

[0077] For example, Figure 2B is a schematic diagram of the structure of a spatiotemporal graph convolutional neural network provided in an embodiment of the present invention. The graph convolutional neural network includes multiple stacked spatiotemporal graph convolutional units, a global pooling layer, and a softmax layer. Each spatiotemporal graph convolutional unit includes a graph convolutional network and a temporal convolutional network (TCN). The spatiotemporal graph convolutional neural network described in this example can be implemented by repeatedly stacking the spatiotemporal graph convolutional unit. As shown in Figure 2B, the processing procedure of the spatiotemporal graph convolutional neural network is as follows:

[0078] 1. Spatial features are extracted from keypoint sequences using graph convolutional networks.

[0079] 1) Calculate the adjacency matrix of the human skeleton diagram:

[0080] For example, in this embodiment of the invention, the topological graph of the designed human skeleton (a graph topological structure composed of 6 key points: left eye, right eye, nose, left shoulder, right shoulder, and neck) is directly used as the undirected graph topological structure for calculating the adjacency matrix. In this example, there are 6 key points, so the size of the adjacency matrix A is a 6*6 matrix, where the values ​​satisfy:

[0081]

[0082] i and j represent key points. i ≠ j means that i and j are different key points, and i = j means that i and j are the same key point.

[0083] In practical applications, normalization (standardization) of the adjacency matrix A is also considered. This invention adopts the following approach: The calculation is performed in the manner described above, where I is the identity matrix, D is the degree matrix of A, and D... -1 That is, the inverse matrix of D. This represents the normalized adjacency matrix (also known as the Laplace matrix).

[0084] 2) Extract features from the key points of the same student in each frame of a series of target images to obtain key point features that represent the key point node information of each frame.

[0085] For example, in this embodiment of the invention, the input data has an N*C*T*V format, where N is the batch size, C is the keypoint feature dimension, V represents the keypoint dimension (number of keypoints), and T is the number of frames. Features representing the keypoint information of each keypoint are extracted using a 1×1 convolutional layer. The input keypoint information may include the location of the keypoint and the confidence level of the predicted keypoint representation of the left eye, right eye, nose, left shoulder, right shoulder, or neck.

[0086] 3) Calculate the product of the adjacency matrix and the key point features of the same student in each frame of multiple consecutive target images to obtain the spatial features.

[0087] In some embodiments of the present invention, calculating the product of the adjacency matrix and the keypoint features of each frame of the same student in multiple consecutive target images requires perming and reshaping the adjacency matrix. Simultaneously, the keypoint features of each frame of the same student in multiple consecutive target images are also permed and reshaped to accommodate the matrix product requirements. The product process is essentially the process of extracting the spatial features of the keypoint skeleton graph.

[0088] In summary, steps 1), 2), and 3) constitute the feature extraction of a frame of human skeleton data, which is the calculation process of graph convolution. The mathematical expression of graph convolution operation is shown in the following formula:

[0089]

[0090] Where H represents spatial features, X represents keypoint features of the same student in multiple consecutive frames of target images, and W represents the graph convolution parameters of the graph convolutional layer. Let be the normalized adjacency matrix, and σ be the activation function. For example, in this embodiment of the invention, the ReLU function is selected as the activation function. The expression for the ReLU function is:

[0091] f(x) = max(0,x)

[0092] When the input is negative, there is no activation at all, and the ReLU function fails. The output of the ReLU function is either 0 or a positive number. ReLU can overcome the vanishing gradient problem and speed up training. It should be noted that in other embodiments of the present invention, the activation function in the graph convolutional layer can also be other activation functions, such as the Sigmoid function or the Tanh function, and the embodiments of the present invention are not limited thereto.

[0093] 2. Temporal features are extracted from spatial features using a temporal convolutional network.

[0094] In some embodiments of the present invention, temporal feature extraction of the skeleton sequence is performed by using a TCN network. Specifically, for an input dimension of N*C*T*V format, where N is the batch size, C is the key point feature dimension, V represents the key point dimension (number of key points), and T is the number of frames, a convolutional kernel with n*1 kernel and stride s is used to perform convolution calculation on the input dimension T, and then the input is normalized by a batch normalization layer. Finally, the output is processed by the ReLU function to achieve temporal feature extraction.

[0095] In summary, steps 1, 2, 3, and 4 above enable the extraction of spatiotemporal information from keypoint data of the same student in different frames of a multi-frame target image using the spatiotemporal graph convolution module.

[0096] 3. Determine student behavior categories based on features output from multiple stacked spatiotemporal graph convolutional units.

[0097] By repeatedly stacking the spatiotemporal graph convolutional units designed in steps 1-2 above N times (e.g., N=4 times), and passing them through a GAP (Global Average Pooling) layer and a softmax classification layer, the probability values ​​of each student's behavior category are inferred and output. The behavior category corresponding to the highest probability value is taken as the student's behavior category. For example, in this application, the behavior category includes standing up, sitting down, or no change in behavior.

[0098] The student classroom behavior recognition method provided in this invention pre-screens classroom images to identify sequences of images suspected of showing posture changes. It then further uses a spatiotemporal graph convolutional neural network (SPCNN) to recognize student classroom behavior, thereby reducing the data processing load on the subsequent SPCNN and improving recognition accuracy. By combining the spatial dimension features of key points in a single frame with the temporal dimension features of key points for the same student across multiple consecutive classroom images, the SPCNN effectively distinguishes target behavior from other interfering behaviors, thus improving scene generalization ability and recognition accuracy.

[0099] Example 3

[0100] Figure 3 is a schematic diagram of a student classroom behavior recognition device provided in an embodiment of the present invention. As shown in Figure 3, the device includes:

[0101] Image acquisition module 301 is used to acquire classroom images of students during class.

[0102] The target detection module 302 is used to perform target detection processing on the classroom image to obtain detection boxes representing each student;

[0103] The key point detection module 303 is used to detect key points representing human posture from the regions within each of the detection frames.

[0104] The spatiotemporal graph convolution processing module 304 is used to process the key point sequence of the same student in multiple consecutive frames of classroom images using a spatiotemporal graph convolutional neural network to determine the student's behavior category.

[0105] In some embodiments of the present invention, the key point detection module includes:

[0106] The key point extraction submodule is used to detect six key points, including those representing the left eye, right eye, nose, left shoulder, right shoulder, and neck, from the area within the detection box.

[0107] In some embodiments of the present invention, the classroom image is subjected to target detection processing to obtain the pose category of each student. The spatiotemporal graph convolution processing module 304 includes:

[0108] The target tracking submodule is used to perform target tracking processing on multiple frames of classroom images to identify the same students in multiple frames of classroom images.

[0109] The first determination result determination submodule is used to determine whether the student has undergone a posture change based on the posture category of the same student in multiple consecutive classroom images.

[0110] The first target image determination submodule is used to determine multiple frames of classroom images within a preset range, centered on the classroom image where the posture change occurred, as target images when the first determination result is that the student's posture has changed.

[0111] The first spatiotemporal graph convolution processing submodule is used to process the key point sequence of the same student in multiple consecutive frames of target images using a spatiotemporal graph convolutional neural network to determine the student's behavior category.

[0112] In some embodiments of the present invention, the target tracking submodule includes:

[0113] The intersection-union ratio (IUU) calculation unit is used to calculate the IUU of the detection boxes in one frame of the classroom image with all detection boxes in the adjacent frame of the classroom image;

[0114] A student identification unit is used to identify the same student in multiple consecutive frames of classroom images based on the intersection-union ratio.

[0115] In some embodiments of the present invention, the spatiotemporal graph convolution processing module 304 further includes:

[0116] The displacement calculation submodule is used to calculate the displacement of the key points of the student in multiple consecutive frames of classroom images when the first determination result is that the student has not changed posture.

[0117] The second determination result determination submodule is used to determine whether the student has undergone a posture change based on the displacement of the student's key points.

[0118] The second target image determination submodule is used to determine multiple classroom images within a preset range as target images when the second determination result is that the student's posture has changed.

[0119] The second spatiotemporal graph convolution processing submodule is used to process the key point sequence of the same student in multiple consecutive frames of target images using a spatiotemporal graph convolutional neural network to determine the student's behavior category.

[0120] In some embodiments of the present invention, the second determination result determination submodule includes:

[0121] The displacement vector calculation unit is used to calculate the displacement vectors of key points in the first and last frames of a series of classroom images.

[0122] A ratio calculation unit is used to calculate the ratio of the mean of the magnitudes of each displacement vector to the width of the detection frame;

[0123] The second determination result unit is used to determine that the student has changed posture when the ratio is greater than a preset threshold, and to determine that the student has not changed posture when the ratio is less than or equal to the preset threshold.

[0124] In some embodiments of the present invention, the spatiotemporal graph convolutional network includes multiple stacked spatiotemporal graph convolutional units, each spatiotemporal graph convolutional unit including a graph convolutional network and a temporal convolutional network, and a first spatiotemporal graph convolutional processing submodule or a second spatiotemporal graph convolutional processing submodule includes:

[0125] The graph convolution processing unit is used to extract spatial features from the keypoint sequence using a graph convolutional network.

[0126] A temporal convolutional processing unit is used to extract temporal features from the spatial features using a temporal convolutional network;

[0127] A behavior category determination unit is used to determine the student's behavior category based on features output by multiple stacked spatiotemporal graph convolutional units.

[0128] In some embodiments of the present invention, the graph convolution processing unit includes:

[0129] The key point feature extraction subunit is used to extract key points for each frame of the same student in multiple consecutive frames of target images, and obtain key point features that represent key point information of each frame.

[0130] The adjacency matrix calculation subunit is used to calculate the adjacency matrix of the human skeleton diagram;

[0131] The spatial feature calculation subunit is used to calculate the product of the adjacency matrix and the key point features of the same student in each frame of multiple consecutive target images to obtain spatial features.

[0132] The aforementioned student classroom behavior recognition device can execute the student classroom behavior recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the student classroom behavior recognition method.

[0133] Example 4

[0134] Embodiment 4 of the present invention provides a computer device. Figure 4 is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. As shown in Figure 4, the computer device includes:

[0135] The mobile terminal includes a processor 401, a memory 402, a communication module 403, an input device 404, and an output device 405. The number of processors 401 in the mobile terminal can be one or more; Figure 4 shows an example of one processor 401. The processor 401, memory 402, communication module 403, input device 404, and output device 405 in the mobile terminal can be connected via a bus or other means; Figure 4 shows an example of connection via a bus. The aforementioned processor 401, memory 402, communication module 403, input device 404, and output device 405 can be integrated into a computer device.

[0136] The memory 402, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the module corresponding to the student classroom behavior recognition method in the above embodiment. The processor 401 executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory 402, thereby realizing the student classroom behavior recognition method described above.

[0137] Memory 402 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the microcomputer. Furthermore, memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, memory 402 may further include memory remotely located relative to processor 401, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0138] The communication module 403 is used to establish a connection with external devices (such as smart terminals) and to realize data interaction with external devices. The input device 404 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the computer device.

[0139] The computer device provided in this embodiment can execute the student classroom behavior recognition method provided in any of the above embodiments of the present invention, and has corresponding functions and beneficial effects.

[0140] Example 5

[0141] Embodiment 5 of the present invention provides a storage medium containing computer-executable instructions, on which a computer program is stored. When executed by a processor, the program implements the student classroom behavior recognition method provided in any of the above embodiments of the present invention. The method includes:

[0142] Acquire classroom images of students during class;

[0143] The classroom image is subjected to target detection processing to obtain detection boxes representing each student;

[0144] Key points representing human posture are detected from the regions within each of the aforementioned detection frames;

[0145] A spatiotemporal graph convolutional neural network is used to process the key point sequence of the same student in multiple consecutive frames of classroom images to determine the student's behavior category.

[0146] It should be noted that the embodiments of the apparatus, device, and storage medium are basically similar to the method embodiments, so the description is relatively simple. For relevant details, please refer to the description of the method embodiments.

[0147] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a robot, personal computer, server, or network device, etc.) to execute the student classroom behavior recognition method described in any embodiment of the present invention.

[0148] It is worth noting that the various modules, sub-modules, and units included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as they can achieve the corresponding functions; in addition, the specific names of each functional module are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0149] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0150] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0151] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for recognizing student classroom behavior, characterized in that, include: Acquire classroom images of students during class; The classroom images are subjected to target detection processing to obtain detection boxes representing each student; key points representing human posture are detected from the regions within each detection box; a spatiotemporal graph convolutional neural network is used to process the key point sequence of the same student in multiple consecutive classroom images to determine the student's behavior category; the classroom images are subjected to target detection processing to obtain the posture category of each student, and the spatiotemporal graph convolutional neural network is used to process the key point sequence of the same student in multiple consecutive classroom images to determine the student's behavior category, including: performing target tracking processing on multiple classroom images to identify the same student in multiple classroom images; A first determination result is made based on the posture category of the same student in multiple consecutive classroom images to determine whether the student's posture has changed; when the first determination result is that the student's posture has changed, multiple classroom images within a preset range are determined as target images, with the classroom image where the posture change occurred as the center; a spatiotemporal graph convolutional neural network is used to process the key point sequence of the same student in multiple consecutive target images to determine the student's behavior category.

2. The student classroom behavior recognition method according to claim 1, characterized in that, Detecting key points representing human posture from the regions within each of the detection frames includes detecting six key points representing the left eye, right eye, nose, left shoulder, right shoulder, and neck from the regions within the detection frames.

3. The student classroom behavior recognition method according to claim 1, characterized in that, The target tracking process for multiple classroom images to identify the same students in the multiple classroom images includes: calculating the intersection-union ratio (IUR) of the detection boxes in one classroom image with all detection boxes in an adjacent classroom image; and identifying the same students in consecutive classroom images based on the IUR.

4. The student classroom behavior recognition method according to claim 1, characterized in that, The method employs a spatiotemporal graph convolutional neural network to process the keypoint sequence of the same student in multiple consecutive classroom images to determine the student's behavior category. The method further includes: when the first determination result indicates that the student has not changed posture, calculating the displacement of the student's keypoints in the multiple consecutive classroom images; determining a second determination result based on the displacement of the student's keypoints to determine whether the student has changed posture; when the second determination result indicates that the student has changed posture, determining multiple classroom images within a preset range centered on the target frame classroom image where the posture change occurred as target images; and using a spatiotemporal graph convolutional neural network to process the keypoint sequence of the same student in the multiple consecutive target images to determine the student's behavior category.

5. The student classroom behavior recognition method according to claim 4, characterized in that, The second determination result based on the displacement of the student's key points to determine whether the student has undergone a posture change includes: calculating the displacement vectors of the key points in the first and last frames of a series of classroom images; calculating the ratio of the mean of the magnitudes of each displacement vector to the width of the detection box; determining that the second determination result is that the student has undergone a posture change when the ratio is greater than a preset threshold; and determining that the second determination result is that the student has not undergone a posture change when the ratio is less than or equal to the preset threshold.

6. The student classroom behavior recognition method according to any one of claims 1, 4, and 5, characterized in that, The spatiotemporal graph convolutional neural network includes multiple stacked spatiotemporal graph convolutional units, each comprising a graph convolutional network and a temporal convolutional network. The spatiotemporal graph convolutional neural network is used to process keypoint sequences of the same student in multiple consecutive frames of target images to determine the student's behavior category. This includes: extracting spatial features from the keypoint sequences using a graph convolutional network; extracting temporal features from the spatial features using a temporal convolutional network; and determining the student's behavior category based on the features output by the multiple stacked spatiotemporal graph convolutional units.

7. The student classroom behavior recognition method according to claim 6, characterized in that, The method employs a graph convolutional network to extract spatial features from a sequence of key points, including: extracting features from the key points of the same student in each frame of a series of target images to obtain key point features that represent the key point information of each frame; calculating the adjacency matrix of the human skeleton graph; and calculating the product of the adjacency matrix and the key point features of the same student in each frame of a series of target images to obtain spatial features.

8. A student classroom behavior recognition device, characterized in that, The method for recognizing student classroom behavior according to any one of claims 1-7 includes: an image acquisition module for acquiring classroom images of students during class; a target detection module for performing target detection processing on the classroom images to obtain detection boxes representing each student; a key point detection module for detecting key points representing human posture from the regions within each detection box; and a graph convolution processing module for processing the key point sequence of the same student in multiple consecutive frames of classroom images using a spatiotemporal graph convolutional neural network to determine the student's behavior category.

9. A computer device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the student classroom behavior recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the student classroom behavior recognition method as described in any one of claims 1-7.

Images