A method and apparatus for student posture assessment

Abstract

The application discloses a neural network HRNet for student posture evaluation, a method and a device, and relates to the technical field of neural networks.The neural network is an improved HRNet-In network, which is obtained by replacing a 3*3 conv convolution kernel in a bottleneck layer of an original HRNet network with a 3*3 Involution operator.The method comprises the following steps: obtaining a picture containing a student posture, inputting the trained neural network HRNet-In, and obtaining corresponding classroom behavior.The device is a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize corresponding steps in the above method.The speed of identifying student classroom behavior is fast and the accuracy is high by replacing the 3*3 conv convolution kernel with the 3*3 Involution operator.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a neural network HRNet, method, and apparatus for student posture evaluation. Background Technology

[0002] The author conducted a search using the query (TACD = (HRNet AND Student AND (Behavior OR Posture) AND bottleneck)) and obtained the following similar existing technical solutions.

[0003] The authorization announcement number is CN115294652B, and the title is "A Method and System for Calculating Behavioral Similarity Based on Deep Learning". The method includes: based on a human pose estimation model, determining the coordinate information of limb key points for a standard action and a action to be corrected, respectively, according to first image data and second image data; calculating the limb offset of the standard action and the limb offset of the action to be corrected at each limb key point, respectively, based on the coordinate information of the limb key points of the standard action and the limb key points of the action to be corrected; determining the limb offset angle between the action to be corrected and the standard action at each limb key point, based on the limb offset of the standard action and the limb offset of the action to be corrected at each limb key point; and determining the comprehensive similarity between the action to be corrected and the standard action based on the limb offset angle between the action to be corrected and the standard action at each limb key point.

[0004] The authorization announcement number is CN112418195B, entitled "A Method, Apparatus, Electronic Device, and Storage Medium for Facial Keypoint Detection." It aims to improve the low accuracy of keypoint detection in facial images. The method includes: obtaining multiple facial images and image annotations for each facial image, where the image annotations are keypoints annotated based on the contours of sensory organs in the facial image; generating mask labels based on the keypoints in each image annotation, obtaining multiple mask labels, where each mask label represents the probability that a keypoint in the facial image is occluded; using the multiple facial images as training data, and the multiple mask labels and image annotations of the multiple facial images as training labels, training the mask branch and keypoint branch of the keypoint detection network to obtain a trained keypoint detection model, which is used to detect facial keypoints in an image.

[0005] Based on the two patent documents mentioned above and existing technical solutions, the inventors analyze the existing technical solutions as follows.

[0006] The classroom is a crucial channel for students to acquire knowledge, and their behavior reflects their level of understanding. Therefore, classroom behavior recognition is of great significance to teaching. In traditional classrooms, teachers primarily assess teaching effectiveness by observing students' behavior. However, due to limited time and energy, teachers struggle to monitor the engagement of every student. With the rapid development of deep learning technology and edge computing chips, school classrooms are becoming increasingly intelligent. Utilizing object detection algorithms and human pose estimation algorithms for student behavior recognition helps teachers gain a comprehensive understanding of classroom dynamics and improves teaching quality. Existing classroom behavior recognition methods include improved PAF methods, improved Faster R-CNN algorithms, GestureDet detection algorithms, and inter-frame difference methods.

[0007] The improved PAF method is an algorithm for human pose estimation. Based on Convolutional Neural Networks (CNNs) and Part Affinity Fields (PAF), it estimates human pose by detecting the connections between key points on the human body. The improved PAF algorithm adds four different sized bounding boxes (×0.5, ×1, ×1.5, and ×2) to the PAF. The introduction of different bounding box sizes solves the problem of detecting key points on the human skeleton in small targets, thus enabling more accurate detection of students raising their hands. However, this method has certain drawbacks. Because it is highly dependent on the initial parameter settings, parameter tuning is required in practical applications, which increases the algorithm's complexity and computational cost.

[0008] The improved Faster R-CNN algorithm, an object detection algorithm, improves detection speed by introducing a Region Proposal Network (RPN). This algorithm first uses a convolutional neural network to extract features, then uses the RPN to generate candidate regions, and finally uses a classifier and regressor to classify and fine-tune these candidate regions. The improved Faster R-CNN algorithm is based on Faster R-CNN and employs a new fusion strategy to improve the accuracy of behavior detection in low-resolution videos. It was used to design an intelligent student behavior analysis system that can automatically detect student behaviors such as raising hands, standing, and sleeping. However, its detection process is highly dependent on generating accurate candidate boxes. Low-quality or inaccurate candidate boxes may lead to missed or false detections.

[0009] The GestureDet detection algorithm is a gesture recognition algorithm based on object detection. It uses the single-stage object detector YOLOv3 as its base model and improves upon it. GestureDet introduces an attention mechanism on top of YOLOv3 to improve gesture detection accuracy. Furthermore, GestureDet uses multi-scale training and testing techniques to improve the algorithm's ability to detect gestures at different scales. However, GestureDet typically requires processing video or image sequences, including hand detection, tracking, and gesture classification. These steps can lead to high computational complexity, thus reducing the algorithm's real-time performance and resulting in poor real-time performance.

[0010] The inter-frame difference method extracts the region of interest by comparing the poses of two adjacent frames, and then performs face tracking and recognition to determine whether a student is standing. For hand-raising behavior, a skin color detection-based algorithm is used to identify students raising their hands. Since the skin color of hands and faces is similar, the shape of the detected skin color region is used for differentiation. Hands are usually long and thin, while faces are mostly round, thus filtering the hand region to determine whether a hand is raised. However, this method has problems such as sensitivity to brightness and background changes, the need to determine a threshold, sensitivity to deformation of moving targets, and difficulty in handling complex backgrounds.

[0011] Existing technical issues and considerations:

[0012] How to solve the technical problems of slow speed and low accuracy in identifying student classroom behavior. Summary of the Invention

[0013] This invention provides a neural network HRNet, method, and apparatus for student posture assessment, solving the technical problems of slow speed and low accuracy in recognizing student classroom behavior.

[0014] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0015] A neural network for student posture evaluation, HRNet, is an improved version of the neural network HRNet-In. Based on the original HRNet network, the 3×3 convolution kernels in the bottleneck layer of the original HRNet network are replaced with 3×3 involution operators to obtain the improved HRNet-In network.

[0016] A further technical solution is as follows: the HRNet-In network includes a feature layer, the feature layer includes a bottleneck layer, the bottleneck layer includes a first convolution kernel, an involution operator, and a second convolution kernel connected in sequence, the first convolution kernel is a 1×1 convolution kernel, the second convolution kernel is a 1×1 convolution kernel, and the involution operator is a 3×3 involution operator.

[0017] A further technical solution is that the number of feature layers is four, namely, a first feature layer, a second feature layer, a third feature layer, and a fourth feature layer connected in sequence.

[0018] A further technical solution is as follows: the neural network HRNet-In is a pre-trained HRNet-In network. By obtaining student classroom videos, a training set is obtained based on the student classroom videos, and the HRNet-In network is trained based on the training set to obtain a pre-trained HRNet-In network.

[0019] A method for evaluating student posture, based on the aforementioned neural network HRNet-In, includes evaluation steps, which include: obtaining images containing student postures, inputting them into the trained neural network HRNet-In, and obtaining corresponding classroom behaviors.

[0020] A further technical solution includes a network training step, which includes: obtaining classroom videos, dividing and obtaining a training set based on the classroom videos, training an SSD object detection network to obtain a student object detection model, obtaining the bounding box location information of student objects in the training set, and training a neural network HRNet-In for human pose recognition to obtain a trained HRNet-In network, i.e., a student human pose model. In the evaluation step, classroom videos are obtained, images containing student poses are extracted from the classroom videos, student object detection results are obtained based on the images containing student poses, student object bounding boxes are established, and the corresponding classroom behaviors are labeled with the student object bounding boxes through the student human pose model.

[0021] A further technical solution includes a verification step. In the network training step, a test set is obtained based on the classroom video segmentation. The verification step includes: extracting images containing student postures from the test set, obtaining student target detection results based on the images containing student postures, establishing student target bounding boxes, and marking the corresponding classroom behaviors with the student target bounding boxes using the student human posture model.

[0022] An apparatus for student posture assessment includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the corresponding steps in the method described above.

[0023] An apparatus for student posture assessment includes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the corresponding steps in the method described above.

[0024] A student posture assessment device includes a camera, a computer, and an assessment module. The camera is connected to and communicates with the computer, and the assessment module runs on the computer to form a student posture assessment system. The assessment module is used to capture classroom video from the camera and send it to the computer. The computer receives the classroom video from the camera, extracts images containing student postures from the classroom video, and inputs them into the neural network HRNet-In to obtain the corresponding classroom behavior.

[0025] The beneficial effects of adopting the above technical solution are as follows:

[0026] First, a neural network for student posture assessment, HRNet, is an improved version of HRNet-In. Based on the original HRNet network, the 3×3 convolutional kernels in the bottleneck layer of the original HRNet network are replaced with 3×3 involution operators, resulting in the improved HRNet-In network. This technical solution, by replacing the 3×3 convolutional kernels with 3×3 involution operators, achieves faster speed and higher accuracy in recognizing student classroom behavior.

[0027] Second, a method for student posture assessment, based on the aforementioned neural network HRNet-In, includes an assessment step: obtaining an image containing student postures, inputting it into the trained neural network HRNet-In, and obtaining the corresponding classroom behavior. This technical solution, by replacing the 3×3 convolution kernel with a 3×3 involution operator, achieves faster and more accurate recognition of student classroom behavior.

[0028] Third, a device for student posture assessment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the corresponding steps in the aforementioned method. This technical solution, by replacing the 3×3 convolution kernel with a 3×3 involution operator, enables faster and more accurate recognition of student classroom behavior.

[0029] Fourth, a device for student posture assessment includes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the corresponding steps in the above-described method. This technical solution, by replacing the 3×3 convolution kernel with a 3×3 involution operator, enables faster and more accurate recognition of student classroom behavior.

[0030] Fifth, a student posture assessment device includes a camera, a computer, and an assessment module. The camera is connected to and communicates with the computer, and the assessment module runs on the computer, forming a student posture assessment system. The assessment module is used to capture classroom video from the camera and send it to the computer. The computer receives the classroom video from the camera, extracts images containing student postures from the video, and inputs them into the aforementioned neural network HRNet-In to obtain the corresponding classroom behavior. This technical solution, by replacing the 3×3 convolution kernel with a 3×3 involution operator, achieves faster and more accurate recognition of student classroom behavior.

[0031] See the detailed implementation section for further description. Attached Figure Description

[0032] Figure 1 This is a flowchart of Embodiment 4 of the present invention;

[0033] Figure 2 This is a model diagram of the original HRNet network;

[0034] Figure 3 This is a model diagram of the improved HRNet-In network;

[0035] Figure 4a These are images from the dataset;

[0036] Figure 4b These are images from the dataset;

[0037] Figure 4c These are images from the dataset;

[0038] Figure 4d These are images from the dataset;

[0039] Figure 4e These are images from the dataset;

[0040] Figure 4f These are images from the dataset;

[0041] Figure 4g These are images from the dataset;

[0042] Figure 4h These are images from the dataset;

[0043] Figure 4i These are images from the dataset;

[0044] Figure 5 These are images of the SSD experiment results;

[0045] Figure 6 This is a block diagram illustrating the principle of the Involution operator;

[0046] Figure 7 It is a mapping that replaces the Involution operator;

[0047] Figure 8 It is a data plot for predicting the heatmap;

[0048] Figure 9 It is a heatmap of the prediction results;

[0049] Figure 10 These are images of key points on the human body;

[0050] Figure 11 This is an image showing the experimental results of HRNet-In;

[0051] Figure 12a These are images of the experimental results of the traditional OpenPose algorithm;

[0052] Figure 12b The image shows the experimental results of the SSD+HRNet-In algorithm.

[0053] Figure 13 It is an infographic of key points of the human skeleton;

[0054] Figure 14a This is an image of the final experimental results;

[0055] Figure 14b This is an image of the final experimental results;

[0056] Figure 15 This is the flowchart for R&D Example 1;

[0057] Figure 16 This is the flowchart for R&D Example 2. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this application or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0059] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0060] Example 1:

[0061] like Figure 3 As shown, this invention discloses an improved neural network HRNet-In for student posture evaluation. Based on the original HRNet network, the 3×3 convolution kernel in the bottleneck layer of the original HRNet network is replaced with a 3×3 involution operator to obtain the improved HRNet-In network.

[0062] The HRNet-In network includes a feature layer, which includes a bottleneck layer. The bottleneck layer includes a first convolution kernel, an involution operator, and a second convolution kernel connected in sequence. The first convolution kernel is a 1×1 convolution kernel, the second convolution kernel is a 1×1 convolution kernel, and the involution operator is a 3×3 involution operator.

[0063] The number of feature layers is four, namely the first feature layer, the second feature layer, the third feature layer and the fourth feature layer connected in sequence.

[0064] Example 2:

[0065] This invention discloses a neural network HRNet for student posture evaluation, which is a pre-trained HRNet-In network. By obtaining student classroom videos, a training set is obtained based on the student classroom videos, and the improved neural network HRNet-In of Example 1 is trained based on the training set to obtain the pre-trained HRNet-In network.

[0066] Example 3:

[0067] This invention discloses a method for evaluating student posture, including evaluation steps. The evaluation steps include obtaining an image containing student postures, inputting it into a neural network HRNet-In, and obtaining corresponding classroom behaviors. The neural network HRNet is a pre-trained HRNet-In network, i.e., the HRNet neural network in Example 2.

[0068] Example 4:

[0069] like Figure 1 As shown, the present invention discloses a method for student posture assessment, including a network training step and an assessment step. The network training step includes obtaining a classroom video, dividing and obtaining a training set based on the classroom video, training an SSD object detection network to obtain a student object detection model, obtaining the bounding box position information of student objects in the training set, and training a neural network HRNet-In for human posture recognition to obtain a trained HRNet-In network, i.e., a student human posture model.

[0070] The evaluation steps include obtaining classroom videos, extracting images containing student postures from the classroom videos, obtaining student target detection results based on the images containing student postures, establishing student target bounding boxes, inputting the images containing student postures into the student human posture model to obtain corresponding classroom behaviors, and labeling the corresponding classroom behaviors with the student target bounding boxes.

[0071] Example 5:

[0072] like Figure 15 As shown, the present invention discloses a method for student posture assessment, which includes the steps of training a network, evaluation, and verification. The network training step includes obtaining classroom videos, dividing and obtaining training and test sets based on the classroom videos, training an SSD object detection network based on the training set to obtain a student object detection model, obtaining the bounding box position information of student objects in the training set, and training a neural network HRNet-In for human posture recognition to obtain a trained HRNet-In network, i.e., a student human posture model.

[0073] The evaluation steps include extracting images containing student poses from the training set, obtaining student target detection results based on the images containing student poses, establishing student target bounding boxes, inputting the images containing student poses into the student human posture model to obtain corresponding classroom behaviors, and labeling the corresponding classroom behaviors with the student target bounding boxes.

[0074] The verification steps include extracting images containing student poses from the test set, obtaining student target detection results based on the images containing student poses, establishing student target bounding boxes, and labeling the corresponding classroom behaviors with the student target bounding boxes using the student human pose model.

[0075] Example 6:

[0076] The present invention discloses a device for student posture assessment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory and the processor form an electronic terminal, and the processor executes the computer program to implement the steps of Embodiment 3.

[0077] Example 7:

[0078] The present invention discloses a device for student posture assessment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory and the processor form an electronic terminal, and the processor executes the computer program to implement the steps of Embodiment 4.

[0079] Example 8:

[0080] The present invention discloses a device for student posture assessment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory and the processor form an electronic terminal, and the processor executes the computer program to implement the steps of Embodiment 5.

[0081] Example 9:

[0082] The present invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in Embodiment 3.

[0083] Example 10:

[0084] The present invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in Embodiment 4.

[0085] Example 11:

[0086] The present invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in Embodiment 5.

[0087] Example 12:

[0088] This invention discloses a system for evaluating student posture, including a camera, a computer, and an evaluation module. The camera is connected to and communicates with the computer, and the evaluation module runs on the computer.

[0089] The evaluation module is used to capture classroom video from the camera and send it to the computer. The computer receives the classroom video from the camera, extracts images containing student postures from the video, and inputs them into the trained neural network HRNet-In to obtain the corresponding classroom behaviors.

[0090] Compared to the above embodiments, the program module can also be a hardware module made using existing logic operation technology to implement the corresponding logic operation steps, communication steps and control steps, thereby realizing the above-mentioned corresponding steps. The logic operation unit is existing technology and will not be described in detail here.

[0091] Research and development process:

[0092] The invention lies in replacing the 3×3 conv convolution kernel in the bottleneck layer of the original HRNet network structure with the Involution operator, adding a Maxpool2d downsampling to the bottleneck layer in the first feature layer, and then fusing the SSD deep learning object detection algorithm with the improved HRNet-In algorithm.

[0093] 1. The most fundamental technical problem to be solved

[0094] This study aims to address the problems of slow recognition speed, low accuracy, and large recognition models in existing methods for identifying student classroom behavior, and to improve the efficiency of student action recognition in the classroom.

[0095] 2 core technology solutions

[0096] like Figure 1 As shown, this invention proposes a video-based method for recognizing student classroom behavior, the implementation process of which is as follows:

[0097] like Figure 2 The diagram shows the original HRNet algorithm model. The original HRNet network consists of four feature layers: the first feature layer, the second feature layer, the third feature layer, and the fourth feature layer. Each feature layer contains a different number of bottleneck layers: the first feature layer contains one bottleneck layer, the second feature layer contains two bottleneck layers, the third feature layer contains three bottleneck layers, and the fourth feature layer contains four bottleneck layers. Each bottleneck layer consists of one 1×1 convolutional kernel, one 3×3 convolutional kernel, and one 1×1 convolutional kernel.

[0098] like Figure 3The diagram shows the HRNet-In network model obtained by improving the HRNet human pose estimation and detection algorithm according to this invention. The improved HRNet-In network replaces the 3×3 convolutional kernels in the bottleneck layer of the original HRNet structure with 3×3 involution operators to reduce the computational cost of the HRNet network while improving feature extraction capabilities. In the original HRNet structure, a 256×192×3 student classroom image becomes 64×48×64 after being output from the bottleneck layer and input to the subsequent second feature layer. By replacing the 3×3 convolutional kernels in the bottleneck layer with 3×3 involution operators, since the involution operators do not change the size or channels, a Maxpool2d downsampling is added to the bottleneck layer of the first feature layer to ensure that the final image size input to the second feature layer is the same.

[0099] 3. Beneficial technical effects

[0100] This invention provides a lightweight student behavior detection method based on an improved HRNet. The improved HRNet-In model is 15% smaller than the original HRNet model, with a 18.3% reduction in computational FLOPs and a 2.5% improvement in average accuracy (mAP). Finally, fusing the SSD model with the HRNet-In model reduces the size by 67% and increases the detection speed by 30% compared to the original SSD+HRNet model.

[0101] 4 Inventive Concept

[0102] Existing methods for recognizing student actions in the classroom typically combine face recognition with posture recognition, but these methods suffer from low accuracy, low efficiency, and large recognition models. Therefore, this invention proposes a method for recognizing student actions in the classroom that integrates the target detection algorithm SSD and the human posture recognition algorithm HRNet. Furthermore, the traditional HRNet algorithm is improved into the HRNet-In algorithm, achieving fast and effective recognition of student actions in the classroom.

[0103] R&D Example 1: Obtaining a dataset to train an SSD object detection network

[0104] like Figure 15As shown, this invention proposes a video-based method for recognizing student classroom behavior. The implementation process includes the following steps: First, a classroom video production dataset is acquired. Then, an SSD object detection network is trained to obtain a student object detection model. The bounding box position information of student objects in the dataset is obtained. Next, an HRNet-In human pose recognition network is trained to obtain a student human pose model. Multiple human pose data are obtained. Finally, the test set in the dataset is used for image extraction. Based on the student object detection results, student object bounding boxes are established in the extracted images. Through the student human pose model, each student bounding box is labeled with the corresponding classroom behavior.

[0105] like Figures 4a to 4i As shown, the dataset of this invention comes from 80 classroom videos (approximately 40 minutes each) from 38 courses across 13 different subjects in 6 classrooms within a university, totaling 30,000 classroom images. The dataset is divided into a training set and a test set in an 8:2 ratio. The training set contains 64 videos and 24,000 classroom images, while the test set contains 16 videos and 6,000 classroom images. The training and test datasets are independent of each other and do not overlap.

[0106] The experimental environment for this invention uses an Intel i7-8700 2.20GHz CPU, an RTX-2060 12GB GPU, Ubuntu 20.04 as the operating system, PyCharm as the editor, Python as the programming language, and PyTorch 1.8 and Torchvision 0.8.1 as the deep learning libraries. An existing SSD object detection network is deployed on this hardware platform for training to obtain a student object detection model.

[0107] Since the student targets in the video are relatively small, SSD target detection has a relative advantage in detecting small targets. It can accurately identify students sitting in the back row who are relatively far from the monitoring camera. SSD, Single Shot MultiBoxDetector, is a single-shot detector for multiple categories. It adds convolutional feature layers to the underlying VGG conv4-3 network. These convolutional feature layers gradually decrease in size, enabling SSD to predict targets of different sizes at multiple scales. This feature can improve the localization of student targets in classroom videos. SSD uses different convolutional models for prediction for each feature layer in order to extract features with higher semantic meaning. However, the number of bounding boxes set for different feature layers is different. The size of the bounding box includes two parts: scale and aspect ratio. The size sk of the bounding box needs to follow the linear increasing rule. The size of the bounding box increases linearly as the size of the feature map decreases. The ratio of the bounding box is calculated as shown in formula (1):

[0108]

[0109] In equation (1), smin is 0.2, indicating that the scale of the lowest layer is 0.2; smax is 0.9, indicating that the scale of the highest layer is 0.9. In this invention, the SSD algorithm is calculated for classroom videos. First, PriorBox is used to match the borders in the label boxes. Then, positive and negative samples are selected according to the degree of overlap to obtain the true values ​​of classification and offset. After selecting positive and negative samples, the corresponding samples need to be extracted from the deep network, and the classification prediction value and offset prediction value are calculated. The loss of classification and offset is calculated with the true value.

[0110] This invention uses SSD (Structured Target Segmentation) to calibrate the bounding boxes corresponding to students in classroom videos. Each student unit in the video is assigned default boxes of different sizes, and the bounding boxes are predicted based on these default boxes, thereby reducing the training difficulty of the student object detection model to some extent. For example, in the first feature prediction layer, it is set to detect relatively small targets. As the level of abstraction increases, later feature prediction layers are used to detect larger targets. SSD has a total of six feature layers: Conv7, Conv8_2, Conv9_2, Conv10_2, and Conv4_3, extracting six feature layers in total. Conv11_2 is used as the feature layer for detection. The student classroom video is used to predict targets of different sizes in these six prediction layers. This invention uses a training set to train the SSD object detection network, obtaining a student object detection model, and representing the corresponding student location information in the form of bounding boxes.

[0111] To verify the stability of the SSD object detection network model, this invention uses the following loss function for verification: the squared loss function is used to regress and predict the experimental results of the Vision Transformer algorithm. The squared loss function includes two parts: the cumulative mean square error and the error w. The formula of the squared loss function is as follows (2):

[0112]

[0113] In equation (2), c represents the confidence level; l represents the predicted bounding box; g represents the true bounding box; L conf (x,c) represents the confidence loss; L loc This indicates positioning loss.

[0114] As shown in Table 1, during the training of the SSD object detection network model, the loss value of the training set gradually stabilized after 150 epochs, proving that the network training was good. This indicates that the SSD object detection network model has been completed, as shown in the loss function graph.

[0115] Table 1: SSD Training Loss Function Table

[0116] Number of iterations 0 20 40 50 100 150 200 250 300 Loss function value 0.56 0.34 0.15 0.12 0.1 0.96 0.93 0.86 0.84

[0117] like Figure 5 The figure shown is an experimental result diagram of the SSD object detection network model. The SSD network used in this invention is the same as the one used in this invention. The experimental results show that the SSD object detection network model can identify the position of the student in the video very well and represent the student's position as a bounding box.

[0118] Research and Development Example 2: Training the HRNet-In human pose recognition network to obtain a student's human pose model

[0119] like Figure 16 As shown, this invention improves the HRNet-In network model based on the HRNet human pose estimation and detection algorithm. HRNet is based on heatmaps, predicting a heatmap for each keypoint of the human body, i.e., predicting the score of the keypoint at each location. The higher the score, the greater the probability of predicting the keypoint. The HRNet network maintains high resolution throughout the human pose detection process, but high-resolution features have the problem of insufficient receptive field. Therefore, the HRNet network obtains more global information by progressively converting high-resolution features to low-resolution features. On this basis, high-resolution and low-resolution information are exchanged through a feature fusion module. Low-resolution information can enhance the learning of high-resolution features, while high-resolution features that focus on local information are used to enhance the learning of global low-resolution features.

[0120] like Figure 2 As shown in the diagram, the HRNet network structure is as follows: the first path of the HRNet network is high resolution, the second path has half the resolution and double the width, the third path has half the resolution of the second path, and the fourth path has half the resolution of the third path. The HRNet network consists of these four parallel high-resolution and low-resolution networks, and multi-scale fusion is performed between the multi-resolution networks. This results in a huge number of parameters and computational cost for the HRNet network. In order to solve this problem and achieve a lightweight student human pose estimation model and improve detection efficiency, this invention adopts the method of replacing some convolutional layers of HRNet with the Involution operator.

[0121] The involution operator possesses channel invariance and spatial specificity. Within the cube of the feature tensor X, involution kernels are based on individual pixel relationships. Each involution kernel is specifically tailored to a pixel Xi,j at coordinates (i,j), but is shared across channels. The parameters of the involution kernel are H×W×K×K×G, where H represents height, W represents convolution degree, G represents the number of identical kernels shared by each group, and K represents the kernel size.

[0122] As shown in Table 2, the Involution algorithm program.

[0123] Table 2: Involution Algorithm Program

[0124]

[0125] like Figure 6 As shown, the principle of the Involution operator is as follows: First, select the eigenvector of a point in the feature map, apply the function φ to the eigenvector, and then pass through FC-BN-RELU-FC to obtain 1×1×K. 2 Parameters, 1×1×K 2 The parameters are rearranged into a K×K×1 convolution kernel. The K×K×1 convolution kernel is then multiplied by the feature vector of the neighborhood of the point to obtain a K×K×C feature map. The K×K×C features are then summed to obtain the final output feature map.

[0126] The kernel generation function is labeled φ, and the function mapping at each position (i, j) is abstracted to the pixel H at that point. ij The calculation formula is shown in equation (3) below:

[0127]

[0128] In equation (3), ψ ij It is a set of indices of the neighborhood of coordinate (i, j). This indicates that the feature map contains X. i,j A part of it. Size and X i,j It depends on the dimension, generating H corresponding to each pixel. ij It can also be expressed as shown in the following formula (4):

[0129] H ij =W 1σ (w0X ij (4)

[0130] In equation (4), W1σ w0 and w0 are two linear transformations that constitute the bottleneck, and σ represents the nonlinear activation function after normalization (BatchNormalization).

[0131] The formula for calculating Involution is shown in equation (5):

[0132] Y ijk =∑ u,v∈ΔK H i,j,u+[K / 2],v+[K / 2],[kG / C] X i+u,j+v,k (5)

[0133] In equation (5), H iju X is the Involution kernel, C represents the input feature of the convolution, and G represents the number of channels shared by each group.

[0134] The Involution operator reduces computational cost and provides better feature extraction capabilities. Therefore, it replaces some convolutional layers in HRNet with the Involution operator, making the network more lightweight and improving computational efficiency. The original HRNet network consists of four feature layers: the first feature layer, the second feature layer, the third feature layer, and the fourth feature layer. Each feature layer contains a different number of bottleneck layers: the first feature layer contains one bottleneck layer, the second feature layer contains two bottleneck layers, the third feature layer contains three bottleneck layers, and the fourth feature layer contains four bottleneck layers. Each bottleneck layer consists of one 1×1 convolutional kernel, one 3×3 convolutional kernel, and one 1×1 convolutional kernel. The improved HRNet-In network of this invention replaces the 3×3 convolutional kernels in the bottleneck layer of the original HRNet structure with 3×3 involution operators to reduce the computational cost of the HRNet network while improving feature extraction capabilities. In the original HRNet structure, a student classroom image of size 256×192×3 becomes 64×48×64 after being output from the bottleneck layer and then input to the subsequent second feature layer. By replacing the 3×3 convolutional kernels in the bottleneck layer with 3×3 involution operators, since the involution operators do not change the size or channels, a Maxpool2d downsampling is added to the bottleneck layer of the first feature layer to ensure that the final image size input to the second feature layer is the same.

[0135] like Figure 2As shown, the HRNet structure contains many bottleneck structures. This invention replaces the 3×3 convolution in the bottleneck layer with a 3×3 involution operator.

[0136] like Figure 3 As shown, the improved HRNet network is referred to as the HRNet-In network.

[0137] like Figure 7 As shown, the 1×1 convolution in the first layer of Involution still performs dimensionality reduction on the input channels, while the 1×1 convolution in the third layer is a dimensionality increase operation, which aims to expand the dimensionality-reduced features to the required size. The dimensionality reduction and dimensionality increase operations of the two convolution layers enable the network to maintain a strong learning ability even when the number of layers is very deep.

[0138] The improved HRNet-In network predicts key points in the human skeleton by using heatmaps to predict the coordinates of key points.

[0139] like Figure 8 As shown, assuming a predicted heatmap for a certain key point, the point with coordinates (3, 3) can be found by finding the maximum score. Then, the scores on the left and right sides (x direction) and the top and bottom sides (y direction) of this point are compared respectively.

[0140] like Figure 10 As shown, the score on the right side is significantly larger than that on the left side, so the final predicted x-coordinate is shifted to the right by 0.25, hence the final x = 3.25. Similarly, the score on the upper side is larger than that on the lower side, so the y-coordinate is shifted upward by 0.25, hence the final y = 2.75.

[0141] like Figure 9 As shown, the heatmap for each key point of the human body is obtained through HRNet-In network operation. The predicted heatmaps for the joints of the left and right eyes are given. The position with the highest score is found in the heatmap and then mapped back to the original image to obtain the prediction information.

[0142] like Figure 10 As shown in the visualization, each predicted skeletal key point is mapped onto the original image.

[0143] HRNet-In network prediction uses a heatmap for each keypoint. It first divides the coordinates of each keypoint on the human body by 4 before calculating the total loss. However, instead of simply summing the losses for each keypoint, the total loss is multiplied by a different weight before summing.

[0144] Table 3 shows the name of each key point in the HRNet-In network and its corresponding weight.

[0145] Table 3: Weights corresponding to key points

[0146]

[0147] To verify the effectiveness of the improved HRNet-In network, it was tested and trained on a self-made dataset.

[0148] As shown in Table 4, the experimental results show that the improved HRNet-In model is 15% smaller than the original HRNet model, with 18.3% fewer FLOPs and a 2.5% higher average accuracy (mAP). The experimental results verify the feasibility of replacing some convolutional layers of HRNet with Involution. The improved HRNet-In algorithm achieves a lighter network while maintaining the same accuracy as the original HRNet, and in fact, even increases it.

[0149] Table 4: HRNet-In Experimental Data

[0150]

[0151] like Figure 11 The image shown is a result of HRNet-In identifying the human skeleton of students in a classroom.

[0152] The improved HRNet-In single-person human pose estimation algorithm was fused with the SSD human target detection algorithm, and then compared with the unimproved SSD+HRNet algorithm in experiments.

[0153] As shown in Table 5, the experimental results show that after the SSD model and the HRNet-In model are fused, the size of the original SSD+HRNet model is reduced from 303Mb to 98.4Mb, a reduction of 67%, and the detection speed is increased from 23FPS of SSD+HRNet to 30FPS of SSD+HRNet-In, an improvement of 30%.

[0154] Table 5: Experimental Results of the Fusion Algorithm

[0155]

[0156] To make the comparison more obvious, the improved SSD+HRNet-In was compared with the traditional OpenPose algorithm.

[0157] like Figure 12a The image shown is a result of the OpenPose experiment, which shows instances where key points are misconnected between different individuals.

[0158] like Figure 12b The experimental results of the proposed SSD-M3-SFP+HRNet-In algorithm are shown in the figure.

[0159] The experimental results are compared below:

[0160] It is evident that the traditional OpenPose algorithm incorrectly connects the left and right shoulder keypoints of the student on the left in the second row with the left and right eye keypoints of the student in the first row. However, the experimental results of the algorithm in this paper do not show any incorrect connections. This demonstrates that the human pose estimation and detection approach can effectively reduce the probability of incorrect connections between skeletal keypoints of different individuals.

[0161] This lightweight student behavior detection method based on an improved HRNet is deployed on the chosen hardware platform to identify student behaviors. The invention focuses on identifying four classroom behaviors: raising hands, sleeping, using a mobile phone, and writing. After SSD student target detection and HRNet-In human skeletal keypoint detection, the skeletal keypoints of each student can be identified. Then, student actions are distinguished directly based on the features of these keypoints. This invention uses the direct coordinate method of skeletal keypoints to identify student behaviors, comparing the position coordinates of keypoints such as the student's head, wrists, and arms to determine the behavior.

[0162] like Figure 13 As shown, the key information of the human skeleton is as follows.

[0163]

[0164] For example, when the key point of the human hand is higher than the key point of the shoulder, it can be determined as raising the hand; when the key point of the human head is lower than the key point of the shoulder, it can be determined as sleeping; when the key points of the human hands do not cross and the key point of the head tends to be arc-shaped, it can be determined as writing; when the key points of the human hands do not appear and the key point of the head tends to be straight, it can be determined as playing with a mobile phone; since students' postures are not consistent, different postures of the same action can be determined by setting the relationship between multiple human skeletal key points.

[0165] like Figure 14a and Figure 14b The diagram shown is a schematic representation of the final result of the method of the present invention.

[0166] After this application had been running internally for a period of time, the beneficial aspects reported by on-site technicians were:

[0167] In response to existing methods, this invention proposes a lightweight student behavior detection method based on an improved HRNet, which offers faster detection speed, higher accuracy, and greater ease of use. Furthermore, while current research on classroom behavior detection primarily focuses on identifying behaviors such as raising hands, sleeping, and writing, this invention adds the recognition of mobile phone use.

Claims

1. A method for assessing student posture, characterized in that: HRNet, a neural network based on student posture evaluation, includes the steps of training and evaluating the network. The neural network HRNet for student posture evaluation is an improved neural network HRNet-In. Based on the original HRNet network, the 3×3 convolutional kernels in the bottleneck layer of the original HRNet network are replaced with 3×3 involution operators to obtain the improved HRNet-In network. The HRNet-In network includes a feature layer, which includes a bottleneck layer. The bottleneck layer includes a first convolutional kernel, an involution operator, and a second convolutional kernel connected in sequence. The first convolutional kernel is a 1×1 convolutional kernel, the second convolutional kernel is a 1×1 convolutional kernel, and the involution operator is a 3×3 involution operator. The training network steps include: obtaining classroom videos, dividing and obtaining training sets based on classroom videos, training the SSD object detection network to obtain a student object detection model, obtaining the bounding box position information of student objects in the training set, and training the HRNet-In neural network for human pose recognition to obtain the trained HRNet-In network, i.e., the student human pose model. The evaluation steps include: obtaining classroom videos, extracting images containing student poses from the classroom videos, obtaining student target detection results based on the images containing student poses, constructing student target bounding boxes, inputting the images containing student poses into the neural network HRNet-In to obtain corresponding classroom behaviors, and labeling the student target bounding boxes with the corresponding classroom behaviors.

2. The method for student posture assessment according to claim 1, characterized in that: The number of feature layers is four, namely the first feature layer, the second feature layer, the third feature layer and the fourth feature layer connected in sequence.

3. The method for evaluating student posture according to claim 1, characterized in that: The neural network HRNet-In is a pre-trained HRNet-In network. It is obtained by acquiring student classroom videos, obtaining a training set based on the student classroom videos, and training the HRNet-In network based on the training set.

4. The method for evaluating student posture according to claim 1, characterized in that: It also includes a verification step. In the network training step, a test set is obtained based on the classroom video segmentation. The verification step includes: extracting images containing student postures from the test set, obtaining student target detection results based on the images containing student postures, establishing student target bounding boxes, and labeling the corresponding classroom behaviors with the student target bounding boxes through the student human posture model.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the corresponding steps in the student posture assessment method according to any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the corresponding steps in the student posture assessment method according to any one of claims 1 to 4.

7. A device for evaluating student posture, characterized in that: The system includes a camera, a computer, and an evaluation module. The camera is connected to and communicates with the computer, and the evaluation module runs on the computer to form a student posture evaluation system. The evaluation module is used to obtain classroom video from the camera and send it to the computer. The computer obtains the classroom video from the camera, extracts images containing student postures from the classroom video, and inputs them into the neural network HRNet-In of the student posture evaluation method according to any one of claims 1 to 3 to obtain the corresponding classroom behavior.

Images