A lightweight human key point detection method and device based on model pruning

Abstract

The application discloses a kind of light-weight human key point detection method and device based on model pruning, comprising the following steps: (1) the original image is convolved and continuously down-sampled, and different scale low-resolution images are obtained;(2) a multi-branch convolutional neural network is constructed, different resolution images are input into branch network respectively, and information fusion layer is set between different branches;(3) a linear classifier is added to each convolution block in the network, and the contribution of each convolution block is calculated using the accuracy of the linear classifier;(4) the contribution of the convolution block is sorted, and the convolution block with low contribution is deleted according to the pruning ratio to obtain a pruned network;(5) the pruned network and the original convolutional neural network are trained together using the knowledge distillation method;(6) the image to be detected is input into the trained pruned network, and the detection result of human key points is obtained.The application can effectively improve the accuracy and reduce the model parameters.

Description

Technical Field

[0001] This invention belongs to the field of human keypoint detection in computer vision, and in particular relates to a lightweight human keypoint detection method and device based on model pruning. Background Technology

[0002] Human key point detection is an important fundamental task in the field of artificial intelligence, which can provide basic technical support for human action recognition, robot training, augmented reality, and behavior capture in human-computer interaction.

[0003] Human keypoints represent the human body as several joints, such as the neck, nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. The human keypoint detection task uses algorithms and models to identify and classify the limb parts and major joints of pedestrians in images or videos to capture a set of coordinates. These coordinates can then be used to describe the human posture of one or more people in images and videos.

[0004] Research on human key point detection algorithms has significant application value in medical diagnosis and rehabilitation, security, human-computer interaction, and autonomous driving. In short, research on human key point detection has important theoretical and practical significance.

[0005] Human keypoint detection tasks are divided into two types based on the number of people in an image: single-person detection and multi-person detection. The former is single-person pose estimation based on predicting human keypoints in a cropped image with a given bounding box, while the latter requires further identification of the full-body poses of all people in an image for multi-person pose estimation. For example, Chinese patent document CN110084161A discloses a fast method and system for detecting human skeletal keypoints, realizing single-person skeletal keypoint detection. Chinese patent document CN109255783A discloses a method for detecting the positional arrangement of human skeletal keypoints in multi-person images.

[0006] The key to single-person pose detection lies in the accuracy of the detection model. As models with various structures are continuously developed and improved, the room for improvement in detection accuracy is becoming increasingly limited. Multi-person keypoint detection is divided into two technical approaches: bottom-up and top-down. Bottom-up algorithms focus on the stitching problem between different parts of multiple people in an image. With in-depth research, various human body model-related algorithms have been proposed, improving detection accuracy and speed. Many of these models have also provided inspiration for human tracking, pose capture, and 3D human detection methods. Top-down algorithms require first detecting individual bounding boxes for each person, and then performing single-person pose estimation for each bounding box. Therefore, many algorithms are based on single-person pose estimation tasks. The differences are: 1) Multi-person estimation needs to consider bounding boxes at different scales, requiring a model capable of performing good calculations for people at multiple scales. For crowded images, the scale span may be even larger, requiring a better algorithm that can meet the detection accuracy requirements at various scales; 2) The model needs to detect every person in an image, making the efficiency of the multi-person keypoint detection model crucial.

[0007] With the emergence of various multi-scale detection models, the accuracy of related tasks has continued to rise. Although there is still room for growth, as the accuracy of models increases and the detection speed becomes slower, research in related fields has begun to focus on the balance between accuracy and model size. In recent years, the issue of lightweighting related models has gradually become a research hotspot. Summary of the Invention

[0008] This invention provides a lightweight human keypoint detection method and device based on model pruning, which enables the network to adaptively adjust the pruning ratio and reduce the number of model parameters, thereby achieving a balance between the accuracy of the detection results and the number of parameters, effectively improving the accuracy and reducing the number of model parameters.

[0009] A lightweight human keypoint detection method based on model pruning includes the following steps:

[0010] (1) Perform a series of convolution operations on the original image and then perform continuous downsampling to obtain low-resolution images of different scales;

[0011] (2) Construct a multi-branch convolutional neural network, input images of different resolutions into the branches of the convolutional neural network respectively, and set up information fusion layers between different branches to enable information from different resolutions to be fused together.

[0012] (3) Add a fully connected layer as a linear classifier to each convolutional block in the convolutional neural network, train and calculate the detection accuracy of the linear classifier; use the accuracy of the linear classifier to calculate the contribution of each convolutional block;

[0013] (4) Sort the contribution of all convolutional blocks and remove the convolutional blocks with low contribution from the network according to the pruning ratio to obtain the pruned network.

[0014] (5) The pruned network and the original convolutional neural network are trained together using the knowledge distillation method;

[0015] (6) Input the image to be detected into the trained pruning network to obtain the detection results of human body key points.

[0016] In step (1), the continuous downsampling specifically includes downsampling by 1 / 2, 1 / 4 and 1 / 8 times.

[0017] In step (2), each branch network consists of 16 stacked convolutional blocks, and each convolutional block consists of a 3×3 convolutional kernel, a normalization function, and an activation function.

[0018] In step (2), setting up an information fusion layer between different branch networks to enable the fusion of information between different resolutions specifically includes:

[0019] Two multi-scale information fusion modules are added between different branch networks. The two information fusion modules first directly upsample and downsample the intermediate results of each branch network in the process; and then directly add the upsampled and downsampled intermediate results of different branch networks into each branch network.

[0020] In step (3), the contribution of the convolutional block is defined as: in, and These represent convolutional blocks b respectively. i and b i-1 The detection accuracy of the corresponding linear classifier.

[0021] In step (4), the number of convolutional blocks that need to be pruned is calculated based on the pruning ratio, and the convolutional blocks with the least contribution are found and directly deleted.

[0022] The specific process of step (5) is as follows:

[0023] (5-1) The original convolutional neural network is used as the teacher model and the pruned network is used as the student model. During the training process, the classroom model and the student model calculate the heat map of each human body key point.

[0024] (5-2) Calculate the loss function of the original convolutional neural network;

[0025] (5-3) Calculate the loss function of the pruned network;

[0026] (5-4) Calculate the total loss function: The formula for calculating the loss function is Loss. total =αLoss stu +βLoss tea Among them, Loss tea It is the loss value of the original network. stu It is the loss of the network after pruning, and α and β are hyperparameters that balance these two losses;

[0027] (5-5) Update model parameters;

[0028] (5-6) Repeat steps (5-2) to (5-5) for training.

[0029] In step (5-1), the pixel values ​​on the heatmap represent the pixel probability of belonging to the key point. During the knowledge distillation process, the heatmap generated by the student model is aligned with the target heatmap of the teacher model.

[0030] Based on the same inventive concept, the present invention also provides a lightweight human key point detection device based on model pruning, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned lightweight human key point detection method.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. This invention constructs a multi-branch convolutional neural network for human key point detection, which can achieve better detection results.

[0033] 2. This invention proposes a pruning algorithm based on convolutional neural networks, which can greatly reduce the model parameters of related types of networks and ensure detection accuracy while controlling the number of model parameters.

[0034] 3. The pruning algorithm based on convolutional block contribution proposed in this invention can dynamically prune networks of different sizes and generate models with different parameter amounts according to parameter constraints, thereby achieving a balance between detection accuracy and model parameters. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the overall network framework according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of the multi-scale fusion module in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the detection head structure of the convolution block in an embodiment of the present invention;

[0038] Figure 4This is a schematic diagram of the model training process of the present invention;

[0039] Figure 5 This is a schematic diagram illustrating the process of training a lightweight model using knowledge distillation. Detailed Implementation

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0041] like Figure 1 As shown, a lightweight human keypoint detection method based on model pruning includes the following steps:

[0042] S01 performs a series of convolution operations on the original image, followed by continuous downsampling, to obtain low-resolution images at different scales.

[0043] In this embodiment of the invention, the input original image has RGB pixels of 3×256×192. After a series of convolution calculations, it becomes 32×64×48. This image is then downsampled three times (by 1 / 2, 1 / 4, and 1 / 8 times) to generate low-resolution images of 64×32×24, 128×16×12, and 256×8×6. A total of four low-resolution images of different scales are obtained.

[0044] S02, construct a multi-branch convolutional neural network, input images of different resolutions into the branches of the convolutional neural network respectively, and set up information fusion layers between different branches to enable the information between different resolutions to be fused together.

[0045] like Figure 2 As shown in the embodiment of the invention, the constructed convolutional neural network has four branches, each branch network corresponding to a low-resolution image of one scale. Each branch network consists of 16 stacked convolutional blocks, each convolutional block consisting of a 3×3 convolutional kernel, a normalization function, and an activation function.

[0046] like Figure 3 As shown, this illustrates the fusion process of pixel information at different scales in the network. In this embodiment, the network performs fusion in three stages, with each stage fusing different branch networks. By performing upsampling and downsampling operations on each branch, the features of the four branches are summed together to generate a four-row branch network. In this way, each branch retains information from both high-resolution and low-resolution networks during training.

[0047] S03, add a fully connected layer as a linear classifier to each convolutional block in the convolutional neural network, train and calculate the detection accuracy of the linear classifier; use the accuracy of the linear classifier to calculate the contribution of each convolutional block.

[0048] like Figure 4 As shown, a fully connected layer is placed after each convolutional block in each branch as a linear classifier, and these classifiers are trained. The contribution of all blocks in the branch is calculated. The contribution of a convolutional block is defined as: in b i The accuracy of the linear classifier corresponding to the block. This algorithm requires sorting the contributions of each convolution and deleting the convolutional block with the smallest contribution based on the pruning rate.

[0049] S04. Sort the contributions of all convolutional blocks and remove the convolutional blocks with low contributions from the network according to the pruning ratio to obtain the pruned network.

[0050] S05 uses a knowledge distillation method to train both the pruned network and the original convolutional neural network.

[0051] S06, input the image to be detected into the trained pruning network to obtain the detection results of human key points.

[0052] like Figure 5 The diagram illustrates the process of training a lightweight model using knowledge distillation. Knowledge distillation treats a pre-trained heavyweight network as a teacher model, transferring the teacher's knowledge to acquire that knowledge and refining it into a lightweight student model. This distillation procedure can be formulated as follows:

[0053] L kd =d(m stu m tea )

[0054] Where d(·) represents the distance loss function, which measures the difference between two probability distributions. stu and m tea These represent the results generated by the student network and the teacher network, respectively.

[0055] For the supervision loss of a specific task, the entire loss function is as follows:

[0056] L total =L task +λL kd

[0057] The λ hyperparameter is added to balance the two loss terms. General knowledge extraction is a two-stage process: first, a cumbersome teacher model is trained and fixed, and then the knowledge is refined into a compact student model. Given a single RGB image, this paper calculates the heatmap of each human keypoint using a heatmap that represents the keypoint location as a Gaussian peak. To train the network, this paper minimizes the heatmap m predicted from each branch. pred Heatmap of actual values ​​m gt Mean squared error (MSE) between:

[0058]

[0059] Where C represents the total number of human keypoints (i.e., heatmap channels). It is the true value from the c-th channel. It is the predicted value from channel c.

[0060] An appropriate distillation loss function is crucial for the entire training process. Since pixel values ​​on the heatmap represent the probability of pixels belonging to keypoints, the heatmap generated by the student model is aligned with the target heatmap during knowledge distillation. The obtained target heatmap acts as the teacher model in our method, teaching each branch model (student). To convey pose structure knowledge, a pixel-wise loss is used to minimize the divergence between the heatmaps of the teacher and student models, as shown below:

[0061]

[0062] W and H represent the width and height of the heatmap. and Let represent the probability of the i-th pixel in the heatmap generated by the teacher model and the student model, respectively. M = {1, 2, ..., W × H} represents all pixels.

[0063] For the proposed method, the entire objective function consists of a traditional mean squared error loss for attitude estimation and another loss term for online knowledge extraction:

[0064] L total =αL mse +βL kl

[0065] α and β are hyperparameters that balance these two types of losses.

[0066] To demonstrate the technical effects and advantages of the present invention, the method proposed in this invention will be applied to a practical example below, and compared with other similar methods.

[0067] This invention was tested on the publicly available MS COCO2017 dataset. The MS COCO2017 dataset contains approximately 200,000 images with different backgrounds and over 250,000 human instances, each with 17 keypoints. It is divided into training, validation, and test sets: Train2017 contains approximately 57,000 images and 150,000 human instances; Val2017 contains approximately 5,000 images; and test-dev2017 contains approximately 20,000 images. This paper trains the model on the COCOtrain2017 dataset.

[0068] The human keypoint detection metric used in this invention is object keypoint similarity (OKS), calculated using the following formula:

[0069]

[0070] Where i represents the labeled joint number; d i σ is the Euclidean distance between the actual ground truth value and the detected keypoint for each keypoint; s represents the size factor of the current person, which is the square root of the area occupied by this person in the ground truth; k represents the normalization factor of the skeleton point, which is obtained by calculating the standard deviation of all ground truth values ​​in the existing dataset, reflecting the degree of influence of the current skeleton point on the overall dataset. A larger value indicates a worse annotation effect for this point in the entire dataset; a smaller value indicates a better annotation effect for this point in the entire dataset. Generally, 2σ is taken, where... v i This is a keypoint visibility flag, with values ​​of 0, 1, and 2. These values ​​represent: 0 means the keypoint does not exist in the actual value; 1 means the keypoint exists but is not visible or is obscured; and 2 means the keypoint exists and can be seen.

[0071] The model evaluation metric used in this method is mean Average Precision (mAP). Specifically, different values ​​are set for the artificial threshold T in the AP metric, resulting in multiple precision indices. Finally, the average of these multiple precision indices is calculated to obtain the final mean precision, as shown in the formula below:

[0072]

[0073] The pruning effect of the algorithm was first verified on the MS COCO2017 validation set, and the results are shown in Table 1.

[0074] Table 1

[0075] Input pixels Pruning rate AP Parameters 256×192 0.00 73.4 28.50M 256×192 0.00 76.1 43.06M 256×192 0.25 76.9 38.25M 256×192 0.40 77.7 32.61M 256×192 0.50 78.9 28.49M 256×192 0.60 74.9 23.10M 256×192 0.75 67.5 19.56M

[0076] During the pruning process, the pruning ratio affects the pruning effect. A larger pruning ratio results in a smaller and lighter model, but at the same time, the model will lose accuracy. This is a trade-off, so this experiment was conducted to observe the impact of different pruning ratios on model size and accuracy to determine the optimal pruning ratio. The experimental results are shown in Table 1. The results show that as the pruning ratio increases, the number of model parameters decreases, and the detection accuracy (measured by AP) initially decreases and then increases, reaching its optimal level at a pruning ratio of 40%. It was also found that at a pruning ratio of 75%, the model accuracy only decreased by 1.2%. At this point, the model's parameter count is significantly lower than the original network, but the detection accuracy is still acceptable. This model size is suitable for deployment on mobile devices. Table 1 demonstrates that this convolutional contribution-based pruning algorithm can achieve a balance between accuracy and parameter count by generating networks of different sizes and flexibly selecting different pruning ratios.

[0077] The present invention was tested on the COCO2017 validation set, and the detection method proposed in this invention was compared with some networks that have achieved better results. The results are shown in Table 2.

[0078] Table 2

[0079] Model Input pixels Parameters GFLOPs AP AP50 <![CDATA[AP 75 ]]> <![CDATA[AP M ]]> <![CDATA[AP L ]]> AE 256×192 - - 56.6 81.8 61.8 49.8 67.0 Mask ResNet-50 256×192 - - 62.7 87.0 68.4 57.4 71.1 CMU-Pose 256×192 - - 64.2 86.2 70.1 61.0 68.8 G-RMI 353×257 42.6M 57.0 64.9 85.5 71.3 62.3 70.0 RMPE 320×256 - 26.7 72.3 89.2 79.1 68.0 78.6 CPN 384×288 - 29.2 72.1 91.4 80.0 68.7 77.2 SimpleBaseline 256×192 - 28.9 70.0 90.9 77.9 66.8 75.8 SimpleBaseline 384×288 68.6M 35.6 73.7 91.9 79.1 70.3 78.1 HRNet 256×192 28.5M 7.7 73.4 89.5 80.7 70.2 80.1 HigherHRNet 256×192 - - 76.1 92.3 83.4 72.2 82.1 TokenPose 384×288 27.5M 22.1 70.5 89.3 77.2 66.6 75.8 TransPose 256×192 17.5M 21.8 75.0 86.2 80.3 72.3 81.1 HRFormer 384×288 43.2M 26.8 70.9 89.0 78.4 67.2 77.8 This invention 256×192 29.8M 5.86 78.9 91.1 81.2 67.1 79.1

[0080] As shown in Table 2, after training the algorithm in this paper and other human pose estimation algorithms that have achieved good results in recent years on the COCO training set, and then testing them on the COCO validation set, this paper will focus on detection speed, model parameter count, and average detection accuracy (AP, AP50, AP75, AP60, AP75, AP80). M AP L The data in Table 2 shows that the proposed method significantly improves detection accuracy compared to traditional convolutional neural network models, and the proposed network outperforms other convolutional neural network-based models.

[0081] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight human keypoint detection method based on model pruning, characterized in that, Includes the following steps: (1) Perform a series of convolution operations on the original image and then perform continuous downsampling to obtain low-resolution images of different scales; (2) Construct a multi-branch convolutional neural network. Input images of different resolutions into the branch networks of the convolutional neural network respectively. Each branch network consists of 16 stacked convolutional blocks. Each convolutional block consists of a 3×3 convolutional kernel, a normalization function and an activation function. An information fusion layer is set up between different branch networks to enable the fusion of information at different resolutions. Specifically, this includes adding two multi-scale information fusion modules between different branch networks. The two information fusion modules first directly upsample and downsample the intermediate results of each branch network during the process. Then, the upsampled and downsampled intermediate results from different branch networks are directly added to each branch network. (3) Add a fully connected layer as a linear classifier to each convolutional block in the convolutional neural network, train and calculate the detection accuracy of the linear classifier; use the accuracy of the linear classifier to calculate the contribution of each convolutional block; (4) Sort the contribution of all convolutional blocks, calculate the number of convolutional blocks to be pruned according to the pruning ratio, and remove the convolutional blocks with low contribution from the network to obtain the pruned network. (5) The pruned network and the original convolutional neural network are trained together using the knowledge distillation method; the specific process is as follows: (5-1) The original convolutional neural network is used as the teacher model and the pruned network is used as the student model. During the training process, the classroom model and the student model calculate the heat map of each human body key point. (5-2) Calculate the loss function of the original convolutional neural network; (5-3) Calculate the loss function of the pruned network; (5-4) Calculate the total loss function The formula for calculating the loss function is: ,in, It is the loss value of the original network. It is the loss value of the network after pruning. and It is a hyperparameter that balances these two losses; (5-5) Update model parameters; (5-6) Repeat steps (5-2) to (5-5) for training; (6) Input the image to be detected into the trained pruning network to obtain the detection results of human body key points.

2. The lightweight human keypoint detection method based on model pruning according to claim 1, characterized in that, In step (1), the continuous downsampling specifically includes downsampling by 1 / 2, 1 / 4 and 1 / 8 times.

3. The lightweight human keypoint detection method based on model pruning according to claim 1, characterized in that, In step (3), the contribution of the convolutional block is defined as: ,in, and Representing convolutional blocks and The detection accuracy of the corresponding linear classifier.

4. A lightweight human key point detection device based on model pruning, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the lightweight human key point detection method according to any one of claims 1-3.

Images