A multi-person human posture estimation method and system based on knowledge distillation

Abstract

The invention discloses a multi-person human posture estimation method and system based on knowledge distillation, which belongs to a human posture estimation method in the field of computer vision technology. The invention aims to solve the problem that a student network constructed by stacking hourglass networks in the existing knowledge distillation technology is difficult to be made very small and avoid the degradation of model performance due to the reduction of network model complexity. The student network model includes an encoder, a decoder and a joint point regressor, the encoder is a GhostNet convolution layer, the decoder is a stacked lightweight upsampling module, and the joint point regressor is a convolution with a convolution kernel of 1×1; after data is input into a teacher network and a student network, a corresponding heat map is obtained, and a joint point bias is generated by using the student heat map and a target joint point heat map of a data label to dynamically adjust the knowledge transfer from the teacher network to the student network; the acquired real-time human posture data is input into the student network, a real-time joint point heat map is output, and the joint point heat map is converted into real joint point coordinates.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and relates to a method for estimating human body posture, and more specifically to a method for estimating human body posture based on a knowledge distillation method. Background Art

[0002] Human pose estimation (HPE) aims to obtain the coordinates of human pose joints from image input, which is one of the hot issues in the field of computer vision. In recent years, human pose estimation technology has developed rapidly and has been widely used in human tracking, action recognition, action detection and human-computer interaction.

[0003] AlexNet's success in the field of image classification and recognition has pushed computer vision research into the era of deep learning. In 2014, A. Toshev et al. proposed a human pose estimation model DeepPose based on deep learning. This model uses deep convolutional neural networks to achieve global prediction of human pose estimation, marking the entry of human pose estimation into the era of deep learning. With the powerful feature extraction capabilities of deep convolutional networks, human pose estimation has also made great progress. Compared with the method of manually extracting features, the human pose estimation algorithm based on deep learning is more robust. Since then, with the release of the MPII and MSCOCO datasets, a large number of excellent works based on deep learning human pose estimation have emerged.

[0004] With the advancement of science and technology, more and more smart devices are appearing in people's lives, such as self-driving cars, smart surveillance cameras, smart sports aids, and elderly care robots. They all need to have functions such as motion recognition, motion detection, and human-computer interaction. The basis for the realization of these functions is inseparable from the human posture estimation algorithm. It is very difficult to deploy large-scale human posture estimation algorithms on these devices with limited computing resources. Although large-scale human posture estimation algorithms can be deployed to the cloud and use technologies such as cloud computing to enable smart devices to achieve these functions, this can easily cause information leakage and people's privacy security cannot be guaranteed. Therefore, researchers have begun to study lighter and more efficient human posture estimation algorithms, so that the algorithms do not require large GPU hardware and can also run in real time on mobile phones, wearable devices, surveillance cameras, and embedded devices.

[0005] 2D human pose estimation can be divided into two categories, namely single person pose estimation (SPPE) and multi-person pose estimation (MPPE). For multi-person pose estimation, the mainstream methods can generally be divided into top-down and bottom-up methods according to the starting point (abstractness) of the prediction. The top-down method starts from high-level abstraction, first detecting people and generating their positions in the bounding box, and then estimating the pose of each person. The top-down method is more intuitive in thought than the bottom-up method, and is more accurate than the bottom-up method. As for how to detect people, the current mainstream multi-person pose estimation algorithms mostly use common target detection algorithms as human detectors, such as Faster R-CNN, Mask R-CNN, Yolo, etc., and then use single-person pose estimation to generate multi-person poses. How to achieve lightweight multi-person pose estimation, more and more researchers have conducted research on lightweight human pose estimation models. Feng et al. used the knowledge distillation method to extract knowledge into a student hourglass network with a simple network using a complex teacher hourglass network. In addition, more work on lightweight human pose estimation focuses on improving the network structure. Tang et al. proposed a densely connected U-Nets network similar to the hourglass network to achieve high-precision key point positioning. Inspired by the hourglass network, Debnath et al. introduced a novel shunt architecture in the last two layers of MobileNet, reducing the parameters of the model and alleviating overfitting, thereby improving accuracy. Zhang et al. introduced a global attention mechanism and proposed a lightweight bottleneck block to replace the bottleneck block in ResNet, and constructed an LPN with a structure similar to SimpleBaseline. Yu et al. proposed to use channel weighting to replace the point convolution of the channel cleaning module in ShuffleNet, and constructed a Lite-HRNet network with high-resolution feature representation. Ding and Zhang et al. tried to use network space search (Neural Architecture Search, NAS), and respectively constructed a multi-scale feature fusion network HR-NAS similar to HRNet, and EfficientPose similar to SimpleBaseline.

[0006] Knowledge distillation methods for human posture estimation such as FPD and OKDHM are limited by the differences in complexity between the teacher and student networks in knowledge distillation. They mainly use stacked Hourglass networks to construct student networks and teacher networks, and use the network with many stacked layers and large parameters as the teacher network, and the network with few stacked layers and small parameters as the student network, which makes it difficult to make the student network very small. Summary of the invention

[0007] The purpose of the present invention is to solve the problem in the prior art that the student network constructed by stacking hourglass networks is difficult to be made very small and to avoid the degradation of model performance due to the reduction of network model complexity. The present invention provides a multi-person human posture estimation method and system based on knowledge distillation.

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A multi-person human pose estimation method based on knowledge distillation, comprising:

[0010] Obtain human body posture sample data and human body posture sample data labels;

[0011] Construct a pre-trained teacher network model and an untrained student network model. The teacher network model is HRNet-W32, and the student network model includes an encoder, a decoder, and a joint point regressor. The encoder is a GhostNet convolutional layer, the decoder is a stacked lightweight upsampling module, and the joint point regressor is a convolution with a convolution kernel of 1×1.

[0012] The human body posture sample data is input into the pre-trained teacher network to obtain a first output joint point heat map, and the human body posture sample data is input into the student network to obtain a second output joint point heat map. The second output joint point heat map and the target joint point heat map of the human body posture sample data label are used to generate joint point biases, and the joint point biases and the first output joint point heat map are combined to guide the learning of the student network;

[0013] The acquired real-time human posture data is input into the student network, the real-time joint point heat map is output, and the joint point heat map is converted into the real joint point coordinates.

[0014] Preferably, the human body posture sample data label is obtained by a two-dimensional Gaussian function G m (x, y) Calculate the target joint point heat map H m ∈R h×w , where the two-dimensional Gaussian function G m (x, y) is:

[0015]

[0016] H m =G m (x, y)

[0017] Among them, σ is the standard deviation of the Gaussian distribution, w and h are the width and length of the generated heat map, and x m and m are the horizontal and vertical coordinates of the joint points respectively.

[0018] Preferably, the formula for generating joint point offset is:

[0019]

[0020] Where P = {p1, p2, p3, ..., p k} is a hyperparameter, k is the number of human joints, p i As the offset of the corresponding joint point i, Represents the heat map generated by the i-th joint point label, represents the heat map of the student network predicting the i-th joint point; j represents the hyperparameter coefficient, h tar represents the joint point heat map generated by the label, h stu Represents the heat map of joint points generated by the student network;

[0021] for:

[0022]

[0023] Among them, γ represents a hyperparameter.

[0024] Preferably, the teacher network adopts the MSE loss function, specifically:

[0025]

[0026] in, represents the heat map of the teacher network at the i-th joint point, represents the heat map of the student network prediction of the i-th joint point, h tea represents the heat map of the teacher network output, h stu represents the heat map of the student network output, n represents the number of joint points;

[0027] The overall loss function of the teacher network and the student network is:

[0028] L total =MSE(h tea ,h stu )+λD(h tar ,h stu )

[0029] Among them, λ is the hyperparameter coefficient that balances the two loss weights, h tar Heatmap representing label generation.

[0030] A multi-person human pose estimation system based on knowledge distillation, comprising:

[0031] A data acquisition module is used to acquire human body posture sample data and human body posture sample data labels;

[0032] The network model building module is used to build a pre-trained teacher network model and an untrained student network model. The teacher network model is HRNet-W32, and the student network model includes an encoder, a decoder, and a joint point regressor. The encoder is a GhostNet convolutional layer, the decoder is a stacked lightweight upsampling module, and the joint point regressor is a convolution with a convolution kernel of 1×1.

[0033] A network model training module is used to input human posture sample data into a pre-trained teacher network to obtain a first output joint point heat map, input human posture sample data into a student network to obtain a second output joint point heat map, use the second output joint point heat map and the target joint point heat map of the human posture sample data label to generate a joint point bias, and combine the joint point bias and the first output joint point heat map to guide the learning of the student network;

[0034] The heat map real-time module is used to input the acquired real-time human posture data into the student network, output the real-time joint point heat map, and convert the joint point heat map into the real joint point coordinates.

[0035] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0036] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above method.

[0037] The beneficial effects of the present invention are as follows:

[0038] 1. In the present invention, a lightweight human posture estimation network GhostNet is used to generate joint point heat maps; an online joint point knowledge distillation method based on knowledge distillation uses label-generated heat maps and teacher network output as supervision information to guide GhosNet learning; thereby, there is no need to construct a student network by stacking Hourglass networks, and the GhostNet student network can be made very small, and in this process, the model performance will not be reduced due to the reduction of network model complexity, so that the student network can be configured in smart devices such as self-driving cars, smart surveillance cameras, smart motion aids, and elderly robots, greatly improving its application scope and application field.

[0039] 2. In the present invention, in order to improve the performance of the model, an online optimization strategy for joints based on knowledge distillation is proposed. This method uses the output of the GhostNet network and the heat map generated by the label to generate joint biases, and uses the pre-trained large-scale human posture estimation HRNet as the teacher network to "soften" the label information and guide the student network GhostNet to learn; the training of the model is more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the process of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of the present invention;

[0042] Figure 3 It is a structural schematic diagram of the student network model of the present invention;

[0043] Figure 4 It is a schematic diagram of the structure of the three-layer LPB in the present invention. DETAILED DESCRIPTION

[0044] Example 1

[0045] This embodiment provides a method for estimating multi-person human posture based on knowledge distillation, which is used to estimate multi-person human posture. It includes:

[0046] Step S1, obtain sample data

[0047] Obtain human pose sample data and human pose sample data labels. In order to verify the superiority in human pose estimation, the public human pose estimation data MSCOCO is selected for experiments, where the human pose estimation evaluation uses mAP, AP 0.5 、AP 0.75 、AP m 、AP l and AR.

[0048] Step S2: Build a network model

[0049] The network model is a lightweight human posture network model, which includes a pre-trained teacher network model, an untrained student network model, and a matching loss function. The teacher network model uses HRNet-W32, and the student network model includes an encoder, a decoder, and a joint point regressor. The encoder consists of a GhostNet convolution layer, the decoder is a stacked lightweight upsampling module, and the joint point regressor is a convolution with a convolution kernel of 1×1. The data passes through the encoder, decoder, and joint point regressor in sequence. The decoder preferably stacks three layers of LPB, which includes three parts: depth-separable transposed convolution, point convolution, and channel attention module.

[0050] Step S3: training the network model

[0051] The human posture sample data is input into the pre-trained teacher network to obtain the first output joint point heat map, the human posture sample data is input into the student network to obtain the second output joint point heat map, the human posture sample data label is calculated by Gaussian to obtain the target joint point heat map, the second output joint point heat map output by the student network and the target joint point heat map obtained by Gaussian calculation are used to jointly generate the joint point bias, and then the heat map output by the teacher network is used as part of the supervision information, and the jointly generated joint point bias and the first output joint point heat map output by the teacher network are used to guide the learning of the student network, and finally complete the training of the network model.

[0052] When the target joint point heat map is obtained by Gaussian calculation, the human posture sample data label is passed through the two-dimensional Gaussian function G m (x, y) Calculate the target joint point heat map H m ∈R h×w , where the two-dimensional Gaussian function G m (x, y) is:

[0053]

[0054] H m =G m (x, y)

[0055] Among them, σ is the standard deviation of the Gaussian distribution, w and h are the width and length of the generated heat map, and x m and m are the horizontal and vertical coordinates of the joint points respectively.

[0056] When the joint point offset is generated by using the second output joint point heat map and the target joint point heat map, the formula for generating the joint point offset is:

[0057]

[0058] Where P = {p1, p2, p3, ..., p k} is a hyperparameter, k is the number of human joints, p i As the offset of the corresponding joint point i, Represents the heat map generated by the i-th joint point label, represents the heat map of the student network predicting the i-th joint point; j represents the hyperparameter, h tar represents the heat map of label generation, h stu Heatmap representing the output of the student network;

[0059] for:

[0060]

[0061] Among them, γ represents a hyperparameter.

[0062] During training, the teacher network uses the MSE loss function, specifically:

[0063]

[0064] in, represents the heat map of the teacher network at the lth joint point, represents the heat map of the student network prediction of the i-th joint point, h tea represents the heat map of the teacher network output, h stu represents the heat map of the student network output, n represents the number of joint points;

[0065] During training, the overall loss function of the teacher network and the student network is:

[0066] L total =MSE(h tea ,h stu )+λD(h tar ,h stu )

[0067] Among them, λ is the hyperparameter coefficient that balances the two loss weights, h tar Heatmap representing label generation.

[0068] After the network model is trained, only the student network needs to work during testing.

[0069] Step S4, real-time acquisition of joint point coordinates

[0070] The acquired real-time human posture data is input into the student network, the real-time joint point heat map is output, and the joint point heat map is converted into the real joint point coordinates.

[0071] In various heat maps, the larger the value in the heat map, the greater the probability that the joint point exists. Therefore, when converting the joint point heat map into joint point coordinates, it is calculated by the Soft-argmax function. The calculation formula of the Soft-argmax function is as follows:

[0072]

[0073] Among them, β is a hyperparameter to prevent numerical "overflow", h represents the length of the heat map, w represents the width of the heat map, and x m ,y m are the horizontal and vertical coordinates of the joint points, H m Indicates the corresponding m joint, The coordinates corresponding to the joint point m are<h,w> The heat value of (which can be understood as the probability of the existence of the joint point).

[0074] Example 2

[0075] This embodiment also provides a multi-person human posture estimation system based on knowledge distillation, which includes:

[0076] A data acquisition module is used to acquire human body posture sample data and human body posture sample data labels;

[0077] In order to verify the superiority in human pose estimation, the human pose sample data and its labels are selected from the public human pose estimation data MSCOCO for experiments, where the human pose estimation evaluation uses mAP, AP 0.5 、AP 0.75 、AP m 、AP l and AR.

[0078] The network building module is used to build a pre-trained teacher network model and an untrained student network model. The teacher network model uses HRNet-W32, and the student network model includes an encoder, a decoder, and a joint point regressor. The encoder consists of a GhostNet convolution layer, the decoder is a stacked lightweight upsampling module, and the joint point regressor is a convolution with a convolution kernel of 1×1. The data passes through the encoder, decoder, and joint point regressor in sequence. The decoder preferably stacks three layers of LPB, which includes three parts: depth-separable transposed convolution, point convolution, and channel attention module.

[0079] The network training module is used to input human posture sample data into the pre-trained teacher network to obtain the first output joint point heat map, input the human posture sample data into the student network to obtain the second output joint point heat map, obtain the target joint point heat map after Gaussian calculation of the human posture sample data label, use the second output joint point heat map output by the student network and the target joint point heat map obtained by Gaussian calculation to jointly generate joint point biases, and then use the heat map output by the teacher network as part of the supervision information, and jointly generate the joint point biases and the first output joint point heat map output by the teacher network to guide the learning of the student network, and finally complete the training of the network model.

[0080] When the target joint point heat map is obtained by Gaussian calculation, the human posture sample data label is passed through the two-dimensional Gaussian function G m (x, y) Calculate the target joint point heat map H m ∈R k×w , where the two-dimensional Gaussian function G m (x, y) is:

[0081]

[0082] H m =G m (x, y)

[0083] Among them, σ is the standard deviation of the Gaussian distribution, w and h are the width and length of the generated heat map, and x m and m are the horizontal and vertical coordinates of the joint points respectively.

[0084] When the joint point offset is generated by using the second output joint point heat map and the target joint point heat map, the formula for generating the joint point offset is:

[0085]

[0086] Where P = {p1, p2, p3, ..., p k} is a hyperparameter, k is the number of human joints, p i As the offset of the corresponding joint point i, Represents the heat map generated by the i-th joint point label, represents the heat map of the student network predicting the i-th joint point; j is the hyperparameter coefficient (indicates the selection of the first j p i ), h tar represents the heat map of label generation, h stu Heatmap representing the output of the student network;

[0087] for:

[0088]

[0089] Among them, γ represents the joint point penalty hyperparameter.

[0090] During training, the teacher network uses the MSE loss function, specifically:

[0091]

[0092] in, represents the heat map of the teacher network at the i-th joint point, represents the heat map of the student network prediction of the i-th joint point, h tea represents the teacher network heat map, h stu represents the student network heat map, n represents the number of joint points;

[0093] During training, the overall loss function of the teacher network and the student network is:

[0094] L total =MSE(h tea ,h stu )+λD(h tar ,h stu )

[0095] Among them, λ is the hyperparameter coefficient that balances the two loss weights, h tar Heatmap representing label generation.

[0096] After the network model is trained, only the student network needs to work during testing.

[0097] The heat map real-time module is used to input the acquired real-time human posture data into the student network, output the real-time joint point heat map, and convert the joint point heat map into the real joint point coordinates.

[0098] In various heat maps, the larger the value in the heat map, the greater the probability that the joint point exists. Therefore, when converting the joint point heat map into joint point coordinates, it is calculated by the Soft-argmax function. The calculation formula of the Soft-argmax function is as follows:

[0099]

[0100] Among them, β is a hyperparameter to prevent numerical "overflow", h represents the length of the heat map, w represents the width of the heat map, and x m ,y m are the horizontal and vertical coordinates of the joint points, H m represents the heat map of joint point m, Indicates the heat map of m joint points<h,w> Heat map value.

[0101] Example 3

[0102] This embodiment also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned method for multi-person human posture estimation based on knowledge distillation.

[0103] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0104] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code for running the multi-person human posture estimation method based on knowledge distillation, etc. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0105] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the multi-person human posture estimation method based on knowledge distillation.

[0106] Example 4

[0107] This embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned method for multi-person human posture estimation based on knowledge distillation.

[0108] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the multi-person human posture estimation method based on knowledge distillation.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

[0110] Experimental example

[0111] In this experimental example, the above network model is built, and the public human posture estimation data MSCOCO is selected, and the above estimation method is applied to perform posture estimation. Other comparison examples use the same human body detector as other multi-person human posture estimation methods.

[0112] After actual measurement and analysis, the proposed method is compared with existing human pose estimation methods, namely Hourglass, SimpleBaseline, LPN, Lite-HRNet, ShuffleNetV2, DARK, and MobilePoseNet. All methods are compared on MSCOCO data. As shown in the table above, the proposed method has a better performance in terms of AP. 0.5 In terms of performance, the method of this application is higher than other comparative experimental methods, indicating the superiority of the model. In addition, the method of this application achieved an mAP of 68.7 with only 0.6GFLOPs of computation, which is higher than other large models such as Hourglass and CPN. Compared with the lightweight human posture estimation model LPN, the method of this application reduces the computation by 40% while maintaining similar accuracy. Compared with other lightweight networks MobileNetV2, ShuffleNetV2 and Lite-HRNet, the method of this application has a large lead in accuracy and parameter quantity. See Table 1 for details:

[0113] Method Backbone Input #Params GFLOPs AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> <![CDATA[AP M ]]> <![CDATA[AP L ]]> YEAR 8-stage Hourglass Hourglass 256×192 25.6M 26.2 66.9 - - - - - CPN ResNet-50 256×192 27.0M 6.2 68.4 - - - - - SimpleBaseline ResNet-50 256×192 34.0M 8.9 70.4 88.6 78.3 67.1 77.2 76.3 HRNet-W32 ResNet-50 256×192 28.5M 12.4 73.4 89.5 80.7 70.2 80.1 79.8 DARK HRNetV1-W48 128×96 63.6M 3.6 71.9 89.1 79.6 69.2 78 77.9 MobileNetV2 MobileNetV2 256×192 9.6M 1.48 64.6 87.4 72.3 61.1 71.2 70.7 MobileNetV2 1× MobileNetV2 384×288 9.6M 3.33 67.3 87.9 74.3 62.8 74.7 72.9 ShuffleNetV2 ShuffleNetV2 256×192 7.6M 1.28 59.9 85.4 66.3 56.6 66.2 66.4 ShuffleNetV2 1× ShuffleNetV2 384×288 7.6M 2.87 63.6 86.5 70.5 59.5 70.7 69.7 Small HRNet HRNet-W16 256×192 1.3M 0.54 55.2 83.7 62.4 52.3 61 62.1 Small HRNet HRNet-W16 384×288 1.3M 1.21 56 83.8 63 52.4 62.6 62.6 Lite-HRNet Lite-HRNet-18 256×192 1.1M 0.20 64.8 86.7 73 62.1 70.5 71.2 Lite-HRNet Lite-HRNet-18 384×288 1.1M 0.45 67.6 87.8 75 64.5 73.7 73.7 LPN ResNet-50 256×192 2.9M 1.0 69.1 88.1 76.6 65.9 75.7 74.9 MobilePoseNet MobileNetV3 256×192 1.5M 0.55 66.2 87.3 74.2 63.1 72.5 72.4 MobilePoseNet MobileNetV3 384×288 1.5M 1.23 69 88.2 75.9 65.5 75.5 74.9 GhostPoseNet GhostNet 256×192 3.1M 0.26 63.8 88.3 71.6 61.6 67 67.3 GhostPoseNet* GhostNet 384×288 3.1M 0.60 67.9 89.4 74.4 65.3 72.1 71.6 GhostPoseNet GhostNet 384×288 3.1M 0.60 68.7 90.4 76.8 65.7 73.0 71.8

[0114] method BackBone #Params GFLoPs Input Size AP Speed* Speed HRNet HRNetV1-W32 28.5M 7.1 256×192 74.4 7.5 19.2 HRNet HRNetV1-W32 28.5M 16 384×288 75.8 4 18.8 NLite-HRNet-18 HRNet-W16 0.7M 0.19 256×192 62.8 11 18.9 WNLite-HRNet-18 HRNet-W16 1.3M 0.3 256×192 66 12 18.6 ShuffleNetV2 1× ShuffleNetV2 7.6M 1.28 256×192 59.9 17 71.3 ShuffleNetV2 1× ShuffleNetV2 7.6M 2.87 384×288 63.6 10 64.1 MobileNetV2 1× MobileNetV2 9.6M 1.48 256×192 64.6 6.8 83.1 MobileNetV2 1× MobileNetV2 9.6M 3.33 384×288 67.3 4.5 73.1 Lite-HRNet Lite-HRNet-18 1.1M 0.2 256×192 64.8 12 17.4 Lite-HRNet Lite-HRNet-18 1.1M 0.45 384×288 67.6 7.1 16.3 MobilePoseNet MobileNetV3 1.5M 0.55 256×192 66.2 7.8 54.8 MobilePoseNet MobileNetV3 1.5M 1.23 384×288 69.0 5.1 50.8 GhostPoseNet GhostNet 3.1M 0.26 256×192 63.8 9.2 62.0 GhostPoseNet GhostNet 3.1M 0.60 384×288 68.7 6.4 59.4 .

Claims

1. A multi-person human posture estimation method based on knowledge distillation, characterized in that: include: Obtain human body posture sample data and human body posture sample data labels; Construct a pre-trained teacher network model and an untrained student network model. The teacher network model is HRNet-W32, and the student network model includes an encoder, a decoder, and a joint point regressor. The encoder is a GhostNet convolutional layer, the decoder is a stacked lightweight upsampling module, and the joint point regressor is a convolution kernel of Convolution; The human body posture sample data is input into the pre-trained teacher network to obtain a first output joint point heat map, and the human body posture sample data is input into the student network to obtain a second output joint point heat map. The second output joint point heat map and the target joint point heat map of the human body posture sample data label are used to generate joint point biases, and the joint point biases and the first output joint point heat map are combined to guide the learning of the student network; The acquired real-time human posture data is input into the student network, the real-time joint point heat map is output, and the joint point heat map is converted into the real joint point coordinates; The formula for generating joint point offsets is: in, As the corresponding joint point The bias of Indicates Heatmap generated by joint point labels, The student network predicts Heat map of joint points; j represents the hyperparameter coefficient, represents the heat map of label generation, Heatmap representing the output of the student network; for: ； in, represents the hyperparameter coefficient.

2. The method for multi-person human posture estimation based on knowledge distillation according to claim 1, characterized in that: Human body posture sample data labels are obtained through a two-dimensional Gaussian function Calculate the target joint point heat map , where the two-dimensional Gaussian function for: in, is the standard deviation of the Gaussian distribution, and To generate the width and height of the heat map, and are the horizontal and vertical coordinates of the joint points respectively.

3. The method for multi-person human posture estimation based on knowledge distillation according to claim 1, characterized in that: The teacher network uses the MSE loss function, specifically: in, Represents the teacher network in Heatmap of joint points, The student network predicts Heatmap of joint points, represents a heat map of the teacher network output, represents a heat map of the output of the student network, Indicates the number of joint points; The overall loss function of the teacher network and the student network is: in, is the hyperparameter coefficient that balances the two loss weights.

4. A multi-person human posture estimation system based on knowledge distillation, characterized in that: include: A data acquisition module is used to acquire human body posture sample data and human body posture sample data labels; The network model building module is used to build a pre-trained teacher network model and an untrained student network model. The teacher network model is HRNet-W32, and the student network model includes an encoder, a decoder, and a joint point regressor. The encoder is a GhostNet convolutional layer, the decoder is a stacked lightweight upsampling module, and the joint point regressor is a convolution kernel of Convolution; A network model training module is used to input human posture sample data into a pre-trained teacher network to obtain a first output joint point heat map, input human posture sample data into a student network to obtain a second output joint point heat map, use the second output joint point heat map and the target joint point heat map of the human posture sample data label to generate a joint point bias, and combine the joint point bias and the first output joint point heat map to guide the learning of the student network; The heat map real-time module is used to input the acquired real-time human posture data into the student network, output the real-time joint point heat map, and convert the joint point heat map into the real joint point coordinates; The formula for generating joint point offsets is: in, As the corresponding joint point The bias, Indicates Heatmap generated by joint point labels, The student network predicts Heat map of joint points; j represents the hyperparameter coefficient, represents the heat map of label generation, Heatmap representing the output of the student network; for: ； in, represents the hyperparameter coefficient.

5. The multi-person human posture estimation system based on knowledge distillation as claimed in claim 4, characterized in that: Human body posture sample data labels are obtained through a two-dimensional Gaussian function Calculate the target joint point heat map , where the two-dimensional Gaussian function for: in, is the standard deviation of the Gaussian distribution, and To generate the width and height of the heat map, and are the horizontal and vertical coordinates of the joint points respectively.

6. A multi-person human posture estimation system based on knowledge distillation as claimed in claim 4, characterized in that: The teacher network uses the MSE loss function, specifically: in, Represents the teacher network in Heatmap of joint points, The student network predicts Heatmap of joint points, represents a heat map of the teacher network output, represents a heat map of the student network output, Indicates the number of joint points; The overall loss function of the teacher network and the student network is: in, is the hyperparameter coefficient that balances the two loss weights.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 3.

Images