Knowledge distillation-based lightweight human body posture estimation method

Abstract

The invention discloses a lightweight human body posture estimation method based on knowledge distillation, a teacher network and a student network are processed, a lightweight student network is designed, compared with the teacher network, the student network needs fewer computing resources and can be effectively deployed on resource-limited equipment, and meanwhile, the student network can be effectively deployed on the resource-limited equipment. High accuracy is kept while student network parameters are reduced, a teacher network preprocesses pictures or videos to generate a Gaussian heat map, the knowledge distillation technology is utilized to transmit feature and posture information to the student network for learning, and the posture of a human body is processed and recognized, so that the learning efficiency is improved. Action recognition is carried out on a posture estimation result through a convolutional neural network, multi-task combination of human body target detection, 2D key point detection and 3D key point prediction is realized, a more uniform and robust feature space is established by using a knowledge distillation method, so that relatively high accuracy is obtained on an acquired data set, and the accuracy of the human body target detection is improved. The problem that coordinate points do not correspond in the human body posture collecting process is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lightweight human posture estimation method based on knowledge distillation. Background Art

[0002] The posture estimation problem is to determine the orientation of a three-dimensional target object. Posture estimation is used in many fields such as robot vision, motion tracking and single camera calibration. Human posture estimation usually faces problems such as coordinate point errors, while factors such as illumination, obstacle occlusion, human self-occlusion and action category confusion increase the difficulty and accuracy of action recognition. At the same time, human posture estimation in existing technologies only considers how to improve the recognition accuracy of the model, but ignores the complex model structure and the large number of calculation parameters that make it impossible to run efficiently on resource-constrained devices.

[0003] In response to the above problems, the inventors proposed a lightweight human posture estimation method based on knowledge distillation to solve the above problems. Summary of the invention

[0004] In order to solve the above problems, the present invention aims to provide a lightweight human posture estimation method based on knowledge distillation.

[0005] The present invention adopts the following technical solution:

[0006] A lightweight human posture estimation method based on knowledge distillation includes the following steps:

[0007] S1. Network processing

[0008] A: Teacher Network Processing

[0009] The teacher network is preprocessed through the preprocessing module, and then the preprocessed image or video is input into the multi-dimensional scaling fusion module to process the resolution of the image or video and generate a Gaussian heat map;

[0010] B: Student Network Processing

[0011] Repeat the teacher network processing method to generate a Gaussian heat map;

[0012] S2. Knowledge Distillation

[0013] Through feature distillation, the semantic information and overall spatial information in the feature map output by the residual block of the teacher network are transferred from the upper teacher network to the lower student network. At the same time, through posture distillation, the "soft" knowledge and key point labels in the teacher network output provide additional guidance to the student network in training, ensuring that the student network is trained under the guidance of the teacher network knowledge, and realizing the transfer of posture knowledge from the teacher network to the student network;

[0014] S3. Human body posture processing

[0015] Collect human posture data and posture feature information through acquisition equipment and Kinect SDK, reconstruct the skeleton and track the human body, and reconstruct the human skeleton system based on the tracked human joints. Each connection links the parent joint to the child joint.

[0016] Find the feature points (X, Y) from the RGB image, and then obtain the depth information of the feature points. Determine the projection relationship by calling the function MapcolorFrameToDepthFrame in the SDK, and obtain the real distance between the human joints and the acquisition device. Then obtain the data set through the acquisition device. The data set includes a posture estimation map and several feature vector groups.

[0017] S4, Posture Estimation and Action Recognition

[0018] By using the deeper ResNet34 as the backbone network, the final model is calculated, the threshold of the vector dimension and the probability value of the posture action are obtained to achieve action recognition.

[0019] Preferably, in S1, the specific steps of preprocessing the teacher network are as follows:

[0020] Input the image or video to the convolution layer and nonlinear activation layer Relu of the preprocessing module, and the channel is changed from C to 64 to complete the preprocessing of the teacher network.

[0021] Preferably, the multi-dimensional scale fusion module includes nine multi-resolution modules, each of which is composed of four residual blocks and a convolutional layer for multi-dimensional scale fusion;

[0022] The specific steps of the processing process of the multidimensional scaling module are as follows:

[0023] The output feature of the residual block in the multi-resolution module is defined as:

[0024]

[0025] Where i represents the stage, j represents the branch, k represents the sequence number of the residual block, M represents the input channel, N represents the number of output channels, and the output of the residual block It is expressed as:

[0026]

[0027] Among them, conv represents the convolution operation, σ represents the Relu activation function;

[0028] The output f of the multi-resolution module 1 1 (x)64→512 Defined as:

[0029]

[0030] Generate a new branch f 1 1 (x) output The operations are:

[0031]

[0032] Preferably, the specific steps of the characteristic distillation are as follows:

[0033] A1: Matching high-level semantic information is achieved by judging feature similarity, so that the student network obtains the same high-level semantic information as the teacher network;

[0034] A2: Comparing by cosine distance and using neural network to obtain similarity, calculating the cosine distance of feature vectors will consume a lot of computing time. Using a dual-channel convolutional neural network, by judging the similarity of the feature graphs of the teacher network and the student network, the student network learns the features extracted by the teacher network. The definition of feature distillation is as follows:

[0035]

[0036] Among them, F s and F t Represents the feature graph of the student network and the teacher network, S(F S ,f t ) represents the output result after the dual-channel network discriminates the feature map, w is the weight in the network, and λ=0.0005.

[0037] Preferably, the specific steps of posture distillation are as follows:

[0038] The function L that the student network learns the “soft” knowledge of the teacher network soft Defined as:

[0039]

[0040] in, and Represents the prediction heatmap of the kth key point predicted by the student network in training and the pre-trained teacher network, represents the mean squared error (MSE) loss function.

[0041] Preferably, in S3, any two non-overlapping coordinate points in the Kinect coordinate system are marked as A(x 1 ,y 1 , z 1 ), B(x 2,y 2 , z 2 ) Based on the translational properties and directionality of the vector, it is transformed into a mathematical coordinate system. The vector composed of two coordinates The calculation of is as follows:

[0042]

[0043] The above formula converts the three-dimensional coordinate points in the Kinect coordinate system into the mathematical coordinate system and calculates the relevant angles.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. In the present invention, the teacher network and the student network are processed through S1-S2 to design a lightweight student network. Compared with the teacher network, the student network requires less computing resources, so it can be effectively deployed on resource-constrained devices. At the same time, it reduces the number of student network parameters while maintaining high accuracy.

[0046] 2. In the present invention, human posture is processed and recognized through S3-S4, and action recognition is performed on the posture estimation results through convolutional neural network, so as to realize the multi-task combination of human target detection, 2D key point detection and 3D key point prediction, and use the knowledge distillation method to establish a more unified and robust feature space, so as to obtain a higher accuracy on the collected data set, and solve the problem of coordinate point mismatch in the process of collecting human posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is a flow chart of a lightweight human posture estimation method based on knowledge distillation of the present invention.

[0049] Figure 2 Flow chart of the knowledge distillation process of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Example: Figure 1-2 As shown, the present invention provides a lightweight human posture estimation method based on knowledge distillation, comprising the following steps:

[0052] S1. Network processing

[0053] A: Teacher Network Processing

[0054] The teacher network is preprocessed through the preprocessing module, and then the preprocessed image or video is input into the multi-dimensional scaling fusion module to process the resolution of the image or video and generate a Gaussian heat map;

[0055] The specific steps of preprocessing the teacher network are as follows:

[0056] Input the image or video to the convolution layer and nonlinear activation layer Relu of the preprocessing module, and the channel is changed from C to 64 to complete the preprocessing of the teacher network;

[0057] The multi-scale fusion module includes nine multi-resolution modules, each of which consists of four residual blocks and a convolutional layer for multi-scale fusion;

[0058] The specific steps of the processing of the multidimensional scaling module are as follows;

[0059] The output features of the residual block in the multi-resolution module are defined as:

[0060]

[0061] Where i represents the stage, j represents the branch, k represents the sequence number of the residual block, M represents the input channel, N represents the number of output channels, and the output of the residual block It is expressed as:

[0062]

[0063] Among them, conv represents the convolution operation, σ represents the Relu activation function;

[0064] The output f of the multi-resolution module 1 1 (x) 64→512 Defined as:

[0065]

[0066] Generate a new branch f 1 1 (x) output The operations are:

[0067]

[0068] B: Student Network Processing

[0069] Repeat the teacher network processing method to generate a Gaussian heat map;

[0070] S2. Knowledge Distillation

[0071] Through feature distillation, the semantic information and overall spatial information in the feature map output by the residual block of the teacher network are transferred from the upper teacher network to the lower student network. At the same time, through posture distillation, the "soft" knowledge and key point labels in the teacher network output provide additional guidance to the student network in training, ensuring that the student network is trained under the guidance of the teacher network knowledge, and realizing the transfer of posture knowledge from the teacher network to the student network;

[0072] The specific steps of feature distillation are as follows:

[0073] A1: Matching high-level semantic information is achieved by judging feature similarity, so that the student network obtains the same high-level semantic information as the teacher network;

[0074] A2: Comparing by cosine distance and using neural network to obtain similarity, calculating the cosine distance of feature vectors will consume a lot of computing time. Using a dual-channel convolutional neural network, by judging the similarity of the feature graphs of the teacher network and the student network, the student network learns the features extracted by the teacher network. The definition of feature distillation is as follows:

[0075]

[0076] Among them, F s and F t Represents the feature graph of the student network and the teacher network, S(F S ,f t ) represents the output result after the dual-channel network discriminates the feature map, w is the weight in the network, λ=0.0005;

[0077] The specific steps of posture distillation are as follows:

[0078] The function L that the student network learns the “soft” knowledge of the teacher network soft Defined as:

[0079]

[0080] in, and Represents the prediction heatmap of the kth key point predicted by the student network in training and the pre-trained teacher network, represents the mean square error (MSE) loss function;

[0081] S3. Human body posture processing

[0082] Collect human posture data and posture feature information through acquisition equipment and Kinect SDK, reconstruct the skeleton and track the human body, and reconstruct the human skeleton system based on the tracked human joints. Each connection links the parent joint to the child joint.

[0083] Find the feature points (X, Y) from the RGB image, and then obtain the depth information of the feature points. Determine the projection relationship by calling the function MapcolorFrameToDepthFrame in the SDK, and obtain the real distance between the human joints and the acquisition device. Then obtain the data set through the acquisition device. The data set includes a posture estimation map and several feature vector groups.

[0084] Mark any two non-overlapping coordinate points in the Kinect coordinate system as A(x 1 ,y 1 , z 1 ), B(x 2 ,y 2 , z 2 ) Based on the translational properties and directionality of the vector, it is transformed into a mathematical coordinate system. The vector composed of two coordinates The calculation of is as follows:

[0085]

[0086] The above formula converts the three-dimensional coordinate points in the Kinect coordinate system into the mathematical coordinate system and calculates the relevant angles;

[0087] S4, Posture Estimation and Action Recognition

[0088] By using the deeper ResNet34 as the backbone network, the final model is calculated, the threshold of the vector dimension and the probability value of the posture action are obtained to achieve action recognition.

[0089] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A lightweight human posture estimation method based on knowledge distillation, characterized in that: The following steps are involved: S1. Network processing A: Teacher Network Processing The teacher network is preprocessed through the preprocessing module, and then the preprocessed image or video is input into the multi-dimensional scaling fusion module to process the resolution of the image or video and generate a Gaussian heat map; B: Student Network Processing Repeat the teacher network processing method to generate a Gaussian heat map; S2. Knowledge Distillation Through feature distillation, the semantic information and overall spatial information in the feature map output by the residual block of the teacher network are transferred from the upper teacher network to the lower student network. At the same time, through posture distillation, the "soft" knowledge and key point labels in the teacher network output provide additional guidance to the student network in training, ensuring that the student network is trained under the guidance of the teacher network knowledge, and realizing the transfer of posture knowledge from the teacher network to the student network; S3. Human body posture processing Collect human posture data and posture feature information through acquisition equipment and Kinect SDK, reconstruct the skeleton and track the human body, and reconstruct the human skeleton system based on the tracked human joints. Each connection links the parent joint to the child joint. Find the feature points (X, Y) from the RGB image, and then obtain the depth information of the feature points. Determine the projection relationship by calling the function MapcolorFrameToDepthFrame in the SDK, and obtain the real distance between the human joints and the acquisition device. Then obtain the data set through the acquisition device. The data set includes a posture estimation map and several feature vector groups. S4, Posture Estimation and Action Recognition By using the deeper ResNet34 as the backbone network, the final model is calculated, the threshold of the vector dimension and the probability value of the posture action are obtained to achieve action recognition.

2. A lightweight human posture estimation method based on knowledge distillation as claimed in claim 1, characterized in that: In S1, the specific steps of preprocessing the teacher network are as follows: Input the image or video to the convolution layer and nonlinear activation layer Relu of the preprocessing module, and the channel is changed from C to 64 to complete the preprocessing of the teacher network.

3. A lightweight human posture estimation method based on knowledge distillation as claimed in claim 1, characterized in that: The multi-dimensional scale fusion module includes nine multi-resolution modules, each of which includes four residual blocks and a convolution layer for multi-dimensional scale fusion.

4. A lightweight human posture estimation method based on knowledge distillation as claimed in claim 3, characterized in that: The specific steps of the processing process of the multidimensional scaling module are as follows: The output feature of the residual block in the multi-resolution module is defined as: Where i represents the stage, j represents the branch, k represents the sequence number of the residual block, M represents the input channel, N represents the number of output channels, and the output of the residual block It is expressed as: Among them, conv represents the convolution operation, σ represents the Relu activation function; The output of the multi-resolution module Defined as: Generate a new branch The operations are:

5. A lightweight human posture estimation method based on knowledge distillation as claimed in claim 1, characterized in that: The specific steps of the characteristic distillation are as follows: A1: Matching high-level semantic information is achieved by judging feature similarity, so that the student network obtains the same high-level semantic information as the teacher network; A2: Comparing by cosine distance and using neural network to obtain similarity, calculating the cosine distance of feature vectors will consume a lot of computing time. Using a dual-channel convolutional neural network, by judging the similarity of the feature graphs of the teacher network and the student network, the student network learns the features extracted by the teacher network. The definition of feature distillation is as follows: Among them, F s and F t Represents the feature graph of the student network and the teacher network, S(F S ,f t ) represents the output result after the dual-channel network discriminates the feature map, w is the weight in the network, and λ=0.0005.

6. A lightweight human posture estimation method based on knowledge distillation as claimed in claim 1, characterized in that: The specific steps of posture distillation are as follows: The function L that the student network learns the "soft" knowledge of the teacher network soft Defined as: in, and Represents the prediction heatmap of the kth key point predicted by the student network in training and the pre-trained teacher network, represents the mean squared error (MSE) loss function.

7. A lightweight human posture estimation method based on knowledge distillation as claimed in claim 1, characterized in that: In S4, any two non-overlapping coordinate points in the Kinect coordinate system are marked as A(x1, y1, z1) and B(x2, y2, z2). Due to the translational properties and directionality of the vector, it is transformed into the mathematical coordinate system. The vector composed of the two coordinates is The calculation of is as follows: The above formula converts the three-dimensional coordinate points in the Kinect coordinate system into the mathematical coordinate system and calculates the relevant angles.

Images