Lightweight human pose estimation method based on optimal network structure

Abstract

The present invention provides a lightweight human posture estimation method with an optimal network structure, constructs an overall network structure, inputs data into the overall network structure, performs network optimal search, obtains an optimal lightweight network structure, inputs a test set into the optimal lightweight network structure, and outputs a heat map of human posture estimation; the overall network structure is a high-low resolution parallel network structure, which can fully extract human posture features; the optimal lightweight network structure is obtained by selecting candidate modules with an optimal network structure, formulating a search method, and selecting a performance evaluation strategy, and a CPB module, one of the candidate modules, is a module that uses separable convolution and channel parameter reduction, and has the advantages of a small number of network parameters and a small amount of calculation while maintaining a high accuracy rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of human body posture estimation in computer vision, and in particular relates to a lightweight human body posture estimation method with an optimal network structure. Background Art

[0002] With the increasing development of computer vision and image processing, computer vision has made great progress in the fields of image target segmentation, classification, detection and behavior classification. Human posture estimation is an important research direction.

[0003] Traditional human posture estimation methods are generally based on graph structures and deformable component models. They design 2D human body component detectors, use graph models to establish the connectivity of each component, and continuously optimize the graph structure model in combination with relevant constraints of human kinematics to estimate human posture.

[0004] Compared with traditional machine vision methods, deep learning can omit data preprocessing and directly learn abstract and essential features from raw data, thereby replacing manual feature extraction. Deep learning has been widely used in attitude estimation in recent years and has achieved good accuracy. However, the number of parameters and the amount of computation required for existing models using deep learning for attitude estimation are very large. Classic deep learning models for attitude estimation usually only consider how to improve the generalization performance of the model, while ignoring significant efficiency issues. In real-life applications, real-time attitude estimation is required in a limited computing environment, and existing deep learning models cannot well implement real-time attitude estimation in a limited computing environment. Summary of the invention

[0005] In view of the shortcomings in the prior art, the present invention provides a lightweight human posture estimation method with an optimal network structure, which overcomes the problem that the existing deep learning model cannot well implement real-time posture estimation in a limited computing environment.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] The optimal network structured lightweight human posture estimation method is as follows:

[0008] Constructing an overall network structure, wherein the overall network structure includes five modules, each module includes two sub-modules;

[0009] The overall network structure inputs data, performs network optimal search, and obtains an optimal lightweight network structure; the optimal lightweight network structure is obtained by selecting candidate modules of the optimal network structure, formulating a search method, and selecting a performance evaluation strategy;

[0010] Input the test set into the optimal lightweight network structure and output the heat map of human posture estimation;

[0011] The candidate module is a CPB module or a MobileNet or a ShuffleNet or an Xception module, and the CPB module is a module using separable convolution and channel parameter reduction.

[0012] Furthermore, the CPB module is as follows: the input feature map is first subjected to channel reduction by 1×1 convolution, then passed through a BN layer and a ReLU activation function, then a 3×3 depthwise separable convolution is used, then a 1×1 convolution is used, then passed through a BN layer and a ReLU activation function, and finally channel parameter reduction and fusion are performed.

[0013] Furthermore, the channel parameter reduction and fusion are specifically as follows: the input feature map is input into the short link, reduced by α times through 1×1 convolution, and finally the channel parameter reduction is fused with the short link channel through an add operation.

[0014] Furthermore, in the process of channel parameter reduction, global channel features are first obtained through global average pooling, then through two fully connected layers, and then a channel weight is obtained through the Sigmoid activation function, and then the Scale operation is performed. The feature map after the Scale operation is then passed through a 1x1 convolution.

[0015] Furthermore, during the Scale operation, a threshold γ is set. When the channel weight is less than the threshold γ, the channel weight is set to zero. If the channel weight is greater than or equal to the threshold γ, the channel weight maintains the value obtained by the Sigmoid activation function.

[0016] Furthermore, the five modules include module 1, module 2, module 3, module 4 and module 5, module 1 includes sub-module 11 and sub-module 12, module 2 includes sub-module 21 and sub-module 22, module 3 includes sub-module 31 and sub-module 32, module 4 includes sub-module 41 and sub-module 42, and module 5 includes sub-module 51 and sub-module 52; sub-module 11, sub-module 21, sub-module 31, sub-module 41 and sub-module 51 perform convolution operations in sequence; the input, sub-module 11, sub-module 21, sub-module 31, sub-module 41, sub-module 12 and sub-module 22 are all down-sampled to reduce the resolution to 1 / 2 and the number of channels becomes twice the number of output channels of the previous module, and sub-module 12, sub-module 22, sub-module 32, sub-module 42 and sub-module 52 are all up-sampled to improve the resolution and keep the number of channels unchanged.

[0017] Furthermore, the performance evaluation strategy is to add a time-consuming term to the loss function, and the loss function is:

[0018]

[0019] Where: W is the parameter of each candidate module, Q is the weight of each candidate module, x i is the input image, y i is the label of the i-th image, p(x i , W, Q) is the network's i prediction, λ is a constant, and t j is the time consumed by the jth candidate module on the embedded device.

[0020] Furthermore, in the process of obtaining the optimal lightweight network structure, the training set is used to train the overall network structure. During the gradient descent process, the weight Q in the loss function is updated through back propagation. Each sub-module retains the candidate module with the largest weight, and the retained candidate modules are connected, so that the optimal lightweight network structure can be searched eventually.

[0021] The beneficial effects of the present invention are:

[0022] (1) The overall network structure of the present invention includes five modules, each of which includes two sub-modules; the overall network structure of the present invention is a high-low resolution parallel network structure, which has great advantages for position-sensitive human posture estimation, and repeatedly performs multi-scale fusion of high and low resolutions in the whole process, which can fully extract human posture features, and can also effectively solve situations such as occlusion, which greatly improves the accuracy of the network.

[0023] (2) The optimal lightweight network structure of the present invention is obtained by selecting the optimal network structured candidate modules, and the CPB module, one of the candidate modules, is a module that uses separable convolution and channel parameter reduction, which has a small number of network parameters and a small amount of calculation while maintaining a high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A lightweight posture estimation flow chart of the optimal network structure of the present invention;

[0025] Figure 2 This is a schematic diagram of the overall network structure of the present invention;

[0026] Figure 3 This is a schematic diagram of the overall network model of the present invention;

[0027] Figure 4 This is a schematic diagram of the CPB module structure of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of the channel reduction module of the present invention;

[0029] Figure 6This is a human body posture estimation heat map obtained by using the lightweight posture estimation method of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0031] like Figure 1 As shown, the present invention provides an optimal network structured lightweight posture estimation method, which specifically includes the following steps:

[0032] Step (1): Obtain a human posture estimation dataset

[0033] The human pose estimation dataset used in this example is MPII, which is the most advanced benchmark for evaluating human joint pose estimation. The dataset includes approximately 25,000 images, each of which provides an activity label. The dataset contains more than 40,000 people with annotated body joints, and the dataset covers 410 human activities; each image is extracted from a YouTube video.

[0034] Step (2), data preprocessing

[0035] The human pose estimation dataset is divided into three parts, two of which are selected as training sets and the other one as a test set. Compared with the training set, the test set adds annotations of body part occlusion, 3D torso and head direction; the size of each image in the training set and the test set is scaled to 512×512; the training set performs triple cross-validation during training.

[0036] Step (3), build the overall network structure

[0037] Use Simple Pose to build the overall network structure and design the size and number of channels of the input and output feature maps of each module;

[0038] See also Figure 2 and Figure 3 , specifically including the following steps:

[0039] Step (31), dividing the overall network structure into five modules (including module 1, module 2, module 3, module 4 and module 5, module 1 includes sub-module 11 and sub-module 12, module 2 includes sub-module 21 and sub-module 22, module 3 includes sub-module 31 and sub-module 32, module 4 includes sub-module 41 and sub-module 42, module 5 includes sub-module 51 and sub-module 52), and specifying the input and output feature map size (height, width, number of channels) of each module;

[0040] In step (32), submodule 11, submodule 21, submodule 31, submodule 41 and submodule 51 perform convolution operations in sequence, and maintain the high resolution of the input unchanged. The input, submodule 11, submodule 21, submodule 31, submodule 41, submodule 12 and submodule 22 are all down-sampled to reduce the resolution to 1 / 2 and the number of channels is doubled. Submodule 12, submodule 22, submodule 32, submodule 42 and submodule 52 are all up-sampled to improve the resolution, while the number of channels remains unchanged. Through up and down sampling, the mutual fusion of high and low resolutions is achieved in the entire network structure.

[0041] Step (4) selects the candidate modules for the optimal network structure, formulates the search method, selects the performance evaluation strategy, inputs the data of the overall network structure, performs the optimal network search, and obtains the optimal lightweight network structure.

[0042] The candidate modules with the best network structure are selected, and CPB module, MobileNet, ShuffleNet and Xception are selected as candidate modules for each submodule.

[0043] The CPB module: Design a module with separable convolution and channel parameter reduction; see Figure 4 , the CPB module is specifically:

[0044] The input feature map (c channels) is first reduced by 1×1 convolution (reduced to Channels), then through the BN layer (normalization) and ReLU activation function, then use 3×3 depth separable convolution to reduce the number of parameters and extract features from the reduced feature map, and then use 1×1 convolution to restore the reduced number of channels, and then through the BN layer (normalization) and ReLU activation function, finally perform channel parameter reduction and fusion, channel parameter reduction is achieved using the channel parameter reduction module (using residual structure); the input feature map input short link, through 1×1 convolution to reduce α times, finally the channel parameter reduction and short link channel are fused together through the add operation.

[0045] like Figure 5 As shown in the figure, in the process of channel parameter reduction, global channel features are first obtained through global average pooling; then the amount of parameters is reduced through two fully connected layers (FC), and then a channel weight ε between 0 and 1 is obtained through the Sigmoid activation function. Then, during the Scale operation, a threshold γ is set. When the channel weight ε is less than the threshold γ, the channel weight ε is set to zero. If the channel weight ε is greater than or equal to the threshold γ, the channel weight ε maintains the value su obtained by the Sigmoid activation function; the feature map after the Scale operation is then cut off through a 1x1 convolution to cut off the channels with smaller weights.

[0046] The present invention adopts a gradient-based search method to search for the optimal lightweight target network structure.

[0047] The performance evaluation strategy is selected with multiple objectives, including the accuracy of predicting joint points and the time consumption of each candidate module on the embedded device. The main method of selecting the performance evaluation strategy is to add a time-consuming term to the loss function, and update the weight parameters of the candidate modules through back propagation, so as to search for the optimal lightweight network structure.

[0048] The design of the loss function includes the accuracy of the target and the label, and the expectation of the total network time consumption. First, two weight parameters W and Q are designed. W is the parameter (convolution kernel) of each candidate module, and Q is the weight of each candidate module. The values ​​of Q1 to Q4 are between 0 and 1. In order to ensure the value of Q, Q is calculated by the softmax function.

[0049] The accuracy formula of the target and label is:

[0050]

[0051] Where: x i is the input image, y i is the label of the i-th image, p(x i , W, Q) is the network's i prediction.

[0052] Secondly, the time consumed by the network on the embedded device is taken into account in the performance evaluation strategy, and the expectation of the total time consumption is designed as a parameter Consume and added to the loss function. The specific steps are as follows:

[0053] Calculate the time consumed by each candidate module in the optimization space in the embedded device, calculate the weighted average of the time consumed by each module, and calculate the overall expected time consumed by all sub-modules; the expected total time consumed is expressed as:

[0054]

[0055] Where: t j is the time consumed by the jth candidate module on the embedded device.

[0056] The smaller the Consume, the shorter the overall network time consumption. By adding Consume to the loss function, the optimal network structure can search for a sufficiently lightweight target network structure. The overall loss function is as follows. The constant λ can be controlled to obtain a balance between model accuracy and speed. The loss function is:

[0057]

[0058] The training set is used to train the overall network structure. During the gradient descent process, the weight Q in the loss function is updated through back propagation. Each sub-module retains the candidate module with the largest weight, and the retained candidate modules are connected. Finally, the optimal lightweight network structure can be searched.

[0059] The overall network structure is implemented using the pytorch framework, with a learning rate of 0.02 and 200 epochs of training. Considering the high image resolution and GPU memory limitations, only one image is used in each iteration, that is, the batch size is set to 1.

[0060] Step (5), input the test set into the optimal lightweight network structure and output the heat map of human posture estimation.

[0061] Figure 6 This is a specific example of a human body posture estimation heat map output by lightweight posture estimation according to the present invention.

[0062] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A lightweight human posture estimation method with an optimal network structure, characterized by: Constructing an overall network structure, wherein the overall network structure includes five modules, each module includes two sub-modules; The overall network structure inputs data, performs network optimal search, and obtains an optimal lightweight network structure; the optimal lightweight network structure is obtained by selecting candidate modules of the optimal network structure, formulating a search method, and selecting a performance evaluation strategy; Input the test set into the optimal lightweight network structure and output the heat map of human posture estimation; The candidate module is a CPB module or a MobileNet or a ShuffleNet or a Xception module, wherein the CPB module is a module using separable convolution and channel parameter reduction; The CPB module is as follows: the input feature map is firstly subjected to channel reduction by 1×1 convolution, then passed through a BN layer and a ReLU activation function, then a 3×3 depthwise separable convolution is used, then a 1×1 convolution is used, then a BN layer and a ReLU activation function are passed, and finally channel parameter reduction and fusion are performed; The channel parameter reduction and fusion are specifically as follows: the input feature map is input into the short link, reduced by α times through 1×1 convolution, and finally the channel parameter reduction and the short link channel are fused through the add operation; In the process of channel parameter reduction, global average pooling is first performed to obtain global channel features, then two fully connected layers are passed, and then a channel weight is obtained through the Sigmoid activation function, and then a scale operation is performed. The feature map after the scale operation is then passed through a 1x1 convolution; During the Scale operation, a threshold γ is set. When the channel weight is less than the threshold γ, the channel weight is set to zero. If the channel weight is greater than or equal to the threshold γ, the channel weight remains the value obtained by the Sigmoid activation function. The five modules include module 1, module 2, module 3, module 4 and module 5, module 1 includes submodule 11 and submodule 12, module 2 includes submodule 21 and submodule 22, module 3 includes submodule 31 and submodule 32, module 4 includes submodule 41 and submodule 42, and module 5 includes submodule 51 and submodule 52; submodule 11, submodule 21, submodule 31, submodule 41 and submodule 51 perform convolution operations in sequence; input, submodule 11, submodule 21, submodule 31, submodule 41, submodule 12 and submodule 22, all reduce the resolution to 1 / 2 and the number of channels becomes twice the number of output channels of the previous module through downsampling, and submodule 12, submodule 22, submodule 32, submodule 42 and submodule 52, all improve the resolution through upsampling and keep the number of channels unchanged; The performance evaluation strategy is to add a time-consuming term to the loss function, which is: Where: W is the parameter of each candidate module, Q is the weight of each candidate module, x i is the input image, y i is the label of the i-th image, p(x i , W, Q) is the network's i prediction, λ is a constant, and t j is the time consumed by the jth candidate module on the embedded device; In the process of obtaining the optimal lightweight network structure, the training set is used to train the overall network structure. During the gradient descent process, the weight Q in the loss function is updated through back propagation. Each sub-module retains the candidate module with the largest weight, and the retained candidate modules are connected. Finally, the optimal lightweight network structure can be searched.

Images