A human key point detection method combining a lightweight convolutional network with a transformer

Abstract

The application relates to a human key point detection method combining a lightweight convolutional network and a Transformer, adopts a deep separable convolution combined with two residual blocks as a feature extraction backbone network, and adopts a lightweight convolutional network combined with a Transformer for key point detection. Compared with a traditional CNN, the deep separable convolution reduces a large number of redundant parameters, and meanwhile, the two residual blocks combine the advantages of standard convolution, so that the feature extraction network in the present application achieves good effects in image feature extraction while the model parameters are compressed and the operation efficiency is improved, the lightweight convolutional network better extracts image features while reducing the parameter quantity, the self-attention layer in the Transformer can obtain global constraint relationships and reserve fine-grained local feature information, so that the spatial relationships between various parts of a human body can be effectively captured, and through fewer parameters and faster speed, the precision of the human key point detection technology based on the CNN can be comparable.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a human key point detection method combining a lightweight convolutional network with a Transformer. BACKGROUND

[0002] Human key point detection refers to reconstructing human joints and limbs based on images, that is, finding and combining key points of human skeletons. In the field of computer vision (CV), human key point positioning is still a popular research direction. At present, human key point detection has given rise to many applications, and has important significance and wide application prospects in the fields of human-computer interaction, patient monitoring systems, intelligent video surveillance, virtual reality, athlete auxiliary training, smart home, human animation, intelligent security, etc.

[0003] At present, most human key point detection methods are based on traditional convolutional neural networks, which can learn feature detection key points well, but these convolutional neural network structures are complex and have large parameters, and training and reasoning will occupy a large amount of computing resources and time. The Transformer adopts an encoder architecture based on self-attention and multi-layer perceptron, which makes it have a natural advantage over CNN in learning position relationships, and the multi-layer attention module enables the model to capture the relationship between any key point positions, and its network structure is simpler than that of the traditional convolutional network. However, the Transformer is far inferior to the convolutional network in extracting low-level image features, which also leads to a significant reduction in the accuracy of key point detection using the Transformer model. SUMMARY

[0004] In view of the deficiencies of the prior art, the application provides a human key point detection method combining a lightweight convolutional network with a Transformer, which combines the advantages of the lightweight convolutional network in extracting image texture information and the lightweight model with the global modeling characteristics of the Transformer structure, greatly improves the detection speed under the premise of high key point accuracy.

[0005] The application adopts the following technical scheme: a human key point detection method combining a lightweight convolutional network with a Transformer, comprising:

[0006] Step 1: Obtain picture data, which is processed as input;

[0007] Step 2: Construct a lightweight convolutional network as a feature extractor, input the picture data into the feature extractor for feature extraction, and obtain a target feature map;

[0008] Step 3: Position encoding is performed on the target feature map, so that the feature vectors in the target feature map maintain independent spatial position relationships;

[0009] Step 4: The target feature map with position encoding is input into the multi-layer attention module of the Transformer, the attention module captures the dependency between different features from the feature vector sequence, and the feature map with different attention weights is fused with the target feature map again to obtain updated feature vectors;

[0010] Step 5: The updated feature vectors are input into the multi-layer perceptron (MLP) module of the Transformer to obtain the final output features;

[0011] Step 6: The key point coordinates of the output features are predicted by the regression heat map method.

[0012] As a preferred, in the step 2, the feature extractor includes a depth separable convolution block and two depth residual blocks; when performing feature extraction: Stage1, a convolution kernel with a step of 1 is used to perform channel-by-channel convolution on an input picture with an input channel of 3, while maintaining channel independence, and the channel dimension is raised to 128 to obtain a first feature map; stage2: point-by-point convolution, a 1*1*128 convolution kernel is used to weight and combine the first feature map in the depth direction to generate a second feature map; stage3: the channel dimension of the second feature map is reduced to 64 dimensions and input into a 3-layer bottleneck residual block, and a 3*3*64 convolution kernel is used to further extract features to obtain a third feature map; stage4: the channel dimension of the third feature map is raised to 128 dimensions and input into a 4-layer bottleneck residual block, and a 3*3*128 convolution kernel is used to extract features to obtain a target feature map.

[0013] As a preferred, in the step 3, the encoding process is represented as:

[0014]

[0015]

[0016] where P x , P y is the position index along the x or y direction, i represents a newly generated index, T is a scaling factor, and then the position information is embedded into the feature vectors of the target feature map

[0017] As a preferred, in the step 4, the multi-layer attention module has three weight parameter matrices w q , w k , w v ∈R d×d, a matrix A e R N×N The calculation formula is:

[0018]

[0019] wherein the eigenvector X1 is multiplied by w q The weight matrix q i to obtain a new vector q1, q1 represents a vector containing the weight of the Q value matrix; the eigenvector X1 is multiplied by w k The weight matrix k i to obtain a new vector k1, k1 represents a vector containing the weight of the K value matrix; the dot product of q1 and k1 is divided by the square root of d and then weighted average to obtain the attention index matrix, wherein d represents the dimension of the K value matrix vector, and then the weight vector w i The matrix A is weighted and summed respectively to complete the update of the eigenvector.

[0020] As preferred, in step 5, the updated eigenvector is input into a feedforward neural network, after passing through two fully connected layers and a Relu activation layer, LayerNorm operation is performed and the upper layer eigenvector is fused to obtain the final eigenvector.

[0021] As preferred, in step 6, the output of the Transformer is connected to a head network, and the head network is composed of two convolutional layers, the first convolutional layer keeps the dimension of the eigenvector obtained in step 5 consistent with the second convolutional layer, and the second convolutional layer increases the dimension of the output of the Transformer module and then reduces it to k, thereby predicting a k-class key point heat map, wherein k is the number of key points.

[0022] As preferred, the COCO2017 public dataset is used as the source of picture data, and it is scaled to 256*192 size in equal proportion and used as input.

[0023] The present application has the following beneficial effects: 1. The depth separable convolution is combined with two residual blocks as a feature extraction backbone network, the depth separable convolution reduces a large number of redundant parameters compared with the traditional CNN, and the two residual blocks combine the advantages of the standard convolution, so that the feature extraction network in the present application achieves good results in image feature extraction while compressing the model parameters and accelerating the running efficiency. 2. The lightweight convolution network is combined with the Transformer for key point detection, the lightweight convolution network extracts image features better while reducing the parameter amount, the self-attention layer in the Transformer can obtain global constraint relationship and retain fine-grained local feature information, so that the spatial relationship between each part of the human body can be effectively captured, and the present application can achieve the precision comparable to the most advanced human key point detection technology based on the CNN with fewer parameters and faster speed. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and are used to explain the application, and do not constitute a limitation of the application. In the drawings:

[0025] Figure 1 The present application is a human key point detection flowchart;

[0026] Figure 2 The present application is a whole structure diagram; DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0028] A human key point detection method combining a lightweight convolution network with a Transformer, as shown in Figure 1 and Figure 2 , includes the following steps:

[0029] Step 1: Obtain picture data, the present embodiment uses the COCO2017 public dataset as the picture data source, and scales it to 256*192 size as input;

[0030] Step 2: Construct a lightweight convolution network as a feature extractor, input the picture data into the feature extractor for feature extraction, and obtain a target feature map;

[0031] In this embodiment, the feature extractor includes a depth separable convolution block and two deep residual blocks. When performing feature extraction, first, the depth separable convolution block is used: Stage 1, a convolution kernel with a step of 1 is used to perform channel-by-channel convolution on an input picture with an input channel of 3, while keeping the channels independent, and the channel dimension is raised to 128 to obtain a first feature map; stage 2: point-by-point convolution, a 1*1*128 convolution kernel is used to weight and combine the first feature map in the depth direction to generate a second feature map; stage 3: the channel dimension of the second feature map is reduced to 64 dimensions and input into a 3-layer bottleneck residual block, and a 3*3*64 convolution kernel is used to further extract features to obtain a third feature map; stage 4: the channel dimension of the third feature map is raised to 128 dimensions and input into a 4-layer bottleneck residual block, and a 3*3*128 convolution kernel is used to extract features to obtain a target feature map.

[0032] Step 3: Position encoding is performed on the target feature map, so that the feature vectors in the target feature map maintain independent spatial position relationships;

[0033] Since the Transformer module itself lacks position information, position encoding needs to be added to the feature vectors in the target feature map before entering the Transformer module, so that the position information of the feature vectors is independent in the image horizontal x and vertical y directions, and maintains an independent spatial position relationship. The encoding process can be represented as:

[0034]

[0035]

[0036] where P x , P y is the position index along the x or y direction, i represents the newly generated index, T is the scaling ratio, H and W represent the length and width of the input image, and then the position information is embedded into the feature vectors of the target feature map.

[0037] Step 4: The target feature map with position encoding is input into the multi-layer attention module of the Transformer, and the attention module captures the dependency between different features from the feature vector sequence, and the feature map with different attention weights is fused with the target feature map again to obtain updated feature vectors;

[0038] The multi-layer attention module iteratively captures the dependencies between different features, such as keypoint features, from the feature vector sequence through query-key-value pairs. Then, it fuses the feature maps with different attention weights with the feature map obtained in step 3. Specifically, the multi-layer attention module has three weight parameter matrices w. q w k w v ∈R d×d , where A∈R represents the attention index matrix N×N The calculation formula is:

[0039]

[0040] Wherein, the eigenvector X1 is multiplied by w q The weight matrix q i We obtain a new vector q1, where q1 represents a vector containing the weights of the Q-value matrix; the eigenvector X1 is multiplied by w. k The weight matrix k i We obtain a new vector k1, where k1 represents the vector containing the weights of the K-value matrix; the dot product of q1 and k1, divided by the square root of d, is then weighted and averaged to obtain the attention index matrix A, where d represents the dimension of the K-value matrix vector, and then the weight vector w in the V-value matrix is... i The eigenvectors are updated by performing weighted summations on matrix A. The feature maps with different weights for different features are called attention maps, which can also be viewed as dynamic weights for the features.

[0041] The resulting attention map is then fused with the low-dimensional feature map again, and a LayerNorm operation is performed to exchange feature information at different scales. The feature map fusion process involves forward and backward propagation of the feature vectors in the feature maps, followed by normalization.

[0042] Step 5: Input the updated feature vector into the Transformer's Multilayer Perceptron (MLP) module to obtain the final output features;

[0043] The Multilayer Perceptron (MLP) module includes a feed-forward network (FFN) layer, which helps to better construct the spatial relationships between keypoints and improve the model's localization efficiency. The updated feature vector is input into the feed-forward neural network, passes through two fully connected layers and one ReLU activation layer, and then undergoes LayerNorm operation to fuse with the upper-layer feature vector to obtain the final feature vector.

[0044] Step 6: Predict the key point coordinates of the output features using a regression heatmap method.

[0045] The output of the Transformer is connected to a simple head network, which functions to predict the key point coordinates by the method of regressing heat maps. Specifically, the head network is composed of two convolutional layers, the first convolutional layer keeps the dimension of the feature vector obtained in step 5 consistent with the second convolutional layer, and the second convolutional layer functions to classify and regress the features. The output of the Transformer module is upgraded to E∈R c×H×W where H and W represent the length and width of the input image, and c represents the channel dimension of the feature map; the channel dimension of E is reduced from c to k through 1x1 convolution, so as to predict k-class key point heat maps P∈R k×h×w where k is the number of key points, c represents the channel dimension of the feature map, and h and w represent the length and width of the key point heat map and h and w are 1 / 4 of the size of the input image, respectively.

[0046] The average accuracy AP (Average Precision) based on OKS (Object Keypoint Similarity) and FPS (Frames Per Second) are used as evaluation indicators to evaluate the accuracy and processing speed of the model. OKS represents the similarity between key points, average accuracy AP (Average Precision) represents the ratio of the number of successfully detected key points to the total number of targets, and FPS represents the number of pictures processed by the model per second.

[0047] The method of the present application is compared with the mainstream CNN-based human key point detection method AlphaPose and HRNet, and the experimental results are shown in the following table:

[0048] Model AP FPS AlphaPose 72.3 20 HRNet 73.1 27 The method herein 73.5 78

[0049] From the experimental results, compared with the mainstream key point detection method based on CNN, the method of combining lightweight convolution and TransFormer model proposed in the present application greatly improves the detection speed (FPS) and speeds up the model running speed under the premise that the average accuracy is not lower than that of the above two methods.

[0050] The above is only a specific embodiment of the present application, and does not limit the present application in any form. Therefore, any simple modification of the above embodiment according to the technical essence of the present application, which does not deviate from the technical solution of the present application, shall fall within the scope of protection of the technical solution of the present application.

Claims

1. A human key point detection method combining a lightweight convolutional network with a Transformer, characterized in that, Comprise: Step 1: Obtain picture data, process as input; Step 2: Build a lightweight convolutional network as a feature extractor, input the picture data into the feature extractor for feature extraction, and obtain a target feature map; Step 3: Position encoding is performed on the target feature map, so that the feature vectors in the target feature map maintain independent spatial position relationships; Step 4: Input the target feature map with position encoding into the multi-layer attention module of the Transformer, and the attention module captures the dependency between different features from the feature vector sequence, and fuses the feature map with different attention weights with the target feature map again to obtain updated feature vectors; Step 5: Input the updated feature vectors into the multi-layer perceptron (MLP) module of the Transformer to obtain the final output feature; Step 6: Predict the key point coordinates of the output feature by the regression heat map method; in step 2, the feature extractor includes a depth separable convolution block and two depth residual blocks; When performing feature extraction: Stage 1, use a convolution kernel with a step size of 1 to perform channel-by-channel convolution on an input picture with 3 input channels, while maintaining channel independence, and increase the channel dimension to 128 to obtain a first feature map; stage 2: point-by-point convolution, using a 1*1*128 convolution kernel to weight and combine the first feature map in the depth direction to generate a second feature map; stage 3: reduce the channel dimension of the second feature map to 64 and input it into a 3-layer bottleneck residual block, using a 3*3*64 convolution kernel to further extract features to obtain a third feature map; stage 4: increase the channel dimension of the third feature map to 128 and input it into a 4-layer bottleneck residual block, using a 3*3*128 convolution kernel to extract features to obtain a target feature map. 2.The human key point detection method of claim 1, wherein, In the step, the encoding process is represented as: wherein Position index in x or y direction, i represents newly generated index, i∈ Then, the position information is embedded into the feature vector of the target feature map. 3.The human key point detection method of claim 1, wherein, The multi-layer attention module in step 4 has three weight parameter matrices ∈ The matrix A∈ The calculation formula of is: A = softmax wherein the eigenvector is multiplied by the weight matrix to obtain a new vector , is multiplied by the weight matrix to obtain a new vector , , represents a vector containing the K-value matrix weight; the dot product of is weighted and averaged after being divided by the square root of d to obtain an attention index matrix, wherein d represents the dimension of the K-value matrix vector, and then the weight vector in the V-value matrix is weighted and summed respectively to complete the update of the eigenvector.​​ 4. The human key point detection method of claim 1, wherein, In step 5, the updated feature vectors are input into a feedforward neural network, which goes through two fully connected layers and a Relu activation layer, then performs LayerNorm operation and fuses with the upper layer feature vectors to obtain the final feature vectors.

5. The human key point detection method of claim 4, wherein, In step 6, the output of the Transformer is connected to a head network composed of two convolution layers, the first convolution layer maintains the dimension of the feature vectors obtained in step 5 consistent with the second convolution layer, and the second convolution layer increases the dimension of the Transformer module output and then reduces it to k, thereby predicting a k-class key point heat map, where k is the number of key points.

6. The human key point detection method of claim 1, wherein, The COCO2017 public dataset is used as the source of picture data, and it is scaled to 256*192 size as input.

Images