Method for feature and semantic refinement network for human pose estimation

Abstract

The application provides a method of feature and semantic refinement network for human pose estimation, which refines the predicted human pose in two aspects of features and semantics. In the aspect of features, multiple expansion rates are used to realize feature alignment of different receptive fields, compared with general down-sampling and up-sampling operations, the feature loss is reduced; then an attention mechanism is used to weight the auxiliary features, and the loss caused by the wrong alignment is reduced. In the aspect of semantics, the difference information between adjacent heat maps is used as an auxiliary to correct the key heat map after feature correction. The method uses the information features of the two aspects of features and semantics to complete the refinement of the predicted pose, improves the general feature alignment module, and improves the accuracy of human pose estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a feature and semantic refinement network method for human pose estimation. BACKGROUND

[0002] Human pose estimation is one of the typical disciplines in computer vision, and its goal is to accurately detect and mark the positions of human key points (such as head and wrist) in images. In addition, it has great application in various fields such as video monitoring, automatic driving, motion analysis, etc. Through the creation of large data sets, this problem has been improved. However, many high-precision methods are designed based on static images, and when applied to videos, they are difficult to have good performance, and due to the presence of more motion blur, video defocus and frequent pose occlusion in video data, a single frame loses a lot of spatial information, resulting in inaccurate detection results.

[0003] The existing human pose estimation method based on video mainly corrects and refines the heat map at the semantic level, without considering the influence of feature level correction on accuracy. Other fields such as video super-resolution use feature alignment to align auxiliary features to key features and correct at the feature level. However, its network structure is a pyramid structure, and the feature loss caused by multiple downsampling and upsampling reduces the feature accuracy and affects the network performance.

[0004] Therefore, a feature and semantic refinement network is proposed to improve the above-mentioned problems. SUMMARY

[0005] The purpose of the present application is to solve the problems existing in the prior art, and to provide a feature and semantic refinement network method for human pose estimation.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme: the feature and semantic refinement network method for human pose estimation comprises the following steps:

[0007] S101, data preprocessing.

[0008] The present method is based on single human pose estimation. Since there are usually multiple people in an image, the image needs to be preprocessed first, and the picture is cropped into several single person pictures. Then the cropped pictures of the same person in the key frame and the previous and next frames of the key frame are taken as a group, and input into the backbone network to generate features and rough heat maps.

[0009] S102, feature refinement.

[0010] The features obtained in the data preprocessing stage are input into the feature refinement module, and the features are corrected using feature alignment and attention mechanism to obtain the corrected key frame features.

[0011] S103, semantic refinement.

[0012] The heat maps of the front and rear frames obtained in the data preprocessing stage and the heat map generated by the key frame features corrected in the feature refinement stage are input into the semantic refinement module, the difference between the adjacent heat maps is used as auxiliary information to refine the key frame heat map, and the final heat map is generated.

[0013] As a further improvement of the application, S101 is specifically:

[0014] Given three consecutive frames I [t-1,t,t+1] , I t is the key frame, and the others are auxiliary frames. Use the human body detector to detect the people in the key frame I t , obtain the detection box locating the human body, the number of detection boxes is the number of personnel instances in I t , and the size of the detection box is enlarged by 25%. The image I [t-1,t,t+1] is cropped using the enlarged detection box in I t , to obtain a plurality of cropped video clips. The number of groups is equal to the number of personnel instances in I t , and each group is a cropped continuous three-frame video clip C [-1,t,t+1] . The cropped video clip C [-1,t,t+1] is input into the backbone network to generate rough features F [t-1,t,t+1] and heat map H [t-1,t,t+1] .

[0015] This process can be represented by the following formula:

[0016]

[0017] Where represents the backbone network with fixed parameters θ.

[0018] As a further improvement of the application, S102 is specifically:

[0019] (1) Feature alignment

[0020] The multi-expansion rate feature alignment module based on deformable convolution is used to align the auxiliary features with the key features. In this method, the structure of the feature alignment module is shown in Figure 2 .

[0021] F t and F t+1 are reference features and adjacent features, respectively. The module has three parallel branches.

[0022] F t and F t+1 are concatenated and convolved to get feature F is input as offset and mask of deformable convolution, F t+1 is input as x of deformable convolution, F t+1 is aligned with F t

[0023] F t , F t+1 and F are concatenated and convolved to get feature F is input as offset and mask of deformable convolution, F t+1 is input as x of deformable convolution, F t+1 is aligned with F t

[0024] F t , F t+1 and F are concatenated and convolved to get feature F is input as offset and mask of deformable convolution, F t+1 is input as x of deformable convolution, F t+1 is aligned with F t

[0025] Finally, the features aligned at different dilation rates are concatenated and aligned with the current key feature at dilation rate 1. The final aligned feature is obtained

[0026] (2) Attention fusion

[0027] First, the similarity between the key feature and the aligned auxiliary feature is calculated, then the calculated similarity is used as a weight to weight the aligned auxiliary feature, and finally the key feature and the weighted auxiliary feature are fused using convolution to generate the key frame feature after feature refinement.

[0028] We first calculate the similarity between the key feature and the aligned auxiliary feature as follows:

[0029] S(F t+1 , F t ) = sigmoid(f θ1 ​​​(F t+1 ) T *f θ2 (F t ))

[0030] where f θ1 and f θ2 are common convolutional layers. The sigmoid function limits the similarity coefficient to [0, 1].

[0031] Then, the calculated similarity coefficient S(F t+i ,F t ) is used to weight the auxiliary feature F t+i . The specific operation is as shown in the following formula:

[0032] F′ t+i = F t+i ⊙S(F t+i , F t )

[0033] where ⊙ is element-wise multiplication.

[0034] Finally, the aligned auxiliary feature and the key feature are connected and input into the convolutional layer for fusion to generate the fused key feature.

[0035] F final = Conv([F′ t+1 , F t , F′ t-1 ])

[0036] where [] is the connection operation.

[0037] As a further improvement of the present application, the S103 is specifically:

[0038] First, subtract the auxiliary heat map from the corrected key heat map, and input the result into a 3x3 convolutional block to generate difference information D = [D t,t+1 , D t,t-1 ].

[0039]

[0040]

[0041] At the same time, the auxiliary heat map is connected with the heat map generated by the refined feature to input into the convolutional layer to generate the aggregated heat map.

[0042]

[0043] Then, the difference feature and the aggregated heat map are connected to input into the convolutional layer to generate the offset feature and the mask feature.

[0044]

[0045]

[0046] where f θo and f θm are the normal convolutional layer networks that generate the offset and the mask.

[0047]

[0048] Compared with the prior art, the application uses time consistency to modify the key frame results of the feature and semantic level by combining adjacent frames, wherein the feature alignment module uses multiple expansion rates to realize feature alignment of different receptive fields, which reduces feature loss compared with general down-sampling and up-sampling operations, the fusion module uses a similarity coefficient between the aligned features and the key features to weight the auxiliary features, so as to improve the performance of the final fused features, and finally the semantic refinement module uses difference information between the heat maps as an offset of the realizable convolution to complete the refinement of the semantic level, thereby improving the effect and making progress significantly.

[0049] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and attained by means of the instrumentalities particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0050] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other accompanying drawings can be obtained by those skilled in the art without creative labor on the basis of these accompanying drawings.

[0052] Figure 1 The flow chart of the method for feature and semantic refinement network for human pose estimation is proposed for the present application.

[0053] Figure 2 The feature alignment module schematic diagram in the method for feature and semantic refinement network for human pose estimation is proposed for the present application.

[0054] Figure 3The results of the human skeleton image generated by the feature and semantic refinement network for human pose estimation proposed in the application are compared with other networks in defocus, occlusion, multi-person proximity and motion blur scenes. Figures (a), (b) and (c) are images generated by the network in the application, PoseWarper and HRnet respectively. The error recognition part is marked with a thick line circle;

[0055] Figure 4 The results of the feature and semantic refinement network for human pose estimation proposed in the application on the Posetrack2017 dataset are shown. The scenes include dynamic blur, occlusion and multiple persons;

[0056] Figure 5 The results of the feature and semantic refinement network for human pose estimation proposed in the application on the Posetrack2018 dataset are shown. The scenes include dynamic blur, occlusion and multiple persons. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0058] Embodiment one, refer to Figures 1-5 The method for feature and semantic refinement network for human pose estimation includes the following steps:

[0059] S101, data preprocessing.

[0060] The method is based on single human pose estimation. Since there are multiple persons in a general image, the image needs to be preprocessed first to crop the picture into several single person pictures. Then the key frame and the cropped pictures of the same person in the previous and subsequent frames of the key frame are taken as a group and input into the backbone network to generate features and rough heat maps.

[0061] S102, feature refinement.

[0062] The features obtained in the above data preprocessing stage are input into the feature refinement module. The features are corrected by using feature alignment and attention mechanism to obtain the corrected key frame features.

[0063] S103, semantic refinement.

[0064] The front and rear frame heat maps obtained in the data preprocessing stage and the heat map generated by the key frame features corrected in the feature refinement stage are input into the semantic refinement module, and the difference between adjacent heat maps is used as auxiliary information to refine the key frame heat map, and a final heat map is generated.

[0065] The following description part will be described in detail with respect to S101-S103.

[0066] S101 specifically includes:

[0067] Given continuous 3 frames I [t-1,t,t+1] , I t is a key frame, and the others are auxiliary frames. A human body detector is used to detect the human in the key frame I t , obtain a detection box locating the human body therein, the number of detection boxes is equal to the number of personnel instances in I t , and the size of the detection box is enlarged by 25%. The image I [t-1,t,t+1] is cropped using the enlarged detection box in I t , to obtain a plurality of groups of cropped video clips. The number of groups is equal to the number of personnel instances in I t , and each group is a cropped continuous three frames video clip C [-1,t,t+1] . The cropped video clip C [-1,t,t+1] is input into the backbone network to generate rough features F [t-1,t,t+1] and a heat map H [t-1,t,t+1] .

[0068] This process can be represented by the following formula:

[0069]

[0070] Wherein represents the backbone network with fixed parameters theta.

[0071] In the present application, deformable convolution v2 (DCN v2) is used as the deformable convolution used in the design of the two modules. Specifically, DCN v2 has multiple inputs: input features F, offset features offset for calculating each element in the deformable convolution kernel, and mask mask for calculating each offset weight. The deformable convolution kernel is composed of a sampling grid G and a sampling weight W, where G locates the feature position near the sampling point, and W weights the features at each position after sampling. The deformable convolution has k sampling positions, for example, a 3x3 kernel is defined as K=9, W∈{w1,w2,…,w9}, and G∈{(-1,-1),(-1,0),…,(1,1)}. The feature F input into the deformable convolution generates a feature F. In F, the feature value at pixel position p can be calculated as

[0072]

[0073] Where p is the pixel position of the aggregated feature, g k The sampling network is based on the offset of the kernel at the k-th position, Δo. k It is the offset of the k-th position calculated by the kernel based on the input, w k It is the basic weight of the convolution kernel at the k-th position, Δw k It is the weight of the k-th position calculated by the kernel based on the input. Δo k ,Δw k Calculated using the following formula:

[0074] Δo k =f θo (offset)

[0075] Δw k =f θw (mask)

[0076] Where f θo ,f θw It is a learnable network used to calculate kernel offsets and weights.

[0077] S102 specifically includes:

[0078] (1) Feature alignment

[0079] A feature alignment module based on deformable convolution with multiple dilation rates is used to align auxiliary features with key features. The structure of the feature alignment module in this method is shown below. Figure 2 .

[0080] F t and F t+1 These are the reference feature and the adjacent feature. This module has three parallel branches.

[0081] When the deformable convolution dilation rate is 1, F t and F t+1 Features are obtained by convolution and convolution. Will As the offset and mask inputs for deformable convolution, F t+1 As its x input, F is obtained when the expansion rate is 1. t+1 With F t Aligned features

[0082] When the deformable convolution dilation rate is 3, F t F t+1 and Connect the elements and then convolve them to obtain features. Will As the offset and mask inputs for deformable convolution, F t+1As its x input, get F t+1 With F t Aligned feature

[0083] Deformable convolution with dilation rate 5, get F t , F t+1 And Connect and get feature F As offset and mask input of deformable convolution, F t+1 As its x input, get F t+1 With F t Aligned feature

[0084] Finally, concatenate the aligned features under different dilation rates, and align them with the current key feature with dilation rate 1. Get the final aligned feature

[0085] (2) Attention fusion

[0086] First, calculate the similarity between the key feature and the aligned auxiliary feature, then weight the aligned auxiliary feature with the calculated similarity as weight, and finally use convolution to fuse the key feature and the weighted auxiliary feature to generate the key frame feature after feature refinement.

[0087] We first calculate the similarity between the key feature and the aligned auxiliary feature as follows:

[0088] S(F t+1 ,F t )=sigmoid(f θ1 (F t+1 ) T *f θ2 (F t ))

[0089] Where f θ1 and f θ2 are ordinary convolution layers. The sigmoid function limits the similarity coefficient to [0, 1].

[0090] Then, the calculated similarity coefficient S(F t+i ,F t ) is used to weight the auxiliary feature F t+i . The specific operation is as follows:

[0091] F′ t+i =F t+i ⊙S(F t+i ,F t )

[0092] where is element-wise multiplication.

[0093] Finally, the aligned auxiliary features and key features are connected and input into a convolution layer for fusion to generate fused key features.

[0094] F final = Conv([F t+1 ,F t ,F t-1 ])

[0095] where [.] is a connection operation.

[0096] S103 specifically includes:

[0097] The correction feature F final generated in the feature correction module is input into a 1x1 convolution layer to generate key features after feature correction F The corrected key heat map is used to subtract the auxiliary heat map, and the result is input into a 3x3 convolution block to generate difference information D = [D t,t+1 ,D t,t-1 ].

[0098]

[0099]

[0100] At the same time, the auxiliary heat map is connected with the heat map generated by the refinement feature, and input into a convolution layer to generate an aggregated heat map.

[0101]

[0102] Then, the difference feature and the aggregated heat map are connected and input into a convolution layer to generate an offset feature and a mask feature.

[0103]

[0104]

[0105] where f θo and f θm are ordinary convolution layer networks for generating offsets and masks. Finally, the aggregated heat map, the offset, and the mask are input into a deformable convolution to generate a semantic-level refinement heat map.

[0106]

[0107] In this embodiment, the human pose estimation algorithm needs to set a loss function as an index during training. In this method, the loss function is defined as follows:

[0108] The loss function is defined as mean square error, and the difference between the predicted heat map and the real heat map is calculated, the heat map generated by the 1x1 convolution after the feature layer is refined, and the label heat map is calculated, then the final heat map after the semantic layer is refined and the label heat map are calculated, and finally the two losses are added to obtain the final loss,

[0109]

[0110]

[0111] Loss=L1+L2

[0112] Wherein is the heat map generated by the 1x1 convolution after the feature layer is refined, is the heat map generated after the heat map layer is refined, is the label heat map, J is the number of key points, V j The key points in the label are visualized, and l2 is the L2 norm.

[0113] To sum up, the application pre-processes the multi-person picture, converts the multi-person picture into a single person picture, and inputs the continuous three segments as a group into the backbone network to generate features and a rough heat map; in the feature correction stage, the deformable convolution alignment with parallel multiple expansion rates is used instead of the general pyramid alignment structure, which reduces the error caused by the difference between the auxiliary features and the key frame, and also reduces the feature loss caused by multiple up-sampling and down-sampling; the attention mechanism is used to reduce the error caused by alignment errors; in the semantic refinement stage, the difference between the heat maps is used as auxiliary features to refine the final result, and the accuracy of human pose estimation is improved.

[0114] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for feature and semantic refinement network for human pose estimation, characterized in that, The method for the feature and semantic refinement network for human pose estimation comprises the following steps: S101, data preprocessing, the present method is based on single human pose estimation, since there are multiple people in a general image, the image needs to be preprocessed first, the picture is cropped into a plurality of single person pictures, then the key frame and the cropped pictures of the same person in the frames before and after the key frame are taken as a group and input into the backbone network to generate features and a rough heat map; S102, feature refinement, the features obtained in the data preprocessing stage are input into the feature refinement module, the features are corrected by using feature alignment and attention mechanism to obtain corrected key frame features; S103, semantic refinement, the heat maps of the frames before and after the data preprocessing stage and the heat map generated by the key frame features corrected in the feature refinement stage are input into the semantic refinement module, the difference between the adjacent heat maps is used as auxiliary information to refine the key frame heat map, and the final heat map is generated; The S101 specifically comprises: Given continuous 3 frames I [t-1,t,t+1] , I t is the key frame, the others are auxiliary frames, using human body detector to detect the people in the key frame I t , get the detection box locating the human body, the number of detection boxes is the number of personnel instances in I t , enlarge the size of the detection box by 25%, cut the image I [t-1,t,t+1] using the enlarged detection box in I t , get several groups of cropped video clips, the number of groups is equal to the number of personnel instances in I t , each group is a cropped continuous three frames video clip C [-1,t,t+1] , input the cropped video clip C [-1,t,t+1] into the backbone network to generate rough features F [t-1,t,t+1] and heat map H [t-1,t,t+1] , This process can be represented by the following formula: wherein denotes a backbone network with fixed parameters Θ; The S102 specifically comprises: (1) feature alignment The auxiliary features are aligned with the key features by using a multi-expansion rate feature alignment module based on deformable convolution, F t and F t+1 are the reference feature and the neighboring feature, respectively, the module has three parallel branches, F t and F t+1 connect and convolve to get feature F as offset and mask input to deformable convolution, F t+1 as x input, get F t+1 with F t aligned features When the deformable convolution dilation rate is 3, F t F t+1 and Connect the elements and then convolve them to obtain features. Will As the offset and mask inputs for deformable convolution, F t+1 As its x input, F is obtained when the expansion rate is 3. t+1 With F t Aligned features F t , F t+1 and connected and convolved to get feature F as offset and mask input for deformable convolution, F t+1 as its x input, get F t+1 with F t aligned feature Finally, the aligned features at different expansion rates are concatenated, and aligned with the current key features at an expansion rate of 1 to obtain the final aligned features (2) attention fusion First, the similarity between the key features and the aligned auxiliary features is calculated, then the calculated similarity is used as a weight to weight the aligned auxiliary features, finally the key features and the weighted auxiliary features are fused by using convolution to generate the key frame features after feature refinement, We first calculate the similarity between the key features and the aligned auxiliary features in the following way: S(F t+1 ,F t ) = sigmoid(f θ1 (F t+1 ) T *f θ2 (F t )) where f θ1 and f θ2 are ordinary convolutional layers, and the sigmoid function limits the similarity coefficients to [0, 1], The calculated similarity coefficient S(F t+i ,F t ) is then used to weight the auxiliary features F t+i , as follows: F' t+i = F t+i ⊙ S(F t+i , F t ) Where is the element-wise multiplication, Finally, the aligned auxiliary features and the key features are connected and input into the convolution layer for fusion to generate the fused key features, F final = Conv([F′ t+1 ,F t ,F′ t-1 ]) Where is the connection operation, The attention fusion module calculates the similarity coefficient between the key features and the aligned auxiliary features by spatial attention, enhances the perception ability of the effective information of the aligned auxiliary features, and improves the quality of the key features after fusion.

2. The method of claim 1, wherein, The S103 specifically comprises: First, the corrected key heat map is subtracted from the auxiliary heat map, and the result is input into a 3x3 convolution block to generate difference information D = [D t,t+1 ,D t,t-1 ]. At the same time, the auxiliary heat map and the heat map generated by the refined features are connected and input into the convolution layer to generate the aggregated heat map, Then the difference features and the aggregated heat map are connected and input into the convolution layer to generate the offset features and the mask features, where f θo and f θm are common convolutional layer networks that generate offsets and masks, and finally, the aggregated heatmaps, offsets, and masks are input into deformable convolutions to generate semantic-level refined heatmaps,

Images