A human pose estimation method based on multi-level feature fusion module

Abstract

The present invention discloses a method for human pose estimation based on a multi-level feature fusion module, comprising the following steps: S1, constructing a human pose estimation network based on a stack of multi-level feature fusion modules, comprising a feature extraction backbone subnetwork, a multi-level feature fusion module stacking subnetwork, and an output subnetwork; S2, preparing a training set, preprocessing images, creating a label heat map using training set labels, and performing data enhancement on the images; S3, training the human pose estimation network based on the stack of multi-level feature fusion modules using the training set; S4, testing and calculating indicators of the trained human pose estimation network, adjusting network structure parameters and training settings until the indicators meet the requirements, and then using the trained human pose estimation network for actual human pose estimation. The method of the present invention uses a new multi-level feature fusion module as the core structure and stacks the module multiple times to achieve an appropriate scale, thereby achieving better human pose estimation performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of human posture estimation, and in particular relates to a human posture estimation method based on a multi-level feature fusion module. Background Art

[0002] The goal of human pose estimation is to determine the position or spatial location of key points (parts / joints) of a person's body from a given image or video. Human pose estimation can be divided into four subtasks: single-person human pose estimation, multi-person human pose estimation, human pose tracking, and 3D human pose estimation.

[0003] Currently, the development of human pose estimation can be divided into deep learning-based methods and traditional non-deep learning methods. Traditional methods use manual feature extraction techniques. In recent years, deep learning-based human pose estimation methods have developed rapidly, achieving continuous breakthroughs in performance.

[0004] Most convolutional neural networks used for human pose estimation have the following structure: a backbone similar to a classification network, characterized by gradually reducing image resolution and extracting high-level features. The backbone generates feature representations with the same resolution as the input. This portion of the network typically employs a framework where feature maps are scaled down from high to low resolution and then restored to high resolution. Furthermore, multi-scale fusion techniques and relay supervision may be added. Finally, an output subnetwork predicts heatmaps or keypoint coordinates. For example, the Stacked Hourglass network (SHN) is centered around the Hourglass module, which mirrors the downsampling process (downsampling) to increase resolution, and the two processes are performed serially. Hourglass modules are stacked multiple times to improve accuracy, and relay supervision is used between modules. However, in this network structure, information propagation requires backward propagation at each feature level, resulting in long information paths and insufficient information exchange between feature levels. Furthermore, the serial structure hinders the parallel computing capabilities of GPUs. After a certain number of modules are serialized, increasing the number of stacked modules is difficult to achieve. Furthermore, the resulting large computational overhead makes it difficult to deploy and implement in practice. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and propose a human posture estimation method based on a multi-level feature fusion module.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for estimating human posture based on a multi-level feature fusion module comprises the following steps:

[0008] S1. Construct a human posture estimation network based on a multi-level feature fusion module stack, including a feature extraction subnetwork, a multi-level feature fusion module stacking subnetwork, and an output subnetwork;

[0009] S2. Prepare the training set, preprocess the images, use the training set labels to create label heatmaps for subsequent network training, and perform data augmentation on the images;

[0010] S3. Use the training set to train a human pose estimation network based on a multi-level feature fusion module stack;

[0011] S4. Test and calculate indicators of the trained human pose estimation network, adjust the network structure parameters and training settings, repeat steps S3-S4 until the indicators meet the requirements, and apply the trained human pose estimation network to actual human pose estimation.

[0012] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0013] 1. The method of the present invention achieves less computation and higher accuracy. Compared with the classic method StackedHourglass, the floating-point operation amount of the present invention is reduced by about 50%, and the key point prediction index pkh@0.5 is improved by 0.2%.

[0014] 2. In the present invention, the feature map downsampling and upsampling structures are connected in parallel. The overall network presents a multi-resolution branch parallel structure in the horizontal direction, and a multi-stage module series structure in the vertical direction. The network based on the stacking of multi-level feature fusion modules has two-dimensional scalability in the horizontal and vertical directions.

[0015] 3. Due to the existence of the highest resolution branch, the high-resolution feature map is maintained. Compared with the method of downsampling first and then upsampling to restore the resolution, the solution of the present invention effectively avoids the accuracy loss caused by quantization error.

[0016] 4. The multi-level feature fusion module of the present invention directly samples the multi-resolution feature maps to the same resolution for fusion, thereby shortening the transmission path length of information between feature levels and strengthening the information exchange between different levels of the pyramid. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method of the present invention;

[0018] Figure 2 2 is a schematic structural diagram of a multi-level feature fusion module in an embodiment;

[0019] Figure 3 2 is a schematic diagram of the structure of the multi-level feature fusion module stacking sub-network in the embodiment;

[0020] Figure 4 2 is a schematic structural diagram of a CBL module in an embodiment;

[0021] Figure 5 Schematic diagram of the structure of the CSP module in the embodiment;

[0022] Figure 6 is a schematic structural diagram of the residual module A in the embodiment;

[0023] Figure 7 Schematic diagram of the structure of the residual module B in the embodiment. DETAILED DESCRIPTION

[0024] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0025] Example

[0026] like Figure 1 As shown, the present invention provides a human posture estimation method based on a multi-level feature fusion module, comprising the following steps:

[0027] S1. Build a human posture estimation network based on multi-level feature fusion module stacking, including a feature extraction backbone subnetwork, a multi-level feature fusion module stacking subnetwork, and an output subnetwork;

[0028] In this embodiment, the feature extraction backbone subnetwork is specifically composed of two CBL (Conv-Bn-LeakyRelu, i.e., convolution-batch normalization-activation function) modules with a convolution step size of 2, which extract features while reducing the length and width of the feature map to 1 / 4 of the input; Figure 4 As shown, it is a structural diagram of the CBL module;

[0029] The output subnetwork specifically uses a 1×1 convolution to integrate channel information, and the number of output channels can be adjusted according to the actual task requirements.

[0030] The multi-level feature fusion module stacking sub-network is composed of multi-level feature fusion (MFF) modules stacked together; Figure 2 As shown in, it is a structural diagram of the multi-level feature fusion module; Figure 3 As shown in Figure 2, it is a structural diagram of the multi-level feature fusion module stacking sub-network.

[0031] In this embodiment, the sub-network is divided into multiple stages, each stage is composed of the same MFF modules stacked together; for the i-th stage, it consists of multiple MFF_n iStacking; Between two adjacent stages, for the output feature map sequence of the previous stage, the feature map with the lowest resolution is downsampled once using a CBL module with a convolution step size of 2 and the resulting new feature map is added to the sequence, thereby increasing the number of input feature maps of the next stage, and the number of internal branches of the MFF module of the next stage is also increased accordingly, that is, n i =n i-1 +1. The network structure used in this embodiment has 4 stages, where the first stage is composed of 4 residual modules A connected in series. Its module structure is as follows: Figure 6 As shown in Figure 3, the remaining three stages are composed of MFF modules, with the number being 1, 4, and 2. The last MFF module in the last stage only includes the highest resolution branch, so the sub-network finally outputs a lowest-level feature map with the same shape as the input feature map of the sub-network.

[0032] Let the multi-level feature fusion module be denoted as MFF_n. Its input is feature maps of n different levels. It has n branches inside, and each branch corresponds to a feature level. For the i-th branch, its structure includes:

[0033] n parallel sampling modules sample j，i , j = 1, 2, ... n, respectively used to process the n feature maps of the input;

[0034] Among them, j represents the feature level index, sample j，i That is, the feature map of the j-th branch is propagated to the sampling module of the i-th branch; when i>j, sample j，i It is composed of the nearest neighbor interpolation upsampling module; when i and j are equal, sample j，i It is a Cross Stage Partial (CSP) module that does not change the shape of the feature map tensor; when i < j, sample j，i It is composed of ij CBL modules with a convolution step size of 2 in series for downsampling;

[0035] The output of each branch after sampling is spliced ​​along the channel dimension and then enters a CBL module with a convolution step size of 1, that is, feature extraction and fusion of all branch information.

[0036] The calculation process of data in MFF_n is as follows:

[0037] Assume that the feature maps of multiple levels of the feature pyramid are F1, F2, ..., F n , and the corresponding resolutions are Res1, Res2, ..., Res n, number the feature maps from high to low according to resolution, and input the feature maps into the multi-level feature fusion module MFF_n, where n represents the number of resolutions of the feature map group, that is, the number of branches of the MFF module;

[0038] Each branch of MFF_n takes all multi-resolution feature maps as input, and the calculation process for the i-th branch is:

[0039] Use the sampling module sample j，i , j=1,2,...n,j≠i, process F j , so that its resolution is transformed to Res i ; For F1, F2, ..., F i-1 , that is, high resolution Res i Based on the feature map, use sample j，i Downsampling is performed; for F i , sample j，i Perform feature extraction without changing the resolution; for F i+1 , F i+2 ,...,F n , that is, the resolution is lower than F i The feature map of j，i Up-sample; then all feature maps are spliced ​​in the channel dimension, and then processed by the CSP module to obtain the output feature map F_out i , the outputs of all branches constitute a new feature map sequence F_out1, F_out2, ..., F_out n ;

[0040] The last MFF module in the last stage of the multi-level feature fusion module stacking sub-network contains only the Res1 branch, which is the highest resolution branch. At this time, the multi-level feature maps are fused into a single feature tensor F_final1 with a resolution of Res1;

[0041] The CSP module consists of k serially connected residual modules B and a 3×3 stride1 convolution. The CSP module divides the input feature map into two parts in a certain ratio in the channel dimension (1:1 in this embodiment). One part is calculated by k residual modules B, and then spliced ​​with the other part in the channel dimension to reconstruct the feature map. Finally, the output is obtained by a 3×3 stride1 convolution module. Such a module is recorded as CSP_k. In this embodiment, k is 4. Figure 5 As shown in the figure, it is a schematic diagram of the CSP module structure; Figure 7 As shown, it is a structural diagram of the residual module B;

[0042] In summary, for the input feature maps F1, F2, ..., Fn , the output feature map F_out of the MFF module i The calculation formula is:

[0043] F_out i =CSP_k i (cat(sample 1，i (F1), sample 2，i (F2), ..., sample n，i (F n )))

[0044] Among them, cat means splicing in the channel dimension;

[0045] Corresponding to input F1, F2, ..., F n , the MFF_n module will output F_out1, F_out2, ..., F_out n .

[0046] S2. Prepare the training set, preprocess the images, use the training set labels to create label heatmaps for subsequent network training, and perform data augmentation on the images; specifically:

[0047] S21, pre-processing the image, specifically:

[0048] The image is scaled and padded to a fixed resolution. Considering the actual scenario, different input image resolutions are selected according to different datasets. For the COCO dataset (original resolution 640×480), the recommended low-resolution input data is 256×192, and the recommended high-resolution input data is 384×288. For the MPII dataset, the recommended input data resolution is 256×256.

[0049] S22. Create a label heat map using the training set labels. In this embodiment, specifically:

[0050] For each sample, 17 label heat maps are produced, one for each type of key point. The pixel values ​​of the label heat map show a two-dimensional Gaussian distribution with a peak of 1 and a variance of 2. The mean (center point) is the coordinate of the key point.

[0051] S23. Perform data enhancement on the image, specifically including:

[0052] There is a 50% probability of flipping the image horizontally and a 30% probability of cropping the half-body image.

[0053] S3. Use the training set to train a human pose estimation network based on a multi-level feature fusion module stack; specifically, the following steps are included:

[0054] S31. Set network structure parameters. In this embodiment, the parameters are:

[0055] Since the human posture estimation network is based on the stacking of multi-level feature fusion modules and is stackable, the network scale is adjusted according to task requirements and actual effects, including setting the maximum resolution branch width and the number of stacked multi-level feature fusion modules in each stage. In this embodiment, the maximum resolution branch width is set to 32, and the MFF module stacking subnetwork is set to 4 stages. The first stage is composed of 4 residual modules A connected in series, and the remaining three stages are composed of multiple MFF modules connected in series. The number of MFF modules stacked in the three stages is 1, 4, and 2, respectively.

[0056] S32: Load the training image and input it into the network for forward calculation. In this embodiment, specifically:

[0057] For the MPII human pose estimation task, the number of output channels of the network is set to 17, which corresponds to the 17 target key points of the MPII human pose estimation task;

[0058] The loss is calculated for the output prediction heat map and label heat map. The loss function uses Mean Squared Error (MSE), and the formula is:

[0059]

[0060] Among them, n joint is the total number of key points, hm target，i is the label heat map of the i-th key point, hm pred，i is the predicted heat map of the i-th key point.

[0061] S33, using ADAM optimizer to update parameters;

[0062] The Adaptive Moment Estimation (ADAM) derivation formula is as follows:

[0063]

[0064] m t =β1m t-1 +(1-β1)g t

[0065]

[0066]

[0067]

[0068]

[0069] in, represents the loss function, in this case the mean square error loss is used; θ t is the parameter to be updated at time t, m t is the first-order moment estimate of the gradient in momentum form at time t; is the first-order moment estimate after bias correction; v t Second-order moment estimation of gradient in momentum form; is the second-order moment estimate after correction of the deviation; α, β1, β2, and ε are all parameters of ADAM: α is the learning rate (0.001 in this embodiment), β1 and β2 are the exponential decay rates of the moment estimate, with values ​​between [0, 1) (0.99 and 0, respectively, in this embodiment), and ε is 1e-8.

[0070] The parameter update process using the ADAM optimizer is:

[0071] (1) Initialize α, β1, β2, ε and loss function;

[0072] (2) Initialize the parameter tensor θ0, the first-order moment vector m0, the second-order moment vector v0 and the time step t;

[0073] (3) When the parameters do not converge (the objective function value is not optimal), update each part iteratively:

[0074] t plus 1, update the gradient of the objective function with respect to the parameters at this time step Update the first-order raw moment estimate m t and the second-order raw moment estimate v t , and then calculate the bias-corrected first-order moment estimate and bias-corrected second-order moment estimates Then use the above numerical values ​​to update the model parameters θ t ;

[0075] (4) When the parameters converge, the training ends.

[0076] S34. Repeat multiple rounds of training. Once all the data in the training set is input into the network, a round of training is completed. The appropriate number of training rounds is selected based on the accuracy. For example, 240 rounds of training are the best results for the COCO and MPII datasets.

[0077] S4. Test and calculate the indicators of the trained human pose estimation network, continuously adjust the network structure parameters and training settings, repeat steps S3 and S4 until the indicators meet the requirements (this embodiment can achieve an accuracy of up to 89.4% in the MPII human pose estimation task), and use the trained human pose estimation network for actual human pose estimation; specifically:

[0078] Input the test set images into the human pose estimation network, perform non-maximum suppression on the heat map output by the network, and then take the peak point as the predicted key point coordinate;

[0079] The percentage of correct keypoints (PCK) is used as an indicator to evaluate the prediction accuracy. The formula is:

[0080]

[0081] Among them, i represents the key point with id i, k represents the index of threshold T, p represents the index of the human body on the image, d pi represents the Euclidean distance between the predicted value and the label of the key point with id i in the pth person, represents the scale factor of the pth person, the calculation method of which depends on the dataset, T k Indicates a manually set threshold;

[0082] If head length is used as the normalized reference, the index is:

[0083]

[0084] Where h is the head length of the human instance, Indicates T k PCK indicator under threshold.

[0085] As shown in Tables 1, 2, and 3 below, respectively, are a comparison table of the MPII human pose estimation performance of the method of the present invention and other methods, a comparison table of the network parameter amount and computational amount, and a comparison table of the human pose estimation performance of the COCO dataset.

[0086]

[0087] Table 1

[0088]

[0089] Table 2

[0090]

[0091] Table 3

[0092] It should also be noted that, in this specification, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or apparatus comprising the element.

[0093] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A human posture estimation method based on a multi-level feature fusion module, characterized in that: The following steps are involved: S1. Construct a human posture estimation network based on a multi-level feature fusion module stack, including a feature extraction subnetwork, a multi-level feature fusion module stacking subnetwork, and an output subnetwork; The multi-level feature fusion module stacking sub-network is composed of a stack of multi-level feature fusion modules MFF; Let the multi-level feature fusion module be denoted as MFF_n, whose input is feature maps of n different levels, and it has n branches inside, each branch corresponding to a feature level; for The structure of the i-th branch includes: n parallel sampling modules sample j，i , j = 1, 2, ..., n, respectively used to process the n feature maps of the input; Among them, j represents the feature level index, sample j，i That is, the feature map of the j-th branch is propagated to the sampling module of the i-th branch; when i>j, sample j，i It is composed of the nearest neighbor interpolation upsampling module; when i and j are equal, sample j，i is a cross-stage local module that does not change the shape of the feature map tensor; when i < j, sample j，i It is composed of ij CBL modules with a convolution step size of 2 in series for downsampling; The output of each branch after sampling is spliced ​​along the channel dimension and then enters a CBL module with a convolution step size of 1, that is, feature extraction and fusion of all branch information; S2. Prepare the training set, preprocess the images, use the training set labels to create label heatmaps for subsequent network training, and perform data augmentation on the images; S3. Use the training set to train a human pose estimation network based on a multi-level feature fusion module stack; S4. Test and calculate indicators of the trained human pose estimation network, adjust the network structure parameters and training settings, repeat steps S3-S4 until the indicators meet the requirements, and apply the trained human pose estimation network to actual human pose estimation.

2. A method for estimating human posture based on a multi-level feature fusion module according to claim 1, characterized in that: The feature extraction subnetwork is composed of two CBL modules with a convolution step of 2 in series, which extracts features while reducing the length and width of the feature map to 1 / 4 of the input; The output subnetwork specifically uses a 1×1 convolution to integrate channel information, and the number of output channels can be adjusted according to the actual task requirements.

3. A method for estimating human posture based on a multi-level feature fusion module according to claim 1, characterized in that: The calculation process of data in MFF_n is as follows: Assume that the feature maps of multiple levels of the feature pyramid are F1, F2, ..., F n , and the corresponding resolutions are Res1, Res2, ..., Res n ; Number the feature maps from high to low resolution and input them into the multi-level feature fusion module MFF_n, where n represents the number of resolutions of the feature map group, that is, the number of branches of the MFF module; Each branch of MFF_n takes all multi-resolution feature maps as input, and the calculation process for the i-th branch is: Use the sampling module sample j,i , j=1,2,...,n,j≠i, process F j , so that its resolution is transformed to Res i ; For F1, F2, ..., F i-1 , that is, high resolution Res i Based on the feature map, use sample j，i Downsampling is performed; for F i , sample j，i Perform feature extraction without changing the resolution; for F i+1 , F i+2 ,...,F n , that is, the resolution is lower than F i The feature map of j，i Up-sample; then all feature maps are spliced ​​in the channel dimension, and then processed by the CSP module to obtain the output feature map F_out i , the outputs of all branches constitute a new feature map sequence F_out1, F_out2, ..., F_out n ; The last MFF module in the last stage of the multi-level feature fusion module stacking sub-network contains only the Res1 branch, which is the highest resolution branch. At this time, the multi-level feature maps are fused into a single feature tensor F_final1 with a resolution of Res1; The CSP module consists of k serially connected residual modules B and a 3×3 stride1 convolution. The data calculation process in the CSP module is as follows: the input feature map is split into two parts in a certain ratio in the channel dimension. One part is calculated by k residual modules B. The obtained new feature map is then spliced ​​with the other part in the channel dimension to reconstruct the feature map. Finally, the output is obtained by 3×3 stride1 convolution. Such a module is denoted as CSP_k. In summary, for the input feature maps F1, F2, ..., F n , the output feature map F_out of the MFF module i The calculation formula is: F_out i =CSP_k i (cat(sample 1,i (F1),sample 2，i (F2),...,sample n,i (F n ))) Among them, cat means splicing in the channel dimension; Corresponding to input F1, F2, ..., F n , the MFF_n module will output F_out1, F_out2, ..., F_out n .

4. A method for estimating human posture based on a multi-level feature fusion module according to claim 1, characterized in that: Step S2 is specifically as follows: S21, pre-processing the image, specifically: Scale and pad the image to a fixed resolution. Considering the actual scenario, different input image resolutions are selected according to different data sets. S22. Use the training set labels to create a label heat map, specifically: For each sample, multiple label heat maps are produced, one for each type of key point. The pixel values ​​of the label heat map show a two-dimensional Gaussian distribution, and the starting mean is the key point coordinate; S23. Perform data enhancement on the image, specifically including: There is a 50% probability of flipping the image horizontally and a 30% probability of cropping the half-body image.

5. A method for estimating human posture based on a multi-level feature fusion module according to claim 1, characterized in that: Step S3 specifically includes: S31, setting network structure parameters; S32, load the training image and input it into the network for forward calculation; S33, using ADAM optimizer to update parameters; S34. Repeat multiple rounds of training.

6. A method for estimating human posture based on a multi-level feature fusion module according to claim 5, characterized in that: Step S31 is specifically as follows: Since the human pose estimation network is based on the stacking of multi-level feature fusion modules and is stackable, the network scale is adjusted according to task requirements and actual results, including setting the maximum resolution branch width and the number of stacked multi-level feature fusion modules in each stage; Step S32 is specifically as follows: For the MPII human pose estimation task, set the number of output channels of the network, which corresponds to the target key points of the MPII human pose estimation task; The loss is calculated for the output prediction heat map and label heat map. The loss function uses mean square loss, and the formula is: Among them, n joint is the total number of key points, hm target,i is the label heat map of the i-th key point, hm pred,i is the predicted heat map of the i-th key point.

7. A method for estimating human posture based on a multi-level feature fusion module according to claim 1, characterized in that: Step S4 is specifically as follows: Input the test set images into the human pose estimation network, perform non-maximum suppression on the heat map output by the network, and then take the peak point as the predicted key point coordinate; The correct key point ratio is used as an indicator to evaluate the prediction accuracy. The formula is: Among them, i represents the key point with id i, k represents the index of threshold T, p represents the index of the human body on the image, d pi represents the Euclidean distance between the predicted value and the label of the key point with id i in the pth person, represents the scale factor of the pth person, the calculation method of which depends on the dataset, T k Indicates a manually set threshold; If the head length is used as the normalized reference, the evaluation index is: Where h is the head length of the human instance, Indicates T k PCK indicator under threshold.

Images