A small target pose estimation method combined with saliency detection

Abstract

This invention provides a small target pose estimation method combined with saliency detection, comprising: S1: generating a pseudo-saliency map based on labeled human or animal pose points; S2: constructing a pose estimation network based on saliency detection, including a feature extraction module, two branches for saliency detection and pose estimation, and an attention upsampling module based on depth parameterized convolution; S3: training the model using labeled human or animal pose data and the generated pseudo-saliency map; S4: using the trained model to predict the input image, including a saliency target perception stage and a target pose estimation stage. This invention decouples fine-grained small-sized object pose point prediction into two parts: fast object position perception and target block-based pose estimation. Compared to existing technologies, this invention effectively improves the accuracy of pose estimation for small-sized objects and has higher prediction efficiency and fewer model parameters than pose estimation methods based on target detection.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and computer vision technology, and specifically relates to a small target pose estimation method that combines saliency detection. Background Technology

[0002] Pose estimation refers to the detection and identification of key points of a human or animal from an input 2D image. It is a major task in the field of computer vision and has wide applications in brain-computer interfaces, autonomous driving, and surveillance and security.

[0003] Traditional pose estimation methods are generally based on template matching or graph structure modeling. However, due to the limitations of the handheld features they rely on and the diversity of human and animal pose spaces, the application scenarios of these methods are very limited.

[0004] In recent years, with the development of deep learning, current mainstream pose estimation methods are all based on neural networks and have achieved good prediction results. However, the main mechanism of deep learning-based pose estimation models is as follows: first, features are extracted through a pre-trained backbone network, and then keypoint coordinates or their heatmaps are regressed through fully connected or upsampling modules. Because the backbone network performs five downsampling operations when extracting features, the length and width of the feature map output by the network are only a fraction of the input image. When the target for attitude estimation is small, it may only occupy a few pixels or one pixel in the feature map. Key points in different parts may overlap, affecting the accuracy of attitude estimation.

[0005] Furthermore, most existing pose estimation models process the features output by the backbone network through convolution in the spatial dimension, while the large amount of useful information contained in the channel dimension is not fully utilized. When the target being faced by the model is relatively small, the model is unlikely to achieve optimal prediction performance. Summary of the Invention

[0006] To address the aforementioned problems of existing attitude estimation models for small targets, this invention provides a small target attitude estimation method that incorporates saliency detection.

[0007] A small target pose estimation method combining saliency detection is proposed. The input of the network model is a two-dimensional image, and the output is the saliency map of the object in the image and the coordinates and category of each key point of the object.

[0008] This invention provides a small target pose estimation method that combines saliency detection, implemented through the following steps:

[0009] S1: Generate a pseudo-saliency map based on labeled human or animal pose points;

[0010] S2: Construct a pose estimation network based on saliency detection, which includes a feature extraction module, two branches for saliency detection and pose estimation, and an upsampling module based on deep hyperparameterized convolutional attention;

[0011] S3: Train the model using labeled human or animal posture data and generated pseudosalience maps;

[0012] S4: Use the trained model to predict the input image, which includes two parts: the salient target perception stage and the target pose estimation stage.

[0013] Step S1 specifically includes:

[0014] S11: Obtain the closed loop region of the object based on the contour connection generation method;

[0015] S12: Binarize and fill the closed-loop region;

[0016] S13: A pseudo-saliency map is obtained by performing two-dimensional convolution filtering on the filled map based on the Gaussian kernel function.

[0017] The method for obtaining the closed-loop region of an object based on contour connection specifically includes: initializing the labeled keypoints into three parts: a sequence of salient target contour points P. s {*}, the current search node p i and candidate point set P k |*|. Then, based on the points in the candidate point set and the current search node p... i The ranking score of each candidate point is calculated by weighting the angle information between the vector formed and the connected edges and the magnitude information of the vector, and P is updated according to the score results. s {*} and p i Until the salient target contour point sequence P s The points in {*} form a closed loop, such as Figure 2 As shown.

[0018] The ranking score for each candidate point is calculated as follows:

[0019]

[0020] in, For p i The connection vector with the i-th candidate point, v0, is p. i With contour point sequence P s The connection vector of the tail point, where α is the weight value. This is a vector normalization operation.

[0021] The update of P based on the score results s {*} and p iUntil the salient target contour point sequence P s The points in {*} form a closed loop, specifically including: initializing P. s and p i For each point P in the candidate point set k Calculate the sorting score, select the point with the highest score as the optimal point p0; set the current search node p... i Added to the salient target contour point sequence P s In the middle; update the optimal point p0 to the current search node p. i From candidate point set P k Delete p0; repeat the above steps until the salient target contour point sequence P is reached. s The points in the loop form a closed loop.

[0022] Step S2 specifically includes:

[0023] S21: Construct a feature extraction module to extract deep semantic features of the input image. The input of the feature extraction module is the two-dimensional image or image patch to be estimated, and the output is its deep semantic feature map.

[0024] S22: Construct a saliency detection branch to obtain a saliency map. The saliency branch consists of an upsampling module based on depth hyperparameterized convolutional attention and a convolutional layer with a kernel size of 1*1. Its input is the feature map of the image output by the feature extraction module, and its output is the saliency map of the image.

[0025] S23: Construct a pose estimation branch to obtain a heatmap of key points of the target. The pose estimation branch consists of three attention-based upsampling modules and a convolutional layer with a kernel size of 1*1. Its input is the feature map obtained by the feature extraction module from the image patch obtained in the previous step, and the output is a set of heatmaps, where the number of heatmaps is equal to the number of key points.

[0026] Furthermore, the attention-based upsampling module consists of two parts: a deep hyperparameterized convolutional upsampling part and a deep hyperparameterized attention module. Its input is the feature map output from the previous layer. The input features are first passed through the upsampling layer and deep hyperparameterized convolution to obtain the upsampled feature map, and then fed into the deep hyperparameterized attention module to obtain the attention features. Finally, the attention features and the upsampled feature map are added pixel by pixel to obtain the output of the attention-based upsampling module.

[0027] The depth hyperparameterized convolution upsampling part consists of an upsampling layer with a stride of 2, a depth hyperparameterized convolution layer, a batch regularization layer, and a ReLU activation layer in sequence.

[0028] The aforementioned deep hyperparameterized attention module consists of a channel attention module and a spatial attention module improved based on deep hyperparameterized convolution.

[0029] Step S3 specifically includes:

[0030] S31: Generate a set of heatmaps using the labeled attitude point data as the learning target for the model's attitude estimation branch;

[0031] S31: Use the pseudo-significance map generated in step S1 as the learning target of the model significance detection branch;

[0032] S31: Constructing the total error of model training using the mean squared loss function in For attitude estimation branch error, For the saliency detection branch error, α and β are the weights of the attitude estimation branch error and the saliency detection branch error in the total error, respectively.

[0033] Step S4 specifically includes:

[0034] S41: Convert the input image into tensor format and extract deep semantic features using the feature extraction module;

[0035] S42: Input the feature map of the image into the saliency detection branch to obtain its saliency map;

[0036] S43: Binarize the saliency map according to the preset threshold, obtain the largest connected region, and crop the image patch containing the target in the original image according to the coordinates of the region range;

[0037] S44: Convert the cropped image patch into tensor format and use the feature extraction module to extract deep semantic features;

[0038] S45: Input the feature map of the image patch into the pose estimation branch to obtain a set of heatmaps;

[0039] S46: Calculate the coordinates of the point with the maximum response in each heatmap based on the non-maximum suppression method, and use them as the coordinates of the corresponding key points; S47: Obtain the corresponding key point coordinates in the input image by mapping the key point coordinates in the image patch obtained above.

[0040] The network model in this method consists of three parts: a feature extraction module, a saliency detection branch, and a pose estimation branch. The feature extraction module is a pre-trained convolutional neural network used to extract features from the input image. The saliency detection branch is mainly used to predict the saliency map of the input image and obtain image patches containing the target region based on the response values ​​in the saliency map. The pose estimation branch is mainly used to predict the key point heatmap of the object in the output image patch.

[0041] The saliency detection branch consists of a set of attention-based upsampling modules and a convolutional layer with a kernel size of 1*1. Its input is the feature map output by the feature extraction module, and the target output is the saliency map of the image, where the target region has a high response value while other regions have low response values. The range of the target region can be obtained according to a preset threshold, and the expanded bounding box of the target region can be calculated sequentially. Then, it can be mapped onto the input image and cropped to obtain an image patch containing the target region.

[0042] The pose estimation branch consists of three attention-based upsampling modules and a 1x1 convolutional layer. The first set of upsampling modules is shared with the saliency detection branch; the other two sets of upsampling modules are unique to the pose estimation branch and are used for feature processing and upsampling the feature map to the input size. Its input is the feature map obtained by the feature extraction module from the image patch obtained in the previous step, and its target output is a set of heat maps, the number of which is determined by the number of key points. The two-dimensional coordinates of each key point can be obtained by non-maximum suppression calculation.

[0043] The beneficial effects of this invention are as follows:

[0044] 1. This invention addresses the pose estimation problem of small target objects by proposing a small target pose estimation method based on saliency detection. This invention decouples the pose estimation problem of small targets into a fast object position perception based on saliency detection and a pose estimation problem of normal-sized objects, thereby improving the pose estimation accuracy of small target objects.

[0045] 2. This invention addresses the issue of labeling the target location perception module in attitude estimation tasks by proposing a pseudo-saliency map generation method based on key points. The method, based on directed graph generation, first generates a boundary line that fits the target contour, and then obtains a pseudo-saliency map through binary padding and Gaussian kernel 2D convolution filtering operations, thereby reducing the reliance on manual annotation for model training.

[0046] 3. This invention addresses the problem that traditional upsampling modules only focus on feature processing in a single spatial dimension, proposing an attention-based upsampling module. By fusing deep hyperparameterized convolution and convolutional block attention mechanisms, it enables feature analysis across multiple dimensions such as channels and space using a small number of parameters, enhancing the model's feature analysis capabilities and thus improving the network's performance in saliency detection and pose estimation. Attached Figure Description

[0047] Figure 1 This is the overall flowchart of the present invention.

[0048] Figure 2This is a schematic diagram illustrating the generation of the target contour when generating a pseudo-saliency map based on key point labels.

[0049] Figure 3 This is a network structure diagram of the model of the present invention.

[0050] Figure 4 This is a schematic diagram of a deep hyperparameterized attention mechanism.

[0051] Figure 5 This is a flowchart of the reasoning stage of the present invention. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0053] Example 1

[0054] like Figure 1 As shown, the small target pose estimation method based on salient target perception provided by this invention mainly includes the following four steps:

[0055] Step S1: Generate a pseudo-salience map based on the labeled human or animal pose points;

[0056] Step S2: Construct a small target pose estimation network based on saliency detection, which includes a feature extraction module, two branches for saliency detection and pose estimation, and an upsampling module based on deep hyperparameterized convolutional attention.

[0057] Step S3: Train the model using labeled human or animal pose data and the generated pseudosalience map;

[0058] Step S4 involves using the trained model to predict the input image, including a salient target perception stage and a target pose estimation stage.

[0059] Furthermore, step S1 specifically includes:

[0060] Step S11: Obtain the closed loop region of the object based on the contour connection generation method;

[0061] Step S12: Binarize and fill the closed-loop region;

[0062] Step S13: Perform two-dimensional convolution on the filled map based on the Gaussian kernel function to obtain the pseudo-saliency map.

[0063] Furthermore, such as Figure 2 As shown, step S11 specifically includes:

[0064] Step A1, initialize P s and p i ;

[0065] Step A2, for each point P in the candidate point set k The scores are calculated based on the ranking and are calculated as follows:

[0066]

[0067] Step A3: Select the point with the highest score as the optimal point p0;

[0068] Step A4, set the current search node p i Added to the salient target contour point sequence P s middle;

[0069] Step A5: Update the optimal point p0 to the current search node p. i ;

[0070] Step A6, from the candidate point set P k Delete p0;

[0071] Step A7, repeat steps 2 to 5 until the salient target contour point sequence P is obtained. s The points in the loop form a closed loop.

[0072] Furthermore, such as Figure 3 As shown, step S2 specifically includes:

[0073] Step S21: Construct a feature extraction module to extract deep semantic features of the input image. The input of the feature extraction module is the two-dimensional image or image patch to be estimated, and the output is its deep semantic feature map.

[0074] Step S22: Construct a saliency detection branch to obtain a saliency map. The saliency branch consists of an attention-based upsampling module and a convolutional layer with a kernel size of 1*1. Its input is the feature map of the image output by the feature extraction module, and its output is the saliency map of the image.

[0075] Step S23: Construct a pose estimation branch to obtain a heatmap of key points of the target. The pose estimation branch consists of three upsampling modules based on depth hyperparameterized convolutional attention and a convolutional layer with a kernel size of 1*1. Its input is the feature map obtained by the feature extraction module from the image patch obtained in the previous step, and the output is a set of heatmaps, where the number of heatmaps is equal to the number of key points.

[0076] Preferably, such as Figure 4As shown, the upsampling module based on deep hyperparameterized convolutional attention consists of two parts: a deep hyperparameterized convolutional upsampling part and a deep hyperparameterized attention module. Its input is the feature map output from the previous layer. The input features are first passed through the deep hyperparameterized convolutional upsampling part to obtain an upsampled feature map, which is then fed into the deep hyperparameterized attention module to obtain attention features. Finally, the attention features and the upsampled feature map are concatenated along the channel dimension as the output of the attention-based upsampling module.

[0077] Preferably, the depth hyperparameterized convolutional upsampling part is composed of an upsampling layer with a stride of 2, a depth hyperparameterized convolutional layer, a batch regularization layer, and a ReLU activation layer in sequence.

[0078] Preferably, the deep hyperparameterized attention module consists of a channel attention module and a spatial attention module improved based on deep hyperparameterized convolution.

[0079] Furthermore, step S3 specifically includes:

[0080] Step S31: Generate a set of heatmaps using the labeled attitude point data as the learning target for the model attitude estimation branch.

[0081] Step S32: Use the pseudo-saliency map generated in step 1 as the learning target of the model saliency detection branch;

[0082] Step S33: Construct the total error of model training using the mean squared loss function. in For attitude estimation branch error, For the saliency detection branch error, α and β are the weights of the attitude estimation branch error and the saliency detection branch error in the total error, respectively.

[0083] Furthermore, such as Figure 5 As shown, step S4 specifically includes:

[0084] Step S41: Convert the input image into tensor format and extract deep semantic features using the feature extraction module;

[0085] Step S42: Input the feature map of the image into the saliency detection branch to obtain its saliency map;

[0086] Step S43: Binarize the saliency map according to the preset threshold and obtain the largest connected region. Then, crop the image patch containing the target in the original image according to the coordinates of the region.

[0087] Step S44: Convert the cropped image patch into tensor format and extract deep semantic features using the feature extraction module;

[0088] Step S45: Input the feature map of the image patch into the pose estimation branch to obtain a set of heatmaps;

[0089] Step S46: Calculate the coordinates of the point with the maximum response in each heatmap based on the non-maximum suppression method, and use them as the coordinates of the corresponding key points;

[0090] Step S47: The key point coordinates in the image block calculated above are used to obtain the corresponding key point coordinate values ​​in the input image through mapping calculation.

Claims

1. A small target pose estimation method combining saliency detection, characterized in that, Includes the following steps: S1: Generate a pseudo-saliency map based on labeled human or animal pose points; S2: Construct a pose estimation network based on saliency detection, which includes three parts: feature extraction module, saliency detection branch and pose estimation branch. The saliency detection branch and pose estimation branch are composed of an upsampling module based on depth hyperparameterized convolutional attention and a convolutional layer with a kernel size of 1*1. S3: Train the model using labeled human or animal posture data and generated pseudosalience maps; S4: Use the trained model to predict the input image, which includes two parts: the salient target perception stage and the target pose estimation stage. Step S1 specifically includes: S11: Obtain the closed loop region of the object based on the contour connection generation method; S12: Binarize and fill the closed-loop region; S13: A pseudo-saliency map is obtained by performing two-dimensional convolution filtering on the filled map based on the Gaussian kernel function; The method for obtaining the closed-loop region of an object based on contour connection specifically includes: initializing the labeled keypoints into three parts: a sequence of salient target contour points P. s {*}, the current search node p i and candidate point set P k |*|, then based on the points in the candidate point set and the current search node p i The ranking score of each candidate point is calculated by weighting the angle information between the vector formed and the connected edges and the magnitude information of the vector, and P is updated according to the score results. s {*} and p i Until the salient target contour point sequence P s The points in {*} form a closed loop; The method for calculating the ranking score of each candidate point by weighting the vector formed by the candidate point and the current search node with the angle information between the vector and the connected edge, and the magnitude information of the vector, is as follows: in, For p i The connection vector with the i-th candidate point, v0, is p. i With contour point sequence P s The connection vector of the tail point, where α is the weight value. This is a vector normalization operation; The update of P based on the score results s {*} and p i Until the salient target contour point sequence P s The points in {*} form a closed loop, specifically including: initializing P. s and p i For each point P in the candidate point set k Calculate the sorting score, select the point with the highest score as the optimal point p0; set the current search node p... i Added to the salient target contour point sequence P s In the middle; update the optimal point p0 to the current search node p. i From candidate point set P k Delete p0; repeat the above steps until the salient target contour point sequence P is reached. s The points in the loop form a closed loop.

2. The small target pose estimation method combining saliency detection according to claim 1, characterized in that, Step S2 specifically includes: S21: Construct a feature extraction module to extract deep semantic features of the input image. The input of the feature extraction module is the two-dimensional image or image patch to be estimated, and the output is its deep semantic feature map. S22: Construct a saliency detection branch to obtain a saliency map. The saliency branch consists of an upsampling module based on depth hyperparameterized convolutional attention and a convolutional layer with a kernel size of 1*1. Its input is the feature map of the image output by the feature extraction module, and its output is the saliency map of the image. S23: Construct a pose estimation branch to obtain a heatmap of key points of the target. The pose estimation branch consists of three attention-based upsampling modules and a convolutional layer with a kernel size of 1*1. Its input is the feature map obtained by the feature extraction module from the image patch obtained in the previous step, and the output is a set of heatmaps, where the number of heatmaps is equal to the number of key points.

3. The small target pose estimation method combining saliency detection as described in claim 2, characterized in that, The upsampling module based on deep hyperparameterized convolutional attention consists of two parts: a deep hyperparameterized convolutional upsampling part and a deep hyperparameterized attention module. Its input is the feature map output from the previous layer. The input features are first passed through the deep hyperparameterized convolutional upsampling part to obtain an upsampled feature map, which is then fed into the deep hyperparameterized attention module to obtain attention features. Finally, the attention features and the upsampled feature map are concatenated along the channel dimension as the output of the attention-based upsampling module.

4. The small target pose estimation method combining saliency detection as described in claim 3, characterized in that, The deep hyperparameterized convolution upsampling module consists of an upsampling layer with a stride of 2, a deep hyperparameterized convolution layer, batch regularization, and a ReLU activation layer in sequence; the deep hyperparameterized attention module consists of a channel attention module and a spatial attention module based on deep hyperparameterized convolution.

5. A small target pose estimation method combining saliency detection as described in claim 1, characterized in that, Step S3 specifically includes: S31: Generate a set of heatmaps using the labeled attitude point data as the learning target for the model's attitude estimation branch; S31: Use the pseudo-significance map generated in step S1 as the learning target of the model significance detection branch; S31: Constructing the total error of model training using the mean squared loss function in For attitude estimation branch error, For the saliency detection branch error, α and β are the weights of the attitude estimation branch error and the saliency detection branch error in the total error, respectively.

6. A small target pose estimation method combining saliency detection as described in claim 1, characterized in that, Step S4 specifically includes: S41: Convert the input image into tensor format and extract deep semantic features using the feature extraction module; S42: Input the feature map of the image into the saliency detection branch to obtain its saliency map; S43: Binarize the saliency map according to the preset threshold, obtain the largest connected region, and crop the image patch containing the target in the original image according to the coordinates of the region range; S44: Convert the cropped image patch into tensor format and use the feature extraction module to extract deep semantic features; S45: Input the feature map of the image patch into the pose estimation branch to obtain a set of heatmaps; S46: Calculate the coordinates of the point with the maximum response in each heatmap based on the non-maximum suppression method, and use them as the coordinates of the corresponding key points; S47: The key point coordinates in the image patch calculated above are used to calculate the corresponding key point coordinates in the input image.

Images