A pose estimation method and device, electronic equipment and storage medium

Abstract

The application provides a pose estimation method and device, electronic equipment and a storage medium, wherein the pose estimation method comprises: extracting a feature map of a to-be-tested image by using a backbone network; obtaining an instance center of each target of the feature map by using an instance branch decoder; obtaining initial skeleton points of the feature map by using a pose estimation branch decoder; grouping the initial skeleton points by using the instance center to determine target skeleton points of each target; and obtaining a pose estimation of each target according to the target skeleton points of each target. The distance between the initial skeleton points and the instance center is used to obtain the target skeleton points of each target, which is conducive to reducing the misgrouping rate and improving the pose estimation accuracy of the pose estimation method.

Description

Technical Field

[0001] This application relates to the field of attitude detection technology, and more specifically, to an attitude estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Pose estimation is a task that involves estimating the pose of a target object in three-dimensional space based on its appearance and motion information in a given image or video.

[0003] Most pose estimation tasks in related technologies are based on convolutional neural networks (CNNs). However, in scenarios with poor hardware processing capabilities, the pose estimation schemes in these technologies have low accuracy in predicting the pose of the target object. Summary of the Invention

[0004] The purpose of this application is to provide an attitude estimation method, apparatus, electronic device, and storage medium to improve the attitude prediction accuracy of attitude estimation tasks.

[0005] In a first aspect, embodiments of this application provide a pose estimation method, the method comprising: extracting a feature map of a test image using a backbone network, the test image including a target whose pose is to be estimated; obtaining an instance center of each target in the feature map using an instance branch decoder; obtaining initial skeleton points of the feature map using a pose estimation branch decoder; grouping the initial skeleton points using the instance centers to determine the target skeleton points of each target; and obtaining a pose estimate of each target based on the target skeleton points of each target.

[0006] In the implementation of the above scheme, the initial skeleton points can be grouped by the distance between the initial skeleton points and the instance center and the preset distance threshold, thereby determining the target skeleton points of each target. On the one hand, the above scheme can quickly complete the grouping of target skeleton points, which is conducive to improving the running efficiency of the above pose estimation method. On the other hand, the above scheme uses the distance between the initial skeleton points and the instance center to obtain the target skeleton points of each target, which is conducive to reducing the misgrouping rate and improving the pose estimation accuracy of the above pose estimation method.

[0007] In one implementation of the first aspect, the step of using an instance branch decoder to obtain the instance center of each target in the feature map includes: using a first upsampling layer in the instance branch decoder to upsample the feature map; using a first convolutional layer in the instance branch decoder to convolve the feature map after the upsampling process by the first upsampling layer to obtain an instance center prediction map; and determining the instance center of each target in the feature map based on the instance center prediction map.

[0008] In the implementation of the above scheme, an instance branch decoder containing upsampling layers and convolutional layers is used to obtain the instance center of each target in the feature map. The accuracy of the obtained instance centers is high, thereby further improving the pose estimation accuracy of the above pose estimation method.

[0009] In one implementation of the first aspect, the step of using a pose estimation branch decoder to obtain the initial skeleton points of the feature map includes: using a second upsampling layer in the pose estimation branch decoder to upsample the feature map; using a second convolutional layer in the pose estimation branch decoder to perform convolution processing on the feature map after the upsampling processing by the second upsampling layer to obtain an initial skeleton point prediction map; and obtaining the initial skeleton points of the feature map based on the initial skeleton point prediction map.

[0010] In the implementation of the above scheme, a pose estimation branch decoder containing upsampling layers and convolutional layers is used to obtain the initial skeleton points. The accuracy of the obtained initial skeleton points is relatively high, thereby further improving the pose estimation accuracy of the above pose estimation method.

[0011] In one implementation of the first aspect, the step of using the instance center to group the initial skeleton points and determine the target skeleton point of each target includes: calculating the distance between the initial skeleton points and the instance center; grouping the initial skeleton points according to the distance and a preset distance threshold to determine the target skeleton point of each target.

[0012] In the implementation of the above scheme, the distance between the initial skeleton point and the instance center is used to obtain the target skeleton point of each target, which helps to reduce the misgrouping rate and improve the pose estimation accuracy of the above pose estimation method.

[0013] In one implementation of the first aspect, the step of extracting the feature map of the image to be tested using a backbone network includes: performing convolution processing on the image to be tested using the third convolutional layer of the backbone network; sequentially inputting the image to be tested after convolution processing by the third convolutional layer into multiple feature extraction blocks in the backbone network; and performing feature extraction operations on the input image using the feature extraction blocks to obtain the feature map of the image to be tested.

[0014] In the implementation of the above scheme, compared with the use of convolutional neural networks (CNNs) for feature extraction in related technologies, the embodiments of this application use a backbone network containing multiple feature extraction blocks to extract feature maps of the image to be tested. This enables the pose estimation method to extract more accurate image features, thereby improving the prediction accuracy of the instance branch decoder for instance centers and the prediction accuracy of the pose estimation branch decoder for initial bone points, thus improving the accuracy of the pose estimation obtained by the above pose estimation method.

[0015] In one implementation of the first aspect, the step of performing feature extraction on the input image using the feature extraction block includes: using a local feature extraction module of the feature extraction block to obtain local features of the input image; using a global feature extraction module of the feature extraction block to obtain global features of the input image based on the local features of the input image; using a fusion module of the feature extraction block to fuse the local features and the global features to obtain a first fused feature; and using the fusion module of the feature extraction block to fuse the first fused feature with the input image to obtain a second fused feature.

[0016] In the implementation of the above scheme, compared with the method of directly fusing global features with the input image of the feature extraction block in related technologies, the above scheme adopts the method of first fusing global features with local features to obtain the first fused feature, and then fusing the first fused feature with the input image of the feature extraction block to obtain the second fused feature. This fully utilizes the correlation between local features and global features, which is conducive to improving the effect of fused features, thereby further improving the pose estimation accuracy of the above pose estimation method.

[0017] In one implementation of the first aspect, before employing an instance branch decoder to obtain the instance center of each target in the feature map, the method further includes: employing a dilated convolutional network to perform dilated convolution processing on the feature map to obtain a feature map processed by dilated convolution.

[0018] In the implementation of the above scheme, the embodiments of this application use dilated convolutional networks to expand the receptive field, which is beneficial to improving the ability of the pose estimation model to understand the input image, thereby further improving the pose estimation accuracy of the above pose estimation method.

[0019] Secondly, embodiments of this application provide a pose estimation device, the device comprising: a feature map extraction unit, an instance center acquisition unit, an initial skeleton point acquisition unit, a skeleton point grouping unit, and a pose estimation acquisition unit, wherein:

[0020] The feature map extraction unit is used to extract feature maps of the image to be tested using a backbone network;

[0021] The instance center acquisition unit is used to acquire the instance center of each target in the feature map using an instance branch decoder;

[0022] The initial skeleton point acquisition unit is used to acquire the initial skeleton points of the feature map using a pose estimation branch decoder.

[0023] The skeleton point grouping unit is used to group the initial skeleton points using the instance center to determine the target skeleton points for each target.

[0024] The pose estimation acquisition unit is used to acquire the pose estimate of each target based on each target skeleton point.

[0025] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other through the communication bus; the memory stores computer program instructions that can be executed by the processor, and the computer program instructions are read and executed by the processor to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0027] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic flowchart illustrating the attitude estimation method provided in this application embodiment;

[0030] Figure 2 This is a schematic diagram of the structure of the attitude estimation model provided in the embodiments of this application;

[0031] Figure 3 This is a schematic diagram of the backbone network provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of the structure of the MobileViT block, a feature extraction block provided in an embodiment of this application.

[0033] Figure 5 This is a schematic diagram of the attitude estimation device provided in the embodiments of this application;

[0034] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of this application, and are therefore merely examples and should not be used to limit the scope of protection of this application.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0037] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0038] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0039] Currently, pose estimation tasks in related technologies are mostly based on convolutional neural networks (CNNs). However, since CNNs are quite complex, applying them to scenarios with poor hardware processing capabilities can lead to both low pose estimation efficiency and decreased accuracy in predicting the target pose.

[0040] Based on this, this application provides a pose estimation method. This method uses the distance between the initial skeleton point and the instance center to obtain the target skeleton point of each target, which helps to reduce the misgrouping rate between the target skeleton points of multiple targets and improve the pose estimation accuracy of the above pose estimation method.

[0041] The above attitude estimation method can be applied to scenarios where the hardware devices have poor processing capabilities, such as intelligent security systems for elevators. Of course, it can also be applied to scenarios where the hardware devices have good processing capabilities.

[0042] Please see Figure 1 This application provides an attitude estimation method, which includes:

[0043] Step S110: Use a backbone network to extract feature maps from the image to be tested, which includes the target whose pose needs to be estimated;

[0044] Step S120: Use the instance branch decoder to obtain the instance center of each target in the feature map;

[0045] Step S130: Use the pose estimation branch decoder to obtain the initial skeleton points of the feature map;

[0046] Step S140: Using instance centers, group the initial skeleton points and determine the target skeleton points for each target;

[0047] Step S150: Obtain the pose estimate of each target based on the target skeleton points of each target.

[0048] Before describing steps S110 to S150 in detail, the attitude estimation model used in the embodiments of this application is first described:

[0049] Please see Figure 2 The attitude estimation model 200 includes:

[0050] Backbone network 210 is used to extract feature maps from the image under test;

[0051] Instance branch decoder 220 is used to extract the instance center of each target in the feature map of the image under test;

[0052] Pose estimation branch decoder 230 is used to extract initial bone points from the feature map of the image to be tested;

[0053] Grouping module 240 is used to group the initial skeleton points using instance center and determine the target skeleton points for each target;

[0054] The pose estimation acquisition module 250 is used to obtain the pose estimate of each target based on the target skeleton points of each target.

[0055] It is understood that the pose estimation model 200 described above adopts an encoder-decoder architecture, and the backbone network 210 refers to the neural network used to extract image or video features in deep learning. In the embodiments of this application, the backbone network 210 is used as an encoder.

[0056] The following is a detailed description of steps S110 to S150:

[0057] First, step S110 will be described in detail:

[0058] As an optional implementation of the above pose estimation method, step S110 uses a backbone network to extract feature maps of the image to be tested, including: performing convolution processing on the image to be tested using the third convolutional layer of the backbone network; sequentially inputting the image to be tested after convolution processing by the third convolutional layer into multiple feature extraction blocks in the backbone network; and performing feature extraction operations on the input image using the feature extraction blocks to obtain feature maps of the image to be tested.

[0059] The backbone network 210 uses the MobileViT network (Mobile Vision Transformer). Please refer to [link / reference]. Figure 3 The backbone network 210 includes a third convolutional layer Conv3 and multiple feature extraction blocks MobileViT blocks. After the backbone network 210 performs convolution processing on the image to be tested using the third convolutional layer Conv3, the image to be tested after processing by the third convolutional layer Conv3 is sequentially input into multiple feature extraction blocks MobileViT blocks, and the feature extraction blocks MobileViT blocks perform feature extraction operations on the input image.

[0060] In the implementation of the above scheme, compared with the use of convolutional neural networks (CNNs) for feature extraction in related technologies, the embodiments of this application use a backbone network containing multiple feature extraction blocks to extract feature maps of the image to be tested. This enables the pose estimation method to extract more accurate image features, thereby improving the prediction accuracy of the instance branch decoder for instance centers and the prediction accuracy of the pose estimation branch decoder for initial bone points, thus improving the accuracy of the pose estimation obtained by the above pose estimation method.

[0061] Please see Figure 3 It is understandable that after the backbone network 210 performs convolution processing on the image to be tested using the third convolutional layer Conv3, it can pass the output of the third convolutional layer Conv3 to the first feature extraction block MobileViT block through residual blocks. Specifically, in this embodiment of the application, after the backbone network 210 performs convolution processing on the image to be tested using the third convolutional layer Conv3, it passes the output of the third convolutional layer Conv3 to the first feature extraction block MobileViT block in sequence through five residual blocks. During the passing process, the second residual block and the fifth residual block perform downsampling operations respectively. Each downsampling operation can reduce the resolution of the image to half of the original resolution.

[0062] Please see Figure 3 It is understandable that after the backbone network 210 performs feature extraction on the input image using the first feature extraction block MobileViTblock, it can pass the output of the first feature extraction block MobileViTblock to the second feature extraction block MobileViTblock through a residual block. Specifically, the output of the first feature extraction block MobileViTblock is passed to the second feature extraction block MobileViTblock through a residual block. During the passing process, the residual block performs a downsampling operation, which can reduce the resolution of the image to half of the original resolution.

[0063] Please see Figure 3 It is understandable that after the backbone network 210 performs feature extraction on the input image using the second feature extraction block MobileViTblock, it can pass the output of the second feature extraction block MobileViTblock to the third feature extraction block MobileViTblock through a residual block. Specifically, the output of the second feature extraction block MobileViTblock is passed to the third feature extraction block MobileViTblock through a residual block. During the passing process, the residual block performs a downsampling operation, which can reduce the resolution of the image to half of the original resolution.

[0064] In summary, the backbone network 210 in this embodiment includes a third convolutional layer Conv3, seven residual blocks, and three feature extraction blocks MobileViT blocks. The input image passes through a third convolutional layer Conv3, five residual blocks, a first feature extraction block MobileViT block, a residual block, a second feature extraction block MobileViT block, a residual block, and a third feature extraction block in sequence before outputting the feature map of the image to be tested.

[0065] Understandably, the aforementioned residual blocks can be inverted residual blocks, such as the inverted residual block in the MobileNet-V2 network. Figure 3 The MV2 shown is the Inverted Residual Block in the MobileNet-V2 network. For the structure of the Inverted Residual Block, please refer to the relevant technology. The embodiments in this application will not be described in detail.

[0066] in addition, Figure 3 In this context, "2X" indicates two, and L indicates the number of Transformers in the MobileViT feature extraction block.

[0067] As an optional implementation of the above pose estimation method, step S110 uses a feature extraction block to perform feature extraction on the input image, including: using the local feature extraction module of the feature extraction block to obtain local features of the input image; using the global feature extraction module of the feature extraction block to obtain global features of the input image based on the local features of the input image; using the fusion module of the feature extraction block to fuse the local features and the global features to obtain a first fused feature; and using the fusion module of the feature extraction block to fuse the first fused feature with the input image to obtain a second fused feature.

[0068] Please see Figure 4 The MobileViT feature extraction block includes:

[0069] The local feature extraction module is used to obtain local features of the input image;

[0070] The global feature extraction module is used to obtain the global features of the input image based on the local features of the input image;

[0071] The fusion module is used to fuse local features with global features to obtain a first fused feature, and to fuse the first fused feature with the input image to obtain a second fused feature.

[0072] The following sections describe each layer in the MobileViT feature extraction block in detail:

[0073] The local feature extraction module first uses a depthwise convolutional layer (DW) with an n×n kernel and a convolutional layer with a 1×1 kernel to adjust the channels and obtain local features. These local features are then passed to the global feature extraction module. The global feature extraction module uses an Unfold structure, L Transformers, and a Fold structure to obtain global features, which are then output to the fusion module. The fusion module first uses a 1×1 convolutional layer to adjust the channels to their original size. Then, it concatenates the local and global features to obtain the first fused feature. Next, it uses a 1×1 convolutional layer to adjust the channels of the first fused feature to their original size. Finally, it concatenates the adjusted first fused feature with the input image of the feature extraction block to obtain and output the second fused feature.

[0074] It should be noted that the Unfold structure, Transformer and Fold structure mentioned above are all MobileViT blocks in MobileViT networks in related technologies. For details, please refer to the related technologies. The embodiments in this application will not be repeated.

[0075] Furthermore, it should be noted that, compared to the MobileViT block in related technologies, the MobileViT block in this embodiment sets up a deep convolutional layer (DW) with an n×n kernel in the local feature extraction module. This further reduces the number of network parameters, thereby reducing the number of parameters in the pose estimation model and making it more lightweight. On the other hand, compared to the method of directly fusing global features with the input image of the feature extraction block in related technologies, this embodiment first fuses global features with local features to obtain a first fused feature, and then fuses the first fused feature with the input image of the feature extraction block to obtain a second fused feature. This fully utilizes the correlation between local features and global features, which is beneficial to improving the effect of fused features.

[0076] As an optional implementation of the above pose estimation method, before using the instance branch decoder in step S120 to obtain the instance center of each target in the feature map, the pose estimation method further includes: using a dilated convolutional network to perform dilated convolution processing on the feature map to obtain the feature map after dilated convolution processing. At this time, step S120 uses the instance branch decoder to obtain the instance center of each target in the feature map after dilated convolution processing, and step S130 uses the pose estimation branch decoder to obtain the initial skeleton points of the feature map after dilated convolution processing.

[0077] Please see Figure 2 The attitude estimation model 200 also includes:

[0078] Dilated Convolutional Network 260 is used to perform dilated convolution processing on feature maps.

[0079] In the implementation of the above scheme, the embodiments of this application use dilated convolutional networks to expand the receptive field, which is beneficial to improving the ability of the pose estimation model to understand the input image, thereby further improving the pose estimation accuracy of the above pose estimation method.

[0080] Steps S120 and S130 will be described in detail below:

[0081] It is understandable that steps S120 and S130 are used only to distinguish between different steps. Steps S120 and S130 can be performed simultaneously or sequentially. For example, step S120 can be executed first and then step S130, or step S130 can be executed first and then step S120.

[0082] As an optional implementation of the above pose estimation method, step S120 uses an instance branch decoder to obtain the instance center of each target in the feature map, including: using the first upsampling layer in the instance branch decoder to upsample the feature map; using the first convolutional layer in the instance branch decoder to convolve the feature map after the first upsampling layer to obtain an instance center prediction map; and determining the instance center of each target in the feature map based on the instance center prediction map.

[0083] Please see Figure 2 The instance branch decoder 220 includes:

[0084] The first upsampling layer 221 is used to upsample the feature map;

[0085] The first convolutional layer 222 is used to perform convolution processing on the feature map after the first upsampling layer 221, to obtain the instance center prediction map, and to determine the instance center of each target in the feature map based on the instance center prediction map, thereby obtaining the instance center prediction.

[0086] The first convolutional layer 222 can perform two convolution operations with a kernel size of 5×5 and one convolution operation with a kernel size of 1×1 on the feature map after the first upsampling layer 221, to obtain one instance center prediction map and two instance center regression maps (one instance center regression map in the x direction and the other instance center regression map in the y direction).

[0087] Understandably, instance center regression is performed to prevent overfitting of the loss predicted by instance centers during the iteration process.

[0088] In the implementation of the above scheme, an instance branch decoder containing upsampling layers and convolutional layers is used to obtain the instance center of each target in the feature map. The accuracy of the obtained instance centers is high, thereby further improving the pose estimation accuracy of the above pose estimation method.

[0089] As an optional implementation of the above pose estimation method, step S130 uses a pose estimation branch decoder to obtain the initial skeleton points of the feature map, including: using the second upsampling layer in the pose estimation branch decoder to upsample the feature map; using the second convolutional layer in the pose estimation branch decoder to convolve the feature map after the upsampling process of the second upsampling layer to obtain the initial skeleton point prediction map; and obtaining the initial skeleton points of the feature map based on the initial skeleton point prediction map.

[0090] Please see Figure 2 The pose estimation branch decoder 230 includes:

[0091] The second upsampling layer 231 is used to upsample the feature map;

[0092] The second convolutional layer 232 is used to perform convolution processing on the feature map after it has been upsampled by the second upsampling layer 231 to obtain the initial skeleton point prediction map, and to obtain the initial skeleton points of the feature map based on the initial skeleton point prediction map.

[0093] The second convolutional layer 232 can perform one convolution operation with a 5×5 kernel and one convolution operation with a 1×1 kernel on the feature map after the second upsampling layer 231, to obtain N initial skeletal point prediction maps, where N is the number of skeletal keypoints required to describe a target. The value of N can be 17, that is, 17 skeletal keypoints are used to determine a target.

[0094] In the implementation of the above scheme, a pose estimation branch decoder containing upsampling layers and convolutional layers is used to obtain the initial skeleton points. The accuracy of the obtained initial skeleton points is relatively high, thereby further improving the pose estimation accuracy of the above pose estimation method.

[0095] Understandably, the first upsampling layer 221 and the second upsampling layer 231 perform three upsampling operations on the input feature map, successively restoring the feature map to the original image size. During the upsampling process, the outputs of the three feature extraction blocks MobileViT from the backbone network 210 are obtained and concatenated to the image output of each upsampling operation, thereby enabling the pose estimation model 200 to capture contextual information at different scales.

[0096] The training methods for the above-mentioned instance branch decoder 220 and pose estimation branch decoder 230 include:

[0097] First, obtain the training dataset;

[0098] Obtain a first training dataset for training the instance branch decoder 220 and a second training dataset for training the pose estimation branch decoder 230 in the corresponding application scenario (e.g., elevator intelligent security scenario). In the first training dataset, each image is labeled with human figures and the labeled pixel coordinates are saved in a JSON file. In the second training dataset, the skeletal keypoint coordinates of each target in each image are labeled and saved in a JSON file in a certain order. Therefore, both the first and second training datasets include training images and JSON files. The difference is the data stored in the JSON file.

[0099] Next, the instance branch decoder 220 and the pose estimation branch decoder 230 are trained using the first training dataset and the second training dataset, respectively, and the network loss is calculated.

[0100] The instance center prediction part in instance branch decoder 220 and the initial skeleton point prediction part in pose estimation branch decoder 230 use the same loss function, which can be the mean squared error loss (MSE, also called L2 paradigm). The calculation method is as follows:

[0101]

[0102] In instance branch decoder 220, the loss function for instance center regression can be the mean absolute error loss (MAE, also called L1 norm), calculated as follows:

[0103]

[0104] Where m is the sample size; Y i Indicates the true label; Indicates the predicted label;

[0105] Steps S140 and S150 are described in detail below:

[0106] It is understandable that each initial skeleton point prediction map corresponds to a type of skeletal keypoint, and each initial skeleton point prediction map includes that type of skeletal keypoint for all targets. For example, if there are three targets in the image to be tested, then the initial skeleton point prediction map belonging to the left eye in the 17 initial skeleton point prediction maps will have three skeleton points, and these three skeleton points correspond to the three targets mentioned above. Therefore, step S140 in this application needs to group the initial skeleton points.

[0107] As an optional implementation of the above posture estimation method, step S140 uses instance center to group the initial skeleton points and determine the target skeleton point of each target, including: calculating the distance between the initial skeleton point and the instance center; grouping the initial skeleton points according to the distance and a preset distance threshold, and determining the target skeleton point of each target.

[0108] The initial bone points obtained in step S130 contain N bone point groups. Each bone point group only distinguishes different types of bone key points, but does not distinguish the bone points of each target. Since the initial bone points can be used, the target to which each bone point in each bone point group belongs can be determined by the distance between the initial bone point in each bone point group and the instance center, as well as the preset distance threshold, thereby determining the target bone points belonging to each target.

[0109] Understandably, the distance between the initial bone point and the instance center can be calculated using Euclidean distance, as follows:

[0110]

[0111] Where n represents the coordinate dimension, for example, n = 2 means two-dimensional coordinates; xi and y i These represent the coordinates of the i-th dimension of the instance center point and the initial skeleton point, respectively.

[0112] It is understandable that the aforementioned preset distance threshold can be determined based on the distance from the keypoint of each bone in one or more existing target skeletons to the center of the instance.

[0113] In the implementation of the above scheme, the initial skeleton points can be grouped by the distance between the initial skeleton points and the instance center and the preset distance threshold, thereby determining the target skeleton points of each target. On the one hand, the above scheme can quickly complete the grouping of target skeleton points, which is conducive to improving the running efficiency of the above pose estimation method. On the other hand, the above scheme uses the distance between the initial skeleton points and the instance center to obtain the target skeleton points of each target, which is conducive to reducing the misgrouping rate and improving the pose estimation accuracy of the above pose estimation method.

[0114] Finally, step S150 outputs the pose estimate of the target based on the target skeleton points of each target. It can be understood that the pose estimate of each target can be represented by the coordinates of the skeleton keypoints.

[0115] It should be noted that other processes may be performed between steps S110 and S150, or before step S110, or after step S150, or between the processing sub-steps in each step, or before or after them.

[0116] Please see Figure 5 Based on the same inventive concept, this application also provides a pose estimation device 300, which includes: a feature map extraction unit 310, an instance center acquisition unit 320, an initial skeleton point acquisition unit 330, a skeleton point grouping unit 340, and a pose estimation acquisition unit 350, wherein:

[0117] Feature map extraction unit 310 is used to extract feature maps of the image to be tested using a backbone network;

[0118] The instance center acquisition unit 320 is used to acquire the instance center of each target in the feature map using the instance branch decoder;

[0119] The initial skeleton point acquisition unit 330 is used to acquire the initial skeleton points of the feature map using the pose estimation branch decoder;

[0120] Skeleton point grouping unit 340 is used to group the initial skeleton points using instance centers to determine the target skeleton points for each target;

[0121] The pose estimation acquisition unit 350 is used to acquire the pose estimate of each target based on each target skeleton point.

[0122] Optionally, the instance center acquisition unit 320 is specifically used to: upsample the feature map using the first upsampling layer in the instance branch decoder; convolve the feature map after upsampling using the first convolutional layer in the instance branch decoder to obtain the instance center prediction map; and determine the instance center of each target in the feature map based on the instance center prediction map.

[0123] Optionally, the initial skeleton point acquisition unit 330 is specifically used to: upsample the feature map using the second upsampling layer in the pose estimation branch decoder; convolve the feature map after upsampling using the second convolutional layer in the pose estimation branch decoder to obtain the initial skeleton point prediction map; and obtain the initial skeleton points of the feature map based on the initial skeleton point prediction map.

[0124] Optionally, the skeleton point grouping unit 340 is specifically used to: calculate the distance between the initial skeleton point and the instance center; group the initial skeleton points according to the distance and a preset distance threshold, and determine the target skeleton point for each target.

[0125] Optionally, the feature map extraction unit 310 is specifically used to: perform convolution processing on the image to be tested using the third convolutional layer of the backbone network; input the image to be tested after convolution processing by the third convolutional layer into multiple feature extraction blocks in the backbone network in sequence, and perform feature extraction operation on the input image using the feature extraction blocks to obtain the feature map of the image to be tested.

[0126] Optionally, the feature map extraction unit 310 is specifically used to: use the local feature extraction module of the feature extraction block to obtain local features of the input image; use the global feature extraction module of the feature extraction block to obtain global features of the input image based on the local features of the input image; use the fusion module of the feature extraction block to fuse the local features and global features to obtain a first fused feature; and use the fusion module of the feature extraction block to fuse the first fused feature with the input image to obtain a second fused feature.

[0127] Optionally, the attitude estimation device 300 further includes:

[0128] The dilated convolutional unit is used to perform dilated convolution processing on the feature map using a dilated convolutional network to obtain the feature map after dilated convolution processing.

[0129] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. (Refer to...) Figure 6 The electronic device 400 includes a processor 410, a memory 420, and a communication interface 430. These components are interconnected and communicate with each other via a communication bus 440 and / or other forms of connection mechanism (not shown).

[0130] The memory 420 includes one or more (only one is shown in the figure), which may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The processor 410 and other possible components may access the memory 420 to read and / or write data therein.

[0131] Processor 410 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 410 can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a special-purpose processor, including a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0132] Communication interface 430 includes one or more (only one is shown in the figure) and can be used to communicate directly or indirectly with other devices to exchange data. For example, communication interface 430 can be an Ethernet interface; it can be a mobile communication network interface, such as an interface for 3G, 4G, or 5G networks; or it can be other types of interfaces with data transmission and reception functions.

[0133] One or more computer program instructions may be stored in memory 420, and processor 410 may read and run these computer program instructions to implement the attitude estimation method provided in the embodiments of this application and other desired functions.

[0134] Understandable. Figure 6The structure shown is for illustrative purposes only; the electronic device 400 may also include more than [other components]. Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown. Figure 6 The components shown can be implemented using hardware, software, or a combination thereof. For example, electronic device 400 can be a single server (or other device with computing power), a combination of multiple servers, a cluster of a large number of servers, etc., and can be either a physical device or a virtual device.

[0135] This application also provides a computer-readable storage medium storing computer program instructions. These instructions are read and executed by a computer's processor to perform the attitude estimation method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 6 The memory 420 in the electronic device 400.

[0136] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0137] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0139] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A pose estimation method, characterized in that, The method includes: A backbone network is used to extract feature maps from the image to be tested, which includes the target whose pose needs to be estimated. An instance branch decoder is used to obtain the instance center of each target in the feature map; The initial skeletal points of the feature map are obtained by using a pose estimation branch decoder. Using the instance center, the initial skeleton points are grouped to determine the target skeleton points for each target; Based on the target skeleton points of each target, obtain the pose estimate of each target; The step of extracting feature maps of the image under test using a backbone network includes: performing convolution processing on the image under test using the third convolutional layer of the backbone network; sequentially inputting the image under test after convolution processing by the third convolutional layer into multiple feature extraction blocks in the backbone network; and performing feature extraction operations on the input image using the feature extraction blocks to obtain the feature maps of the image under test. Before employing an instance branch decoder to obtain the instance center of each target in the feature map, the method further includes: employing a dilated convolutional network to perform dilated convolution processing on the feature map to obtain a feature map after dilated convolution processing.

2. The attitude estimation method according to claim 1, characterized in that, The step of using an instance branch decoder to obtain the instance center of each target in the feature map includes: The feature map is upsampled using the first upsampling layer in the instance branch decoder; The first convolutional layer in the instance branch decoder is used to perform convolution processing on the feature map after upsampling by the first upsampling layer to obtain the instance center prediction map. The instance center of each target in the feature map is determined based on the instance center prediction map.

3. The attitude estimation method according to claim 1, characterized in that, The step of using a pose estimation branch decoder to obtain the initial skeleton points of the feature map includes: The feature map is upsampled using the second upsampling layer in the pose estimation branch decoder; The second convolutional layer in the pose estimation branch decoder is used to perform convolution processing on the feature map after upsampling by the second upsampling layer to obtain the initial bone point prediction map. Based on the initial skeleton point prediction map, the initial skeleton points of the feature map are obtained.

4. The attitude estimation method according to claim 1, characterized in that, The step of using the instance center to group the initial skeleton points and determine the target skeleton points for each target includes: Calculate the distance between the initial skeleton point and the center of the instance; Based on the distance and a preset distance threshold, the initial skeleton points are grouped to determine the target skeleton points for each target.

5. The attitude estimation method according to claim 1, characterized in that, The step of performing feature extraction on the input image using the feature extraction block includes: The local feature extraction module of the aforementioned feature extraction block is used to obtain local features of the input image; The global feature extraction module of the aforementioned feature extraction block obtains the global features of the input image based on the local features of the input image; The fusion module of the feature extraction block is used to fuse the local features with the global features to obtain the first fused feature; The fusion module of the feature extraction block is used to fuse the first fusion feature with the input image to obtain the second fusion feature.

6. An attitude estimation device, characterized in that, The device includes: a feature map extraction unit, an instance center acquisition unit, an initial skeleton point acquisition unit, a skeleton point grouping unit, and a pose estimation acquisition unit, wherein: The feature map extraction unit is used to extract feature maps of the image to be tested using a backbone network; The instance center acquisition unit is used to acquire the instance center of each target in the feature map using an instance branch decoder; The initial skeleton point acquisition unit is used to acquire the initial skeleton points of the feature map using a pose estimation branch decoder. The skeleton point grouping unit is used to group the initial skeleton points using the instance center to determine the target skeleton points for each target. The pose estimation acquisition unit is used to acquire the pose estimate of each target based on each target skeleton point; The feature map extraction unit is specifically used to: perform convolution processing on the image to be tested using the third convolutional layer of the backbone network; input the image to be tested after convolution processing by the third convolutional layer into multiple feature extraction blocks in the backbone network in sequence; and perform feature extraction operations on the input image using the feature extraction blocks to obtain the feature map of the image to be tested. The device further includes: A dilated convolutional unit is used to perform dilated convolution processing on the feature map using a dilated convolutional network to obtain a feature map that has undergone dilated convolution processing.

7. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1 to 6 by calling the program instructions.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images