Single-target pose estimation method, device and electronic equipment

Abstract

The application provides a single-target pose estimation method and device and electronic equipment, and relates to the technical field of computer vision. The single-target pose estimation method comprises: acquiring a to-be-processed image of a target; learning image features within a receptive field range in the to-be-processed image to obtain at least one local feature map; the local feature map reflects local features of the to-be-processed image at different positions; based on the at least one local feature map, a connection between image features outside the receptive field range in the to-be-processed image is established to obtain a global feature map; through feature fusion of each local feature map and the global feature map, a key point information map is obtained; based on the key point information map, pose estimation of the target is performed to obtain a pose estimation result; the pose estimation result comprises a key point position of the target in the to-be-processed image. The application solves the technical problems of low pose estimation accuracy and high model training cost in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular, the present application relates to a single target pose estimation method and device and electronic equipment. BACKGROUND

[0002] Target pose estimation refers to locating the key points of a target from a two-dimensional image or video to accurately understand the behavior of the target. Target pose estimation has a wide range of applications in many fields and can help us better understand and control objects and movements. The accuracy of target pose estimation has an important influence on the effect of downstream tasks. For example, one important topic in modern biology is to clarify the relationship between neural activity and behavior, and pose estimation is an important step in this process. Accurate pose estimation can ensure the accuracy of behavior-neural activity analysis.

[0003] Therefore, pose estimation requires sufficient accuracy. However, current pose estimation methods are based on convolutional neural networks, and convolution operations have certain limitations. They can only capture image feature relationships within a certain range and cannot establish relationships between pixel points beyond the convolution kernel range, resulting in point drift and low accuracy of target key point detection. In addition, some pose estimation models have a large number of parameters and high training costs.

[0004] As can be seen from the above, the prior art has the technical problems of low pose estimation accuracy and high model training cost. SUMMARY

[0005] The present application provides a single target pose estimation method, device and electronic equipment, which can solve the technical problems of low pose estimation accuracy and high model training cost in related technologies. The technical solution is as follows:

[0006] According to one aspect of the present application, a single target pose estimation method includes: obtaining a target image to be processed; learning image features within a receptive field range in the image to be processed to obtain at least one local feature map; the local feature map reflects the local features of the image to be processed at different positions; based on at least one local feature map, the relationship between image features beyond the receptive field range in the image to be processed is established to obtain a global feature map; through feature fusion of each local feature map and the global feature map, a key point information map is obtained; based on the key point information map, pose estimation of the target is performed to obtain a pose estimation result; the pose estimation result includes the key point position of the target in the image to be processed.

[0007] According to an aspect of the present application, a single-target pose estimation device comprises: an image acquisition module configured to acquire a to-be-processed image of a target; a local feature map acquisition module configured to learn image features within a receptive field range of the to-be-processed image to obtain at least one local feature map; the local feature map reflects local features of the to-be-processed image at different positions; a global feature map acquisition module configured to establish a connection between image features beyond the receptive field range of the to-be-processed image based on the at least one local feature map to obtain a global feature map; a feature fusion module configured to obtain a key point information map through feature fusion of the local feature map and the global feature map; and a pose estimation module configured to perform pose estimation of the target based on the key point information map to obtain a pose estimation result; the pose estimation result comprises key point positions of the target in the to-be-processed image.

[0008] In an example embodiment, the local feature map is obtained by invoking a CNN network; the CNN network comprises at least one residual module, at least one convolutional layer, and at least one pooling layer; and the local feature map acquisition module comprises: an initial feature extraction unit configured to perform initial feature extraction on the to-be-processed image by the residual module to obtain an initial feature map; and a secondary feature extraction unit configured to perform secondary feature extraction on the initial feature map by the convolutional layer and / or the pooling layer to obtain a plurality of local feature maps.

[0009] In an example embodiment, the global feature map is obtained by invoking a Transformer network; and the global feature map acquisition module comprises: a flattening unit configured to flatten the at least one local feature map into a one-dimensional vector; and a position embedding unit configured to input the one-dimensional vector after position embedding into the Transformer network to learn a dependency relationship between long-distance pixel points in the to-be-processed image to obtain the global feature map.

[0010] In an example embodiment, the position embedding unit comprises: a fusion subunit configured to assign a corresponding position embedding vector to each element in the one-dimensional vector, and fuse the position embedding vector with the one-dimensional vector to obtain the one-dimensional vector containing position information; and a Transformer subunit configured to input the one-dimensional vector containing position information into the Transformer network, so that the Transformer network learns a relationship between pixel points at different positions in the to-be-processed image by using the position information to obtain the global feature map.

[0011] In an example embodiment, the feature fusion module comprises: an image association unit configured to associate the local feature map and the global feature map to obtain a target feature map; and a key point information map acquisition unit configured to perform convolution operation on the target feature map to obtain the key point information map.

[0012] In an example embodiment, the key point information map comprises a key point heat map and a key point position optimization map; and the pose estimation module comprises: a key point position identification unit configured to identify a key point position of the target in the to-be-processed image according to the key point heat map and the key point position optimization map to obtain the pose estimation result.

[0013] In an example embodiment, the key point position identification unit comprises: a maximum value index determination subunit configured to determine a maximum value index of each of the key point heat maps; a search subunit configured to search a value of the key point position optimization map at the maximum value index; and a key point position acquisition subunit configured to obtain the key point position of the target according to the maximum value index and the value of the key point position optimization map at the maximum value index.

[0014] In an example embodiment, the key point information map further comprises a skeleton map formed by connecting each of the key point positions according to a skeleton relationship; and the single-target pose estimation device further comprises: a category prediction module configured to predict a category of the target according to the skeleton map to obtain the category of the target.

[0015] According to an aspect of the present application, an electronic device comprises at least one processor and at least one memory, wherein the memory has stored thereon computer readable instructions; the computer readable instructions are executed by one or more of the processors, so that the electronic device implements the single-target pose estimation method as described above.

[0016] According to an aspect of the present application, a storage medium has stored thereon computer readable instructions, which are executed by one or more processors to implement the single-target pose estimation method as described above.

[0017] According to an aspect of the present application, a computer program product comprises computer readable instructions stored in a storage medium, which are read by one or more processors of an electronic device, loaded and executed, so that the electronic device implements the single-target pose estimation method as described above.

[0018] The beneficial effects brought by the technical scheme provided in the application are as follows: in the technical scheme, an image of a target is acquired, image features within a receptive field range in the image are learned to obtain a local feature map (which can reflect local features of the image to be processed at different positions), a connection between image features beyond the receptive field range in the image is learned to obtain a global feature map, key point information is obtained according to the local feature map and the global feature map, and finally, pose estimation is performed based on the key point information. Different features of the image to be processed are learned to obtain a plurality of feature maps with different focused features, the key point information is obtained by combining the feature maps, and then the pose estimation of the target is realized, which overcomes the deficiency in the prior art that only the image feature relationship within a certain range of the image to be processed can be captured, the connection between long-distance image features beyond the receptive field range is established, the point drift phenomenon is reduced, the accuracy of the pose estimation is improved, and the application scheme is simple and easy to implement, the model training amount is not high, and the training cost is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the description of the embodiments of the application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0020] Figure 1 is a schematic diagram according to the implementation environment involved in the application;

[0021] Figure 2 is a flowchart of a single-target pose estimation method according to an exemplary embodiment;

[0022] Figure 3 is Figure 2 the flowchart of step 220 in the corresponding embodiment in one embodiment;

[0023] Figure 4 is Figure 2 the flowchart of step 240 in the corresponding embodiment in one embodiment;

[0024] Figure 5 is Figure 4 the flowchart of step 420 in the corresponding embodiment in one embodiment;

[0025] Figure 6 is Figure 2 the flowchart of step 260 in the corresponding embodiment in one embodiment;

[0026] Figure 7 is Figure 2 the flowchart of step 280 in the corresponding embodiment in one embodiment;

[0027] Figure 8 is Figure 7 corresponding to step 700 in an embodiment of the flowchart of the embodiment;

[0028] Figure 9 is Figure 2 corresponding to step 280 in an embodiment of the flowchart of the embodiment;

[0029] Figure 10 is a structural schematic diagram of a single target pose estimation network according to an example embodiment;

[0030] Figure 11 is a specific implementation schematic diagram of a single target pose estimation method in an application scenario;

[0031] Figure 12 is a schematic diagram showing experimental results of the present application;

[0032] Figure 13 is a structural block diagram of a single target pose estimation device according to an example embodiment;

[0033] Figure 14 is a hardware structural diagram of a server according to an example embodiment;

[0034] Figure 15 is a structural block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0035] The embodiments of the present application are described in detail below with reference to the accompanying drawings. The embodiments described below are examples for explaining the present application and should not be construed as limiting the present application.

[0036] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an", and "the" as used herein include plural forms. It should be further understood that the use of the term "including" in the specification of the present disclosure means that the features, integers, steps, operations, elements, and / or components described in the specification exist, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" as used herein can include wireless connection or wireless coupling. The phrase "and / or" as used herein includes all or any single unit and all combinations of the associated listed items.

[0037] The following is an introduction and explanation of several terms related to this application:

[0038] The ImageNet dataset is a large-scale image dataset containing over 15 million labeled images covering more than 22,000 different categories. It is widely used in the field of deep learning for image classification, object detection, image segmentation, and other tasks.

[0039] CNN (Convolutional Neural Network) is a deep learning model that can extract features from images. Through convolution and pooling operations, CNN can extract important features from images and generate feature maps. These feature maps can be used for classification, detection, segmentation, and other tasks. Therefore, using CNN to extract features from images is a common method.

[0040] ResNet50 is a deep residual network with deeper layers and more complex structures. Compared to general convolutional neural networks, ResNet50 can better solve problems such as gradient vanishing and gradient explosion, thus better extracting high-level features. In addition, ResNet50 uses shortcut connections to help information pass faster, thus improving the efficiency and accuracy of feature extraction. Therefore, using Stack1 and Stack2 of ResNet50 for feature extraction can usually obtain better feature representation, thus improving the performance of the model.

[0041] As mentioned earlier, there are still technical problems of low pose estimation accuracy and high model training cost in the prior art.

[0042] Although the network structures used in the prior art pose estimation method are different, they essentially belong to the traditional convolutional neural network category. Limited by the size of the convolution kernel receptive field, the convolution kernel can only cover a certain range of pixels around a certain pixel point in the image, but cannot cover pixels far away from that pixel point. Therefore, the convolution operation can only capture local image features, but cannot establish relationships between pixels beyond the convolution kernel range, which can easily cause point drift, resulting in low detection accuracy of target key points and poor pose estimation results.

[0043] Some solutions have made technical improvements to promote the network to learn the relevance of key points, thereby improving network accuracy, but this often results in too many model parameters and increased training costs.

[0044] As can be seen from the above, the technical problems of low pose estimation accuracy and high model training cost in the related art still need to be solved.

[0045] To this end, the single-target pose estimation method provided in the present application can effectively improve the accuracy of pose estimation. Correspondingly, the single-target pose estimation method is applicable to a single-target pose estimation device, which can be deployed on an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a notebook computer, a server, etc. The electronic device can also be a portable mobile electronic device, for example, the electronic device includes a smart phone, a tablet computer, etc.

[0046] To make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0047] Figure 1 An illustrative diagram of an implementation environment for a single-target pose estimation method. It should be noted that this implementation environment is only an example adapted to the present application and should not be considered as providing any limitation on the use range of the present application.

[0048] The implementation environment includes a collection end 110 and a service end 130.

[0049] Specifically, the collection end 110, which can also be regarded as an image collection device, includes but is not limited to a camera, a camera, a camcorder, etc. The collection end 110 is used to obtain image data for a target to be pose estimated. For example, the collection end 110 collects images for a person to obtain an RGB image including a single person.

[0050] The service end 130 can be a desktop computer, a notebook computer, a server, etc. The service end 130 can also be a computer cluster composed of multiple servers, or even a cloud computing center composed of multiple servers. The service end 130 is used to provide background services, for example, the background services include but are not limited to single-target pose estimation, etc.

[0051] The service end 130 and the collection end 110 are pre-established in network communication connection through wired or wireless means, and the data transmission between the service end 130 and the collection end 110 is realized through the network communication connection. The transmitted data includes but is not limited to images including targets, etc.

[0052] In an application scenario, through the interaction between the collection end 110 and the service end 130, the collection end 110 shoots and collects the target to obtain a to-be-processed image of the target, and uploads the to-be-processed image to the service end 130 to request the service end 130 to provide single-target pose estimation services.

[0053] For server 130, after receiving the target image uploaded by acquisition terminal 110, it calls the target pose estimation service to extract features from different scales of the target's image to be processed, and obtains multiple multi-scale feature maps. The multi-scale feature maps are combined to obtain the target's key point information map, and the target's pose is estimated based on the key point information map. Furthermore, the skeleton map in the key point information map can also be used as the target's semantic segmentation map for target category detection.

[0054] Please see Figure 2 This application provides a single-target attitude estimation method, which is applicable to electronic devices, and the electronic device can be... Figure 1 The server 130 in the implementation environment is shown.

[0055] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0056] like Figure 2 As shown, the method may include the following steps:

[0057] Step 200: Obtain the target image to be processed.

[0058] The target refers to the single object for which pose estimation is to be performed. The target can be a single person or a single animal, such as a mouse, without any specific limitation.

[0059] Optionally, the image to be processed can be an RGB image, without any specific limitation.

[0060] In one possible implementation, the original image of the target is preprocessed to obtain the image of the target to be processed. The preprocessing method is not unique and is not specifically limited here.

[0061] In one possible implementation, different preprocessing methods can be flexibly adopted according to the actual situation. For example, when training the model, preprocessing includes image enhancement, ResNet preprocessing, and scaling; when using the model for pose estimation, preprocessing only includes ResNet preprocessing.

[0062] Image enhancement can include rotation (±25°), scaling (0.75–1.25), cropping and padding (±0.15), motion blur, and color enhancement, among others. ResNet's preprocessing methods include image resizing and image normalization. Scaling can be used to resize the target image to a size suitable for GPU processing.

[0063] The preprocessing can make the image of the target more convenient to process, improve the quality of the image, reduce noise, enhance the features of the image, and make the image better recognized and processed by a subsequent processing algorithm, thereby facilitating feature extraction in the subsequent step and improving the effect and accuracy of the pose estimation.

[0064] In step 220, image features in the receptive field range of the to-be-processed image are learned to obtain at least one local feature map.

[0065] The local feature map reflects the local features of the to-be-processed image at different positions.

[0066] In one possible implementation manner, the image region in the receptive field range is subjected to a convolution operation by sliding a convolution kernel on the to-be-processed image, and an activation function is applied to generate the local feature map, which reflects the local features of the to-be-processed image at different positions.

[0067] In one possible implementation manner, a convolutional neural network can be used to extract features of the to-be-processed image, which can learn the image features in the receptive field range of the to-be-processed image through convolution layers, pooling layers, fully connected layers, and the like, to obtain the local feature map. For example, through layer-by-layer convolution and down-sampling operations, multi-scale and multi-level local feature information of the to-be-processed image is extracted and represented, which is used for subsequent tasks.

[0068] A small convolution kernel can be used to extract small-range features, while a large convolution kernel can be used to extract large-range features, and a pooling operation can help extract main features in the image. In one possible implementation manner, in the convolutional neural network, different convolution operations and / or pooling operations are performed on the to-be-processed image to obtain multiple local feature maps with different feature expression capabilities, for example, the multiple local feature maps can be 4 times, 8 times, and 16 times down-sampling feature maps, respectively.

[0069] A convolutional neural network (CNN) can be used to obtain multiple feature maps that reserve different multiple features. Specifically, multiple convolution layers can be added in the CNN, and each convolution layer uses different convolution kernel sizes and steps to extract features of different multiples. For example, the first convolution layer uses a 3x3 convolution kernel and a step of 1, the second convolution layer uses a 5x5 convolution kernel and a step of 2, the third convolution layer uses a 7x7 convolution kernel and a step of 3, and so on. In this way, each convolution layer can extract features of different multiples, thereby obtaining multiple feature maps. In addition, a pooling layer can be added after each convolution layer to further compress the size of the feature map.

[0070] The plurality of local feature maps can be obtained by alternately performing convolution operations and pooling operations, or can be obtained using a specific convolutional neural network model (such as ResNet50), which is not specifically limited herein.

[0071] In one possible implementation, the local feature map is obtained by invoking a CNN convolutional neural network, the CNN network including at least one residual module, at least one convolutional layer, and at least one pooling layer; specifically, as shown in Figure 3 The step 220 can include the following steps:

[0072] In step 300, the initial feature extraction is performed on the image to be processed by the residual module, and an initial feature map is obtained.

[0073] The residual module is an important component of a deep residual neural network (a special type of convolutional neural network).

[0074] The basic structure of the residual module includes one or more convolutional layers and batch normalization layers (Batch Normalization), and a skip connection. Specifically, the residual module first performs a series of convolutional operations and batch normalization operations on the input, then adds the obtained feature map and the input at the element level, and finally performs a nonlinear transformation through an activation function.

[0075] Optionally, the residual module can be part of a ResNet (a type of deep residual neural network), for example, it can be stack1 and stack2 parts of ResNet50, which are not specifically limited herein.

[0076] The introduction of the residual module can effectively solve the problems of gradient vanishing and gradient explosion in deep network training, and make the network easier to train and optimize. In a traditional convolutional neural network, information is processed through unidirectional transmission between layers. However, when the depth of the network increases, the transmission of information between layers becomes difficult, resulting in a decline in network performance. The residual module introduces a skip connection, which directly adds the input to the output of the network, making it easier for the network to learn residual information.

[0077] Using the residual module to perform initial feature extraction on the image to be processed improves the efficiency and accuracy of feature extraction, and can obtain better feature representation, thereby improving the performance of the model.

[0078] In step 320, the initial feature map is subjected to secondary feature extraction by the convolutional layer and / or the pooling layer, and a plurality of local feature maps are obtained.

[0079] Based on the initial feature map, a plurality of different local feature maps are obtained through convolution layers and / or pooling layers, and each local feature map can reserve different multiples of features to improve the effect of subsequent pose estimation.

[0080] At step 240, based on the at least one local feature map, a connection between image features beyond the receptive field range in the image to be processed is established to obtain a global feature map.

[0081] In one possible implementation, the global feature map is obtained by calling a Transformer network.

[0082] The Transformer network is a sequence-to-sequence model based on an attention mechanism, which consists of an encoder and a decoder, and both the encoder and the decoder are stacked by multiple identical modules. The Transformer maps the input features to different feature spaces through a multi-head attention mechanism, calculates multiple attention distributions, and finally weights and sums multiple output features to obtain the final output feature. This structure can effectively capture the relevant information in the sequence and improve the performance of the model. Possibly, the Transformer is composed of LN (Layer Normalization), MultiHeadAttention (multi-head attention mechanism), and Dense (full connection layer).

[0083] It is worth noting that the Transformer can extract global features through a self-attention mechanism. The self-attention mechanism can perform attention calculation on each position in the input sequence to determine the relationship between the position and other positions. This allows the Transformer to capture global dependencies in the input sequence and extract global features. In addition, the Transformer also uses a multi-head attention mechanism to further enhance its global feature extraction capability.

[0084] Specifically, as shown in Figure 4 step 240 can include the following steps:

[0085] At step 400, the at least one local feature map is flattened into a one-dimensional vector.

[0086] Before entering the Transformer network, the local feature map needs to be shape-transformed to adapt to the input requirements of the Transformer. For example, a convolution operation can be used to flatten the local feature map into a one-dimensional vector, which is then input into the Transformer for processing.

[0087] Specifically, the flattening operation takes each pixel point in the feature map as a feature vector, and then connects these feature vectors to obtain a long vector.

[0088] Step 420, after position embedding of the one-dimensional vector, input the Transformer network to learn the dependence relationship of long-distance pixels in the image to be processed, and obtain the global feature map.

[0089] After converting the image into a one-dimensional vector, in order to introduce position information, the method of position embedding can be used.

[0090] By embedding the position information into the one-dimensional vector, the Transformer network can use the position embedding to understand the relationship between pixels at different positions in the image, thereby better performing feature extraction and representation learning.

[0091] As shown in Figure 5 Step 420 can include the following steps:

[0092] Step 500, assign a corresponding position embedding vector to each element in the one-dimensional vector, and fuse the position embedding vector with the one-dimensional vector to obtain a one-dimensional vector containing position information.

[0093] The elements in the one-dimensional vector refer to the pixel values at each position after the image is converted into a one-dimensional vector. During the conversion of the image into a one-dimensional vector, each pixel value is arranged in a certain order into a one-dimensional vector.

[0094] The position embedding vector can be a fixed-length vector that contains information about the position of the element in the one-dimensional vector. In this way, each element in the one-dimensional vector will have a unique position embedding vector corresponding to it.

[0095] The position embedding vector and the one-dimensional vector can be fused by element-wise addition or concatenation. Specifically, assume that the one-dimensional vector is represented as x, and the position embedding vector is represented as p. Their dimensions should be the same to allow element-wise operations.

[0096] Element-wise addition: the position embedding vector p can be added to the one-dimensional vector x element by element to obtain the processed vector. In this way, the value of the position embedding vector is added to the one-dimensional vector element at the corresponding position, thereby embedding the position information into the one-dimensional vector, and the processed vector is represented as: x' = x + p.

[0097] Element-wise concatenation: the position embedding vector p can be concatenated with the one-dimensional vector x, connecting them in the dimension. In this way, the position embedding vector is added as an additional dimension to the one-dimensional vector, so that the position information is passed to the Transformer model together with the one-dimensional vector, and the processed vector is represented as: x' = [x, p].

[0098] After processing the position embedding vector and the one-dimensional vector, the resulting one-dimensional vector x' will contain position information.

[0099] Step 520: Input a one-dimensional vector containing positional information into the Transformer network, so that the Transformer network can use the positional information to learn the relationship between pixels at different locations in the image to be processed, and obtain a global feature map.

[0100] A one-dimensional vector containing location information is passed as input to the Transformer network for feature extraction and representation learning. The Transformer network uses the location embedding vector to understand the relationship between elements at different positions in the one-dimensional vector, thereby learning the relationship between pixels at different positions in the image to be processed and obtaining a global feature map.

[0101] Step 260: By fusing the features of each local feature map with the global feature map, a key point information map is obtained.

[0102] The purpose of feature fusion is to combine different feature information to provide a more comprehensive and accurate feature representation. By fusing features from different levels or sources, image information from different levels and angles can be captured, improving the model's perceptual ability and performance. Feature fusion can also improve the model's robustness and generalization ability, and reduce redundancy and noise between features.

[0103] Feature fusion methods can include feature concatenation, feature addition, feature averaging, feature weighted fusion, attention mechanisms, and feature map concatenation.

[0104] like Figure 6 As shown, step 260 may include the following steps:

[0105] Step 600: Connect the local feature map and the global feature map to obtain the target feature map.

[0106] Local and global feature maps are features extracted using different methods, encompassing features from low to high levels and retaining different proportions of features. Each feature map focuses on different features. Concatenating local and global feature maps enriches feature representation, facilitates information transfer, improves model robustness, and effectively utilizes computational resources, thereby enhancing model performance and effectiveness.

[0107] It should be noted that before linking images, you must ensure that the images to be linked have the same size or have been scaled and cropped to make them the same size.

[0108] Step 620: Perform a convolution operation on the target feature map to obtain a key point information map.

[0109] Optionally, the key point information map can include a key point heat map, a key point position optimization map, and a target skeleton map.

[0110] At step 280, pose estimation of the target is performed based on the key point information map, and a pose estimation result is obtained.

[0111] The pose estimation result includes the key point position of the target in the to-be-processed image.

[0112] The key point information map includes a key point heat map and a key point position optimization map, as shown in Figure 7 Step 280 can include the following steps:

[0113] At step 700, the key point position of the target in the to-be-processed image is identified according to the key point heat map and the key point position optimization map, and a pose estimation result is obtained.

[0114] Under the action of the above embodiments, an image of a target is acquired, image features within a receptive field range in the image are learned to obtain a local feature map, a connection between image features beyond the receptive field range in the image is learned to obtain a global feature map, a key point information map is obtained according to the local feature map and the global feature map, and finally pose estimation is performed based on the key point information map. Different features of the to-be-processed image are learned to obtain a plurality of feature maps with different focused features, the key point information map is obtained by combining these feature maps, and then pose estimation of the target is implemented. The present application overcomes the deficiency in the prior art that only the relationship between image features within a certain range in the to-be-processed image can be captured, establishes a connection between long-distance image features beyond the receptive field range, reduces the point drift phenomenon, improves the accuracy of pose estimation, and the present application is simple and easy to implement, has low model training volume, and reduces training cost.

[0115] In an exemplary embodiment, as Figure 8 Step 700 can include the following steps:

[0116] At step 800, the maximum value index of each key point heat map is determined.

[0117] The maximum value index refers to the position or index of the maximum value in an array or matrix. It can be understood that the maximum value index is positioning information (i.e., position information).

[0118] At step 820, the value of the key point position optimization map at the maximum value index is searched.

[0119] The key point position optimization map includes a key point horizontal and vertical direction position optimization map, and the value of the key point horizontal and vertical direction position optimization map at the maximum value index is searched.

[0120] Step 840, according to the maximum value index and the key point position optimization map, the value at the maximum value index is optimized to obtain the key point position of the target.

[0121] The maximum value index of the key point heat map and the value at the maximum value index of the key point position optimization map are added, and then multiplied by a certain multiple to obtain the key point coordinates.

[0122] The certain multiple depends on the global feature map and the local feature map for feature fusion. For example, if the global feature map and the local feature map are both 8 times down-sampling feature maps, the certain multiple is 8.

[0123] In an exemplary embodiment, the key point information map further includes a skeleton map formed by connecting each key point position according to the skeleton relationship, as shown in Figure 9 After step 280, the method can further include the following steps:

[0124] Step 900, predicting the category of the target according to the skeleton map to obtain the category of the target.

[0125] The skeleton map can be used as a semantic segmentation map of the target, and the category of the target can be detected according to the semantic segmentation map.

[0126] The Faster R-CNN and the semantic segmentation map are combined to detect the category of the target. Specifically, the semantic segmentation map can be used to provide more accurate target boundary information, thereby helping the Faster R-CNN to more accurately locate and classify the target.

[0127] In a possible implementation, the target in the image can be detected by the following steps:

[0128] According to the semantic segmentation map, the pixel coordinates of the target are extracted; the extracted pixel coordinates are subjected to connected region analysis to obtain the bounding box of the target; the bounding box is filtered and optimized, such as removing overlapping bounding boxes, adjusting the size of the bounding box, etc.; according to the bounding box of the target, the target in the image is detected, and a target detection algorithm such as Faster R-CNN, YOLO, etc. can be used.

[0129] Under the action of the above embodiment, the skeleton map obtained by the single target pose estimation method is used as a semantic segmentation map to predict the category of the target, providing an application mode and further widening the application scenarios.

[0130] Please refer to Figure 10This application also provides a single-target pose estimation network to implement the single-target pose estimation method provided in this application. The single-target pose estimation network includes: CNN layer 1, MaxPool layer 2, C1 layer 3, C2 layer 4, C3 layer 5, C4 layer 6, Reshape layer 7, Patch encoder layer 8, Transformer layer 9, Concate and Conv layers 12, Detail convolution layer 11, and MLP layer 10.

[0131] CNN layer 1 consists of ResNet50 stack1 and stack2. ResNet50 is a deep residual convolutional neural network composed of multiple residual blocks. Stack1 and Stack2 are two stacked residual blocks in ResNet50. Stack1 performs preliminary feature extraction on the input feature map while reducing its spatial size. Stack2 further extracts features and further reduces the spatial size of the feature map. By using ResNet50's Stack1 and Stack2 to perform multi-level feature extraction on the preprocessed map and progressively reduce its spatial size, the model's performance can be effectively improved while reducing the number of parameters and computational cost.

[0132] MaxPool layer 2 includes maxpooling2D with stride = 2. MaxPool layer 2 is used for 2x downsampling, which reduces the feature map size while preserving important features, thereby reducing computational cost.

[0133] Layers C1 (3), C2 (4), C3 (5), and C4 (6) are all convolutions with a stride of 1. For example... Figure 8 As shown, layers C1 (3) and C2 (4) are cascaded together and located after MaxPool layer 2. This means that after max pooling, two convolutional operations with a stride of 1 are performed. These two convolutional operations with a stride of 1 increase the depth of the convolution, improving the model's expressive power and accuracy. Simultaneously, these two convolutional operations with a stride of 1 can better capture detailed information in the image, enhancing the model's feature extraction capabilities. Layers C3 (5) and C4 (6) are also cascaded together, similar to layers C1 (3) and C2 (4).

[0134] Reshape layer 7 is used to transform the shape of the feature map; Reshape layer 7, located before Patch encoder layer 8, is used to flatten the feature map into a one-dimensional vector.

[0135] The patch encoder layer 8 is used for position embedding of one-dimensional vectors. Position embedding is needed before the one-dimensional vectors enter the Transformer layer 9, and the position embedding gives each vector a position information.

[0136] The Transformer layer 9 is used for processing the one-dimensional vectors after position embedding to extract global features. The Transformer layer 9 can distinguish the different roles of the vectors in the sequence according to their position information, so as to extract global features.

[0137] The MLP layer 10 is used for further processing the global features and restoring the global features to feature maps, and deconvolution to obtain global feature maps. The MLP layer 10 includes LN (Layer Normalization), Dense (full connection layer), reshaping unit, and deconvolution.

[0138] The Detail convolution layer 11 includes: conv2D, stride = 2, Padding = same; conv2D, stride = 1, Padding = same; BN (Batch normalization), Activation = Relu. The Detail convolution layer 11 can realize 2 times down-sampling.

[0139] The Concate and Conv layer 12 is used for splicing the second type of feature maps and the global feature maps, and performing convolution operation on the target feature maps obtained by splicing. The convolution operation in the Concate and Conv layer 12 includes: conv2D*2, Padding = same; conv2D, stride = 1.

[0140] In the embodiment, the second type of feature maps includes the feature maps output by the Detail convolution layer 11 and the feature maps output by the C2 layer 4, and the global feature maps include the feature maps output by the MLP layer 10.

[0141] Possibly, the Transformer layer 9 is composed of LN (Layer Normalization), MultiHeadAttention (multi-head attention mechanism), and Dense (full connection layer).

[0142] In a possible implementation, the single-target pose estimation network further comprises an Intermediate supervision unit 13 for adding an additional supervision signal in the single-target pose estimation network to help the single-target pose estimation network to better learn and adjust. The supervision signal can be the intermediate layer output of the prediction result, or the output of other related tasks.

[0143] Through the above embodiment, the network structure of the scheme is simple, and the parameter quantity is not large, so that the training cost can be reduced; the finally spliced feature map has three groups, which can be considered as three groups of features to be spliced obtained by three-way feature calculation, and the features calculated by a single way are beneficial to retain the features at high resolution, therefore, the feature map output by the Detail convolution layer 11 retains 4 times the features, the feature map output by the C2 layer 4 retains 8 times the features, and the feature map output by the MLP layer 10 retains 16 times the features, and finally the 4 times, 8 times and 16 times features are combined to obtain the key point heat map, the key point position optimization map and the skeleton map of the target, so that the single target can be effectively pose estimated.

[0144] Figure 11 is a specific implementation schematic diagram of a single-target pose estimation method in an application scenario. In the application scenario, a mouse is taken as a target for pose estimation, that is, according to Figure 11 the image a of the mouse, the pose information b, c, d and e of the mouse are obtained.

[0145] The hardware configuration of the application scenario is: i9 12900K, 4090GPU, 5200MHz DDR5 32G*4 memory, 2TBSUMSUNG 980PRO solid state disk; the environment configuration is: window 11 professional version 22H2, programming is performed using python3.9 language, network building is performed based on tensorflow2.9.3, and image data enhancement is performed using imgaug.

[0146] After the mouse image is collected by using a collection device (such as a CMOS camera), the mouse image is subjected to image enhancement, specifically, rotation (±25°), scaling (0.75-1.25), cropping and padding (±0.15), motion blur, and color enhancement, to obtain the mouse image a.

[0147] The mouse image a is preprocessed according to the preprocessing mode of resnet, and then the mouse image a is scaled to a picture size suitable for GPU processing (the resolution of 640*480 is used in the application scenario), and then the scaled mouse image a is sent to the stack1 and stack2 parts of the ResNet50 pre-trained on the imagenet dataset, to obtain the 4 times down-sampled intermediate feature map F4x (160x120); on F 4x perform max pooling to obtain an 8 times down-sampling feature map F 8x_0 (80x60); on the other hand, F 4x perform a convolution operation with a step size of 2 (perform down-sampling) to obtain a high-resolution detail feature map F 8x_1 ; on F 8x_0 perform two convolution operations with a step size of 1 to obtain F 8x_2 ; on F 8x_2 perform max pooling and two convolution operations with a step size of 1, reshape to a one-dimensional vector and perform position embedding, and then input into 6 Transformer blocks for calculation, and then restore the obtained features to a feature map, and then deconvolve to obtain a feature map F 8x_3 ; finally, F 8x_1 , F 8x_2 , and F 8x_3 are combined to obtain F 8x (80x60), and a convolution operation is performed on F 8x to obtain the pose information of the mouse, such as Figure 11 , where b represents the sum of the 16 key point heat maps of the mouse; c and d represent the sum of the horizontal position optimization map and the vertical position optimization map of the 16 key points of the mouse, respectively; and e represents the animal skeleton map.

[0148] In the above process, the processing from the scaled mouse image a to the pose information b, c, d, and e can be implemented by a single target pose estimation network as shown in Figure 10 .

[0149] Figure 11 In the above process, the pose information b is the sum of the 16 key point heat maps of the mouse; the pose information c and d are the sum of the horizontal position optimization map and the vertical position optimization map of the 16 key points of the mouse, respectively; and the pose information e is the target (mouse) skeleton map, which is obtained by connecting the target (mouse) key points.

[0150] Further, searching for the maximum value index of each key point heat map and multiplying by 8 can obtain the key point position (x, y); searching for the value of each key point horizontal and vertical position optimization map at the index and multiplying by 8, and adding (x, y) can obtain the optimized (x, y), which is the final key point position.

[0151] The present application scheme achieves better results compared to the prior art. The effects of the present application scheme are described below.

[0152] DeepLabCut and SLEAP are pose estimation techniques with similar accuracy (mean average precision, mAP), while the present application achieves a 0.5% higher mAP in the fly32 dataset. The performance difference (root mean square error, RMSE) is evaluated using a self-built mouse dataset, and the RMSE of the present application is 10% lower than that of DeepLabCut, which is the best of the two techniques. The present application significantly reduces the point fluctuation.

[0153] Please refer to Figure 12 which shows a schematic diagram of the implementation result of the present application.

[0154] Among them, a is the accuracy comparison chart of the present application scheme and other schemes (DeepLabCut and SLEAP) in single fruit fly pose estimation, it can be seen that the mAP of the present application scheme on the fruit fly dataset is 93.5±0.5%, which is higher than 92.8% and 92.7% of DeepLabCut and SLEAP.

[0155] b is the RMSE comparison chart of the present application scheme and other schemes (DeepLabCut and SLEAP) in single mouse pose estimation, wherein the RMSE of the present application scheme is 3.44±0.19, 3.73±0.20, and the RMSE of other schemes is 3.83±0.41, 4.40±0.52, it can be seen that in the mouse dataset, the RMSE of the present application scheme is lower than that of other schemes.

[0156] c is the inference speed comparison chart of the present application scheme and other schemes (DeepLabCut and SLEAP) in single mouse pose estimation, on the mouse dataset, the inference speed of the present application scheme is 90±4fps, which is lower than 106±4fps of SLEAP and higher than 44±2fps of DeepLabCut.

[0157] d and e are performance charts of the present application scheme under different training amounts, when the training data amount is about 350, the pose estimation network obtains better prediction effect.

[0158] f is a key connection diagram of the mouse used skeleton label.

[0159] g is a schematic diagram of key point inference of the present application scheme and other schemes (DeepLabCut and SLEAP) on mouse video, the results show that the present application model can greatly reduce the point fluctuation compared with DeepLabCut, and can retain more key point prediction information (pcutoff=0.6) compared with SLEAP, as shown in the box.

[0160] h is the model performance diagram of adding bone labels and baseline (without adding bone labels), as shown in the box, after adding bone labels, the slight jitter of key point prediction is reduced.

[0161] In the above experimental effect display, the accuracy, inference speed, stability and other performances of the pose estimation scheme of the present application are better than those of the prior art. In addition, when the training data amount is about 350, the pose estimation network obtains good prediction effect, which can indicate that the training amount of the pose estimation network provided by the present application is relatively low, that is, the training cost is low, that is, the present application effectively solves the technical problem of high model training cost.

[0162] The following is an embodiment of the device of the present application, which can be used to execute the single target pose estimation method and target category detection method involved in the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the single target pose estimation method and target category detection method involved in the present application.

[0163] Please refer to Figure 13 In the embodiment of the present application, a single target pose estimation device 1300 is provided, which includes but is not limited to: an image acquisition module 1310, a local feature map acquisition module 1330, a global feature map acquisition module 1350, a feature fusion module 1370, and a pose estimation module 1390.

[0164] The image acquisition module 1310 is configured to acquire a target image to be processed.

[0165] The local feature map acquisition module 1330 is configured to learn image features within a receptive field range in the image to be processed to obtain at least one local feature map.

[0166] The global feature map acquisition module 1350 is configured to establish a connection between image features beyond the receptive field range in the image to be processed based on the at least one local feature map to obtain a global feature map.

[0167] The feature fusion module 1370 is configured to obtain a key point information map through feature fusion of each local feature map and the global feature map.

[0168] The pose estimation module 1390 is configured to perform pose estimation of the target based on the key point information map to obtain a pose estimation result; the pose estimation result includes the key point position of the target in the image to be processed.

[0169] It should be noted that the single-target pose estimation device provided by the above embodiment is only used for illustrating the division of the above functional modules when performing single-target pose estimation, and in actual application, the above functions can be distributed to different functional modules according to needs, that is, the internal structure of the single-target pose estimation device is divided into different functional modules to complete all or part of the above-described functions.

[0170] In addition, the single-target pose estimation device and the single-target pose estimation method provided by the above embodiment belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiment, which will not be described here.

[0171] Figure 14 According to an exemplary embodiment, a structure of a server is shown. The server is suitable for Figure 1 The server 130 in the shown implementation environment.

[0172] It should be noted that the server is only an example suitable for the present application, and should not be considered as providing any limitation on the use range of the present application. The server should also not be interpreted as needing to rely on or must have Figure 14 One or more components in the shown exemplary server 2000.

[0173] The hardware structure of the server 2000 can vary greatly due to different configurations or performance, such as Figure 12 As shown, the server 2000 includes a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0174] Specifically, the power supply 210 is used to provide working voltage for each hardware device on the server 2000.

[0175] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. For example, performing Figure 1 The interaction between the collection end 110 and the server 130 in the shown implementation environment.

[0176] Of course, in other examples suitable for the present application, the interface 230 can further include at least one serial-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc., such as Figure 14 As shown, this is not a specific limitation on this structure.

[0177] The memory 250 can be a read-only memory, a random access memory, a magnetic disk, an optical disk, or the like, and stores resources including an operating system 251, an application program 253, and data 255, and the like. The storage manner can be temporary storage or permanent storage.

[0178] The operating system 251 is used to manage and control each hardware device on the server 2000 and the application program 253, so as to realize the operation and processing of the central processing unit 270 on the mass data 255 in the memory 250. The operating system 251 can be Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0179] The application program 253 is computer readable instructions for completing at least one specific work based on the operating system 251. The application program 253 can include at least one module (not shown), and each module can include computer readable instructions for the server 2000. For example, the single target pose estimation device can be regarded as an application program 253 deployed on the server 2000. Figure 14

[0180] The data 255 can be photos, pictures, or the like stored in a disk, and can also be an image of a target to be pose estimated, a pose estimation result, or the like, and is stored in the memory 250.

[0181] The central processing unit 270 can include one or more processors, and is configured to communicate with the memory 250 through at least one communication bus, so as to read the computer readable instructions stored in the memory 250, and then realize the operation and processing of the mass data 255 in the memory 250. For example, the single target pose estimation method is completed by the central processing unit 270 reading a series of computer readable instructions stored in the memory 250.

[0182] In addition, the present application can also be realized by hardware circuit or hardware circuit combined with software, and therefore, the realization of the present application is not limited to any specific hardware circuit, software, and combination of the two.

[0183] Please refer to Figure 15 The electronic device 4000 provided in the embodiments of the present application can include a desktop computer, a notebook computer, a server, or the like.

[0184] In Figure 15 the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0185] ​The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 can include a channel for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 Only one thick line is used in the middle, but it does not mean that there is only one bus or only one type of bus.

[0186] Optionally, the electronic device 4000 can also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that the transceiver 4004 is not limited to one in actual application, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0187] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 4001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0188] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions in the form of instructions or data structures and that can be accessed by the electronic device 4000, but is not limited thereto.

[0189] The memory 4003 stores computer readable instructions, and the processor 4001 can read the computer readable instructions stored in the memory 4003 through the communication bus 4002.

[0190] The computer readable instructions are executed by the one or more processors 4001 to implement the single target pose estimation method in the above embodiments.

[0191] In addition, the present application provides a storage medium, and the storage medium stores computer readable instructions, and the computer readable instructions are executed by one or more processors to implement the single target pose estimation method as described above.

[0192] The present application provides a computer program product, and the computer program product includes computer readable instructions stored in a storage medium, and one or more processors of an electronic device read the computer readable instructions from the storage medium, load and execute the computer readable instructions, so that the electronic device implements the single target pose estimation method as described above.

[0193] Compared with the related art, the present application can produce the following beneficial effects:

[0194] 1. An image of a target is obtained, image features within a receptive field range in the image are learned to obtain a local feature map, connections between image features beyond the receptive field range in the image are learned to obtain a global feature map, key point information is obtained according to the local feature map and the global feature map, and finally, pose estimation is performed based on the key point information map. Different features of the image to be processed are learned to obtain multiple feature maps with different focused features, the key point information map is obtained by combining these feature maps, and then the pose of the target is estimated, which overcomes the shortcomings of the prior art that can only capture the image feature relationship within a certain range of the image to be processed, establishes the connection between long-distance image features beyond the receptive field range, reduces the point drift phenomenon, improves the accuracy of pose estimation, and the present application is simple and easy to implement, the model training amount is not high, and the training cost is reduced.

[0195] 2. For the skeleton map obtained by the single target pose estimation method, the skeleton map is used as a semantic segmentation map to predict the category of the target, providing an application mode, and further widening the application scenarios.

[0196] 3. The preprocessed image is subjected to multi-level feature extraction by Stack1 and Stack2 of ResNet50, and the spatial size of the feature map is gradually reduced, which can effectively improve the performance of the model, reduce the number of parameters and the amount of calculation. The two convolution operations with a step size of 1 can increase the depth of convolution, improve the expression ability and accuracy of the model; at the same time, the two convolution operations with a step size of 1 can better capture the detailed information in the image, and enhance the ability of the model in feature extraction.

[0197] 4、obtain the image of the target, use different convolution operations and / or pooling operations to extract features from the image, obtain a plurality of feature maps with different focused features, and further use a Transformer to obtain global features, and obtain key point heat maps, key point position optimization maps, and skeleton maps and other pose information according to the feature maps obtained by using the convolutional neural network and the Transformer. The technical scheme of the present application uses a network structure combining a convolutional neural network and a Transformer, wherein the convolutional neural network is responsible for extracting the features of the target image at high resolution and reducing the image dimension, and the Transformer is responsible for processing the image features output by the convolutional neural network. The feature map obtained by the Transformer is high-level, and the feature of each pixel point is associated with the features of other pixel points in the sequence, that is, the relationship between a certain pixel point and all other pixel points can be established to obtain global features. Finally, the different feature map information obtained by the convolutional neural network and the Transformer is combined and convolved to generate a pose key point heat map, a key point position optimization map, and a body skeleton map. The network structure combining the convolutional neural network and the Transformer can learn global-local context to strengthen feature representation, overcome the limitation that the convolution operation can only capture local image features, reduce the point drift situation, and improve the accuracy of single target pose estimation. Moreover, the technical scheme of the present application is simple and easy to implement, the corresponding single target pose estimation network parameters are relatively few, and the structure is not complex, so that high-precision effects can be achieved with fewer parameters and training costs. Thus, the technical problems of low pose estimation accuracy and high model training cost in related technologies can be effectively solved.

[0198] 5、in Figure 12 The experimental effect diagram shows that the accuracy, inference speed, stability, and other performances of the pose estimation of the technical scheme of the present application are better than those of the prior art. In addition, Figure 12 It is shown that when the training data amount is about 350, the pose estimation network obtains good prediction effect, which can indicate that the training amount of the pose estimation network provided by the present application is relatively low, that is, the training cost is low, and thus the technical problem of high model training cost is effectively solved.

[0199] It should be understood that although the steps in the flowcharts of the drawings are shown in a sequential order following the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated otherwise herein, the execution of the steps is not strictly limited to the order indicated by the arrows, and can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of which is not necessarily sequential, but can be round-robin or alternating with at least some of the other steps or sub-steps or stages of other steps.

[0200] The above only describes some embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A single-target pose estimation method, characterized by, The method comprises: obtaining a to-be-processed image of a target; learning image features within a receptive field range in the to-be-processed image to obtain at least one local feature map; the local feature map reflects local features of the to-be-processed image at different positions; based on at least one of the local feature maps, the relationship between image features beyond the receptive field range in the to-be-processed image is established to obtain a global feature map; through feature fusion of each local feature map and the global feature map, a key point information map is obtained; based on the key point information map, pose estimation of the target is performed to obtain a pose estimation result; the pose estimation result includes the key point position of the target in the to-be-processed image; the key point information map includes a key point heat map and a key point position optimization map; based on the key point information map, the pose estimation of the target is performed to obtain a pose estimation result, which includes: identifying the key point position of the target in the to-be-processed image according to the key point heat map and the key point position optimization map to obtain the pose estimation result; determining the maximum value index of each key point heat map; searching the value of the key point position optimization map at the maximum value index; according to the maximum value index and the value of the key point position optimization map at the maximum value index, the key point position of the target is obtained.

2. The method of claim 1, wherein, the local feature map is obtained by calling a CNN network; the CNN network includes at least one residual module, at least one convolutional layer, and at least one pooling layer; the learning of the image features within the receptive field range in the to-be-processed image to obtain at least one local feature map includes: performing initial feature extraction on the to-be-processed image through the residual module to obtain an initial feature map; performing secondary feature extraction on the initial feature map through the convolutional layer and / or the pooling layer to obtain a plurality of local feature maps.

3. The method of claim 1, wherein, the global feature map is obtained by calling a Transformer network; based on at least one of the local feature maps, the relationship between image features beyond the receptive field range in the to-be-processed image is established to obtain a global feature map, which includes: flattening at least one of the local feature maps into a one-dimensional vector; after position embedding of the one-dimensional vector, inputting the one-dimensional vector into the Transformer network to learn the dependency relationship of long-distance pixel points in the to-be-processed image, and obtaining the global feature map.

4. The method of claim 3, wherein, after position embedding of the one-dimensional vector, inputting the one-dimensional vector into the Transformer network to learn the dependency relationship of long-distance pixel points in the to-be-processed image, and obtaining the global feature map, which includes: allocating a corresponding position embedding vector to each element in the one-dimensional vector, and fusing the position embedding vector with the one-dimensional vector to obtain the one-dimensional vector containing position information; The one-dimensional vector containing the position information is input into the Transformer network, so that the Transformer network learns the relationship between pixels at different positions in the image to be processed by using the position information, and a global feature map is obtained.

5. The method of claim 1, wherein, The feature fusion of each local feature map and the global feature map obtains a key point information map, which includes: The local feature map and the global feature map are connected by image connection to obtain a target feature map; The target feature map is subjected to a convolution operation to obtain the key point information map.

6. The method of any one of claims 1 to 5, wherein the key point information map further comprises a skeleton map formed by connecting each key point position according to a skeleton relationship; The method further comprises: According to the skeleton map, the category of the target is predicted to obtain the category of the target.

7. A single-target pose estimation apparatus characterized by comprising: It includes: An image acquisition module is configured to acquire a target image to be processed; A local feature map acquisition module is configured to learn image features within a receptive field range in the image to be processed to obtain at least one local feature map; the local feature map reflects local features of the image to be processed at different positions; A global feature map acquisition module is configured to establish a connection between image features beyond the receptive field range in the image to be processed based on at least one local feature map to obtain a global feature map; A feature fusion module is configured to fuse features of each local feature map and the global feature map to obtain a key point information map; An attitude estimation module is configured to estimate the attitude of the target based on the key point information map to obtain an attitude estimation result; the attitude estimation result includes key point positions of the target in the image to be processed; The key point information map includes a key point heat map and a key point position optimization map; the attitude estimation of the target based on the key point information map to obtain the attitude estimation result includes: identifying the key point positions of the target in the image to be processed according to the key point heat map and the key point position optimization map to obtain the attitude estimation result; The identification of the key point positions of the target in the image to be processed according to the key point heat map and the key point position optimization map to obtain the attitude estimation result includes: determining the maximum value index of each key point heat map; searching the value of the key point position optimization map at the maximum value index; and obtaining the key point positions of the target according to the maximum value index and the value of the key point position optimization map at the maximum value index.

8. An electronic device, comprising: It includes: At least one processor and at least one memory, The memory has computer readable instructions stored thereon; The computer readable instructions are executed by one or more processors, so that the electronic device implements the single target attitude estimation method of any one of claims 1 to 6.

Images