Single-target pose estimation method and apparatus, and electronic device

Abstract

The present application provides a single-target pose estimation method and apparatus, and an electronic device. The single-target pose estimation method comprises: acquiring an image to be processed of a target; performing learning on image features in a receptive field in said image, and obtaining at least one local feature map, the local feature map reflecting local features of said image at different positions; on the basis of the at least one local feature map, establishing a relationship between image features in said image which exceed the receptive field, and obtaining a global feature map; performing feature fusion on each local feature map and the global feature map, and obtaining a key point information map; and on the basis of the key point information map, performing pose estimation of the target, and obtaining a pose estimation result, the pose estimation result comprising a key point position of the target in said image to be processed. In the present application, the technical problems in related technologies of low pose estimation accuracy and high model training costs can be solved.

Description

Single target attitude estimation method, device and electronic equipment Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a single-target posture estimation method, device, and electronic device. Background Art

[0002] Object pose estimation involves locating the key points of an object from a two-dimensional image or video to accurately understand its behavior. Object pose estimation has broad applications in many fields, helping us better understand and control objects and motion. Its accuracy significantly impacts the effectiveness of downstream tasks. For example, a key issue in modern biology is clarifying the relationship between neural activity and behavior, and pose estimation is a crucial step in this process. Accurate pose estimation ensures the precision of behavioral-neural analysis.

[0003] Therefore, pose estimation requires sufficient accuracy. However, current pose estimation methods are all based on convolutional neural networks. The convolution operation itself has certain limitations and can only capture image feature relationships within a certain range. It cannot establish relationships between pixel points beyond the range of the convolution kernel, resulting in point drift, low target key point detection accuracy, and low pose estimation accuracy. In addition, some models used for pose estimation have more parameters and high training costs.

[0004] From the above, it can be seen that the existing technology has technical problems such as low posture estimation accuracy and high model training cost.

[0005] Summary of the Invention

[0006] This application provides a single-target attitude estimation method, device, and electronic device that can solve the technical problems of low attitude estimation accuracy and high model training costs in related technologies. The technical solution is as follows:

[0007] According to one aspect of the present application, a single-target posture estimation method includes: obtaining an image to be processed of a target; learning image features within a receptive field in the image to be processed to obtain at least one local feature map; the local feature map reflects local features of the image to be processed at different positions; based on at least one local feature map, establishing a connection between image features exceeding the receptive field in the image to be processed to obtain a global feature map; obtaining a key point information map by performing feature fusion between each of the local feature maps and the global feature map; performing posture estimation of the target based on the key point information map to obtain a posture estimation result; the posture estimation result includes the key point positions of the target in the image to be processed.

[0008] According to one aspect of the present application, a single-target posture estimation device includes: an image acquisition module for acquiring an image to be processed of a target; a local feature map acquisition module for learning image features within a receptive field in the image to be processed to obtain at least one local feature map; the local feature map reflects local features of the image to be processed at different positions; a global feature map acquisition module for establishing a connection between image features exceeding the receptive field in the image to be processed based on at least one local feature map to obtain a global feature map; a feature fusion module for obtaining a key point information map by performing feature fusion of each local feature map with the global feature map; a posture estimation module for performing posture estimation of the target based on the key point information map to obtain a posture estimation result; the posture estimation result includes the key point positions of the target in the image to be processed.

[0009] In an exemplary embodiment, the local feature map is obtained by calling a CNN network; the CNN network includes at least one residual module, at least one convolutional layer, and at least one pooling layer; the local feature map acquisition module includes: an initial feature extraction unit, used to perform initial feature extraction on the image to be processed through the residual module to obtain an initial feature map; a secondary feature extraction unit, used to perform secondary feature extraction on the initial feature map through the convolutional layer and / or pooling layer to obtain multiple local feature maps.

[0010] In an exemplary embodiment, the global feature map is obtained by calling the Transformer network; the global feature map acquisition module includes: a flattening unit, used to flatten at least one of the local feature maps into a one-dimensional vector; a position embedding unit, used to perform position embedding on the one-dimensional vector and then input it into the Transformer network to learn the dependency relationship of long-distance pixel points in the image to be processed to obtain the global feature map.

[0011] In an exemplary embodiment, the position embedding unit includes: a fusion subunit, used to assign a corresponding position embedding vector to each element in the one-dimensional vector, and fuse the position embedding vector with the one-dimensional vector to obtain the one-dimensional vector containing position information; a Transformer subunit, used to input the one-dimensional vector containing position information into the Transformer network, so that the Transformer network uses the position information to learn the relationship between pixels at different positions in the image to be processed to obtain the global feature map.

[0012] In an exemplary embodiment, the feature fusion module includes: an image connection unit, used to perform image connection on the local feature map and the global feature map to obtain a target feature map; and a key point information map acquisition unit, used to perform a convolution operation on the target feature map to obtain the key point information map.

[0013] In an exemplary embodiment, the key point information map includes a key point heat map and a key point position optimization map; the posture estimation module includes: a key point position identification unit, which is used to identify the key point position of the target in the image to be processed based on the key point heat map and the key point position optimization map to obtain the posture estimation result.

[0014] In an exemplary embodiment, the key point position identification unit includes: a maximum index determination subunit, used to determine the maximum index of each key point heat map; a search subunit, used to search for the value of the key point position optimization map at the maximum index; a key point position acquisition subunit, used to obtain the key point position of the target based on the maximum index and the value of the key point position optimization map at the maximum index.

[0015] In an exemplary embodiment, the key point information map also includes a skeleton map formed by connecting the key point positions according to a skeleton relationship; the single target posture estimation device also includes: a category prediction module, which is used to predict the category of the target based on the skeleton map to obtain the category of the target.

[0016] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the single-target pose estimation method as described above.

[0017] According to one aspect of the present application, a storage medium stores computer-readable instructions thereon, and the computer-readable instructions are executed by one or more processors to implement the single-target pose estimation method as described above.

[0018] According to one aspect of the present application, a computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the single-target pose estimation method as described above.

[0019] The beneficial effects of the technical solution provided by the present application are as follows: in the above technical solution, an image of the target is acquired, the image features within the receptive field of the image are learned, and a local feature map (which can reflect the local features of the image to be processed at different positions) is obtained. Then, the connection between the image features exceeding the receptive field in the image is learned to obtain a global feature map. A key point information map is obtained based on the local feature map and the global feature map, and finally, posture estimation is performed based on the key point information map. Different features of the image to be processed are learned separately to obtain multiple feature maps with different emphasis features. These feature maps are combined to obtain a key point information map, thereby realizing posture estimation of the target. This overcomes the deficiency of the existing technical solution that can only capture the relationship between image features within a certain range in the image to be processed, establishes the connection between long-distance image features beyond the receptive field, reduces the point drift phenomenon, and improves the accuracy of posture estimation. In addition, the present application solution is simple and easy to implement, the model training volume is not high, and the training cost is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts.

[0021] FIG1 is a schematic diagram of an implementation environment according to the present application;

[0022] FIG2 is a flow chart of a single target posture estimation method according to an exemplary embodiment;

[0023] FIG3 is a flow chart of step 220 in one embodiment of the embodiment corresponding to FIG2 ;

[0024] FIG4 is a flow chart of step 240 in one embodiment of the embodiment corresponding to FIG2 ;

[0025] FIG5 is a flow chart of step 420 in one embodiment of the embodiment corresponding to FIG4 ;

[0026] FIG6 is a flow chart of step 260 in one embodiment of the embodiment corresponding to FIG2 ;

[0027] FIG7 is a flow chart of step 280 in one embodiment of the embodiment corresponding to FIG2 ;

[0028] FIG8 is a flow chart of step 700 in the embodiment corresponding to FIG7 in one embodiment;

[0029] FIG9 is a flow chart of an embodiment following step 280 in the embodiment corresponding to FIG2 ;

[0030] FIG10 is a schematic diagram showing the structure of a single target pose estimation network according to an exemplary embodiment;

[0031] FIG11 is a schematic diagram of a specific implementation of a single target posture estimation method in an application scenario;

[0032] [Corrected 20.07.2023 according to Rule 91] Figure 12a is a schematic diagram showing the comparison of the accuracy of the solution of the present invention and other solutions (Deeplabcut and Sleap) in single fruit fly posture estimation; [0032.1] [Corrected 20.07.2023 according to Rule 91] Figure 12b is a schematic diagram comparing the RMSE of the present invention and other schemes (DeepLabCut and Sleap) in single mouse posture estimation; [0032.2] [Corrected 20.07.2023 according to Rule 91] Figure 12c is a schematic diagram comparing the inference speed of the present invention and other solutions (DeepLabCut and Sleap) for single mouse posture estimation; [0032.3] [Corrected 20.07.2023 according to Rule 91] Figures 12d and 12e are schematic diagrams showing the performance of the scheme of the present invention under different training amounts; [0032.4] [Corrected 20.07.2023 according to Rule 91] Figure 12f is a diagram of the key connections of the skeletal labels used in mice; [0032.5] [Corrected 20.07.2023 according to Rule 91] Figure 12g is a schematic diagram of key point reasoning on mouse videos using the present invention and other schemes (DeepLabCut and Sleap); [0032.6] [Corrected 20.07.2023 according to Rule 91] Figure 12h is a schematic diagram of the model with and without skeleton labels (baseline);

[0033] FIG13 is a structural block diagram of a single target posture estimation device according to an exemplary embodiment;

[0034] FIG14 is a hardware structure diagram of a server according to an exemplary embodiment;

[0035] Fig. 15 is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0037] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0038] The following is an introduction and explanation of several terms involved in this application:

[0039] The ImageNet dataset is a large-scale image dataset containing over 15 million labeled images covering over 22,000 different categories. This dataset is widely used in the research and evaluation of deep learning tasks such as image classification, object detection, and image segmentation.

[0040] A CNN (convolutional neural network) is a deep learning model that can extract features from images. Through convolution and pooling operations, a CNN extracts important features from an image and generates feature maps. These feature maps can be used for tasks such as classification, detection, and segmentation. Therefore, using a CNN for feature extraction is a common method.

[0041] ResNet50 is a deep residual network with deeper layers and a more complex structure. Compared to typical convolutional neural networks, ResNet50 can better address issues such as vanishing and exploding gradients, thereby better extracting high-level features. Furthermore, ResNet50 uses shortcut connections to facilitate faster information transfer, thereby improving the efficiency and accuracy of feature extraction. Therefore, using ResNet50's Stack1 and Stack2 for feature extraction generally results in better feature representation, thereby improving model performance.

[0042] As mentioned above, the existing technology still has technical problems such as low posture estimation accuracy and high model training cost.

[0043] While existing pose estimation methods employ varying network structures, they are essentially traditional convolutional neural networks. Limited by the receptive field of the convolution kernel, the kernel can only cover pixels within a certain range around a specific pixel in the image, but cannot capture pixels farther away. Consequently, the convolution operation can only capture local image features and cannot establish relationships between pixels beyond the kernel's range. This can easily lead to pixel drift, resulting in low detection accuracy for target keypoints and poor pose estimation.

[0044] There are also plans to make technical improvements to encourage the network to learn the correlation between key points, thereby improving network accuracy, but this often results in too many model parameters and increases training costs.

[0045] From the above, we can see that the technical problems of low posture estimation accuracy and high model training cost in related technologies still need to be solved.

[0046] To this end, the single-target attitude estimation method provided in the present application can effectively improve the accuracy of attitude estimation. Accordingly, the single-target attitude estimation method is applicable to a single-target attitude estimation device, which can be deployed in an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a laptop computer, a server, etc.; the electronic device can also refer to a portable and mobile electronic device, for example, the electronic device includes a smart phone, a tablet computer, etc.

[0047] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0048] Figure 1 is a schematic diagram of an implementation environment involved in a single target pose estimation method. It should be noted that this implementation environment is only an example adapted to the present invention and should not be considered as providing any limitation on the scope of application of the present invention.

[0049] The implementation environment includes a collection end 110 and a service end 130 .

[0050] Specifically, acquisition terminal 110 can also be considered an image acquisition device, including but not limited to electronic devices with a camera function, such as a webcam, a still camera, and a video camera. Acquisition terminal 110 is used to acquire image data of an object for pose estimation. For example, acquisition terminal 110 acquires an image of a person and obtains an RGB image of the person.

[0051] Server 130 can be an electronic device such as a desktop computer, laptop computer, or server. It can also be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide background services, such as, but not limited to, single target pose estimation.

[0052] A network communication connection is pre-established between the server 130 and the acquisition terminal 110 via a wired or wireless method, and data transmission between the server 130 and the acquisition terminal 110 is achieved via the network communication connection. The transmitted data includes, but is not limited to, images of the target.

[0053] In one application scenario, through the interaction between the acquisition terminal 110 and the server 130, the acquisition terminal 110 shoots and acquires the target to obtain the target's image to be processed, and uploads the image to be processed to the server 130 to request the server 130 to provide a single target pose estimation service.

[0054] For the server 130, after receiving the target image uploaded by the acquisition end 110, it calls the target posture estimation service, extracts the different scale features of the target image to be processed, and obtains multiple multi-scale feature maps. The multi-scale feature maps are combined to obtain the key point information map of the target, and the posture of the target is estimated based on the key point information map; further, the skeleton map in the key point information map can also be used as the semantic segmentation map of the target for category detection of the target.

[0055] Please refer to FIG. 2 . An embodiment of the present application provides a single target posture estimation method. The method is applicable to an electronic device, which may be the server 130 in the implementation environment shown in FIG. 1 .

[0056] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0057] As shown in FIG2 , the method may include the following steps:

[0058] Step 200: Obtain the target image to be processed.

[0059] The target refers to a single object for which posture estimation is to be performed. The target can be a single person or a single animal, such as a mouse, and is not specifically limited here.

[0060] Optionally, the image to be processed may be an RGB image, which is not specifically limited here.

[0061] In a possible implementation, the original image of the target is preprocessed to obtain the target image to be processed. The preprocessing method is not unique and is not specifically limited here.

[0062] In one possible implementation, different preprocessing methods can be flexibly adopted according to actual conditions. For example, when training a model, preprocessing includes image enhancement, ResNet preprocessing, and scaling; when using the model for pose estimation, preprocessing only includes ResNet preprocessing.

[0063] Image augmentation can include rotation (±25°), scaling (0.75-1.25), cropping and padding (±0.15), motion blur, and color enhancement. ResNet preprocessing includes image resizing and normalization. Scaling involves scaling the target image to a size suitable for GPU processing.

[0064] Among them, preprocessing can make the target image easier to process, improve image quality, reduce noise, and enhance image features, so that the image can be better recognized and processed by subsequent processing algorithms, which is beneficial to feature extraction in subsequent steps to improve the effect and accuracy of posture estimation.

[0065] Step 220 : learning image features within the receptive field of the image to be processed to obtain at least one local feature map.

[0066] The local feature map reflects the local features of the image to be processed at different locations.

[0067] In one possible implementation, a convolution kernel is slid over the image to be processed, a convolution operation is performed on the image area within the receptive field, and an activation function is applied to generate local feature maps. These local feature maps reflect the local features of the image to be processed at different positions.

[0068] In one possible implementation, a convolutional neural network can be used to extract features from the image being processed. Through convolutional layers, pooling layers, and fully connected layers, it can learn image features within the receptive field of the image being processed to obtain local feature maps. For example, through layer-by-layer convolution and downsampling operations, multi-scale and multi-level local feature information of the image being processed can be extracted and represented for subsequent tasks.

[0069] Using a smaller convolution kernel can extract small-scale features, while using a larger convolution kernel can extract larger-scale features, and pooling operations can help extract the main features in the image. In one possible implementation, in a convolutional neural network, by performing different convolution operations and / or pooling operations on the processed image, multiple local feature maps with different feature expression capabilities are obtained. For example, the multiple local feature maps can be 4x, 8x, and 16x downsampled feature maps, respectively.

[0070] Convolutional neural networks (CNNs) can be used to obtain multiple feature maps that retain features of different multiples. Specifically, multiple convolutional layers can be added to the CNN, with each convolutional layer using a different convolution kernel size and stride to extract features of different multiples. For example, the first convolutional layer uses a 3x3 convolution kernel with a stride of 1, the second convolutional layer uses a 5x5 convolution kernel with a stride of 2, the third convolutional layer uses a 7x7 convolution kernel with a stride of 3, and so on. In this way, each convolutional layer can extract features of different multiples, thereby obtaining multiple feature maps. In addition, a pooling layer can be added after each convolutional layer to further compress the size of the feature map.

[0071] Multiple local feature maps can be obtained by alternating convolution operations and pooling operations. A specific convolutional neural network model (such as ResNet50) can also be used to obtain multiple local feature maps, which is not specifically limited here.

[0072] In one possible implementation, the local feature map is obtained by calling a CNN convolutional neural network, which includes at least one residual module, at least one convolutional layer, and at least one pooling layer. Specifically, as shown in FIG3 , step 220 may include the following steps:

[0073] In step 300 , initial features are extracted from the image to be processed through a residual module to obtain an initial feature map.

[0074] The residual module is an important component of the deep residual neural network (a special convolutional neural network).

[0075] The basic structure of a residual module is one or more convolutional layers and batch normalization layers, as well as skip connections. Specifically, the residual module first performs a series of convolution and batch normalization operations on the input, then performs an element-wise addition operation on the resulting feature map and the input, and finally performs a nonlinear transformation using an activation function.

[0076] Optionally, the residual module may be part of ResNet (a type of deep residual neural network), for example, it may be stack1 and stack2 parts of ResNet50, which is not specifically limited here.

[0077] The introduction of the residual module can effectively solve the problems of vanishing and exploding gradients in deep network training, and makes the network easier to train and optimize. In traditional convolutional neural networks, information is processed through a one-way transmission between layers. However, as the depth of the network increases, the transmission of information between layers becomes difficult, resulting in a decrease in network performance. The residual module introduces skip connections, which directly add the input to the network's output, making it easier for the network to learn residual information.

[0078] The residual module is used to perform initial feature extraction on the processed image, which improves the efficiency and accuracy of feature extraction and obtains better feature representation, thereby improving the performance of the model.

[0079] Step 320: Perform secondary feature extraction on the initial feature map through a convolutional layer and / or a pooling layer to obtain multiple local feature maps.

[0080] Based on the initial feature map, multiple different local feature maps are obtained through convolutional layers and / or pooling layers. Each local feature map can retain different multiples of features to improve the effect of subsequent posture estimation.

[0081] Step 240: Based on at least one local feature map, establish a connection between image features exceeding the receptive field range in the image to be processed to obtain a global feature map.

[0082] In one possible implementation, the global feature map is obtained by calling the Transformer network.

[0083] The Transformer network is a sequence-to-sequence model based on an attention mechanism. It consists of an encoder and a decoder, both of which are stacked with multiple identical modules. Using a multi-head attention mechanism, the Transformer maps input features to different feature spaces, calculates multiple attention distributions, and finally performs a weighted summation of multiple output features to obtain the final output features. This structure effectively captures relevant information in the sequence and improves model performance. Typically, the Transformer is composed of an LN (Layer Normalization), a MultiHeadAttention (multi-head attention mechanism), and a Dense (fully connected) layer.

[0084] It's worth noting that the Transformer can extract global features through a self-attention mechanism. This mechanism applies attention to each position in the input sequence to determine its relationship with other positions. This allows the Transformer to capture global dependencies in the input sequence and thus extract global features. Furthermore, the Transformer uses a multi-head attention mechanism to further enhance its global feature extraction capabilities.

[0085] Specifically, as shown in FIG4 , step 240 may include the following steps:

[0086] Step 400: flatten at least one local feature map into a one-dimensional vector.

[0087] Before entering the Transformer network, the local feature map needs to be transformed to adapt to the Transformer input requirements. For example, a convolution operation can be used to flatten the local feature map into a one-dimensional vector, and then input into the Transformer for processing.

[0088] Specifically, the flattening operation treats each pixel in the feature map as a feature vector, and then concatenates these feature vectors to obtain a long vector.

[0089] Step 420 , after position embedding, the one-dimensional vector is input into the Transformer network to learn the dependency relationship of long-distance pixels in the image to be processed, and obtain a global feature map.

[0090] After converting the image into a one-dimensional vector, in order to introduce position information, the positional embedding method can be used.

[0091] By integrating position information into a one-dimensional vector through position embedding, the Transformer network can use position embedding to understand the relationship between pixels at different positions in the image, thereby better performing feature extraction and representation learning.

[0092] As shown in FIG5 , step 420 may include the following steps:

[0093] Step 500: assign a corresponding position embedding vector to each element in the one-dimensional vector, and fuse the position embedding vector with the one-dimensional vector to obtain a one-dimensional vector containing position information.

[0094] The elements in a one-dimensional vector refer to the pixel values ​​at each position after the image is converted to a one-dimensional vector. In the process of converting the image to a one-dimensional vector, each pixel value is arranged in a certain order to form a one-dimensional vector.

[0095] The position embedding vector can be a fixed-length vector that contains information about the position of the element in the one-dimensional vector. In this way, each element in the one-dimensional vector will have a unique position embedding vector corresponding to it.

[0096] The position embedding vector and the one-dimensional vector can be fused by element-wise addition or concatenation. Specifically, suppose the one-dimensional vector is represented as x and the position embedding vector is represented as p. Their dimensions should be the same so that element-wise operations can be performed.

[0097] Element-by-element addition: The position embedding vector p can be added to the one-dimensional vector x element-by-element to obtain the processed vector. In this way, the value of the position embedding vector is added to the one-dimensional vector element at the corresponding position, thereby incorporating the position information into the one-dimensional vector. The processed vector is expressed as: x' = x + p.

[0098] Element-by-element concatenation: The position embedding vector p can be concatenated with the one-dimensional vector x to connect them in dimension. In this way, the position embedding vector is added as an extra dimension to the one-dimensional vector, thereby passing the position information to the Transformer model together with the one-dimensional vector. The processed vector is represented as: x' = [x, p].

[0099] After processing the position embedding vector and the one-dimensional vector, the processed one-dimensional vector x' will contain position information.

[0100] In step 520, the one-dimensional vector containing the position information is input into the Transformer network, so that the Transformer network uses the position information to learn the relationship between pixels at different positions in the image to be processed, and obtains a global feature map.

[0101] The one-dimensional vector containing position information is passed as input to the Transformer network for feature extraction and representation learning. The Transformer network uses the position embedding vector to understand the relationship between elements at different positions in the one-dimensional vector, thereby learning the relationship between pixels at different positions in the image to be processed and obtaining a global feature map.

[0102] In step 260 , a key point information map is obtained by fusing the features of each local feature map with the global feature map.

[0103] The goal of feature fusion is to combine different feature information to provide a more comprehensive and accurate representation. By fusing features from different levels or sources, we can capture image information from different levels and angles, improving the model's perception and performance. Feature fusion can also improve the model's robustness and generalization capabilities by reducing redundancy and noise between features.

[0104] Feature fusion methods can be feature concatenation, feature addition, feature averaging, feature weighted fusion, attention mechanism, etc., or feature graph connection.

[0105] As shown in FIG6 , step 260 may include the following steps:

[0106] Step 600: perform image concatenation on the local feature map and the global feature map to obtain a target feature map.

[0107] Local and global feature maps use different methods to extract features, ranging from low-level to high-level features, retaining different multiples of features. Each feature map focuses on different features. Combining local and global feature maps enriches feature representation, facilitates information transfer, improves model robustness, and efficiently utilizes computing resources, thereby improving model performance and effectiveness.

[0108] It should be noted that before joining images, it is necessary to ensure that the images to be joined have the same size or have been scaled and cropped accordingly to make them consistent in size.

[0109] Step 620: Perform a convolution operation on the target feature map to obtain a key point information map.

[0110] Optionally, the key point information map may include: a key point heat map, a key point position optimization map, and a target skeleton map.

[0111] Step 280 , performing pose estimation of the target based on the key point information graph to obtain a pose estimation result.

[0112] The pose estimation result includes the key point positions of the target in the image to be processed.

[0113] The key point information map includes a key point heat map and a key point position optimization map. As shown in FIG7 , step 280 may include the following steps:

[0114] Step 700: Identify the key point positions of the target in the image to be processed based on the key point heat map and the key point position optimization map, and obtain a pose estimation result.

[0115] Under the action of the above embodiment, an image of the target is obtained, the image features within the receptive field of the image are learned to obtain a local feature map, and then the connection between the image features exceeding the receptive field is learned to obtain a global feature map. A key point information map is obtained based on the local feature map and the global feature map, and finally, posture estimation is performed based on the key point information map. Different features of the image to be processed are learned separately to obtain multiple feature maps with different focuses. These feature maps are combined to obtain a key point information map, thereby achieving posture estimation of the target. This overcomes the deficiency of the existing technical solution that can only capture the relationship between image features within a certain range in the image to be processed, establishes the connection between long-distance image features beyond the receptive field, reduces the point drift phenomenon, and improves the accuracy of posture estimation. In addition, the present application solution is simple and easy to implement, the model training volume is not high, and the training cost is reduced.

[0116] In an exemplary embodiment, as shown in FIG8 , step 700 may include the following steps:

[0117] Step 800: Determine the maximum value index of each key point heat map.

[0118] Maximum index refers to finding the position or index of the maximum value in an array or matrix. It can be understood that the maximum index is positioning information (i.e., position information).

[0119] Step 820: Search the key point position optimization graph for the value at the maximum value index.

[0120] The key point position optimization graph includes the key point position optimization graphs in the horizontal and vertical directions, and searches for the value of the key point position optimization graphs in the horizontal and vertical directions at the maximum value index.

[0121] Step 840 , obtaining the key point position of the target according to the maximum value index and the value of the key point position optimization graph at the maximum value index.

[0122] The key point coordinates can be obtained by adding the maximum value index of the key point heat map and the value of the key point position optimization map at the maximum value index, and then multiplying it by a specific multiple.

[0123] The specific multiple depends on the global feature map and the local feature map for feature fusion. For example, if both the global feature map and the local feature map are 8-fold downsampled feature maps, then the specific multiple is 8.

[0124] In an exemplary embodiment, the key point information graph further includes a skeleton graph formed by connecting the key point positions according to a skeleton relationship, as shown in FIG9 . Step 280 may further include the following steps:

[0125] Step 900 : predicting the category of the target based on the skeleton image to obtain the category of the target.

[0126] Among them, the skeleton map can be used as the semantic segmentation map of the target, and the category of the target can be detected according to the semantic segmentation map.

[0127] Faster R-CNN and semantic segmentation maps are combined to perform object category detection. Specifically, semantic segmentation maps can be used to provide more accurate object boundary information, thereby helping Faster R-CNN to more accurately locate and classify objects.

[0128] In one possible implementation, the following steps can be used to detect objects in an image:

[0129] Based on the semantic segmentation map, the pixel coordinates of the target are extracted; connected region analysis is performed on the extracted pixel coordinates to obtain the target's bounding box; the bounding box is screened and optimized, such as removing overlapping bounding boxes and adjusting the bounding box size; based on the target's bounding box, the image is detected using an object detection algorithm such as Faster R-CNN and YOLO.

[0130] Under the effect of the above embodiment, the skeleton graph obtained by the single target posture estimation method is used as a semantic segmentation graph to predict the category of the target, which provides an application method and further broadens the application scenario.

[0131] Referring to Figure 10, an embodiment of the present application further provides a single target pose estimation network for implementing the single target pose estimation method provided by the present application. The single target pose estimation network includes: a CNN layer 1, a MaxPool layer 2, a C1 layer 3, a C2 layer 4, a C3 layer 5, a C4 layer 6, a Reshape layer 7, a Patch encoder layer 8, a Transformer layer 9, a Concate and Conv layer 12, a Detail convolution layer 11, and an MLP layer 10;

[0132] CNN layer 1 includes stack1 and stack2 of ResNet50. ResNet50 is a deep residual convolutional neural network composed of multiple residual blocks. Stack1 and Stack2 are two stacked residual blocks in ResNet50. Stack1 performs preliminary feature extraction on the input feature map while reducing the spatial size of the feature map. Stack2 further extracts features and further reduces the spatial size of the feature map. By using Stack1 and Stack2 of ResNet50 to perform multi-level feature extraction on the preprocessed image and gradually reduce the spatial size of the feature map, the performance of the model can be effectively improved while reducing the number of parameters and the amount of computation.

[0133] MaxPool layer 2 includes maxpooling2D with stride = 2. MaxPool layer 2 is used to perform 2x downsampling, which reduces the size of the feature map while retaining important features to reduce the amount of computation.

[0134] Layers C1-3, C2-4, C3-5, and C4-6 all use convolutions with a stride of 1. As shown in Figure 8, layers C1-3 and C2-4 are cascaded and placed after MaxPool layer 2. That is, after max pooling, two convolutions with a stride of 1 are performed. These two convolutions with a stride of 1 increase the depth of the convolutions, improving the model's expressiveness and accuracy. Furthermore, these two convolutions with a stride of 1 better capture detailed information in the image, enhancing the model's feature extraction capabilities. Layers C3-5 and C4-6 are also cascaded, similar to layers C1-3 and C2-4.

[0135] The Reshape layer 7 is used to transform the shape of the feature map; the Reshape layer 7 located before the Patch encoder layer 8 is used to flatten the feature map into a one-dimensional vector.

[0136] The Patch Encoder layer 8 is used to perform position embedding on the one-dimensional vector. Before the one-dimensional vector enters the Transformer layer 9, position embedding is required to give each vector a position information.

[0137] Transformer layer 9 processes the one-dimensional vectors after position embedding to extract global features. Transformer layer 9 can distinguish the different roles of vectors in the sequence based on their position information, thereby extracting global features.

[0138] The MLP layer 10 is used to further process the global features and restore the global features to a feature map, and then perform deconvolution to obtain a global feature map. The MLP layer 10 includes an LN (Layer Normalization), a Dense (fully connected layer), a reshaping unit, and a deconvolution.

[0139] The Detail convolution layer 11 includes: conv2D, stride = 2, Padding = same; conv2D, stride = 1, Padding = same; BN (Batch normalization); Activation = Relu. The Detail convolution layer 11 can achieve 2x downsampling.

[0140] The Concate and Conv layers 12 are used to concatenate the second-category feature map with the global feature map and perform a convolution operation on the concatenated target feature map. The convolution operations in the Concate and Conv layers 12 include: conv2D*2, Padding=same; conv2D, stride=1.

[0141] In this embodiment, the second type of feature map includes the feature map output by the Detail convolution layer 11 and the feature map output by the C2 layer 4, and the global feature map includes the feature map output by the MLP layer 10.

[0142] Possibly, Transformer layer 9 is composed of LN (Layer Normalization), MultiHeadAttention (multi-head attention mechanism), and Dense (fully connected layer).

[0143] In one possible implementation, the single-target pose estimation network further includes an intermediate supervision unit 13 for adding an additional supervision signal to the single-target pose estimation network to help the single-target pose estimation network better learn and adjust. The supervision signal can be an intermediate layer output of the prediction result or an output of other related tasks.

[0144] Through the above embodiments, the network structure of the solution of the present application is simple, the number of parameters is not large, and the training cost can be reduced; there are three groups of feature maps that are finally spliced, which can be considered as three groups of feature maps to be spliced ​​obtained through three-way feature calculation. The feature calculation through a single way is conducive to retaining the features at high resolution. Therefore, the feature map output by the Detail convolution layer 11 retains 4 times the features, the feature map output by the C2 layer 4 retains 8 times the features, and the feature map output by the MLP layer 10 retains 16 times the features. Finally, the 4-fold, 8-fold, and 16-fold features are connected to obtain the key point heat map, key point position optimization map, and skeleton map of the target, which can effectively estimate the posture of a single target.

[0145] Figure 11 is a schematic diagram of a specific implementation of a single-target pose estimation method in an application scenario. In this application scenario, a mouse is used as the target for pose estimation. That is, based on the mouse image a in Figure 11, the mouse pose information b, c, d, and e are obtained.

[0146] The hardware configuration for this application scenario is: i912900K, 4090 GPU, 5200MHz DDR532G*4 memory, 2TB SUMSUNG 980PRO solid-state drive; the environment configuration is: Windows 11 Professional Edition 22H2, programming is done in Python 3.9, the network is built based on TensorFlow 2.9.3, and imgaug is used for image data enhancement.

[0147] After acquiring the mouse image using an acquisition device (such as a CMOS camera), the mouse image is enhanced, specifically, rotated (±25°), scaled (0.75-1.25), cropped and filled (±0.15), motion blurred, and color enhanced to obtain the mouse image a.

[0148] The mouse image a is preprocessed according to the ResNet preprocessing method, and then scaled to a size suitable for GPU processing (640×480 resolution is used in this application scenario). The scaled mouse image a is then fed into the stack1 and stack2 parts of the ResNet50 pre-trained on the ImageNet dataset to obtain a 4x downsampled intermediate feature map F. 4x (160×120); for F 4x Perform maximum pooling to obtain 8 times downsampled feature map F 8x_0 (80×60); on the other hand, F 4x Perform a convolution operation with a step size of 2 (downsampling) to obtain a high-resolution detail feature map F 8x_1 ; for F 8x_0 After performing two convolution operations with a step size of 1, we get F 8x_2 ; for F 8x_2 Perform maximum pooling and two convolution operations with a step size of 1, then reshape it into a one-dimensional vector and embed the position before sending it to 6 Transformer blocks for calculation. Then restore the obtained features to feature maps, and then deconvolve to obtain feature maps F 8x_3 ;Finally connect F 8x_1 、F 8x_2 、F 8x_3 Get F 8x (80×60), for F 8x The convolution operation can obtain the posture information of the mouse. As shown in Figure 11, b shows the sum of the heat map of the 16 key points of the mouse; c and d show the sum of the horizontal position optimization map and vertical position optimization map of the 16 key points of the mouse, respectively; e shows the animal skeleton map.

[0149] In the above process, the processing from the scaled mouse image a to the mouse posture information b, c, d and e can be implemented by the single target posture estimation network shown in FIG10 .

[0150] In Figure 11, posture information b is the sum of the heat maps of the 16 key points of the mouse; posture information c and d are the sum of the horizontal position optimization map and the vertical position optimization map of the 16 key points of the mouse, respectively; posture information e is the target (mouse) skeleton map, which is obtained by connecting the target (mouse) key points.

[0151] Furthermore, the key point position (x, y) can be obtained by searching the maximum value index index of each key point heat map and multiplying it by 8; the value of the horizontal and vertical position optimization map of each key point at index is searched and multiplied by 8, and then added to (x, y) to obtain the optimized (x, y), which is the final key point position.

[0152] Compared with the prior art, the present invention has achieved better results. The following describes the effects of the present invention.

[0153] DeepLabCut and Sleap offer similar pose estimation accuracy (mean average precision, mAP). Our method achieved a 0.5% higher mAP on the Fly32 dataset. We evaluated performance differences (root mean square error, RMSE) using a self-built mouse dataset. Our method achieved a 10% lower RMSE than the best of the two methods, DeepLabCut. We also significantly reduced point drift.

[0154] [Corrected 20.07.2023 according to Rule 91] Please refer to Figures 12a to 12h, which show schematic diagrams of the implementation results of the present invention.

[0155] [Corrected 20.07.2023 according to Rule 91] Figure 12a is a comparison of the accuracy of the scheme of the present invention and other schemes (Deeplabcut and Sleap) in single fruit fly posture estimation. It can be seen that the mAP of the scheme of the present invention on the fruit fly dataset is 93.5±0.5%, which is higher than the 92.8% and 92.7% of Deeplabcut and Sleap respectively.

[0156] [Corrected 20.07.2023 according to Rule 91] Figure 12b is a RMSE comparison chart of the scheme of the present invention and other schemes (Deeplabcut and Sleap) in single mouse posture estimation, among which, the RMSE of the scheme of the present invention is 3.44±0.19 and 3.73±0.20, and the RMSE of other schemes is 3.83±0.41 and 4.40±0.52. It can be seen that in terms of the RMSE of the mouse dataset, the scheme of the present invention is lower than that of other schemes.

[0157] [Corrected 20.07.2023 according to Rule 91] Figure 12c is a comparison of the inference speed of the scheme of the present invention and other schemes (Deeplabcut and Sleap) in single mouse posture estimation. On the mouse dataset, the inference speed of the scheme of the present invention is 90±4fps, which is lower than SLEAP's 106±4fps and higher than Deeplabcut's 44±2fps.

[0158] [Corrected 20.07.2023 according to Rule 91] Figures 12d and 12e are performance diagrams of the solution of the present invention under different training data amounts. When the amount of training data is about 350, the pose estimation network achieves better prediction results.

[0159] [Corrected 20.07.2023 according to rule 91] Figure 12f is a diagram of the key connections of the skeletal labels used in mice.

[0160] [Corrected 20.07.2023 according to Rule 91] Figure 12g is a schematic diagram of the key point inference of mouse videos using the scheme of the present invention and other schemes (Deeplabcut and Sleap), respectively. The results show that the model of the present invention can significantly reduce the oscillation of points compared with Deeplabcut, and can retain more key point prediction information than SLEAP (pcutoff = 0.6), as shown in the box.

[0161] [Corrected 20.07.2023 according to Rule 91] Figure 12h is a schematic diagram of the model performance with and without skeleton labels (baseline). As shown in the box, after adding skeleton labels, the slight oscillation of key point prediction is alleviated.

[0162] In the above experimental results, the accuracy, inference speed, stability, etc. of the posture estimation of the present application solution are better than the existing technology. In addition, when the amount of training data is about 350, the posture estimation network obtains better prediction results, which can be explained that the training amount of the posture estimation network provided by the present application is relatively low, that is, the training cost is low, which means that the present application solution effectively solves the technical problem of high model training cost.

[0163] The following are embodiments of the apparatus of the present application, which can be used to perform the single target pose estimation method and target category detection method involved in the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the single target pose estimation method and target category detection method involved in the present application.

[0164] Please refer to Figure 13. An embodiment of the present application provides a single-target posture estimation device 1300, including but not limited to: an image acquisition module 1310, a local feature map acquisition module 1330, a global feature map acquisition module 1350, a feature fusion module 1370 and a posture estimation module 1390.

[0165] The image acquisition module 1310 is used to acquire the target image to be processed.

[0166] The local feature map acquisition module 1330 is used to learn image features within the receptive field of the image to be processed to obtain at least one local feature map.

[0167] The global feature map acquisition module 1350 is used to establish a connection between image features exceeding the receptive field range in the image to be processed based on at least one local feature map to obtain a global feature map.

[0168] The feature fusion module 1370 is used to obtain a key point information map by fusing the features of each local feature map with the global feature map.

[0169] The posture estimation module 1390 is used to estimate the posture of the target based on the key point information map to obtain a posture estimation result; the posture estimation result includes the key point positions of the target in the image to be processed.

[0170] It should be noted that the single-target posture estimation device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing single-target posture estimation. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the single-target posture estimation device will be divided into different functional modules to complete all or part of the functions described above.

[0171] In addition, the single target posture estimation device and the single target posture estimation method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0172] Fig. 14 shows a schematic diagram of the structure of a server according to an exemplary embodiment. The server is applicable to the server 130 in the implementation environment shown in Fig. 1 .

[0173] It should be noted that the server is only an example adapted for the present application and cannot be considered to provide any limitation on the scope of use of the present application. The server cannot be interpreted as needing to rely on or necessarily having one or more components in the exemplary server 2000 shown in Figure 14.

[0174] The hardware structure of the server 2000 may vary greatly due to different configurations or performances. As shown in FIG. 12 , the server 2000 includes a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0175] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the server 2000 .

[0176] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices, for example, the interaction between the acquisition terminal 110 and the service terminal 130 in the implementation environment shown in FIG1 .

[0177] Of course, in other examples adapted by this application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, as shown in FIG14 , which is not specifically limited here.

[0178] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system 251, application 253 and data 255, etc. The storage method can be temporary storage or permanent storage.

[0179] Among them, the operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 2000 to enable the central processing unit 270 to calculate and process the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0180] The application 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251. It may include at least one module (not shown in FIG14 ), each of which may include computer-readable instructions for the server 2000. For example, a single-target pose estimation device may be considered as an application 253 deployed on the server 2000.

[0181] The data 255 may be photos, pictures, etc. stored in a disk, or may be images of a target to be subjected to posture estimation, posture estimation results, etc., stored in the memory 250 .

[0182] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on the massive data 255 in the memory 250. For example, the single target pose estimation method is completed by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0183] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.

[0184] Please refer to FIG. 15 . An embodiment of the present application provides an electronic device 4000 . The electronic device 4000 may include a desktop computer, a laptop computer, a server, etc.

[0185] In FIG. 15 , the electronic device 4000 includes at least one processor 4001 and at least one memory 4003 .

[0186] Data exchange between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in FIG15 , but this does not mean that there is only one bus or one type of bus.

[0187] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0188] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0189] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to these.

[0190] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0191] The computer-readable instructions are executed by one or more processors 4001 to implement the single target pose estimation method in the above embodiments.

[0192] In addition, an embodiment of the present application provides a storage medium having computer-readable instructions stored thereon, and the computer-readable instructions are executed by one or more processors to implement the single target pose estimation method as described above.

[0193] In an embodiment of the present application, a computer program product is provided. The computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the single-target pose estimation method as described above.

[0194] Compared with related technologies, this application can produce the following beneficial effects:

[0195] 1. Acquire an image of the target, learn the image features within the receptive field of the image to obtain a local feature map, then learn the connections between the image features beyond the receptive field to obtain a global feature map, obtain a key point information map based on the local feature map and the global feature map, and finally perform pose estimation based on the key point information map. By learning different features of the image to be processed separately, multiple feature maps with different focuses are obtained, and these feature maps are combined to obtain a key point information map, thereby achieving pose estimation of the target. This overcomes the deficiency of existing technical solutions that can only capture the relationship between image features within a certain range in the image to be processed, establishes connections between long-distance image features beyond the receptive field, reduces point drift, and improves the accuracy of pose estimation. In addition, the present application solution is simple and easy to implement, the model training volume is not high, and the training cost is reduced.

[0196] 2. For the skeleton map obtained by the single target pose estimation method, the skeleton map is used as a semantic segmentation map to predict the target category, which provides an application method and further broadens the application scenarios.

[0197] 3. Using ResNet50's Stack1 and Stack2 to perform multi-level feature extraction on the preprocessed image and gradually reduce the spatial size of the feature map can effectively improve model performance while reducing the number of parameters and computational complexity. Two convolution operations with a stride of 1 increase the depth of the convolution, improving the model's expressiveness and accuracy. Furthermore, these two convolution operations with a stride of 1 better capture detailed information in the image, enhancing the model's feature extraction capabilities.

[0198] 4. Obtain an image of the target, use different convolution operations and / or pooling operations to extract features from the image, obtain multiple feature maps with different focuses, and further use Transformer to obtain global features. Based on the feature maps obtained using the convolutional neural network and Transformer, obtain key point heat maps, key point position optimization maps, skeleton maps and other posture information. The technical solution of this application uses a convolutional neural network combined with a Transformer network structure, wherein the convolutional neural network is responsible for extracting the features of the target image at high resolution and reducing the image dimension, and the Transformer is responsible for processing the image features output by the convolutional neural network. The feature map obtained by the Transformer is advanced, and the features of each pixel are associated with the features of other pixels in the sequence, that is, the relationship between a certain pixel and all other pixels can be established to obtain global features. Finally, the different feature map information obtained by the convolutional neural network and the Transformer are combined, and after convolution, a posture key point heat map, a key point position optimization map and a body skeleton map are generated. The convolutional neural network combined with the Transformer network structure can learn the global-local context to strengthen feature representation, overcome the limitation that the convolution operation can only capture local image features, reduce the point drifting situation, and improve the accuracy of single-target posture estimation; and the technical solution of the present application is simple and easy, the corresponding single-target posture estimation network parameters are relatively small, and the structure is not complicated, and high-precision effects can be achieved with fewer parameters and training costs; thereby effectively solving the technical problems of low posture estimation accuracy and high model training cost in related technologies.

[0199] 5. In the experimental effect diagram shown in Figure 12, the accuracy, inference speed, stability, etc. of the posture estimation of the present application scheme are better than the existing technology. In addition, Figure 12 shows that when the amount of training data is about 350, the posture estimation network obtains better prediction results, which can explain that the training amount of the posture estimation network provided by the present application is relatively low, that is, the training cost is low, which means that the present application scheme effectively solves the technical problem of high model training cost.

[0200] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0201] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A single target attitude estimation method, characterized in that: include: Obtain the target image to be processed; Learning image features within the receptive field of the image to be processed to obtain at least one local feature map; The local feature map reflects the local features of the image to be processed at different positions; Based on at least one of the local feature maps, establishing a connection between image features exceeding the receptive field range in the image to be processed to obtain a global feature map; Obtaining a key point information map by fusing the features of each of the local feature maps with the global feature map; The target's posture is estimated based on the key point information graph to obtain a posture estimation result; the posture estimation result includes the key point position of the target in the image to be processed.

2. The method according to claim 1, characterized in that The local feature map is obtained by calling a CNN network; the CNN network includes at least one residual module, at least one convolutional layer, and at least one pooling layer; The learning of the image features within the receptive field of the image to be processed to obtain at least one local feature map includes: Performing initial feature extraction on the image to be processed through the residual module to obtain an initial feature map; The initial feature map is subjected to secondary feature extraction through the convolution layer and / or the pooling layer to obtain a plurality of local feature maps.

3. The method according to claim 1, characterized in that The global feature map is obtained by calling the Transformer network; The step of establishing a connection between image features exceeding the receptive field range in the image to be processed based on at least one of the local feature maps to obtain a global feature map includes: Flattening at least one of the local feature maps into a one-dimensional vector; After position embedding is performed on the one-dimensional vector, the vector is input into the Transformer network to learn the dependency relationship of long-distance pixel points in the image to be processed, so as to obtain the global feature map.

4. The method according to claim 3, characterized in that The step of embedding the one-dimensional vector into a Transformer network to learn the dependency relationship of long-distance pixels in the image to be processed to obtain the global feature map includes: Assigning a corresponding position embedding vector to each element in the one-dimensional vector, and fusing the position embedding vector with the one-dimensional vector to obtain the one-dimensional vector containing position information; The one-dimensional vector containing the position information is input into the Transformer network, so that the Transformer network uses the position information to learn the relationship between pixels at different positions in the image to be processed, and obtains the global feature map.

5. The method according to claim 1, characterized in that The step of obtaining a key point information map by fusing the features of each of the local feature maps with the global feature map includes: Performing image concatenation of the local feature map and the global feature map to obtain a target feature map; A convolution operation is performed on the target feature map to obtain the key point information map.

6. The method according to claim 1, characterized in that The key point information map includes a key point heat map and a key point position optimization map; The step of performing attitude estimation of the target based on the key point information graph to obtain an attitude estimation result includes: According to the key point heat map and the key point position optimization map, the key point positions of the target in the image to be processed are identified to obtain the posture estimation result.

7. The method according to claim 6, characterized in that The step of identifying the key point positions of the target in the image to be processed according to the key point heat map and the key point position optimization map to obtain the posture estimation result includes: Determine the maximum value index of each key point heat map; Searching the key point position optimization graph for a value at the maximum value index; The key point position of the target is obtained according to the maximum value index and the value of the key point position optimization map at the maximum value index.

8. The method according to any one of claims 1 to 7, wherein the key point information graph further comprises a skeleton graph formed by connecting the key point positions according to a skeleton relationship; The method further comprises: The category of the target is predicted according to the skeleton image to obtain the category of the target.

9. A single target posture estimation device, characterized in that: include: An image acquisition module, used to acquire the target image to be processed; A local feature map acquisition module, used to learn the image features within the receptive field of the image to be processed to obtain at least one local feature map; the local feature map reflects the local features of the image to be processed at different positions; A global feature map acquisition module, used to establish a connection between image features exceeding the receptive field range in the image to be processed based on at least one of the local feature maps, to obtain a global feature map; A feature fusion module, used for obtaining a key point information map by fusing the features of each of the local feature maps with the global feature map; The posture estimation module is used to perform posture estimation of the target based on the key point information map to obtain a posture estimation result; the posture estimation result includes the key point position of the target in the image to be processed.

10. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors so that the electronic device implements the single target posture estimation method according to any one of claims 1 to 8.

Images