Attitude estimation method, attitude estimation device, and computer-readable storage medium

Abstract

The present application provides a posture estimation method, posture estimation device, and computer-readable storage medium. The method includes: obtaining a first key point detection result of an image to be detected; extracting instance-aware posture features based on predicted key point coordinates; fusing the instance-aware posture features, preset global shared posture features, and a convolutional feature map to obtain a fused posture feature; performing key point detection on the fused posture feature to obtain a second key point detection result; combining the first key point detection result and the second key point detection result to obtain a final key point detection result; and determining the target posture in the image to be detected using the final key point detection result. Through the above-described method, the posture estimation device enhances the original feature map by separately encoding features of specific instance posture information and shared posture information, and finally further performs posture detection on the enhanced feature map, thereby achieving simple and accurate posture estimation of all human targets in the image.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a posture estimation method, a posture estimation device, and a computer-readable storage medium. Background Art

[0002] Human skeleton keypoint detection is mainly aimed at detecting key points of the human body, such as facial features and joints, and describing the human skeleton through key points. Human skeleton keypoints are crucial for describing human posture and predicting human behavior.

[0003] Human skeleton keypoint detection is fundamental to many computer vision tasks and plays a fundamental role in research in related fields, such as action recognition, abnormal behavior detection, AR / VR, human-computer interaction, smart homes, sports assistance, and autonomous driving. In recent years, with the development of deep learning technology, the effectiveness of human skeleton keypoint detection has continued to improve and has begun to be widely used in related fields of computer vision. However, due to the diversity of human postures and the impact of environmental factors such as occlusion and lighting changes in real-world applications, human skeleton keypoint detection currently still suffers from low detection accuracy. Summary of the Invention

[0004] The present application provides a posture estimation method, a posture estimation device, and a computer-readable storage medium.

[0005] The present application provides a posture estimation method, which includes:

[0006] Extract the convolution feature map of the image to be detected;

[0007] Obtaining a first key point detection result of the image to be detected using the convolution feature map, wherein the first key point detection result includes predicted key point coordinates of each feature point in the convolution feature map;

[0008] extracting instance-aware pose features from the convolutional feature map based on the predicted keypoint coordinates;

[0009] Fusing the instance-perceived posture feature, the preset global shared posture feature, and the convolutional feature map to obtain a fused posture feature;

[0010] Performing key point detection on the fused posture feature to obtain a second key point detection result;

[0011] Combining the first key point detection result and the second key point detection result to obtain a final key point detection result;

[0012] The final key point detection result is used to determine the target posture in the image to be detected.

[0013] The step of extracting instance-aware posture features from the convolutional feature map based on the predicted key point coordinates includes:

[0014] Performing feature sampling at a position of a key point corresponding to the predicted key point coordinates in the convolution feature map to obtain a key point feature set, wherein the key point feature set includes key point features corresponding to each feature point in the convolution feature map;

[0015] The key point feature set is determined as the instance-aware gesture feature.

[0016] The global shared posture feature is used to encode the shared information of each key point between different target instances, and the global shared posture feature is obtained by learning and optimizing through network training after random initialization.

[0017] The step of fusing the instance-aware posture feature, the preset global shared posture feature, and the convolutional feature map to obtain the fused posture feature includes:

[0018] Obtaining instance features at each position in the convolutional feature map;

[0019] Acquire a first feature relationship between the instance feature and the instance perception posture feature at the same position;

[0020] Acquire a second feature relationship between the instance feature and the global shared posture feature at the same position;

[0021] The convolution feature map is concatenated with the first feature relationship and the second feature relationship in a channel dimension to obtain the fused posture feature.

[0022] Wherein, obtaining the first feature relationship between the instance feature and the instance perception posture feature at the same position includes:

[0023] Obtaining a transposed matrix of the instance-perceived posture feature;

[0024] Obtaining normalized parameters of the transposed matrix and the instance features;

[0025] The first feature relationship is calculated based on the normalized parameter and the instance-aware posture feature.

[0026] The second key point detection result includes a key point classification result, a key point prediction offset, and a key point existence probability; wherein the key point classification result includes a key point prediction category and its confidence.

[0027] The combining of the first key point detection result and the second key point detection result to obtain a final key point detection result includes:

[0028] Acquire predicted key point coordinates based on the first key point detection result;

[0029] The predicted key point coordinates are adjusted using the key point prediction offset to obtain the final key point detection result.

[0030] Wherein, determining the target posture in the image to be detected by using the final key point detection result includes:

[0031] According to the final key point detection result, the position of each key point is obtained;

[0032] The minimum bounding rectangle of all key points is used as the detection frame of the target;

[0033] Output the target detection box as the detection result of the target posture.

[0034] The present application also provides a posture estimation device, which includes a processor and a memory, wherein program data is stored in the memory, and the processor is used to execute the program data to implement the posture estimation method as described above.

[0035] The present application also provides a computer-readable storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the above-mentioned posture estimation method.

[0036] The beneficial effects of the present application are as follows: the posture estimation device extracts a convolution feature map of the image to be detected; uses the convolution feature map to obtain a first key point detection result of the image to be detected, wherein the first key point detection result includes the predicted key point coordinates of each feature point in the convolution feature map; extracts instance-aware posture features from the convolution feature map based on the predicted key point coordinates; performs feature fusion on the instance-aware posture features, the preset global shared posture features and the convolution feature map to obtain a fused posture feature; performs key point detection on the fused posture feature to obtain a second key point detection result; combines the first key point detection result and the second key point detection result to obtain a final key point detection result; and uses the final key point detection result to determine the target posture in the image to be detected. In the above manner, the posture estimation device enhances the original feature map by respectively encoding the features of specific instance posture information and shared posture information, and finally further performs posture detection on the enhanced feature map, thereby achieving simple and accurate posture estimation of all human targets in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0038] Figure 1 This is a flow chart of an embodiment of a posture estimation method provided by the present application;

[0039] Figure 2 It is a schematic diagram of key points of the human body provided by this application;

[0040] Figure 3 This is a schematic diagram of the network structure of the human body posture estimation method based on instance-aware feature fusion provided by this application;

[0041] Figure 4 yes Figure 1 The specific flow diagram of step S14 of the posture estimation method shown;

[0042] Figure 5 1 is a structural diagram of an embodiment of a posture estimation device provided by the present application;

[0043] Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0045] Please refer to the following for details: Figure 1 , Figure 1 This is a flow chart of an embodiment of a posture estimation method provided by this application.

[0046] The posture estimation method of the present application is applied to a posture estimation device, wherein the posture estimation device of the present application can be a server or a system comprising a server and a terminal device. Accordingly, the various components of the posture estimation device, such as the various units, subunits, modules, and submodules, can be all provided in the server or separately provided in the server and the terminal device.

[0047] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the posture estimation method of the embodiment of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0048] Specifically, if Figure 1 As shown, the posture estimation method of the embodiment of the present application specifically includes the following steps:

[0049] Step S11: extracting the convolution feature map of the image to be detected.

[0050] In an embodiment of the present application, a posture estimation device installs cameras at multiple positions and angles in a target scene, collects a number of images containing human targets, and divides the collected images into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0051] Furthermore, the staff annotated the images they collected, marking the coordinates of 14 key points on the human body of all the human targets in each image. Each coordinate is represented by (x, y, v), where (x, y) represents the position of the key point, and v represents the visibility of the key point. v = 1 means that the key point has been annotated and is visible, v = 2 means that the key point has been annotated but is not visible due to occlusion, etc., and v = 0 means that the key point has not been annotated because it is not in the image, etc. The 14 key points are the right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, top of the head, and nose. Their distribution on the human body is as follows: Figure 2 shown.

[0052] The convolutional neural network used in the embodiment of the present application is a human posture estimation complete network structure of an instance-aware feature fusion. Figure 3 As shown in Figure 3, the network consists of three parts: a one-stage key point detection module, an instance-aware-shared pose feature fusion module, and a two-stage detection module.

[0053] The posture estimation device uses the feature extraction network in the first-stage key point detection module to extract the image to be detected, that is, the convolution feature map of the input image. Specifically, the image to be detected I is input into the feature extraction network in the first-stage key point detection module to extract the convolution feature F∈R H×W×CWhere H, W, and C represent the spatial height, width, and number of channels of the convolutional feature, respectively. The feature extraction network uses lightweight feature extractors such as MobileNetV2, MobileOne, and GhostNet.

[0054] Step S12: using the convolution feature map, obtaining a first key point detection result of the image to be detected, wherein the first key point detection result includes the predicted key point coordinates of each feature point in the convolution feature map.

[0055] In the embodiment of the present application, the posture estimation device processes the extracted convolution feature F with a convolution layer with a kernel size of , a step size of 1, an input channel number of C, and an output channel number of 14×2 to obtain a key point detection result A∈R H ×W×(14×2) . A (i,j) ∈R 14×2 It represents the two-dimensional coordinates of 14 key points predicted by the network for the instance target corresponding to the feature point with coordinates (i, j) on the feature map F.

[0056] Step S13: Extracting instance-aware pose features from the convolutional feature map based on the predicted key point coordinates.

[0057] In the embodiment of the present application, the posture estimation device inputs the detection results of the key point detection module of the first stage into the instance perception-shared posture feature fusion module, and performs feature sampling on the key point coordinates corresponding to the position of the key point on the convolution feature map F by the bilinear interpolation method to obtain the key point feature set E∈R H×W×14×C Among them, E (i,j) ∈R 14×C The 14 key point features of the instance target corresponding to the feature point with coordinates (i, j) on the feature map F are recorded as instance-aware posture features.

[0058] Step S14: Fusing the instance-aware posture feature, the preset global shared posture feature, and the convolutional feature map to obtain a fused posture feature.

[0059] In the embodiment of the present application, different from the instance-aware posture features sampled in step S13, which are used to encode the posture information of a specific instance target, a global shared posture feature tokens∈R is set. 14×C , which is used to encode the shared information of posture between different target instances. The global shared posture feature tokens are randomly initialized and learned and optimized during the network training process.

[0060] Among them, the global shared pose feature is used to encode the shared information of each key point between different target instances, while the instance-aware pose feature is used to encode the feature information of each key point in the same target instance.

[0061] Furthermore, the posture estimation device uses instance-aware posture features and global shared posture features to enhance the convolution feature map from the two perspectives of target instance and posture sharing respectively, and obtains the fused posture features after the convolution feature map feature enhancement through feature fusion.

[0062] For more details on the generation process of fusion posture features, please refer to Figure 4 , Figure 4 yes Figure 1 Schematic diagram of the specific flow of step S14 of the posture estimation method shown.

[0063] Specifically, if Figure 4 As shown, the posture estimation method of the embodiment of the present application specifically includes the following steps:

[0064] Step S141: Obtain instance features at each position in the convolution feature map.

[0065] In the embodiment of the present application, the posture estimation device traverses the convolution feature map and determines the instance feature F of each position in the convolution feature map. (i,j) ∈R 1×C .

[0066] Step S142: Obtain a first feature relationship between the instance feature and the instance-perceived posture feature at the same position.

[0067] In the embodiment of the present application, the posture estimation device extracts the instance-perceived posture feature E of the corresponding position from the key point feature set E obtained by sampling according to the instance feature. (i,j) ∈R 14×C , and calculate the first feature relationship F between the instance feature and the instance-perceived posture feature (i,j) ′:

[0068]

[0069] Among them, T represents the matrix transpose operation of the instance-aware posture features, softmax() represents the calculation of the normalization parameter, and c represents the number of channels.

[0070] Step S143: Obtain a second feature relationship between the instance feature at the same position and the global shared posture feature.

[0071] In the embodiment of the present application, the posture estimation device extracts the global shared posture feature tokens and calculates the second feature relationship F between the instance feature and the global shared posture feature. (i,j) ″:

[0072]

[0073] Step S144: Concatenate the convolution feature map with the first feature relationship and the second feature relationship in the channel dimension to obtain a fused posture feature.

[0074] In the embodiment of the present application, the posture estimation device combines the convolution feature map F with the calculated first feature relationship F (i,j )′ and the second characteristic relation F (i,j) ″F is obtained by concatenating features in the channel dimension fusion ∈R H×W×(3×C) :

[0075] F fusion =concat(F, F′, F″)

[0076] Then, the pose estimation device uses a convolutional layer with a kernel size of 1×1, a stride of 1, an input channel number of 3×C, and an output channel number of C to train F. fusion Perform feature dimensionality reduction to obtain F fusion ′∈R H×W×C .

[0077] Step S15: Perform key point detection on the fused posture features to obtain a second key point detection result.

[0078] In an embodiment of the present application, the posture estimation device performs key point detection on the fused posture features by means of quadratic regression in the two-stage detection module to obtain a second key point detection result.

[0079] Specifically, the posture estimation device uses two consecutive convolutional layers with a kernel size of 5×5, a stride of 1, and a number of input channels and output channels of C to form a convolutional layer F. fusion ′Further feature extraction is performed to obtain F fusion ″∈R H×W×C .

[0080] Among them, each convolutional layer is followed by a batch normalization BN layer and an activation function LeakyReLU.

[0081] Then, the pose estimation device uses a convolutional layer with a kernel size of 1×1, a stride of 1, an input channel number of C, and an output channel number of C′ to train F fusion ″Perform convolution to obtain the network output result p out ∈R H×W×C′ .in:

[0082] C′=num classes +num kpts +num kpts_conf

[0083] Among them, num classes =1 indicates the number of categories, num kpts=28 is used to represent the two-dimensional coordinate offset of the predicted 14 key points, num kpts_conf =14 is used to indicate the probability of the existence of 14 key points.

[0084] Finally, the posture estimation device outputs the network result p out Decompose from the channel dimension and assign the 1st dimension, 2nd to 29th dimensions, and 30th to 43rd dimensions to P respectively. cls 、P kpt and P kpt_conf Among them, P cls Represents the classification result, P kpt Represents the offset compared to the key point prediction of the first stage and P kpt_conf Indicates the probability of the key point existing.

[0085] Step S16: combining the first key point detection result and the second key point detection result to obtain a final key point detection result.

[0086] In the embodiment of the present application, the posture estimation device obtains the key point coordinates A predicted by the first stage key point detection module according to the first key point detection result. Then, the offset P predicted by the second stage detection module compared with the first stage key point prediction is used. kpt Adjust the key point coordinate A to achieve the final key point detection result:

[0087] P kpt ′=A+P kpt

[0088] Step S17: Determine the target posture in the image to be detected using the final key point detection result.

[0089] In the embodiment of the present application, the posture estimation device uses the classification result P in the second key point detection result cls And the probability of the key point P kpt_conf Determine the final category of each key point of the target, then obtain the minimum enclosing rectangle of each target's 14 key points as the target's detection rectangle, and output the final posture detection result.

[0090] Furthermore, the posture estimation device can also use non-maximum suppression (NMS) to filter the generated multiple minimum bounding rectangles to obtain the final network detection result of the input image.

[0091] In an embodiment of the present application, a posture estimation device extracts a convolution feature map of an image to be detected; uses the convolution feature map to obtain a first key point detection result of the image to be detected, wherein the first key point detection result includes the predicted key point coordinates of each feature point in the convolution feature map; extracts instance-aware posture features from the convolution feature map based on the predicted key point coordinates; performs feature fusion on the instance-aware posture features, the preset global shared posture features, and the convolution feature map to obtain a fused posture feature; performs key point detection on the fused posture feature to obtain a second key point detection result; combines the first key point detection result and the second key point detection result to obtain a final key point detection result; and uses the final key point detection result to determine the target posture in the image to be detected. In the above manner, the posture estimation device enhances the original feature map by respectively encoding the features of specific instance posture information and shared posture information, and finally further performs posture detection on the enhanced feature map, thereby achieving simple and accurate posture estimation of all human targets in the image.

[0092] This application proposes an instance-aware shared posture feature fusion module in the process of building a convolutional neural network. This module predicts the key point positions of each instance target through a one-stage key point detection module, and samples the features of all key point positions on the original convolution feature map using bilinear interpolation, which are recorded as instance-aware posture features. At the same time, global shared posture features are set in advance and adaptively optimized during network training. These two features, which encode specific instance posture information and shared posture information respectively, are then used to enhance the original feature map. Finally, posture detection is further performed on the enhanced feature map, achieving simple and accurate posture estimation of all human targets in the image.

[0093] This application estimates accurate human posture by performing secondary regression on the key points of all human targets in the image. Specifically, it first performs a first-stage key point detection on the extracted convolutional feature map, and then enhances the original convolutional feature map by extracting instance-aware posture features and pre-set global shared posture features based on the detection results. Secondary key point detection is performed on the enhanced feature map to correct the deviation of the key point positions predicted in the first stage, thereby achieving accurate and fast posture estimation of human targets in the image.

[0094] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0095] In order to implement the posture estimation method of the above embodiment, this application also proposes a posture estimation device, please refer to Figure 5 , Figure 5 It is a structural diagram of an embodiment of a posture estimation device provided by this application.

[0096] The posture estimation device 300 of the embodiment of the present application includes a memory 31 and a processor 32 , wherein the memory 31 and the processor 32 are coupled.

[0097] The memory 31 is used to store program data, and the processor 32 is used to execute the program data to implement the posture estimation method described in the above embodiment.

[0098] In this embodiment, the processor 32 may also be referred to as a CPU (Central Processing Unit). The processor 32 may be an integrated circuit chip having signal processing capabilities. The processor 32 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor, or the processor 32 may be any conventional processor.

[0099] In order to implement the posture estimation method of the above embodiment, the present application also provides a computer-readable storage medium, such as Figure 6 As shown, the computer-readable storage medium 400 is used to store program data 41. When the program data 41 is executed by the processor, it is used to implement the posture estimation method described in the above embodiment.

[0100] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the posture estimation method as described in the embodiment of the present application. The computer program product can be a software installation package.

[0101] The posture estimation method described in the above embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0102] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A posture estimation method, characterized in that: The posture estimation method comprises: Extract the convolution feature map of the image to be detected; Obtaining a first key point detection result of the image to be detected using the convolution feature map, wherein the first key point detection result includes predicted key point coordinates of each feature point in the convolution feature map; extracting instance-aware pose features from the convolutional feature map based on the predicted keypoint coordinates; Fusing the instance-perceived posture feature, the preset global shared posture feature, and the convolutional feature map to obtain a fused posture feature; Performing key point detection on the fused posture feature to obtain a second key point detection result; Combining the first key point detection result and the second key point detection result to obtain a final key point detection result; Determining the target posture in the image to be detected using the final key point detection result; The extracting instance-aware posture features from the convolutional feature map based on the predicted key point coordinates includes: Performing feature sampling at a position of a key point corresponding to the predicted key point coordinates in the convolution feature map to obtain a key point feature set, wherein the key point feature set includes key point features corresponding to each feature point in the convolution feature map; Determining the key point feature set as the instance-aware posture feature; The global shared posture feature is used to encode the shared information of each key point between different target instances. The global shared posture feature is obtained by learning and optimizing through network training after random initialization.

2. The posture estimation method according to claim 1, wherein The step of fusing the instance-perceived posture feature, the preset global shared posture feature, and the convolutional feature map to obtain a fused posture feature includes: Obtaining instance features at each position in the convolutional feature map; Acquire a first feature relationship between the instance feature and the instance perception posture feature at the same position; Acquire a second feature relationship between the instance feature and the global shared posture feature at the same position; The convolution feature map is concatenated with the first feature relationship and the second feature relationship in a channel dimension to obtain the fused posture feature.

3. The posture estimation method according to claim 2, wherein The obtaining of a first feature relationship between the instance feature and the instance perception posture feature at the same position includes: Obtaining a transposed matrix of the instance-perceived posture feature; Obtaining normalized parameters of the transposed matrix and the instance features; The first feature relationship is calculated based on the normalized parameter and the instance-aware posture feature.

4. The posture estimation method according to claim 1, wherein The second key point detection result includes a key point classification result, a key point prediction offset, and a key point existence probability; wherein the key point classification result includes a key point prediction category and its confidence.

5. The posture estimation method according to claim 4, characterized in that Combining the first key point detection result and the second key point detection result to obtain a final key point detection result includes: Acquire predicted key point coordinates based on the first key point detection result; The predicted key point coordinates are adjusted using the key point prediction offset to obtain the final key point detection result.

6. The posture estimation method according to claim 5, characterized in that Determining the target posture in the image to be detected by using the final key point detection result includes: According to the final key point detection result, the position of each key point is obtained; The minimum bounding rectangle of all key points is used as the detection frame of the target; Output the target detection box as the detection result of the target posture.

7. A posture estimation device, characterized in that: The posture estimation device includes a processor and a memory, wherein program data is stored in the memory, and the processor is configured to execute the program data to implement the posture estimation method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program data, and when the program data is executed by the processor, it is used to implement the posture estimation method according to any one of claims 1 to 6.

Images