A multi-person 3D pose estimation method and system based on explicit limb representation

Abstract

The present application belongs to the field of 3D human posture estimation, and particularly relates to a multi-person 3D posture estimation method and system based on explicit limb representation, comprising: acquiring a multi-person scene image, inputting the multi-person scene image into a trained multi-person 3D posture estimation model to obtain a key point heat map, a limb orientation vector field, a limb relative depth map and a root key point absolute depth map, and post-processing the key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map to obtain an estimated multi-person 3D posture; the present application encodes the limb information of a person into a two-dimensional limb orientation vector field and a limb relative depth map to ensure that the matching between candidate key points can directly utilize the most intuitive limb information, so as to improve the phenomenon of incorrect matching of candidate key points of different human bodies and improve the quality of the matched posture.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of multi-person 3D pose estimation in the sub-problem of 3D human pose estimation, and particularly relates to a multi-person 3D pose estimation method and system based on explicit limb representation. BACKGROUND

[0002] Human pose estimation is an important research direction in the field of computer vision, and the task aims to locate human body parts and construct human representation (such as human skeleton) according to input data such as images and videos. In the single-person scene, the pose estimation algorithm needs to locate the specified parts of the human body, such as the neck, left / right shoulder, left / right elbow, which are also called human pose key points. In a more general case, the number of people in the scene is unknown, and the pose estimation algorithm not only needs to detect the key points of all human bodies, but also needs to associate each key point to the corresponding human body instance. According to different task requirements, human pose estimation can also be divided into two-dimensional pose estimation and three-dimensional pose estimation, and there are also great differences between the two. The two-dimensional pose estimation task includes predicting the positions of the key points in the two-dimensional space, and usually only involves the prediction of the X, Y pixel coordinates of the key points. The three-dimensional pose estimation usually needs to predict the X, Y, Z coordinates of the key points in the camera space coordinate system, and needs to additionally predict the depth information of each key point. In some complex scenes, the number of people is usually unknown, therefore, it is very important for multi-person 3D pose estimation to fully utilize the limb features to simultaneously predict the three-dimensional poses of all people.

[0003] However, in the actual process, multi-person 3D pose estimation simultaneously faces the problems brought by multi-person scenes and depth prediction: multi-person scenes will inevitably have occlusion and interaction between people, when the spatial positions of the key points of different human bodies are highly intertwined and overlapped, the model is difficult to accurately detect all the key points, and it is even more difficult to distinguish different human body instances; since the three-dimensional information projected to the two-dimensional image has lost one dimension, recovering the three-dimensional pose from the two-dimensional is an inverse problem that does not have a stable unique solution, and one two-dimensional pose can correspond to multiple three-dimensional poses without determining the depth information, thus causing depth ambiguity. Therefore, multi-person 3D pose estimation still faces many difficulties.

[0004] Existing multi-person 3D pose estimation algorithms are mostly divided into three types: top-down method, bottom-up method and single-stage method. The top-down method first detects a set of human region frames from the input image through an existing human detector, and then detects the key points of each person in the human boundary frame area using a heat map to generate multi-person pose. However, the human detector will bring greater computational complexity, and the complexity is related to the number of people in the scene, which exists redundant calculation and is difficult to cope with real-time scenes. The bottom-up method and the single-stage method do not need an additional human detector. The bottom-up method detects all the key points of the people in the first stage, and groups the key points detected in the first stage in the second stage to realize the association of the key points and the human instances. Therefore, the computational complexity is greatly reduced, and there is no detection frame constraint, which can fully utilize the global context clues to locate the body key points. However, the detection effect of the key points that are occluded or difficult to identify is not ideal because it is difficult to capture useful information at the corresponding pixel position, and the association of the candidate key points is prone to error association. The single-stage method usually locates the body key points based on the vector at the center of the human body. However, it is difficult to capture useful information when processing long-distance vectors, and the limited features at the center are difficult to encode the flexible pose of the whole human body.

[0005] Therefore, how to estimate the 3D poses of all people in a complex multi-person scene at the same time is a problem to be solved. SUMMARY

[0006] To solve the above technical problems, the present application adopts a multi-person 3D pose estimation method based on explicit limb representation, which is characterized by comprising: acquiring a multi-person scene image, inputting the multi-person scene image into a trained multi-person 3D pose estimation model to obtain a key point heat map, a limb orientation vector field, a limb relative depth map and a root key point absolute depth map, and post-processing the key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map to obtain an estimated multi-person 3D pose; the multi-person 3D pose estimation model comprises: an image pose feature extraction module, a two-dimensional pose representation encoding module and a depth representation encoding module.

[0007] The training process of the multi-person 3D pose estimation model comprises:

[0008] S1, acquiring a multi-person scene image training set, the training set comprising a multi-person scene image and a real label; pre-processing the multi-person scene image;

[0009] S2, inputting the pre-processed multi-person scene image into the image pose feature extraction module to extract features and obtain a feature map;

[0010] S3, inputting the feature map into the two-dimensional pose representation encoding module to extract two-dimensional pose information and obtain a predicted key point heat map and a limb orientation vector field;

[0011] S4, inputting the feature map and the predicted limb orientation vector field into a depth representation encoding module to extract human depth information, to obtain a predicted limb relative depth map and a root key point absolute depth map;

[0012] S5, calculating a loss function value according to the real label and the predicted key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map, and updating the multi-person 3D pose estimation model parameters according to the loss function value, and obtaining the trained multi-person 3D pose estimation model when the loss function value is the minimum.

[0013] The preprocessing of the multi-person scene image includes:

[0014] S11, performing random rotation and flipping on the multi-person scene image to obtain an enhanced multi-person scene image;

[0015] S12, performing same-proportion scaling on the width and height of the enhanced multi-person scene image to ensure that at least one of the width and height is aligned with the input size of the image pose feature extraction module;

[0016] S13, performing center padding on the scaled image.

[0017] The image pose feature extraction module is an HRNet network, and the HRNet network includes four branches of different resolutions.

[0018] The two-dimensional pose representation encoding module includes a feature extraction unit, a first channel conversion output module and a second channel conversion output module; the processing of the two-dimensional pose representation encoding module on the feature map includes: inputting the feature map into the feature extraction unit to obtain a two-dimensional pose feature map; inputting the two-dimensional pose feature map into the first channel conversion output module and the second channel conversion output module respectively to obtain a predicted key point heat map and a limb orientation vector field.

[0019] The feature map output by the image pose feature extraction module includes four feature maps F b ,b∈[1,4]; the feature extraction unit includes four branches and a channel conversion convolution, each branch includes a feature extraction module, and the feature extraction module is stacked by four Basic Blocks; the processing of the feature extraction unit on the feature map includes: inputting the four feature maps F b into the feature extraction module of the corresponding bth branch respectively to obtain four feature maps F′ b , and selecting the feature map F′ b with the highest resolution from the four feature maps F′ , and using bilinear interpolation to upsample the resolutions of the remaining three feature maps F′ b to the resolution of the feature map F′ three up-sampled feature maps F" b and three up-sampled feature maps F" b are combined, and the combined result is input into a channel conversion convolution to obtain a two-dimensional pose feature map; wherein b is the index of the feature map and the branch, and Basic Block is a basic block, which is a basic building unit of a residual network.

[0020] The depth representation encoding module comprises a feature extraction unit, an absolute depth output module and a channel conversion output module; the processing of the feature map and the predicted limb orientation vector field by the depth representation encoding module comprises: inputting the feature map into the feature extraction unit to obtain a depth information feature map, inputting the depth information feature map into the channel conversion output module to obtain a predicted limb relative depth map; and inputting the depth information feature map and the predicted limb orientation vector field into the absolute depth output module to obtain a predicted root key point absolute depth map.

[0021] The absolute depth output module comprises a first convolution block, a second convolution block and a Warp layer; the processing of the depth information feature map and the predicted limb orientation vector field by the absolute depth output module comprises: inputting the depth information feature map into the first convolution block, inputting the output of the first convolution block and the predicted limb orientation vector field into the Warp layer, inputting the output of the Warp layer into the second convolution block to obtain the predicted root key point absolute depth map.

[0022] The calculation of the loss function value according to the real label and the predicted key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map comprises: generating a key point heat map, a limb orientation vector field, a limb relative depth map and a root key point absolute depth map of a real label; and calculating a loss function value according to the key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map of the real label and the predicted key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map; wherein the calculation formula of the loss function value is:

[0023]

[0024] wherein, is the final loss function, represents an L2 loss for the key point heat map, represents a smooth L1 loss for the limb orientation vector field, the smooth L1 loss being a smooth L1 loss, represents a smooth L1 loss for the limb relative depth map, represents an L1 loss for the root key point absolute depth map, λ o , λ rel , λ​abs To balance the hyperparameters of different loss terms.

[0025] The post-processing of the keypoint heat map, the limb orientation vector field, the limb relative depth map, and the root keypoint absolute depth map includes:

[0026] Step 1, using the NMS algorithm to detect candidate key points from the keypoint heat map, and calculating the matching weight between the candidate key points from the limb orientation vector field; wherein the NMS algorithm is a non-maximum suppression algorithm;

[0027] Step 2, using the depth preferred matching algorithm to match the candidate key points according to the matching weight between the candidate key points and the absolute depth map of the root keypoint, and calculating the absolute depth value of each matched key point according to the limb relative depth map;

[0028] Step 3, using the limb orientation vector field and the matched key points in step 2 to infer the key points not detected in the keypoint heat map, to obtain the pixel position and absolute depth value of the inferred key points;

[0029] Step 4, using the perspective camera model to reconstruct the multi-person 3D pose according to the pixel position and absolute depth value of the matched key points and the inferred key points.

[0030] A system using the above-mentioned multi-person 3D pose estimation method based on explicit limb representation, comprising: an image input module, an image enhancement preprocessing module, an image pose feature extraction module, a two-dimensional pose representation encoding module, a depth representation encoding module, and a post-processing module;

[0031] The image input module is used to input the image;

[0032] The image enhancement preprocessing module is used to preprocess the image;

[0033] The image pose feature extraction module is used to extract the feature map of the preprocessed image;

[0034] The two-dimensional pose representation encoding module is used to generate the keypoint heat map and the limb orientation vector field according to the feature map;

[0035] The depth representation encoding module is used to generate the limb relative depth map and the root keypoint absolute depth map according to the feature map and the limb orientation vector field;

[0036] The post-processing module is used to reconstruct the multi-person 3D pose according to the keypoint heat map, the limb orientation vector field, the limb relative depth map, and the root keypoint absolute depth map.

[0037] Beneficial effects:

[0038] This invention ensures that the matching between candidate keypoints can directly utilize the most intuitive limb information by encoding human limb information into a two-dimensional limb orientation vector field and a limb relative depth map. This improves the phenomenon of incorrect matching of candidate keypoints for different human bodies and enhances the quality of the matched pose. Furthermore, the limb orientation vector is redundantly encoded on two keypoints of the limb. When one keypoint is not detected due to occlusion or human-to-human interaction, the location of the undetected keypoint can be inferred from the limb orientation vector field at the detected keypoint location. This greatly improves the robustness of the model to occluded images and also improves the estimation accuracy. Attached Figure Description

[0039] Figure 1 A flowchart of a multi-person 3D pose estimation method based on explicit limb representation provided in an embodiment of the present invention;

[0040] Figure 2 This is an overall framework diagram of the multi-person 3D pose estimation model provided in the embodiments of the present invention;

[0041] Figure 3 A flowchart of image preprocessing provided in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the structure of a two-dimensional pose representation encoding module provided in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the structure of the depth representation encoding module provided in an embodiment of the present invention;

[0044] Figure 6 A schematic diagram of the limb orientation vector field provided in an embodiment of the present invention;

[0045] Figure 7 This invention provides a visualization interface for a multi-person 3D pose estimation system based on explicit limb representation, as provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] like Figure 1 , Figure 2 As shown, this invention employs a multi-person 3D pose estimation method based on explicit limb representation, comprising the following steps:

[0048] The multi-person scene image is input into the trained multi-person 3D pose estimation model to obtain a key point heat map, a limb orientation vector field, a limb relative depth map and a root key point absolute depth map, and the key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map are post-processed to obtain an estimated multi-person 3D pose; the multi-person 3D pose estimation model comprises an image pose feature extraction module, a two-dimensional pose representation encoding module and a depth representation encoding module;

[0049] The training process of the multi-person 3D pose estimation model comprises:

[0050] S1, obtaining a multi-person scene image training set, the training set comprising multi-person scene images and real labels; the multi-person scene images are preprocessed;

[0051] S2, inputting the preprocessed multi-person scene images into the image pose feature extraction module to extract features to obtain a feature map;

[0052] S3, inputting the feature map into the two-dimensional pose representation encoding module to extract two-dimensional pose information to obtain a predicted key point heat map and a limb orientation vector field;

[0053] S4, inputting the feature map and the predicted limb orientation vector field into the depth representation encoding module to extract human depth information to obtain a predicted limb relative depth map and a root key point absolute depth map;

[0054] S5, calculating a loss function value according to the real labels and the predicted key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map, updating the multi-person 3D pose estimation model parameters according to the loss function value, and obtaining the trained multi-person 3D pose estimation model when the loss function value is the minimum.

[0055] The multi-person scene image training set comprises a MuCo-3DHP dataset and a COCO dataset, the MuCo-3DHP containing 200,000 synthetic images with 3D pose labels, and the COCO dataset containing 2D pose labels being used to mix training to enhance generalization performance, and 50% of the COCO dataset being mixed in each small batch.

[0056] As shown in Figure 3 , the preprocessing of the multi-person scene image comprises:

[0057] S11, the multi-person scene image is x∈R H×W×C , wherein H, W and C represent height, width and channel number respectively; the multi-person scene image is randomly rotated and flipped to obtain an enhanced multi-person scene image;

[0058] S12, scale the width and height of the enhanced multi-person scene image by the same ratio to ensure that at least one of the width and height is aligned with the input size (scale) 512x832x3 of the image pose feature extraction module;

[0059] scale = min(H / 512, W / 832)

[0060] S13, center-fill the scaled image to 512x832x3 size.

[0061] The image pose feature extraction module is an HRNet network (high resolution network). The HRNet network adopts a convolutional network HRNet-W32 suitable for dense prediction tasks. The HRNet-W32 includes four parallel branches of different resolutions (128x208, 64x104, 32x52, 16x26). The output of each branch is a feature map F b , b e [1, 4]; wherein the branch with lower resolution is obtained by gradually fusing the features of other branches from the high-resolution branch and down-sampling. Each branch uses a stacked Basic Block as a feature extraction unit. Each Basic Block consists of a 3x3 convolution, a normalization layer (2D BatchNorm), and a ReLU activation layer. Basic Block is a basic building block in a residual network (ResNet).

[0062] The initial weights of the backbone network are ImageNet pre-training weights.

[0063] As shown in Figure 4 , the two-dimensional pose representation encoding module includes a feature extraction unit, a first channel conversion output module, and a second channel conversion output module. The processing of the feature map by the two-dimensional pose representation encoding module includes: inputting the feature map into the feature extraction unit to obtain a two-dimensional pose feature map; inputting the two-dimensional pose feature map into the first channel conversion output module and the second channel conversion output module, respectively, to obtain a predicted J-channel key point heat map and a 2x(J-1)-channel limb orientation vector field; J is the number of key point types.

[0064] The feature extraction unit includes four parallel branches b and a channel conversion convolution. Each branch includes a feature extraction module, which is stacked by Basic Blocks. The processing of the feature map by the feature extraction unit includes: inputting the four feature maps F b into the feature extraction module of the corresponding b-th branch, respectively, to obtain four feature maps F′ b of different resolutions. Among the four feature maps F′ b of different resolutions, the feature map F′ And bilinear interpolation is used to transform the remaining three feature maps F′ b Upsampled to feature map at resolution At a resolution of 128×208, three upsampled feature maps F″ were obtained. b ;Will and three upsampled feature maps F″ b By concatenating the inputs along the channel dimension into a channel transformation convolution (1×1Conv2d), the number of channels is transformed to 256, resulting in a two-dimensional pose feature map.

[0065] Both the first-channel conversion output module and the second-channel conversion output module include multi-layer convolution.

[0066] In one embodiment, both the first channel conversion output module and the second channel conversion output module include (C) out (Number of output channels for different encodings): First 1×1 convolution (Conv2d), with 256 input and 256 output channels; First normalization layer BatcNorm2d, with 256 input channels; First activation layer ReLU, with 256 input channels; Second 3×3 convolutional layer (Conv2d), with 256 input and 256 output channels. out The second normalization layer, BatcNorm2d, has C input channels. out .

[0067] like Figure 5 As shown, the depth representation encoding module includes: a feature extraction unit, an absolute depth output module, and a channel conversion output module; the depth representation encoding module processes the feature map and the predicted limb orientation vector field by: inputting the feature map into the feature extraction unit to obtain a depth information feature map; inputting the depth information feature map into the channel conversion output module to obtain a predicted limb relative depth map; and inputting the depth information feature map and the predicted limb orientation vector field into the absolute depth output module to obtain a predicted root keypoint absolute depth map.

[0068] The absolute depth output module structure includes: a first convolutional block, a second convolutional block, and a Warp layer; the absolute depth output module processes the depth information feature map and the predicted limb orientation vector field by: inputting the depth information feature map into the first convolutional block, inputting the output of the first convolutional block and the predicted limb orientation vector field into the Warp layer (mapping layer), and inputting the output of the Warp layer into the second convolutional block to obtain the predicted root keypoint absolute depth map.

[0069] The Warp layer takes out the limb vector field corresponding to the root key point (pelvis) and the neck key point in the limb orientation vector field. The pelvis pixel position of the vector field encodes the two-dimensional vector from the pelvis to the neck, and the Warp layer uses the two-dimensional vector from the pelvis to the neck to perform a mapping transformation on the output feature map of the first convolution block: the depth information feature at the neck pixel position is mapped to the pelvis pixel position to obtain the absolute depth feature map F w , which encodes the depth feature at the neck key point at the root key point pelvis pixel position. w The absolute depth feature map F in after the mapping transformation is combined with the output feature map F out of the first convolution block to obtain the predicted root key point absolute depth map F w , and the output channel is 512.

[0070] Specifically, the Warp layer can be represented as:

[0071] F w (x,y)=Warp(x+O x (x,y),y+O y (x,y))

[0072] F out =Concat(F w ,F in )

[0073] Where O x , O y are two component channels of the limb two-dimensional vector from the pelvis to the neck, and F in is the output of the first convolution block.

[0074] In one embodiment, the first convolution block is sequentially connected with: 1x1Conv2d (input and output channel numbers are 256 and 256 respectively), BatcNorm2d (input channel number is 256), RELU (input channel is 256). The second convolution block is sequentially connected with: 1x1Conv2d (input and output channel numbers are 512 and 256 respectively), BatcNorm2d (input channel number is 256), RELU (input channel is 256), 3x3Conv2d (input and output channel numbers are 256), BatcNorm2d (input channel number is 1).

[0075] Generating the key point heat map of the real label, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map includes:

[0076] The label in the pose estimation dataset is a series of key point coordinate arrays, which mainly includes two-dimensional pixel coordinates (x, y) of each key point, real coordinates (X, Y, Z) in the three-dimensional camera coordinate system, and camera intrinsic matrix K. When loading the dataset, the real labels of the matching four intermediate representations are further generated according to the label in the form of these coordinates:

[0077] Generating a key point heat map includes: generating a heat map real label H* of J channels with the same size as the predicted key point heat map H according to the two-dimensional pixel coordinates (x, y) of the key point, specifically, encoding the confidence in the form of a Gaussian kernel in the region around the key point, and setting it to 0 outside the region.

[0078] Figure 6 For the limb orientation vector field between the two key points of the neck and the top of the head, generating the limb orientation vector field and the limb relative depth map includes: generating J-1 corresponding 3D vector fields for the J-1 limbs of the human body, each vector field corresponds to the two-dimensional vector and the relative depth between the two key points on the limb, the two-dimensional vector is obtained by subtracting the pixel coordinates x and y of the two key points, and the relative depth is obtained by subtracting the Z coordinates of the two key points. The two-dimensional vector and the relative depth are not consistent in units and have large semantic differences in prediction, so they are supervised respectively as two intermediate feature maps, and the two-dimensional vector and the relative depth are also encoded only in the key point region, and the other pixel positions are 0, to obtain a two-dimensional limb orientation vector field and a relative depth map for each limb.

[0079] Generating the absolute depth map of the root key point: because only the absolute depth on the pelvic pixel position is supervised, there is no need to generate additional feature map labels for large-scale regional supervision, and only the original label Z on the pelvic pixel (x, y) is used for supervision.

[0080] Since the image is randomly rotated and scaled, the two-dimensional pixel position of the key point changes, but the absolute depth remains unchanged, so the pixel position transformation of the key point heat map, the limb orientation vector field, and the limb relative depth map is also needed, and the pelvic real pixel position (x, y) required for supervision also needs to be modified.

[0081] Loss function of the model is:

[0082]

[0083] wherein, represents the L2 loss of the key point heat map, J is the number of key points, H k and are the predicted key point heat map of the key point k and the real key point heat map of the key point k, respectively; represents the smooth L1 loss function for limb orientation vector field, S is the set of pixels i around two key points on the limb, o i and are the predicted and real limb orientation vector of pixel i respectively; represents the smooth L1 loss function for limb relative depth map, Δz i and are the predicted and real relative depth of pixel i respectively; represents the L1 loss for root key point absolute depth map, z n and are the predicted and real absolute depth of n root key points pixels in root key point absolute depth map respectively, N represents the number of people in the multi-person scene image; λ0, λ rel , λ abs are the hyper-parameters to balance different loss terms.

[0084] L2 loss is also known as Mean Squared Error (MSE) loss, L1 loss is also known as Mean Absolute Error (MAE) loss, and Smooth L1 loss is known as Huber loss or Smooth L1 loss, which is a loss function combining the advantages of L1 loss and L2 loss.

[0085] The post-processing of the key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map includes:

[0086] Step 1, using Non-Maximum Suppression (NMS) algorithm to detect the pixel position (x, y) of the candidate key point from the key point heat map, and calculating the matching weight between the candidate key points from the limb orientation vector field;

[0087] The matching weight calculation method between two candidate key points i, j on the limb is as follows:

[0088] p′ j = p i + o i

[0089] p′ i = p j + o j

[0090]

[0091] where p i , p j are the 2D pixel positions of the two candidate key points, o i and oj are the limb orientation vectors read out at the pixel positions of the p i , p j are the limb orientation vectors read out at the pixel positions of the p j is the position of the jth keypoint on the ith limb inferred from the position of the candidate keypoint i and the limb orientation vector at the position, p i Similarly, To calculate the pixel distance between the inferred position and the detected position, which is used to measure the matching degree of two candidate points.

[0092] Step 2, match the candidate key points according to the matching weight between the candidate key points and the absolute depth map of the root key point using a depth-preferred matching algorithm, and calculate the absolute depth value of each matched key point according to the limb relative depth map;

[0093] Matching candidate key points using a depth-preferred matching algorithm includes:

[0094] Construct a weighted bipartite graph composed of matching weights between candidate key points, determine the number of people in the scene based on the number of root key points, and preferentially match the human body with smaller absolute depth value in order of the absolute depth value of the root key point; Specifically, according to the weighted bipartite graph, start matching from the root key point with the smallest absolute depth value, select the key point with the matching weight Optimal key point, and then sequentially match from the pelvic bone to the end key point of the limbs along the human skeleton path.

[0095] In the process of matching, the absolute depth Z of each key point is also gradually inferred: first, the absolute depth value of the root key point is directly read out from the absolute depth map at the pixel position of the candidate root key point, in the process of matching, the absolute depth value of the matched child key point can be calculated by the formula , wherein are the absolute depth values of the matched parent and child key points on the lth limb, and in each round of matching In the last round, Δz l represents the relative depth difference of the two key points on the limb read from the limb relative depth map, and thus the absolute depth value Z of each matched key point is recursively obtained by accumulating the relative depth value along the skeleton path from the known absolute depth value of the root key point.

[0096] Step 3, infer the key points in the key point heat map that are not detected using the limb orientation vector field and the matched key points in step 2, to obtain the pixel position and absolute depth value of the inferred key points.

[0097] Reasoning undetected key points by using limb orientation vector field: In the last depth-first matching process, due to occlusion, human interaction and other complex reasons, part of the key points may not be detected by the key point heat map, so in the two-part graph, there will be a problem that the number of parent key points is more than that of child key points, that is, part of the child key points are not detected, resulting in that the parent key points with low depth priority have no matching child key points, at this time, the limb orientation vector field is redundantly encoded on the pixel position of the parent key point, the vector from the parent key point to the child key point can be read from the vector field, and the undetected key points can be reasoned by relying on the vector to obtain the pixel position and absolute depth value of the reasoned key points.

[0098] Step 4, reconstructing the multi-person 3D pose by using the perspective camera model according to the pixel position and absolute depth value of the matched key points and the reasoned key points:

[0099] [X,Y,Z] T =ZK -1 [x,y,1] T

[0100] Wherein, K is the camera intrinsic matrix, Z is the absolute depth value of the key point, (x, y) is the pixel position of the key point, [X, Y, Z] T is the multi-person 3D coordinates to be solved in the camera coordinate system, and T represents transposition.

[0101] The application also provides a multi-person 3D pose estimation system based on explicit limb representation, comprising: an image input module, an image enhancement preprocessing module, an image pose feature extraction module, a two-dimensional pose representation encoding module, a depth representation encoding module and a post-processing module.

[0102] The image input module is used for inputting images.

[0103] The image enhancement preprocessing module is used for pre-processing the images.

[0104] The image pose feature extraction module is used for extracting the feature map of the pre-processed images.

[0105] The two-dimensional pose representation encoding module is used for generating the key point heat map and the limb orientation vector field by using the feature map of the pre-processed images.

[0106] The depth representation encoding module is used for generating the limb relative depth map and the root key point absolute depth map by using the feature map of the pre-processed images.

[0107] The post-processing module comprises a candidate key point detection module and a key assembly and reasoning module.

[0108] The key point detection module is used for extracting the candidate key points from the key point heat map and calculating the matching weight between the candidate key points by using the limb orientation vector field.

[0109] The key assembly and reasoning module is used for assembling the detected candidate key points and reasoning the undetected key points to obtain the positions of the final key points, and reconstructing the multi-person 3D pose according to the positions of the final key points.

[0110] As shown in Figure 7 The application further provides a visualization interface of a multi-person 3D pose estimation system based on explicit limb representation, which comprises a file selection interface, an image selection interface, a start prediction button, an image preview interface and a prediction result interface.

[0111] The file selection interface is used for selecting a local multi-person 3D pose estimation model and displaying the storage path of the multi-person 3D pose estimation model; the image selection button is used for selecting an input image of the multi-person 3D pose estimation model and displaying the storage path of the input image; the start prediction button is used for starting the estimation of the multi-person 3D pose; the image preview interface is used for displaying the input image; and the prediction result interface is used for displaying the reconstructed multi-person 3D pose.

[0112] The multi-person 3D pose estimation based on explicit limb representation of the application can be applied to multiple scenes:

[0113] 1. Sports training

[0114] The multi-person 3D pose estimation technology can track and analyze the posture and action of athletes, help coaches evaluate the technical level of athletes, and provide targeted training suggestions. By accurately capturing and analyzing the actions of athletes, coaches can timely find and correct the wrong actions of athletes, and improve the training effect.

[0115] 2. Virtual reality system

[0116] In a virtual reality environment, 3D multi-person pose estimation can capture and reconstruct the body posture of users in real time, thereby providing a more natural and smooth interactive experience. Users can directly interact with the virtual environment through body movements, enhancing the sense of immersion; the technology can also dynamically adjust the lighting, shadow and other effects in the virtual scene according to the posture information of the user, making the scene rendering more realistic.

[0117] 3. Game development

[0118] In game development, 3D multi-person pose estimation technology can allow players to control game characters through body movements, improving the interactivity and interest of the game; the technology can also realize real-time interaction between game characters and game scenes, enhancing the sense of immersion and reality of the game.

[0119] 4. Medical health

[0120] In the medical field, this technology can be used for rehabilitation training to help patients recover physical functions. By capturing the patient's movements in real time, doctors can assess the patient's rehabilitation progress and adjust the rehabilitation plan; combined with telemedicine technology, doctors can also remotely guide patients in rehabilitation training, improving the accessibility and efficiency of medical services.

[0121] 5、Robotics and intelligent manufacturing

[0122] In the field of robotics, multi-person 3D pose estimation can assist robots in understanding and responding to human instructions and actions, enabling more intelligent interaction and operation; in the field of intelligent manufacturing, this technology can be used to monitor the postures and actions of workers on the production line, ensuring the standardization and safety of the production process.

[0123] The above examples further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and do not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for estimating the pose of multiple people in 3D based on explicit limb representation, characterized in that, include: Acquire multi-person scene images, input the multi-person scene images into the trained multi-person 3D pose estimation model, and obtain key point heatmaps, limb orientation vector fields, limb relative depth maps and root key point absolute depth maps. Post-process the key point heatmaps, limb orientation vector fields, limb relative depth maps and root key point absolute depth maps to obtain the estimated multi-person 3D poses. The multi-person 3D pose estimation model includes: an image pose feature extraction module, a two-dimensional pose representation encoding module, and a depth representation encoding module; The training process of a multi-person 3D pose estimation model includes: S1. Obtain a training set of multi-person scene images, which includes multi-person scene images and real labels; preprocess the multi-person scene images. S2. Input the preprocessed multi-person scene image into the image pose feature extraction module to extract features and obtain a feature map; S3. Input the feature map into the two-dimensional pose representation encoding module to extract two-dimensional pose information and obtain the predicted key point heatmap and limb orientation vector field. S4. Input the feature map and the predicted limb orientation vector field into the depth representation encoding module to extract human body depth information and obtain the predicted limb relative depth map and root key point absolute depth map. S5. Calculate the loss function value based on the real labels and the predicted key point heatmap, limb orientation vector field, limb relative depth map and root key point absolute depth map. Update the parameters of the multi-person 3D pose estimation model based on the loss function value. When the loss function value is minimized, the trained multi-person 3D pose estimation model is obtained. The two-dimensional pose representation encoding module includes: a feature extraction unit, a first channel conversion output module, and a second channel conversion output module; the two-dimensional pose representation encoding module processes the feature map by: inputting the feature map into the feature extraction unit to obtain a two-dimensional pose feature map; and inputting the two-dimensional pose feature map into the first channel conversion output module and the second channel conversion output module respectively to obtain a predicted key point heatmap and a limb orientation vector field; The feature maps output by the image pose feature extraction module include four feature maps F at different resolutions. b ,b∈[1,4]; The feature extraction unit includes four branches and a channel transformation convolution. Each branch includes a feature extraction module, which is composed of four Basic Blocks stacked together; The feature extraction unit processes the feature maps including: converting the four feature maps F b Input the corresponding feature extraction module of the b-th branch to obtain four feature maps F′ with different resolutions. b In four feature maps F′ at different resolutions b Select the highest resolution feature map And bilinear interpolation is used to transform the remaining three feature maps F′ b Upsampled to feature map at resolution At a resolution of [resolution], three upsampled feature maps F″ are obtained. b ;Will and three upsampled feature maps F″ b The combined result is then input into the channel for convolution to obtain a two-dimensional pose feature map; where b is the index of the feature map and branch, and Basic Block is a basic building block, which is a basic building unit of the residual network. The depth representation encoding module includes a feature extraction unit, an absolute depth output module, and a channel conversion output module. The depth representation encoding module processes the feature map and the predicted limb orientation vector field by: inputting the feature map into the feature extraction unit to obtain a depth information feature map; inputting the depth information feature map into the channel conversion output module to obtain a predicted relative limb depth map; and inputting the depth information feature map and the predicted limb orientation vector field into the absolute depth output module to obtain a predicted absolute depth map of the root keypoints. The absolute depth output module includes a first convolutional block, a second convolutional block, and a Warp layer. The absolute depth output module processes the depth information feature map and the predicted limb orientation vector field by: inputting the depth information feature map into the first convolutional block, inputting the output of the first convolutional block and the predicted limb orientation vector field into the Warp layer, and inputting the output of the Warp layer into the second convolutional block to obtain the predicted root keypoint absolute depth map; wherein, the Warp layer is a mapping layer. The loss function value is calculated based on the real labels and predicted keypoint heatmaps, limb orientation vector fields, limb relative depth maps, and root keypoint absolute depth maps. This includes generating the real-label keypoint heatmaps, limb orientation vector fields, limb relative depth maps, and root keypoint absolute depth maps; and calculating the loss function value based on the real-label keypoint heatmaps, limb orientation vector fields, limb relative depth maps, and root keypoint absolute depth maps, as well as the predicted keypoint heatmaps, limb orientation vector fields, limb relative depth maps, and root keypoint absolute depth maps. The formula for calculating the loss function value is as follows: in, For the final loss function, This represents the L2 loss of the keypoint heatmap. This represents the smooth L1 loss applied to the limb orientation vector field. The smooth L1 loss is the equivalent of a smoothed L1 loss. This represents the smoothL1 loss applied to the relative depth map of the limbs. λ represents the L1 loss on the absolute depth map of the root keypoints. o , λ rel , λ abs To balance the hyperparameters of different loss terms.

2. The method for estimating the pose of multiple people in 3D based on explicit limb representation according to claim 1, characterized in that, Preprocessing of multi-person scene images includes: S11. Randomly rotate and flip the multi-person scene image to obtain the enhanced multi-person scene image; S12. Scale the width and height of the enhanced multi-person scene image proportionally to ensure that at least one of the width and height is aligned with the input size of the image pose feature extraction module. S13. Center-fill the scaled image.

3. The method for estimating the pose of multiple people in 3D based on explicit limb representation according to claim 1, characterized in that, The image pose feature extraction module uses the HRNet network, which includes four branches with different resolutions; among them, the HRNet network is a high-resolution network.

4. The method for estimating the pose of multiple people in 3D based on explicit limb representation according to claim 1, characterized in that, Post-processing of keypoint heatmaps, limb orientation vector fields, limb relative depth maps, and root keypoint absolute depth maps includes: Step 1: Use the NMS algorithm to detect candidate key points from the key point heatmap and calculate the matching weights between candidate key points from the limb orientation vector field; where the NMS algorithm is a non-maximum suppression algorithm. Step 2: Use the depth-optimized matching algorithm to match candidate keypoints based on the matching weights between candidate keypoints and the absolute depth map of the root keypoint, and calculate the absolute depth value of each matched keypoint based on the relative depth map of the limbs. Step 3: Using the limb orientation vector field and the key points matched in Step 2, infer the key points that were not detected in the key point heatmap, and obtain the pixel position and absolute depth value of the inferred key points. Step 4: Reconstruct the 3D poses of multiple people using a perspective camera model based on the pixel positions and absolute depth values ​​of the matched and inferred key points.

5. A system employing a multi-person 3D pose estimation method based on explicit limb representation as described in any one of claims 1 to 4, characterized in that, include: The system includes an image input module, an image enhancement and preprocessing module, an image pose feature extraction module, a two-dimensional pose representation encoding module, a depth representation encoding module, and a post-processing module. The image input module is used to input images; The image enhancement preprocessing module is used to preprocess images; The image pose feature extraction module is used to extract feature maps from the preprocessed image; The two-dimensional pose representation encoding module is used to generate key point heatmaps and limb orientation vector fields based on feature maps; The depth representation encoding module is used to generate relative depth maps of limbs and absolute depth maps of root keypoints based on feature maps and limb orientation vector fields. The post-processing module is used to reconstruct the 3D poses of multiple people based on keypoint heatmaps, limb orientation vector fields, limb relative depth maps, and root keypoint absolute depth maps.

Images