Multi-person 3D attitude estimation method and system based on explicit limb representation

Abstract

The invention belongs to the field of 3D human body posture estimation, and particularly relates to a multi-person 3D posture estimation method and system based on explicit limb representation, and the method comprises the steps: obtaining a multi-person scene image, inputting the multi-person scene image into a trained multi-person 3D posture estimation model, a key point heat map, a limb orientation vector field, a limb relative depth map and a root key point absolute depth map are obtained, post-processing is carried out on the key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map, and an estimated multi-person 3D posture is obtained; according to the method, the most intuitive limb information can be directly utilized for matching of the candidate key points by displaying and coding the limb information of the human body into the two-dimensional limb azimuth vector field and the limb relative depth map, so that the phenomenon that the candidate key points of different human bodies are wrongly matched is improved, and the quality of matched postures is improved.

Description

Technical Field

[0001] The present invention belongs to the field of multi-person 3D posture estimation in the sub-problem of 3D human body posture estimation, and specifically relates to a multi-person 3D posture estimation method and system based on explicit limb representation. Background Art

[0002] Human pose estimation is an important research direction in the field of computer vision. The task aims to locate human body parts and construct human body representation (such as human skeleton) based on input data such as images and videos. In a single-person scene, the pose estimation algorithm needs to locate the specified parts of the human body, such as the neck, left / right shoulders, and left / right elbows. These parts are also called human pose key points. In more general cases, the number of people in the scene is unknown. The pose estimation algorithm not only needs to detect the key points of all human bodies, but also associate each key point with the corresponding human body instance. According to different task requirements, human pose estimation can also be divided into two-dimensional pose estimation and three-dimensional pose estimation, and there are also huge differences between the two. Among them, the two-dimensional pose estimation task includes predicting the position of the key points of the body in two-dimensional space, which usually only involves the prediction of the X and Y pixel coordinates of the key points. Three-dimensional pose estimation usually requires predicting the X, Y, and Z coordinates of the key points in the camera space coordinate system, and requires additional prediction of the depth information of each key point. In some complex scenes, the number of people is usually unknown. Therefore, it is very important to make full use of limb features to simultaneously predict the three-dimensional pose of all people for multi-person 3D pose estimation.

[0003] However, in practice, multi-person 3D pose estimation faces problems with multi-person scenes and depth prediction: multi-person scenes inevitably involve occlusion and communication interactions between people. When the spatial positions of key points of different human bodies are highly intertwined and overlapped, it is difficult for the model to accurately detect all key points, and it is even more difficult to distinguish different human instances. Since one dimension has been lost when projecting three-dimensional information onto a two-dimensional image, recovering three-dimensional pose from two dimensions is an inverse problem that does not have a stable unique solution. A two-dimensional pose can correspond to multiple three-dimensional poses without certain depth information, thus causing depth ambiguity. Therefore, there are still many difficulties in the implementation of multi-person 3D pose estimation.

[0004] Most of the existing multi-person 3D pose estimation algorithms are divided into three types: top-down methods, bottom-up methods, and single-stage methods. The top-down method first detects a set of human region boxes from the input image through the existing human detector, and then uses the heat map to detect the key points of a single person in each human bounding box area to generate multi-person poses. However, the human detector will bring greater computational complexity, and the complexity is related to the number of people in the scene. There are redundant calculations and it is difficult to cope with real-time scenes; the bottom-up method and the single-stage method do not require additional human detectors. The bottom-up method detects the key points of all people at the same time in the first stage, and groups the key points detected in the first stage in the second stage to achieve the association between the key points and the human instance. Therefore, the computational complexity is greatly reduced, and there is no constraint on the detection box. The global context clues can be fully utilized to locate the body key points, but the detection effect of occluded key points or key points that are difficult to identify is not ideal, because it is difficult to capture useful information at the corresponding pixel position, and it is easy to cause wrong association when associating candidate key points; the single-stage method usually locates the body key points based on the vector at the center point of the human body. However, it is difficult to capture useful information when processing long-distance vectors, and the limited features at the center make it difficult to encode the flexible posture of the entire human body.

[0005] Therefore, how to simultaneously estimate the 3D posture of all people in complex multi-person scenes is an urgent problem to be solved. Summary of the invention

[0006] In order to solve the above-mentioned problems in the prior art, the present invention adopts a multi-person 3D posture estimation method based on explicit limb representation, which is characterized by comprising: obtaining a multi-person scene image, inputting the multi-person scene image into a trained multi-person 3D posture estimation model, obtaining a key point heat map, a limb orientation vector field, a limb relative depth map and a root key point absolute depth map, and post-processing the key point heat map, the limb orientation vector field, the limb relative depth map and the root key point absolute depth map to obtain an estimated multi-person 3D posture; the multi-person 3D posture estimation model comprises: an image posture feature extraction module, a two-dimensional posture representation encoding module and a depth representation encoding module;

[0007] The training process of the multi-person 3D pose estimation model includes:

[0008] S1. Obtain a multi-person scene image training set, which includes multi-person scene images and true labels; and pre-process the multi-person scene images;

[0009] S2, inputting the preprocessed multi-person scene graph into an image posture feature extraction module to extract features and obtain a feature graph;

[0010] S3, inputting the feature map into a two-dimensional posture representation encoding module to extract two-dimensional posture information, and obtaining a predicted key point heat map and limb orientation vector field;

[0011] S4, inputting the feature map and the predicted limb orientation vector field into the depth representation encoding module to extract the human body depth information, and obtaining the predicted limb relative depth map and the root key point absolute depth map;

[0012] S5. Calculate the loss function value according to the real label and the predicted key point heat map, limb orientation vector field, limb relative depth map and root key point absolute depth map, and update the multi-person 3D pose estimation model parameters according to the loss function value. When the loss function value is minimized, a trained multi-person 3D pose estimation model is obtained.

[0013] Preprocessing of multi-person scene images includes:

[0014] S11, randomly rotating and flipping the multi-person scene image to obtain an enhanced multi-person scene image;

[0015] S12, scaling the width and height of the enhanced multi-person scene image in the same proportion to ensure that at least one of the width and the height is aligned with the input size of the image posture feature extraction module;

[0016] S13, filling the center of the scaled image.

[0017] The image posture feature extraction module is the HRNet network, which includes four branches with different resolutions.

[0018] The two-dimensional posture representation encoding module includes: a feature extraction unit, a first channel conversion output module and a second channel conversion output module; the two-dimensional posture representation encoding module processes the feature map including: inputting the feature map into the feature extraction unit to obtain a two-dimensional posture feature map; inputting the two-dimensional posture feature map into the first channel conversion output module and the second channel conversion output module respectively to obtain a predicted key point heat map and a limb orientation vector field.

[0019] The feature maps output by the image posture feature extraction module include four feature maps F with different resolutions: b ,b∈[1,4]; the feature extraction unit includes four branches and a channel conversion convolution, each branch includes a feature extraction module, and the feature extraction module is composed of four Basic Blocks stacked together; the feature extraction unit processes the feature map by: converting the four feature maps F b Input the corresponding feature extraction module of the b-th branch respectively to obtain four feature maps F′ with different resolutions b , in four feature maps F′ with different resolutions b Select the highest resolution feature map And use bilinear interpolation to transform the remaining three feature maps F′ b Upsample to feature map The resolution of three upsampled feature maps F″ is obtained b ;Will And three upsampled feature maps F″ b The combined results are input into the channel conversion convolution to obtain a two-dimensional posture feature map; where b is the index of the feature map and branch, and Basic Block is a basic block, which is a basic building unit of the residual network.

[0020] The depth representation encoding module includes: a feature extraction unit, an absolute depth output module and a channel conversion output module; the depth representation encoding module processes the feature map and the predicted limb orientation vector field, including: inputting the feature map into the feature extraction unit to obtain a depth information feature map, inputting the depth information feature map into the channel conversion output module to obtain a predicted limb relative depth map; inputting the depth information feature map and the predicted limb orientation vector field into the absolute depth output module to obtain a predicted root key point absolute depth map.

[0021] The absolute depth output module includes: a first convolution block, a second convolution block and a Warp layer; the absolute depth output module processes the depth information feature map and the predicted limb orientation vector field, including: inputting the depth information feature map into the first convolution block, inputting the output of the first convolution block and the predicted limb orientation vector field into the Warp layer, and inputting the output of the Warp layer into the second convolution block to obtain the predicted root key point absolute depth map.

[0022] Calculating the loss function value according to the real label and the predicted key point heat map, limb orientation vector field, limb relative depth map, and root key point absolute depth map includes: generating the real label key point heat map, limb orientation vector field, limb relative depth map, and root key point absolute depth map; calculating the loss function value according to the real label key point heat map, limb orientation vector field, limb relative depth map, and root key point absolute depth map and the predicted key point heat map, limb orientation vector field, limb relative depth map, and root key point absolute depth map; wherein the calculation formula of the loss function value is:

[0023]

[0024] in, is the final loss function, represents the L2 loss of the key point heat map, Represents the smooth L1 loss of the limb orientation vector field. The smooth L1 loss is the smooth L1 loss. represents the smooth L1 loss of the relative depth map of the limbs, represents the L1 loss of the absolute depth map of the root keypoint, λ o , rel ,abs A hyperparameter to balance different loss terms.

[0025] Post-processing of keypoint heatmaps, limb orientation vector fields, limb relative depth maps, and root keypoint absolute depth maps includes:

[0026] Step 1: Use the NMS algorithm to detect candidate key points from the key point heat map, and calculate the matching weights between the candidate key points from the limb orientation vector field; wherein the NMS algorithm is a non-maximum suppression algorithm;

[0027] Step 2: Use the depth optimization matching algorithm to match the candidate key points according to the matching weights between the candidate key points and the absolute depth map of the root key point, and calculate the absolute depth value of each matched key point according to the relative depth map of the limbs;

[0028] Step 3, using the limb orientation vector field and the key points matched in step 2 to infer the key points that are not detected in the key point heat map, and obtain the pixel position and absolute depth value of the inferred key points;

[0029] Step 4: Reconstruct the 3D postures of multiple people using the perspective camera model based on the pixel positions and absolute depth values ​​of the matched key points and the inferred key points.

[0030] A system using the above-mentioned multi-person 3D posture estimation method based on explicit limb representation, comprising: an image input module, an image enhancement preprocessing module, an image posture feature extraction module, a two-dimensional posture representation encoding module, a depth representation encoding module and a post-processing module;

[0031] The image input module is used to input images;

[0032] The image enhancement preprocessing module is used to preprocess the image;

[0033] The image posture feature extraction module is used to extract the feature map of the preprocessed image;

[0034] The 2D posture representation encoding module is used to generate key point heat map and limb orientation vector field based on the feature map;

[0035] The depth representation encoding module is used to generate a limb relative depth map and a root keypoint absolute depth map according to the feature map and the limb orientation vector field;

[0036] The post-processing module is used to reconstruct multi-person 3D poses based on key point heat maps, limb orientation vector fields, limb relative depth maps, and root key point absolute depth maps.

[0037] Beneficial effects:

[0038] The present invention ensures that the matching between candidate key points can directly utilize the most intuitive limb information by encoding the display of human limb information as a two-dimensional limb orientation vector field and a limb relative depth map, so as to improve the phenomenon of incorrect matching of candidate key points of different human bodies and improve the quality of matched postures; and redundantly encodes the limb orientation vectors on two key points of the limb. When one of the key points is not detected due to occlusion or interaction between people, the position of the undetected key point can be inferred through the limb orientation vector field at the detected key point position, which greatly improves the robustness of the model to occluded images and also improves the estimation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a multi-person 3D posture estimation method based on explicit limb representation provided by an embodiment of the present invention;

[0040] Figure 2 The overall framework diagram of the multi-person 3D posture estimation model provided by the embodiment of the present invention;

[0041] Figure 3 A flowchart of image preprocessing provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the structure of a two-dimensional posture representation encoding module provided in an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of the structure of a depth representation encoding module provided by an embodiment of the present invention;

[0044] Figure 6 A schematic diagram of a limb orientation vector field provided by an embodiment of the present invention;

[0045] Figure 7 A visualization interface of a multi-person 3D posture estimation system based on explicit limb representation is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] like Figure 1 , Figure 2 As shown, the present invention adopts a multi-person 3D posture estimation method based on explicit limb representation, comprising the following steps:

[0048] Acquire a multi-person scene image, input the multi-person scene image into a trained multi-person 3D pose estimation model, obtain a key point heat map, a limb orientation vector field, a limb relative depth map, and a root key point absolute depth map, and post-process the key point heat map, the limb orientation vector field, the limb relative depth map, and the root key point absolute depth map to obtain an estimated multi-person 3D pose; the multi-person 3D pose estimation model includes: an image pose feature extraction module, a two-dimensional pose representation encoding module, and a depth representation encoding module;

[0049] The training process of the multi-person 3D pose estimation model includes:

[0050] S1. Obtain a multi-person scene image training set, which includes multi-person scene images and true labels; and pre-process the multi-person scene images;

[0051] S2, inputting the preprocessed multi-person scene graph into an image posture feature extraction module to extract features and obtain a feature graph;

[0052] S3, inputting the feature map into a two-dimensional posture representation encoding module to extract two-dimensional posture information, and obtaining a predicted key point heat map and limb orientation vector field;

[0053] S4, inputting the feature map and the predicted limb orientation vector field into the depth representation encoding module to extract the human body depth information, and obtaining the predicted limb relative depth map and the root key point absolute depth map;

[0054] S5. Calculate the loss function value according to the real label and the predicted key point heat map, limb orientation vector field, limb relative depth map and root key point absolute depth map, and update the multi-person 3D pose estimation model parameters according to the loss function value. When the loss function value is minimized, a trained multi-person 3D pose estimation model is obtained.

[0055] The multi-person scene image training set includes: MuCo-3DHP dataset and COCO dataset. MuCo-3DHP contains 200,000 synthetic images with 3D pose labels. In order to enhance the generalization performance, mixed training is performed using part of the COCO dataset containing 2D pose labels, and 50% of the COCO dataset is mixed in each small batch.

[0056] like Figure 3 As shown, the preprocessing of multi-person scene images includes:

[0057] S11, multi-person scene image is x∈R H×W×C , where H, W, and C represent the height, width, and number of channels, respectively; the multi-person scene image is randomly rotated and randomly flipped to obtain an enhanced multi-person scene image;

[0058] S12, scaling the width and height of the enhanced multi-person scene image in the same proportion to ensure that at least one of the width and height is aligned with the input size (scale) of the image posture feature extraction module, 512×832×3;

[0059] scale=min(H / 512,W / 832)

[0060] S13, filling the center of the scaled image to a size of 512×832×3.

[0061] The image posture feature extraction module is HRNet network (high-resolution network). HRNet network adopts HRNet-W32 convolutional network suitable for dense prediction tasks. HRNet-W32 contains four parallel branches with different resolutions (128×208, 64×104, 32×52, 16×26). The output of each branch is a feature map F with different resolutions. b ,b∈[1,4]; the branches with lower resolution are gradually obtained by fusing the features of other branches from the high-resolution branches and downsampling them. Each branch uses stacked Basic Blocks as feature extraction units. Each Basic Block consists of a 3×3 convolution, a normalization layer (2DBatchNorm) and a ReLU activation layer. BasicBlock is a basic block and a basic building unit in the residual network (ResNet).

[0062] The initial weights of the backbone network are ImageNet pre-trained weights.

[0063] like Figure 4 As shown, the two-dimensional posture representation encoding module includes: a feature extraction unit, a first channel conversion output module and a second channel conversion output module; the two-dimensional posture representation encoding module processes the feature map including: inputting the feature map into the feature extraction unit to obtain a two-dimensional posture feature map; inputting the two-dimensional posture feature map into the first channel conversion output module and the second channel conversion output module respectively to obtain the predicted key point heat map of J channels and the limb orientation vector field of 2×(J-1) channels; J is the number of key point types.

[0064] The feature extraction unit includes four parallel branches b and a channel conversion convolution, each branch includes a feature extraction module, and the feature extraction module is stacked by Basic Block; the feature extraction unit processes the feature map by: converting the four feature maps F b Input the corresponding feature extraction module of the b-th branch respectively to obtain four feature maps F′ with different resolutions b , in four feature maps F′ with different resolutions b Select the highest resolution feature map And use bilinear interpolation to transform the remaining three feature maps F′ b Upsample to feature map The resolution (128×208) is used to obtain three upsampled feature maps F″ b ;Will And three upsampled feature maps F″ b They are connected in series in the channel dimension and input into a channel conversion convolution (1×1Conv2d) to convert the number of channels to 256 and obtain a two-dimensional posture feature map.

[0065] The first channel conversion output module and the second channel conversion output module both include multiple layers of convolution.

[0066] In one embodiment, the first channel conversion output module and the second channel conversion output module both include (C out is the number of output channels for different encodings): the first 1×1 convolution (Conv2d), the number of input and output channels is 256 and 256 respectively; the first normalization layer BatcNorm2d, the number of input channels is 256; the first activation layer RELU, the number of input channels is 256; the second 3×3 convolution layer (Conv2d), the number of input and output channels is 256 and C respectively. out ; The second normalization layer BatcNorm2d, the number of input channels is C out .

[0067] like Figure 5 As shown, the depth representation encoding module includes: a feature extraction unit, an absolute depth output module and a channel conversion output module; the depth representation encoding module processes the feature map and the predicted limb orientation vector field, including: inputting the feature map into the feature extraction unit to obtain a depth information feature map, inputting the depth information feature map into the channel conversion output module to obtain a predicted limb relative depth map; inputting the depth information feature map and the predicted limb orientation vector field into the absolute depth output module to obtain a predicted root key point absolute depth map.

[0068] The absolute depth output module structure includes: a first convolution block, a second convolution block and a Warp layer; the absolute depth output module processes the depth information feature map and the predicted limb orientation vector field, including: inputting the depth information feature map into the first convolution block, inputting the output of the first convolution block and the predicted limb orientation vector field into the Warp layer (mapping layer), and inputting the output of the Warp layer into the second convolution block to obtain the predicted root key point absolute depth map.

[0069] The Warp layer extracts the limb vector field corresponding to the root key point (pelvis) and the neck key point in the limb orientation vector field. The pelvic pixel position of the vector field encodes a two-dimensional vector pointing from the pelvis to the neck. The Warp layer uses the two-dimensional vector pointing from the pelvis to the neck to map the output feature map of the first convolutional block: the depth information feature at the neck pixel position is mapped to the pelvic pixel position, and the absolute depth feature map F is obtained, which encodes the depth feature at the neck key point at the root key point pelvic pixel position. w , the absolute depth feature map F after mapping transformation w The output feature map F of the first convolutional block in Combine them to get the predicted root key point absolute depth map F out , the output channels are 512.

[0070] Specifically, the Warp layer can be expressed as:

[0071] F w (x,y)=Warp(x+O x (x,y),y+O y (x,y)

[0072] F out =Concat(F w ,F in )

[0073] Among them, O x , O y are the two-component channels of the 2D vector of the limb pointing from the pelvis to the neck, F in is the output of the first convolutional block.

[0074] In one embodiment, the first convolution block is sequentially connected with: 1×1Conv2d (the number of input and output channels is 256 and 256 respectively), BatcNorm2d (the number of input channels is 256), and RELU (the number of input channels is 256). The second convolution block is sequentially connected with: 1×1Conv2d (the number of input and output channels is 512 and 256 respectively), BatcNorm2d (the number of input channels is 256), RELU (the number of input channels is 256), 3×3Conv2d (the number of input and output channels is 256 respectively), and BatcNorm2d (the number of input channels is 1).

[0075] Generating the keypoint heatmap of the real label, the limb orientation vector field, the limb relative depth map and the root keypoint absolute depth map includes:

[0076] The labels in the pose estimation dataset are a series of key point coordinate arrays, which mainly include the two-dimensional pixel coordinates (x, y) of each key point, the real coordinates (X, Y, Z) in the three-dimensional camera coordinate system, and the camera intrinsic parameter matrix K. When loading the dataset, the real labels matching the four intermediate representations are further generated according to these coordinate labels:

[0077] Generating a key point heat map includes: generating a heat map true label H* of J channels of the same size as the predicted key point heat map H according to the two-dimensional pixel coordinates (x, y) of the key point. Specifically, the confidence in the form of a Gaussian kernel is encoded in the area around the key point, and all confidences outside the area are set to 0.

[0078] Figure 6 For the limb orientation vector field between the two key points of the neck and the top of the head, generating a limb orientation vector field and a limb relative depth map includes: generating corresponding J-1 3D vector fields for J-1 limbs of the human body, each vector field corresponds to the two-dimensional vector and relative depth between the two key points on the limb, the two-dimensional vector is obtained by subtracting the pixel coordinates x and y of the two key points, and the relative depth is obtained by subtracting the Z coordinates of the two key points. The units of the two-dimensional vector and the relative depth are inconsistent and the semantic differences involved in the prediction are large. Therefore, supervision is performed for two intermediate feature maps respectively. Similarly, the two-dimensional vector and relative depth are only encoded in the key point area, and the other pixel positions are 0, so as to obtain a two-dimensional limb orientation vector field and a relative depth map for each limb.

[0079] Generate root keypoint absolute depth map: Since only the absolute depth at the pelvic pixel position is supervised, there is no need to generate additional feature map labels for large-scale regional supervision. Only the original label Z is used for supervision on the pelvic pixel (x, y).

[0080] Since the image is randomly rotated and scaled, the two-dimensional pixel position of the key point has changed, but the absolute depth remains unchanged. Therefore, the key point heat map, limb orientation vector field, and limb relative depth map also need to be transformed in pixel position. The root key point absolute depth is a single pixel point supervision, and the actual pixel position (x, y) of the pelvis required for supervision also needs to be modified.

[0081] The loss function of the model for:

[0082]

[0083] in, represents the L2 loss of the key point heat map, J is the number of key points, H k and They are the key point heat map of the predicted key point k and the key point heat map of the actual key point k; represents the smooth L1 loss function for the limb orientation vector field, S is the set of pixels i around the two key points on the limb, o i and are the predicted and true limb orientation vectors of pixel i, respectively; Represents the smooth L1 loss function for the relative depth map of the limbs, Δz i and is the relative depth of the predicted and true pixel i; represents the L1 loss of the absolute depth map of the root keypoint, z n and is the predicted and actual absolute depth of the human body at n root key point pixels in the root key point absolute depth map, where N represents the number of people in the multi-person scene image; λ0, λ rel , abs A hyperparameter to balance different loss terms.

[0084] L2 loss is also called mean squared error loss (MSE), L1 loss is also called mean absolute error loss (MAE), and Smooth L1 loss is called Huber loss or smooth L1 loss. It is a loss function that combines the advantages of L1 loss and L2 loss.

[0085] Post-processing of keypoint heatmaps, limb orientation vector fields, limb relative depth maps, and root keypoint absolute depth maps includes:

[0086] Step 1: Use the non-maximum suppression (NMS) algorithm to detect the pixel positions (x, y) of the candidate key points from the key point heat map, and calculate the matching weights between the candidate key points from the limb orientation vector field;

[0087] The matching weight between two candidate key points i and j on the limb is calculated as follows:

[0088] p′ j =p i +o i

[0089] p′ i =p j +o j

[0090]

[0091] where p i 、p j are the 2D pixel positions of the two candidate key points, o i andj In p i 、p j Limb orientation vector read out at pixel position, p′ j The position of j on the limb is inferred based on the position of the candidate key point i and the limb orientation vector at that position, p′ i Similarly, To calculate the pixel distance between the inferred position and the detected position, it is used to measure the matching degree of two candidate points.

[0092] Step 2: Use the depth optimization matching algorithm to match the candidate key points according to the matching weights between the candidate key points and the absolute depth map of the root key point, and calculate the absolute depth value of each matched key point according to the relative depth map of the limbs;

[0093] The candidate key points matched using the deep optimization matching algorithm include:

[0094] Construct a weighted bipartite graph consisting of the matching weights between candidate key points, determine the number of people in the scene based on the number of root key points, and prioritize matching human bodies with smaller absolute depth values ​​according to the absolute depth value order of the root key points; specifically, match starting from the root key point with smaller absolute depth value according to the weighted bipartite graph, and select the human body with matching weights with the root key point. The optimal key points are then matched from the pelvis along the human skeleton path to the key points at the ends of the limbs.

[0095] While matching, the absolute depth Z of each key point is also gradually inferred: first, the absolute depth value of the root key point is directly read from the pixel position of the candidate root key point in the absolute depth map. During the matching process, the absolute depth value of the successfully matched sub-key point can be obtained by the formula Calculate, where are the absolute depth values ​​of the successfully matched parent and child key points on the lth limb, respectively. In the previous round, it was calculated that Δz l Represents the relative depth difference between the two key points on the limb read from the limb relative depth map, and the absolute depth value Z of each key point that completes the matching is recursively obtained by accumulating the relative depth values ​​along the skeleton path from the known absolute depth value of the root key point.

[0096] Step 3: Use the limb orientation vector field and the key points matched in step 2 to infer the key points that are not detected in the key point heat map, and obtain the pixel position and absolute depth value of the inferred key points.

[0097] Inferring undetected key points using the limb orientation vector field: In the previous depth-first matching process, due to complex reasons such as occlusion and human body interaction, some key points may not be detected by the key point heat map. Therefore, there will be a problem in the bipartite graph where there are more parent key points than child key points, that is, some child key points are not detected, resulting in the parent key points with low depth priority having no matching child key points. At this time, since the limb orientation vector field is redundantly encoded at the pixel position of the parent key point, the vector pointing from the parent key point to the child key point can be read from the vector field. This vector can be used to realize the inference of the undetected key points and obtain the pixel position and absolute depth value of the inferred key points.

[0098] Step 4: Reconstruct the 3D poses of multiple people using the perspective camera model based on the pixel positions and absolute depth values ​​of the matched key points and the inferred key points:

[0099] [X,Y,Z] T =ZK -1 [x,y,1] T

[0100] Among them, K is the camera internal parameter matrix, Z is the absolute depth value of the key point, (x, y) is the pixel position of the key point, [X, Y, Z] T are the multi-person 3D coordinates required in the camera coordinate system, and T represents transpose.

[0101] The present invention also provides a multi-person 3D posture estimation system based on explicit limb representation, comprising: an image input module, an image enhancement preprocessing module, an image posture feature extraction module, a two-dimensional posture representation encoding module, a depth representation encoding module and a post-processing module;

[0102] The image input module is used to input images;

[0103] The image enhancement preprocessing module is used to preprocess the image;

[0104] The image posture feature extraction module is used to extract the feature map of the preprocessed image;

[0105] The 2D posture representation encoding module is used to generate key point heat map and limb orientation vector field using the feature map of the preprocessed image;

[0106] The depth representation encoding module is used to generate a limb relative depth map and a root keypoint absolute depth map using the feature map of the preprocessed image;

[0107] The post-processing module includes a candidate keypoint detection module and a key assembly and reasoning module;

[0108] The key point detection module is used to extract candidate key points from the key point heat map and calculate the matching weights between the candidate key points through the limb orientation vector field;

[0109] The key assembly and reasoning module is used to assemble the detected candidate key points and reason about the undetected key points to obtain the final key point positions and reconstruct the multi-person 3D postures based on the final key point positions.

[0110] like Figure 7 As shown, the present invention also provides a visualization interface of a multi-person 3D posture estimation system based on explicit limb representation, including: a file selection interface, an image selection interface, a start prediction button, an image preview interface and a prediction result interface.

[0111] The file selection interface is used to select the local multi-person 3D pose estimation model and display the storage path of the multi-person 3D pose estimation model; the image selection button is used to select the input image of the multi-person 3D pose estimation model and display the storage path of the input image; the start prediction button is used to start the estimation of multi-person 3D pose; the image preview interface is used to display the input image; the prediction result interface is used to display the reconstructed multi-person 3D pose.

[0112] The multi-person 3D pose estimation based on explicit limb representation of the present invention can be applied to multiple scenarios:

[0113] 1. Physical training

[0114] Multi-person 3D posture estimation technology can track and analyze the postures and movements of athletes, helping coaches assess the technical level of athletes and provide targeted training suggestions. By accurately capturing and analyzing the movements of athletes, coaches can promptly detect and correct the athletes' incorrect movements and improve training results.

[0115] 2. Virtual Reality System

[0116] In a virtual reality environment, 3D multi-person posture estimation can capture and reconstruct the user's body posture in real time, thus providing a more natural and smooth interactive experience. Users can directly interact with the virtual environment through body movements to enhance the sense of immersion; the technology can also dynamically adjust the lighting, shadows and other effects in the virtual scene based on the user's posture information, making the scene rendering more realistic.

[0117] 3. Game Development

[0118] In game development, 3D multi-person pose estimation technology can allow players to control game characters through body movements, improving the interactivity and fun of the game; this technology can also realize real-time interaction between game characters and game scenes, enhancing the immersion and realism of the game.

[0119] 4. Healthcare

[0120] In the medical field, this technology can be used for rehabilitation training to help patients recover their physical functions. By capturing the patient's movements in real time, doctors can evaluate the patient's rehabilitation progress and adjust the rehabilitation plan; combined with telemedicine technology, doctors can also remotely guide patients in rehabilitation training, improving the accessibility and efficiency of medical services.

[0121] 5. Robotics and smart manufacturing

[0122] In the field of robotics, multi-person 3D posture estimation can assist robots in understanding and responding to human commands and actions, achieving more intelligent interactions and operations; in the field of smart manufacturing, this technology can be used to monitor the postures and movements of workers on the production line to ensure the standardization and safety of the production process.

[0123] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-person 3D pose estimation method based on explicit limb representation, characterized in that: include: Acquire a multi-person scene image, input the multi-person scene image into a trained multi-person 3D pose estimation model, obtain a key point heat map, a limb orientation vector field, a limb relative depth map, and a root key point absolute depth map, post-process the key point heat map, the limb orientation vector field, the limb relative depth map, and the root key point absolute depth map to obtain an estimated multi-person 3D pose; The multi-person 3D posture estimation model includes: an image posture feature extraction module, a two-dimensional posture representation encoding module, and a depth representation encoding module; The training process of the multi-person 3D pose estimation model includes: S1. Obtain a multi-person scene image training set, which includes multi-person scene images and true labels; and pre-process the multi-person scene images; S2, inputting the preprocessed multi-person scene graph into an image posture feature extraction module to extract features and obtain a feature graph; S3, inputting the feature map into a two-dimensional posture representation encoding module to extract two-dimensional posture information, and obtaining a predicted key point heat map and limb orientation vector field; S4, inputting the feature map and the predicted limb orientation vector field into the depth representation encoding module to extract the human body depth information, and obtaining the predicted limb relative depth map and the root key point absolute depth map; S5. Calculate the loss function value according to the real label and the predicted key point heat map, limb orientation vector field, limb relative depth map and root key point absolute depth map, and update the multi-person 3D pose estimation model parameters according to the loss function value. When the loss function value is minimized, a trained multi-person 3D pose estimation model is obtained.

2. The method for multi-person 3D pose estimation based on explicit limb representation according to claim 1, characterized in that: Preprocessing of multi-person scene images includes: S11, randomly rotating and flipping the multi-person scene image to obtain an enhanced multi-person scene image; S12, scaling the width and height of the enhanced multi-person scene image in the same proportion to ensure that at least one of the width and the height is aligned with the input size of the image posture feature extraction module; S13, filling the center of the scaled image.

3. The method for multi-person 3D pose estimation based on explicit limb representation according to claim 1, characterized in that: The image posture feature extraction module is the HRNet network, which includes four branches with different resolutions; among them, the HRNet network is a high-resolution network.

4. The method for multi-person 3D posture estimation based on explicit limb representation according to claim 1, characterized in that: The two-dimensional posture representation encoding module includes: a feature extraction unit, a first channel conversion output module and a second channel conversion output module; the two-dimensional posture representation encoding module processes the feature map including: inputting the feature map into the feature extraction unit to obtain a two-dimensional posture feature map; inputting the two-dimensional posture feature map into the first channel conversion output module and the second channel conversion output module respectively to obtain a predicted key point heat map and a limb orientation vector field.

5. The method for multi-person 3D posture estimation based on explicit limb representation according to claim 4, characterized in that: The feature maps output by the image posture feature extraction module include four feature maps F with different resolutions: b ,b∈[1,4]; the feature extraction unit includes four branches and a channel conversion convolution, each branch includes a feature extraction module, and the feature extraction module is composed of four Basic Blocks stacked together; the feature The extraction unit processes the feature map by: converting the four feature maps F b Input the corresponding feature extraction module of the b-th branch respectively to obtain four feature maps F′ with different resolutions b , in four feature maps F′ with different resolutions b Select the highest resolution feature map And use bilinear interpolation to transform the remaining three feature maps F′ b Upsample to feature map The resolution of three upsampled feature maps F″ is obtained b ;Will And three upsampled feature maps F″ b The combined results are input into the channel conversion convolution to obtain a two-dimensional posture feature map; where b is the index of the feature map and branch, and Basic Block is a basic block, which is a basic building unit of the residual network.

6. The method for multi-person 3D posture estimation based on explicit limb representation according to claim 1, characterized in that: The depth representation encoding module includes: a feature extraction unit, an absolute depth output module and a channel conversion output module; the depth representation encoding module processes the feature map and the predicted limb orientation vector field, including: inputting the feature map into the feature extraction unit to obtain a depth information feature map, inputting the depth information feature map into the channel conversion output module to obtain a predicted limb relative depth map; inputting the depth information feature map and the predicted limb orientation vector field into the absolute depth output module to obtain a predicted root key point absolute depth map.

7. The method for multi-person 3D posture estimation based on explicit limb representation according to claim 1, characterized in that: The absolute depth output module includes: a first convolution block, a second convolution block and a Warp layer; the absolute depth output module processes the depth information feature map and the predicted limb orientation vector field including: inputting the depth information feature map into the first convolution block, inputting the output of the first convolution block and the predicted limb orientation vector field into the Warp layer, and inputting the output of the Warp layer into the second convolution block to obtain a predicted root key point absolute depth map; wherein the Warp layer is a mapping layer.

8. The method for multi-person 3D posture estimation based on explicit limb representation according to claim 1, characterized in that: Calculating the loss function value according to the real label and the predicted key point heat map, limb orientation vector field, limb relative depth map, and root key point absolute depth map includes: generating the real label key point heat map, limb orientation vector field, limb relative depth map, and root key point absolute depth map; calculating the loss function value according to the real label key point heat map, limb orientation vector field, limb relative depth map, and root key point absolute depth map and the predicted key point heat map, limb orientation vector field, limb relative depth map, and root key point absolute depth map; wherein the calculation formula of the loss function value is: in, is the final loss function, represents the L2 loss of the key point heat map, Represents the smooth L1 loss of the limb orientation vector field. The smooth L1 loss is the smooth L1 loss. represents the smoothL1 loss for the relative depth map of the limbs, represents the L1 loss of the absolute depth map of the root keypoint, λ o , rel , abs A hyperparameter to balance different loss terms.

9. The method for multi-person 3D posture estimation based on explicit limb representation according to claim 1, characterized in that: Post-processing of keypoint heatmaps, limb orientation vector fields, limb relative depth maps, and root keypoint absolute depth maps includes: Step 1: Use the NMS algorithm to detect candidate key points from the key point heat map, and calculate the matching weights between the candidate key points from the limb orientation vector field; wherein the NMS algorithm is a non-maximum suppression algorithm; Step 2: Use the depth optimization matching algorithm to match the candidate key points according to the matching weights between the candidate key points and the absolute depth map of the root key point, and calculate the absolute depth value of each matched key point according to the relative depth map of the limbs; Step 3, using the limb orientation vector field and the key points matched in step 2 to infer the key points that are not detected in the key point heat map, and obtain the pixel position and absolute depth value of the inferred key points; Step 4: Reconstruct the 3D postures of multiple people using the perspective camera model based on the pixel positions and absolute depth values ​​of the matched key points and the inferred key points.

10. A system using a multi-person 3D posture estimation method based on explicit limb representation as claimed in any one of claims 1 to 9, characterized in that: include: Image input module, image enhancement preprocessing module, image posture feature extraction module, two-dimensional posture representation encoding module, depth representation encoding module and post-processing module; The image input module is used to input images; The image enhancement preprocessing module is used to preprocess the image; The image posture feature extraction module is used to extract the feature map of the preprocessed image; The 2D posture representation encoding module is used to generate key point heat map and limb orientation vector field based on the feature map; The depth representation encoding module is used to generate a limb relative depth map and a root keypoint absolute depth map according to the feature map and the limb orientation vector field; The post-processing module is used to reconstruct multi-person 3D poses based on key point heat maps, limb orientation vector fields, limb relative depth maps, and root key point absolute depth maps.

Images