A pedestrian detection and pose estimation method based on 2D-3D coordinate matching strategy

Abstract

This invention relates to a pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy. It takes an image containing multiple people as input and inputs it into a pre-constructed and trained detection network, outputting pedestrian detection and 2D human pose estimation results. The detection network includes: a backbone network for outputting multiple first feature maps at different scales; an information recovery branch for performing multiple upsampling operations to generate multiple second feature maps at different scales; a detector for performing detection tasks and outputting single-person detection boxes and single-person keypoint coordinates; a 2D prediction branch for outputting 2D human representation parameters based on all second feature maps and single-person detection boxes; and a correction and optimization module for optimizing the 2D human representation parameters output by the 2D prediction branch, outputting the final 2D human pose coordinates. Compared with existing technologies, this invention can correct keypoint coordinates, thereby alleviating the mismatch problem in existing 2D pose estimation technologies and improving detection efficiency.

Description

Technical Field

[0001] This invention relates to the field of pedestrian detection and pose estimation technology, and in particular to a pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy. Background Technology

[0002] With the development of autonomous driving technology, its safety has received increasing attention. Compared with non-autonomous vehicles, autonomous vehicles need to accurately identify obstacles in the environment to avoid collisions. In traffic scenarios, pedestrians are the most variable in their movement trajectory. Therefore, models not only need to be able to detect pedestrian positions, but also need to have a certain understanding of their trajectories.

[0003] Existing pedestrian detection and pose estimation methods often directly employ a multi-object tracking framework to simultaneously track pedestrians and other obstacles within the field of view, without considering prior information about the pedestrians themselves. Many previous works focused on model-based 3D pose estimation methods, using parameterized SMPL human model parameters as the output target to leverage prior statistical information about body structure. With the advancement of deep learning technology, more and more research is focusing on learning-based models, using deep learning networks to estimate pose and shape parameters. However, learning-based multi-person pose detection methods suffer from the 2D-3D keypoint coordinate transformation problem, which has not yet been fully resolved, often leading to mismatches in traditional 2D pose estimation. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy. This invention can correct the coordinates of key points, thereby alleviating the problem of mismatch in the existing 2D pose estimation technology and improving detection efficiency.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] This invention provides a pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy. The method takes an image containing multiple people as the input image and inputs it into a pre-constructed and trained detection network, and outputs pedestrian detection and 2D human pose estimation results.

[0007] The detection network includes:

[0008] Backbone network: Used to receive input images and output multiple first feature maps at different scales;

[0009] Information recovery branch: used to upsample the first feature map at the smallest scale multiple times, thereby generating multiple second feature maps at different scales;

[0010] Detector: Used to receive all first feature maps and perform detection tasks, outputting single-person detection boxes and single-person key point coordinates;

[0011] 2D prediction branch: Used to receive and recover all the second feature maps output by the branch and the single person detection box output by the detector based on the information, predict the single person's 2D pose, and output 2D human representation parameters.

[0012] Correction and optimization module: It is used to receive and describe the 3D human body representation parameters based on the single person key point coordinates output by the detector, thereby obtaining the relative relationship of each coordinate point of the human body in 3D. Based on the relative relationship of each coordinate point of the human body in 3D, it optimizes the 2D human body representation parameters output by the 2D prediction branch and outputs the final 2D human body pose coordinate points.

[0013] Preferably, the 2D prediction branch selects a second feature map from all the second feature maps output by the information recovery branch that matches the scale of the single-person detection box output by the detector, and crops the second feature map to obtain a prediction heatmap for single-person 2D pose prediction.

[0014] The process of the 2D prediction branch predicting the 2D pose of a single person is as follows:

[0015] 1a) For the predicted heatmap, sum the values ​​in both the UV and UV directions to obtain the coordinate voting weight z of each key point in the UV direction;

[0016] 2a) Calculate the normalized weight distribution z′=Softmax(z) based on the coordinate voting weight z;

[0017] 3a) Define a fixed enumeration sequence E, calculate the product of the normalized weight distribution z′ and the fixed enumeration sequence E, and obtain the voting value corresponding to each key point;

[0018] 4a) Calculate the mean and variance based on the voting values ​​to obtain the predicted UV coordinates and their uncertainties for each key point.

[0019] Preferably, the 2D pose loss function in the 2D prediction branch prediction process is:

[0020]

[0021] In the formula, u i and v i c represents the true values ​​of the x and y coordinates of the i-th keypoint, respectively. i Let w represent the uncertainty of the predicted value of the i-th keypoint, and w represent the weight of the regularization term. and These represent the predicted horizontal and vertical coordinates of the i-th key point, respectively.

[0022] Preferably, the 2D prediction branch is constructed based on a K-Block structure and outputs the 2D coordinate prediction values ​​of human SMPL key points and the uncertainty of the 2D coordinate prediction values ​​of the key points.

[0023] Preferably, the correction and optimization module describes the 3D human body representation parameters based on the SMPL model and single-person keypoint coordinates, outputs the 3D coordinate prediction values ​​of the SMPL keypoints, and optimizes the 2D coordinate prediction values ​​of the human body SMPL keypoints output by the 2D prediction branch based on the 3D coordinate prediction values ​​of the SMPL keypoints. The optimization equation based on the 2D-3D coordinate matching strategy is as follows:

[0024]

[0025] In the formula (R) * ,t * To find the optimal rotation matrix and translation vector for 2D-3D coordinate matching that minimizes the loss function, p i and q i For the corresponding coordinates in SMPL vector form, considering the uncertainty c i and weight w i Negative correlation, with uncertainty c i and weight w i Construct diagonal matrices W and C for their diagonal elements, with W = C. -1 Substituting the values, we can simplify to:

[0026]

[0027] In the formula, P satisfies SVD decomposition T C -1 Q=U∑V T Two sets of corresponding coordinate points form a point set P = {p1, p2, ..., p...} k} and Q = {q1,q2,…,q k Let (R, t) be the rotation matrix and translation vector of a pair of Euclidean transformations, respectively. This transformation minimizes the errors of P and Q. U and V are two orthogonal matrices that satisfy P T C -1 Q=U∑V T .

[0028] Preferably, the optimization process based on the 2D-3D coordinate matching strategy is as follows:

[0029] Input the 2D coordinate prediction values ​​(KPS) of human SMPL keypoints 2D Uncertainty C of the predicted 2D coordinates of key points 2D And the 3D coordinate prediction value KPS of SMPL key points 3D ;

[0030] 1b) Calculate KPS 2D The mean and variance var are used for subsequent recovery of the coordinates of 2D points, assuming KPS 2D The z-axis coordinates are all 0;

[0031] 2b) Regarding KPS 2D and KPS 3D Perform normalization to align the model;

[0032] 3b) Calculate KPS 2D C 2D and KPS 3D The outer product S = (KPS) 2D (C) 2D ) -1 (KPS 3D );

[0033] 4b) Calculate the SVD decomposition of S, S = UΣV', to obtain the rotation matrix R = VU';

[0034] 5b) Set the determinant of R to 1, i.e., det(R) = 1, to ensure that R is a rotation matrix and not a reflection matrix;

[0035] 6b) Calculate KPS based on mean and var 3D The coordinates of the corresponding 2D point;

[0036] 7b) Assign the transformed z-axis coordinates to KPS 2D ;

[0037] 8b) Repeat steps 3b) to 7b) to optimize the rotation matrix R and obtain the final 2D human pose coordinates.

[0038] Preferably, during the optimization process, the keypoint regression loss function Loss KPS3D for:

[0039]

[0040] In the formula, u i and v i and represent the true values ​​of the x and y coordinates of the i-th key point, respectively. and These represent the predicted horizontal and vertical coordinates of the i-th key point, respectively.

[0041] Preferably, the detector is a Transformer-based DETR detector.

[0042] Preferably, the loss function of the detector during the detection task includes the L1 loss function calculated directly between the single person detection box and the ground truth box, the IoU optimization loss function, the pedestrian prediction confidence loss function, and the GIoU value loss function corresponding to the predicted detection box.

[0043] Preferably, the backbone network is a simplified multilayer neural network based on ResNet, and the information recovery branch is a feature pyramid network.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) This invention feeds the preprocessed original image into the backbone network; mines the image features extracted by the backbone network, predicts the bounding box (BBox) where the pedestrian is located, outputs the detection confidence and the coordinates of the detection box, and outputs a set of predicted SMPL parameter values ​​to correct the joint coordinates, thus alleviating the mismatch in traditional 2D pose estimation; extracts features from different detection boxes; directly feeds the extracted low-resolution features into a specific structure after transformation for pose prediction, reducing the computational load and improving detection efficiency; and uses the parameters output by the model to further optimize human joint detection under special conditions such as occlusion or low light.

[0046] (2) This invention uses a simplified multi-layer neural network to avoid the difficulties in gradient propagation, and adds an information recovery branch to adapt to inputs of different sizes, while providing corresponding parameters for use by the single-person 2D pose prediction module.

[0047] (3) The human pose estimation mode based on K-Block designed in this invention greatly reduces the waste of gradients and enables end-to-end training of the model. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the detection network structure for a pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy provided in this embodiment.

[0049] Figure 2 for Figure 1 The illustrated embodiment is a schematic diagram of the 2D prediction branch predicting the 2D pose of a single person. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0051] refer to Figure 1As shown, this embodiment provides a pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy, including the following steps:

[0052] Step 1: Obtain images to be detected that contain multiple people, and divide the images to be detected into a test set and a training set;

[0053] Step 2: Construct a detection network. Input the training set into the detection network and train it so that the trained detection network can perform pedestrian detection and pose estimation.

[0054] The detection network includes:

[0055] Backbone network: Used to receive input images and output multiple first feature maps at different scales.

[0056] Information recovery branch: used to upsample the first feature map at the smallest scale multiple times, thereby generating multiple second feature maps at different scales.

[0057] Detector: Used to receive all first feature maps and perform detection tasks, outputting single-person detection bounding boxes, single-person keypoint coordinates, and single-person detection confidence scores.

[0058] 2D prediction branch: Used to receive and recover all the second feature maps output by the branch and the single person detection box output by the detector based on the information, predict the single person's 2D pose, and output 2D human representation parameters.

[0059] Correction and optimization module: It is used to receive and describe the 3D human body representation parameters based on the single person key point coordinates output by the detector, thereby obtaining the relative relationship of each coordinate point of the human body in 3D. Based on the relative relationship of each coordinate point of the human body in 3D, it optimizes the 2D human body representation parameters output by the 2D prediction branch and outputs the final 2D human body pose coordinate points.

[0060] As an alternative implementation, the backbone network is a simplified multilayer neural network based on ResNet.

[0061] As an optional implementation, the information recovery branch is a Feature Pyramid Network. The information recovery branch performs multiple upsamplings on the first feature map at the smallest scale. For each upsampling process, the second feature map at the current scale and the first feature map one level larger than the current scale are used as inputs for the current upsampling, and the output is a second feature map one level larger than the current scale.

[0062] As an optional implementation, the detector is a Transformer-based DETR detector.

[0063] As an optional implementation, the loss function of the detector during the detection task includes:

[0064] (1) L1 loss function: It is calculated directly between the single person detection box and the ground truth box. The formula describing the L1 loss is as follows:

[0065]

[0066] In the formula, x i y i w i and h i These are the true values ​​of the horizontal and vertical positions and the width and height of the j-th detection box, respectively. and Let $\mathbf$ be the predicted values ​​of the horizontal and vertical positions and the four parameters of the $j$-th detection box, and $\mathbf$ be the predicted values ​​of the four parameters. BBox This is the L1 loss function.

[0067] (2) IoU optimization loss function:

[0068] Introducing the main evaluation metric IoU:

[0069]

[0070] In the formula, A and B are the areas occupied by the single-person detection box and the ground truth box, respectively.

[0071]

[0072] In the formula, Loss GIoU The loss function is optimized for IoU, where GIoU(A,B) is a function that measures the distance between A and B, and C is the area occupied by the smallest bounding rectangle of A and B.

[0073] (3) Pedestrian prediction confidence loss function:

[0074]

[0075] In the formula, α = 0.25 and γ = 2 are the corresponding hyperparameters. 'p' represents the true label, and 'p' represents the predicted result.

[0076] (4) Loss function for predicting the GIoU value of the detection box:

[0077]

[0078] In the formula, GIoU is the GIoU value of the true bounding box. This represents the GIoU value of the predicted single-person detection box.

[0079] As an optional implementation, the 2D prediction branch selects a second feature map from all the second feature maps output by the information recovery branch that matches the scale of the single-person detection box output by the detector, and crops the second feature map to obtain a prediction heatmap for single-person 2D pose prediction.

[0080] refer to Figure 2 As shown, the process of the 2D prediction branch predicting the 2D pose of a single person is as follows:

[0081] 1a) For a single predicted heatmap T with size W×H i The coordinate voting weights z of each key point in the UV direction are obtained by summing the values ​​in both directions, with a size of 1H and W×1.

[0082] 2a) Calculate the normalized weight distribution z′=Softmax(z) based on the coordinate voting weight z;

[0083] 3a) For a given vector, let its length be N, then there is a fixed enumeration sequence vector E = (1,2,…N), whose physical meaning is a vector composed of the index of each row or column pixel. Calculate the product of the normalized weight distribution z′ and the fixed enumeration sequence E to obtain the voting value corresponding to each key point.

[0084] 4a) Calculate the mean and variance based on the voting values ​​to obtain the predicted UV coordinates u of each key point. i v i and its uncertainty c i,1 c i,2 .

[0085] As an optional implementation, the 2D pose loss function in the 2D prediction branch prediction process is:

[0086]

[0087] In the formula, u i and v i Let represent the true values ​​of the x and y coordinates of the i-th keypoint, respectively; c represents the uncertainty of the coordinate prediction; and w represents the weight of the regularization term. and These represent the predicted horizontal and vertical coordinates of the i-th key point, respectively.

[0088] As an optional implementation, the 2D prediction branch is built based on a K-Block structure, which simultaneously calculates the coordinate values ​​of human joints from all heatmap pixels and outputs the 2D coordinate prediction values ​​KPS of human SMPL keypoints in a non-argmax output format. 2D The uncertainty C of the 2D coordinate prediction of key points 2D .

[0089] As an optional implementation, the correction and optimization module receives the single-person key point coordinates output by the detector. Based on the SMPL model and the single-person key point coordinates, the correction and optimization module describes the 3D human body representation parameters, outputs the 3D coordinate prediction values ​​of the SMPL key points, optimizes the 2D coordinate prediction values ​​of the human body SMPL key points output by the 2D prediction branch based on the 3D coordinate prediction values ​​of the SMPL key points, and outputs the final 2D human body pose coordinate points.

[0090] The optimization equation based on the 2D-3D coordinate matching strategy is:

[0091]

[0092] In the formula (R) * ,t * The optimal rotation matrix and translation vector for 2D-3D coordinate matching that minimizes the loss function (Loss). i and q i These are the corresponding coordinate points in SMPL vector form. Considering the uncertainty c... i and weight w i Negative correlation, with uncertainty c i and weight w i Construct diagonal matrices W and C for their diagonal elements, with W = C. -1 Substituting the values, we can simplify to:

[0093]

[0094] In the formula, P satisfies SVD decomposition T C -1 Q=U∑V T Two sets of corresponding coordinate points form a point set P = {p1, p2, ..., p...} k} and Q = {q1,q2,…,q k Let (R, t) be the rotation matrix and translation vector of a pair of Euclidean transformations, respectively. This transformation minimizes the errors of P and Q. U and V are two orthogonal matrices that satisfy P T C -1 Q=U∑V T .

[0095] As an optional implementation method, the optimization process based on the 2D-3D coordinate matching strategy is as follows:

[0096] The 2D prediction branch outputs the predicted 2D coordinates (KPS) of human SMPL keypoints. 2D The uncertainty C of the 2D coordinate prediction of key points 2D 3D coordinate prediction values ​​(KPS) of SMPL key points3D As input, the output is a weighted 2D coordinate SMPL form, i.e., Weighted KPS. 2D .

[0097] 1b) Calculate KPS 2D The mean and variance var are used for subsequent recovery of the coordinates of 2D points, assuming KPS 2D The z-axis coordinates are all 0;

[0098] 2b) Regarding KPS 2D and KPS 3D Normalize (remove mean and variance) to align the model;

[0099] 3b) Calculate KPS 2D C 2D and KPS 3D The outer product S = (KPS) 2D (C) 2D ) -1 (KPS 3D );

[0100] 4b) Calculate the SVD decomposition of S, S = UΣV', to obtain the rotation matrix R = VU';

[0101] 5b) Set the determinant of R to 1, i.e., det(R) = 1, to ensure that R is a rotation matrix and not a reflection matrix;

[0102] 6b) Calculate KPS based on mean and var 3D The coordinates of the corresponding 2D point;

[0103] 7b) Assign the transformed z-axis coordinates to KPS 2D ;

[0104] 8b) Repeat steps 3b to 7b to optimize the rotation matrix R and obtain weighted 2D coordinates based on the predicted 3D point values.

[0105] As an optional implementation method, the keypoint regression loss function during the optimization process is:

[0106]

[0107] Step 3: Input the test set into the trained detection network and output the pedestrian detection and 2D human pose estimation results.

[0108] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy, characterized in that, The image to be detected, which contains multiple people, is used as the input image and fed into a pre-built and trained detection network. The output is the pedestrian detection and 2D human pose estimation results. The detection network includes: Backbone network: Used to receive input images and output multiple first feature maps at different scales; Information recovery branch: used to upsample the first feature map at the smallest scale multiple times, thereby generating multiple second feature maps at different scales; Detector: Used to receive all first feature maps and perform detection tasks, outputting single-person detection boxes and single-person key point coordinates; 2D prediction branch: Used to receive and recover all the second feature maps output by the branch and the single person detection box output by the detector based on the information, predict the single person's 2D pose, and output 2D human representation parameters. Correction and optimization module: It is used to receive and describe the 3D human body representation parameters based on the single person key point coordinates output by the detector, thereby obtaining the relative relationship of each coordinate point of the human body in 3D, optimizing the 2D human body representation parameters output by the 2D prediction branch based on the relative relationship of each coordinate point of the human body in 3D, and outputting the final 2D human body pose coordinate points. The 2D prediction branch selects a second feature map from all the second feature maps output by the information recovery branch that matches the scale of the single-person detection box output by the detector, and crops the second feature map to obtain a prediction heatmap for single-person 2D pose prediction. The process of the 2D prediction branch predicting the 2D pose of a single person is as follows: 1a) For the predicted heatmap, sum the values ​​in both the UV and UV directions to obtain the coordinate voting weights of each key point in the UV direction. ; 2a) Based on coordinate voting weights Calculate the normalized weight distribution ; 3a) Define a fixed enumeration sequence Calculate the normalized weight distribution With fixed enumeration sequence The product of these values ​​yields the voting value for each key point; 4a) Calculate the mean and variance based on the voting values ​​to obtain the predicted UV coordinates of each key point. Its uncertainty; The 2D prediction branch is constructed based on the K-Block structure and outputs the 2D coordinate prediction values ​​of human SMPL key points and the uncertainty of the 2D coordinate prediction values ​​of the key points. The correction and optimization module describes the 3D human body representation parameters based on the SMPL model and single-person keypoint coordinates, outputs the 3D coordinate prediction values ​​of the SMPL keypoints, and optimizes the 2D coordinate prediction values ​​of the human body SMPL keypoints output by the 2D prediction branch based on the 3D coordinate prediction values ​​of the SMPL keypoints. The optimization equation based on the 2D-3D coordinate matching strategy is as follows: In the formula To satisfy the minimum loss function The optimal rotation matrix and translation vector for 2D-3D coordinate matching. and For the corresponding coordinates in SMPL vector form, considering the uncertainty... and weight Negative correlation, in terms of uncertainty and weight Create a diagonal matrix for each diagonal element. and ,by Substituting and simplifying, we get: In the formula, SVD decomposition is satisfied. Two sets of corresponding coordinate points form a point set. and , ( R , t Let be the rotation matrix and translation vector of a pair of Euclidean transformations, respectively. After this transformation, we can achieve the following: P and Q The error is the smallest. U , V Let be two orthogonal matrices, satisfying .

2. The pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy according to claim 1, characterized in that, The 2D pose loss function in the 2D prediction branch prediction process is: In the formula, and Representing the first i The true values ​​of the x and y coordinates of each key point. c i Representing the i Uncertainty of predicted values ​​for key points w The weights representing the regularization terms. Representing the first i Predicted horizontal and vertical coordinates of key points.

3. The pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy according to claim 1, characterized in that, The optimization process based on the 2D-3D coordinate matching strategy is as follows: Input the 2D coordinate prediction values ​​(KPS) of human SMPL keypoints 2D Uncertainty C of the predicted 2D coordinates of key points 2D And the 3D coordinate prediction value KPS of SMPL key points 3D ; 1b) Calculate KPS 2D The mean and variance var are used for subsequent recovery of the coordinates of 2D points, assuming KPS 2D The z-axis coordinates are all 0; 2b) Regarding KPS 2D and KPS 3D Perform normalization to align the model; 3b) Calculate KPS 2D C 2D and KPS 3D The outer product S = (KPS) 2D (C) 2D ) -1 (KPS 3D ); 4b) Calculate the SVD decomposition of S, S=UΣV', to obtain the rotation matrix R=VU'; 5b) Set the determinant of R to 1, i.e., det(R) = 1, to ensure that R is a rotation matrix and not a reflection matrix; 6b) Calculate KPS based on mean and var 3D The coordinates of the corresponding 2D point; 7b) Assign the transformed z-axis coordinates to KPS 2D ; 8b) Repeat steps 3b) to 7b) to optimize the rotation matrix R and obtain the final 2D human pose coordinates.

4. The pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy according to claim 3, characterized in that, During the optimization process, the key point regression loss function for: In the formula, and Representing the first i The true values ​​of the x and y coordinates of each key point. Representing the first i Predicted horizontal and vertical coordinates of key points.

5. The pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy according to claim 1, characterized in that, The detector is a Transformer-based DETR detector.

6. The pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy according to claim 1, characterized in that, The loss functions of the detector during the detection task include the L1 loss function calculated directly between the single person detection box and the ground truth box, the IoU optimization loss function, the pedestrian prediction confidence loss function, and the GIoU value loss function corresponding to the predicted detection box.

7. The pedestrian detection and pose estimation method based on a 2D-3D coordinate matching strategy according to claim 1, characterized in that, The backbone network is a simplified multilayer neural network based on ResNet, and the information recovery branch is a feature pyramid network.

Images