A method for detecting pseudo 3D bounding boxes of vehicles based on key points

Abstract

This invention discloses a method for detecting vehicle pseudo-3D bounding boxes based on keypoints, comprising the following steps: inputting an image; extracting the vehicle ROI through object detection; training multiple detection head branches using subpixel coordinate encoding; selecting the optimal detection head branch through online distillation with self-attention; outputting keypoint coordinates; obtaining the vehicle pseudo-3D category information; and connecting the predicted keypoints using the vehicle pseudo-3D category information to form a pseudo-3D bounding box. The above technical solution features a simple model structure, is easy to train, calculates self-attention in the channel dimension and scale space for the results of multiple branches, and finally fuses the two dimensions, enabling the capture of global information between independent branches.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a method for detecting a vehicle's pseudo 3D bounding box based on key points. Background Technology

[0002] Keypoint detection is an important branch of object detection, widely used in pose estimation, face recognition, autonomous driving, and other fields. With the continuous development of deep learning, the mainstream keypoint detection techniques currently fall into two categories: heatmap-based keypoint detection and regression-based keypoint detection.

[0003] Heatmap-based methods dominate keypoint detection due to their superior performance. These methods predict a heatmap for each keypoint, utilize local information on the heatmap to perform feature matching in the spatial dimension, predict the probability distribution of each point, and then use argmax to obtain extreme points. The prediction accuracy is generally higher than methods that directly regress coordinate values. The drawbacks are that maintaining accuracy requires upsampling operations (such as transposed convolution) and additional post-processing (such as DARK (distribution-aware coordinate representation)), increasing network computation and GPU memory requirements. Furthermore, accuracy drops sharply as input resolution decreases.

[0004] Regression-based methods utilize neural networks to directly regress the coordinates of keypoints in the original image, enabling end-to-end network training without post-processing and offering fast forward speeds. However, this method heavily relies on the spatial distribution of the input, is susceptible to noise interference in blurred scenes such as occlusion and truncation, has weak spatial generalization ability, struggles to fit the network, and typically results in lower detection accuracy compared to heatmap-based methods.

[0005] Data shows that current mainstream keypoint detection methods primarily rely on heatmap-assisted regression. Because manually labeled keypoint coordinates rarely accurately represent the true target location, Gaussian kernels are typically used to smooth the labels, smoothly unfolding the target point region into a two-dimensional Gaussian distribution. A probability value is assigned to each keypoint region on the heatmap, representing the confidence level that the point is a true target point; the target point has the highest probability value, typically set to 1. CenterNet extracts features from the input image, downsamples it to generate a quarter-sized heatmap, then performs classification and regression on the heatmap, finally converting the coordinate values ​​back to the original image. However, the accuracy of this method is affected by the feature map resolution, and quantization errors occur during downsampling, making it impossible to directly output coordinate values. DARK proposed a new distribution-aware keypoint coordinate representation method, approximating the heatmap's distribution information as a Taylor expansion distribution, generating an unbiased heatmap to reduce coordinate quantization errors introduced during feature map downsampling, but this also adds additional post-processing steps.

[0006] Chinese patent document CN107671414A discloses a "method, apparatus, and electronic device for constructing pseudo-3D bounding boxes." This method acquires video data of traffic scenes from traffic cameras and extracts vehicle images corresponding to target vehicles from the video data. Based on the 2D detection boxes of the vehicle body, upper half of the vehicle body, and end of the vehicle from the acquired vehicle images, a pseudo-3D bounding box corresponding to the target vehicle is constructed. However, this approach requires regressing multiple 2D detection boxes, containing redundant feature information, and necessitates filtering for overlapping points among the boxes, increasing the network complexity. Summary of the Invention

[0007] This invention primarily addresses the technical problems of quantization errors and high network complexity inherent in existing heatmap downsampling methods. It provides a method for detecting pseudo-3D bounding boxes of vehicles based on key points. By simplifying the regression coordinate point format into a simple classification task, it leverages the advantages of heatmap spatial information, uses sub-pixel coordinates to remove quantization errors during downsampling, and directly outputs key point coordinate values ​​using soft-argmax. This eliminates the need to maintain a high-resolution heatmap, resulting in a simple model structure that is easy to train. The method calculates self-attention in the channel dimension and scale space for the results of multiple branches, and finally fuses the two dimensions to capture global information between independent branches.

[0008] The above-mentioned technical problems of the present invention are mainly solved by the following technical solution: The present invention includes the following steps:

[0009] S1 Input image;

[0010] S2 target detection extracts vehicle ROI;

[0011] S3 uses subpixel coordinate encoding to train multiple detection head branches;

[0012] S4 selects the optimal detection head branch through self-attention online distillation;

[0013] S5 outputs the coordinates of key points;

[0014] S6 obtains the pseudo-3D category information of the vehicle;

[0015] S7 uses the vehicle's pseudo-3D category information to connect predicted key points to form a pseudo-3D bounding box.

[0016] Preferably, step S3 specifically includes:

[0017] S3.1 First, the coordinates of the key points P(x,y) on the original image are rounded to P(round(x),round(y)), and then downsampled to create heatmap labels;

[0018] During the training process of S3.2, the heatmap is transformed into a one-dimensional space, and the probability distribution of each point is obtained through a fully connected layer. The position information of point P on the current feature map is calculated using soft-argmax.

[0019] Preferably, step S3.1 specifically includes: first assuming the coordinates of point P, rounding the coordinates of key point P, and then setting the coordinates of point P in the heat map to (Hx, Hy) after downsampling by k times. The label value of this position is set to 1, and other background categories are set to 0.

[0020] Taking an original image size of 12×12 and k as an example, assume that the coordinates of point P are (6.2, 7.3). The coordinates of key point P are rounded to (6, 7). After being downsampled by k times, the coordinates of point P in the heat map are (1, 1). Set the label value of this position to 1 and set the other background values ​​to 0.

[0021] Preferably, step S3.2 converts the two-dimensional heatmap into a one-dimensional space, classifies each point using a fully connected layer, calculates the loss value with the generated labels, and obtains the probability value P of each point representing a key point. I (i = 0, 1, 2, ...), use the soft-argmax method to calculate the position C of the keypoint P in the one-dimensional vector. p for:

[0022]

[0023] Where j represents the same meaning as i, β is a scaling factor used to improve the accuracy of approaching the target point, and then the obtained one-dimensional coordinates are transformed into two-dimensional coordinates on the heatmap (H). x H y ),

[0024]

[0025]

[0026] Where W and H represent the width and height of the heatmap, respectively.

[0027] Preferably, step S3.1 converts the quantization error value of the heatmap into sub-pixel coordinates using a sub-pixel coordinate point method, and then uses a classification method to obtain the sub-pixel coordinate values ​​to solve the problem of non-integer coordinates after the original image coordinates are reduced by a factor of k when making heatmap labels. The sub-pixel label making process specifically includes expanding the unit point in the heatmap by a factor of k at the sub-pixel level, using convolution to map the point to a k×k feature map, treating each heatmap unit point as k sub-pixel points, i.e., mapping to the original image size level, and the decimal part of the quantized heatmap coordinates is the true label coordinate (S) under sub-pixel coordinates. x ′, Sy ′).

[0028] Because the creation of heatmap labels involves coordinate quantization, meaning the original image coordinates are not integers after being reduced by a factor of k, this invention uses sub-pixel coordinates to convert the quantization error values ​​of the heatmap into sub-pixel coordinates. Then, a classification method is still used to obtain the sub-pixel coordinate values. In this example:

[0029] S x =6%4=2

[0030] S y =7%4 =3.

[0031] Preferably, the true label coordinates (S) under the sub-pixel coordinates are... x ′, S y After obtaining the coordinates, similar to the method used to calculate the heatmap coordinates, the network again goes through a fully connected layer to classify the sub-pixel coordinates. The soft-argmax method is then used to transform the sub-pixel coordinates to (S). x S y The final output is the predicted key point P(P). x ,P y The coordinates in the original graph are:

[0032] P x =H x ×k+S x

[0033] P y =H y ×k+S y

[0034] The entire process of the keypoint detection head is completely differentiable, with only one rounding operation when labeling the original image. The error is at the level of a single pixel in the original image, which is much smaller than the quantization error level of multiple downsampling operations, or the error size of regression rounding can be achieved by adding a branch.

[0035] Preferably, step S4 specifically includes: first, expanding the keypoint detection head into multiple branches; jointly calculating self-attention in the channel dimension and self-attention in the scale dimension using the output feature maps of multiple branches; then fusing them to obtain the final integrated features; using the integrated model and each branch to calculate the coordinate loss to achieve the distillation process; and finally selecting the branch with the best performance based on the training results as the detection head of the network model. Since the detection head has a simple structure, distillation is used to enhance the performance of the detection head network.

[0036] Preferably, step S6, vehicle pseudo-3D classification, is a regular classification task, which categorizes vehicles into eight types based on their orientation in the image: head left, head right, tail left, tail right, head, tail, left, and right.

[0037] The beneficial effects of this invention are: by simplifying the form of regression coordinate points into a simple classification task, it combines the advantages of heatmap spatial information, uses sub-pixel coordinates to remove quantization errors in the downsampling process, and uses soft-argmax to directly output key point coordinate values ​​without maintaining a high-resolution heatmap. The model structure is simple and easy to train. It calculates self-attention in the channel dimension and self-attention in the scale space for the results of multiple branches, and finally fuses the two dimensions, which can capture global information between independent branches. Attached Figure Description

[0038] Figure 1 This is a flowchart of the present invention.

[0039] Figure 2 This is a diagram illustrating the calculation process of soft-argmax according to the present invention.

[0040] Figure 3 This is a schematic diagram of a subpixel tag according to the present invention.

[0041] Figure 4 This is a schematic diagram of a self-attention online knowledge distillation method according to the present invention. Detailed Implementation

[0042] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0043] Example: This example illustrates a method for detecting pseudo-3D bounding boxes of vehicles based on key points. Figure 1 As shown, the input image is first processed by a 2D detection network to obtain the region of interest (ROI) of the vehicle in the image. Then, key point detection and vehicle pseudo 3D classification are performed on the ROI. Finally, the predicted key points are connected to form a pseudo 3D bounding box using the vehicle pseudo 3D category information.

[0044] Similar to the heatmap method, the detection head uses a convolutional neural network to downsample the ROI to generate a heatmap. The heatmap size is 1 / k of the original ROI. The difference is that the detection task is transformed into multiple classification tasks for supervised learning. The sub-pixel coordinate concept is used to re-encode the coordinates to remove quantization errors during downsampling, and the network can directly output coordinate point information. First, the coordinates of the keypoint P(x,y) on the original image are rounded to P(round(x),round(y)), and then downsampled to create heatmap labels. During training, the heatmap is transformed into a one-dimensional space, and the probability distribution of each predicted point is obtained through a fully connected layer. Soft-argmax is used to calculate the position information of point P on the current feature map unit point. This process is as follows: Figure 2 As shown, taking an original image size of 12×12 and k=4 as an example, assuming the coordinates of point P are (6.2, 7.3), the coordinates of keypoint P are rounded to (6, 7). After downsampling by a factor of k, the coordinates of point P in the heatmap are (1, 1). The label value of this location is set to 1, and other background classes are set to 0. During the network prediction process, the two-dimensional heatmap is converted to a one-dimensional space, and a fully connected layer is used to classify each point. The loss value is calculated with the generated labels to obtain the probability value P of each point representing a keypoint. I (i = 0, 1, 2, ...), to make the network differentiable during the process of acquiring coordinate values, the soft-argmax method is used to calculate the position C of the keypoint P in the one-dimensional vector. p for:

[0045]

[0046] Here, j represents the same meaning as i, and β is a scaling factor used to improve the accuracy of approaching the target point. The obtained one-dimensional coordinates are then transformed into two-dimensional coordinates on the heatmap (H). x H y ),

[0047]

[0048]

[0049] Where W and H represent the width and height of the heatmap, respectively.

[0050] Because the creation of heatmap labels involves coordinate quantization, meaning the original image coordinates are not integers after being reduced by a factor of k, this invention utilizes sub-pixel coordinates to convert the quantization error values ​​of the heatmap into sub-pixel coordinates. Then, a classification method is still used to obtain the sub-pixel coordinate values. The sub-pixel label creation process is as follows: Figure 3As shown, the unit points in the heatmap are enlarged by a factor of k at the sub-pixel level, and then mapped to a k×k feature map using convolution. Each heatmap unit point is considered to be k sub-pixel in size, i.e., mapped to the original image size level. The decimal part of the heatmap coordinate quantization is the true label coordinate (S) at the sub-pixel coordinate level. x ′, S y In this example:

[0051] S x =6%4=2

[0052] S y ′=7%4=3

[0053] Similar to the method used to calculate heatmap coordinates, the network again passes through a fully connected layer to classify sub-pixel coordinates. The soft-argmax method is then used to transform the sub-pixel coordinates to (S... x S y The final output is the predicted key point P(P). x ,P y The coordinates in the original graph are:

[0054] P x =H x ×k+S x

[0055] P y =H y ×k+S y

[0056] The entire process of the keypoint detection head is fully differentiable, with rounding occurring only once when labeling the original image. The error is only at the level of a single pixel in the original image, far smaller than the quantization error of multiple downsampling steps. Alternatively, a branch can be added to regress the rounding error.

[0057] Because the detection head has a simple structure, distillation is used to enhance the performance of the detection head network. The proposed self-attention online distillation method first expands the keypoint detection head into multiple branches, such as... Figure 4 As shown, the output feature maps of multiple branches are jointly calculated to perform self-attention in the channel dimension and self-attention in the scale dimension, and then fused to obtain the final integrated feature. The distillation process involves using the integrated model and each branch to calculate the coordinate loss separately, and finally selecting the branch with the best performance as the detection head of the network model based on the training results.

[0058] The classification of vehicle pseudo 3D is a regular classification task. Based on the vehicle's orientation in the image, the vehicle pseudo 3D categories are divided into eight categories: head left, head right, tail left, tail right, head, tail, left, and right.

[0059] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0060] Although this paper uses terms such as ROI and subpixel coordinate encoding frequently, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A method for detecting a vehicle pseudo-3D bounding box based on key points, characterized in that, The method comprises the following steps: S1 inputting a picture; S2 target detection and extraction of a vehicle ROI; S3 training of a multi-detection head branch using sub-pixel coordinate coding; S3 comprises: S3.1 rounding the coordinates of key points P(x, y) on an original graph to P(round(x), round(y)) and making a heat map label after k times of down-sampling; S3.1 converting the quantization error value of the heat map into a sub-pixel coordinate point by using the method of the sub-pixel coordinate point, and then obtaining the sub-pixel coordinate value by using a classification method; S3.2 Convert the two-dimensional heat map to one-dimensional space during the training process, use a fully connected layer to classify each point, calculate the loss value with the label made, and get the probability value P of each point representing the key point i , use the soft-argmax method to calculate the position of the key point P in the one-dimensional vector; convert the obtained one-dimensional coordinates to two-dimensional coordinate points on the heat map ; the real label coordinates under the sub-pixel coordinates are obtained, and the same soft-argmax method is used to convert the coordinate points under the sub-pixel coordinates to , and the predicted key point , is multiplied by k and added , is multiplied by k and added ; S4 selection of an optimal detection head branch through self-attention online distillation; S5 output of key point coordinates; S6 acquisition of vehicle pseudo-3D class information; S7 connection of the predicted key points to form a pseudo-3D bounding box by using the vehicle pseudo-3D class information.

2. The method of claim 1, wherein, The step S3.1 specifically comprises the following steps: first, assuming the coordinates of P, rounding the coordinate value of the key point P, and obtaining the coordinates in the heat map after k times of down-sampling of the point P.

3. The method of claim 1, wherein, Method to compute the position of a keypoint P in a one-dimensional vector using soft-argmax is: ， where i and j represent the ith key point and the jth key point, respectively, an expansion factor for improving the accuracy of approaching the target point, and then converting the obtained one-dimensional coordinate to a two-dimensional coordinate point on the heat map , ， ， Wherein W and H represent the width and height of the heat map, respectively.

4. The method of claim 2, wherein, The sub-pixel label making process specifically includes expanding a unit point in the heat map by k times at a sub-pixel level, mapping the point to a k*k feature map by convolution, regarding each heat map unit point as k sub-pixel point sizes, i.e., mapping to the original map size level, and the decimal part of the heat map coordinate quantization being the real label coordinate under the sub-pixel coordinate .

5. The method of claim 4, wherein, The whole process of the key point detection head is completely differentiable, and the rounding is performed only once when the label is made on the original graph, the error is the size of a single pixel level of the original graph, which is much smaller than the quantization error level of multiple times of down-sampling.

6. The method of claim 1, wherein, The step S4 specifically comprises the following steps: first, expanding the key point detection head into multiple branches, jointly calculating the self-attention in the channel dimension and the self-attention in the scale dimension of the output feature maps of the multiple branches, then fusing the integrated features to obtain the final integrated features, using the integrated model and each branch to respectively calculate the coordinate loss to realize the distillation process, and finally selecting a branch with the best performance as the detection head of the network model through the training result.

7. The method of claim 1, wherein, The step S6 is a common classification task, and the vehicle pseudo-3D class is divided into eight classes, i.e., head left, head right, tail left, tail right, head, tail, left and right, according to the orientation of the vehicle in the graph.

Images