Adaptive 3D target detection method and device for optimizing BEV features and computer equipment

Abstract

The application relates to a 3D target detection method and device for adaptively optimizing BEV features, computer equipment and a storable medium. The 3D target detection method for adaptively optimizing BEV features comprises the following steps: acquiring ring-view image data from a Nuscenes dataset; performing feature extraction on the ring-view image by using a feature extraction backbone network to obtain image features; dimensionally upgrading the image features to a BEV space and compressing a z-axis to obtain BEV features; uniformly setting N 3D query boxes to be queried in the BEV space, projecting a candidate frame to a BEV plane to obtain a 2D rotating frame, extracting BEV features in the 2D rotating frame as prior features, using a center point of the query box as a query point, projecting the query point to an image feature plane by using camera parameters to obtain corresponding query features, and fusing the prior features and the query features at a channel level; predicting a perd box and a label score label by using the fused features, manually generating an optimization point at the center of the 6 faces of the top-k pred boxes, and updating the BEV features by using corresponding fused features of the optimization point after adding an offset; and feeding the optimized BEV features into a positioning detection head and a classification detection head to respectively predict a 3D object bounding box and an object category. The application has the advantages that the accuracy of 3D target detection of the ring-view image is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision target detection, and specifically to a 3D target detection method, apparatus, computer device, and readable storage medium that adaptively optimizes BEV features. Background Technology

[0002] As a key component of 3D perception, 3D object detection has wide applications in fields such as autonomous driving and robotics. While many LiDAR-based 3D detection methods have demonstrated significant performance, camera-based methods have received increasing attention in recent years. This shift is due not only to the lower deployment cost of cameras but also to their advantages in long-range detection and recognition of visual road elements. However, unlike LiDAR sensors which directly provide accurate depth information, relying solely on camera sensor images for target detection presents significant challenges. Therefore, how to utilize multi-view images to construct effective BEV (Battery Elevated Vehicle) features has become a crucial issue.

[0003] Currently, mainstream BEV feature construction methods fall into two branches: one is based on the Lift-Splat-Shoot method, which elevates 2D panoramic image features to 3D frustum point cloud features through explicit depth estimation and coordinate system transformation, and finally obtains BEV features through BEVPool operations. However, due to the characteristics of depth estimation, the feature information obtained from BEV rasters with low depth probability confidence is invalid, which will cause sparsity of BEV features; at the same time, due to the characteristics of camera imaging, the distribution density of the constructed frustum point cloud in BEV space is uneven, showing a trend of near-dense and far-sparse, resulting in uneven distribution of the constructed BEV feature information. The other branch is based on the Transformer method. This type of method does not rely on depth estimation, but interacts with 2D panoramic image features through queries to complete the end-to-end 3D object detection task. Furthermore, based on the query density, it can be divided into dense queries and sparse queries, with typical examples being BEVFormer and PETR, respectively. The former method constructs dense BEV (Browser Active Vehicle) grid features beforehand, performs numerous interactions between queries and 2D panoramic image features, and ultimately completes the construction of BEV features. However, the dense queries result in slow inference speed and high computational cost. The latter method uses sparse queries, which lightens the entire network structure and allows the query mechanism to focus more on foreground target information. However, by omitting the process of constructing BEV features, it leads to the loss of holistic scene information. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a 3D target detection method, apparatus, computer equipment and readable storage medium that adaptively optimizes BEV features, which greatly improves the accuracy of 3D target detection of panoramic images.

[0005] The present invention achieves the above-mentioned objectives by adopting the following technical solution: Firstly, the present invention provides a 3D target detection method that adaptively optimizes BEV features, comprising:

[0006] S1. Obtain panoramic image data from the Nuscenes dataset;

[0007] S2. Use a feature extraction backbone network to extract features from the panoramic image to obtain image features;

[0008] S3. Input the image features into the viewpoint conversion module to generate BEV features;

[0009] S4, the BEV feature optimizer includes 6 transformer decoder layers. Query boxes are pre-set evenly in 3D space to query and fuse BEV features and image features respectively, resulting in adaptive fused features. The fused features are then fed into the localization detection head and the classification detection head to obtain the 3D bounding box attribute pred box and the label score. At the same time, the query box is updated using the pred box and then fed into the next decoder layer.

[0010] S5. Take the top N pred boxes with high label scores and use their corresponding fusion features to update and optimize the BEV features.

[0011] S6. Using the optimized BEV features, generate heatmaps for each category in the Nuscenes dataset, as well as the target's center point height, bounding box length, width, height, yaw angle around the z-axis, and velocity along the x-axis and y-axis, etc.

[0012] Furthermore, step S3 specifically includes:

[0013] S31. Manually generate depth preset values ​​from 1 to 60 at the image feature level, with a spacing of 1 meter. At this point, the coordinates of each depth point in pixel space can be represented as... By utilizing the camera's intrinsic and extrinsic parameters, depth points can be transformed from the pixel coordinate system to the vehicle coordinate system, thus obtaining pseudo-point cloud coordinates in the BEV space. The conversion formula is:

[0014]

[0015] in, Indicates camera external parameters. This indicates the camera's internal parameters.

[0016] S32. The image features are fed into the semantic feature extractor and the depth generator respectively to obtain the semantic features and depth probability confidence. The two are then cross-productted to obtain the pseudo-point cloud semantic features in the BEV space.

[0017] S33, constructing a BEV grid with a length and width ranging from -51.2 to 51.2 and a spacing of 0.8m, can be divided into: 28 cubic pillars with equal base area and infinite height are called Pillars. The semantic features of all pseudo-point clouds in the same Pillar are summed to obtain the BEV Feature.

[0018] Furthermore, step S4 specifically includes:

[0019] S41. Map the query box in 3D space to the BEV plane, extract the features of all BEV grids within the directed box, and calculate the mean value to obtain the RoI features.

[0020] S42. All RoI features interact with each other through self-attention to obtain global information and prevent multiple query boxes from converging to the same RoI feature. The self-attention formula is:

[0021]

[0022] Where Softmax() represents the normalization function, and d represents the dimension of K. It is a learnable parameter, where D represents the distance between the center points of any two query boxes.

[0023] S43. Feed the RoI features into a linear layer to predict the offset and couple it with the query box center point to form a reference point. Project the reference point onto the image feature plane and use bilinear interpolation to sample features. The projection formula is:

[0024]

[0025] in, Indicates camera external parameters. This indicates the camera's internal parameters.

[0026] S44. At the channel level, RoI features and sampled image features are fused as query features. After passing through the ffn layer, they are fed into the localization detection head and the classification detection head to obtain the target attribute pred box and the target category score label. The pred box is used to update the query box and is used as the input of the next decoder layer. A total of 6 iterations are performed.

[0027] Furthermore, step S5 specifically includes:

[0028] S51. Manually generate the center points of the query box output by the last decoder layer in the above six faces. The RoI features corresponding to the query box are fed into a linear layer to predict the offset, and then coupled... ,get The projection formula of S33 is used to sample planar features. ;

[0029] S52, will Mapping to the BEV plane yields two-dimensional coordinates Its corresponding BEV raster index is fusion and , to obtain optimized features The formula is:

[0030]

[0031] Where n is the number of cells falling within the current BEV grid. The quantity.

[0032] In a second aspect, the present invention provides a 3D target detection device for a panoramic image, used to implement the 3D target detection method for adaptively optimizing BEV features as described above, the detection device comprising:

[0033] The view is like a dataset loading module, specifically used for:

[0034] Obtain the panoramic image based on the scene sequence;

[0035] Obtain the 3D target bounding box information in the global coordinate system corresponding to the scene sequence, and transform the 3D target bounding box positioning information to the forward-looking camera coordinate system according to the transformation matrix. Use this as the reference coordinate system, and calculate the yaw angle and velocity along the xy axis in this coordinate system.

[0036] Filter out targets that do not have lidar or ladar point clouds in the 3D target bounding box to prevent visually unreasonable targets from interfering with the training of the network model;

[0037] The perspective conversion module is used to select BEV features for BEV perception to construct a model for perspective conversion. Specifically, it is used for:

[0038] Lift-Splat was chosen as the model architecture for viewpoint transformation;

[0039] The adaptive BEV feature optimization module is used to select a method based on the transformer query mechanism to optimize BEV features. Specifically, it is used for:

[0040] SparseBEV was chosen as the model architecture for adaptive BEV feature optimization.

[0041] The training module is used to select a neural network model for 3D object detection in a panoramic image for training. Specifically, it is used for:

[0042] ResNet50 and FPN were selected as the backbone networks for feature extraction, and the CenterPoint model was used as the 3D object detection trainer.

[0043] The image features are input into the aforementioned viewpoint conversion module and adaptive BEV feature optimization module to obtain BEV features;

[0044] Input BEV features to train the CenterPoint model;

[0045] The performance evaluation module is used for model performance evaluation, specifically for:

[0046] The input loop image is detected in the BEV space, and non-maximum suppression merging is used to obtain the final prediction result;

[0047] The test images from the Nuscenes panoramic image dataset were used as the evaluation dataset, and mAP and NDS were calculated as the model performance evaluation metrics.

[0048] Input the evaluation dataset and obtain the evaluation results.

[0049] Thirdly, the present invention provides a computer device including a memory storing program instructions, wherein when the program instructions are executed, the 3D target detection method for adaptively optimizing BEV features as described above is performed.

[0050] The beneficial effects of this invention are as follows:

[0051] This invention effectively preserves the deep semantic information of the image by projecting pixels in the panoramic image into three-dimensional space and further mapping them to the BEV plane, providing a richer semantic foundation for subsequent processing.

[0052] This invention introduces an advanced sparse query mechanism to adaptively optimize BEV features, greatly enriching feature representation and effectively alleviating the problems of feature sparsity and uneven information distribution in traditional methods, thus significantly improving data representativeness. Simultaneously, this invention introduces an offset to the fixed points of the bounding boxes during adaptive optimization, enabling the network to focus more precisely on target instances rather than general 3D bounding boxes.

[0053] This invention utilizes a neural network model trained on a surround-view image dataset, which can accurately identify targets in complex autonomous driving scenarios, achieving efficient and stable 3D target detection of surround-view images and demonstrating excellent scene understanding capabilities. Attached Figure Description

[0054] Figure 1 This is a flowchart of a 3D target detection method for adaptively optimizing BEV features provided by an embodiment of the present invention;

[0055] Figure 2 This is a structural block diagram of a 3D target detection device for a panoramic image provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0057] Example 1

[0058] This invention utilizes the Nusenens dataset to construct and optimize BEV features to train a convolutional neural network 3D object detection model, such as... Figure 1 As shown, the 3D target detection method based on adaptive optimization of BEV features, also known as the 3D target detection method based on adaptive optimization of BEV features, specifically includes the following steps:

[0059] Step 1: Obtain the panoramic image data from the Nuscenes dataset. The specific steps are as follows:

[0060] 1) Read the surround view image data from the Nuscenes dataset, which contains rich 3D target annotation data and surround view images, suitable for autonomous driving tasks. For each scene sequence, extract the corresponding surround view image and obtain the 3D target bounding box information in the global coordinate system of the scene.

[0061] 2) Transform the 3D target bounding box information from the global coordinate system to the forward-looking camera coordinate system, and use this coordinate system as the reference coordinate system for subsequent processing.

[0062] 3) Filter out targets that do not contain LiDAR or LaDAR point clouds to ensure that only valid target data is used and to avoid invalid targets that interfere with training.

[0063] Step 2: Use a feature extraction backbone network to extract features from the panoramic image to obtain image features. The specific steps are as follows:

[0064] 1) Use pre-trained ResNet50 and FPN to extract features from the panoramic image to obtain the feature map of the panoramic image;

[0065] Step 3: Input the image features into the viewpoint conversion module to generate BEV features. The specific steps are as follows:

[0066] 1) Manually generate preset depth values ​​from 1 to 60 at 1-meter intervals at the image feature level. In this case, the coordinates of each depth point in pixel space can be represented as... By utilizing the camera's intrinsic and extrinsic parameters, depth points can be transformed from the pixel coordinate system to the vehicle coordinate system, thus obtaining pseudo-point cloud coordinates in the BEV space. The conversion formula is:

[0067]

[0068] in, Indicates camera external parameters. This indicates the camera's internal parameters.

[0069] 2) The image features are fed into the semantic feature extractor and the depth generator respectively to obtain the semantic features and depth probability confidence. The two are then cross-productted to obtain the pseudo-point cloud semantic features in the BEV space.

[0070] 3) Construct a BEV grid with a length and width range of -51.2~51.2 and a spacing of 0.8m, which can be divided into: 28 cubic pillars with equal base area and infinite height are called Pillars. The semantic features of all pseudo-point clouds in the same Pillar are summed to obtain the BEV Feature.

[0071] Step 4: BEV feature optimization and update, the specific steps are as follows:

[0072] 1) Pre-set 900 query boxes (x, y, z, w, l, h, sin) evenly in three-dimensional space. cos , , );

[0073] 2) Map the query box in 3D space to the BEV plane, extract the features of all BEV grids within the directed box, and calculate the mean to obtain the RoI features;

[0074] 3) All RoI features interact with each other through self-attention to obtain global information and prevent multiple queryboxes from converging to the same RoI feature. The self-attention formula is:

[0075]

[0076] Where Softmax() represents the normalization function, and d represents the dimension of K. It is a learnable parameter, where D represents the distance between the center points of any two query boxes.

[0077] 4) Feed the RoI features into a linear layer to predict the offset and couple it with the query box center point to form a reference point. Project the reference point onto the image feature plane and use bilinear interpolation to sample features. The projection formula is:

[0078]

[0079] in, Indicates camera external parameters. This indicates the camera's internal parameters.

[0080] 5) At the channel level, RoI features and sampled image features are fused as query features. After passing through the ffn layer, they are fed into the localization detection head and the classification detection head to obtain the target attribute pred box and the target category score label. The pred box is used to update the query box and is used as the input of the next decoder layer. A total of 6 iterations are performed.

[0081] 6) Manually generate the center points of the query box output from the last decoder layer on the six faces. The RoI features corresponding to the query box are fed into a linear layer to predict the offset, and then coupled... ,get The projection formula of S33 is used to sample planar features. ;

[0082] 7) Mapping to the BEV plane yields two-dimensional coordinates Its corresponding BEV raster index is fusion and , to obtain optimized features The formula is:

[0083]

[0084] Where n is the number of cells falling within the current BEV grid. The quantity.

[0085] Step 5: Select a convolutional neural network model for 3D object detection in panoramic images and train it. The specific steps are as follows:

[0086] 1) Selecting a basic convolutional neural network model, this invention uses CenterPoint to describe the training process and performance evaluation;

[0087] 2) Input the optimized BEV features and train the model;

[0088] Step 6: Model performance evaluation:

[0089] 1) During prediction, the input loop view image is detected in the BEV space and non-maximum suppression (NMS) is used to merge the images to obtain the final prediction result;

[0090] 2) Use the test set of the Nuscenes dataset as the evaluation dataset, and use mAP and NDS as the model performance evaluation metrics;

[0091] 3) Input the evaluation panoramic image and compile the evaluation results;

[0092] This invention first constructs BEV (Browser-Eye-Driven) features using a panoramic image and designs an adaptive optimization method for these features, resulting in BEV features with accurate scene and target information. This alleviates the problems of feature sparsity and uneven information distribution in traditional methods. For convolutional network models used for 3D object detection, training on the Nuscenes dataset yields a high-performance 3D object detection model. This model is suitable for 3D image object detection tasks and can be applied to numerous fields such as robot vision, consumer electronics, security, autonomous driving, human-to-human interaction, image retrieval, intelligent monitoring, augmented reality, and virtual reality. It also provides a foundation for more complex tasks such as semantic segmentation and scene understanding.

[0093] Example 2

[0094] like Figure 2 The aforementioned 3D target detection device for panoramic images is used to implement the 3D target detection method with adaptively optimized BEV features as described in Embodiment 1. The device includes a panoramic image dataset loading module, a viewpoint conversion module, an adaptive BEV feature optimization module, a training module, a performance evaluation module, and an input evaluation dataset.

[0095] The surround view image dataset loading module is used to obtain surround view images based on scene sequences;

[0096] Specifically, the 3D target bounding box information in the global coordinate system corresponding to the scene sequence is obtained, and the 3D target bounding box positioning information is transformed to the forward-looking camera coordinate system according to the transformation matrix. This is used as the reference coordinate system, and the yaw angle and velocity along the xy axis in this coordinate system are calculated at the same time.

[0097] Filter out targets that do not have lidar or ladar point clouds in the 3D target bounding box to prevent visually unreasonable targets from interfering with the training of the network model.

[0098] The perspective conversion module is used to select BEV features for BEV perception to build a model for perspective conversion. Specifically, it is used to select Lift-Splat as the model architecture for perspective conversion.

[0099] The adaptive BEV feature optimization module is used to select a method based on the transformer query mechanism to optimize BEV features. Specifically, it is used to select SparseBEV as the model architecture for adaptive BEV feature optimization.

[0100] The training module is used to select a neural network model for 3D object detection in a panoramic image for training, specifically for:

[0101] ResNet50 and FPN were selected as the backbone network for feature extraction, and the CenterPoint model was used as the 3D object detection trainer. Next, the image features were input into the above-mentioned viewpoint conversion module and adaptive BEV feature optimization module to obtain BEV features. Then, the CenterPoint model was trained by inputting the BEV features.

[0102] The performance evaluation module is used to evaluate the model's performance. Specifically, it is used to: perform detection on the input aroundview image in the BEV space and use non-maximum suppression merging to obtain the final prediction result; and use test images from the Nuscenes aroundview image dataset as the evaluation dataset to calculate mAP and NDS as model performance evaluation metrics.

[0103] Input the evaluation dataset to obtain the evaluation results.

[0104] Example 3

[0105] This embodiment provides a computer device, including a memory storing program instructions. When the program instructions are executed, they perform a 3D object detection method for adaptively optimizing BEV features as described in Embodiment 1. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by instructing related hardware using program instructions.

[0106] Example 4

[0107] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the 3D target detection method of adaptively optimizing BEV features according to Embodiment 1.

[0108] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A 3D target detection method that adaptively optimizes BEV features, characterized in that, Includes the following steps: S1. Obtain panoramic image data from the Nuscenes dataset; S2. Use a feature extraction backbone network to extract features from the panoramic image in S1 to obtain image features; S3. Input the image features from S2 into the viewpoint conversion module to generate BEV features; S4. The BEV feature optimizer includes 6 transformer decoder layers. Query boxes are pre-set evenly in 3D space to query and fuse the BEV features and the image features respectively, resulting in adaptive fused features. The fused features are then fed into the localization detection head and the classification detection head to obtain the 3D bounding box attribute pred box and the label score. At the same time, the query box is updated using the pred box and fed into the next decoder layer. S5. Take the top N pred boxes with high label scores from S4 and use their corresponding fusion features to update and optimize the BEV features. S6. Using the optimized BEV features, generate heatmaps for each category in the Nuscenes dataset and the attributes of the target, including center point height, length, width, and height of the bounding box, yaw angle around the z-axis, and velocity attributes along the x-axis and y-axis. Step S4 specifically includes: S41. Map the query box in 3D space to the BEV plane, extract the features of all BEV grids within the directed box, and calculate the mean value to obtain the RoI features. S42. All RoI features interact with each other through self-attention to obtain global information and prevent multiple queryboxes from converging to the same RoI feature. The self-attention formula is: ； Where Softmax() represents the normalization function, and d represents the dimension of K. These are learnable parameters, where D represents the distance between the center points of any pair of queryboxes; S43. Feed the RoI features into a linear layer to predict the offset and couple it with the query box center point to form a reference point. Project the reference point onto the image feature plane and use bilinear interpolation to sample features. The projection formula is: ； in, Indicates camera external parameters. Indicates camera intrinsic parameters; S44. At the channel level, RoI features and sampled image features are fused as query features. After passing through the ffn layer, they are fed into the localization detection head and the classification detection head to obtain the target attribute pred box and the target category score label. The pred box is used to update the query box and is used as the input of the next decoder layer. A total of 6 iterations are performed.

2. The 3D target detection method with adaptively optimized BEV features according to claim 1, characterized in that, S3 specifically includes: S31. Manually generate depth preset values ​​from 1 to 60 at the image feature level, with a spacing of 1 meter. At this point, the coordinates of each depth point in pixel space can be represented as... By utilizing the camera's intrinsic and extrinsic parameters, depth points can be transformed from the pixel coordinate system to the vehicle coordinate system, thus obtaining pseudo-point cloud coordinates in the BEV space. The conversion formula is: ； in, Indicates camera external parameters. Indicates camera intrinsic parameters; S32. The image features are fed into the semantic feature extractor and the depth generator respectively to obtain the semantic features and depth probability confidence. The two are then cross-productted to obtain the pseudo-point cloud semantic features in the BEV space. S33, constructing a BEV grid with a length and width ranging from -51.2 to 51.2 and a spacing of 0.8m, can be divided into: A cube with an equal base area and infinite height is called a Pillar. The semantic features of all pseudo-point clouds in the same Pillar are summed to obtain the BEV Feature.

3. The 3D target detection method with adaptively optimized BEV features according to claim 1, characterized in that, Step S5 specifically includes: S51. Manually generate the center points of the query box output by the last decoder layer in the above six faces. The RoI features corresponding to the query box are fed into a linear layer to predict the offset, and then coupled... ,get The projection formula of S33 is used to sample planar features. ; S52, will Mapping to the BEV plane yields two-dimensional coordinates Its corresponding BEV raster index is fusion and , to obtain optimized features The formula is: ； Where n is the number of cells falling within the current BEV grid. The quantity.

4. A 3D target detection device for a panoramic image, used to implement the 3D target detection method for adaptively optimizing BEV features as described in any one of claims 1-3, characterized in that, The detection device includes: The view is like a dataset loading module, specifically used for: Obtain the panoramic image based on the scene sequence; Obtain the 3D target bounding box information in the global coordinate system corresponding to the scene sequence, and transform the 3D target bounding box positioning information to the forward-looking camera coordinate system according to the transformation matrix. Use this as the reference coordinate system, and calculate the yaw angle and velocity along the xy axis in this coordinate system. Filter out targets that do not have lidar or ladar point clouds in the 3D target bounding box to prevent visually unreasonable targets from interfering with the training of the network model; The perspective conversion module is used to select BEV features for BEV perception to construct a model for perspective conversion. Specifically, it is used for: Lift-Splat was chosen as the model architecture for viewpoint transformation; The adaptive BEV feature optimization module is used to select a method based on the transformer query mechanism to optimize BEV features. Specifically, it is used for: SparseBEV was chosen as the model architecture for adaptive BEV feature optimization. The training module is used to select a neural network model for 3D object detection in a panoramic image for training. Specifically, it is used for: ResNet50 and FPN were selected as the backbone networks for feature extraction, and the CenterPoint model was used as the 3D object detection trainer. The image features are input into the aforementioned viewpoint conversion module and adaptive BEV feature optimization module to obtain BEV features; Input BEV features to train the CenterPoint model; The performance evaluation module is used for model performance evaluation, specifically for: The input loop image is detected in the BEV space, and non-maximum suppression merging is used to obtain the final prediction result; The test images from the Nuscenes panoramic image dataset were used as the evaluation dataset, and mAP and NDS were calculated as the model performance evaluation metrics. Input the evaluation dataset and obtain the evaluation results.

5. A computer device, comprising a memory storing program instructions, characterized in that, When the program instructions are executed, the adaptive optimization of BEV features 3D target detection method as described in any one of claims 1-3 is performed.

Images