Visual three-dimensional target detection method and device based on normal distribution weighted feature mapping
Patent Information
- Application Number
- CN202510708791.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-29
- Publication Date
- 2025-08-12
AI Technical Summary
The existing pure visual three-dimensional object detection method lacks depth information guidance when mapping two-dimensional features to three-dimensional space, resulting in insufficient detection accuracy and robustness. The traditional method has high calculation cost and low efficiency.
A reverse projection module based on normal distribution weighted feature mapping is designed. Through depth estimation and rapid lookup tables, a weight function is constructed to project two-dimensional features into three-dimensional space, enhance the bird's-eye view feature expression ability, and adopt space-to-channel operator compression and timing fusion to improve detection accuracy and real-time.
It improves the accuracy and robustness of three-dimensional object detection, reduces the calculation cost, meets the needs of real-time application, has good scalability and a feature expression that takes into account global information and local details.
Smart Images
Figure CN120472431A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of three-dimensional target detection based on deep learning, and in particular to a method and device for visual three-dimensional target detection based on normal distribution weighted feature mapping. Background Art
[0002] In the fields of autonomous driving and computer vision, 3D object detection is a key task, aiming to identify and localize objects in 3D space, such as vehicles, pedestrians, and bicycles, from sensor data. To achieve comprehensive perception and understanding of the environment, algorithms based on a bird's-eye view have attracted widespread attention due to their advantages in spatial representation and path planning.
[0003] Traditional 3D object detection methods typically rely on point cloud data acquired by sensors such as LiDAR. However, the high dimensionality, sparsity, disorder, and irregularity of point cloud data make direct application of traditional image processing and machine learning methods challenging. Consequently, pure vision-based 3D object detection methods have begun to gain widespread application. These methods directly utilize 2D images for 3D object detection, reducing reliance on expensive sensors like LiDAR. These methods can be broadly categorized into two types: forward projection-based and back-projection-based.
[0004] The Lift, Splat, Shoot algorithm (LSS), a classic forward-projection-based method, was the first to utilize depth information to construct bird's-eye-view features. The LSS algorithm first predicts the depth distribution of 2D pixels, then generates the corresponding camera's view frustum. Each frustum is then aggregated into 3D voxels, and finally voxel pooling is performed to obtain bird's-eye-view features. This forward-projection approach is cumbersome and has high inference costs. Furthermore, the LSS algorithm predicts a depth probability distribution for each pixel, and the depth prediction is supervised by the final detection loss, resulting in very blurry and low-quality depth predictions.
[0005] Methods based on back-projection include the Simple-BEV algorithm and the Fast-BEV algorithm. Simple-BEV provides a direct and efficient method for generating bird's-eye view features, directly mapping two-dimensional image features to the bird's-eye view feature space through a convolutional neural network. This method has low computational complexity and is easy to implement. Fast-BEV is an optimization of Simple-BEV, improving the efficiency of bird's-eye view feature generation by improving the network structure and feature fusion strategy. Although these methods are very efficient in reasoning and deployment, due to the lack of guidance from depth information, the mapping of two-dimensional features to three-dimensional space is based only on a fixed projection method, which easily leads to confusion of spatial features and cannot accurately reflect the true position and shape of objects in three-dimensional space. Therefore, how to effectively integrate depth information in pure visual three-dimensional target detection, improve the mapping accuracy of two-dimensional features to three-dimensional space, and enhance the expressiveness of three-dimensional features has become a hot topic and difficulty in current research. Summary of the Invention
[0006] In response to the above problems, the present invention provides a method and device for visual three-dimensional target detection based on normally distributed weighted feature mapping. This method designs a depth-guided three-dimensional to two-dimensional reverse projection module, which accurately estimates the depth value of each feature point in the two-dimensional feature map and constructs a normally distributed weight function to perform weighted mapping on the features. Compared with the traditional forward projection method, this method avoids the tedious steps of constructing the viewing cone and aggregating voxel space, and quickly fills in dense bird's-eye view features through feature indexing, thereby improving the efficiency and real-time performance of the reasoning process. At the same time, during the reverse projection process, this method provides depth guidance based on normally distributed weighting for feature projection, enhances the expressive power of the bird's-eye view features, and significantly improves the accuracy and robustness of three-dimensional target detection.
[0007] Specifically, the present invention adopts the following technical solutions:
[0008] A method for visual three-dimensional object detection based on normal distribution weighted feature mapping includes the following steps:
[0009] S1: The surround view camera image is input into the backbone network for feature extraction to obtain multi-scale two-dimensional feature maps. is the number of surround view cameras;
[0010] S2: Use a pre-trained depth estimation network to predict the depth of the two-dimensional feature map and obtain A depth map;
[0011] S3: Based on the internal and external parameters of the camera, pre-calculate the mapping relationship between each voxel in the three-dimensional space and the corresponding two-dimensional feature point in the two-dimensional feature map, and establish a fast lookup table based on this mapping relationship;
[0012] S4: For each 2D feature point, a weight function is constructed based on its depth value. Different weights are assigned to the 3D voxels corresponding to the same 2D feature point based on their distance. The 2D feature is projected into 3D space to obtain a 4D voxel tensor.
[0013] S5: Compress the 4D voxel tensor using the space-to-channel (S2C) operator to obtain the bird's-eye view features;
[0014] S6: The bird's-eye view features of several historical frames are spatially aligned, concatenated and fused with the current frame to obtain the temporally fused bird's-eye view features;
[0015] S7: The final bird's-eye view features are input into the bird's-eye view feature encoder and connected to a two-stage three-dimensional object detection head CenterPoint network to obtain the final object detection results.
[0016] A visual three-dimensional target detection device based on normal distribution weighted feature mapping, comprising:
[0017] Feature extraction module, used to The surround view camera image is input into the backbone network for feature extraction to obtain multi-scale two-dimensional feature maps. is the number of surround view cameras;
[0018] The depth prediction module is used to use a pre-trained depth estimation network to predict the depth of the two-dimensional feature map. A depth map;
[0019] A calculation module is used to pre-calculate the mapping relationship between each voxel in the three-dimensional space and the corresponding two-dimensional feature point in the two-dimensional feature map based on the internal and external parameters of the camera, and establish a fast lookup table based on this mapping relationship;
[0020] The back-projection module is used to construct a weight function for each 2D feature point based on its depth value, assign different weights to the 3D voxels corresponding to the same 2D feature point according to their distance, and project the 2D features into 3D space to obtain a 4D voxel tensor;
[0021] Compression module, used to compress the 4D voxel tensor using the space-to-channel (S2C) operator to obtain the bird's-eye view features;
[0022] The concatenation module is used to spatially align the bird's-eye view features of several historical frames with the current frame, and then concatenate and fuse them to obtain the temporally fused bird's-eye view features.
[0023] The result output module is used to input the final bird's-eye view features into the bird's-eye view feature encoder and connect a two-stage three-dimensional target detection head CenterPoint network to obtain the final target detection results.
[0024] A computing device comprises: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device executes a visual three-dimensional target detection method based on a normal distribution weighted feature map.
[0025] A readable storage medium storing program instructions, when the program instructions are read and executed by a computing device, enables the computing device to perform a visual three-dimensional target detection method based on normal distribution weighted feature mapping.
[0026] Compared with the existing algorithms, the technical solution of the present invention has the following beneficial effects:
[0027] The present invention designs a depth-guided three-dimensional to two-dimensional back projection module to improve the existing bird's-eye view feature mapping and generation strategy. This module is based on a depth-based weight calculation method, which can more accurately reflect the true position and shape of objects in three-dimensional space. By depth-weighting two-dimensional features, the feature expression of the bird's-eye view feature space is enriched, taking into account both global information and local details. By adopting a fast lookup table and an efficient calculation method, while improving the detection performance, the high computational efficiency of the algorithm is maintained to meet the requirements of real-time applications. This method has good scalability and can be combined with other technologies that improve the accuracy of depth prediction, such as multimodal fusion, self-supervised learning, etc., to further improve the overall performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 Flowchart of the visual three-dimensional target detection method based on normal distribution weighted feature mapping in the present invention;
[0029] Figure 2 Schematic diagram of the three-dimensional to two-dimensional back projection process based on depth guidance in the present invention. DETAILED DESCRIPTION
[0030] The following will be combined Figure 1 、 Figure 2 , describing in detail the technical solutions adopted in the embodiments of the present invention. Obviously, the embodiments only represent some possible implementations of the present invention, not all of them. Based on the technical ideas and methods demonstrated in the embodiments of the present invention, ordinary technicians in this field can deduce all other possible implementations based on the embodiments of the present invention without involving creative work, and these methods should all fall within the scope of protection of the present invention.
[0031] like Figure 1The figure below is a flowchart of the visual 3D target detection method based on normally distributed weighted feature mapping in this invention. In this method, depth guidance based on normally distributed weighting is introduced during the back-projection process, which improves the accuracy of feature projection and effectively enhances the expressiveness of bird's-eye view features. The algorithm includes seven steps: extracting multi-scale features, depth prediction, establishing mapping relationships, feature projection, constructing bird's-eye view features, temporal fusion, and detecting targets:
[0032] S1: Input N surround camera images into the backbone network for feature extraction to obtain multi-scale two-dimensional feature maps. is the number of surround view cameras;
[0033] Extracting multi-scale features is to input the surround camera image into the backbone network for feature extraction. The input surround camera image is A tensor of is the number of surround-view cameras, are the height and width of the image respectively, and 3 represents the RGB channels of the image. The backbone network includes the feature extraction network ResNet-50 and the feature pyramid network. The ResNet-50 network is used to obtain the four-stage features. In the output part, a three-layer multi-scale feature pyramid network structure is used for feature fusion, and finally three layers of two-dimensional feature maps of different scales are obtained. , whose scale is , is the number of channels of the feature.
[0034] S2: Use a pre-trained depth estimation network to predict the depth of the two-dimensional feature map to obtain N depth maps;
[0035] Depth prediction is the extraction of two-dimensional feature maps For each pixel in , a pre-trained depth estimation network is used to estimate its depth value in three-dimensional space. For this depth estimation network, the generated autonomous driving scene dataset is combined with the real autonomous driving scene dataset to fine-tune the network. The network adopts an encoder-decoder structure and outputs a depth map of the same size as the input image, with a scale of .
[0036] S3: Based on the internal and external parameters of the camera, pre-calculate the mapping relationship between each voxel in the three-dimensional space and the corresponding two-dimensional feature point in the two-dimensional feature map, and establish a fast lookup table based on this mapping relationship;
[0037] Constructing a mapping relationship is to use the internal and external parameters of the camera to construct the projection relationship between the three-dimensional space voxels and the two-dimensional image pixels. Define a three-dimensional space area around the vehicle and divide it into two areas of size A voxel grid where are the number of voxels in the vertical and horizontal directions on the feature space plane of the bird’s-eye view, The projection matrix from the world coordinate system to the image coordinate system can be obtained through the camera's internal and external parameter matrices, and the dimension is , where 3 corresponds to the homogeneous coordinates of the two-dimensional image plane , 4 corresponds to the homogeneous coordinates in three-dimensional space The mapping relationship between the voxel grid and the corresponding two-dimensional feature points in the two-dimensional feature map is obtained according to the projection matrix, and the mapping relationship between the two is saved as a quick lookup table. The two-dimensional pixel coordinates corresponding to the voxels in the three-dimensional space are calculated according to the mapping relationship, and the two-dimensional pixel coordinates corresponding to each voxel in the three-dimensional space are recorded in the quick lookup table.
[0038] S4: For each 2D feature point, a weight function is constructed based on its depth value. Different weights are assigned to the 3D voxels corresponding to the same feature point according to their distance. The 2D features are projected into 3D space to obtain a 4D voxel tensor.
[0039] In one embodiment, during the projection of two-dimensional features into three-dimensional space, a depth-based feature weighting module is added to improve the accuracy of the projection of two-dimensional features into three-dimensional space and enhance the expressiveness of bird's-eye view features. Figure 2 , the feature projection process in the present invention is described in detail.
[0040] Figure 2 This is a schematic diagram of the depth-guided 2D to 3D back-projection process in the present invention. , taking the largest scale feature as an example.
[0041] S4-1, for each three-dimensional voxel in the three-dimensional space region defined above, , find the corresponding two-dimensional pixel coordinates through a quick lookup table , obtain the two-dimensional feature map through the coordinate And the corresponding depth on the depth map .
[0042] S4-2, with depth As the mean, construct a normal distribution weight function, the standard deviation of the weight function Dynamically adjust according to application requirements to control the width of the weight distribution. The weight function of this normal distribution is expressed as:
[0043] ,in, is the distance from the voxel to the origin of the ego vehicle.
[0044] S4-3, the calculated coordinates are Voxels To the origin of the vehicle Euclidean distance :
[0045] ;
[0046] S4-4: Sampling the sampling point on the weight function to obtain the weight corresponding to each voxel ;
[0047] ;
[0048] S4-5, for this 3D voxel , multiply the two-dimensional feature map by the corresponding weight to obtain the feature representation of the three-dimensional voxel:
[0049] ,
[0050] Calculate the feature representation corresponding to all three-dimensional voxels to obtain a complete three-dimensional voxel feature representation with a scale of A 4D voxel tensor.
[0051] S5: Compress the 4D voxel tensor using the space-to-channel (S2C) operator to obtain the bird's-eye view features;
[0052] Compress the 4D voxel tensor to scale using the space-to-channel operator For smaller-scale bird's-eye view features, we upsample to the same scale along the X and Y dimensions, concatenate the multi-scale and multi-frame features in the channel dimension, and fuse them from higher to lower parameter amounts. The resulting bird's-eye view feature size is .
[0053] S6: The bird's-eye view features of several historical frames are spatially aligned, concatenated and fused with the current frame to obtain the temporally fused bird's-eye view features;
[0054] Perform temporal fusion on the bird's-eye view features at the current time T. Align the bird's-eye view features of the three historical frames, namely the bird's-eye view features at time T-3, T-2, and T-1, and make the coordinate system of the historical frame correspond to the correct position of the current frame through translation and rotation. Then, splice the bird's-eye view features of the four moments along the channel direction to obtain a The concatenated tensor of .
[0055] S7: The final bird's-eye view features are input into the bird's-eye view feature encoder and connected to a two-stage three-dimensional object detection head CenterPoint network to obtain the final object detection results.
[0056] The concatenated tensor is fed into the bird’s-eye view feature encoder, which contains several residual modules. Each residual module consists of The convolution layer and ReLU activation function are used to obtain the final bird's-eye view features with a scale of , and fed into a two-stage 3D object detection head network to obtain the final object detection result.
[0057] According to the present invention, there is also provided a visual three-dimensional target detection device based on normal distribution weighted feature mapping, comprising:
[0058] Feature extraction module, used to The surround view camera image is input into the backbone network for feature extraction to obtain multi-scale two-dimensional feature maps. is the number of surround view cameras;
[0059] The depth prediction module is used to use a pre-trained depth estimation network to predict the depth of the two-dimensional feature map. A depth map;
[0060] A calculation module is used to pre-calculate the mapping relationship between each voxel in the three-dimensional space and the corresponding two-dimensional feature point in the two-dimensional feature map based on the internal and external parameters of the camera, and establish a fast lookup table based on this mapping relationship;
[0061] The back-projection module is used to construct a weight function for each 2D feature point based on its depth value, assign different weights to the 3D voxels corresponding to the same 2D feature point according to their distance, and project the 2D features into 3D space to obtain a 4D voxel tensor;
[0062] Compression module, used to compress the 4D voxel tensor using the space-to-channel (S2C) operator to obtain the bird's-eye view features;
[0063] The concatenation module is used to spatially align the bird's-eye view features of several historical frames with the current frame, and then concatenate and fuse them to obtain the temporally fused bird's-eye view features.
[0064] The result output module is used to input the final bird's-eye view features into the bird's-eye view feature encoder and connect a two-stage three-dimensional target detection head CenterPoint network to obtain the final target detection results.
[0065] A computing device is also provided, comprising: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device executes the visual three-dimensional target detection method based on normal distribution weighted feature mapping as described above.
[0066] A readable storage medium storing program instructions, when the program instructions are read and executed by a computing device, enables the computing device to execute the visual three-dimensional target detection method based on normal distribution weighted feature mapping as described above.
[0067] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A visual 3D target detection method based on normal distribution weighted feature mapping, characterized in that: The following steps are involved: S1: The surround view camera image is input into the backbone network for feature extraction to obtain multi-scale two-dimensional feature maps. is the number of surround view cameras; S2: Use a pre-trained depth estimation network to predict the depth of the two-dimensional feature map and obtain A depth map; S3: Based on the internal and external parameters of the camera, pre-calculate the mapping relationship between each voxel in the three-dimensional space and the corresponding two-dimensional feature point in the two-dimensional feature map, and establish a fast lookup table based on this mapping relationship; S4: For each 2D feature point, a weight function is constructed based on its depth value. Different weights are assigned to the 3D voxels corresponding to the same 2D feature point based on their distance. The 2D feature is projected into 3D space to obtain a 4D voxel tensor. S5: Use the space-to-channel (S2C) operator to compress the dimension of the 4D voxel tensor on the z-axis to obtain the bird's-eye view feature; S6: The bird's-eye view features of several historical frames are spatially aligned, concatenated and fused with the current frame to obtain the temporally fused bird's-eye view features; S7: The final bird's-eye view features are input into the bird's-eye view feature encoder and connected to a two-stage three-dimensional object detection head CenterPoint network to obtain the final object detection result.
2. The method for visual three-dimensional target detection based on normal distribution weighted feature mapping according to claim 1, characterized in that: In S1, the backbone network includes the feature extraction network ResNet-50 and the feature pyramid network. The ResNet-50 network is used to obtain the four-stage features. In the output part, a three-layer multi-scale feature pyramid network structure is used for feature fusion, and finally three layers of two-dimensional feature maps of different scales are obtained. .
3. The method for visual three-dimensional target detection based on normal distribution weighted feature mapping according to claim 1, characterized in that: The depth estimation network adopts an encoder-decoder structure and outputs a depth map with the same size as the input image. The depth estimation network is fine-tuned and trained using the generated autonomous driving scene dataset combined with the real autonomous driving scene dataset.
4. The method for visual three-dimensional target detection based on normal distribution weighted feature mapping according to claim 1, wherein S3 include: Define a three-dimensional space area around the vehicle, divided into two parts of size A voxel grid where are the number of voxels in the vertical and horizontal directions on the feature space plane of the bird’s-eye view, is the number of voxels in the height direction; The projection matrix from the world coordinate system to the image coordinate system is obtained through the camera's internal and external parameter matrices, and the dimension is , where 3 corresponds to the homogeneous coordinates of the two-dimensional image plane , 4 corresponds to the homogeneous coordinates in three-dimensional space ; According to the projection matrix, the mapping relationship between the above voxel grid and the corresponding two-dimensional feature points in the two-dimensional feature map is obtained, and the mapping relationship between the two is saved as a quick lookup table. The two-dimensional pixel coordinates corresponding to the voxels in the three-dimensional space are calculated according to the mapping relationship, and the two-dimensional pixel coordinates corresponding to each voxel in the three-dimensional space are recorded in the quick lookup table.
5. The method for visual three-dimensional target detection based on normal distribution weighted feature mapping according to claim 1, wherein S4 include: S4-1: For mapping to the same two-dimensional feature All three-dimensional voxels are obtained from the fast lookup table to obtain their corresponding two-dimensional pixel coordinates , obtain the two-dimensional feature map through the two-dimensional pixel coordinates And the corresponding depth on the depth map ; S4-2: Construct a normal distribution weight function with a mean is the depth value corresponding to the feature point , the standard deviation of the weight function Dynamically adjusted according to application requirements, the weight function of the normal distribution is expressed as: ,in, is the distance from the voxel to the origin of the ego vehicle; S4-3: Calculate the coordinates Voxels To the origin of the vehicle Euclidean distance : ； S4-4: Sampling the sampling point on the weight function to obtain the weight corresponding to each voxel ; S4-5: For 3D voxels , multiply the two-dimensional feature map by the corresponding weight , and obtain three-dimensional voxels The feature representation of: ； Calculate the feature representation corresponding to all three-dimensional voxels to obtain a complete three-dimensional voxel feature representation with a scale of A 4D voxel tensor.
6. The method for visual three-dimensional target detection based on normal distribution weighted feature mapping according to claim 1, wherein S6 include: Perform temporal fusion operation on the bird's-eye view features at the current time T, align the bird's-eye view features of the three historical frames, namely the bird's-eye view features at time T-3, T-2, and T-1, and make the coordinate system of the historical frame correspond to the correct position of the current frame through translation and rotation. Then, splice the bird's-eye view features of the four moments along the channel direction to obtain a The concatenated tensor of .
7. The method for visual three-dimensional target detection based on normal distribution weighted feature mapping according to claim 6, characterized in that: S7 includes: The concatenated tensor is input into the bird's-eye view feature encoder for fusion. The bird's-eye view feature encoder contains several residual modules, each of which consists of The convolutional layer and ReLU activation function are used to obtain the final bird's-eye view features, which are then fed into a two-stage 3D target detection head network to obtain the final target detection results.
8. A visual three-dimensional target detection device based on normal distribution weighted feature mapping, characterized in that: include: Feature extraction module, used to The surround view camera image is input into the backbone network for feature extraction to obtain multi-scale two-dimensional feature maps. is the number of surround view cameras; The depth prediction module is used to use a pre-trained depth estimation network to predict the depth of the two-dimensional feature map. A depth map; A calculation module is used to pre-calculate the mapping relationship between each voxel in the three-dimensional space and the corresponding two-dimensional feature point in the two-dimensional feature map based on the internal and external parameters of the camera, and establish a fast lookup table based on this mapping relationship; The back-projection module is used to construct a weight function for each 2D feature point based on its depth value, assign different weights to the 3D voxels corresponding to the same 2D feature point according to their distance, and project the 2D features into 3D space to obtain a 4D voxel tensor; Compression module, which compresses the 4D voxel tensor using the space-to-channel (S2C) operator to obtain bird's-eye view features; The concatenation module is used to spatially align the bird's-eye view features of several historical frames with the current frame, and then concatenate and fuse them to obtain the temporally fused bird's-eye view features. The result output module is used to input the final bird's-eye view features into the bird's-eye view feature encoder and connect a two-stage three-dimensional target detection head CenterPoint network to obtain the final target detection results.
9. A computing device, characterized in that include: at least one processor and memory storing program instructions; When the program instructions are read and executed by the processor, the computing device executes the visual three-dimensional target detection method based on normal distribution weighted feature mapping according to any one of claims 1 to 7.
10. A readable storage medium storing program instructions, characterized in that: When the program instructions are read and executed by a computing device, the computing device is caused to execute the visual three-dimensional target detection method based on normal distribution weighted feature mapping as described in any one of claims 1 to 7.