A 3D target detection method based on a basic model and point cloud image fusion

Abstract

The application discloses a kind of 3D target detection methods of point cloud image fusion based on basic model, applied to the field of automatic driving technology, comprising: inputting multi-view image into basic model, obtain image depth map;Image depth map and radar point cloud are input into semantic projection model, obtain radar depth map;Image depth map and radar depth map are input into filtering module, obtain enhanced radar point cloud;Respectively obtain the multi-scale image BEV feature and multi-scale radar BEV feature corresponding to multi-view image and enhanced radar point cloud, and input into multi-scale feature fusion module, obtain multi-scale fusion feature;Fusion feature is input into detection head, obtain detection result.The application realizes the densification of sparse radar point cloud by basic model, and realizes the complete fusion of multi-scale and multi-modal BEV feature by multi-scale feature fusion module, improves the accuracy and robustness of target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a 3D target detection method based on point cloud image fusion of a basic model. BACKGROUND

[0002] In recent years, 3D target detection has attracted great attention in the field of automatic driving, which aims to detect the position, size and category of objects in a 3D scene. Based on a single sensor, such as a camera-based 3D target detection method, is susceptible to environmental influences, resulting in limited performance. Therefore, a single detection method has limitations. Therefore, a multi-sensor fusion 3D target detection method has appeared later, which usually combines camera and radar technology, but since the radar points collected by the radar technology are sparse, the multi-sensor fusion 3D target detection method still has deficiencies in detection robustness and accuracy.

[0003] Therefore, how to improve the accuracy and robustness of the 3D target detection method based on point cloud image fusion is a problem that needs to be solved at present. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a 3D target detection method based on point cloud image fusion of a basic model, which solves the problem of insufficient accuracy and lack of robustness of the 3D target detection method based on point cloud image fusion in the prior art.

[0005] To solve the above technical problems, the present application provides a 3D target detection method based on point cloud image fusion of a basic model, comprising:

[0006] inputting a multi-view image into a basic model to obtain an image depth map;

[0007] inputting the image depth map and radar point cloud into a semantic projection model to obtain a radar depth map;

[0008] inputting the image depth map and the radar depth map into a filtering module to obtain enhanced radar point cloud;

[0009] respectively acquiring multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and inputting them into a multi-scale feature fusion module to obtain multi-scale fusion features;

[0010] inputting the fusion features into a detection head to obtain a detection result.

[0011] Optionally, it further comprises:

[0012] inputting the multi-view image into the basic model to predict the probability of each pixel at each predefined depth interval;

[0013] a standard deviation is calculated according to the probability corresponding to each pixel;

[0014] Correspondingly, the image depth map and the radar depth map are input into a filtering module to obtain an enhanced radar point cloud, including:

[0015] The image depth map, the radar depth map and the standard deviation are input into the filtering module to obtain the enhanced radar point cloud.

[0016] Optionally, the image depth map, the radar depth map and the standard deviation are input into a filtering module to obtain the enhanced radar point cloud, including:

[0017] ;

[0018] wherein, represents the enhanced radar point cloud at position c; represents the depth value of position c in the radar depth map; represents the depth value of position c in the image depth map; represents the standard deviation of position c; is a threshold value of the standard deviation and is a hyperparameter; represents the radar depth map; represents the image depth map.

[0019] Optionally, the image depth map and the radar point cloud are input into a semantic projection model to obtain a radar depth map, including:

[0020] The background part in the projection area is filtered using the depth value in the image depth map to obtain a target projection area;

[0021] The radar point cloud is projected onto the target projection area in the form of an image block to obtain the radar depth map; the depth values in the image block are the same.

[0022] Optionally, the multi-scale image BEV feature and the multi-scale radar BEV feature are input into a multi-scale feature fusion module to obtain a multi-scale fusion feature, including:

[0023] The multi-scale query vector and the multi-scale offset vector corresponding to the multi-scale radar BEV feature are obtained using the WQ linear layer and the WO linear layer of the multi-scale feature fusion module, respectively;

[0024] The multi-scale key vector and the multi-scale value vector corresponding to the multi-scale image BEV feature are obtained using the WK linear layer and the WV linear layer of the multi-scale feature fusion module, respectively;

[0025] The multi-scale query vector, the multi-scale offset vector, the multi-scale key vector, and the multi-scale value vector are fused to obtain the fused feature.

[0026] Optionally, after inputting the image depth map and the radar depth map into the filtering module to obtain the enhanced radar point cloud, the method further includes:

[0027] The enhanced radar point cloud is used to update the radar point cloud, and the number of updates is recorded.

[0028] The enhanced radar point cloud is updated using the updated radar point cloud and the image depth map. The update stops when the number of updates reaches a preset number.

[0029] Optionally, the base model is the Depth Anything model.

[0030] The present invention also provides a 3D target detection device based on point cloud image fusion using a base model, comprising:

[0031] The image depth map acquisition module is used to input multi-view images into the base model to obtain image depth maps;

[0032] The radar depth map acquisition module is used to input the image depth map and radar point cloud into the semantic projection model to obtain the radar depth map;

[0033] The radar point cloud enhancement module is used to input the image depth map and the radar depth map into the filtering module to obtain the enhanced radar point cloud;

[0034] The fusion module is used to acquire the multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, respectively, and input them into the multi-scale feature fusion module to obtain multi-scale fused features.

[0035] The detection module is used to input the fused features into the detection head to obtain the detection results.

[0036] This invention also provides a 3D target detection device based on point cloud image fusion using a base model, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute the computer program to implement the 3D target detection method based on point cloud image fusion of the underlying model as described above.

[0039] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the 3D target detection method based on point cloud image fusion of the basic model as described above.

[0040] As can be seen, this invention obtains an image depth map by inputting multi-view images into a base model; it then inputs the image depth map and radar point cloud into a semantic projection model to obtain a radar depth map; finally, it inputs the image depth map and radar depth map into a filtering module to obtain an enhanced radar point cloud; it acquires multi-scale image BEV (Bird's Eye View) features and multi-scale radar BEV features corresponding to the multi-view images and enhanced radar point cloud, respectively, and inputs them into a multi-scale feature fusion module to obtain multi-scale fused features; finally, it inputs the fused features into a detection head to obtain the detection result. This invention utilizes the base model to densify sparse radar point clouds, enabling the radar point cloud to provide stronger and more reliable distance information when further fused with multi-view images, while the filtering module can also reduce additional noise; to better utilize the densified radar point cloud, the multi-scale feature fusion module adaptively interacts with multi-scale image and radar features in the BEV space, achieving thorough fusion of multi-scale and multi-modal BEV features, thus improving the accuracy and robustness of target detection.

[0041] In addition, the present invention also provides a 3D target detection device, equipment and computer-readable storage medium based on point cloud image fusion of a basic model, which also has the above-mentioned beneficial effects. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a 3D target detection method based on point cloud image fusion using a base model, provided in an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of the framework of a semantic projection module provided in an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of the architecture of a multi-scale feature fusion module provided in an embodiment of the present invention;

[0046] Figure 4A schematic diagram of a 3D target detection framework based on point cloud image fusion according to an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the structure of a 3D target detection device based on point cloud image fusion according to an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the structure of a 3D target detection device based on point cloud image fusion according to an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In recent years, 3D object detection has received considerable attention in the field of autonomous driving. It aims to predict the position, size, and category of objects in 3D scenes. Cameras can provide high-resolution images containing rich color and texture information, which can be used for object recognition and classification. However, camera performance is limited under adverse weather, low light, or strong light conditions, and complex image processing algorithms are required to obtain target distance information. Radar can provide high-precision distance information; therefore, combining radar distance and velocity measurements with the rich appearance and texture details in camera images can provide more comprehensive and reliable perception for 3D object detection.

[0051] Existing optimization methods for 3D object detection using point cloud image fusion can be broadly categorized into two types. One type focuses on improving the representativeness of single-mode radar by designing more advanced radar feature extractors. For example, RCFusion (an open-source software library for medical image registration) emphasizes the differences in radar features across different regions by decoding the projection layer. The other type leverages cross-modal correlations between camera and radar signals, filtering noise within the original radar points by referencing semantically meaningful camera images. For instance, HVDetFusion (a multimodal detection algorithm) can explicitly suppress radar noise by using proposals (candidate regions that may contain the target object in object detection). Neither of these methods fundamentally addresses the sparsity problem of radar data; therefore, how to compact sparse radar data remains an unsolved challenge. Furthermore, existing methods only perform single-scale fusion of multimodal data, lacking utilization of the multi-granularity information within the multimodal data. Consequently, they still fall short in terms of detection robustness and accuracy.

[0052] To address the aforementioned problems, this invention provides a 3D target detection method based on point cloud image fusion using a base model. This method leverages general scene understanding knowledge embedded in the base model to unlock the potential of radar measurements before fusion with multi-view images. Please refer to [link / reference] for details. Figure 1 , Figure 1 This is a flowchart illustrating a 3D object detection method based on point cloud image fusion using a base model, provided as an embodiment of the present invention. The method may include:

[0053] S101: Input multi-view images into the base model to obtain image depth maps.

[0054] The execution subject in this embodiment is a terminal. This embodiment does not limit the type of terminal, as long as it can perform the operation of the 3D target detection method based on point cloud image fusion using a base model. The multi-view images in this embodiment can be camera images captured by a camera, which can provide high-resolution images containing rich color and texture information. This embodiment does not limit the base model; the base model in this embodiment is a general, large-scale artificial intelligence model. The base model has witnessed a large number of training images and understands the general distribution of real-world images. Therefore, the basic knowledge learned by the base model is general and can be effectively transferred to various downstream tasks. This embodiment can leverage the general scene understanding knowledge embedded in the base model to unlock the potential of radar measurement before fusing with multi-view images.

[0055] Furthermore, the base model in this embodiment is the Depth Anything model. This model is an advanced monocular depth estimation technique that can extract depth information from 2D images and convert them into 3D images, achieving significant progress in improving depth map quality and enabling it to estimate scene depth information more accurately. The Depth Anything model expands the dataset by collecting and automatically annotating a large amount of unlabeled data, improving the model's generalization ability. It also uses auxiliary supervision techniques, forcing the model to inherit rich semantic priors from the pre-trained encoder, thereby improving the accuracy of depth estimation. In addition, this model provides pre-trained models of different sizes, which can be easily integrated into other deep learning frameworks to provide depth information for various applications. This embodiment adopts the encoder-decoder architecture in the Depth Anything model and adds a depth metric head that follows ZoeDepth (the name of a deep learning model or framework used to handle depth estimation-related tasks). During model training, the learning rate allocated to the backbone and the depth metric head is adjusted to adapt the model to the depth distribution of the scene while preserving the powerful content understanding capabilities of the base model as much as possible.

[0056] S102: The image depth map and radar point cloud are input into the semantic projection model to obtain the radar depth map.

[0057] This embodiment does not limit the method of acquiring radar point clouds. For example, radar point clouds can be acquired using lidar; or they can be acquired using millimeter-wave radar. Considering the limited resolution of lidar, millimeter-wave radar can provide distance, velocity, and direction information and is less affected by the external environment. Therefore, this embodiment can use millimeter-wave radar to acquire radar point clouds. In this embodiment, the acquired radar point cloud and the image depth map obtained in step S101 are input into the semantic projection model to obtain the radar depth map.

[0058] Furthermore, the process of inputting the image depth map and radar point cloud into the semantic projection model to obtain the radar depth map can specifically include the following steps:

[0059] Step 21: Use the depth values ​​in the image depth map to filter the background in the projection area to obtain the target projection area.

[0060] Specifically, this embodiment considers a problem with direct projection: radar points initially illuminating the object's outline edge will be projected onto an area simultaneously covering both the foreground and background. The portion projected onto the background introduces additional noise to the network because semantic projection models often confuse the depth of these background points with the depth of foreground objects during training. Therefore, this embodiment leverages the powerful semantic understanding capabilities of the base model, using the depth values ​​of the generated image depth map to distinguish between foreground and background, thereby filtering out the background portion in the projected area, such as... Figure 2 As shown, each pixel in the image depth map records the distance from the viewpoint to the surface of the corresponding scene object.

[0061] Step 22: Project the radar point cloud onto the target projection area in the form of image patches to obtain a radar depth map; the depth values ​​in the image patches are the same.

[0062] Traditionally, a one-to-one correspondence exists between radar points and pixels in a radar point cloud, meaning that each radar point naturally matches a unique pixel in its projection. This one-to-one correspondence implies that when radar points are projected onto other radar points to provide depth information, only a few pixels have the corresponding radar point and depth information. However, these sparse depth values ​​from the radar may not provide enough information for the underlying model to accurately learn depth information. Therefore, this embodiment projects radar points onto patches on the image instead of individual pixels. The area of ​​the patch is a square region whose side length is related to the radar cross-section (RCS) value, as shown in the formula:

[0063]

[0064] in, Represents the radar RCS value, and γ represents the hyperparameter. This represents half the side length of the square expansion block centered on the projection point. In this embodiment, all pixels within the square area centered on the original projection point have a side length of 2a and share the depth information of that radar point.

[0065] S103: Input the image depth map and radar depth map into the filtering module to obtain the enhanced radar point cloud.

[0066] Step S101 obtains the image depth map based on the basic model, and step S102 obtains the radar depth map based on the semantic projection model and the image depth map, thus expanding the radar points. To improve the reliability of the radar points, the image depth map and the radar depth map are input into the filtering module to reliably select depth points, thereby increasing the number of reliable radar points compared to the original radar point cloud.

[0067] S104: Obtain the multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, respectively, and input them into the multi-scale feature fusion module to obtain the multi-scale fused features.

[0068] Multi-scale image BEV features are obtained based on multi-view images, and multi-scale radar BEV features are obtained based on the enhanced radar point cloud. This embodiment does not limit the feature extraction process; specific feature extraction methods can be referred to in existing technologies. For example, multi-view images can be converted into image BEV features through perspective transformation or depth estimation. Multi-scale radar BEV features are then extracted directly from the enhanced radar point cloud data using a deep learning model (such as PointPillars, a deep learning model for 3D object detection). The deep learning model typically includes an encoder network and a decoder network. The encoder is responsible for converting the point cloud data into intermediate feature representations, while the decoder maps these features to the BEV space to generate the final radar BEV feature map.

[0069] Existing methods fuse multi-scale features into a single-scale feature during feature extraction for each modality. In the cross-modal fusion stage, unified-scale BEV features from both modalities are further integrated. However, this invention considers that low-level features capture fine-grained details, while high-level semantic features provide cohesive contextual interpretation. Early fusion limits the diversity of semantic granularity between modalities and hinders appropriate interaction between different levels of semantic granularity. Therefore, this embodiment does not merge them prematurely but retains multi-scale semantic information in the cross-modal fusion stage, utilizing a multi-scale feature fusion module (MFF) to achieve dynamic alignment and thorough interaction between radar BEV features and image BEV features at different scales. In this way, the multi-scale feature fusion module adaptively interacts with multi-resolution images and radar features within the BEV space, integrating information from different semantic levels, enhancing perception capabilities and robustness to radar point noise.

[0070] Furthermore, the above-mentioned input of multi-scale image BEV features and multi-scale radar BEV features into the multi-scale feature fusion module to obtain multi-scale fused features may include the following steps:

[0071] Step 31: Use the WQ linear layer and WO linear layer of the multi-scale feature fusion module to obtain the multi-scale query vector and multi-scale offset vector corresponding to the multi-scale radar BEV features respectively;

[0072] Step 32: Use the WK linear layer and WV linear layer of the multi-scale feature fusion module to obtain the multi-scale key vector and multi-scale value vector corresponding to the BEV features of the multi-scale image respectively;

[0073] Step 33: Fuse the multi-scale query vector, multi-scale offset vector, multi-scale key vector, and multi-scale value vector to obtain the fused feature.

[0074] To thoroughly integrate multi-scale and multi-modal features, this embodiment explicitly retains single-modal multi-scale features for the cross-modal feature fusion stage and designs a corresponding multi-scale feature fusion module. For details, please refer to... Figure 3 The multi-scale feature fusion module consists of four separate linear layers: WQ (Query linear layer), WO (Output linear layer), WK (Key linear layer), and WV (Value linear layer). WQ generates query vectors, WO generates offset vectors, WK generates key vectors, and WV generates value vectors. This embodiment organizes the results into a multi-scale form, followed by an LCM (Leaky Concatenation of Max-pooling, a layer in deep learning models) block to obtain fused features, which are used as input to the detection head. LCM includes LN (Low Normalization), Concat (Connection), and Maxpooling (Max Pooling).

[0075] This embodiment calculates the query vector Q and sampling offset ∆q of the radar BEV feature, as well as the key vector K and value vector V of the image BEV feature at position q, where position q is a two-dimensional reference point Fr = {Frs}3s=1 in the multi-scale radar BEV feature. The output of position q can be calculated as follows:

[0076] ;

[0077] in, express( ; express ; , , , H represents the coefficient matrix; W represents the number of attention heads; H represents the number of attention heads. h The result matrix is ​​represented by S; different feature scales are represented by P; P is the number of sampling points used at each scale. and Representing the BEV features of radar and images at different scales; softmax function. (·) is used to determine attention weights.

[0078] S105: Input the fused features into the detection head to obtain the detection results.

[0079] This step inputs the multimodal BEV representation (i.e., fused features) generated in step S104 into the detection head to generate detection results.

[0080] Furthermore, to address the uncertainties in the depth prediction process, this embodiment may further include the following steps when performing step S101:

[0081] Step 41: Input the multi-view images into the base model to predict the probability of each pixel at each predefined depth interval;

[0082] Step 42: Calculate the standard deviation based on the probability corresponding to each pixel.

[0083] Specifically, this embodiment also utilizes the base model to calculate another metric, σ (standard deviation), to prepare for the filtering module. Specifically, during fine-tuning, the range of depth values ​​is divided into several intervals, each corresponding to a different depth range. The base model, after training, can predict the probability that the depth of each pixel falls within each predefined depth interval, thus enabling more effective learning at different depth levels. For each pixel, there is a corresponding probability p in n depth intervals. The predicted probability σ of the standard deviation of the depth of each pixel within the predefined depth intervals across the entire multi-view image is calculated as follows:

[0084]

[0085] Where C represents the total number of pixels in the multi-view image, N is the number of depth intervals, and p represents the probability of each depth interval.

[0086] Accordingly, step S103 may specifically include the following steps:

[0087] The image depth map, radar depth map, and standard deviation are input into the filtering module to obtain the enhanced radar point cloud.

[0088] For details, please refer to Figure 3 The filtering module receives two inputs: a radar depth map from the semantic projection model and an output from the base model (including an image depth map and standard deviation σ). This embodiment implements cross-validation between the two input depth maps to filter depth information. It's important to note that the fine-tuned depth points (referring to the image depth map) provide comprehensive coverage of the entire image, while the depth points from the semantic projection model (referring to the radar depth map) are local, covering only a specific region. Therefore, this embodiment employs different filtering methods based on whether the two depth maps overlap, as described below:

[0089] ;

[0090] in, This represents the enhanced radar point cloud at position c; This represents the depth value at position c in the radar depth map; This represents the depth value at position c in the image depth map; This represents the standard deviation of position c; It is the threshold of the standard deviation and is a hyperparameter; Represents a radar depth map; This represents the image depth map.

[0091] Furthermore, after inputting the image depth map and radar depth map into the filtering module to obtain the enhanced radar point cloud, the following steps may also be included:

[0092] Step 51: Update the radar point cloud using the enhanced radar point cloud and record the number of updates;

[0093] Step 52: Update the enhanced radar point cloud using the updated radar point cloud and image depth map. Stop updating when the preset number of updates is reached.

[0094] To improve the reliability and accuracy of newly added radar point clouds, this embodiment implements iterative densification. In each densification iteration, radar depth is used as a sparsity guide to extract dense depth knowledge embedded in the base model. By iteratively executing the above steps, the radar depth gradually develops, ultimately providing more robust and reliable range information when further fused with multi-view images. This achieves a "from sparse to dense" transformation of the radar point cloud, converting radar points that are sparser than "instance-level" into dense and accurate "pixel-level" depth maps. After each forward pass, the original radar points are updated in the next iteration using the newly acquired reliable points, i.e., the enhanced radar point cloud is used to update the radar point cloud. The entire process is similar to a snowball effect, accumulating more depth points through an iterative and self-reinforcing module, achieving progressive radar signal densification.

[0095] This invention provides a 3D target detection method based on a base model and point cloud image fusion. The method involves inputting multi-view images into a base model to obtain an image depth map; inputting the image depth map and radar point cloud into a semantic projection model to obtain a radar depth map; inputting the image depth map and radar depth map into a filtering module to obtain an enhanced radar point cloud; acquiring multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view images and enhanced radar point cloud, respectively, and inputting them into a multi-scale feature fusion module to obtain multi-scale fused features; and inputting the fused features into a detection head to obtain the detection result. This invention utilizes a base model to densify sparse radar point clouds, enabling the radar point cloud to provide stronger and more reliable distance information when further fused with multi-view images. Simultaneously, the filtering module reduces additional noise. To better utilize the densified radar point cloud, the multi-scale feature fusion module adaptively interacts with multi-scale image and radar features in the BEV space, achieving thorough fusion of multi-scale and multi-modal BEV features, thus improving the accuracy and robustness of target detection. Furthermore, a progressive, cyclical approach is used to enhance sparse radar points, which benefit from the base model while reducing its additional noise.

[0096] For a clearer understanding of this invention, please refer to the following details. Figure 4 , Figure 4 The schematic diagram of a 3D target detection framework based on a base model for point cloud image fusion provided in this embodiment of the invention specifically includes: (1) a progressive radar densification module and (2) a multi-scale multimodal detection module. The progressive radar densification module is used to transform the radar point cloud "from sparse to dense", converting radar points that are sparser than "instance level" into dense and accurate "pixel level" depth maps. The multi-scale multimodal detection module is used to thoroughly fuse multi-scale and multimodal features, explicitly retaining the multi-scale features of single-modality features for cross-modal feature fusion. The multi-scale feature fusion module of this method considers the scale hierarchy of features and retains multi-scale information starting from the characteristics at the sample level. At the same time, in terms of multimodal fusion, a powerful base model is used to enhance the radar signal. In particular, sparse radar depth is used as a guiding prompt, and mature scene understanding knowledge is derived from the base model for improvement.

[0097] The following describes the 3D target detection device based on point cloud image fusion based on a basic model provided in the embodiments of the present invention. The 3D target detection device based on point cloud image fusion based on a basic model described below can be referred to in correspondence with the 3D target detection method based on point cloud image fusion based on a basic model described above.

[0098] Please refer to the details. Figure 5 , Figure 5A schematic diagram of a 3D target detection device based on point cloud image fusion using a base model, provided in an embodiment of the present invention, may include:

[0099] The image depth map acquisition module 100 is used to input multi-view images into the base model to obtain image depth maps;

[0100] The radar depth map acquisition module 200 is used to input the image depth map and radar point cloud into the semantic projection model to obtain the radar depth map;

[0101] The radar point cloud enhancement module 300 is used to input the image depth map and the radar depth map into the filtering module to obtain the enhanced radar point cloud;

[0102] The fusion module 400 is used to acquire the multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, respectively, and input them into the multi-scale feature fusion module to obtain multi-scale fused features.

[0103] The detection module 500 is used to input the fused features into the detection head to obtain the detection result.

[0104] Based on the above embodiments, the 3D target detection device based on point cloud image fusion of the basic model may further include:

[0105] The probability prediction module is used to input the multi-view image into the base model and predict the probability of each pixel at each predefined depth interval;

[0106] The standard deviation calculation module is used to calculate the standard deviation based on the probability corresponding to each pixel.

[0107] Accordingly, the radar point cloud enhancement module 300 may include:

[0108] The radar point cloud enhancement unit is used to input the image depth map, the radar depth map, and the standard deviation into the filtering module to obtain the enhanced radar point cloud.

[0109] Based on the above embodiments, the radar point cloud enhancement module 300 may specifically include:

[0110] ;

[0111] in, This represents the enhanced radar point cloud at position c; This represents the depth value at position c in the radar depth map; This represents the depth value at position c in the image depth map; This represents the standard deviation of position c; It is the threshold of the standard deviation and is a hyperparameter; Represents a radar depth map; This represents the image depth map.

[0112] Based on the above embodiments, the radar depth map acquisition module 200 may include:

[0113] The filtering unit is used to filter the background portion in the projection area using the depth value in the image depth map to obtain the target projection area;

[0114] The projection unit is used to project the radar point cloud onto the target projection area in the form of image blocks to obtain the radar depth map; the depth values ​​in the image blocks are the same.

[0115] Based on the above embodiments, the fusion module 400 may include:

[0116] The first unit is used to obtain the multi-scale query vector and multi-scale offset vector corresponding to the multi-scale radar BEV feature by utilizing the WQ linear layer and WO linear layer of the multi-scale feature fusion module, respectively.

[0117] The second unit is used to obtain the multi-scale key vector and multi-scale value vector corresponding to the BEV feature of the multi-scale image by utilizing the WK linear layer and WV linear layer of the multi-scale feature fusion module, respectively.

[0118] The third unit is used to fuse the multi-scale query vector, the multi-scale offset vector, the multi-scale key vector, and the multi-scale value vector to obtain the fused feature.

[0119] Based on any of the above embodiments, the 3D target detection device based on point cloud image fusion of the base model may further include:

[0120] A radar point cloud update module is used to update the radar point cloud using the enhanced radar point cloud and record the number of updates.

[0121] The enhanced radar point cloud update module is used to update the enhanced radar point cloud using the updated radar point cloud and the image depth map. When the number of updates reaches a preset number, the update stops.

[0122] Based on the above embodiments, the underlying model can specifically be the Depth Anything model.

[0123] It should be noted that the order of the modules and units in the above-mentioned 3D target detection device based on point cloud image fusion of the basic model can be changed without affecting the logic.

[0124] The 3D target detection device based on point cloud image fusion using the embodiment of the present invention comprises the following modules: an image depth map acquisition module 100, which inputs multi-view images into a base model to obtain an image depth map; a radar depth map acquisition module 200, which inputs the image depth map and the radar point cloud into a semantic projection model to obtain a radar depth map; a radar point cloud enhancement module 300, which inputs the image depth map and the radar depth map into a filtering module to obtain an enhanced radar point cloud; a fusion module 400, which acquires the multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view images and the enhanced radar point cloud, respectively, and inputs them into a multi-scale feature fusion module to obtain multi-scale fused features; and a detection module 500, which inputs the fused features into a detection head to obtain a detection result. This invention utilizes a basic model to densify sparse radar point clouds, enabling the radar point clouds to provide more powerful and reliable distance information when further fused with multi-view images. At the same time, the filtering module can reduce additional noise. To better utilize the densified radar point clouds, the multi-scale feature fusion module adaptively interacts with multi-scale image and radar features in the BEV space, achieving thorough fusion of multi-scale and multi-modal BEV features, thereby improving the accuracy and robustness of target detection.

[0125] The following describes the 3D target detection device based on point cloud image fusion based on a basic model provided in the embodiments of the present invention. The 3D target detection device based on point cloud image fusion based on a basic model described below can be referred to in correspondence with the 3D target detection method based on point cloud image fusion based on a basic model described above.

[0126] Please refer to Figure 6 , Figure 6 A schematic diagram of a 3D target detection device based on point cloud image fusion using a base model, provided in an embodiment of the present invention, may include:

[0127] Memory 10 is used to store computer programs;

[0128] The processor 20 is used to execute computer programs to implement the above-described 3D target detection method based on point cloud image fusion of the basic model.

[0129] The memory 10, processor 20, and communication interface 31 all communicate with each other through the communication bus 32.

[0130] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions:

[0131] By inputting multi-view images into the base model, an image depth map is obtained;

[0132] The image depth map and radar point cloud are input into the semantic projection model to obtain the radar depth map;

[0133] The image depth map and radar depth map are input into the filtering module to obtain the enhanced radar point cloud;

[0134] The multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud are obtained respectively, and then input into the multi-scale feature fusion module to obtain the multi-scale fused features.

[0135] The fused features are input into the detection head to obtain the detection results.

[0136] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0137] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0138] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0139] Communication interface 31 can be an interface for the communication module, used to connect with other devices or systems.

[0140] Of course, it should be noted that, Figure 6 The structure shown does not constitute a limitation on the 3D target detection device based on point cloud image fusion of the base model in the embodiments of the present invention. In practical applications, the 3D target detection device based on point cloud image fusion of the base model can include more than Figure 6 More or fewer components as shown, or combinations of certain components.

[0141] The readable storage medium provided in the embodiments of the present invention is described below. The readable storage medium described below can be referred to in correspondence with the 3D target detection method based on point cloud image fusion of the basic model described above.

[0142] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described 3D target detection method based on point cloud image fusion of a base model.

[0143] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0145] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0146] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0147] The foregoing has provided a detailed description of a 3D target detection method, apparatus, device, and computer-readable storage medium based on point cloud image fusion using a fundamental model, as provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A 3D target detection method based on point cloud image fusion using a fundamental model, characterized in that, include: Multi-view images are input into the base model to obtain image depth maps; The image depth map and radar point cloud are input into the semantic projection model to obtain the radar depth map; The image depth map and the radar depth map are input into the filtering module to obtain the enhanced radar point cloud; The multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud are obtained respectively, and input into the multi-scale feature fusion module to obtain multi-scale fused features. The fused features are input into the detection head to obtain the detection result; Also includes: The multi-view image is input into the base model to predict the probability of each pixel at each predefined depth interval; The standard deviation is calculated based on the probability corresponding to each pixel. Accordingly, the step of inputting the image depth map and the radar depth map into the filtering module to obtain the enhanced radar point cloud includes: The image depth map, the radar depth map, and the standard deviation are input into the filtering module to obtain the enhanced radar point cloud.

2. The 3D target detection method based on point cloud image fusion according to claim 1, characterized in that, The image depth map, the radar depth map, and the standard deviation are input into the filtering module to obtain the enhanced radar point cloud, including: ； in, This represents the enhanced radar point cloud at position c; This represents the depth value at position c in the radar depth map; This represents the depth value at position c in the image depth map; This represents the standard deviation of position c; It is the threshold of the standard deviation and is a hyperparameter; Represents a radar depth map; This represents the image depth map.

3. The 3D target detection method based on point cloud image fusion according to claim 1, characterized in that, The image depth map and radar point cloud are input into the semantic projection model to obtain the radar depth map, including: The background portion in the projection area is filtered out using the depth values ​​in the image depth map to obtain the target projection area; The radar point cloud is projected onto the target projection area in the form of image patches to obtain the radar depth map; the depth values ​​in the image patches are the same.

4. The 3D target detection method based on point cloud image fusion according to claim 1, characterized in that, Multi-scale image BEV features and multi-scale radar BEV features are input into a multi-scale feature fusion module to obtain multi-scale fused features, including: The multi-scale query vector and multi-scale offset vector corresponding to the multi-scale radar BEV feature are obtained by using the WQ linear layer and WO linear layer of the multi-scale feature fusion module, respectively. The multi-scale key vector and multi-scale value vector corresponding to the BEV features of the multi-scale image are obtained by using the WK linear layer and WV linear layer of the multi-scale feature fusion module, respectively. The multi-scale query vector, the multi-scale offset vector, the multi-scale key vector, and the multi-scale value vector are fused to obtain the fused feature.

5. The 3D target detection method based on point cloud image fusion according to any one of claims 1 to 4, characterized in that, After inputting the image depth map and the radar depth map into the filtering module to obtain the enhanced radar point cloud, the process further includes: The enhanced radar point cloud is used to update the radar point cloud, and the number of updates is recorded. The enhanced radar point cloud is updated using the updated radar point cloud and the image depth map. The update stops when the number of updates reaches a preset number.

6. The 3D target detection method based on point cloud image fusion according to claim 1, characterized in that, The underlying model is the Depth Anything model.

7. A 3D target detection device based on point cloud image fusion using a fundamental model, characterized in that, include: The image depth map acquisition module is used to input multi-view images into the base model to obtain image depth maps; The radar depth map acquisition module is used to input the image depth map and radar point cloud into the semantic projection model to obtain the radar depth map; The radar point cloud enhancement module is used to input the image depth map and the radar depth map into the filtering module to obtain the enhanced radar point cloud; The fusion module is used to acquire the multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, respectively, and input them into the multi-scale feature fusion module to obtain multi-scale fused features. The detection module is used to input the fused features into the detection head to obtain the detection result; Also includes: The probability prediction module is used to input the multi-view image into the base model and predict the probability of each pixel at each predefined depth interval; The standard deviation calculation module is used to calculate the standard deviation based on the probability corresponding to each pixel. Accordingly, the radar point cloud enhancement module includes: The radar point cloud enhancement unit is used to input the image depth map, the radar depth map, and the standard deviation into the filtering module to obtain the enhanced radar point cloud.

8. A 3D target detection device based on point cloud image fusion using a fundamental model, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the 3D target detection method based on point cloud image fusion according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the 3D target detection method based on point cloud image fusion according to any one of claims 1 to 6.

Images