3D target detection method based on point cloud image fusion of basic model

Abstract

The invention discloses a 3D target detection method for point cloud image fusion based on a basic model, which is applied to the technical field of automatic driving and comprises the following steps: inputting a multi-view image into the basic model to obtain an image depth map; inputting the image depth map and the radar point cloud into a semantic projection model to obtain a radar depth map; inputting the image depth map and the radar depth map into a filtering module to obtain an enhanced radar point cloud; obtaining multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and inputting the multi-scale image BEV features and the multi-scale radar BEV features into a multi-scale feature fusion module to obtain multi-scale fusion features; and inputting the fusion feature into a detection head to obtain a detection result. The sparse radar point cloud densification is realized by using the basic model, the multi-scale and multi-mode BEV features are thoroughly fused by using the multi-scale feature fusion module, and the accuracy and robustness of target detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a 3D target detection method based on point cloud image fusion of a basic model. Background Art

[0002] In recent years, 3D (three-dimensional) object detection has received great attention in the field of autonomous driving. It aims to detect the position, size and category of objects in 3D scenes. 3D object detection methods based on a single sensor, such as camera-based methods, are easily affected by the environment, resulting in limited performance. Therefore, a single detection method has limitations. Therefore, a multi-sensor fusion 3D object detection method has emerged later, which usually combines camera and radar technology. However, due to the sparseness of radar points collected by radar technology, the multi-sensor fusion 3D object detection method still has deficiencies in detection robustness and accuracy.

[0003] Therefore, how to improve the accuracy and robustness of 3D object detection methods based on point cloud image fusion is an urgent problem that needs to be solved. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a 3D target detection method based on point cloud image fusion of a basic model, which solves the problems of inaccuracy and lack of robustness of the 3D target detection method based on point cloud image fusion in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a 3D target detection method based on point cloud image fusion of a basic model, comprising:

[0006] Input the multi-view image into the basic model to obtain the image depth map;

[0007] Inputting the image depth map and the radar point cloud into a semantic projection model to obtain a radar depth map;

[0008] Inputting the image depth map and the radar depth map into a filtering module to obtain an enhanced radar point cloud;

[0009] Respectively obtaining multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and inputting them into a multi-scale feature fusion module to obtain multi-scale fusion features;

[0010] The fused features are input into the detection head to obtain the detection result.

[0011] Optionally, also include:

[0012] Inputting the multi-view image into the basic model, and predicting the probability of each pixel at each predefined depth interval;

[0013] The standard deviation is obtained by calculating the probability corresponding to each pixel;

[0014] Accordingly, the image depth map and the radar depth map are input into a filtering module to obtain an enhanced radar point cloud, including:

[0015] The image depth map, the radar depth map and the standard deviation are input into the filtering module to obtain the enhanced radar point cloud.

[0016] Optionally, inputting the image depth map, the radar depth map and the standard deviation into a filtering module to obtain the enhanced radar point cloud includes:

[0017] ;

[0018] in, represents the enhanced radar point cloud at position c; represents the depth value of position c in the radar depth map; represents the depth value of position c in the image depth map; represents the standard deviation of position c; is the threshold of the standard deviation and is a hyperparameter; Represents a radar depth map; Represents an image depth map.

[0019] Optionally, the image depth map and the radar point cloud are input into a semantic projection model to obtain a radar depth map, including:

[0020] Filtering the background portion in the projection area using the depth value in the image depth map to obtain a target projection area;

[0021] The radar point cloud is projected onto the target projection area in the form of an image block to obtain the radar depth map; the depth values ​​in the image blocks are the same.

[0022] Optionally, the multi-scale image BEV features and the multi-scale radar BEV features are input into a multi-scale feature fusion module to obtain multi-scale fusion features, including:

[0023] The multi-scale query vector and the multi-scale offset vector corresponding to the multi-scale radar BEV feature are obtained by using the WQ linear layer and the WO linear layer of the multi-scale feature fusion module respectively;

[0024] The multi-scale key vector and the multi-scale value vector corresponding to the BEV feature of the multi-scale image are obtained by using the WK linear layer and the WV linear layer of the multi-scale feature fusion module respectively;

[0025] The multi-scale query vector, the multi-scale offset vector, the multi-scale key vector and the multi-scale value vector are fused to obtain the fused feature.

[0026] Optionally, after inputting the image depth map and the radar depth map into a filtering module to obtain an enhanced radar point cloud, the method further includes:

[0027] Updating the radar point cloud using the enhanced radar point cloud, and recording the number of updates;

[0028] The enhanced radar point cloud is updated using the updated radar point cloud and the image depth map, and when the number of updates reaches a preset number, the updating is stopped.

[0029] Optionally, the basic model is a Depth Anything model.

[0030] The present invention also provides a 3D target detection device based on point cloud image fusion of a basic model, comprising:

[0031] An image depth map acquisition module is used to input a multi-view image into a basic model to obtain an image depth map;

[0032] A radar depth map acquisition module, used for inputting the image depth map and the radar point cloud into a semantic projection model to obtain a radar depth map;

[0033] A radar point cloud enhancement module, used for inputting the image depth map and the radar depth map into a filtering module to obtain an enhanced radar point cloud;

[0034] A fusion module, used to respectively obtain the multi-scale image BEV features and the multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and input them into a multi-scale feature fusion module to obtain multi-scale fusion features;

[0035] The detection module is used to input the fusion features into the detection head to obtain the detection result.

[0036] The present invention also provides a 3D target detection device based on point cloud image fusion of a basic model, comprising:

[0037] Memory for storing computer programs;

[0038] A processor is used to implement the 3D target detection method based on point cloud image fusion of the basic model as described above when executing the computer program.

[0039] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the 3D target detection method based on point cloud image fusion of the basic model as described above is implemented.

[0040] It can be seen that the present invention obtains an image depth map by inputting a multi-view image into the basic model; inputting the image depth map and the radar point cloud into the semantic projection model to obtain a radar depth map; inputting the image depth map and the radar depth map into the filtering module to obtain an enhanced radar point cloud; respectively obtaining the multi-scale image BEV (Bird's Eye View) features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and inputting them into the multi-scale feature fusion module to obtain multi-scale fusion features; inputting the fusion features into the detection head to obtain the detection results. The present invention uses the basic model to realize the densification of sparse radar point clouds, so that the radar point cloud can provide more powerful and reliable distance information when further fused with the multi-view image, and the filtering module can also reduce additional noise; in order to better utilize the densified radar point cloud, the multi-scale feature fusion module interacts multi-scale image and radar features in an adaptive manner in the BEV space, realizing the thorough fusion of multi-scale and multi-modal BEV features, and improving the accuracy and robustness of target detection.

[0041] In addition, the present invention also provides a 3D target detection device, equipment and computer-readable storage medium based on point cloud image fusion of a basic model, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flowchart of a 3D target detection method based on point cloud image fusion of a basic model provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a framework of a semantic projection module provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of the architecture of a multi-scale feature fusion module provided in an embodiment of the present invention;

[0046] Figure 4A schematic diagram of the structure of a 3D object detection framework based on point cloud image fusion of a basic model provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the structure of a 3D target detection device based on point cloud image fusion of a basic model provided by an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the structure of a 3D target detection device based on point cloud image fusion of a basic model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In recent years, 3D object detection has received great attention in the field of autonomous driving, which aims to predict the position, size, and category of objects in 3D scenes. Cameras can provide high-resolution images with rich color and texture information, which can be used for object recognition and classification, but the performance of cameras is limited in bad weather, low light, or strong light conditions, and complex image processing algorithms are required to obtain the distance information of the target. Radar can provide high-precision distance information, so the distance and speed measurements of radar can be combined with the rich appearance and texture details in the camera image to provide more comprehensive and reliable perception for 3D object detection.

[0051] The existing optimization methods for 3D target detection with point cloud image fusion can be roughly divided into two categories. One type of methods focuses on improving the representativeness of single-mode radar by designing more advanced radar feature extractors. For example, RCFusion (an open source software library for medical image registration) emphasizes the differences in radar features in different regions by decoding the projection layer. Another type of method uses the cross-modal association between camera and radar signals to filter the noise in the original radar points by referencing semantically meaningful camera images. For example, HVDetFusion (a multimodal detection algorithm) can suppress radar noise in an explicit way by explicitly filtering noisy points using proposals (candidate areas that may contain target objects in target detection). None of the above methods fundamentally solves the sparsity problem of radar data, so how to make sparse radar data dense remains an unresolved challenge. In addition, the existing methods only perform single-scale fusion of multimodal data, lacking the use of multi-granularity information in multimodal data. Therefore, there are still deficiencies in detection robustness and accuracy.

[0052] To solve the above problems, this paper provides a 3D object detection method based on point cloud image fusion based on a basic model, which uses the general scene understanding knowledge embedded in the basic model to release the potential of radar measurement before fusion with multi-view images. Figure 1 , Figure 1 A flowchart of a 3D target detection method based on point cloud image fusion of a basic model provided by an embodiment of the present invention. The method may include:

[0053] S101: Input the multi-view image into the basic model to obtain an image depth map.

[0054] The executor of this embodiment is a terminal. This embodiment does not limit the type of terminal, as long as it can complete the operation of the 3D target detection method based on the point cloud image fusion of the basic model. The multi-view image in this embodiment can be a camera image captured by a camera, and the camera can provide high-resolution images containing rich color and texture information. This embodiment does not limit the basic model. The basic model in this embodiment is a general, large-scale artificial intelligence model. The basic model has witnessed a large number of training images and understands the general distribution of real-world images. Therefore, the basic knowledge learned by the basic model is general and can be effectively transferred to various downstream tasks. This embodiment can use the general scene understanding knowledge embedded in the basic model to release the potential of radar measurement before fusion with multi-view images.

[0055] Further, the basic model in this embodiment is the Depth Anything model. This model is an advanced monocular depth estimation technology that can extract depth information from 2D (two-dimensional) images and convert them into 3D images. It has made significant progress in improving the quality of depth maps, enabling it to more accurately estimate the depth information of the scene. The Depth Anything model expands the data set by collecting and automatically annotating a large amount of unlabeled data, thereby improving the generalization ability of the model. It also uses auxiliary supervision technology to force the model to inherit rich semantic priors from the pre-trained encoder, thereby improving the accuracy of depth estimation. In addition, the model also provides pre-trained models of different scales, which can be easily integrated into other deep learning frameworks to provide depth information for various applications. This embodiment adopts the encoder-decoder architecture in the Depth Anything model and adds a metric depth head (head) following ZoeDepth (the name of a deep learning model or framework for processing tasks related to depth estimation). During the model training process, the learning rate assigned to the backbone and the metric depth head is to adapt the model to the depth distribution of this scene while retaining the powerful content understanding ability of the basic model as much as possible.

[0056] S102: The image depth map and the radar point cloud are input into a semantic projection model to obtain a radar depth map.

[0057] This embodiment does not limit the method for obtaining the radar point cloud. For example, the radar point cloud can be obtained by a laser radar; or the radar point cloud can be obtained by a millimeter-wave radar. Considering that the resolution of the laser radar is limited, the millimeter-wave radar can provide distance, speed and direction information, and is not easily affected by the external environment. Therefore, this embodiment can use a millimeter-wave radar to obtain a radar point cloud. This embodiment inputs the obtained radar point cloud and the image depth map obtained in step S101 into the semantic projection model to obtain a radar depth map.

[0058] Furthermore, the image depth map and the radar point cloud are input into the semantic projection model to obtain the radar depth map, which may specifically include the following steps:

[0059] Step 21: Filter the background part in the projection area using the depth value in the image depth map to obtain the target projection area.

[0060] Specifically, this embodiment takes into account that direct projection will bring a problem: the radar points that initially illuminate the edge of the object's outline will be projected to an area that covers both the foreground and the background. The part projected onto the background will bring additional noise to the network, because the semantic projection model tends to confuse the depth of these background points with the depth of the foreground object during training. Therefore, this embodiment uses the powerful semantic understanding ability of the basic model and uses the depth value of the image depth map it generates to distinguish between the foreground and the background, thereby filtering out the background part in the projection area, such as Figure 2 As shown in Figure 2, each pixel in the image depth map records the distance from the viewpoint to the surface of the corresponding scene object.

[0061] Step 22: Project the radar point cloud onto the target projection area in the form of an image block to obtain a radar depth map; the depth values ​​in the image blocks are the same.

[0062] Traditionally, there is a one-to-one correspondence between individual radar points and pixels in a radar point cloud, meaning that each radar point naturally matches a unique pixel of its projection. This one-to-one correspondence means that when radar points are projected onto radar points to provide depth information, only a few pixels have corresponding radar points and depth information. However, these sparse depth values ​​from the radar may not provide enough information for the underlying model to accurately learn depth information. Therefore, this embodiment projects radar points onto patches on the image instead of individual pixels. The area of ​​a patch is a square area whose side length is related to the radar cross section (RCS) value, as follows:

[0063]

[0064] in, represents the radar RCS value, γ represents the hyperparameter, Indicates half of the side length of the square expansion block centered on the projection point. In this embodiment, all pixels in the square area centered on the original projection point, with a side length of 2a, share the depth information of the radar point.

[0065] S103: Input the image depth map and the radar depth map into a filtering module to obtain an enhanced radar point cloud.

[0066] Step S101 obtains the image depth map based on the basic model, and step S102 obtains the radar depth map based on the semantic projection model and the image depth map, thereby realizing the expansion of radar points. In order to improve the reliability of radar points, the image depth map and the radar depth map are input into the filtering module so that depth points can be reliably selected. Compared with the original radar point cloud, the number of reliable radar points is increased.

[0067] S104: respectively obtaining multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and inputting them into a multi-scale feature fusion module to obtain multi-scale fusion features.

[0068] Multi-scale image BEV features are obtained based on multi-view images, and multi-scale radar BEV features are obtained based on the enhanced radar point cloud. This embodiment does not limit the process of feature extraction, and specific reference may be made to the feature extraction means of the prior art. For example, multi-view images are converted into image BEV features through perspective transformation or depth estimation. Multi-scale radar BEV features are directly extracted from enhanced radar point cloud data using deep learning models (such as PointPillars (a deep learning model for 3D object detection), etc.). Among them, the deep learning model usually includes an encoder network and a decoder network. The encoder is responsible for converting the point cloud data into an intermediate feature representation, while the decoder maps these features into the BEV space to generate the final radar BEV feature map.

[0069] Existing methods fuse multi-scale features into a single-scale feature during the feature extraction process of each modality, and in the cross-modal fusion stage, unified-scale BEV features from the two modalities are further integrated. The present invention takes into account that low-level features capture fine-grained details, and high-level semantic features provide cohesive contextual interpretations. Early fusion limits the diversity of semantic granularity between modalities and hinders the proper interaction of semantic granularity at different levels. Therefore, this embodiment does not merge it prematurely, but retains the multi-scale semantic information until the cross-modal fusion stage, and uses MFF (multi-scale feature fusion module) to achieve dynamic alignment and thorough interaction between radar BEV features and image BEV features of different scales. In this way, the multi-scale feature fusion module interacts multi-resolution image and radar features in an adaptive manner in the BEV space, integrates information from different semantic levels, and enhances perception capabilities and robustness to radar point noise.

[0070] Furthermore, the multi-scale image BEV features and the multi-scale radar BEV features are input into the multi-scale feature fusion module to obtain the multi-scale fusion features, which may include the following steps:

[0071] Step 31: respectively use the WQ linear layer and the WO linear layer of the multi-scale feature fusion module to obtain the multi-scale query vector and the multi-scale offset vector corresponding to the multi-scale radar BEV feature;

[0072] Step 32: respectively use the WK linear layer and the WV linear layer of the multi-scale feature fusion module to obtain the multi-scale key vector and the multi-scale value vector corresponding to the BEV feature of the multi-scale image;

[0073] Step 33: Fuse the multi-scale query vector, the multi-scale offset vector, the multi-scale key vector and the multi-scale value vector to obtain a fused feature.

[0074] In order to completely integrate multi-scale and multi-modal features, this embodiment explicitly retains the multi-scale features of a single modality for the cross-modal feature fusion stage, and designs a corresponding multi-scale feature fusion module. For details, please refer to Figure 3 , the multi-scale feature fusion module consists of four separate linear layers WQ (Query linear layer), WO (Output linear layer), WK (Key linear layer), and WV (Value linear layer). WQ is used to generate query vectors, WO is used to generate offset vectors, WK is used to generate key vectors, and WV is used to generate value vectors. This embodiment organizes the results into a multi-scale form, followed by an LCM (Leaky Concatenation of Max-pooling, a layer in a deep learning model) block to obtain fused features, which are used as input to the detection head. Among them, LCM includes LN (normalization), Concat (connection), and Maxpooling (maximum pooling).

[0075] This embodiment calculates the query vector Q and sampling offset ∆q of the radar BEV feature, and the key vector K and value vector V of the image BEV feature at position q, where position q is a two-dimensional reference point Fr = {Frs}3s=1 in the multi-scale radar BEV feature. The output at position q can be calculated as follows:

[0076] ;

[0077] in, express( ; express ; , , , represents the coefficient matrix; H represents the number of attention heads; W h represents the result matrix; S represents different feature scales; P is the number of sampling points used in each scale; and Represents the BEV features of radar and image at different scales; softmax function (·) is used to determine the attention weight.

[0078] S105: Input the fused features into the detection head to obtain the detection results.

[0079] In this step, the multimodal BEV representation (ie, fusion feature) generated in step S104 is input into the detection head to generate a detection result.

[0080] Furthermore, in order to resolve the uncertainty in the depth prediction process, this embodiment may further include the following steps when executing step S101:

[0081] Step 41: Input the multi-view image into the basic model to predict the probability of each pixel at each predefined depth interval;

[0082] Step 42: Calculate the standard deviation based on the probability corresponding to each pixel.

[0083] Specifically, this embodiment also uses the basic model to calculate another indicator σ (standard deviation) in preparation for the filtering module. Specifically, during the fine-tuning process, the range of depth values ​​is divided into several intervals, each of which corresponds to a different depth range. The basic model is trained to predict the probability that the depth of each pixel falls into each predefined depth interval, enabling more efficient learning at different depth levels. For each pixel, there is a corresponding probability p at n depth intervals. The standard deviation σ prediction probability of the depth of each pixel in the predefined depth interval on the entire multi-view image is calculated as follows:

[0084]

[0085] Where C represents the total number of pixels in the multi-view image, N is the number of depth intervals, and p represents the probability of each depth interval.

[0086] Accordingly, step S103 may specifically include the following steps:

[0087] The image depth map, radar depth map and standard deviation are input into the filtering module to obtain the enhanced radar point cloud.

[0088] For details, please refer to Figure 3 , the input of the filtering module includes two parts: the radar depth map from the semantic projection model and the output from the base model (including the image depth map and the standard deviation σ). This embodiment implements cross-validation between the two input depth maps to filter the depth information. It should be noted that the depth points generated by fine-tuning (referring to the image depth map) provide comprehensive coverage of the entire image, while the depth points of the semantic projection model (referring to the radar depth map) are local and only cover specific areas. Therefore, this embodiment adopts different filtering methods according to whether the two depths overlap, which can be expressed as follows:

[0089] ;

[0090] in, represents the enhanced radar point cloud at position c; represents the depth value of position c in the radar depth map; represents the depth value of position c in the image depth map; represents the standard deviation of position c; is the threshold of the standard deviation and is a hyperparameter; Represents a radar depth map; Represents an image depth map.

[0091] Furthermore, after the image depth map and the radar depth map are input into the filtering module to obtain the enhanced radar point cloud, the following steps may be further included:

[0092] Step 51: Update the radar point cloud using the enhanced radar point cloud, and record the number of updates;

[0093] Step 52: Use the updated radar point cloud and image depth map to update the enhanced radar point cloud, and stop updating when the number of updates reaches a preset number.

[0094] In order to improve the reliability and accuracy of the newly added radar point cloud, this embodiment implements iterative densification. And in each densification iteration, the radar depth is used as a sparse guide to elicit dense depth knowledge embedded in the basic model. By iteratively executing the above steps, the radar depth gradually develops, and finally provides more powerful and reliable distance information when further fused with multi-view images, realizing the radar point cloud "from sparse to dense", and converting radar points that are sparser than "instance level" into dense and accurate "pixel level" depth maps. After each forward pass, the newly obtained reliable points are used to update the original radar points in the next iteration, that is, the radar point cloud is updated with the enhanced radar point cloud. The whole process is similar to the snowball effect, accumulating more depth points through an iterative and self-enhancing module, and realizing progressive radar signal densification.

[0095] The 3D target detection method based on point cloud image fusion of the basic model provided by the embodiment of the present invention is applied, by inputting the multi-view image into the basic model to obtain the image depth map; inputting the image depth map and the radar point cloud into the semantic projection model to obtain the radar depth map; inputting the image depth map and the radar depth map into the filtering module to obtain the enhanced radar point cloud; respectively obtaining the multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and inputting them into the multi-scale feature fusion module to obtain the multi-scale fusion features; inputting the fusion features into the detection head to obtain the detection results. The present invention uses the basic model to realize the densification of sparse radar point clouds, so that the radar point cloud can provide more powerful and reliable distance information when further fused with the multi-view image, and the filtering module can also reduce additional noise. In order to better utilize the densified radar point cloud, the multi-scale feature fusion module interacts the multi-scale image and radar features in an adaptive manner in the BEV space, realizing the thorough fusion of multi-scale and multi-modal BEV features, and improving the accuracy and robustness of target detection. Furthermore, a progressive and cyclical approach is adopted to enhance the sparse radar points, benefiting from the base model while reducing its additional noise.

[0096] In order to make the present invention easier to understand, please refer to Figure 4 , Figure 4 A schematic diagram of the structure of a 3D target detection framework based on point cloud image fusion of a basic model provided in an embodiment of the present invention may specifically include: (1) a progressive radar densification module and (2) a multi-scale multi-modal detection module. The progressive radar densification module is used to transform the radar point cloud from "sparse to dense", and convert radar points that are sparser than "instance level" into dense and accurate "pixel level" depth maps. The multi-scale multi-modal detection module is used to thoroughly fuse multi-scale and multi-modal features, explicitly retain the multi-scale features of a single modality, and is used in the cross-modal feature fusion stage. The multi-scale feature fusion module of this method takes into account the scale hierarchy of features and retains multi-scale information based on the characteristics of the sample level. At the same time, in the multi-modal fusion method, a powerful basic model is used to enhance the radar signal. In particular, the sparse radar depth is used as a guiding prompt to derive mature scene understanding knowledge from the basic model for improvement.

[0097] The following is an introduction to the 3D target detection device based on point cloud image fusion of a basic model provided in an embodiment of the present invention. The 3D target detection device based on point cloud image fusion of a basic model described below and the 3D target detection method based on point cloud image fusion of a basic model described above can be referenced to each other.

[0098] Please refer to Figure 5 , Figure 5A schematic structural diagram of a 3D object detection device based on point cloud image fusion of a basic model provided by an embodiment of the present invention may include:

[0099] The image depth map acquisition module 100 is used to input the multi-view image into the basic model to obtain the image depth map;

[0100] A radar depth map acquisition module 200 is used to input the image depth map and the radar point cloud into a semantic projection model to obtain a radar depth map;

[0101] A radar point cloud enhancement module 300, configured to input the image depth map and the radar depth map into a filtering module to obtain an enhanced radar point cloud;

[0102] A fusion module 400 is used to obtain multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, respectively, and input them into a multi-scale feature fusion module to obtain multi-scale fusion features;

[0103] The detection module 500 is used to input the fusion feature into the detection head to obtain the detection result.

[0104] Based on the above embodiment, the 3D object detection device based on point cloud image fusion of the basic model may further include:

[0105] A probability prediction module, used to input the multi-view image into the basic model to predict the probability of each pixel at each predefined depth interval;

[0106] A standard deviation calculation module, used to calculate the standard deviation according to the probability corresponding to each pixel;

[0107] Accordingly, the radar point cloud enhancement module 300 may include:

[0108] The radar point cloud enhancement unit is used to input the image depth map, the radar depth map and the standard deviation into the filtering module to obtain the enhanced radar point cloud.

[0109] Based on the above embodiment, the radar point cloud enhancement module 300 may specifically include:

[0110] ;

[0111] in, represents the enhanced radar point cloud at position c; represents the depth value of position c in the radar depth map; represents the depth value of position c in the image depth map; represents the standard deviation of position c; is the threshold of the standard deviation and is a hyperparameter; Represents a radar depth map; Represents an image depth map.

[0112] Based on the above embodiment, the radar depth map acquisition module 200 may include:

[0113] A filtering unit, configured to filter the background portion in the projection area using the depth value in the image depth map to obtain a target projection area;

[0114] A projection unit is used to project the radar point cloud onto the target projection area in the form of an image block to obtain the radar depth map; the depth values ​​in the image blocks are the same.

[0115] Based on the above embodiment, the fusion module 400 may include:

[0116] The first unit is used to obtain a multi-scale query vector and a multi-scale offset vector corresponding to the multi-scale radar BEV feature by using the WQ linear layer and the WO linear layer of the multi-scale feature fusion module respectively;

[0117] The second unit is used to obtain the multi-scale key vector and the multi-scale value vector corresponding to the multi-scale image BEV feature by using the WK linear layer and the WV linear layer of the multi-scale feature fusion module respectively;

[0118] The third unit is used to fuse the multi-scale query vector, the multi-scale offset vector, the multi-scale key vector and the multi-scale value vector to obtain the fused feature.

[0119] Based on any of the above embodiments, the 3D object detection device based on point cloud image fusion of the basic model may further include:

[0120] A radar point cloud updating module, used to update the radar point cloud using the enhanced radar point cloud and record the number of updates;

[0121] The enhanced radar point cloud update module is used to update the enhanced radar point cloud using the updated radar point cloud and the image depth map, and stop updating when the number of updates reaches a preset number.

[0122] Based on the above embodiment, the basic model may specifically be a Depth Anything model.

[0123] It should be noted that the order of the modules and units in the above-mentioned 3D object detection device based on point cloud image fusion of the basic model can be changed without affecting the logic.

[0124] The 3D target detection device using point cloud image fusion based on the basic model provided in the embodiment of the present invention is applied, and the image depth map acquisition module 100 is used to input the multi-view image into the basic model to obtain the image depth map; the radar depth map acquisition module 200 is used to input the image depth map and the radar point cloud into the semantic projection model to obtain the radar depth map; the radar point cloud enhancement module 300 is used to input the image depth map and the radar depth map into the filtering module to obtain the enhanced radar point cloud; the fusion module 400 is used to respectively obtain the multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and input them into the multi-scale feature fusion module to obtain the multi-scale fusion features; the detection module 500 is used to input the fusion features into the detection head to obtain the detection results. The present invention uses the basic model to realize the densification of sparse radar point cloud, so that the radar point cloud can provide more powerful and reliable distance information when further fused with multi-view images. At the same time, the filtering module can also reduce additional noise. In order to better utilize the densified radar point cloud, the multi-scale feature fusion module interacts multi-scale images and radar features in an adaptive manner in the BEV space, realizing the thorough fusion of multi-scale and multi-modal BEV features, and improving the accuracy and robustness of target detection.

[0125] The following is an introduction to the 3D target detection device based on point cloud image fusion of the basic model provided in an embodiment of the present invention. The 3D target detection device based on point cloud image fusion of the basic model described below and the 3D target detection method based on point cloud image fusion of the basic model described above can be referenced to each other.

[0126] Please refer to Figure 6 , Figure 6 A schematic structural diagram of a 3D target detection device based on point cloud image fusion of a basic model provided by an embodiment of the present invention may include:

[0127] A memory 10, used for storing computer programs;

[0128] The processor 20 is used to execute a computer program to implement the above-mentioned 3D target detection method based on point cloud image fusion of the basic model.

[0129] The memory 10 , the processor 20 , and the communication interface 31 all communicate with each other via the communication bus 32 .

[0130] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0131] By inputting multi-view images into the basic model, the image depth map is obtained;

[0132] Input the image depth map and radar point cloud into the semantic projection model to obtain the radar depth map;

[0133] The image depth map and radar depth map are input into the filtering module to obtain the enhanced radar point cloud;

[0134] The multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud are obtained respectively, and input into the multi-scale feature fusion module to obtain the multi-scale fusion features;

[0135] The fused features are input into the detection head to obtain the detection results.

[0136] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0137] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0138] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .

[0139] The communication interface 31 may be an interface of a communication module, and is used to connect to other devices or systems.

[0140] Of course, it should be noted that Figure 6 The structure shown does not constitute a limitation on the 3D target detection device based on the point cloud image fusion of the basic model in the embodiment of the present invention. In actual applications, the 3D target detection device based on the point cloud image fusion of the basic model may include Figure 6 More or fewer components than shown, or combinations of certain components.

[0141] The following is an introduction to the readable storage medium provided in an embodiment of the present invention. The readable storage medium described below and the 3D target detection method based on point cloud image fusion of the basic model described above can be referenced to each other.

[0142] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned 3D target detection method based on point cloud image fusion of a basic model are implemented.

[0143] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0144] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0145] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0146] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0147] The above is a detailed introduction to the 3D target detection method, device, equipment and computer-readable storage medium based on point cloud image fusion of a basic model provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A 3D object detection method based on point cloud image fusion of a basic model, characterized in that: include: Input the multi-view image into the basic model to obtain the image depth map; Inputting the image depth map and the radar point cloud into a semantic projection model to obtain a radar depth map; Inputting the image depth map and the radar depth map into a filtering module to obtain an enhanced radar point cloud; Respectively obtaining multi-scale image BEV features and multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and inputting them into a multi-scale feature fusion module to obtain multi-scale fusion features; The fused features are input into the detection head to obtain the detection result.

2. The 3D object detection method based on point cloud image fusion of the basic model according to claim 1 is characterized in that: Also includes: Input the multi-view image into the basic model to predict the probability of each pixel at each predefined depth interval; The standard deviation is obtained by calculating the probability corresponding to each pixel; Accordingly, the image depth map and the radar depth map are input into a filtering module to obtain an enhanced radar point cloud, including: The image depth map, the radar depth map and the standard deviation are input into the filtering module to obtain the enhanced radar point cloud.

3. The 3D object detection method based on point cloud image fusion of basic model according to claim 2 is characterized in that: Inputting the image depth map, the radar depth map and the standard deviation into a filtering module to obtain the enhanced radar point cloud, including: ； in, represents the enhanced radar point cloud at position c; represents the depth value of position c in the radar depth map; represents the depth value of position c in the image depth map; represents the standard deviation of position c; is the threshold of the standard deviation and is a hyperparameter; Represents a radar depth map; Represents an image depth map.

4. The 3D object detection method based on point cloud image fusion of basic model according to claim 1, characterized in that: The image depth map and the radar point cloud are input into the semantic projection model to obtain the radar depth map, including: Filtering the background portion in the projection area using the depth value in the image depth map to obtain a target projection area; The radar point cloud is projected onto the target projection area in the form of an image block to obtain the radar depth map; the depth values ​​in the image blocks are the same.

5. The 3D object detection method based on point cloud image fusion of basic model according to claim 1, characterized in that: The multi-scale image BEV features and multi-scale radar BEV features are input into the multi-scale feature fusion module to obtain multi-scale fusion features, including: The multi-scale query vector and the multi-scale offset vector corresponding to the multi-scale radar BEV feature are obtained by using the WQ linear layer and the WO linear layer of the multi-scale feature fusion module respectively; The multi-scale key vector and the multi-scale value vector corresponding to the BEV feature of the multi-scale image are obtained by using the WK linear layer and the WV linear layer of the multi-scale feature fusion module respectively; The multi-scale query vector, the multi-scale offset vector, the multi-scale key vector and the multi-scale value vector are fused to obtain the fused feature.

6. The 3D object detection method based on point cloud image fusion of basic model according to any one of claims 1 to 5, characterized in that: After the image depth map and the radar depth map are input into the filtering module to obtain the enhanced radar point cloud, the method further includes: Updating the radar point cloud using the enhanced radar point cloud, and recording the number of updates; The enhanced radar point cloud is updated using the updated radar point cloud and the image depth map, and when the number of updates reaches a preset number, the updating is stopped.

7. The 3D object detection method based on point cloud image fusion of basic model according to claim 1, characterized in that: The basic model is the Depth Anything model.

8. A 3D object detection device based on point cloud image fusion of a basic model, characterized in that: include: An image depth map acquisition module is used to input a multi-view image into a basic model to obtain an image depth map; A radar depth map acquisition module, used for inputting the image depth map and the radar point cloud into a semantic projection model to obtain a radar depth map; A radar point cloud enhancement module, used for inputting the image depth map and the radar depth map into a filtering module to obtain an enhanced radar point cloud; A fusion module, used to respectively obtain the multi-scale image BEV features and the multi-scale radar BEV features corresponding to the multi-view image and the enhanced radar point cloud, and input them into a multi-scale feature fusion module to obtain multi-scale fusion features; The detection module is used to input the fusion features into the detection head to obtain the detection result.

9. A 3D object detection device based on point cloud image fusion of a basic model, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the 3D target detection method based on point cloud image fusion of a basic model as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the 3D target detection method based on point cloud image fusion of the basic model as described in any one of claims 1 to 7 is implemented.

Images