An improved three-dimensional target detection method of multi-modal fusion

Abstract

The application relates to a target detection technology, and specifically discloses a three-dimensional target detection method improved by multi-modal fusion, which extracts features of point clouds and images, defines a local attention correlation function by the feature difference of each point cloud and points in the field of the point cloud, obtains an edge perception graph of the point cloud, and further obtains a center heat map of an object from the edge information. In the image and point cloud feature fusion stage, a depth perception BEV grid division strategy is used to project the image features to the BEV grid, the image-point cloud feature heat map after fusion and the point cloud heat map are combined, and an initial query point index is generated. The corresponding points are located in the point cloud BEV feature through the index, and the attention fusion is sequentially performed on the point cloud features and the image features, so that the final fusion query point is obtained. The application utilizes the spatial structure characteristics of the point cloud, simultaneously utilizes the depth perception BEV grid to fuse the image information, combines the advantages of different modes, and improves the detection precision and stability in complex road conditions.

Description

Technical Field

[0001] This invention relates to target detection and autonomous driving technology, specifically to an improved three-dimensional target detection method using multimodal fusion. Background Technology

[0002] With the continuous improvement of chip performance and the rapid development of new energy vehicles, autonomous driving technology is entering a stage of rapid evolution. Currently, most new energy vehicles are equipped with driver assistance systems, which significantly enhance driving safety while improving driving convenience. To achieve higher levels of autonomous driving, improving the accuracy and stability of road target detection has become one of the key technologies. LiDAR and cameras are two important object detection sensors in autonomous driving. Due to the sparsity of point clouds, using LiDAR alone is clearly insufficient for robust 3D detection. For example, it is difficult to detect small or overlapping objects in LiDAR mode. In contrast, these objects remain clearly visible and distinguishable in high-resolution images. The complementary nature of point clouds and images prompts the design of detectors that leverage the advantages of both modalities.

[0003] Currently, one of the most representative methods in multimodal feature fusion is the cross-attention mechanism. Unlike traditional single-modal attention mechanisms, when dealing with inconsistent data formats across different modalities (such as point clouds and images), it is usually necessary to construct a unified spatial representation to guide the interaction and fusion of features from different modalities. Among these, the Bird's Eye View (BEV) space is widely used as a common representation domain for cross-modal feature fusion due to its spatial partitioning capabilities and representational advantages adapted to downstream tasks.

[0004] When initializing query points for BEV spatial localization, traditional center heatmap methods, while demonstrating good center localization capabilities in image tasks, have significant limitations in point cloud scenarios. First, heatmaps essentially estimate target centers based on pixel density. This method is more effective on regular, dense two-dimensional grids, but point cloud data is typically sparse and unstructured, making the definition of the center inherently unstable. Second, center heatmaps ignore the spatial geometry of the point cloud, relying solely on the distribution of points. This makes it difficult to fully utilize the local relationships and geometric features between points, leading to insufficient understanding of target morphology. The robustness of center heatmaps decreases significantly, especially in the presence of occlusion, small targets, or asymmetrical objects. Therefore, relying solely on center heatmaps can easily lead to localization errors or missed target detections.

[0005] When constructing the BEV space of an image, traditional methods typically employ a uniformly divided grid structure, i.e., dividing the entire space equidistantly with a fixed number of grids. However, this strategy does not consider the actual perception depth of the camera, resulting in an implicit variation in the actual size of each grid in physical space, significantly affected by the overall perception range. This approach has obvious drawbacks: First, uniform grids ignore the differences in importance between different regions in the perception task. In reality, the area near the vehicle is the most sensitive to perception accuracy, while distant areas are more tolerant. Second, with uniform division, distant areas waste a lot of computational resources without providing meaningful features, while near-field areas may lose details due to insufficient resolution. Uniform grids treat high-value and low-value areas uniformly in terms of computational resources, causing image information to be "averaged" and diluted in the near-field region, making it difficult to form a clear and focused response, ultimately affecting detection accuracy. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies and provide an improved three-dimensional target detection method based on multimodal fusion.

[0007] Technical solution: An improved multimodal fusion method for three-dimensional target detection according to the present invention includes the following steps:

[0008] Step 1: Construct a dataset consisting of point clouds and images. Extract image features from the images in the dataset using the ResNet+FPN network model, and extract point cloud features from the point clouds in the dataset using the SECOND+SECONDFPN network model.

[0009] Step 2: Based on the point cloud features obtained in Step 1, calculate the feature difference for each point cloud and its k nearest neighbors. According to feature difference To construct a local attention correlation function And scale to obtain attention relevance map ;

[0010] Step 3: Analyze the attention correlation map obtained in Step 2. Standard deviation is calculated to obtain the edge sensing map E. The center heatmap of the point cloud is obtained by taking the local mean of the edge sensing map E and subtracting it from the original edge sensing map. ;

[0011] Step 4: For the image features obtained in Step 1, combine their length and width dimensions to obtain the image BEV features. And construct a BEV mesh with depth perception capabilities. Project the image to obtain the corresponding BEV position code;

[0012] Simultaneously, the point cloud features obtained in step 1 are merged with their length and width dimensions to obtain the point cloud BEV features. And construct a uniform BEV mesh. Project the image to obtain the corresponding BEV position code;

[0013] Then, the image BEV features and point cloud BEV features, along with their respective corresponding BEV location codes, are fused using cross-attention. The fused features are then... The center heatmap of the fused features was calculated. ;

[0014] Step 5: Obtain the central heat map and central heat map Combined, the initial query point index is obtained from the 200 points with the highest values ​​(by adding the two heatmaps together and taking the highest value as the target point).

[0015] Next, the point cloud corresponding to the obtained index is found on the point cloud BEV features. Attention is first calculated with the point cloud BEV features to generate a query point with an initial prediction. Then, cross-attention is calculated with the image BEV features to generate a fused query point.

[0016] Finally, the fused query points are input into prediction_head for final prediction. The prediction classification includes the detected target type and the target's 3D bounding box.

[0017] Further, step 2 obtains the attention relevance map. The detailed method is as follows:

[0018] For the point cloud data after feature extraction, the last dimension is expanded to a length k. First, the Euclidean distance between each point cloud is calculated to find the k nearest neighbors of each point cloud. Based on the index, the features of these k points are found, and then a local attention correlation function is constructed. :

[0019] ;

[0020] Where Q and K are linearly varying layers, For the features of point cloud i, The features of the neighboring points of point cloud i are used; the correlation function is scaled to obtain the attention correlation map. ;

[0021] ;

[0022] Where d is the feature dimension.

[0023] Furthermore, an attention correlation map is obtained. Next, step 3 calculates the standard deviation on the neighborhood channel of each point cloud to obtain the edge-aware map. The larger the value, the more likely it is to be the center of the object;

[0024] ;

[0025] In the above formula, dim=-1 refers to the last dimension;

[0026] Obtain edge-aware map Then, a local average is performed using 5×5 average pooling to obtain E. 1 The original edge-aware image and Subtraction yields the center score map G;

[0027] ;

[0028] Next, the center score map G is convolved once (with a kernel size of 1) to obtain its classification. To avoid destroying the original score, a center heatmap of the point cloud is obtained. ;

[0029] ;

[0030] in, For a convolution with 1 input channel and 10 output channels, , where the kernel size is 1.

[0031] Furthermore, in step 4, the BEV mesh shape is 1200×1200, the camera depth range of the dataset is set to 50 meters, a gradient-dense mesh is applied to the important foreground features in the first X meters, while a uniform mesh is used for the less important background features in the last 50-X meters. The formula for front-end mesh division is as follows:

[0032] ;

[0033] ;

[0034] ;

[0035] in, The minimum depth (usually 0), The number of grids for important areas, The total number of grids to divide the y-axis (1200); The magnitude of the y-axis direction of the i-th grid in the foreground (from near to far);

[0036] For the features in the last 50-X meters, a uniform grid is applied, and the remaining grid is used, as shown in the following formula:

[0037] ;

[0038] ;

[0039] in, The maximum depth is 50 meters, based on the camera's depth range. This represents the number of grid cells along the y-axis in the background. For the background The size of each grid cell along its y-axis;

[0040] Then, the image BEV features and point cloud BEV features, along with their corresponding BEV location codes, are input into cross-attention for calculation to obtain the center heatmap of the fused features. :

[0041] ;

[0042] ;

[0043] Here, heatmap is a two-layer convolution with 10 output channels, corresponding to the 10 categories in the dataset. BEV_L is the uniform BEV grid position encoding of the point cloud, and BEV_C is the BEV grid position encoding obtained through depth perception.

[0044] This invention utilizes the camera's perception range, with image features corresponding to different sizes of BEV (Body Image Module) grids for different perception ranges. After projecting image features into the depth-sensing BEV grid, the output is a feature in the same space as the point cloud BEV. Similarly, the height and width dimensions of the image are concatenated to obtain [BS, C, total_width], providing a positional code, which is then replaced by the BEV grid. The point cloud BEV features and the BEV positional code are input into a Transformer. During aggregation with point cloud queries, smaller grids yield more refined features. Smaller grids involve more and more refined attention interactions in the Transformer, resulting in richer extracted information. The projection process essentially merges dimensions and then provides a positional code for the BEV grid.

[0045] Further, in step 5, the heatmap of the point cloud is combined with the heatmap of the fused features to obtain a dense map, the index of the high-response points is obtained, and the index of the first 200 points on the BEV feature is set as the center position of the prediction box.

[0046] ;

[0047] The obtained query points are subjected to a multi-head attention mechanism with point cloud and image features respectively. Finally, the prediction box with rich semantics is input into the prediction_head detection head for final prediction.

[0048] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0049] (1) This invention proposes an edge-aware module that generates query points from the edge-aware map of a point cloud. It makes full use of the local geometric relationships between point clouds and, by constructing connections between points and their neighbors, mines regions where structural boundaries may exist. This approach not only preserves the three-dimensional spatial features of the point cloud but also accurately captures the contour information of the target.

[0050] Unlike traditional heatmap methods that rely solely on "central probability distribution," this invention starts from the edges, establishes a structure-aware inference path, and gradually aggregates back to the target region, thus overcoming the shortcomings of central heatmaps in spatial structure modeling. This edge-guided structure enhancement mechanism significantly improves the model's ability to identify small and occluded targets in complex scenes and provides a processing approach for point cloud data that better suits its sparse and unstructured characteristics.

[0051] (2) The depth-sensing BEV mesh of the present invention uses a higher resolution in the near-vehicle area, which effectively improves the fineness of image features and semantic expression ability. This spatial resolution allocation makes the information of key areas (such as the vehicle's periphery) richer and more accurate. When fused with radar and other modalities, the cross-attention of key areas is more discriminative, thereby significantly improving the response quality and focusing ability of the subsequent center heatmap at the target location, and enhancing the accuracy and stability of target center prediction.

[0052] (3) This invention combines edge perception technology with depth perception BEV mesh, utilizing the spatial structure of point cloud features while also taking advantage of the high resolution of image features. At the same time, the designed depth perception BEV mesh is not only effective in road recognition, but can also be used in other target recognition fields in a plug-and-play manner. Attached Figure Description

[0053] Figure 1 A flowchart for generating the initial query in this invention;

[0054] Figure 2 This is a schematic diagram of the point cloud edge perception module in the embodiment;

[0055] Figure 3 This is a schematic diagram of the depth-sensing BEV module in the embodiment;

[0056] Figure 4This is a schematic diagram of the BEV mesh effect in the embodiment. Detailed Implementation

[0057] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0058] This invention extracts features from point clouds and images, defines a local attention correlation function for the feature difference between each point cloud and its neighboring point clouds, and calculates the standard deviation of the obtained correlation map to obtain an edge-aware map of the point clouds. Then, it extracts the center heatmap of the object from the edge information. In the image and point cloud feature fusion stage, a depth-aware BEV mesh partitioning strategy is used. When projecting image features onto the BEV mesh, higher spatial resolution is given to near-field regions to retain richer image detail information. The fused feature heatmap is then combined with the point cloud heatmap to generate an initial query point index. Using this index, the corresponding point is located in the point cloud BEV features, and attention fusion is performed sequentially with the point cloud features and image features to obtain the final fused query point.

[0059] like Figure 1 As shown, the improved multimodal fusion-based 3D target detection method of this embodiment includes the following steps:

[0060] Step 1: Construct a dataset consisting of point clouds and images. Extract image features using the ResNet+FPN network model and extract point cloud features using the SECOND+SECONDFPN network model.

[0061] Step 2: Based on the point cloud features obtained in Step 1, calculate the feature difference for each point cloud and its k nearest neighbors. According to feature difference To construct a local attention correlation function And scale to obtain attention relevance map ;

[0062] Step 3: Analyze the attention correlation map obtained in Step 2. Standard deviation is calculated to obtain the edge sensing map E. The center heatmap of the point cloud is obtained by taking the local mean of the edge sensing map E and subtracting it from the original edge sensing map. ;

[0063] Step 4: For the image features obtained in Step 1, combine their length and width dimensions to obtain the image BEV features. And construct a BEV mesh with depth perception capabilities. Project;

[0064] Simultaneously, the point cloud features obtained in step 1 are merged with their length and width dimensions to obtain the point cloud BEV features. And construct a uniform BEV mesh. Project;

[0065] Then, the image BEV features and point cloud BEV features, along with their respective corresponding BEV location codes, are fused using cross-attention. The fused features are then... The center heatmap of the fused features was calculated. ;

[0066] Step 5: Obtain the central heat map and central heat map Combined, the initial query point index is obtained from the 200 points with the highest values ​​(by adding the two heatmaps together and taking the highest value as the target point).

[0067] Next, the obtained index is used to find the corresponding point on the point cloud BEV feature. Attention is first calculated with the point cloud BEV feature to generate a query point with an initial prediction. Then, cross-attention is calculated with the image BEV feature to generate a fused query point.

[0068] Finally, the query point is input into prediction_head for final prediction. The prediction classification includes the detected target type and the target's 3D bounding box.

[0069] like Figure 2 As shown, step 2 obtains the attention correlation map. The detailed method is as follows:

[0070] For the point cloud data after feature extraction, the last dimension is expanded to a length of k. First, the Euclidean distance between each point cloud is calculated to find the k nearest neighbors of each point cloud. Based on the index, the features of these k points are found, and then a local attention correlation function is constructed. :

[0071] ;

[0072] Where Q and K are linearly varying layers, For the characteristics of point i, The features of the points adjacent to point i.

[0073] Scaling the correlation function yields an attention correlation graph. :

[0074] ;

[0075] Where d is the feature dimension;

[0076] Obtain attention correlation map Next, step 3 calculates the standard deviation on the neighborhood channel of each point cloud to obtain the edge-aware map. The larger the value, the more likely it is to be the center of the object;

[0077] ;

[0078] In the above formula, dim=-1 refers to the last dimension;

[0079] After obtaining the edge-aware map E, a local average is performed on it using 5×5 average pooling to obtain E. 1 Then the original edge-aware image E and E 1 Subtraction yields the center score map G;

[0080] ;

[0081] Next, the center score map G is convolved once (with a kernel size of 1) to obtain its classification. To avoid destroying the original score, a center heatmap of the point cloud is obtained. ;

[0082] ;

[0083] in, For a convolution with 1 input channel and 10 output channels, , where the kernel size is 1.

[0084] like Figure 3 and Figure 4 As shown, the BEV mesh in this embodiment has a shape of 1200×1200. The camera depth range of the dataset is set to 50 meters. Gradient-dense meshes are used for important foreground features in the first X meters, while uniform meshes are used for less important background features in the last 50-X meters. The formula for front-end mesh division is as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] in, The minimum depth (usually 0), The number of grids for important areas, The total number of grids for the y-axis is 1200; The magnitude of the y-axis direction of the i-th grid in the foreground (from near to far);

[0089] For the features in the last 50-X meters, a uniform grid is applied, and the remaining grid is used, as shown in the following formula:

[0090] ;

[0091] ;

[0092] in, The maximum depth is 50 meters, based on the camera's depth range. This represents the number of grid cells along the y-axis in the background. For the background The size of each grid cell along its y-axis;

[0093] Then, the image BEV features and point cloud BEV features, along with their corresponding BEV location codes, are input into cross-attention for calculation to obtain the center heatmap of the fused features. :

[0094] ;

[0095] ;

[0096] Here, heatmap is a two-layer convolution with 10 output channels, corresponding to the 10 categories in the dataset. BEV_L is the uniform BEV grid position encoding of the point cloud, and BEV_C is the BEV grid position encoding obtained through depth perception.

[0097] This invention utilizes the camera's perception range, with image features corresponding to different sizes of BEV (Browser Active Value) grids for different perception ranges. After projecting image features into the depth-sensing BEV grid, the output is a feature in the same space as the point cloud BEV. Similarly, the height and width dimensions of the image are concatenated to obtain [BS, C, total_width], and the positional encoding is replaced by the depth-sensing BEV grid. The point cloud BEV features and the BEV positional encoding are input into the Transformer together. During aggregation with point cloud queries, smaller grids yield more refined features. Smaller grids involve more and more refined attention interactions in the Transformer, resulting in richer extracted information. The projection process essentially merges dimensions and then provides a positional encoding for the BEV grid.

[0098] Step 5 combines the heatmap of the point cloud with the heatmap of the fused features to obtain a dense map, obtains the index of high-response points, and sets the index of the first 200 points on the BEV feature as the center position of the prediction box:

[0099] ;

[0100] The obtained query points are subjected to a multi-head attention mechanism with point cloud and image features respectively. Finally, the prediction box with rich semantics is input into the prediction_head detection head for final prediction.

[0101] To verify the technical effectiveness of this invention, this embodiment applies the method of this invention to perform 3D road target detection in the following scenarios. The experimental dataset used in this embodiment is nuScenes, a large-scale autonomous driving benchmark containing 10,000 driving scenarios, divided into 700, 150, and 150 scenarios for training, validation, and testing, respectively. For detection, nuScenes defines an evaluation protocol, including the nuScenes Detection Score (NDS), mean accuracy (mAP), and five true measure (TP) metrics: mean translation error (mATE), mean scale error (mASE), mean orientation error (mAOE), mean velocity error (mAVE), and mean attribute error (mAAE). In this embodiment, mAP is the average of the mean accuracies of ten classes at distance thresholds of 0.5m, 1m, 2m, and 4m. NDS is a weighted combination of mAP, mATE, mASE, mAOE, mAVE, and mAAE. Specific comparison results are shown in Table 1.

[0102] Table 1. Accuracy comparison with baseline network on the validation set.

[0103]

[0104] In the table above, Car and Ped represent the accuracy of detecting cars and pedestrians.

[0105] The experimental results show that this invention utilizes the spatial structure characteristics of point clouds and integrates image information from depth-sensing BEV meshes, combining the advantages of different modalities to improve detection accuracy and stability in complex road conditions.

Claims

1. A method for improved multi-modal fusion based three-dimensional object detection, the method comprising: Comprising the following steps: Step 1, constructing a dataset composed of point cloud and image, extracting features of image in the dataset through ResNet+FPN network model to obtain image features, and extracting features of point cloud in the dataset through SECOND+SECONDFPN network model to obtain point cloud features; Step 2, based on the point cloud features obtained in step 1, calculate the feature difference for each point cloud and its k nearest point clouds , construct a local attention correlation function according to the feature difference , and scale to obtain an attention correlation map ; Step 3, attention correlation map of step 2 The standard deviation is calculated to obtain the edge perception map E, and the center heat map of the point cloud is obtained by locally averaging the edge perception map E and subtracting the original edge perception map ; Step 4, for the image feature obtained in step 1, merge the length and width dimensions to obtain the image BEV feature and construct a BEV grid with depth perception capability Projection is performed to obtain the corresponding BEV position encoding; At the same time, for the point cloud features obtained in step 1, the length and width dimensions thereof are combined to obtain point cloud BEV features , and a uniform BEV grid is constructed Projection is performed to obtain corresponding BEV position encodings; Then, cross-attention fusion is performed on the image BEV feature and the point cloud BEV feature and their respective corresponding BEV position encodings, and a fused feature A center heat map of the fused feature is calculated ; Step 5, combining the center heat map and the center heat map to obtain an initial query point index from the 200 points with the highest values; Then, find the corresponding point cloud of the obtained index on the point cloud BEV feature, first perform attention calculation with the point cloud BEV feature to generate a query point with initial prediction; then perform cross-attention calculation with the image BEV feature to generate a fused query point; Finally, input the fused query point into prediction_head for final prediction, and the prediction classification includes the detected target type and the three-dimensional bounding box of the target.

2. The method of claim 1, wherein, The step 2 obtains the attention correlation graph The detailed method is that: For the point cloud data after feature extraction, the last dimension is expanded to length k; first calculate the Euclidean distance between each point cloud, find the nearest k neighbors of each point cloud, find the features of the k points according to the index, and then construct a local attention correlation function : ； wherein Q, K are linearly varying layers, is a feature of the point cloud i, is a feature of a neighboring point of the point cloud i; scaling the correlation function to obtain an attention correlation map ; ； Wherein, d is the feature dimension.

3. The method of claim 1 or 2, wherein, Attention correlation map is obtained After that, step 3 calculates the standard deviation on the domain channel of each point cloud to obtain the edge perception map ; ； In the above formula, dim=-1 means the last dimension; obtain edge-aware map After that, local average is made using 5x5 average pooling to obtain E 1 , and the original edge-aware map Subtracting obtain the center score map G; ； Next, the center score map G is convolved to obtain a center heat map of the point cloud ; ； wherein, is a convolution with input channels 1 and output channels 10, is a convolution kernel size of 1. 4.The method of claim 1, wherein, The shape of the BEV grid in step 4 is 1200x1200, and the camera depth range of the dataset is set to 50 meters. The important features in the foreground of the first X meters are divided into dense grids according to the gradient, while the unimportant features in the background of the last 50-X meters are divided into uniform grids. The formula for dividing the front grid is as follows: ； ； ； wherein, is the minimum depth, is the number of grid of the important region division, is the total number of grid of y axis division; is the size of the i-th grid of the foreground in the y axis direction. For the features of the last 50-X meters, a uniform grid is used, and the remaining grid is allocated, and the formula is as follows: ； ； wherein, is the maximum depth; is the number of y-axis grids of the background; is the y-axis direction size of the i-th grid of the background; is the y-axis direction size of the i-th grid of the background; Then, the image BEV feature and the point cloud BEV feature and the respective corresponding BEV position codes are input into cross attention for calculation to obtain a center heat map of the fused feature : ； ； Wherein, heatmap is a two-layer convolution, and the final output channel is 10, corresponding to 10 categories in the dataset, BEV_L is the BEV grid position encoding corresponding to the point cloud, which is uniform, and BEV_C is the BEV grid position encoding obtained by depth perception. 5.The method of claim 1, wherein, Step 5 combines the heat map of the point cloud with the heat map of the fused features to obtain a densemap, and then obtains the index of the high response point. The index on the first 200 point cloud BEV features is set as the center position of the prediction box: ； The obtained query point is subjected to multi-head attention mechanism with the point cloud and image features respectively, and finally the prediction box with fused rich semantics is input into the prediction_head detection head for final prediction.

Images