Three-dimensional target detection method for improving multi-modal fusion

Abstract

The invention relates to a target detection technology, and particularly discloses an improved multi-modal fusion three-dimensional target detection method, which comprises the steps of extracting features of point clouds and images, defining a local attention correlation function for feature differences of each point cloud and points in the field, obtaining an edge perception graph of the point clouds, and further obtaining a center thermodynamic graph of an object from edge information. In an image and point cloud feature fusion stage, a depth perception BEV grid division strategy is used, image features are projected to an image-point cloud feature thermodynamic diagram after BEV grid fusion, the image-point cloud feature thermodynamic diagram is combined with a point cloud thermodynamic diagram, and an initial query point index is generated. And positioning a corresponding point in the point cloud BEV feature through the index, and sequentially performing attention fusion with the point cloud feature and the image feature to obtain a final fusion query point. According to the method, the spatial structure characteristics of the point cloud are utilized, the depth perception BEV grid is utilized to fuse the image information, and the advantages of different modes are combined, so that the detection precision and stability in the complex road condition are improved.

Description

Technical Field

[0001] This invention relates to target detection and autonomous driving technology, specifically to an improved three-dimensional target detection method using multimodal fusion. Background Technology

[0002] With the continuous improvement of chip performance and the rapid development of new energy vehicles, autonomous driving technology is entering a stage of rapid evolution. Currently, most new energy vehicles are equipped with driver assistance systems, which significantly enhance driving safety while improving driving convenience. To achieve higher levels of autonomous driving, improving the accuracy and stability of road target detection has become one of the key technologies. LiDAR and cameras are two important object detection sensors in autonomous driving. Due to the sparsity of point clouds, using LiDAR alone is clearly insufficient for robust 3D detection. For example, it is difficult to detect small or overlapping objects in LiDAR mode. In contrast, these objects remain clearly visible and distinguishable in high-resolution images. The complementary nature of point clouds and images prompts the design of detectors that leverage the advantages of both modalities.

[0003] Currently, one of the most representative methods in multimodal feature fusion is the cross-attention mechanism. Unlike traditional single-modal attention mechanisms, when dealing with inconsistent data formats across different modalities (such as point clouds and images), it is usually necessary to construct a unified spatial representation to guide the interaction and fusion of features from different modalities. Among these, the Bird's Eye View (BEV) space is widely used as a common representation domain for cross-modal feature fusion due to its spatial partitioning capabilities and representational advantages adapted to downstream tasks.

[0004] When initializing query points for BEV spatial localization, traditional center heatmap methods, while demonstrating good center localization capabilities in image tasks, have significant limitations in point cloud scenarios. First, heatmaps essentially estimate target centers based on pixel density. This method is more effective on regular, dense two-dimensional grids, but point cloud data is typically sparse and unstructured, making the definition of the center inherently unstable. Second, center heatmaps ignore the spatial geometry of the point cloud, relying solely on the distribution of points. This makes it difficult to fully utilize the local relationships and geometric features between points, leading to insufficient understanding of target morphology. The robustness of center heatmaps decreases significantly, especially in the presence of occlusion, small targets, or asymmetrical objects. Therefore, relying solely on center heatmaps can easily lead to localization errors or missed target detections.

[0005] When constructing the BEV space of an image, traditional methods typically employ a uniformly divided grid structure, i.e., dividing the entire space equidistantly with a fixed number of grids. However, this strategy does not consider the actual perception depth of the camera, resulting in an implicit variation in the actual size of each grid in physical space, significantly affected by the overall perception range. This approach has obvious drawbacks: First, uniform grids ignore the differences in importance between different regions in the perception task. In reality, the area near the vehicle is the most sensitive to perception accuracy, while distant areas are more tolerant. Second, with uniform division, distant areas waste a lot of computational resources without providing meaningful features, while near-field areas may lose details due to insufficient resolution. Uniform grids treat high-value and low-value areas uniformly in terms of computational resources, causing image information to be "averaged" and diluted in the near-field region, making it difficult to form a clear and focused response, ultimately affecting detection accuracy. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies and provide an improved three-dimensional target detection method based on multimodal fusion.

[0007] Technical solution: An improved multimodal fusion method for three-dimensional target detection according to the present invention includes the following steps:

[0008] Step 1: Construct a dataset consisting of point clouds and images. Extract image features from the images in the dataset using the ResNet+FPN network model, and extract point cloud features from the point clouds in the dataset using the SECOND+SECONDFPN network model.

[0009] Step 2: Based on the point cloud features obtained in Step 1, calculate the feature difference for each point cloud and its k nearest neighbors. According to feature difference To construct a local attention correlation function And scale to obtain attention relevance map ;

[0010] Step 3: Analyze the attention correlation map obtained in Step 2. Standard deviation is calculated to obtain the edge sensing map E. The center heatmap of the point cloud is obtained by taking the local mean of the edge sensing map E and subtracting it from the original edge sensing map. ;

[0011] Step 4: For the image features obtained in Step 1, combine their length and width dimensions to obtain the image BEV features. And construct a BEV mesh with depth perception capabilities. Project the image to obtain the corresponding BEV position code;

[0012] Simultaneously, the point cloud features obtained in step 1 are merged with their length and width dimensions to obtain the point cloud BEV features. And construct a uniform BEV mesh. Project the image to obtain the corresponding BEV position code;

[0013] Then, the image BEV features and point cloud BEV features, along with their respective corresponding BEV location codes, are fused using cross-attention. The fused features are then... The center heatmap of the fused features was calculated. ;

[0014] Step 5: Obtain the central heat map and central heat map Combined, the initial query point index is obtained from the 200 points with the highest values ​​(by adding the two heatmaps together and taking the highest value as the target point).

[0015] Next, the point cloud corresponding to the obtained index is found on the point cloud BEV features. Attention is first calculated with the point cloud BEV features to generate a query point with an initial prediction. Then, cross-attention is calculated with the image BEV features to generate a fused query point.

[0016] Finally, the fused query points are input into prediction_head for final prediction. The prediction classification includes the detected target type and the target's 3D bounding box.

[0017] Further, step 2 obtains the attention relevance map. The detailed method is as follows:

[0018] For the point cloud data after feature extraction, the last dimension is expanded to a length k. First, the Euclidean distance between each point cloud is calculated to find the k nearest neighbors of each point cloud. Based on the index, the features of these k points are found, and then a local attention correlation function is constructed. :

[0019] ;

[0020] Where Q and K are linearly varying layers, For the features of point cloud i, The features of the neighboring points of point cloud i are used; the correlation function is scaled to obtain the attention correlation map. ;

[0021] ;

[0022] Where d is the feature dimension.

[0023] Furthermore, an attention relevance map is obtained. Next, step 3 calculates the standard deviation on the neighborhood channel of each point cloud to obtain the edge-aware map. The larger the value, the more likely it is to be the center of the object;

[0024] ; In the above formula, dim=-1 refers to the last dimension;

[0025] Obtain edge-aware map Then, a local average is performed using 5×5 average pooling to obtain E. 1 The original edge-aware image and Subtraction yields the center score map G;

[0026] ;

[0027] Next, the center score map G is convolved once (with a kernel size of 1) to obtain its classification. To avoid destroying the original score, a center heatmap of the point cloud is obtained. ;

[0028] ;

[0029] in, For a convolution with 1 input channel and 10 output channels, , where the kernel size is 1.

[0030] Furthermore, in step 4, the BEV mesh shape is 1200×1200, the camera depth range of the dataset is set to 50 meters, a gradient-dense mesh is applied to the important foreground features in the first X meters, while a uniform mesh is used for the less important background features in the last 50-X meters. The formula for front-end mesh division is as follows:

[0031] ;

[0032] ;

[0033] ;

[0034] in, The minimum depth (usually 0), The number of grids for important areas, The total number of grids to divide the y-axis (1200); The magnitude of the y-axis direction of the i-th grid in the foreground (from near to far);

[0035] For the features in the last 50-X meters, a uniform grid is applied, and the remaining grid is used, as shown in the following formula:

[0036] ;

[0037] ;

[0038] in, The maximum depth is 50 meters, based on the camera's depth range. This represents the number of grid cells along the y-axis in the background. For the background The size of each grid cell along its y-axis;

[0039] Then, the image BEV features and point cloud BEV features, along with their corresponding BEV location codes, are input into cross-attention for calculation to obtain the center heatmap of the fused features. :

[0040] ;

[0041] ;

[0042] Here, heatmap is a two-layer convolution with 10 output channels, corresponding to the 10 categories in the dataset. BEV_L is the uniform BEV grid position encoding of the point cloud, and BEV_C is the BEV grid position encoding obtained through depth perception.

[0043] This invention utilizes the camera's perception range, with image features corresponding to different sizes of BEV (Body Image Module) grids for different perception ranges. After projecting image features into the depth-sensing BEV grid, the output is a feature in the same space as the point cloud BEV. Similarly, the height and width dimensions of the image are concatenated to obtain [BS, C, total_width], providing a positional code, which is then replaced by the BEV grid. The point cloud BEV features and the BEV positional code are input into a Transformer. During aggregation with point cloud queries, smaller grids yield more refined features. Smaller grids involve more and more refined attention interactions in the Transformer, resulting in richer extracted information. The projection process essentially merges dimensions and then provides a positional code for the BEV grid.

[0044] Further, in step 5, the heatmap of the point cloud is combined with the heatmap of the fused features to obtain a dense map, the index of the high-response points is obtained, and the index of the first 200 points on the BEV feature is set as the center position of the prediction box.

[0045] ;

[0046] The obtained query points are subjected to a multi-head attention mechanism with point cloud and image features respectively. Finally, the prediction box with rich semantics is input into the prediction_head detection head for final prediction.

[0047] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0048] (1) This invention proposes an edge-aware module that generates query points from the edge-aware map of a point cloud. It makes full use of the local geometric relationships between point clouds and, by constructing connections between points and their neighbors, mines regions where structural boundaries may exist. This approach not only preserves the three-dimensional spatial features of the point cloud but also accurately captures the contour information of the target.

[0049] Unlike traditional heatmap methods that rely solely on "central probability distribution," this invention starts from the edges, establishes a structure-aware inference path, and gradually aggregates back to the target region, thus overcoming the shortcomings of central heatmaps in spatial structure modeling. This edge-guided structure enhancement mechanism significantly improves the model's ability to identify small and occluded targets in complex scenes and provides a processing approach for point cloud data that better suits its sparse and unstructured characteristics.

[0050] (2) The depth-sensing BEV mesh of the present invention uses a higher resolution in the near-vehicle area, which effectively improves the fineness of image features and semantic expression ability. This spatial resolution allocation makes the information of key areas (such as the vehicle's periphery) richer and more accurate. When fused with radar and other modalities, the cross-attention of key areas is more discriminative, thereby significantly improving the response quality and focusing ability of the subsequent center heatmap at the target location, and enhancing the accuracy and stability of target center prediction.

[0051] (3) This invention combines edge perception technology with depth perception BEV mesh, utilizing the spatial structure of point cloud features while also taking advantage of the high resolution of image features. At the same time, the designed depth perception BEV mesh is not only effective in road recognition, but can also be used in other target recognition fields in a plug-and-play manner. Attached Figure Description

[0052] Figure 1 A flowchart for generating the initial query in this invention;

[0053] Figure 2 This is a schematic diagram of the point cloud edge perception module in the embodiment;

[0054] Figure 3 This is a schematic diagram of the depth-sensing BEV module in the embodiment;

[0055] Figure 4 This is a schematic diagram of the BEV mesh effect in the embodiment. Detailed Implementation

[0056] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0057] This invention extracts features from point clouds and images, defines a local attention correlation function for the feature difference between each point cloud and its neighboring point clouds, and calculates the standard deviation of the obtained correlation map to obtain an edge-aware map of the point clouds. Then, it extracts the center heatmap of the object from the edge information. In the image and point cloud feature fusion stage, a depth-aware BEV mesh partitioning strategy is used. When projecting image features onto the BEV mesh, higher spatial resolution is given to near-field regions to retain richer image detail information. The fused feature heatmap is then combined with the point cloud heatmap to generate an initial query point index. Using this index, the corresponding point is located in the point cloud BEV features, and attention fusion is performed sequentially with the point cloud features and image features to obtain the final fused query point.

[0058] like Figure 1 As shown, the improved multimodal fusion-based 3D target detection method of this embodiment includes the following steps:

[0059] Step 1: Construct a dataset consisting of point clouds and images. Extract image features using the ResNet+FPN network model and extract point cloud features using the SECOND+SECONDFPN network model.

[0060] Step 2: Based on the point cloud features obtained in Step 1, calculate the feature difference for each point cloud and its k nearest neighbors. According to feature difference To construct a local attention correlation function And scale to obtain attention relevance map ;

[0061] Step 3: Analyze the attention correlation map obtained in Step 2. Standard deviation is calculated to obtain the edge sensing map E. The center heatmap of the point cloud is obtained by taking the local mean of the edge sensing map E and subtracting it from the original edge sensing map. ;

[0062] Step 4: For the image features obtained in Step 1, combine their length and width dimensions to obtain the image BEV features. And construct a BEV mesh with depth perception capabilities. Project;

[0063] Simultaneously, the point cloud features obtained in step 1 are merged with their length and width dimensions to obtain the point cloud BEV features. And construct a uniform BEV mesh. Project;

[0064] Then, the image BEV features and point cloud BEV features, along with their respective corresponding BEV location codes, are fused using cross-attention. The fused features are then... The center heatmap of the fused features was calculated. ;

[0065] Step 5: Obtain the central heat map and central heat map Combined, the initial query point index is obtained from the 200 points with the highest values ​​(by adding the two heatmaps together and taking the highest value as the target point).

[0066] Next, the obtained index is used to find the corresponding point on the point cloud BEV feature. Attention is first calculated with the point cloud BEV feature to generate a query point with an initial prediction. Then, cross-attention is calculated with the image BEV feature to generate a fused query point.

[0067] Finally, the query point is input into prediction_head for final prediction. The prediction classification includes the detected target type and the target's 3D bounding box.

[0068] like Figure 2 As shown, step 2 obtains the attention correlation map. The detailed method is as follows:

[0069] For the point cloud data after feature extraction, the last dimension is expanded to a length of k. First, the Euclidean distance between each point cloud is calculated to find the k nearest neighbors of each point cloud. Based on the index, the features of these k points are found, and then a local attention correlation function is constructed. :

[0070] ;

[0071] Where Q and K are linearly varying layers, For the characteristics of point i, The features of the points adjacent to point i.

[0072] Scaling the correlation function yields an attention correlation graph. :

[0073] ;

[0074] Where d is the feature dimension;

[0075] Obtain attention correlation map Next, step 3 calculates the standard deviation on the neighborhood channel of each point cloud to obtain the edge-aware map. The larger the value, the more likely it is to be the center of the object;

[0076] ; In the above formula, dim=-1 refers to the last dimension;

[0077] After obtaining the edge-aware map E, a local average is performed on it using 5×5 average pooling to obtain E. 1 Then the original edge-aware image E and E 1 Subtraction yields the center score map G;

[0078] ;

[0079] Next, the center score map G is convolved once (with a kernel size of 1) to obtain its classification. To avoid destroying the original score, a center heatmap of the point cloud is obtained. ;

[0080] ;

[0081] in, For a convolution with 1 input channel and 10 output channels, , where the kernel size is 1.

[0082] like Figure 3 and Figure 4 As shown, the BEV mesh in this embodiment has a shape of 1200×1200. The camera depth range of the dataset is set to 50 meters. Gradient-dense meshes are used for important foreground features in the first X meters, while uniform meshes are used for less important background features in the last 50-X meters. The formula for front-end mesh division is as follows:

[0083] ;

[0084] ;

[0085] ;

[0086] in, The minimum depth (usually 0), The number of grids for important areas, The total number of grids for the y-axis is 1200; The magnitude of the y-axis direction of the i-th grid in the foreground (from near to far);

[0087] For the features in the last 50-X meters, a uniform grid is applied, and the remaining grid is used, as shown in the following formula:

[0088] ;

[0089] ;

[0090] in, The maximum depth is 50 meters, based on the camera's depth range. This represents the number of grid cells along the y-axis in the background. For the background The size of each grid cell along its y-axis;

[0091] Then, the image BEV features and point cloud BEV features, along with their corresponding BEV location codes, are input into cross-attention for calculation to obtain the center heatmap of the fused features. :

[0092] ;

[0093] ;

[0094] Here, heatmap is a two-layer convolution with 10 output channels, corresponding to the 10 categories in the dataset. BEV_L is the uniform BEV grid position encoding of the point cloud, and BEV_C is the BEV grid position encoding obtained through depth perception.

[0095] This invention utilizes the camera's perception range, with image features corresponding to different sizes of BEV (Browser Active Value) grids for different perception ranges. After projecting image features into the depth-sensing BEV grid, the output is a feature in the same space as the point cloud BEV. Similarly, the height and width dimensions of the image are concatenated to obtain [BS, C, total_width], and the positional encoding is replaced by the depth-sensing BEV grid. The point cloud BEV features and the BEV positional encoding are input into the Transformer together. During aggregation with point cloud queries, smaller grids yield more refined features. Smaller grids involve more and more refined attention interactions in the Transformer, resulting in richer extracted information. The projection process essentially merges dimensions and then provides a positional encoding for the BEV grid.

[0096] Step 5 combines the heatmap of the point cloud with the heatmap of the fused features to obtain a dense map, obtains the index of high-response points, and sets the index of the first 200 points on the BEV feature as the center position of the prediction box:

[0097] ;

[0098] The obtained query points are subjected to a multi-head attention mechanism with point cloud and image features respectively. Finally, the prediction box with rich semantics is input into the prediction_head detection head for final prediction.

[0099] To verify the technical effectiveness of this invention, this embodiment applies the method of this invention to perform 3D road target detection in the following scenarios. The experimental dataset used in this embodiment is nuScenes, a large-scale autonomous driving benchmark containing 10,000 driving scenarios, divided into 700, 150, and 150 scenarios for training, validation, and testing, respectively. For detection, nuScenes defines an evaluation protocol, including the nuScenes Detection Score (NDS), mean accuracy (mAP), and five true measure (TP) metrics: mean translation error (mATE), mean scale error (mASE), mean orientation error (mAOE), mean velocity error (mAVE), and mean attribute error (mAAE). In this embodiment, mAP is the average of the mean accuracies of ten classes at distance thresholds of 0.5m, 1m, 2m, and 4m. NDS is a weighted combination of mAP, mATE, mASE, mAOE, mAVE, and mAAE. Specific comparison results are shown in Table 1.

[0100] Table 1. Accuracy comparison with baseline network on the validation set.

[0101]

[0102] In the table above, Car and Ped represent the accuracy of detecting cars and pedestrians.

[0103] The experimental results show that this invention utilizes the spatial structure characteristics of point clouds and integrates image information from depth-sensing BEV meshes, combining the advantages of different modalities to improve detection accuracy and stability in complex road conditions.

Claims

1. An improved multimodal fusion method for three-dimensional target detection, characterized in that, Includes the following steps: Step 1: Construct a dataset consisting of point clouds and images. Extract image features from the images in the dataset using the ResNet+FPN network model, and extract point cloud features from the point clouds in the dataset using the SECOND+SECONDFPN network model. Step 2: Based on the point cloud features obtained in Step 1, calculate the feature difference for each point cloud and its k nearest neighbors. According to feature difference To construct a local attention correlation function And scale to obtain attention relevance map ; Step 3: Analyze the attention correlation map obtained in Step 2. Standard deviation is calculated to obtain the edge sensing map E. The center heatmap of the point cloud is obtained by taking the local mean of the edge sensing map E and subtracting it from the original edge sensing map. ; Step 4: For the image features obtained in Step 1, combine their length and width dimensions to obtain the image BEV features. And construct a BEV mesh with depth perception capabilities. Project the image to obtain the corresponding BEV position code; Simultaneously, the point cloud features obtained in step 1 are merged with their length and width dimensions to obtain the point cloud BEV features. And construct a uniform BEV mesh. Project the image to obtain the corresponding BEV position code; Then, the image BEV features and point cloud BEV features, along with their respective corresponding BEV location codes, are fused using cross-attention. The fused features are then... The center heatmap of the fused features was calculated. ; Step 5: Obtain the central heat map and central heat map Combined, obtain the initial query point index from the 200 points with the highest values; Then, the point cloud corresponding to the obtained index is found on the point cloud BEV feature. Attention is first calculated with the point cloud BEV feature to generate a query point with initial prediction. Then, cross-attention is calculated with the image BEV feature to generate a fused query point. Finally, the fused query points are input into prediction_head for final prediction. The prediction classification includes the detected target type and the target's 3D bounding box.

2. The improved multimodal fusion-based three-dimensional target detection method according to claim 1, characterized in that, Step 2 obtains the attention correlation map. The detailed method is as follows: For the point cloud data after feature extraction, the last dimension is expanded to a length k. First, the Euclidean distance between each point cloud is calculated to find the k nearest neighbors of each point cloud. Based on the index, the features of these k points are found, and then a local attention correlation function is constructed. : ； Where Q and K are linearly varying layers, For the features of point cloud i, Features of the neighboring points of point cloud i; Scaling the correlation function yields the attention correlation map. ; ； Where d is the feature dimension.

3. The improved multimodal fusion three-dimensional target detection method according to claim 1 or 2, characterized in that, Obtain attention correlation map Next, step 3 calculates the standard deviation on the neighborhood channel of each point cloud to obtain the edge-aware map. ; ； In the above formula, dim=-1 refers to the last dimension; Obtain edge-aware map Then, a local average is performed using 5×5 average pooling to obtain E. 1 The original edge-aware image and Subtraction yields the center score map G; ； Next, the center score map G is convolved once to obtain the center heatmap of the point cloud. ; ； in, For a convolution with 1 input channel and 10 output channels, , where the kernel size is 1.

4. The improved multimodal fusion-based three-dimensional target detection method according to claim 1, characterized in that, In step 4, the BEV mesh has a shape of 1200×1200. The camera depth range of the dataset is set to 50 meters. Gradient-dense meshes are used for important foreground features in the first X meters, while uniform meshes are used for less important background features in the last 50-X meters. The formula for front-end mesh division is as follows: ； ； ； in, Minimum depth, The number of grids for important areas, The total number of grids to divide the y-axis; The size of the i-th grid in the foreground along the y-axis; For the features in the last 50-X meters, a uniform grid is applied, and the remaining grid is used, as shown in the following formula: ； ； in, Maximum depth; This represents the number of grid cells along the y-axis in the background. For the background The size of each grid cell along its y-axis; Then, the image BEV features and point cloud BEV features, along with their corresponding BEV location codes, are input into cross-attention for calculation to obtain the center heatmap of the fused features. : ； ； Here, heatmap is a two-layer convolution with 10 output channels, corresponding to the 10 categories in the dataset. BEV_L is the uniform BEV grid position encoding of the point cloud, and BEV_C is the BEV grid position encoding obtained through depth perception.

5. The improved multimodal fusion-based three-dimensional target detection method according to claim 1, characterized in that, Step 5 combines the heatmap of the point cloud with the heatmap of the fused features to obtain a dense map, and then obtains the index of high response points. The index on the BEV feature of the first 200 point clouds is set as the center position of the prediction box. ； The obtained query points are subjected to a multi-head attention mechanism with point cloud and image features respectively. Finally, the prediction box with rich semantics is input into the prediction_head detection head for final prediction.

Images