Multimodal detection method based on local density perception and dynamic sparse attention

Abstract

The application discloses a multi-modal detection method based on local density perception and dynamic sparse attention, and belongs to the technical field of intelligent driving and computer vision. The method is aimed at the non-uniform characteristics of point cloud spatial distribution, and an adaptive neighborhood selection mechanism based on local density estimation is constructed. The neighborhood size of each point is dynamically adjusted according to the point density of different regions, so that a smaller neighborhood is used in a high-density region to avoid information redundancy, and a larger neighborhood is used in a low-density region to enhance the context perception ability. On this basis, a multi-source context-driven neighborhood reliability evaluation module is further designed, and factors such as local geometric structure, image projection quality and semantic consistency are comprehensively considered to generate the confidence weight of each neighborhood point. Finally, a soft mask guided by the reliability weight is introduced into the self-attention mechanism to realize the modeling of conditional sparse attention, and the invalid information interaction between low-confidence neighbors is effectively inhibited.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of intelligent driving and computer vision, and in particular relates to a multimodal detection method based on local density perception and dynamic sparse attention. Background Technology

[0002] With the development of autonomous driving technology, single sensors are no longer sufficient to meet the perception needs in complex traffic scenarios. LiDAR can provide accurate 3D geometric information, while cameras possess rich texture and semantic details. Therefore, multimodal fusion has become a key path to improve the performance of 3D target detection. In recent years, graph-based cross-modal feature alignment methods such as GraphAlign have made significant progress. They construct a point cloud nearest neighbor graph and project image features onto the locations of point cloud neighbors to achieve spatial alignment and information complementarity. However, existing methods generally use a fixed neighborhood size (e.g., or ) for feature aggregation, ignoring the significant non-uniformity of the spatial distribution of LiDAR point clouds: high point density in near-distance areas and sparse points in far-distance areas. Using large values ​​in high-density areas can easily introduce redundant or even noisy neighbors, leading to over-smoothing of features; while in low-density areas, small values ​​cannot capture enough contextual information, affecting the detection capability of small targets, especially pedestrians at a distance. In addition, during cross-modal alignment, some projection areas may be located at image edges, occluded areas, or blurred areas, resulting in a decrease in the reliability of image features. If these are still given high attention weights, incorrect fusion bias will be introduced. Existing methods lack a dynamic evaluation mechanism for neighborhood quality, making it difficult to achieve fine-grained attention control. For example, GraphAlign uses a fixed nearest neighbor graph structure, which struggles to capture effective context in sparse, distant regions, while leading to overly smoothed features in dense, near-field regions. Furthermore, its attention mechanism does not consider quality differences in image projection areas, lacking the ability to suppress occluded or blurred regions. Therefore, designing a fusion strategy that can adapt to the spatial heterogeneity of point clouds while suppressing low-confidence interactions has become a key challenge in current multimodal 3D detection. Summary of the Invention

[0003] The purpose of this invention is to overcome the problem of unbalanced feature fusion caused by fixed neighborhood selection and homogenized attention mechanism in the prior art, and to propose a three-dimensional target multimodal detection method based on local density perception and dynamic sparse attention.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal detection method based on local density sensing and dynamic sparse attention, comprising the following steps:

[0005] Step S1: Use the KD-Tree spatial indexing structure to perform a neighborhood search on the input raw LiDAR point cloud data and the synchronously acquired multi-view camera images; for each point in the point cloud, calculate the number of points contained in its local neighborhood within a preset radius R, which is defined as the local density index of that point. ;

[0006] Step S2: Based on statistical analysis of the point cloud density distribution patterns in the KITTI and nuScenes datasets, establish... To neighborhood size Piecewise linear mapping function, to implement Adaptive regulation;

[0007] Step S3: Adaptive determination based on step S2 Re-execute the nearest neighbor query to obtain the results for each point. Furthermore, a domain set is constructed, and using the camera intrinsic and extrinsic parameter matrix, the center point of the point cloud and its neighboring points are projected onto the image plane to obtain the corresponding cloud features and image feature vectors, thereby achieving dimensional alignment and obtaining the preliminary fused features.

[0008] Step S4: Construct a five-dimensional context feature vector Used to quantify the alignment confidence of each neighboring point;

[0009] Step S5: Convert the five-dimensional context feature vector from step S4... Input a lightweight fully connected network to generate a neighborhood reliability weight vector. ;

[0010] Step S6: Construct a self-attention feature enhancement module, introducing the feature generated in step S5. Construct an attention mask matrix Interactions are only preserved when both neighbors have high confidence levels.

[0011] Step S7: Calculate the attention score matrix from the standard attention mechanism. and Perform element-wise multiplication to obtain the modulated attention distribution, and then perform Softmax normalization on each row;

[0012] Step S8: Use the attention distribution modulated and normalized in step S7 to perform weighted aggregation on the preliminary fusion features of step S3, and output the final enhanced features.

[0013] Furthermore, in step S1 The mathematical expression is: ,in, Indicates the input point cloud set. For the first The three-dimensional coordinates of the points For the first j The three-dimensional coordinates of the points The radius of the spherical query.

[0014] Furthermore, the piecewise linear mapping function in step S2 is: when When this occurs, it indicates that the point is in an extremely sparse region, and a setting is made. ;when At that time, it was considered that the density of the area was moderate, and a setting was made. ;when When a high-density area is identified, it is set up. The formal representation of this mapping relationship is as follows: .

[0015] Furthermore, in step S3, the image feature vectors of each neighboring point are obtained by bilinear interpolation in the depth feature map extracted by the 2D backbone network, forming a cross-modal neighborhood feature set. Point cloud features were extracted using a 3D voxel encoder and then copied and expanded to... ,in Indicates the first i Each center point A cross-modal image feature set consisting of neighboring points, where each row is an image feature vector with dimension C; Indicates the first i Each center point The feature matrix formed by copying and expanding the point cloud features corresponding to each neighbor point has the same dimensions as the image features and is used to achieve dimension alignment.

[0016] Furthermore, the five-dimensional context feature vector of step S4 Includes: First component The second component is the ratio of the average Euclidean distance from the neighboring points to the center point to the standard deviation of the distance; The minimum eigenvalue obtained from principal component analysis (PCA) of the neighborhood points; the third component Mean Sobel gradient response of the projected image patch: fourth component The category prediction entropy value output by the pre-trained semantic segmentation model; the fifth component. It is the normalized value of the area of ​​the minimum bounding rectangle of K projection points.

[0017] Furthermore, the lightweight fully connected network in step S5 contains two hidden layers, uses LeakyReLU as the activation function, and uses the Sigmoid function for the output layer.

[0018] Furthermore, the neighborhood reliability weight vector in step S5 The calculation process is expressed as follows: ,in This represents the learnable weight matrix of the first fully connected layer. This represents the learnable weight matrix of the second fully connected layer. b 1 This indicates the bias top of the first fully connected layer. b 2 This indicates the bias top of the second fully connected layer. This is the Sigmoid function, which compresses the output to the range (0,1).

[0019] Furthermore, the attention mask matrix in step S6 The elements satisfy ,in k Represents the attention mask matrix The row index, corresponding to the first row. k Neighboring points, l Represents the attention mask matrix The column index, corresponding to the first l 1 neighboring point.

[0020] Furthermore, the expression for the modulated attention distribution in step S7 is: The normalized probability distribution is expressed as follows: ,in This represents the normalized attention probability distribution ultimately used for aggregation, derived from... The result is obtained by performing Softmax on each row; The first term of the modulated attention matrix is... k Line, i.e., the first k The unnormalized, modulated attention score of each point to all its neighbors; Indicating the first modulated attention distribution k Line 1 l The element of the column represents the first element. k The neighbor to the first l Modulated attention score of each neighbor.

[0021] Furthermore, step S8 outputs the enhanced feature representation: ,in To initially fuse features, the final output is As input to subsequent detection heads, it completes the 3D bounding box regression and classification task, whereby... The enhanced feature representation for the final output is used as input for subsequent detection heads; To initially fuse features, point cloud features and image features Obtained by splicing or adding together.

[0022] Through the above design scheme, the present invention can bring the following beneficial effects:

[0023] This invention addresses the non-uniformity of point cloud spatial distribution by constructing an adaptive neighborhood selection mechanism based on local density estimation. This mechanism dynamically adjusts the neighborhood size of each point according to the point density in different regions, using smaller neighborhoods in high-density areas to avoid information redundancy and larger neighborhoods in low-density areas to enhance context awareness. Furthermore, a multi-source context-driven neighborhood reliability assessment module is designed, comprehensively considering factors such as local geometry, image projection quality, and semantic consistency to generate confidence weights for each neighboring point. Finally, a soft mask guided by reliability weights is introduced into the self-attention mechanism to achieve conditional sparse attention modeling, effectively suppressing invalid information interaction between low-confidence neighbors and improving the accuracy and robustness of feature fusion. The entire method requires no additional labeled data, is computationally efficient, has clear geometric and physical meaning, and can be seamlessly integrated into mainstream multimodal detection frameworks. Attached Figure Description

[0024] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0025] Figure 1 This is a flowchart of the multimodal detection method based on local density sensing and dynamic sparse attention according to the present invention;

[0026] Figure 2 This is a network framework diagram of the multimodal detection method based on local density sensing and dynamic sparse attention in this invention;

[0027] Figure 3 This is a schematic diagram of the neighborhood reliability assessment module of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of the dynamic sparse attention fusion module of the present invention. Detailed Implementation

[0029] The present invention will be further illustrated below with reference to specific embodiments, but the embodiments do not limit the present invention in any way. Example

[0030] The specific implementation process of this method includes the following steps:

[0031] A. Acquire raw 3D point cloud data from the vehicle-mounted LiDAR and simultaneously acquired multi-view color images (RGB images), ensuring that both have been accurately calibrated in time and space. Input the point cloud data into a KD-Tree spatial index structure to accelerate subsequent neighborhood query operations.

[0032] The KD-Tree spatial index structure is a highly efficient multi-dimensional spatial index structure. By alternately partitioning the multi-dimensional space to construct a binary tree, it can quickly achieve nearest neighbor search and range query, which is a prior art in the field. A 3D backbone is used to process point cloud data, and a 2D backbone is used to process RGB images.

[0033] Specifically, for each point cloud point Set a fixed radius around it. The local density index of a point is defined as the number of all points within its spherical neighborhood, excluding itself. Its mathematical expression is as follows:

[0034]

[0035] in This represents the current frame's point cloud set. This density index reflects the density of the local point distribution and forms the basis for subsequent adaptive neighborhood control.

[0036] B. Based on the statistical analysis results of the KITTI and nuScenes datasets, establish a model from local density. arrive Number of neighbors The piecewise linear mapping relationship is used to achieve differentiated neighborhood construction for different spatial regions.

[0037] Specifically, when When this occurs, it indicates that the point is in an extremely sparse region. To enhance context awareness, a setting is made. ;when At that time, it was considered that the density of the region was moderate, balancing computational efficiency and feature expressiveness, and the following settings were adopted. ;when When a region is identified as high-density, to prevent over-smoothing of features and interference from redundant information, a setting is implemented. This mapping rule can be formally expressed as:

[0038] This strategy requires no additional training, has good generalization ability, and is applicable to various urban road and highway scenarios.

[0039] KITTI and nuScenes are both highly authoritative large-scale public datasets in the field of autonomous driving. They are often used to train, test, and evaluate core autonomous driving algorithms such as 3D object detection, trajectory estimation, and sensor fusion, and belong to the existing technologies in the field.

[0040] C. Based on the above adaptive determination Value, re-execute Nearest neighbor search to obtain each point dynamic neighborhood set The center point and its neighbors are projected onto the image plane using the camera's intrinsic and extrinsic parameter matrices to obtain the corresponding pixel coordinates. Then, image feature vectors of each neighboring point are obtained from the depth feature map extracted from the pre-trained ResNet-50 using bilinear interpolation, forming a cross-modal neighborhood image feature set. Simultaneously, point cloud features are extracted using a VoxelNet encoder and then copied and expanded to... This completes geometric alignment and cross-modal alignment.

[0041] The Neighbor refers to... The number of nearest neighbors is dynamically determined. Indicates the first i Each center point A cross-modal image feature set consisting of neighboring points, where each row is an image feature vector with dimension C; Indicates the first i Each center point The feature matrix, formed by copying and expanding the point cloud features corresponding to each neighboring point, has the same dimensions as the image features and is used to achieve dimension alignment. The camera intrinsic and extrinsic parameter matrix is ​​a mathematical model of a set of transformation rules, which belongs to existing technology.

[0042] D. To further improve the fusion quality, a neighborhood reliability assessment module is designed to quantify the reliability of each neighboring point in cross-modal alignment. Figure 3 This is a schematic diagram of the neighborhood reliability assessment module of the present invention;

[0043] Specifically, this involves constructing a five-dimensional context feature vector. Each component represents different prior information: the first component The first component is the ratio of the average Euclidean distance from neighboring points to the center point to its standard deviation, reflecting the local clustering tendency; the second component... The minimum eigenvalue obtained from principal component analysis (PCA) reflects surface curvature and smoothness; the third component... The fourth component is the mean Sobel gradient response of the projected image patch, which measures the local texture sharpness; The fifth component is the category prediction entropy value output by the pre-trained semantic segmentation model; the lower the entropy, the clearer the semantics. It is the normalized value of the area of ​​the minimum bounding rectangle of all projection points, describing the compactness of the spatial distribution.

[0044] Sobel gradient is a classic edge detection algorithm. Its core function is to locate regions in an image where brightness changes drastically by calculating the rate of change (gradient) of the gray values ​​of image pixels. These regions usually correspond to the edges of objects (such as contours, boundaries, etc.) and are existing technologies.

[0045] E. Input the above five-dimensional context feature vector into a lightweight fully connected (MLP) network containing two hidden layers with 64 and 64 neurons respectively. The activation function is LeakyReLU, and the output layer uses the Sigmoid function to generate a neighborhood reliability weight vector. The calculation process can be expressed as follows:

[0046] in, This represents the learnable weight matrix of the first fully connected layer. This represents the learnable weight matrix of the second fully connected layer. b 1 This indicates the bias top of the first fully connected layer. b 2 This indicates the bias top of the second fully connected layer. The sigmoid function compresses the output to the range (0,1). This module has less than 50K parameters, allowing for efficient optimization during end-to-end training without significantly increasing the inference burden.

[0047] F. In the subsequent self-attention feature enhancement module, a soft mask mechanism generated by reliability weights is introduced. Figure 4 This is a schematic diagram of the structure of the dynamic sparse attention fusion module of the present invention. The input layer takes point cloud features and image features as input. In the intermediate layer, a reliability weight vector (RVW) for each neighborhood is generated through neighborhood reliability evaluation. Based on this, a soft mask is constructed to modulate the original attention, thereby effectively suppressing the interaction between low-confidence neighbors. As a result, the output layer obtains the enhanced feature representation.

[0048] Interactions between high-confidence neighbors are represented by the product of two reliability weights: the product is large only when both are high, and the interaction is preserved. It is a joint high-confidence judgment method, and the judgment is implicit, relative and learnable.

[0049] Specifically: for the first Construct an attention mask matrix using points. Its elements are determined by the outer product of the weights of their corresponding neighbors:

[0050]

[0051] in k Represents the attention mask matrix The row index, corresponding to the first row. k Neighboring points, l Represents the attention mask matrix The column index, corresponding to the first l 1 neighboring point.

[0052] The mask matrix reflects the joint confidence between neighbor pairs: their interaction is preserved only when both neighbors have high confidence, thus achieving fine-grained sparsity control.

[0053] G. The attention score matrix in the standard self-attention mechanism With soft mask Performing element-wise multiplication, we obtain the modulated attention distribution:

[0054] Then, Softmax normalization is performed on each row to ensure that it is a valid probability distribution:

[0055]

[0056] in This represents the normalized attention probability distribution ultimately used for aggregation, derived from... The result is obtained by performing Softmax on each row; The first term of the modulated attention matrix is... k Line, i.e., the first k The unnormalized, modulated attention score of each point to all its neighbors; Indicating the first modulated attention distribution k Line 1 l The element of the column represents the first element. k The neighbor to the first l Modulated attention score of each neighbor.

[0057] Standard attention score matrix The unnormalized fraction matrix obtained by multiplying the query by the key.

[0058] This operation enables the attention mechanism to dynamically ignore invalid interactions between low-confidence neighbors, improving the accuracy of feature aggregation.

[0059] H. Utilize the modulated attention weights to perform weighted aggregation of the neighborhood fusion features, and output the enhanced feature representation:

[0060]

[0061] in This represents the initial fusion characteristics.

[0062] I. Final Output It serves as input for subsequent detection heads, used to complete 3D bounding box regression and classification tasks.

[0063] in The enhanced feature representation for the final output is used as input for subsequent detection heads; To initially fuse features, point cloud features and image features Obtained by splicing or adding together.

Claims

1. A multimodal detection method based on local density sensing and dynamic sparse attention, characterized in that, Includes the following steps: Step S1: Use the KD-Tree spatial indexing structure to perform a neighborhood search on the input raw LiDAR point cloud data and the synchronously acquired multi-view camera images; for each point in the point cloud, calculate the number of points contained in its local neighborhood within a preset radius R, which is defined as the local density index of that point. ; Step S2: Based on statistical analysis of the point cloud density distribution patterns in the KITTI and nuScenes datasets, establish... To neighborhood size Piecewise linear mapping function, to implement Adaptive regulation; Step S3: Adaptive determination based on step S2 Re-execute the nearest neighbor query to obtain the results for each point. Furthermore, a domain set is constructed, and using the camera intrinsic and extrinsic parameter matrix, the center point of the point cloud and its neighboring points are projected onto the image plane to obtain the corresponding point cloud features and image feature vectors, thereby achieving dimensional alignment and obtaining the preliminary fused features. Step S4: Construct a five-dimensional context feature vector Used to quantify the alignment confidence of each neighboring point; the five-dimensional context feature vector Includes: First component The second component is the ratio of the average Euclidean distance from the neighboring points to the center point to the standard deviation of the distance; The minimum eigenvalue obtained from principal component analysis (PCA) of the neighborhood points; the third component Mean Sobel gradient response of the projected image patch: fourth component The category prediction entropy value output by the pre-trained semantic segmentation model; the fifth component. The normalized value of the area of ​​the minimum bounding rectangle for K projection points; Step S5: Convert the five-dimensional context feature vector from step S4... Input a lightweight fully connected network to generate a neighborhood reliability weight vector. ; Step S6: Construct a self-attention feature enhancement module, introducing the feature generated in step S5. Construct an attention mask matrix The interaction is preserved only when both neighbors have high confidence; the attention mask matrix The elements satisfy ,in k Represents the attention mask matrix The row index, corresponding to the first row. k Neighboring points, l Represents the attention mask matrix The column index, corresponding to the first l 1 neighboring point; Step S7: Calculate the attention score matrix from the standard attention mechanism. and Perform element-wise multiplication to obtain the modulated attention distribution, and then perform Softmax normalization on each row; Step S8: Use the attention distribution modulated and normalized in step S7 to perform weighted aggregation on the preliminary fusion features of step S3, and output the final enhanced features.

2. The multimodal detection method based on local density sensing and dynamic sparse attention according to claim 1, characterized in that: In step S1 The mathematical expression is: ,in, Indicates the input point cloud set. For the first The three-dimensional coordinates of the points For the first j The three-dimensional coordinates of the points The radius of the spherical query.

3. The multimodal detection method based on local density sensing and dynamic sparse attention according to claim 1, characterized in that, The piecewise linear mapping function in step S2 is: when When this occurs, it indicates that the point is in an extremely sparse region, and a setting is made. ;when At that time, it was considered that the density of the area was moderate, and a setting was made. ;when When it is determined to be a high-density area, set The formal representation of this mapping relationship is as follows: .

4. The multimodal detection method based on local density sensing and dynamic sparse attention according to claim 1, characterized in that: In step S3, the image feature vectors of each neighboring point are obtained by bilinear interpolation in the depth feature map extracted by the 2D backbone network, forming a cross-modal neighborhood feature set. Point cloud features were extracted using a 3D voxel encoder and then copied and expanded to... ,in Indicates the first i Each center point A cross-modal image feature set consisting of neighboring points, where each row is an image feature vector with dimension C; Indicates the first i Each center point The feature matrix formed by copying and expanding the point cloud features corresponding to each neighbor point has the same dimensions as the image features and is used to achieve dimension alignment.

5. The multimodal detection method based on local density sensing and dynamic sparse attention according to claim 1, characterized in that: The lightweight fully connected network in step S5 contains two hidden layers, uses LeakyReLU as the activation function, and uses the Sigmoid function for the output layer.

6. The multimodal detection method based on local density sensing and dynamic sparse attention according to claim 1, characterized in that, The neighborhood reliability weight vector in step S5 The calculation process is expressed as follows: ,in This represents the learnable weight matrix of the first fully connected layer. This represents the learnable weight matrix of the second fully connected layer. b 1 This indicates the bias top of the first fully connected layer. b 2 This indicates the bias top of the second fully connected layer. This is the Sigmoid function, which compresses the output to the range (0,1).

7. The multimodal detection method based on local density sensing and dynamic sparse attention according to claim 1, characterized in that, The expression for the modulated attention distribution in step S7 is: The normalized probability distribution is expressed as follows: ,in, This represents the normalized attention probability distribution ultimately used for aggregation, derived from... The result is obtained by performing Softmax on each row; The first term of the modulated attention matrix is... k Line, i.e., the first k The unnormalized, modulated attention score of each point to all its neighbors; Indicating the first modulated attention distribution k Line number l The element of the column represents the first element. k The neighbor to the first l Modulated attention score of each neighbor.

8. The multimodal detection method based on local density sensing and dynamic sparse attention according to claim 1, characterized in that, Step S8 outputs the enhanced feature representation: ,in To initially fuse features, the final output is As input to subsequent detection heads, it completes the 3D bounding box regression and classification task, whereby... The enhanced feature representation for the final output is used as input for subsequent detection heads; To initially fuse features, point cloud features and image features Obtained by splicing or adding together.

Images