A road garbage detection method and system based on multi-modal information fusion

Abstract

The application discloses a kind of road garbage detection method and system based on multi-modal information fusion, method includes: feature extraction multi-scale image feature to camera two-dimensional image;Feature extraction is carried out to LiDAR point cloud data and obtains BEV feature map;Multi-scale feature and BEV feature map are fused to form multi-modal BEV fusion feature, and generate class perception object query vector;Class perception object query vector and BEV feature map are input to the first layer Transformer decoder and obtain enhanced query vector and initial 3D boundary box prediction vector;Multi-scale image feature, enhanced query vector and initial 3D boundary box prediction vector are input to the second layer Transformer decoder, and output garbage recognition result, the present application makes up the information loss of single mode data, can more comprehensively capture target information, significantly improve the adaptability of detection system under various environmental conditions.

Description

Technical Field

[0001] This invention belongs to the field of visual target detection technology, specifically relating to a road waste detection method and system based on multimodal information fusion. Background Technology

[0002] With the development of urbanization and autonomous driving technology, unmanned sanitation vehicles have become an important direction for upgrading sanitation operations, and the accurate and real-time detection of garbage on the road by the environmental perception system is the core prerequisite for its efficient and safe operation.

[0003] As a core technology of intelligent sanitation sensing systems, road litter detection faces numerous technical challenges due to the characteristics of litter itself and the complex road environment: First, the target size is extremely small; beverage bottles, paper scraps, etc., often account for less than 32×32 pixels in the image, and the point cloud contains only a few dozen sparse points, making feature extraction difficult. Second, occlusion interference is severe; litter is easily obscured by road textures, shadows, and foreign objects, resulting in incomplete feature information. Third, there are significant differences in categories; litter has diverse shapes, colors, and materials, requiring high generalization ability of the model. Fourth, real-time requirements are stringent; sanitation vehicle operations require high frame rate detection results to support subsequent action planning.

[0004] To address the inherent limitations of visual sensors in adverse weather and lighting conditions, the industry has adopted a technology approach that integrates cameras and LiDAR. Existing fusion detection models mostly employ hard correlation fusion strategies, which are highly sensitive to sensor calibration accuracy and image quality, exhibiting poor robustness. Furthermore, some models have excessively large parameter sets, making them difficult to adapt to the limited computing resources available on vehicles, thus restricting their engineering applications. Therefore, there is an urgent need for a road debris detection method that can achieve depth-adaptive fusion of cameras and LiDAR, achieve high detection accuracy, and is suitable for vehicle deployment. Summary of the Invention

[0005] This invention provides a road waste detection method and system based on multimodal information fusion. It utilizes a camera to capture rich texture and color semantic information, and uses a lidar to provide accurate 3D spatial geometric information. Deep fusion is performed at the feature layer to make up for the information loss of single-modal data, which can capture target information more comprehensively and significantly improve the adaptability of the detection system under various environmental conditions.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The first aspect of this invention provides a road waste detection method based on multimodal information fusion, comprising:

[0008] Acquire two-dimensional images of the road surface and LiDAR point cloud data generated by scanning the road surface using vehicle-mounted cameras and LiDAR, and perform joint calibration and time alignment on the two-dimensional images of the road surface and the LiDAR point cloud data;

[0009] A combination of ResNet-50 backbone network and Feature Pyramid Network (FPN) is used to extract multi-scale image features from 2D images from the camera; and a sparse convolution backbone network based on sparse embedded convolution detection (SECOND) is used to extract features from LiDAR point cloud data to obtain BEV feature maps.

[0010] Multi-scale image features are transformed from the camera coordinate system to the lidar coordinate system, and multi-scale features and BEV feature maps are fused to form multimodal BEV fusion features;

[0011] The class heatmaps are predicted for the multimodal BEV fusion features and BEV feature maps respectively, and then fused by average to obtain the fusion heatmap; the top N fusion heatmaps of all classes are selected by local maxima, and class embedding is used to generate class-aware object query vectors;

[0012] The category-aware object query vector and BEV feature map are input into the first-layer Transformer decoder to obtain the enhanced query vector and the initial 3D bounding box prediction vector; the multi-scale image features, the enhanced query vector, and the initial 3D bounding box prediction vector are input into the second-layer Transformer decoder to output the garbage identification result.

[0013] Furthermore, a combination of the ResNet-50 backbone network and the feature pyramid network is used to extract multi-scale image features from the camera's 2D images, specifically including:

[0014] The 2D image from the camera is input into the image preprocessing layer. The image preprocessing layer performs preliminary feature extraction by using a 2D convolutional layer, a 2D batch normalization layer, a ReLU activation function, and a 2D max pooling layer to obtain a preprocessed feature map.

[0015] The preprocessed feature maps are input into the ResNet-50 backbone network and sequentially processed through four stages of Bottleneck residual block stacking from Stage 1 to Stage 4 to obtain backbone feature maps containing information from different receptive fields. ;

[0016] Main feature map The input to the feature pyramid network yields multi-scale image features, expressed as follows:

[0017]

[0018] In the formula, The output features of the feature pyramid network; for convolution; for convolution; This indicates an upsampling operation.

[0019] Furthermore, Stage 1 consists of three stacked Bottleneck residual blocks and outputs the backbone feature map. Stage 2 consists of 4 Bottleneck residual blocks and outputs the backbone feature map. Stage 3 consists of 6 Bottleneck residual blocks, which form the output backbone feature map. Stage 4 consists of three Bottleneck residual blocks, which form the output backbone feature map. The Bottleneck residual block contains 1×1 convolutions, 3×3 convolutions, and 1×1 convolutions.

[0020] Furthermore, a sparse convolutional backbone network based on sparse embedded convolutional detection is used to extract features from LiDAR point cloud data to obtain BEV feature maps, specifically including:

[0021] The LiDAR point cloud data is input into the voxel feature extractor for 3D spatial voxel mesh division. Based on the three-dimensional coordinates of each point in the LiDAR point cloud data, it is assigned to the corresponding voxel. T points are randomly sampled for each non-empty voxel to obtain the sampled voxel in-point set.

[0022] The set of voxel in-points is input into the VFE layer, and each point is processed by a fully connected network consisting of a linear layer, a BatchNorm layer, and a ReLU activation function to obtain point-by-point feature vectors. The point-by-point feature vectors are then processed by point-by-point MaxPool operations to obtain local aggregated features. The local aggregated features are concatenated with the point-by-point feature vectors, and then processed by the first fully connected layer and MaxPooling aggregation to obtain sparse voxel features.

[0023] The sparse voxel features are input into a sparse convolution backbone network based on sparse embedded convolution detection. In the first stage, the sparse voxel features are processed sequentially through three sub-manifold sparse convolution modules to obtain deep features. The deep features are then processed by Z-downsampling through one ordinary sparse convolution module to obtain sparse voxel compressed features.

[0024] In the second stage, the sparse voxel compressed features are sequentially processed through 5 layers of submanifold sparse convolution modules for depth extraction, and then processed through 1 layer of ordinary sparse convolution module to complete Z-axis compression to obtain the BEV feature map. The submanifold sparse convolution module includes SubM Conv3D (submanifold sparse 3D convolution), BatchNorm3D, and ReLU activation; the ordinary sparse convolution module includes Sparse Conv3D (sparse convolution), BatchNorm3D, and ReLU activation.

[0025] Furthermore, the multi-scale image features are transformed from the camera coordinate system to the LiDAR coordinate system, specifically including:

[0026]

[0027] In the formula, These are the coordinates of pixels in the camera coordinate system within multi-scale image features; These are the coordinates of pixels in the LiDAR coordinate system within the multi-scale image features. This is the rotation matrix from the camera coordinate system to the lidar coordinate system; This is the translation vector from the camera coordinate system to the lidar coordinate system.

[0028] Furthermore, multi-scale features and BEV feature maps are fused to form multimodal BEV fusion features, specifically including:

[0029] Folded image features are obtained by folding multi-scale image features along the height axis. The folded image features are used as key-value sequences and BEV feature maps are used as query sequences. These sequences are then input into a multi-head attention mechanism for cross-modal attention computation to obtain multi-modal BEV fusion features that integrate LiDAR and camera information.

[0030] Furthermore, the category-aware object query vector and BEV feature map are input into the first-layer Transformer decoder to obtain the enhanced query vector and the initial 3D bounding box prediction vector, specifically including:

[0031] The first Transformer decoder layer includes a first multi-head self-attention layer, a first residual connection and layer normalization layer, a second multi-head cross-attention layer, a second residual connection and layer normalization layer, a fifth feedforward network layer, and a third residual connection and layer normalization layer connected in sequence.

[0032] The category-aware object query vector is input into the first multi-head self-attention layer to perform self-attention calculation processing between object queries, thereby obtaining self-attention enhanced features; the self-attention enhanced features and the category-aware object query vector are then subjected to the first residual connection and layer normalization processing to obtain the first normalized features;

[0033] The first normalized feature is used as the query sequence and the BEV feature map is used as the key value sequence. The two are then input into the second multi-head cross-attention layer to perform cross-modal attention calculation to obtain cross-attention features. The cross-attention features and the first normalized feature are then connected by a second residual and processed by layer normalization to obtain the second normalized features.

[0034] The second normalized feature is input into the first feedforward network layer, and then processed sequentially through the second fully connected layer, ReLU activation function, first Dropout layer, third fully connected layer, and second Dropout layer to obtain the feedforward network feature; the feedforward network feature and the second normalized feature are then processed by the third residual connection and layer normalization to obtain the enhanced query vector;

[0035] The enhanced query vector is input into FFN detector head 1 and processed in parallel through center offset branch, height branch, size branch, orientation branch, velocity branch and classification branch to obtain the initial 3D bounding box prediction vector.

[0036] Furthermore, the multi-scale image features, enhanced query vector, and initial 3D bounding box prediction vector are input into the second-layer Transformer decoder to output the garbage identification result, specifically including:

[0037] The second-layer Transformer decoder includes a third multi-head self-attention layer, a fourth residual connection and layer normalization, a spatial modulation cross-attention layer, a fifth residual connection and layer normalization, a second feedforward network layer, and a sixth residual connection and layer normalization.

[0038] The enhanced query vector is input into the third multi-head self-attention layer to perform self-attention calculation between object queries, thereby obtaining the second self-attention enhanced feature; the second self-attention enhanced feature and the enhanced query vector are then subjected to a fourth residual connection and layer normalization to obtain the third normalized feature;

[0039] The initial 3D bounding box prediction vector is projected onto the camera coordinate system to obtain the 2D center coordinates and the radius of the circumscribed circle. The 2D center coordinates and the radius of the circumscribed circle are input into the spatial modulation cross-attention layer to construct a 2D circular Gaussian mask to obtain the spatial weight mask. The third normalized feature is used as the query sequence, the multi-scale image features are used as the key sequence, and the spatial weight mask is used as the modulation weight. These are then input into the spatial modulation cross-attention layer for spatial modulation attention calculation to obtain the spatial modulation attention feature. The spatial modulation attention feature and the third normalized feature are then connected by a fifth residual and processed by layer normalization to obtain the fourth normalized feature.

[0040] The fourth normalized feature is input into the second feedforward network layer to obtain the second feedforward network feature; the second feedforward network feature and the fourth normalized feature are then connected by the sixth residual and processed by layer normalization to obtain the final object query vector.

[0041] The final object query vector is input into FFN detector head 2, and the garbage identification result is obtained after processing through regression branch and classification branch.

[0042] Furthermore, the two-dimensional center coordinates and the radius of the circumcircle are input into the spatial modulation cross-attention layer to construct a two-dimensional circular Gaussian mask to obtain a spatial weight mask, specifically including:

[0043]

[0044] In the formula, For spatial weight mask, For the row index of the weight mask, For the column index of the weight mask, To query the predicted 2D center x-coordinate projected onto the camera image plane, To query the predicted 2D center ordinate projected onto the camera image plane, To adjust the hyperparameters of the Gaussian distribution bandwidth, The minimum circumcircle radius of the 3D bounding box projection corner point.

[0045] Furthermore, the training process for the first and second layer Transformer decoders includes:

[0046] Road debris image data from the training dataset is input into the ResNet-50 backbone network and feature pyramid network to obtain multi-scale image features; LiDAR point cloud data from the training dataset is input into a sparse convolutional backbone network based on sparse embedded convolutional detection (SECOND) to obtain BEV feature maps; and ground truth labels are added to the BEV feature maps and multi-scale image features.

[0047] Adaptive feature fusion is performed on BEV feature maps and multi-scale image features to obtain multimodal BEV fusion features; a training sample set is constructed from multi-scale image features, BEV feature maps, and multimodal BEV fusion features;

[0048] The first and second Transformer decoders are trained using training samples to obtain spam detection results; the total training loss is calculated based on the spam detection results and the ground truth labels, expressed by the following formula:

[0049]

[0050]

[0051]

[0052]

[0053] In the formula, For total training loss, For binary cross-entropy loss, For bounding box regression loss, The IoU loss is the difference between the predicted bounding box and the ground truth bounding box. , and These are the weighting coefficients of the loss function; To predict the total number of garbage objects, The number of training samples. To predict the number of waste categories, For the first The training sample of the th training sample Similar to real-world labels, For the first The first sample The predicted classification probability of waste category; h is the difference between the predicted bounding box and the ground truth bounding box; IOU is the ratio of the area of ​​the intersection region between the predicted bounding box and the ground truth bounding box to the area of ​​the merged region;

[0054] With the goal of minimizing the total training loss L, backpropagation is performed to calculate the gradient, update the parameters of the first and second layer Transformer decoders, and repeat the iteration until the loss converges or the preset number of training rounds is reached to output the trained first and second layer Transformer decoders.

[0055] A second aspect of the present invention provides a road waste detection system based on multimodal information fusion, comprising:

[0056] The acquisition module is used to acquire two-dimensional images of the road surface and LiDAR point cloud data generated by scanning the road surface using vehicle-mounted cameras and LiDAR, and to perform joint calibration and time alignment on the two-dimensional images of the road surface and the LiDAR point cloud data.

[0057] The feature extraction module is used to extract multi-scale image features from 2D images from the camera by combining the ResNet-50 backbone network and the feature pyramid network; and to extract features from LiDAR point cloud data to obtain BEV feature maps by using a sparse convolution backbone network based on sparse embedded convolution detection.

[0058] The feature fusion module transforms multi-scale image features from the camera coordinate system to the LiDAR coordinate system, fuses the multi-scale features and BEV feature maps to form multimodal BEV fusion features; predicts category heatmaps for the multimodal BEV fusion features and BEV feature maps respectively, and fuses them by averaging to obtain a fusion heatmap; selects the top N fusion heatmaps for all categories through local maxima, and generates category-aware object query vectors with category embedding;

[0059] The feature recognition module inputs the category-aware object query vector and BEV feature map into the first-layer Transformer decoder to obtain the enhanced query vector and the initial 3D bounding box prediction vector; it inputs the multi-scale image features, the enhanced query vector, and the initial 3D bounding box prediction vector into the second-layer Transformer decoder to output the garbage recognition result.

[0060] A third aspect of the present invention provides an electronic terminal, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the road waste detection method of the first aspect.

[0061] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0062] This invention employs a dual-modal data fusion strategy using both camera and LiDAR, fully leveraging the complementary characteristics of the two sensors. The camera provides rich color, texture, and semantic information, while the LiDAR provides precise 3D spatial geometric information. By transforming multi-scale image features from the camera coordinate system to the LiDAR coordinate system, a unified multimodal BEV fusion feature is formed. This overcomes the inherent limitations of a single visual sensor under adverse weather and lighting conditions, while also compensating for information loss caused by the sparsity and disorder of LiDAR point clouds, achieving reliable perception in all weather and all scenarios.

[0063] This invention employs a query-driven Transformer decoder architecture, selecting initial object queries by fusing local maxima from heatmaps and incorporating category embedding to enhance semantic awareness. The first-layer decoder predicts initial 3D bounding boxes based on BEV feature maps, establishing coarse spatial localization. The second-layer decoder introduces a spatial modulation cross-attention mechanism, using a two-dimensional Gaussian weighted mask to project and constrain the 3D bounding boxes onto local regions of the image plane, achieving soft correlation and fine-grained feature fusion. This progressive detection strategy, from coarse to fine, significantly improves the detection capability of small-sized, long-distance, and heavily occluded road debris. Attached Figure Description

[0064] Figure 1 This is a structural diagram of the ResNet-50 backbone network and feature pyramid network provided in Embodiment 1 of the present invention;

[0065] Figure 2 This is a diagram of the voxel feature extraction network structure provided in Embodiment 1 of the present invention.

[0066] Figure 3 This is a diagram of the sparse convolutional intermediate layer network structure provided in Embodiment 1 of the present invention.

[0067] Figure 4This is a network framework diagram of fusing multi-scale features of camera images and BEV features of lidar provided in Embodiment 1 of the present invention;

[0068] Figure 5 This is a flowchart of road waste detection provided in Embodiment 1 of the present invention. Detailed Implementation

[0069] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0070] Example 1

[0071] like Figures 1 to 5 As shown, this embodiment provides a road waste detection method based on multimodal information fusion, including:

[0072] Acquire 2D road surface images and LiDAR point cloud data generated by vehicle-mounted cameras and LiDAR scanning, and perform joint calibration and time alignment of the 2D road surface images and LiDAR point cloud data; specifically including:

[0073] The Network Time Protocol (NTP) is used to stamp the same timestamp on the image frames of the camera and the corresponding point cloud frames of the LiDAR, so as to achieve point cloud and image matching; the camera and LiDAR fixedly installed on the sanitation vehicle take pictures and capture point clouds at the same position on the checkerboard calibration board placed on the road surface, find at least three corresponding two-dimensional feature points and three-dimensional feature points, and use the open source algorithm lidar_camera_calibration to solve the transformation relationship between the camera coordinate system and the LiDAR coordinate system;

[0074] like Figure 1 As shown, a combination of ResNet-50 backbone network and feature pyramid network is used to extract multi-scale image features from 2D camera images, specifically including:

[0075] The 2D image from the camera is input into the image preprocessing layer. The image preprocessing layer performs preliminary feature extraction by using a 2D convolutional layer, a 2D batch normalization layer, a ReLU activation function, and a 2D max pooling layer to obtain a preprocessed feature map.

[0076] The preprocessed feature maps are input into the ResNet-50 backbone network and sequentially processed through four stages of Bottleneck residual block stacking from Stage 1 to Stage 4 to obtain backbone feature maps containing information from different receptive fields. ;

[0077] Stage 1 consists of three stacked Bottleneck residual blocks and outputs the backbone feature map. Stage 2 consists of 4 Bottleneck residual blocks and outputs the backbone feature map. Stage 3 consists of 6 Bottleneck residual blocks, which form the output backbone feature map. Stage 4 consists of three Bottleneck residual blocks, which form the output backbone feature map. The Bottleneck residual block contains 1×1 convolutions, 3×3 convolutions, and 1×1 convolutions.

[0078] Main feature map The input is fed into the feature pyramid network to obtain multi-scale image features, and the output of the ResNet-50 backbone network is... , , The feature map is uniformly processed by a 1×1 convolution, resulting in 256 channels. Starting from the deepest layer... Start upsampling (nearest neighbor interpolation) and merge it with lateral connections. After 1×1 convolution, we get , After upsampling and The horizontal connections are added together, and then smoothed by a 3×3 convolution to obtain the result. , After upsampling and The horizontal connections are added together, and then smoothed by a 3×3 convolution to obtain the result. , Upsampling and Fusion The formula is as follows:

[0079]

[0080] In the formula, The output features of the feature pyramid network; for convolution; for convolution; This indicates an upsampling operation.

[0081] This application employs a ResNet-50 backbone network combined with an FPN structure in the image branch. Through residual connections and feature pyramid fusion, it effectively extracts multi-scale semantic information, adapting to the detection needs of road debris of different sizes. In the point cloud branch, it adopts a SECOND-based sparse convolution backbone network. By utilizing an alternating design of submanifold sparse convolution and ordinary sparse convolution, convolution operations are performed only on non-empty voxels. While maintaining data sparsity, this significantly reduces computational load and memory consumption, solving the problem of excessive computational load in traditional 3D convolution and meeting the computational constraints of real-time detection for unmanned sanitation vehicles.

[0082] like Figure 2 and Figure 3 As shown, a sparse convolutional backbone network based on sparse embedded convolutional detection (SECOND) is used to extract features from LiDAR point cloud data to obtain BEV feature maps, specifically including:

[0083] LiDAR point cloud data is input into a voxel feature extractor for 3D spatial voxel mesh generation. Each point in the LiDAR point cloud data is assigned to a corresponding voxel based on its 3D coordinates. For each non-empty voxel, T points are randomly sampled to obtain the sampled voxel in-point set. The formula is:

[0084]

[0085] In the formula, Let d be the three-dimensional coordinates of the d-th point. Let d be the feature vector of the d-th sampling point; Let be the reflectance at point d. The coordinates of the voxel center are 3D coordinates.

[0086] The set of voxel in-points is input into the VFE layer, and each point is processed by a fully connected network consisting of a linear layer, a BatchNorm layer, and a ReLU activation function to obtain point-by-point feature vectors. The point-by-point feature vectors are then processed by point-by-point MaxPool operations to obtain local aggregated features. The local aggregated features are concatenated with the point-by-point feature vectors, and then processed by the first fully connected layer and MaxPooling aggregation to obtain sparse voxel features.

[0087] The sparse voxel features are input into a sparse convolution backbone network based on sparse embedded convolution detection. In the first stage, the sparse voxel features are processed sequentially through three sub-manifold sparse convolution modules to obtain deep features. The deep features are then processed by Z-downsampling through one ordinary sparse convolution module to obtain sparse voxel compressed features.

[0088] In the second stage, the sparse voxel compressed features are sequentially processed through 5 layers of submanifold sparse convolution modules for depth extraction, and then processed through 1 layer of ordinary sparse convolution module to complete Z-axis compression to obtain the BEV feature map. The submanifold sparse convolution module includes SubM Conv3D (submanifold sparse 3D convolution), BatchNorm3D, and ReLU activation; the ordinary sparse convolution module includes Sparse Conv3D (sparse convolution), BatchNorm3D, and ReLU activation.

[0089] This application employs a submanifold sparse convolution module, performing convolution operations only on non-empty voxels and their neighboring regions, strictly maintaining data sparsity and avoiding computational explosion. A standard sparse convolution module expands the receptive field and performs downsampling at key locations, progressively compressing the Z-axis dimension to efficiently convert 3D voxel features into 2D BEV representations, significantly reducing memory usage and inference latency. Furthermore, the first stage maintains sparsity and initially compresses the Z-axis, while the second stage deeply extracts global contextual information. The stacked design of the submanifold sparse convolution achieves local-to-global feature aggregation without disrupting the sparse structure, effectively capturing the geometric shape and spatial distribution characteristics of the point cloud. The progressive Z-axis compression strategy preserves key spatial information from the top-down view, generating BEV feature maps with both semantic expressiveness and positional accuracy.

[0090] The multi-scale image features are transformed from the camera coordinate system to the LiDAR coordinate system, expressed by the following formula:

[0091]

[0092] In the formula, These are the coordinates of pixels in the camera coordinate system within multi-scale image features; These are the coordinates of pixels in the LiDAR coordinate system within the multi-scale image features. This is the rotation matrix from the camera coordinate system to the lidar coordinate system; This is the translation vector from the camera coordinate system to the lidar coordinate system.

[0093] Multi-scale features and BEV feature maps are fused to form multimodal BEV fusion features, specifically including:

[0094] Folded image features are obtained by folding multi-scale image features along the height axis. The folded image features are used as key-value sequences and BEV feature maps are used as query sequences. These sequences are then input into a multi-head attention mechanism for cross-modal attention computation to obtain multi-modal BEV fusion features that integrate LiDAR and camera information.

[0095] This application folds multi-scale image features along the height axis and then fuses them with LiDAR BEV feature maps using multi-head attention, achieving implicit alignment from the camera coordinate system to the LiDAR coordinate system. Compared to traditional methods that rely on the accuracy of depth estimation for explicit projection transformation, the attention mechanism in this application adaptively learns cross-modal feature associations, avoiding the impact of calibration errors and depth estimation errors on fusion accuracy and improving robustness to sensor misregistration.

[0096] The class heatmaps are predicted for the multimodal BEV fusion features and BEV feature maps respectively, and then averaged and fused to obtain the fusion heatmap. The top N fusion heatmaps of all classes are selected by local maxima and equipped with class embedding to generate class-aware object query vectors.

[0097] The category-aware object query vector and BEV feature map are input into the first-layer Transformer decoder to obtain the enhanced query vector and the initial 3D bounding box prediction vector, specifically including:

[0098] The first Transformer decoder layer includes a first multi-head self-attention layer, a first residual connection and layer normalization layer, a second multi-head cross-attention layer, a second residual connection and layer normalization layer, a fifth feedforward network layer, and a third residual connection and layer normalization layer connected in sequence.

[0099] The category-aware object query vector is input into the first multi-head self-attention layer to perform self-attention calculation processing between object queries, thereby obtaining self-attention enhanced features; the self-attention enhanced features and the category-aware object query vector are then subjected to the first residual connection and layer normalization processing to obtain the first normalized features;

[0100] The first normalized feature is used as the query sequence and the BEV feature map is used as the key value sequence. The two are then input into the second multi-head cross-attention layer to perform cross-modal attention calculation to obtain cross-attention features. The cross-attention features and the first normalized feature are then connected by a second residual and processed by layer normalization to obtain the second normalized features.

[0101] The second normalized feature is input into the first feedforward network layer, and then processed sequentially through the second fully connected layer, ReLU activation function, first Dropout layer, third fully connected layer, and second Dropout layer to obtain the feedforward network feature; the feedforward network feature and the second normalized feature are then processed by the third residual connection and layer normalization to obtain the enhanced query vector;

[0102] The enhanced query vector is input into FFN detector head 1 and processed in parallel through center offset branch, height branch, size branch, orientation branch, velocity branch and classification branch to obtain the initial 3D bounding box prediction vector.

[0103] The multi-scale image features, enhanced query vector, and initial 3D bounding box prediction vector are input into the second-layer Transformer decoder to output the garbage identification result, specifically including:

[0104] The second-layer Transformer decoder includes a third multi-head self-attention layer, a fourth residual connection and layer normalization, a spatial modulation cross-attention layer, a fifth residual connection and layer normalization, a second feedforward network layer, and a sixth residual connection and layer normalization.

[0105] The enhanced query vector is input into the third multi-head self-attention layer to perform self-attention calculation between object queries, thereby obtaining the second self-attention enhanced feature; the second self-attention enhanced feature and the enhanced query vector are then subjected to a fourth residual connection and layer normalization to obtain the third normalized feature;

[0106] The initial 3D bounding box prediction vector is projected onto the camera coordinate system to obtain the 2D center coordinates and the radius of the circumscribed circle. These 2D center coordinates and the radius of the circumscribed circle are then input into the spatial modulation cross-attention layer to construct a 2D circular Gaussian mask, obtaining the spatial weight mask. The formula is as follows:

[0107]

[0108] In the formula, For spatial weight mask, For the row index of the weight mask, For the column index of the weight mask, To query the predicted 2D center x-coordinate projected onto the camera image plane, To query the predicted 2D center ordinate projected onto the camera image plane, To adjust the hyperparameters of the Gaussian distribution bandwidth, The minimum circumcircle radius of the 3D bounding box projection corner point;

[0109] The third normalized feature is used as the query sequence, the multi-scale image features are used as the key sequence, and the spatial weight mask is used as the modulation weight. These are then input into the spatial modulation cross-attention layer for spatial modulation attention calculation to obtain the spatial modulation attention feature. The spatial modulation attention feature and the third normalized feature are then connected by the fifth residual and processed by layer normalization to obtain the fourth normalized feature.

[0110] The fourth normalized feature is input into the second feedforward network layer to obtain the second feedforward network feature; the second feedforward network feature and the fourth normalized feature are then connected by the sixth residual and processed by layer normalization to obtain the final object query vector.

[0111] The final object query vector is input into FFN detector head 2, and the garbage identification result is obtained after processing through regression branch and classification branch.

[0112] This application employs a query-driven Transformer decoder architecture, selecting initial object queries by fusing local maxima from heatmaps and incorporating category embeddings to enhance semantic awareness. The first-layer decoder predicts initial 3D bounding boxes based on BEV feature maps, establishing coarse spatial localization. The second-layer decoder introduces a spatial modulation cross-attention mechanism, using a two-dimensional Gaussian weighted mask to project and constrain the 3D bounding boxes onto local regions of the image plane, achieving soft correlation and fine-grained feature fusion. This progressive detection strategy, from coarse to fine, significantly improves the detection capability of small-sized, long-distance, and heavily occluded road debris.

[0113] like Figure 4 and Figure 5 As shown, the training process for the first and second layer Transformer decoders includes:

[0114] Road debris image data from the training dataset is input into the ResNet-50 backbone network and feature pyramid network to obtain multi-scale image features; LiDAR point cloud data from the training dataset is input into a sparse convolutional backbone network based on sparse embedded convolutional detection (SECOND) to obtain BEV feature maps; and ground truth labels are added to the BEV feature maps and multi-scale image features.

[0115] Adaptive feature fusion is performed on BEV feature maps and multi-scale image features to obtain multimodal BEV fusion features; a training sample set is constructed from multi-scale image features, BEV feature maps, and multimodal BEV fusion features;

[0116] The first and second Transformer decoders are trained using training samples to obtain spam detection results; the total training loss is calculated based on the spam detection results and the ground truth labels, expressed by the following formula:

[0117]

[0118]

[0119]

[0120]

[0121] In the formula, For total training loss, For binary cross-entropy loss, For bounding box regression loss, The IoU loss is the difference between the predicted bounding box and the ground truth bounding box. , and These are the weighting coefficients of the loss function; To predict the total number of garbage objects, The number of training samples. To predict the number of waste categories, For the first The training sample of the th training sample Similar to real-world labels, For the first The first sample The predicted classification probability of waste category; h is the difference between the predicted bounding box and the ground truth bounding box; IOU is the ratio of the area of ​​the intersection region between the predicted bounding box and the ground truth bounding box to the area of ​​the merged region;

[0122] With the goal of minimizing the total training loss L, backpropagation is performed to calculate the gradient, update the parameters of the first and second layer Transformer decoders, and repeat the iteration until the loss converges or the preset number of training rounds is reached to output the trained first and second layer Transformer decoders.

[0123] A training strategy employs a two-layer Transformer decoder for separate supervision. The first layer uses 3D bounding box annotations for intermediate supervision, while the second layer uses 2D bounding box annotations for final supervision. A multi-task loss function integrates classification loss, regression loss, and IoU loss, balancing the optimization objectives of class prediction accuracy and localization precision. An auxiliary detection head design allows intermediate layer features to participate in loss calculation, mitigating the gradient vanishing problem in deep networks and accelerating model convergence.

[0124] The method of this invention is compared with the performance of LiDAR-based detection models PointPillar and CenterPoint, camera-based detection models YOLOv10, YOLOv11, and YOLO26, and LiDAR-camera fusion models MVP, BEVFusion, and FusionFormer. The comparison results are shown in Table 1. The target detection performance is evaluated using mAP and the quantified parameters as evaluation metrics; mAP and parameters represent the average AP50:95 across all categories and the total number of learnable weights in the model, respectively.

[0125] Table 1. Performance Comparison of Different Detection Algorithms on Road Waste Detection Task

[0126]

[0127] As can be seen from Table 1, the detection performance of the fusion algorithm of this invention is better than that of other known target detection algorithms. The mAP evaluation index of this invention reaches 74.6%, which shows the effectiveness of the detection method of this invention in improving the detection performance of occluded objects and small objects at a distance. The number of parameters of this invention is significantly lower than that of conventional LiDAR-camera fusion algorithms while improving detection accuracy, which is conducive to deployment in autonomous driving vehicle equipment such as unmanned sanitation vehicles and realizing engineering applications.

[0128] Example 2

[0129] This embodiment provides a road waste detection system based on multimodal information fusion. The road waste detection system is used to execute the road waste detection method described in Embodiment 1. The road waste detection system includes:

[0130] The acquisition module is used to acquire two-dimensional images of the road surface and LiDAR point cloud data generated by scanning the road surface using vehicle-mounted cameras and LiDAR, and to perform joint calibration and time alignment on the two-dimensional images of the road surface and the LiDAR point cloud data.

[0131] The feature extraction module is used to extract multi-scale image features from 2D images from the camera by combining the ResNet-50 backbone network and the feature pyramid network; and to extract features from LiDAR point cloud data to obtain BEV feature maps by using a sparse convolution backbone network based on sparse embedded convolution detection.

[0132] The feature fusion module transforms multi-scale image features from the camera coordinate system to the LiDAR coordinate system, fuses the multi-scale features and BEV feature maps to form multimodal BEV fusion features; predicts category heatmaps for the multimodal BEV fusion features and BEV feature maps respectively, and fuses them by averaging to obtain a fusion heatmap; selects the top N fusion heatmaps for all categories through local maxima, and generates category-aware object query vectors with category embedding;

[0133] The feature recognition module inputs the category-aware object query vector and BEV feature map into the first-layer Transformer decoder to obtain the enhanced query vector and the initial 3D bounding box prediction vector; it inputs the multi-scale image features, the enhanced query vector, and the initial 3D bounding box prediction vector into the second-layer Transformer decoder to output the garbage recognition result.

[0134] Example 3

[0135] This embodiment provides an electronic terminal, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the road waste detection method described in Embodiment 1.

[0136] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A road waste detection method based on multimodal information fusion, characterized in that, include: Acquire two-dimensional images of the road surface and LiDAR point cloud data generated by scanning the road surface using vehicle-mounted cameras and LiDAR, and perform joint calibration and time alignment on the two-dimensional images of the road surface and the LiDAR point cloud data; A combination of ResNet-50 backbone network and feature pyramid network is used to extract multi-scale image features from 2D images from the camera; a sparse convolution backbone network based on sparse embedded convolution detection is used to extract features from LiDAR point cloud data to obtain BEV feature maps. Multi-scale image features are transformed from the camera coordinate system to the lidar coordinate system, and multi-scale features and BEV feature maps are fused to form multimodal BEV fusion features; The class heatmaps are predicted for the multimodal BEV fusion features and BEV feature maps respectively, and then fused by average to obtain the fusion heatmap; the top N fusion heatmaps of all classes are selected by local maxima, and class embedding is used to generate class-aware object query vectors; The category-aware object query vector and BEV feature map are input into the first-layer Transformer decoder to obtain the enhanced query vector and the initial 3D bounding box prediction vector; the multi-scale image features, the enhanced query vector, and the initial 3D bounding box prediction vector are input into the second-layer Transformer decoder to output the garbage identification result.

2. The road waste detection method according to claim 1, characterized in that, This study employs a combination of a ResNet-50 backbone network and a feature pyramid network to extract multi-scale image features from 2D camera images, specifically including: The 2D image from the camera is input into the image preprocessing layer. The image preprocessing layer performs preliminary feature extraction by using a 2D convolutional layer, a 2D batch normalization layer, a ReLU activation function, and a 2D max pooling layer to obtain a preprocessed feature map. The preprocessed feature maps are input into the ResNet-50 backbone network and sequentially processed through four stages of Bottleneck residual block stacking from Stage 1 to Stage 4 to obtain backbone feature maps containing information from different receptive fields. ; Main feature map The input to the feature pyramid network yields multi-scale image features, expressed as follows: ； In the formula, The output features of the feature pyramid network; for convolution; for convolution; This indicates an upsampling operation.

3. The road waste detection method according to claim 2, characterized in that, Stage 1 consists of three stacked Bottleneck residual blocks and outputs the backbone feature map. Stage 2 consists of 4 Bottleneck residual blocks and outputs the backbone feature map. Stage 3 consists of 6 Bottleneck residual blocks, which form the output backbone feature map. Stage 4 consists of three Bottleneck residual blocks, which form the output backbone feature map. The Bottleneck residual block contains 1×1 convolutions, 3×3 convolutions, and 1×1 convolutions.

4. The road waste detection method according to claim 1, characterized in that, BEV feature maps are obtained by extracting features from LiDAR point cloud data using a sparse convolutional backbone network based on sparse embedded convolutional detection. Specifically, this includes: The LiDAR point cloud data is input into the voxel feature extractor for 3D spatial voxel mesh division. Based on the three-dimensional coordinates of each point in the LiDAR point cloud data, it is assigned to the corresponding voxel. T points are randomly sampled for each non-empty voxel to obtain the sampled voxel in-point set. The set of voxel in-points is input into the VFE layer, and each point is processed by a fully connected network consisting of a linear layer, a BatchNorm layer, and a ReLU activation function to obtain point-by-point feature vectors. The point-by-point feature vectors are then processed by point-by-point MaxPool operations to obtain local aggregated features. The local aggregated features are concatenated with the point-by-point feature vectors, and then processed by the first fully connected layer and MaxPooling aggregation to obtain sparse voxel features. The sparse voxel features are input into a sparse convolution backbone network based on sparse embedded convolution detection. In the first stage, the sparse voxel features are processed sequentially through three sub-manifold sparse convolution modules to obtain deep features. The deep features are then processed by Z-downsampling through one ordinary sparse convolution module to obtain sparse voxel compressed features. In the second stage, the sparse voxel compressed features are sequentially processed through 5 layers of sub-manifold sparse convolution modules for depth extraction, and then processed through 1 layer of ordinary sparse convolution module to complete Z-axis compression to obtain the BEV feature map. The sub-manifold sparse convolution module includes SubM Conv3D, BatchNorm3D and ReLU activation; the ordinary sparse convolution module includes Sparse Conv3D, BatchNorm3D and ReLU activation.

5. The road waste detection method according to claim 1, characterized in that, Transforming multi-scale image features from the camera coordinate system to the LiDAR coordinate system specifically includes: ； In the formula, These are the coordinates of pixels in the camera coordinate system within multi-scale image features; These are the coordinates of pixels in the LiDAR coordinate system within the multi-scale image features. This is the rotation matrix from the camera coordinate system to the lidar coordinate system; This is the translation vector from the camera coordinate system to the lidar coordinate system.

6. The road waste detection method according to claim 1, characterized in that, Multi-scale features and BEV feature maps are fused to form multimodal BEV fusion features, specifically including: Folded image features are obtained by folding multi-scale image features along the height axis. The folded image features are used as key-value sequences and BEV feature maps are used as query sequences. These sequences are then input into a multi-head attention mechanism for cross-modal attention computation to obtain multi-modal BEV fusion features that integrate LiDAR and camera information.

7. The road waste detection method according to claim 1, characterized in that, The category-aware object query vector and BEV feature map are input into the first-layer Transformer decoder to obtain the enhanced query vector and the initial 3D bounding box prediction vector, specifically including: The first Transformer decoder layer includes a first multi-head self-attention layer, a first residual connection and layer normalization layer, a second multi-head cross-attention layer, a second residual connection and layer normalization layer, a fifth feedforward network layer, and a third residual connection and layer normalization layer connected in sequence. The category-aware object query vector is input into the first multi-head self-attention layer to perform self-attention calculation processing between object queries, thereby obtaining self-attention enhanced features; the self-attention enhanced features and the category-aware object query vector are then subjected to the first residual connection and layer normalization processing to obtain the first normalized features; The first normalized feature is used as the query sequence and the BEV feature map is used as the key value sequence. The two are then input into the second multi-head cross-attention layer to perform cross-modal attention calculation to obtain cross-attention features. The cross-attention features and the first normalized feature are then connected by a second residual and processed by layer normalization to obtain the second normalized features. The second normalized feature is input into the first feedforward network layer, and then processed sequentially through the second fully connected layer, ReLU activation function, first Dropout layer, third fully connected layer, and second Dropout layer to obtain the feedforward network feature; the feedforward network feature and the second normalized feature are then processed by the third residual connection and layer normalization to obtain the enhanced query vector; The enhanced query vector is input into FFN detector head 1 and processed in parallel through center offset branch, height branch, size branch, orientation branch, velocity branch and classification branch to obtain the initial 3D bounding box prediction vector.

8. The road waste detection method according to claim 1, characterized in that, The multi-scale image features, enhanced query vector, and initial 3D bounding box prediction vector are input into the second-layer Transformer decoder to output the garbage identification result, specifically including: The second-layer Transformer decoder includes a third multi-head self-attention layer, a fourth residual connection and layer normalization, a spatial modulation cross-attention layer, a fifth residual connection and layer normalization, a second feedforward network layer, and a sixth residual connection and layer normalization. The enhanced query vector is input into the third multi-head self-attention layer to perform self-attention calculation between object queries, thereby obtaining the second self-attention enhanced feature; the second self-attention enhanced feature and the enhanced query vector are then subjected to a fourth residual connection and layer normalization to obtain the third normalized feature; The initial 3D bounding box prediction vector is projected onto the camera coordinate system to obtain the 2D center coordinates and the radius of the circumscribed circle. The 2D center coordinates and the radius of the circumscribed circle are input into the spatial modulation cross-attention layer to construct a 2D circular Gaussian mask to obtain the spatial weight mask. The third normalized feature is used as the query sequence, the multi-scale image features are used as the key sequence, and the spatial weight mask is used as the modulation weight. These are then input into the spatial modulation cross-attention layer for spatial modulation attention calculation to obtain the spatial modulation attention feature. The spatial modulation attention feature and the third normalized feature are then connected by a fifth residual and processed by layer normalization to obtain the fourth normalized feature. The fourth normalized feature is input into the second feedforward network layer to obtain the second feedforward network feature; the second feedforward network feature and the fourth normalized feature are then connected by the sixth residual and processed by layer normalization to obtain the final object query vector. The final object query vector is input into FFN detector head 2, and the garbage identification result is obtained after processing through regression branch and classification branch.

9. The road waste detection method according to claim 8, characterized in that, The two-dimensional center coordinates and the radius of the circumcircle are input into the spatial modulation cross-attention layer to construct a two-dimensional circular Gaussian mask to obtain the spatial weight mask, specifically including: ； In the formula, For spatial weight mask, For the row index of the weight mask, For the column index of the weight mask, To query the predicted 2D center x-coordinate projected onto the camera image plane, To query the predicted 2D center ordinate projected onto the camera image plane, To adjust the hyperparameters of the Gaussian distribution bandwidth, The minimum circumcircle radius of the 3D bounding box projection corner point.

10. The road waste detection method according to claim 1, characterized in that, The training process for the first and second layer Transformer decoders includes: Road debris image data from the training dataset is input into the ResNet-50 backbone network and feature pyramid network to obtain multi-scale image features; LiDAR point cloud data from the training dataset is input into a sparse convolutional backbone network based on sparse embedded convolutional detection to obtain BEV feature maps; and ground truth labels are added to the BEV feature maps and multi-scale image features. Adaptive feature fusion is performed on BEV feature maps and multi-scale image features to obtain multimodal BEV fusion features; a training sample set is constructed from multi-scale image features, BEV feature maps, and multimodal BEV fusion features; The first and second Transformer decoders are trained using training samples to obtain spam detection results; the total training loss is calculated based on the spam detection results and the ground truth labels, expressed by the following formula: ； ； ； ； In the formula, For total training loss, For binary cross-entropy loss, For bounding box regression loss, The IoU loss is the difference between the predicted bounding box and the ground truth bounding box. , and These are the weighting coefficients of the loss function; To predict the total number of garbage objects, The number of training samples. To predict the number of waste categories, For the first The training sample of the th training sample Similar to real-world labels, For the first The first sample The predicted classification probability of waste category; h is the difference between the predicted bounding box and the ground truth bounding box; IOU is the ratio of the area of ​​the intersection region between the predicted bounding box and the ground truth bounding box to the area of ​​the merged region; With the goal of minimizing the total training loss L, backpropagation is performed to calculate the gradient, update the parameters of the first and second layer Transformer decoders, and repeat the iteration until the loss converges or the preset number of training rounds is reached to output the trained first and second layer Transformer decoders.

11. A road waste detection system based on multimodal information fusion, characterized in that, include: The acquisition module is used to acquire two-dimensional images of the road surface and LiDAR point cloud data generated by scanning the road surface using vehicle-mounted cameras and LiDAR, and to perform joint calibration and time alignment on the two-dimensional images of the road surface and the LiDAR point cloud data. The feature extraction module is used to extract multi-scale image features from 2D images from the camera by combining the ResNet-50 backbone network and the feature pyramid network; and to extract features from LiDAR point cloud data to obtain BEV feature maps by using a sparse convolution backbone network based on sparse embedded convolution detection. The feature fusion module transforms multi-scale image features from the camera coordinate system to the LiDAR coordinate system, fuses the multi-scale features and BEV feature maps to form multimodal BEV fusion features; predicts category heatmaps for the multimodal BEV fusion features and BEV feature maps respectively, and fuses them by averaging to obtain a fusion heatmap; selects the top N fusion heatmaps for all categories through local maxima, and generates category-aware object query vectors with category embedding; The feature recognition module inputs the category-aware object query vector and BEV feature map into the first-layer Transformer decoder to obtain the enhanced query vector and the initial 3D bounding box prediction vector; it inputs the multi-scale image features, the enhanced query vector, and the initial 3D bounding box prediction vector into the second-layer Transformer decoder to output the garbage recognition result.

12. An electronic terminal, characterized in that, It includes a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to perform the steps of the road waste detection method according to any one of claims 1 to 10.

Images