A target detection method based on multi-sensor context perception fusion

Abstract

This invention discloses a target detection method based on multi-sensor context-aware fusion. The method includes: acquiring images from a left camera, a right camera, and LiDAR point cloud data; projecting the LiDAR point cloud into a bird's-eye view and inputting it along with the left and right camera images into a shared backbone module to extract initial spatial features; using a context-aware gating module to rank candidate detection branches based on their expected performance in the current scene, and dynamically selecting the top-ranked detection branch; performing target detection on the selected single-sensor branch and the early fusion branch; and then using Soft-NMS or weighted bounding box fusion to perform post-fusion on the outputs of each branch to obtain the final detection result. This invention can adaptively adjust the fusion timing and fusion path for complex scenes such as rain, snow, fog, and nighttime, preserving the ability to utilize intermediate features from multiple modalities while also considering the flexibility of post-fusion, thus improving the accuracy and robustness of target detection in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving target detection technology, specifically, it relates to a target detection method based on multi-sensor context perception fusion. Background Technology

[0002] When autonomous vehicles operate in complex open road environments, they typically rely on multiple sensors, such as cameras and LiDAR, to perceive their surroundings. Cameras provide high-resolution texture, color, and semantic information, while LiDAR provides more reliable distance and spatial geometry information; the two exhibit significant modal complementarity. Efficiently fusing data from these two heterogeneous sensors is a key challenge in improving the accuracy and robustness of target detection in autonomous driving systems.

[0003] Existing multimodal fusion schemes can generally be divided into three categories: early-stage data-level fusion, feature-level fusion, and late-stage decision-level fusion. Early-stage fusion processes data at the raw data level, which retains a lot of low-level information, but is sensitive to spatiotemporal registration accuracy and data quality. Feature-level fusion can enhance intermodal interaction, but has high computational complexity, and the fusion position, fusion weight, and alignment method have a significant impact on the results. Late-stage decision-level fusion has the advantages of flexibility, modularity, and strong fault tolerance, but it usually discards a lot of intermediate features and cross-modal correlation information, making it difficult to fully utilize the complementary advantages of multimodal fusion.

[0004] Furthermore, most existing fusion methods employ static fusion strategies, continuing to use fixed fusion paths under varying weather, lighting, road types, and sensor disturbance conditions. This makes it difficult to adaptively select the optimal modality combination and fusion timing based on the scene context. When cameras are obstructed by rain or snow, exposed to strong light, or affected by low illumination at night, or when LiDAR performance degrades in foggy weather, with distant targets, or sparse point cloud conditions, fixed fusion rules can easily cause fluctuations in detection performance, especially lacking robustness in complex scenarios. Summary of the Invention

[0005] To address the problems of fixed multimodal fusion rules, poor adaptability to complex scenarios, and insufficient utilization of intermediate features in existing autonomous driving target detection methods, this invention provides a target detection method based on multi-sensor context-aware fusion.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows: A target detection method based on multi-sensor context-aware fusion includes the following steps: S1, acquire the left camera image, the right camera image and the LiDAR point cloud data, project the LiDAR point cloud data into a bird's-eye view, and input it into the shared backbone module along with the left camera image and the right camera image respectively, and extract the initial spatial features of each modality; S2, input the initial spatial features into the context-aware gating module, predict the expected performance of each candidate detection branch in the current scene and sort them, and select the top k detection branches; S3, target detection is performed on the detection branches respectively. The single sensor branch directly uses the initial spatial features of the corresponding modality. The early fusion branch performs channel stitching and convolution fusion on the initial spatial features of multiple modalities before target detection, and obtains the candidate detection results of each branch. S4, perform post-fusion processing on the candidate detection results output by each selected detection branch to obtain the target category, bounding box position and confidence level; S5 outputs the final target detection result.

[0007] Compared with the prior art, the present invention has the following advantages: The initial spatial features of the left and right camera images and the LiDAR bird's-eye view are extracted by the shared backbone module. The current scene context is identified by the context-aware gating module, and the performance of candidate detection branches is predicted and ranked. The top-ranked detection branch is dynamically selected based on the ranking results. Target detection is performed on the selected single-sensor branch and the early fusion branch. Finally, Soft-NMS or weighted bounding box fusion is used to perform post-fusion on the output of each branch to obtain the final detection result.

[0008] Instead of using the same fusion path for all scenarios, the gating module predicts and ranks the performance of candidate detection branches based on the scene context, and then dynamically activates the subset of branches that are most suitable for the current scene, thereby improving the robustness of target detection in complex scenarios. Attached Figure Description

[0009] Figure 1 This is an overall flowchart of the target detection method based on multi-sensor context-aware fusion in an embodiment of the present invention; Figure 2 This is a schematic diagram of the shared backbone module structure in an embodiment of the present invention; Figure 3 This is a schematic diagram of the gating module structure based on prior knowledge in an embodiment of the present invention; Figure 4 This is a schematic diagram of the depth gating module structure in an embodiment of the present invention; Figure 5 This is a schematic diagram of the self-attention gating module structure in an embodiment of the present invention; Figure 6 This is a schematic diagram of the fusion module processing flow in an embodiment of the present invention; Figure 7 This is a schematic diagram comparing the average mAP of different fusion methods in the embodiments of the present invention; Figure 8This is a visual comparison diagram of rainy day scenes in an embodiment of the present invention; Figure 9 This is a visual comparison diagram of snowy scenes in an embodiment of the present invention. Detailed Implementation

[0010] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0011] like Figure 1 As shown, this embodiment provides a target detection method based on multi-sensor context-aware fusion, including the following steps: S1. Acquire the left camera image, right camera image and LiDAR point cloud data, project the LiDAR point cloud data into a bird's-eye view, and input it into the shared backbone module along with the left camera image and right camera image respectively, and extract the initial spatial features of each modality. S2. Input the initial spatial features into the context-aware gating module, predict the expected performance of each candidate detection branch in the current scene and sort them, and select the top k detection branches. S3. Perform target detection on each detection branch. The single sensor branch directly utilizes the initial spatial features of the corresponding modality. The early fusion branch performs channel stitching and convolutional fusion on the initial spatial features of multiple modalities before performing target detection, thus obtaining the candidate detection results of each branch. S4. Perform post-fusion processing on the candidate detection results output by each selected detection branch to obtain the target category, bounding box position and confidence level; S5. Output the final target detection result.

[0012] The input sensors include a left camera, a right camera, and a LiDAR. The left and right cameras output RGB images, respectively, while the LiDAR outputs a discrete 3D point cloud. Since point cloud data is not suitable for direct collaborative processing with image data in a 2D convolutional network, the LiDAR point cloud is first projected as a bird's-eye view.

[0013] The LiDAR point cloud data is projected into a bird's-eye view at a resolution of 0.1 meters, with the point cloud spatial range limited to [-40, 40] × [0, 70] meters. The bird's-eye view is represented by six channels, where the first five channels consist of the maximum height values ​​corresponding to five equally divided slices in the [0, 2.5] meter interval along the Z-axis, and the sixth channel represents the point cloud density characteristics within the grid cell. Through the above processing, the LiDAR point cloud is converted into a bird's-eye view representation suitable for processing by a two-dimensional convolutional network.

[0014] like Figure 2As shown, the shared backbone module uses the first two residual stages of a segmented ResNet-50 network as the feature extraction structure to extract initial spatial features from the left camera image, right camera image, and LiDAR bird's-eye view, respectively. After each modal input passes through its corresponding backbone module, a spatial feature map with reduced size, increased channel count, and preliminary semantic information is obtained. These initial spatial features retain key information from the original input and provide a more compact and stable feature representation for the subsequent gating module to recognize the scene context.

[0015] In step S2, the context-aware gating module includes any one of prior knowledge gating, deep gating, and self-attention gating, and is used to predict and rank the performance of candidate detection branches based on the scene context.

[0016] like Figure 3 As shown, prior knowledge gating establishes branch selection rules based on a predefined set of scene contexts; the predefined set of scene contexts includes at least city, intersection, rainy, snowy, foggy, and nighttime scenes; and the candidate detection branches are statically encoded and sorted according to the relative performance of each sensor in different scenes.

[0017] like Figure 4 As shown, the depth-gated system includes a convolutional feature extraction part and a multilayer perceptron decision layer. The convolutional feature extraction part includes three depthwise separable convolutional layers, each of which includes depthwise convolution, pointwise convolution, batch normalization, channel attention, and ReLU activation in sequence. The multilayer perceptron decision layer includes a 256-dimensional hidden layer, which outputs a performance prediction value corresponding to the number of candidate detection branches.

[0018] like Figure 5 As shown, self-attention gating introduces a self-attention module between the convolutional feature extraction part and the multilayer perceptron decision layer. By globally modeling the spatial dependency of the output features of the shared backbone module, it generates attention-enhanced feature representations for evaluating the performance of candidate detection branches. This allows for more accurate identification of key regions related to branch selection and improves branch ranking capabilities in complex scenarios.

[0019] The candidate detection branches are set to five branches, including three single-sensor branches and two early fusion branches; the three single-sensor branches are the left camera branch, the right camera branch and the LiDAR branch, and the two early fusion branches are the left and right camera fusion branch and the camera and LiDAR fusion branch.

[0020] For the single-sensor branch, the initial spatial features of the corresponding sensor are directly used as the branch input. For the early fusion branch, the initial spatial features of multiple modalities are first concatenated along the channel dimension, and then the concatenated result is fused through two-dimensional convolution. This fusion result is then input into the remaining network of the branch to complete object detection. Each branch can be connected to a region proposal network after the convolutional layer to generate corresponding candidate detection boxes.

[0021] By setting up single-modal paths and multi-modal fusion paths in parallel, this invention can still rely on robust branch output detection results when the sensor is subjected to local disturbances, and fully leverage the advantages of collaborative sensing when all multi-modal approaches are effective.

[0022] like Figure 6 As shown, in step S4, the post-fusion processing employs either Soft-NMS or Weighted Bounding Facilitation (WBF). Soft-NMS retains potentially valid boxes by attenuating the confidence of overlapping detection boxes, while WBF obtains the final detection box by weighted coordinate fusion of multiple overlapping detection boxes of the same target. The fusion module uses Soft-NMS, which no longer directly deletes boxes overlapping with high-confidence detection boxes, but instead attenuates their confidence based on the degree of overlap, thereby retaining potentially valid candidate boxes and improving recall in dense scenes. The fusion module uses Weighted Bounding Facilitation (WBF), which aggregates overlapping detection boxes according to category and weights their coordinates based on the confidence of each detection box, obtaining more accurate output bounding boxes. Compared to traditional non-maximum suppression, WBF is more advantageous for comprehensively utilizing the localization information of multiple branches for the same target.

[0023] In step S2, the context-aware gating module adopts a phased training method. First, the shared backbone module and candidate detection branches are trained. Then, the parameters of the shared backbone module and candidate detection branches are fixed. The gating module is trained separately with the initial spatial features output by the shared backbone module as input and the classification loss and regression loss corresponding to each candidate detection branch as supervision signals. The training objective of the gating module is a regression task. The mean absolute error loss function is used to constrain the predicted loss value output by the gating module and the true loss of each candidate detection branch. The gating module outputs the predicted loss value of each candidate detection branch during the inference phase and selects the k branches with the highest predicted loss values ​​to perform detection and post-fusion. This reduces the execution overhead of unnecessary branches while maintaining detection performance, balancing accuracy and real-time requirements.

[0024] Compared with the prior art, the present invention has the following advantages: The initial spatial features of the left and right camera images and the LiDAR bird's-eye view are extracted by the shared backbone module. The current scene context is identified by the context-aware gating module, and the performance of candidate detection branches is predicted and ranked. The top-ranked detection branch is dynamically selected based on the ranking results. Target detection is performed on the selected single-sensor branch and the early fusion branch. Finally, Soft-NMS or weighted bounding box fusion is used to perform post-fusion on the output of each branch to obtain the final detection result.

[0025] A multi-stage collaborative detection framework for target detection in complex scenes is proposed. By constructing an overall technical route of "shared backbone feature extraction - context-aware gating and sorting - candidate branch dynamic activation - post-fusion of detection results", scene context modeling, detection branch selection and multi-branch result fusion are unified in the same detection framework, realizing the integrated design of the detection process.

[0026] Using the initial spatial features output by the shared backbone module as the scene context representation carrier, the expected loss of each candidate detection branch is predicted using the initial spatial features. This eliminates the need to set up an additional independent scene recognition network, thereby reducing network structure redundancy, lowering additional computational overhead, and improving the tight coupling between context modeling and object detection.

[0027] In the design of candidate detection branches, both single-sensor detection branches and multi-sensor early fusion detection branches are retained. The candidate detection paths are limited to five representative branches: left camera branch, right camera branch, LiDAR branch, left and right camera fusion branch, and camera and LiDAR fusion branch. While ensuring sufficient coverage of visual information and point cloud information, the scale of branch combination is effectively controlled, and the structural complexity of the multi-branch detection framework is reduced.

[0028] In the design of the gating module, three replaceable structures are set up: prior knowledge gating, deep gating, and self-attention gating. Among them, prior knowledge gating can complete candidate branch selection based on a preset rule base, deep gating can complete branch priority prediction based on a continuous feature space, and self-attention gating can further enhance the ability to model context relevance. Thus, the present invention has both rule-based interpretability and data-driven adaptive capability.

[0029] The training and deployment strategy adopts a phased training approach, and in the inference phase, only the top k candidate detection branches are activated to perform target detection. Then, the detection results of the k candidate detection branches are subjected to post-fusion processing, which is completed by Soft-NMS or weighted bounding box fusion. This ensures robustness and accuracy in complex scene detection while taking into account model inference efficiency and engineering deployment feasibility.

[0030] To verify the effectiveness of the contextual multi-branch object detection method described in this invention, the publicly available RADIATE multimodal autonomous driving perception dataset was selected as the experimental data source. This dataset contains labeled samples collected by a Velodyne 32-line LiDAR, a ZED stereo camera, and a Navtech radar. The labeled categories include eight types of targets: cars, vans, trucks, buses, motorcycles, bicycles, pedestrians, and groups of pedestrians. In the experiment, the dataset was divided into training and testing sets in a 7:3 ratio for model training and performance evaluation.

[0031] Experimental Implementation Images from both the left and right cameras, along with a bird's-eye view from the LiDAR, are used as input modalities. The LiDAR point cloud is first converted into a bird's-eye view representation, and then fed into a shared backbone module along with the left and right camera images to extract initial spatial features. Each candidate detection branch is implemented using a Faster R-CNN detection structure based on a ResNet-50 backbone network to ensure consistent feature extraction and detection capabilities across different branches. The average precision (mAP) across all categories is used as the primary evaluation metric for object detection performance.

[0032] Training parameter settings The learning rate for training the shared backbone module and candidate detection branches was set to 0.005; the learning rate for training the gating module separately was set to 5×10^-5; the batch size for training the overall Select-Fusion model was set to 4, the training epochs were set to 80, the optimizer used was stochastic gradient descent (SGD), and the initial learning rate was set to 0.0006. The experimental environment included PyTorch 1.8.0, Ubuntu 18.04 64-bit operating system, Intel i7-11700F processor, NVIDIA GeForce RTX 3090 24GB graphics card, 32GB of memory, and CUDA 11.1 and cuDNN 8.0 acceleration libraries.

[0033] Basic performance comparison experiment The context-aware multi-branch fusion object detection method of this invention was comprehensively evaluated on the RADIATE validation set. Table 1 shows the average accuracy (mAP) across all classes achieved by different modal configurations on the dataset. The camera is represented by C, and L / RC represents the fusion of left and right cameras.

[0034] To verify the effectiveness of the method of this invention, the following comparative experiments were set up: (I) no fusion, single sensor; (II) early fusion between sensors; (III) late fusion between specific sensor branches; (IV) the Select-Fusion method of this invention. For the results of (I) and (II), this embodiment uses the corresponding single sensor data as input and uses the Faster R-CNN model based on ResNet-50 to obtain the mAP. The result of late fusion in (III) is obtained by inputting each sensor modality into the Faster R-CNN model based on ResNet-50 for processing, and finally using one of the two fusion algorithms (Soft-NMS and WBF) to fuse the output of each model. Table 1 shows the best performance result. (IV) is the result obtained by running Select-Fusion to select the optimal branch and using the fusion algorithm to fuse the results. It shows the best result of the selective sensor fusion method used in this embodiment, which uses a self-attention gating method to select the top two branches for each input.

[0035] Table 1. Target detection performance (mAP) of different fusion methods in RADIATE As shown in Table 1, the Select-Fusion method described in this invention outperforms single-sensor, early fusion, and late fusion methods in target detection performance. The best performing method is the Select-Fusion method using Soft-NMS in the fusion block, with an mAP of 71.26%. This demonstrates that the Select-Fusion architecture combines the advantages of early and late fusion, fully utilizing scene context to dynamically select the branches to be fused, thus reducing the impact of complex scenes and scene changes on the performance of the target detection model.

[0036] Table 1 shows that all fusion variants outperform the results in (I), validating the advantages of multimodal fusion over single-sensor methods. The table also compares early and late fusion methods. In the experiments conducted in this embodiment, the late fusion method performed slightly better than the early fusion method under the same perceptual modality fusion. This is attributed to the presence of more challenging scenarios in the RADIATE dataset, resulting in slightly lower quality raw data from the sensors. In contrast, late fusion is more robust to lower-quality data. Furthermore, the late fusion method incorporates a multimodal combination where all branches participate in the fusion. However, its performance is worse than all results in (IV), demonstrating that simply stacking sensor data does not improve target detection performance.

[0037] Figure 7The average mAP of each fusion method is compared with Select-Fusion (showing the best performance). Select-Fusion significantly outperforms the early and late fusion methods, exceeding them by 5.25% and 4.81% respectively, achieving a peak mAP of 71.26%. Experimental results demonstrate that, compared with existing fusion methods, the context-aware selective sensor fusion architecture effectively improves the robustness and accuracy of target detection.

[0038] ablation experiment Since Select-Fusion dynamically selects the fusion branch based on the scene recognition context, the ablation experiment mainly verifies the effectiveness of the context-aware gating module in improving target detection performance. This embodiment will conduct an ablation experiment on the gating perception module, and selects the Select-Fusion algorithm with fusion of all branches as the benchmark for comparison. That is, after each modality's data passes through the shared backbone module, it directly enters each branch to output its own results, which are then finally processed by the fusion module. Table 2 shows the impact of each gating method in the gating perception module, and the table shows the best performing results.

[0039] Table 2. Analysis of the contribution of the ablation experiment to the gated sensing module. Among them, self-attention gating achieved the best mAP of 71.26%. This is because it can identify the regions in the backbone input features that are most relevant to the output. The depth gating module ranked second with an mAP of 69.47%, demonstrating that it can effectively utilize backbone features to identify contextual information.

[0040] Furthermore, this embodiment also evaluates the impact of different optimal branch sizes k on each gating method, wherein... The results of the best-performing fusion algorithm for each gating method are shown in Table 3.

[0041] Table 3 Evaluation of gating modules with different k values As shown in Table 3, when k=2, the self-attention gating and depth gating using Soft-NMS achieve the best mAP during fusion. In contrast, the prior knowledge gating performs best when k=1. This is likely because domain knowledge is sufficient to determine the optimal modality in each context. However, when k=1, its robustness to sensor clogging or failure is insufficient, limiting its practical applicability. Compared to the other two gating methods, the mAP is lower than depth gating and attention gating for all other k values, meaning that knowledge gating typically performs worse in dynamic environments. This is due to its reliance on fixed context definitions, such as limited scene types like city, rainy day, and snowy day. In real-world applications, scenarios are complex and varied, with many situations that cannot be precisely categorized into these predefined scenarios. This fixed context definition struggles to adapt to these complex and varied situations, leading to the inability to accurately select the appropriate branch.

[0042] Visualization Analysis To visually demonstrate the effectiveness of Select-Fusion in improving detection model performance, this section selects two types of complex scenarios for visualization analysis. This example compares the detection results of camera-only, LiDAR-only, late fusion, and Select-Fusion. A unified annotation standard is used for the visualization scheme. Yellow boxes represent ground truth, green detection boxes represent predicted boxes for camera-only, light blue detection boxes represent predicted boxes for LiDAR-only, predicted boxes for late fusion are marked in orange, and purple prediction boxes represent the detection results of Select-Fusion.

[0043] (1) Rainy day scene like Figure 8 As shown, from the camera's perspective, rainwater accumulation blurs the image information collected by the sensor, significantly increasing the difficulty of target detection. A car traveling in the opposite direction and a bus traveling in the same direction are present on the road. In single-modal camera detection, rainwater on the lens prevents the collection of clear features of the distant bus, resulting in missed detections. For single-modal LiDAR, due to rain noise and the relatively long distance of the target, the collected point cloud information is very sparse, leading to the same missed detection of the distant bus. Late-mode fusion, which fuses the prediction results of two modalities, also failed to successfully detect the bus. However, the Select-Fusion method in this embodiment maintains good detection accuracy, without any missed or false detections.

[0044] (2) Snowy scene Figure 9The image depicts a snowy scene where snowflakes partially obscure the camera's view, affecting the sensor's field of view. In the point cloud data collected by the LiDAR, many snowflakes are also scanned as target points, interfering with the detection of real targets. There are four ground truth targets in this scene: one car in the left lane and three in the right lane. Due to the snow accumulation, only the headlights of the left-hand vehicle are faintly visible in the image data, resulting in missed detections in the single-modal camera detection. In LiDAR detection, the obfuscation caused by the dense snowflake projection leads to a significant deviation in the predicted position of the nearest white car. Furthermore, the farthest car is also missed due to the sparse point cloud. While late-stage fusion successfully predicts the occluded car on the left compared to the single-modal camera detection, its predicted position is not accurate. It also fails to detect the farthest vehicle, resulting in performance degradation. In the method of this invention, both the snow-occluded vehicle on the left and the farthest vehicle are predicted well.

[0045] In summary, the method presented in this embodiment demonstrates excellent detection performance in both of the aforementioned complex scenarios. This fully illustrates that Select-Fusion can efficiently combine feature information from different modalities, overcoming the limitations of single-modal perception. Compared to other fusion methods, it can dynamically adjust the fusion strategy using scene context information, effectively improving the accuracy and robustness of the detection model in challenging scenarios.

[0046] The above provides a detailed description of a target detection method based on multi-sensor context-aware fusion provided in this application. The specific embodiments are described only to aid in understanding the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A target detection method based on multi-sensor context-aware fusion, characterized in that, Including the following steps: S1, acquire the left camera image, the right camera image and the LiDAR point cloud data, project the LiDAR point cloud data into a bird's-eye view, and input it into the shared backbone module along with the left camera image and the right camera image respectively, and extract the initial spatial features of each modality; S2, input the initial spatial features into the context-aware gating module, predict the expected performance of each candidate detection branch in the current scene and sort them, and select the top k detection branches; S3, target detection is performed on the detection branches respectively. The single sensor branch directly uses the initial spatial features of the corresponding modality. The early fusion branch performs channel stitching and convolution fusion on the initial spatial features of multiple modalities before target detection, and obtains the candidate detection results of each branch. S4, perform post-fusion processing on the candidate detection results output by each selected detection branch to obtain the target category, bounding box position and confidence level; S5 outputs the final target detection result.

2. The target detection method based on multi-sensor context-aware fusion according to claim 1, characterized in that, The shared backbone module uses the first two residual stages of the segmented ResNet-50 network as the feature extraction structure to extract the initial spatial features of the left camera image, the right camera image, and the LiDAR bird's-eye view, respectively.

3. The target detection method based on multi-sensor context-aware fusion according to claim 1, characterized in that, In step S2, the context-aware gating module includes any one of prior knowledge gating, deep gating, and self-attention gating, and is used to predict and rank the performance of candidate detection branches based on the scene context.

4. The target detection method based on multi-sensor context-aware fusion according to claim 3, characterized in that, Prior knowledge gating establishes branch selection rules based on a predefined set of scene contexts; the predefined set of scene contexts includes at least urban, intersection, rainy, snowy, foggy, and nighttime scenes; and candidate detection branches are statically encoded and sorted according to the relative performance of each sensor in different scenes.

5. The target detection method based on multi-sensor context-aware fusion according to claim 4, characterized in that, The depth-gated system comprises a convolutional feature extraction part and a multilayer perceptron decision layer. The convolutional feature extraction part includes three depthwise separable convolutional layers, each of which sequentially includes depthwise convolution, pointwise convolution, batch normalization, channel attention, and ReLU activation. The multilayer perceptron decision layer includes a 256-dimensional hidden layer, which outputs a performance prediction value corresponding to the number of candidate detection branches.

6. The target detection method based on multi-sensor context-aware fusion according to claim 5, characterized in that, Self-attention gating introduces a self-attention module between the convolutional feature extraction part and the multilayer perceptron decision layer. By globally modeling the spatial dependency of the output features of the shared backbone module, it generates attention-enhanced feature representations for evaluating the performance of candidate detection branches.

7. The target detection method based on multi-sensor context-aware fusion according to claim 6, characterized in that, The candidate detection branches are set to five branches, including three single-sensor branches and two early fusion branches; The three single-sensor branches are the left camera branch, the right camera branch, and the LiDAR branch, while the two early fusion branches are the left and right camera fusion branch and the camera and LiDAR fusion branch.

8. The target detection method based on multi-sensor context-aware fusion according to claim 7, characterized in that, In step S4, the post-fusion processing uses Soft-NMS or weighted bounding box fusion (WBF). Soft-NMS retains potential valid boxes by attenuating the confidence of overlapping detection boxes, while WBF obtains the final detection box by weighted coordinate fusion of multiple overlapping detection boxes of the same target.

9. A target detection method based on multi-sensor context-aware fusion according to claim 8, characterized in that, In step S2, the context-aware gating module adopts a phased training method. First, the shared backbone module and candidate detection branches are trained. Then, the parameters of the shared backbone module and candidate detection branches are fixed. The gating module is trained separately with the initial spatial features output by the shared backbone module as input and the classification loss and regression loss corresponding to each candidate detection branch as supervision signals. The training objective of the gating module is a regression task. The mean absolute error loss function is used to constrain the predicted loss value output by the gating module and the true loss of each candidate detection branch. During the inference phase, the gating module outputs the predicted loss value of each candidate detection branch and selects the k branches with the highest predicted loss values ​​to perform detection and post-fusion.

10. A target detection method based on multi-sensor context-aware fusion according to claim 1, characterized in that, The lidar point cloud data is projected into a bird's-eye view at a resolution of 0.1 meters, and the spatial range of the point cloud is limited to [-40,40]×[0,70] meters. The bird's-eye view is represented by six channels, of which the first five channels are composed of the maximum height values ​​corresponding to the five equal slices in the Z-axis direction [0,2.5] meters, and the sixth channel represents the point cloud density characteristics within the grid cell.

Images