A camera and laser radar adaptive gating BEV feature fusion perception method and system for foggy off-road environment

Abstract

This invention discloses a camera- and lidar-adaptive gated BEV feature fusion perception method and system for foggy off-road environments, relating to the field of intelligent driving environment perception technology. The method mainly includes: time synchronization of visible light image data and lidar point cloud data to obtain synchronized image and point cloud data; construction of a multi-task joint perception model; training of the multi-task joint perception model to obtain a trained multi-task joint perception model; and detection of obstacles in the synchronized image and point cloud data using the trained multi-task joint perception model to obtain 3D detection results of obstacles in the off-road environment. Implementing the camera- and lidar-adaptive gated BEV feature fusion perception method and system provided by this invention can improve the deep anti-fog interference capability and high-precision small target perception capability of environmental perception.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving environment perception technology, and more specifically, to a camera and lidar adaptive gated BEV feature fusion perception method and system for foggy off-road environments. Background Technology

[0002] With the widespread application of intelligent off-road vehicles in fields such as wilderness rescue, military reconnaissance, and mining transportation, the unstructured environments they face not only feature undulating terrain and diverse obstacles, but also often suffer from low visibility due to fog and water vapor, posing a significant challenge to sensor stability. However, existing perception solutions often suffer from visual feature loss, radar ghosting, and single-sensor failure due to sparse point clouds in foggy conditions. Traditional post-fusion algorithms, relying on bounding box matching, are prone to missed detections, while forward projection fusion struggles to cope with undulating terrain due to the loss of crucial 3D spatial information. Furthermore, existing multimodal fusion based on Bird's Eye View (BEV) architectures often employ simple channel stitching, which can easily lead to spatiotemporal asymmetric degradation of sensors under foggy conditions, introducing severely degraded modal noise into the fused features and causing feature contamination.

[0003] Improving the deep anti-fog interference capability and high-precision small target perception capability of environmental perception is an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a camera and lidar adaptive gating BEV feature fusion perception method and system for off-road environments in foggy weather, which can improve the deep anti-fog interference capability and high-precision small target perception capability of environmental perception.

[0005] This invention provides a camera and lidar adaptive gating BEV feature fusion perception method for off-road environments in foggy weather, comprising the following steps: S1: Synchronize visible light image data and lidar point cloud data in time to obtain synchronized image data and point cloud data; S2: Construct a multi-task joint perception model, train the multi-task joint perception model, and obtain a trained multi-task joint perception model. S3: Use the trained multi-task joint perception model to detect the synchronized image data and point cloud data to obtain the three-dimensional detection results of obstacles in the off-road environment.

[0006] Furthermore, the aforementioned multi-task joint perception model includes a dual-stream feature extraction module, an adaptive gating fusion module, and a 3D detection head. The dual-stream feature extraction module includes a visual backbone network, an LSS module, and a radar backbone network. The visual backbone network is used to extract multi-scale visual semantic features from synchronized image data. The LSS module is used to project the multi-scale visual semantic features onto a bird's-eye view space to obtain visual BEV features. The radar backbone network is used to perform two-dimensional bird's-eye view feature transformation on synchronized point cloud data to obtain radar geometric features. The adaptive gating fusion module is used to perform feature-level fusion of the visual BEV features and the radar geometric features to obtain multi-modal fusion features. The 3D detection head is used to perform 3D target detection on the multi-modal fusion features to obtain three-dimensional detection results.

[0007] Furthermore, the specific configuration of the aforementioned adaptive gating fusion module is as follows: The visual BEV features and the radar geometric features are spliced ​​together along the channel dimension to obtain spliced ​​features; Based on the splicing features, a spatial attention evaluation network is used to generate a confidence mask; The confidence mask is exponentially normalized along the channel dimension to obtain adaptive visual spatial weights and radar spatial weights. Based on adaptive visual spatial weights and radar spatial weights, visual BEV features are fused with radar geometric features to obtain the final multimodal fusion features.

[0008] Furthermore, the training process of the above-mentioned multi-task joint perception model includes: constructing the depth real labels of the visual backbone network based on the original visible light image and the synchronous dataset of lidar point cloud; The multi-task joint perception model is trained in a distributed, multi-stage, progressive manner using the deep real labels and the multi-task joint loss function to obtain a trained multi-task joint perception model, specifically including: The visual backbone network of the multi-task joint perception model is pre-trained using an image dataset alone to obtain a pre-trained visual backbone network. The radar backbone network and 3D detection head are pre-trained using a point cloud dataset alone to obtain a pre-trained radar backbone network and 3D detection head. The parameters of the radar backbone network are frozen, the pre-trained visual backbone network is connected to the LSS module, and the visual backbone network is optimized separately using the depth real labels to obtain the optimized visual backbone network. Based on the optimized visual backbone network, LSS module, adaptive gating fusion module, pre-trained radar backbone network and 3D detection head, end-to-end joint fine-tuning is performed using synthetic fog noise data to obtain a trained multi-task joint perception model.

[0009] Furthermore, the formula for calculating the joint loss function for the above multi-task tasks is as follows:

[0010]

[0011] in, For multi-task joint loss function, For 3D object detection loss, For LSS depth prediction loss, These are the weighting coefficients for the LSS depth prediction loss. For heatmap-based classification loss, For 3D bounding box regression loss, These are the weighting coefficients for the 3D bounding box regression loss.

[0012] This invention also provides a camera and lidar adaptive gating BEV feature fusion perception system for foggy off-road environments, the system comprising the following modules: The data acquisition and synchronization module is configured to synchronize visible light image data and lidar point cloud data in time to obtain synchronized image data and point cloud data. The model building and training module is configured to: build a multi-task joint perception model, train the multi-task joint perception model, and obtain a trained multi-task joint perception model. The multi-task joint perception and detection module is configured to use a trained multi-task joint perception model to detect the synchronized image data and point cloud data, thereby obtaining the three-dimensional detection results of obstacles in the off-road environment.

[0013] Furthermore, the aforementioned multi-task joint perception model includes a dual-stream feature extraction module, an adaptive gating fusion module, and a 3D detection head. The dual-stream feature extraction module includes a visual backbone network, an LSS module, and a radar backbone network. The visual backbone network is used to extract multi-scale visual semantic features from synchronized image data. The LSS module is used to project the multi-scale visual semantic features onto a bird's-eye view space to obtain visual BEV features. The radar backbone network is used to perform two-dimensional bird's-eye view feature transformation on synchronized point cloud data to obtain radar geometric features. The adaptive gating fusion module is used to perform feature-level fusion of the visual BEV features and the radar geometric features to obtain multi-modal fusion features. The 3D detection head is used to perform 3D target detection on the multi-modal fusion features to obtain three-dimensional detection results.

[0014] Furthermore, the specific configuration of the aforementioned adaptive gating fusion module is as follows: The visual BEV features and the radar geometric features are spliced ​​together along the channel dimension to obtain spliced ​​features; Based on the splicing features, a spatial attention evaluation network is used to generate a confidence mask; The confidence mask is exponentially normalized along the channel dimension to obtain adaptive visual spatial weights and radar spatial weights. Based on adaptive visual spatial weights and radar spatial weights, visual BEV features are fused with radar geometric features to obtain the final multimodal fusion features.

[0015] Furthermore, the training process of the above-mentioned multi-task joint perception model includes: constructing the depth real labels of the visual backbone network based on the original visible light image and the synchronous dataset of lidar point cloud; The multi-task joint perception model is trained in a distributed, multi-stage, progressive manner using the deep real labels and the multi-task joint loss function to obtain a trained multi-task joint perception model, specifically including: The visual backbone network of the multi-task joint perception model is pre-trained using an image dataset alone to obtain a pre-trained visual backbone network. The radar backbone network and 3D detection head are pre-trained using a point cloud dataset alone to obtain a pre-trained radar backbone network and 3D detection head. The parameters of the radar backbone network are frozen, the pre-trained visual backbone network is connected to the LSS module, and the visual backbone network is optimized separately using the depth real labels to obtain the optimized visual backbone network. Based on the optimized visual backbone network, LSS module, adaptive gating fusion module, pre-trained radar backbone network and 3D detection head, end-to-end joint fine-tuning is performed using synthetic fog noise data to obtain a trained multi-task joint perception model.

[0016] Furthermore, the formula for calculating the joint loss function for the above multi-task tasks is as follows:

[0017]

[0018] in, For multi-task joint loss function, For 3D object detection loss, For LSS depth prediction loss, These are the weighting coefficients for the LSS depth prediction loss. For heatmap-based classification loss, For 3D bounding box regression loss, These are the weighting coefficients for the 3D bounding box regression loss.

[0019] The camera- and lidar adaptive gating BEV feature fusion perception method and system for foggy off-road environments provided by this invention has the following beneficial effects: This invention addresses the limitations of traditional single-sensor or simple channel stitching methods in low-visibility and severe weather conditions, and the shortcomings of traditional defogging preprocessing and 3D sparse convolution in real-time performance on edge computing platforms. It provides an environmental perception scheme for intelligent off-road chassis, based on adaptive gating BEV feature fusion using cameras and LiDAR in foggy conditions. Specifically, it acquires strictly synchronized visible light images and LiDAR point clouds through high-precision multi-source data synchronization using the PTP protocol. Specifically, it uses GPS / BeiDou timestamps as a global reference for hardware triggering, establishes an IEEE 1588 PTP time synchronization network, sets the onboard computing platform as the PTP master clock, and the camera and LiDAR as PTP slave clocks. Through hardware triggering and timestamp matching, it achieves real-time alignment of multimodal data, ensuring strict time and spatial synchronization between the visible light image and the LiDAR point cloud. This provides an accurate spatiotemporal reference for subsequent feature extraction and fusion, avoiding fusion errors caused by data asynchrony. This invention utilizes anti-fog dual-stream feature extraction techniques to extract deep features. Specifically, the visual branch employs the ConvNeXt-V2 backbone, leveraging the large receptive field spatial smoothness of 7×7 deep separable convolutions to suppress local high-frequency fog noise. A Global Response Normalization (GRN) module is introduced to enhance inter-channel feature competition, suppressing background channel responses and increasing channel activation values ​​for distant small targets, thus preventing feature channel collapse for small targets. The radar branch employs the PointPillars backbone, employing spatial cylinder partitioning, MLP feature encoding, Scatter projection, and 2D... The CNN extraction process enables high-speed conversion from point cloud to BEV pseudo-image; abandoning the traditional pixel-level dehazing preprocessing, it directly achieves anti-fog feature extraction through the characteristics of the backbone network, effectively suppressing fog noise, capturing weak features of distant small targets in foggy weather, improving the perception robustness of distant small obstacles in unstructured roads, and retaining the geometric information of the point cloud through high-speed conversion from point cloud to BEV features, avoiding the loss of geometric features caused by the sparsity of the point cloud, while meeting real-time requirements, reducing computational latency, and adapting to the computing power constraints of edge computing platforms; This invention utilizes the Lift-Splat-Shoot (LSS) mechanism to project image features into a Bird's-Eye (BEV) space. During the feature fusion stage, an uncertainty-aware gating mechanism is introduced. An adaptive gating BEV fusion method dynamically allocates multimodal feature weights based on the degree of feature degradation in local grids and performs reconstruction and fusion. Specifically, visual features are projected into the BEV space via the LSS mechanism, concatenated with radar BEV features, and input into a spatial attention evaluation network to generate a confidence mask representing the degree of feature degradation in local grids. This mask is then normalized using Softmax to obtain an adaptive spatial weight map. Pixel-level weighted reconstruction is performed on the two BEV features to obtain the fused features. The weights of the two features are dynamically allocated based on the degree of feature degradation in local grids, avoiding the introduction of noise from degraded modes into the fused features. This effectively solves the problems of low information utilization and feature contamination in low signal-to-noise ratio environments, while maximizing the preservation of the camera's semantic features and the radar's geometric information, thus improving fusion accuracy and perceptual robustness in foggy scenes. This invention employs a targeted multi-stage training approach to train the model. First, it utilizes foggy multimodal data augmentation: performing 3D random flipping, global scaling, and translation enhancement in the point cloud space, and synchronously inverting the transformation matrix to the image intrinsic and extrinsic parameters to ensure spatial consistency across the multimodal environment. Second, it employs multi-task joint loss: using binary cross-entropy loss for depth prediction, Gaussian focus loss for detection and classification to address the imbalance between positive and negative samples, and L1 loss for bounding box regression. Third, it employs a three-stage progressive training: first, pre-training two single-modal backbones separately, then training the visual depth prediction branch separately, and finally fine-tuning the entire network end-to-end, incorporating gradient clipping to prevent gradient explosion. This approach ensures spatial consistency after foggy multimodal data augmentation, improves the model's generalization ability to foggy and degraded scenarios, effectively solves the imbalance between positive and negative samples, improves the detection accuracy of small targets at long distances, and enhances detection accuracy and convergence stability in foggy scenarios. In summary, this invention employs an improved adaptive gating feature fusion architecture, effectively solving the problems of low information utilization and feature contamination in fusion algorithms under low signal-to-noise ratio environments. Simultaneously, it utilizes a large receptive field visual backbone structure to suppress fog noise, improving the robustness of perceiving small, distant obstacles in unstructured roads. This enables environmental perception with deep anti-fog interference capabilities and high-precision small target perception, effectively improving the detection accuracy of the intelligent off-road chassis perception system and enhancing its adaptability in fog-covered and unstructured complex environments, making it suitable for the field of intelligent off-road chassis. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of the camera and lidar adaptive gating BEV feature fusion perception method for foggy off-road environments provided by the present invention; Figure 2 This is the overall flowchart of the camera and lidar adaptive gating BEV feature fusion perception method for foggy off-road environments provided by the present invention; Figure 3 This is a schematic diagram of the PTP time synchronization and data acquisition connection provided by the present invention; Figure 4 This is a flowchart of the visual feature extraction process provided by the present invention; Figure 5 This is a flowchart of radar point cloud feature extraction provided by the present invention; Figure 6 This is a schematic diagram of the LSS perspective conversion and feature-level fusion principle provided by the present invention.

[0021] Figure 7 This is a flowchart of the model training method provided by the present invention. Detailed Implementation

[0022] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1 This diagram illustrates the camera-and-LiDAR adaptive gating BEV feature fusion perception method for foggy off-road environments according to this embodiment. In this embodiment, the camera-and-LiDAR adaptive gating BEV feature fusion perception method for foggy off-road environments includes the following steps: S1: Synchronize visible light image data and lidar point cloud data in time to obtain synchronized image data and point cloud data; In one exemplary embodiment, the time synchronization process includes: using GPS or BeiDou timestamps as a reference for hardware triggering to synchronize the time of the vehicle computing platform to the global time; connecting the vehicle computing platform and network switch with the camera and LiDAR via the IEEE 1588 PTP protocol, making the computing platform the PTP master and the camera and LiDAR the PTP slave, thereby completing the time synchronization between the camera and LiDAR and the computing platform. S2: Construct a multi-task joint perception model, train the multi-task joint perception model, and obtain a trained multi-task joint perception model. In one exemplary embodiment, the multi-task joint perception model includes a dual-stream feature extraction module, an adaptive gating fusion module, and a 3D detection head. The dual-stream feature extraction module includes a visual backbone network, an LSS module, and a radar backbone network. The visual backbone network is used to extract multi-scale visual semantic features from synchronized image data. The LSS module is used to project the multi-scale visual semantic features onto a bird's-eye view space to obtain visual BEV features. The radar backbone network is used to perform two-dimensional bird's-eye view feature transformation on synchronized point cloud data to obtain radar geometric features. The adaptive gating fusion module is used to perform feature-level fusion of the visual BEV features and the radar geometric features to obtain multi-modal fusion features. The 3D detection head is used to perform 3D target detection on the multi-modal fusion features to obtain a three-dimensional detection result. In one exemplary embodiment, the adaptive gating fusion module is specifically configured as follows: The visual BEV features and the radar geometric features are spliced ​​together along the channel dimension to obtain spliced ​​features; Based on the splicing features, a spatial attention evaluation network is used to generate a confidence mask; The confidence mask is exponentially normalized along the channel dimension to obtain adaptive visual spatial weights and radar spatial weights. Based on adaptive visual spatial weights and radar spatial weights, visual BEV features are fused with radar geometric features to obtain the final multimodal fusion features. As an exemplary embodiment, the visual backbone network is the ConvNeXt-V2 network; the radar backbone network is the PointPillars network; the LSS module (Lift-Splat-Shoot) is a 2D image to BEV space perspective transformation module proposed by the NVIDIA team, and is a pioneering basic component in the field of BEV perception; the 3D detection head structure is divided into two parts: a feature preprocessing branch, which first performs channel-dimensional feature aggregation and spatial-dimensional feature aggregation on the input BEV feature map through several layers of 2D convolution. The system compresses information and extracts high-order semantic features for detection. It features a dual-task output branch where preprocessed features are simultaneously input into two parallel branches: a classification branch, which outputs a category confidence heatmap corresponding to the BEV grid through convolutional layers, typically using Gaussian focus loss as the supervisory loss to address the extreme imbalance between positive and negative samples in foggy or long-distance scenarios; and a regression branch, which outputs 3D bounding box regression parameters for each candidate obstacle through convolutional layers, including continuous values ​​such as 3D center coordinates, length, width, height, and orientation angle, typically using L1 loss as the supervisory loss to ensure regression accuracy.

[0024] In one exemplary embodiment, the training process of the multi-task joint perception model includes: constructing a depth real label for the visual backbone network based on the original visible light image and the lidar point cloud synchronization dataset; and performing distributed multi-stage progressive training on the multi-task joint perception model using the depth real label and the multi-task joint loss function to obtain the trained multi-task joint perception model. In one exemplary embodiment, the distributed multi-stage progressive training process includes: The visual backbone network of the multi-task joint perception model is pre-trained using an image dataset alone to obtain a pre-trained visual backbone network. The radar backbone network and 3D detection head are pre-trained using a point cloud dataset alone to obtain a pre-trained radar backbone network and 3D detection head. The parameters of the radar backbone network are frozen, the pre-trained visual backbone network is connected to the LSS module, and the visual backbone network is optimized separately using the depth real labels to obtain the optimized visual backbone network. Based on the optimized visual backbone network, LSS module, adaptive gating fusion module, pre-trained radar backbone network and 3D detection head, end-to-end joint fine-tuning is performed using synthetic fog noise data to obtain a trained multi-task joint perception model. In one exemplary embodiment, the process of constructing the depth true label includes: acquiring the original visible light image and LiDAR point cloud synchronous dataset; performing 3D data augmentation in the point cloud space for off-road environments, including 3D random flipping, global scaling, and translation; synchronously inverting the transformation matrix of the 3D data augmentation into the intrinsic and extrinsic parameter matrix in the image domain; using the time-synchronized original sparse LiDAR point cloud, projecting the point cloud onto a two-dimensional image plane through camera intrinsic and extrinsic parameters to obtain the projected continuous depth value; discretizing the projected continuous depth value into multiple depth intervals according to the depth preset range set by LSS, and converting it into a one-hot encoding format as the depth true label of the image depth prediction network; In one exemplary embodiment, the calculation formula for the multi-task joint loss function is as follows:

[0025]

[0026] in, For multi-task joint loss function, For 3D object detection loss, For LSS depth prediction loss, These are the weighting coefficients for the LSS depth prediction loss. For heatmap-based classification loss, For 3D bounding box regression loss, These are the weighting coefficients for the 3D bounding box regression loss; S3: Use the trained multi-task joint perception model to detect the synchronized image data and point cloud data to obtain the three-dimensional detection results of obstacles in the off-road environment.

[0027] This embodiment provides a camera and LiDAR adaptive gating BEV feature fusion perception system for foggy off-road environments. The system includes the following modules: The data acquisition and synchronization module is configured to synchronize visible light image data and lidar point cloud data in time to obtain synchronized image data and point cloud data. The model building and training module is configured to: build a multi-task joint perception model, train the multi-task joint perception model, and obtain a trained multi-task joint perception model. The multi-task joint perception and detection module is configured to use a trained multi-task joint perception model to detect the synchronized image data and point cloud data, thereby obtaining the three-dimensional detection results of obstacles in the off-road environment.

[0028] Furthermore, the aforementioned multi-task joint perception model includes a dual-stream feature extraction module, an adaptive gating fusion module, and a 3D detection head. The dual-stream feature extraction module includes a visual backbone network, an LSS module, and a radar backbone network. The visual backbone network is used to extract multi-scale visual semantic features from synchronized image data. The LSS module is used to project the multi-scale visual semantic features onto a bird's-eye view space to obtain visual BEV features. The radar backbone network is used to perform two-dimensional bird's-eye view feature transformation on synchronized point cloud data to obtain radar geometric features. The adaptive gating fusion module is used to perform feature-level fusion of the visual BEV features and the radar geometric features to obtain multi-modal fusion features. The 3D detection head is used to perform 3D target detection on the multi-modal fusion features to obtain three-dimensional detection results.

[0029] Furthermore, the specific configuration of the aforementioned adaptive gating fusion module is as follows: The visual BEV features and the radar geometric features are spliced ​​together along the channel dimension to obtain spliced ​​features; Based on the splicing features, a spatial attention evaluation network is used to generate a confidence mask; The confidence mask is exponentially normalized along the channel dimension to obtain adaptive visual spatial weights and radar spatial weights. Based on adaptive visual spatial weights and radar spatial weights, visual BEV features are fused with radar geometric features to obtain the final multimodal fusion features.

[0030] Furthermore, the training process of the above-mentioned multi-task joint perception model includes: constructing the depth real labels of the visual backbone network based on the original visible light image and the synchronous dataset of lidar point cloud; The multi-task joint perception model is trained in a distributed, multi-stage, progressive manner using the deep real labels and the multi-task joint loss function to obtain a trained multi-task joint perception model, specifically including: The visual backbone network of the multi-task joint perception model is pre-trained using an image dataset alone to obtain a pre-trained visual backbone network. The radar backbone network and 3D detection head are pre-trained using a point cloud dataset alone to obtain a pre-trained radar backbone network and 3D detection head. The parameters of the radar backbone network are frozen, the pre-trained visual backbone network is connected to the LSS module, and the visual backbone network is optimized separately using the depth real labels to obtain the optimized visual backbone network. Based on the optimized visual backbone network, LSS module, adaptive gating fusion module, pre-trained radar backbone network and 3D detection head, end-to-end joint fine-tuning is performed using synthetic fog noise data to obtain a trained multi-task joint perception model.

[0031] Furthermore, the formula for calculating the joint loss function for the above multi-task tasks is as follows:

[0032]

[0033] in, For multi-task joint loss function, For 3D object detection loss, For LSS depth prediction loss, These are the weighting coefficients for the LSS depth prediction loss. For heatmap-based classification loss, For 3D bounding box regression loss, These are the weighting coefficients for the 3D bounding box regression loss.

[0034] In some embodiments, the above-described camera and lidar adaptive gating BEV feature fusion perception method for foggy off-road environments can also be implemented in the following ways.

[0035] In this embodiment, the camera and lidar adaptive gating BEV feature fusion perception method for foggy off-road environments includes: Multi-source data acquisition and high-precision synchronization: The PTP protocol is used to acquire time-synchronized visible light images and lidar point clouds, and real-time synchronization of multimodal data is achieved through hardware triggering and timestamp matching.

[0036] End-to-end anti-fog dual-stream feature extraction: ConvNeXt-V2 and PointPillars are used as the backbone networks for vision and radar, respectively, to uniformly project image features and point cloud features into the bird's-eye view (BEV) space. In the feature extraction process, traditional high-latency pixel-level preprocessing is abandoned; instead, the characteristics of large receptive field convolutions are utilized to suppress local high-frequency fog noise and capture deep features. This accurately captures the features of weak, small targets in foggy scenes, providing a data foundation for subsequent multimodal gating fusion.

[0037] Adaptive Gated Fusion and Object Detection Output: An adaptive gated fusion mechanism is constructed that dynamically allocates weights based on the degree of feature degradation. A spatial attention evaluation network is applied to the fusion layer to generate confidence masks for the two feature streams. In the degraded regions of the visual and radar modalities, the weights are exponentially normalized. After weighted feature reconstruction, the results are fed into a shared detection head for 3D detection. This approach avoids excessive reliance on asymmetric failure mode noise, maximizing the preservation of semantic features from the camera and geometric information from the radar, achieving robust and accurate perception results.

[0038] In some embodiments, the above-described camera and lidar adaptive gating BEV feature fusion perception method for foggy off-road environments can also be implemented in the following ways.

[0039] Reference Figure 2 This invention proposes a foggy environment perception method based on the BEVFusion architecture, comprising the following steps: Step 21: Use the camera driver and LiDAR driver to acquire visible light image data and LiDAR point cloud data, and perform time synchronization; Step 22: Input the synchronized image data and point cloud data into the dual-stream feature extraction module to obtain visual semantic features and radar geometric features; Step 23: Project the visual features into the bird's-eye view (BEV) space through LSS view transformation, and perform feature-level fusion with radar features to obtain a multimodal fusion feature map; Step 24: Perform 3D target detection based on multimodal fusion feature maps and output the 3D detection results of obstacles in the off-road environment.

[0040] Reference Figure 3 This invention proposes a PTP time synchronization and data acquisition connection method, comprising the following steps: Step 31: Use GPS / BeiDou timestamp as a reference for hardware triggering to synchronize the vehicle computing platform's time to the global time; Step 32: Connect the vehicle computing platform, network switch, camera, and LiDAR via network cables, ensuring that all four support the IEEE 1588 PTP protocol; Step 33: Make the computing platform the PTP master and the camera and LiDAR the PTP slave to complete the time synchronization between the camera and LiDAR and the computing platform.

[0041] Reference Figure 4 This invention discloses a visual feature extraction method, which specifically includes the following steps: Step 41: Acquire the synchronized visible light image data and pass it as input to the visual feature extraction backbone network; Step 42: Perform preliminary feature extraction on the visible light image data using large kernel depth convolution, and effectively suppress local high-frequency fog noise in the image by relying on the natural spatial smoothness characteristics of the large receptive field. Step 43: Input the features processed by deep convolution into the Global Response Normalization (GRN) module to enhance feature competition between network channels and prevent the weak features of small targets at a distance from collapsing under severe weather conditions. The visual backbone network uses ConvNeXt-V2, which utilizes its 7×7 large kernel depth to separate convolutional local noise. The GRN module introduced in each block is calculated using the following formula:

[0042]

[0043] The GRN module dynamically suppresses the response of the background channel and enhances the channel activation values ​​of weak targets (such as small, distant obstacles in fog).

[0044] Step 44: Further utilize deep convolution to extract deep multi-channel features and complete the high-order semantic encoding of visual features; Step 45: Input the extracted deep features from different levels into the Feature Pyramid Network (FPN) to perform top-down multi-scale feature fusion; Step 46: The final output contains multi-scale visual semantic features with rich semantic and texture details, providing reliable visual prior information for subsequent multimodal fusion.

[0045] Reference Figure 5 This invention discloses a radar point cloud feature extraction method, which specifically includes the following steps: Step 51: Acquire synchronized LiDAR point cloud data; Step 52: Divide the point cloud data into spatial pillars, discretizing the continuous and disordered three-dimensional spatial point cloud into uniformly distributed vertical grid pillars on the XY horizontal plane. Step 53: Use the Pillar feature encoding module to extract the local point cloud geometry and reflectivity features contained in each vertical grid cylinder to generate a high-dimensional cylinder feature representation; for each point in the cylinder, extract features through a multilayer perceptron (MLP) and then perform max pooling to generate a dense cylinder tensor.

[0046] Step 54: Perform a two-dimensional pseudo-image mapping operation (Scatter) to project and flatten the extracted high-dimensional cylinder features along the height direction (Z-axis), transforming them into a two-dimensional grid space; Step 55: Input the mapped 2D feature map into the 2D CNN backbone network for further deep feature extraction and receptive field expansion; Step 56: Finally, output radar BEV pseudo-image features with high real-time performance, completing the rapid conversion from 3D point cloud to 2D bird's-eye view features.

[0047] Reference Figure 6 This invention proposes an LSS viewpoint transformation and feature-level fusion method, comprising the following steps: Step 61: Apply a convolutional layer to the extracted visual semantic features to predict the discrete depth probability distribution of pixels in the two-dimensional image; Step 62: Based on the camera intrinsic and extrinsic parameter matrices and discrete depth probability distribution, the two-dimensional visual features are enhanced into three-dimensional view frustum features, and then flattened and projected onto a unified bird's-eye view (BEV) grid to obtain visual BEV features. Step 63: Input the visual BEV features and radar BEV features into the spatial attention evaluation network to generate a confidence mask to characterize the degree of feature degradation in foggy weather. Step 64: Dynamically allocate adaptive weights for vision and radar based on confidence masks, and perform pixel-level weighted collaborative reconstruction of the two features to obtain the final multimodal fusion feature map.

[0048] Dynamic weight allocation and collaborative reconstruction of modal features. Let the visual BEV feature be... Radar BEV characteristics are Both feature dimensions are C×H×W, where C, H, and W represent the number of channels, the height of the BEV grid, and the width of the grid, respectively.

[0049] First, the two features mentioned above are concatenated along the channel dimension to obtain the concatenated features. Its dimensions become 2C×H×W. The specific formula is:

[0050] A confidence mask is then generated using a spatial attention evaluation network. The calculation process is as follows:

[0051] Where δ represents the ReLU activation function. Used to aggregate local spatial information The number of channels is reduced to 2, corresponding to the initial confidence scores of the features for vision and radar, respectively.

[0052] Next, the confidence mask is Softmax exponentially normalized along the channel dimension to obtain an adaptive visual spatial weight map. With radar spatial weight :

[0053]

[0054] Final multimodal fusion features :

[0055] Reference Figure 7 This invention proposes a training method that incorporates data augmentation, deep strong supervision, and multi-stage fine-tuning, comprising the following steps: Step 71: Foggy Multimodal Data Augmentation and Ground Value Label Construction. Obtain the original visible light image and LiDAR point cloud synchronized dataset. For off-road environments, perform 3D data augmentation in the point cloud space using 3D random flipping, global scaling, and translation; simultaneously, to maintain consistency in the multimodal space, the transformation matrix of the 3D data augmentation is synchronously inversely calculated into the intrinsic and extrinsic parameter matrices in the image domain.

[0056] Using the original sparse LiDAR point cloud synchronized with time, it is projected onto a two-dimensional image plane through camera intrinsic and extrinsic parameters; the projected continuous depth values ​​are discretized into multiple depth intervals according to the depth preset range set by LSS, and converted into a one-hot encoding format, which serves as the true depth label for the image depth prediction network.

[0057] Step 72: Construct the multi-task joint loss function. The network's total loss function. Loss due to 3D object detection and LSS depth prediction loss The composition is balanced using weighting coefficients. The specific formula is:

[0058] Among them, depth prediction loss A binary cross-entropy loss is employed to supervise the alignment of the discrete depth probability distribution output by the visual LSS module with the true one-hot depth encoding distribution, ensuring the geometric accuracy of visual feature projection into the BEV space. Object detection loss. Further breakdown into heatmap-based classification loss and 3D bounding box regression loss :

[0059] To address the severe imbalance between positive and negative samples caused by distant small targets and occluded targets in foggy off-road environments, a classification loss method is proposed. Gaussian focus loss; bounding box regression loss. L1 loss is used to regress the three-dimensional center coordinates, length, width, height, and other attributes of obstacles.

[0060] Step 73: Perform distributed, multi-stage progressive training. Due to the large and heterogeneous number of parameters in the multimodal fusion network, a three-stage training strategy is adopted to prevent the "feature-dominated" phenomenon caused by the inconsistent convergence speed of the two backbone networks in the early stage of training.

[0061] The first stage involves single-modal pre-training: disconnecting the multimodal fusion module. The ConvNeXt-V2 visual backbone network is pre-trained separately using an image dataset; the PointPillars radar backbone network and the 3D detection head are pre-trained separately using a point cloud dataset, enabling each to possess basic feature representation capabilities.

[0062] The second stage involves visual depth training: freezing the parameters of the PointPillars network. The ConvNeXt-V2 pre-trained in the first stage is connected to the LSS module. Using the depth real labels constructed in step 6.1, the depth distribution prediction network of the visual branch is optimized separately with a large learning rate, so that it can still output accurate depth frustums in foggy images.

[0063] The third stage involves adaptive gating end-to-end fine-tuning: loading the pre-trained weights from the first and second stages, and opening all network parameters, including the adaptive gating module. Synthetic foggy noise data is introduced, and the AdamW optimizer is used, combined with a cosine annealing learning rate decay strategy for end-to-end joint fine-tuning. Through backpropagation, the spatial attention evaluation network is driven to automatically learn the implicit mapping relationship between fog occlusion and feature degradation, thereby optimizing the optimal gating confidence mask allocation logic. Simultaneously, gradient clipping is added during training to prevent gradient explosion caused by extreme foggy samples, ensuring the stability of model convergence.

[0064] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A camera and lidar adaptive gating BEV feature fusion perception method for foggy off-road environment, characterized in that, Includes the following steps: S1: Synchronize visible light image data and lidar point cloud data in time to obtain synchronized image data and point cloud data; S2: Construct a multi-task joint perception model, train the multi-task joint perception model, and obtain a trained multi-task joint perception model. S3: Use the trained multi-task joint perception model to detect the synchronized image data and point cloud data to obtain the three-dimensional detection results of obstacles in the off-road environment.

2. The camera and lidar adaptive gating BEV feature fusion perception method for foggy off-road environments according to claim 1, characterized in that, The multi-task joint perception model includes a dual-stream feature extraction module, an adaptive gating fusion module, and a 3D detection head. The dual-stream feature extraction module includes a visual backbone network, an LSS module, and a radar backbone network. The visual backbone network is used to extract multi-scale visual semantic features from synchronized image data. The LSS module is used to project the multi-scale visual semantic features onto a bird's-eye view space to obtain visual BEV features. The radar backbone network is used to perform two-dimensional bird's-eye view feature transformation on synchronized point cloud data to obtain radar geometric features. The adaptive gating fusion module is used to perform feature-level fusion of the visual BEV features and the radar geometric features to obtain multi-modal fusion features. The 3D detection head is used to perform 3D target detection on the multi-modal fusion features to obtain three-dimensional detection results.

3. The camera and lidar adaptive gating BEV feature fusion perception method for foggy off-road environments according to claim 2, characterized in that, The specific configuration of the adaptive gating fusion module is as follows: The visual BEV features and the radar geometric features are spliced ​​together along the channel dimension to obtain spliced ​​features; Based on the splicing features, a spatial attention evaluation network is used to generate a confidence mask; The confidence mask is exponentially normalized along the channel dimension to obtain adaptive visual spatial weights and radar spatial weights. Based on adaptive visual spatial weights and radar spatial weights, visual BEV features are fused with radar geometric features to obtain the final multimodal fusion features.

4. The foggy off-road environment oriented camera and lidar adaptive-gated BEV feature fusion perception method according to claim 1, characterized in that, The training process of the multi-task joint perception model includes: constructing the depth real labels of the visual backbone network based on the original visible light image and the synchronous dataset of lidar point cloud; The multi-task joint perception model is trained in a distributed, multi-stage, progressive manner using the deep real labels and the multi-task joint loss function to obtain a trained multi-task joint perception model, specifically including: The visual backbone network of the multi-task joint perception model is pre-trained using an image dataset alone to obtain a pre-trained visual backbone network. The radar backbone network and 3D detection head are pre-trained using a point cloud dataset alone to obtain a pre-trained radar backbone network and 3D detection head. The parameters of the radar backbone network are frozen, the pre-trained visual backbone network is connected to the LSS module, and the visual backbone network is optimized separately using the depth real labels to obtain the optimized visual backbone network. Based on the optimized visual backbone network, LSS module, adaptive gating fusion module, pre-trained radar backbone network and 3D detection head, end-to-end joint fine-tuning is performed using synthetic fog noise data to obtain a trained multi-task joint perception model.

5. The foggy off-road environment oriented camera and lidar adaptive-gated BEV feature fusion perception method according to claim 4, characterized in that, The formula for calculating the multi-task joint loss function is as follows: in, For multi-task joint loss function, For 3D object detection loss, For LSS depth prediction loss, These are the weighting coefficients for the LSS depth prediction loss. For heatmap-based classification loss, For 3D bounding box regression loss, These are the weighting coefficients for the 3D bounding box regression loss.

6. A camera and lidar adaptive gating BEV feature fusion perception system for foggy off-road environment, characterized in that, The system includes the following modules: The data acquisition and synchronization module is configured to synchronize visible light image data and lidar point cloud data in time to obtain synchronized image data and point cloud data. The model building and training module is configured to: build a multi-task joint perception model, train the multi-task joint perception model, and obtain a trained multi-task joint perception model. The multi-task joint perception and detection module is configured to use a trained multi-task joint perception model to detect the synchronized image data and point cloud data, thereby obtaining the three-dimensional detection results of obstacles in the off-road environment.

7. The foggy off-road environment oriented camera and lidar adaptive-gated BEV feature fusion perception system according to claim 6, characterized in that, The multi-task joint perception model includes a dual-stream feature extraction module, an adaptive gating fusion module, and a 3D detection head. The dual-stream feature extraction module includes a visual backbone network, an LSS module, and a radar backbone network. The visual backbone network is used to extract multi-scale visual semantic features from synchronized image data. The LSS module is used to project the multi-scale visual semantic features onto a bird's-eye view space to obtain visual BEV features. The radar backbone network is used to perform two-dimensional bird's-eye view feature transformation on synchronized point cloud data to obtain radar geometric features. The adaptive gating fusion module is used to perform feature-level fusion of the visual BEV features and the radar geometric features to obtain multi-modal fusion features. The 3D detection head is used to perform 3D target detection on the multi-modal fusion features to obtain three-dimensional detection results.

8. The foggy off-road environment oriented camera and lidar adaptive-gated BEV feature fusion perception system according to claim 7, characterized in that, The specific configuration of the adaptive gating fusion module is as follows: The visual BEV features and the radar geometric features are spliced ​​together along the channel dimension to obtain spliced ​​features; Based on the splicing features, a spatial attention evaluation network is used to generate a confidence mask; The confidence mask is exponentially normalized along the channel dimension to obtain adaptive visual spatial weights and radar spatial weights. Based on adaptive visual spatial weights and radar spatial weights, visual BEV features are fused with radar geometric features to obtain the final multimodal fusion features.

9. The foggy off-road environment oriented camera and lidar adaptive-gated BEV feature fusion perception system of claim 6, wherein, The training process of the multi-task joint perception model includes: constructing the depth real labels of the visual backbone network based on the original visible light image and the synchronous dataset of lidar point cloud; The multi-task joint perception model is trained in a distributed, multi-stage, progressive manner using the deep real labels and the multi-task joint loss function to obtain a trained multi-task joint perception model, specifically including: The visual backbone network of the multi-task joint perception model is pre-trained using an image dataset alone to obtain a pre-trained visual backbone network. The radar backbone network and 3D detection head are pre-trained using a point cloud dataset alone to obtain a pre-trained radar backbone network and 3D detection head. The parameters of the radar backbone network are frozen, the pre-trained visual backbone network is connected to the LSS module, and the visual backbone network is optimized separately using the depth real labels to obtain the optimized visual backbone network. Based on the optimized visual backbone network, LSS module, adaptive gating fusion module, pre-trained radar backbone network and 3D detection head, end-to-end joint fine-tuning is performed using synthetic fog noise data to obtain a trained multi-task joint perception model.

10. The foggy off-road environment oriented camera and lidar adaptive-gated BEV feature fusion perception system according to claim 9, characterized in that, The formula for calculating the multi-task joint loss function is as follows: wherein, is a multi-task joint loss function, is a 3D object detection loss, is a LSS depth prediction loss, is a weight coefficient of the LSS depth prediction loss, is a classification loss based on Heatmap, is a 3D bounding box regression loss, is a weight coefficient of the 3D bounding box regression loss.

Images