Adaptive obstacle detection method and device for adverse weather conditions

Abstract

This application discloses an adaptive obstacle detection method and apparatus for adverse weather conditions, relating to the field of autonomous driving technology. The method acquires multi-source sensor data from an onboard vehicle, combines it with pre-calibrated sensor parameters to generate spatiotemporally aligned multi-source sensor data, extracts features to obtain feature maps for each modality, and projects all feature maps onto a bird's-eye view BEV space of the same resolution, using the LiDAR coordinate system as a reference, outputting a multi-modal BEV feature set. The feature set is concatenated along channels to form global state features, which are input into a pre-trained dual-branch adaptive decision network to obtain the output decision result, thereby generating the final image BEV features. Combining the multi-modal feature confidence weight matrix, the final image BEV features and the multi-modal BEV feature set are weighted to obtain globally fused BEV features, which are input into a 3D obstacle detection head for decoding and output detection results. This application can improve the robustness of multi-sensor fusion perception and obstacle detection accuracy under adverse weather conditions.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to an adaptive obstacle detection method for adverse weather conditions, an adaptive obstacle detection device for adverse weather conditions, an electronic device, and a storage medium. Background Technology

[0002] With the continuous development and iteration of autonomous driving technology, environmental perception algorithms have gradually evolved from early decision-level fusion to the more advantageous feature-level fusion stage. The core of this fusion method is to map data collected from multiple heterogeneous sensors such as cameras, LiDAR, and millimeter-wave radar to a unified feature space (such as the bird's-eye view BEV space) for fusion processing. By mining and utilizing richer contextual information, it effectively improves the accuracy and robustness of target detection, providing crucial support for the safe operation of autonomous driving systems.

[0003] However, this fusion method is highly dependent on the quality of the original sensor data. In severe weather conditions such as heavy rain, heavy snow, and dense fog, its inherent defects are significantly amplified, becoming a key bottleneck restricting the perception performance of autonomous driving. Specifically, severe weather causes differentiated performance degradation for different sensors: visible light camera images experience a sharp drop in contrast, blurred or even lost details, making it difficult to identify target outlines; the laser beam emitted by lidar is scattered and absorbed by raindrops, snowflakes, and fog particles, making the collected point cloud data sparse and containing a large number of noise points, failing to accurately reflect the position and shape of the target; in contrast, although millimeter-wave radar has a longer wavelength and stronger penetration ability in rain, snow, and fog, and the target distance and speed information it acquires is relatively stable, single-modal data still cannot meet the perception needs in complex scenarios. The degradation of the aforementioned sensor data directly leads to a decrease in the quality of features extracted from the original data, which in turn causes a sharp decline in the performance of subsequent fusion perception algorithms, posing a serious threat to driving safety.

[0004] To address the aforementioned technical challenges, existing technologies have yielded relevant solutions, primarily involving the addition of independent weather classification modules and single-modal data recovery modules (such as image deraining / defogging modules and point cloud denoising modules) to the detection algorithms. These approaches attempt to mitigate the impact of severe weather by first recovering degraded data and then performing feature fusion, achieving some research progress. However, in-depth analysis reveals that these methods still have significant inherent limitations, specifically in the following three aspects:

[0005] 1. Lack of cross-modal collaborative mechanisms in data recovery: Existing data recovery methods often process single-modal data independently. For example, image dehazing modules only input degraded image data for repair, and point cloud denoising modules only process contaminated point clouds. This independent processing mode ignores the complementary potential inherent in the differences in the degree of damage to different sensors under adverse weather conditions. Millimeter-wave radar, with its strong penetration capability, can still provide stable target spatial information under adverse weather conditions. If this information can be used as a guide, theoretically, the accuracy of recovering degraded data such as images and point clouds can be greatly improved. However, the existing "additional" module design has not built such a deep, context-dependent cross-modal collaborative recovery mechanism, resulting in limited data recovery effects. It may even introduce new noise due to erroneous "repair," further affecting subsequent sensing performance.

[0006] 2. Rigid and unadaptable recovery module triggering strategies: High-quality data recovery algorithms (especially deep learning-based methods) typically have high computational complexity, increasing system power consumption. Most existing systems rely on preset categories or fixed thresholds output by the weather classification module to determine whether to activate the recovery module. This rule-based triggering strategy cannot adapt to the dynamic changes in real-world scenarios: in mild degradation scenarios, activating the recovery module unnecessarily increases system power consumption, resulting in resource waste; while in sudden severe degradation scenarios, improper threshold settings may prevent the recovery module from triggering in a timely manner, failing to effectively alleviate data degradation and thus affecting the perception reliability of the autonomous driving system.

[0007] 3. Lack of Dynamic Adaptability in Multimodal Feature Weight Allocation: In the feature fusion stage, most existing methods adopt fixed weight allocation or only weight based on simple signal-to-noise ratio estimation, failing to fully consider the dynamic attenuation characteristics of different sensor performances under specific weather conditions. For example, in dense fog, the perception performance of lidar will significantly decrease, and the reliability of its extracted features is far lower than that of millimeter-wave radar. If a fixed weight allocation strategy is still used, assigning high weights to the degraded features of lidar, the degraded features will contaminate the overall fusion result, significantly reducing the accuracy of the perception algorithm. An ideal feature fusion strategy should be able to flexibly adjust the contribution of each modal feature according to the real-time environmental state and the dynamic performance of each sensor to optimize the fusion effect. Summary of the Invention

[0008] To address the shortcomings of the existing technologies, the present invention aims to provide an adaptive obstacle detection method, an adaptive obstacle detection device, an electronic device, and a storage medium for adverse weather conditions. The invention aims to construct a multimodal collaborative data recovery mechanism, combine it with reinforcement learning strategies, and adaptively decide the start / stop of the recovery module and the weight allocation of multimodal features based on the final perception effect. This achieves global optimization of perception performance and computational efficiency, effectively solving the problems of insufficient perception accuracy and poor reliability in autonomous driving under adverse weather conditions.

[0009] This invention provides the following solution:

[0010] According to one aspect of this application, an adaptive obstacle detection method for severe weather conditions is provided, comprising the following steps:

[0011] Acquire synchronous raw data from vehicle-mounted multi-source sensors, combine it with pre-calibrated sensor parameters, preprocess the synchronous raw data, and generate spatiotemporally aligned multi-source sensor data.

[0012] Feature extraction is performed on spatiotemporally aligned multi-source sensor data to obtain feature maps of the corresponding modes;

[0013] Using the lidar coordinate system as the global reference, the feature maps of all modes are uniformly projected onto the bird's-eye view BEV space of the same resolution, completing the alignment of spatial and channel dimensions, and outputting the aligned multimodal BEV feature set;

[0014] The multimodal BEV feature set is concatenated along the channel dimension to obtain the global state feature. The global state feature is then input into a pre-trained dual-branch adaptive decision network to generate the output decision result.

[0015] Based on the output decision results, the final image BEV features are generated;

[0016] Obtain the multimodal feature confidence weight matrix, and perform weighted processing on the final image BEV features and the multimodal BEV feature set based on the weight matrix;

[0017] The weighted BEV features are concatenated and fused to obtain the global fused BEV features.

[0018] The global fused BEV features are input into the 3D obstacle detection head, and the detection results are decoded and output.

[0019] Further preprocessing includes: data distortion correction, timestamp alignment, and global coordinate system unification.

[0020] Furthermore, the multimodal BEV feature set includes image BEV features, lidar BEV features, and millimeter-wave radar BEV features.

[0021] Furthermore, the detection results include: obstacle type, 3D spatial location, size, heading angle, and speed.

[0022] Furthermore, the output decision results include:

[0023] Output a binarized start / stop signal to control the start and stop of the image feature recovery module;

[0024] Output a multimodal feature confidence weight matrix that matches the dimensions of the multimodal BEV feature set.

[0025] Furthermore, the dual-branch adaptive decision network is trained based on a deep reinforcement learning framework, and its parameters are optimized using the real-time performance of obstacle detection as the core reward signal.

[0026] Furthermore, perform the corresponding operation based on the start / stop signal:

[0027] If the start / stop signal is enabled, the image BEV feature is used as the query, and the multimodal guidance feature that integrates the millimeter-wave radar BEV feature is used as the key and value. Stable context information is extracted through a multi-head cross-attention mechanism to generate the restored image BEV feature.

[0028] If the start / stop signal is off, the original image BEV features are directly used as the final image BEV features.

[0029] Furthermore, the dual-branch adaptive decision network and image feature recovery module employ a two-stage alternating training strategy, specifically including:

[0030] Phase 1: Pre-training of the basic perception model. The image feature recovery module and the dual-branch adaptive decision network are turned off. A real-world clear weather dataset is used, and the ground truth of obstacle 3D annotation is used as the supervision signal. End-to-end pre-training is performed on the feature extraction network, BEV projection module, fixed weight fusion module and detection head to obtain a converged baseline model and pre-trained weights.

[0031] Phase Two: Core Module Optimization and Full Model Fine-Tuning, consisting of two alternating sub-steps:

[0032] Sub-step 1: Freeze training. Freeze all weights of the baseline model and only open the parameters of the dual-branch adaptive decision network and the image feature recovery module. Use a multi-weather condition dataset generated by a high-fidelity simulation platform to optimize the image feature recovery module with mean squared error loss and optimize the dual-branch adaptive decision network with the goal of maximizing cumulative reward.

[0033] Sub-step 2: Joint fine-tuning of the entire model. Unfreeze all weights of the entire model, and use a mixed reality severe weather dataset and a high-fidelity synthetic dataset to perform end-to-end fine-tuning with a multi-task joint loss function until the total model loss converges.

[0034] According to two aspects of this application, an adaptive obstacle detection device for adverse weather conditions is provided, comprising:

[0035] The system includes a data acquisition and preprocessing module, a feature extraction module, a feature projection module, a feature stitching module, a final feature generation module, a weighted processing module, a stitching and fusion module, and a result output module.

[0036] The data acquisition and preprocessing module is used to acquire synchronous raw data from vehicle-mounted multi-source sensors, and preprocess the synchronous raw data in combination with pre-calibrated sensor parameters to generate spatiotemporally aligned multi-source sensor data.

[0037] The feature extraction module is used to extract features from the spatiotemporally aligned multi-source sensor data and obtain feature maps of the corresponding modes.

[0038] The feature projection module is used to project the feature maps of all modes onto the bird's-eye view BEV space of the same resolution, using the LiDAR coordinate system as the global reference, to complete the alignment of the spatial and channel dimensions, and output the aligned multimodal BEV feature set.

[0039] The feature concatenation module is used to concatenate the multimodal BEV feature set along the channel dimension to obtain global state features. The global state features are then input into a pre-trained dual-branch adaptive decision network to generate output decision results.

[0040] The final feature generation module is used to generate the final image BEV features based on the output decision results;

[0041] The weighted processing module is used to obtain the multimodal feature confidence weight matrix, and to perform weighted processing on the final image BEV features and the multimodal BEV feature set based on the weight matrix.

[0042] The splicing and fusion module is used to perform channel splicing and fusion encoding on the weighted BEV features to obtain the global fused BEV features;

[0043] The results output module is used to input the globally fused BEV features into the 3D obstacle detection head and decode and output the detection results.

[0044] According to three aspects of this application, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0045] The memory stores a computer program that, when executed by a processor, causes the processor to perform steps of an adaptive obstacle detection method for adverse weather conditions.

[0046] According to four aspects of this application, a computer-readable storage medium is provided that stores a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of an adaptive obstacle detection method for adverse weather conditions.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] This application employs a multi-head cross-attention mechanism that uses degraded image BEV features as queries and multimodal fused BEV features integrating millimeter-wave radar information and multiple image information as keys and values. The algorithm can automatically extract complementary information beneficial for degraded image restoration from the fused multimodal information (e.g., relatively stable target contours and distance cues even in adverse weather conditions). This mechanism provides strong structural constraints and semantic guidance for feature recovery of degraded images, significantly improving feature quality in severely degraded scenarios.

[0049] This application employs a dual-decision-making function to determine whether to activate the image restoration module and to output a sensor confidence weight matrix for feature fusion. The training of this mechanism does not rely on preset rules but instead utilizes reinforcement learning, using the final performance of the obstacle detection module as the reward signal. By designing a reasonable reward function (rewarding computational savings and performance improvements while penalizing unnecessary computational consumption and performance degradation), the system can autonomously learn the optimal resource scheduling and feature fusion strategies under different environmental conditions. Attached Figure Description

[0050] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1This is a flowchart of an adaptive obstacle detection method for severe weather conditions provided by one or more embodiments of the present invention.

[0052] Figure 2 This is a structural diagram of an adaptive obstacle detection device for severe weather conditions provided by one or more embodiments of the present invention.

[0053] Figure 3 This is a flowchart of an adaptive obstacle detection method for severe weather conditions provided by a specific embodiment of the present invention.

[0054] Figure 4 This is a block diagram of an electronic device structure for an adaptive obstacle detection method for severe weather conditions, provided by one or more embodiments of the present invention. Detailed Implementation

[0055] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0057] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0058] It should be understood that although the terms first, second, third, etc., may be used in the embodiments of this application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, first may also be referred to as second without departing from the scope of the embodiments of this application, and similarly, second may also be referred to as first.

[0059] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0060] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0061] It should be noted that any symbols and / or numbers present in the specification that are not marked in the accompanying drawings are not reference numerals.

[0062] Figure 1 This is a flowchart of an adaptive obstacle detection method for severe weather conditions provided by one or more embodiments of the present invention.

[0063] like Figure 1 As shown, it includes the following steps:

[0064] Step S1: Obtain the synchronous raw data of the vehicle-mounted multi-source sensors, and preprocess the synchronous raw data in combination with the pre-calibrated parameters of each sensor to generate spatiotemporally aligned multi-source sensor data.

[0065] Specifically, preprocessing includes: data distortion removal, timestamp alignment, and global coordinate system unification.

[0066] By correcting data distortion, aligning timestamps, and unifying global coordinates, the problems of high sensor noise, asynchrony, and coordinate offset under severe weather conditions are solved.

[0067] This provides a high-quality and consistent input foundation for subsequent integration, preventing detection failures due to inaccurate data.

[0068] Step S2: Extract features from spatiotemporally aligned multi-source sensor data to obtain feature maps of the corresponding modes;

[0069] Step S3: Using the lidar coordinate system as the global reference, project the feature maps of all modes onto the bird's-eye view BEV space of the same resolution to complete the alignment of the spatial and channel dimensions, and output the aligned multimodal BEV feature set.

[0070] Specifically, the multimodal BEV feature set includes image BEV features, lidar BEV features, and millimeter-wave radar BEV features.

[0071] By uniformly projecting the features of images, LiDAR, and millimeter-wave radar onto a BEV space of the same resolution, the misalignment of viewpoint, scale, and space is eliminated.

[0072] Achieving multimodal complementarity: LiDAR has high geometric accuracy, millimeter-wave radar has strong resistance to rain and fog, and rich image texture, forming redundant and reliable perception under adverse weather conditions.

[0073] Step S4: Concatenate the multimodal BEV feature set along the channel dimension to obtain the global state feature, and input the global state feature into the pre-trained dual-branch adaptive decision network to generate the output decision result;

[0074] Specifically, the output decision results include:

[0075] Output a binarized start / stop signal to control the start and stop of the image feature recovery module;

[0076] Output a multimodal feature confidence weight matrix that matches the dimensions of the multimodal BEV feature set.

[0077] The dual-branch adaptive decision network is trained based on a deep reinforcement learning framework, and its parameters are optimized using the real-time performance of obstacle detection as the core reward signal.

[0078] Specifically, it outputs a binarized start / stop signal and a multimodal confidence weight matrix, automatically allocating the confidence level of each sensor based on weather / environmental quality. When rain, fog, or other factors cause image degradation, it automatically reduces the image weight and increases the radar weight to prevent a single modal failure from causing overall detection failure.

[0079] Based on deep reinforcement learning, real-time performance is rewarded, ensuring accuracy while also taking into account the operating efficiency of automotive embedded platforms.

[0080] Furthermore, the corresponding operation is performed based on the start / stop signal:

[0081] If the start / stop signal is enabled, the image BEV feature is used as the query, and the multimodal guidance feature that integrates the millimeter-wave radar BEV feature is used as the key and value. Stable context information is extracted through a multi-head cross-attention mechanism to generate the restored image BEV feature.

[0082] If the start / stop signal is off, the original image BEV features are directly used as the final image BEV features.

[0083] Step S5: Based on the output decision results, generate the final image BEV features;

[0084] Step S6: Obtain the multimodal feature confidence weight matrix, and perform weighted processing on the final image BEV features and the multimodal BEV feature set according to the weight matrix;

[0085] Specifically, the weight matrix output by the decision network is used to weight the BEV features, thereby strengthening reliable modes and suppressing noisy modes.

[0086] Channel splicing and fusion coding form a globally fused BEV feature, which is more resistant to interference and more stable than fixed-weight fusion.

[0087] Step S7: Perform channel concatenation and fusion encoding on the weighted BEV features to obtain the global fused BEV features;

[0088] Step S8: Input the globally fused BEV features into the 3D obstacle detection head and decode to output the detection results.

[0089] Specifically, the detection results include: obstacle type, 3D spatial location, size, heading angle, and speed.

[0090] The system outputs complete obstacle information via a 3D obstacle detection head, meeting the practical needs of autonomous driving and supporting subsequent tasks such as traffic control, obstacle avoidance, and prediction. It can still provide continuous, stable, and high-precision 3D perception results even in adverse weather conditions.

[0091] In one specific embodiment, the dual-branch adaptive decision network and the image feature recovery module employ a two-stage alternating training strategy, specifically including:

[0092] Phase 1: Pre-training of the basic perception model. The image feature recovery module and the dual-branch adaptive decision network are turned off. A real-world clear weather dataset is used, and the ground truth of obstacle 3D annotation is used as the supervision signal. End-to-end pre-training is performed on the feature extraction network, BEV projection module, fixed weight fusion module and detection head to obtain a converged baseline model and pre-trained weights.

[0093] Phase Two: Core Module Optimization and Full Model Fine-Tuning, consisting of two alternating sub-steps:

[0094] Sub-step 1: Freeze training. Freeze all weights of the baseline model and only open the parameters of the dual-branch adaptive decision network and the image feature recovery module. Use a multi-weather condition dataset generated by a high-fidelity simulation platform to optimize the image feature recovery module with mean squared error loss and optimize the dual-branch adaptive decision network with the goal of maximizing cumulative reward.

[0095] Sub-step 2: Joint fine-tuning of the entire model. Unfreeze all weights of the entire model, and use a mixed reality severe weather dataset and a high-fidelity synthetic dataset to perform end-to-end fine-tuning with a multi-task joint loss function until the total model loss converges.

[0096] Specifically, by projecting multimodal features from images, LiDAR, and millimeter-wave radar into a unified bird's-eye view (BEV) space, and performing weighted fusion based on a multimodal feature confidence weight matrix, this approach fully leverages the complementary characteristics of LiDAR's high geometric accuracy, millimeter-wave radar's strong resistance to rain and fog interference, and image's rich texture information. Simultaneously, an image feature recovery module is incorporated to repair degraded image features according to environmental conditions, effectively overcoming the technical bottlenecks of single-sensor failure or excessive data noise under adverse weather conditions such as rain, snow, fog, strong light, and obstructions, significantly improving the stability and accuracy of obstacle detection.

[0097] The dual-branch adaptive decision network, based on a deep reinforcement learning framework, can output the confidence weight matrix of multimodal features and the start / stop signals of the image feature recovery module in real time. When good weather conditions are detected, the recovery module is turned off to reduce computational overhead; when severe weather is detected, the recovery module is automatically turned on and the weights of each modality are dynamically adjusted. This ensures detection accuracy while meeting the real-time requirements of the embedded vehicle platform, achieving an optimal balance between accuracy and efficiency.

[0098] By employing a two-stage alternating training strategy, the basic perception model is first pre-trained using a clear weather dataset to ensure good basic convergence performance. Then, multi-weather datasets generated by a high-fidelity simulation platform and mixed reality severe weather datasets are used to jointly fine-tune and optimize the core modules and the entire model. This strategy breaks away from excessive reliance on real extreme weather samples, significantly reduces training costs, and effectively improves the model's generalization ability and deployment adaptability in complex and ever-changing real-world traffic scenarios.

[0099] Furthermore, by directly decoding and outputting complete information such as the type, 3D spatial position, size, heading angle, and speed of obstacles through the 3D obstacle detection head, it can provide a continuous, reliable, and refined environmental perception foundation for path planning, decision control, and risk prediction of autonomous driving systems, thus meeting the practical application needs of high-level intelligent driving.

[0100] Figure 2 This is a structural diagram of an adaptive obstacle detection device for severe weather conditions provided by one or more embodiments of the present invention.

[0101] like Figure 2 As shown, it includes:

[0102] The system includes a data acquisition and preprocessing module, a feature extraction module, a feature projection module, a feature stitching module, a final feature generation module, a weighted processing module, a stitching and fusion module, and a result output module.

[0103] The data acquisition and preprocessing module is used to acquire synchronous raw data from vehicle-mounted multi-source sensors, and preprocess the synchronous raw data in combination with pre-calibrated sensor parameters to generate spatiotemporally aligned multi-source sensor data.

[0104] The feature extraction module is used to extract features from the spatiotemporally aligned multi-source sensor data and obtain feature maps of the corresponding modes.

[0105] The feature projection module is used to project the feature maps of all modes onto the bird's-eye view BEV space of the same resolution, using the LiDAR coordinate system as the global reference, to complete the alignment of the spatial and channel dimensions, and output the aligned multimodal BEV feature set.

[0106] The feature concatenation module is used to concatenate the multimodal BEV feature set along the channel dimension to obtain global state features. The global state features are then input into a pre-trained dual-branch adaptive decision network to generate output decision results.

[0107] The final feature generation module is used to generate the final image BEV features based on the output decision results;

[0108] The weighted processing module is used to obtain the multimodal feature confidence weight matrix, and to perform weighted processing on the final image BEV features and the multimodal BEV feature set based on the weight matrix.

[0109] The splicing and fusion module is used to perform channel splicing and fusion encoding on the weighted BEV features to obtain the global fused BEV features;

[0110] The results output module is used to input the globally fused BEV features into the 3D obstacle detection head and decode and output the detection results.

[0111] It is worth noting that although only some basic functional modules are disclosed in this embodiment, it does not mean that the composition of this system is limited to the above-mentioned basic functional modules. On the contrary, what this embodiment intends to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technology to form an infinite number of embodiments or technical solutions. That is to say, this system is open rather than closed. The fact that this embodiment only discloses a few basic functional modules does not mean that the scope of protection of the claims of this invention is limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above device is described separately according to its functions as various units and modules. Of course, in implementing this invention, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Figure 3 This is a flowchart of an adaptive obstacle detection method for severe weather conditions provided by a specific embodiment of the present invention.

[0114] like Figure 3 As shown, in a specific embodiment, the following steps are included:

[0115] Step 1: Dataset Preparation. The dataset consists of two parts: 1) Real-world dataset: This includes spatiotemporally aligned images, LiDAR point clouds, millimeter-wave radar data, and corresponding sensor calibration parameters. The data must include target annotation information (such as category, size, location, etc.). 2) High-fidelity simulation platform synthetic dataset: In addition to meeting the above requirements, it must also include simulated data from various sensors under multiple controllable weather parameters (such as precipitation intensity and fog concentration) for training and evaluating the system's performance under adverse weather conditions.

[0116] Step 2: Multimodal Feature Extraction and BEV Projection. Image features are extracted using a pre-trained ResNet50 network and a Feature Pyramid Network (FPN); point cloud features from LiDAR and millimeter-wave radar are extracted using two independent CenterPoint networks respectively. Subsequently, the features of all modalities are uniformly projected onto the BEV space centered on the LiDAR, forming an aligned BEV feature map.

[0117] Step 3: Adaptive Decision-Making Based on Reinforcement Learning. The input to the adaptive decision-making module is the BEV spatial features (image, LiDAR, millimeter-wave radar) obtained in Step 2. These three features are directly concatenated and input into the recovery start / stop subnetwork (determining whether the data recovery module needs to be activated in the current state) and the feature weight prediction subnetwork, respectively. The recovery start / stop decision subnetwork ultimately outputs the decision on whether to activate the image recovery module; the feature weight prediction subnetwork outputs the confidence weight matrix of each sensor feature, which will be directly used for the feature weighted fusion of the subsequent obstacle detection module. The reward function is designed to comprehensively consider the changes in detection accuracy and computational power consumption. Specifically, it can be quantified as: Reward = α * Δperformance - β * Recovery Activation Flag. Wherein, Δperformance is the difference in obstacle detection performance index (such as mAP) between the two cases where the recovery module is activated and not activated; α and β are balancing hyperparameters, respectively reflecting the encouragement of performance improvement and the suppression of computational power consumption; the recovery activation flag is 0 or 1, where 1 indicates that the recovery module is activated and consumes computational power.

[0118] Step 4: Multimodal Collaborative Image Feature Restoration. This module is activated when the decision module determines that restoration is needed. The input consists of the BEV features of the degraded image (obtained by projection from the original image features) and the BEV features of the millimeter-wave radar. First, the two are initially fused using a convolutional layer. Then, a multi-head cross-attention mechanism is employed, using the degraded image BEV features as the query and the fused features as the key and value, to extract contextual information useful for image restoration. The extracted information is concatenated with the original degraded image features and then processed by a decoder network to generate the restored image BEV features. The training loss function for this module is the mean squared error (MSE) between the restored features and the corresponding standard image BEV features under clear weather conditions. The restored features replace the original degraded image features and are input into the subsequent obstacle detection module.

[0119] Step 5: Weighted Feature Fusion and Obstacle Detection. The obstacle detection module receives BEV features from LiDAR, millimeter-wave radar, and (raw or restored) images. Before fusion, the features are adaptively weighted using the weight matrix output by the adaptive decision module. Then, the weighted multimodal features are fused in the BEV space and input into the detection head, ultimately outputting information such as the obstacle's category, location, and size.

[0120] Training strategy description: A phased alternating training strategy is adopted.

[0121] In the first stage, using real-world datasets, the feature extractor, fusion network, and detection head are pre-trained end-to-end without enabling the image feature recovery module and decision module, resulting in a basic multimodal perception model.

[0122] In the second stage, using a multi-weather condition dataset generated by a high-fidelity simulation platform, firstly, all network weights except for the adaptive decision-making module and the image restoration module are frozen. Then, through the reinforcement learning training process described above (step three), the strategy of the adaptive decision-making module is optimized separately. During this process, the parameters of the image feature restoration module are also supervised learning through its own MSE loss. Subsequently, all network weights are unfrozen, and end-to-end joint fine-tuning is performed with a small learning rate. At this point, the start / stop signal of the decision-making module acts as a conditional gate, controlling whether the gradient is backpropagated to the restoration module. The total detection loss (including the reinforcement learning loss of the decision-making module, the MSE loss of the restoration module, and the supervised loss of the obstacle detection head) jointly guides model optimization, thereby improving the overall perception robustness of the system under severe weather conditions.

[0123] Specifically, by using the relatively stable millimeter-wave radar BEV features under severe weather conditions as contextual information, and through a cross-attention mechanism that uses degraded image BEV features as a query and BEV features that integrate multimodal information as a key and value, the restoration and enhancement of degraded image features are guided.

[0124] By using a reinforcement learning framework, the online performance evaluation of the obstacle detection module is used as a reward signal. At the same time, the system learns and outputs decision signals for controlling the start and stop of the image restoration module, as well as a confidence weight matrix for weighted fusion of features from various sensors, thereby achieving a dynamic balance between perception performance and computational efficiency.

[0125] Figure 4 This is a block diagram of an electronic device structure for an adaptive obstacle detection method for severe weather conditions, provided by one or more embodiments of the present invention.

[0126] like Figure 4 As shown, this application provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0127] The memory stores a computer program that, when executed by the processor, causes the processor to perform steps of an adaptive obstacle detection method for adverse weather conditions.

[0128] This application also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of an adaptive obstacle detection method for adverse weather conditions.

[0129] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0130] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive obstacle detection method for severe weather conditions, characterized in that, Includes the following steps: Acquire synchronous raw data from vehicle-mounted multi-source sensors, combine it with pre-calibrated sensor parameters, preprocess the synchronous raw data to generate spatiotemporally aligned multi-source sensor data; Feature extraction is performed on the spatiotemporally aligned multi-source sensor data to obtain feature maps of the corresponding modes; Project the feature maps of all modalities onto the bird's-eye view BEV space to align the spatial and channel dimensions, and output the aligned multimodal BEV feature set; The multimodal BEV feature set is concatenated along the channel dimension to obtain global state features. The global state features are then input into a pre-trained dual-branch adaptive decision network to generate output decision results. Based on the output decision results, the final image BEV features are generated; Obtain the multimodal feature confidence weight matrix, and perform weighted processing on the final image BEV features and the multimodal BEV feature set according to the weight matrix; The weighted BEV features are concatenated and fused to obtain the global fused BEV features. The globally fused BEV features are input into the 3D obstacle detection head, and the detection results are decoded and output.

2. The adaptive obstacle detection method for severe weather conditions according to claim 1, characterized in that, The preprocessing includes: data distortion removal, timestamp alignment, and global coordinate system integration.

3. The adaptive obstacle detection method for severe weather conditions according to claim 1, characterized in that, The multimodal BEV feature set includes image BEV features, lidar BEV features, and millimeter-wave radar BEV features.

4. The adaptive obstacle detection method for severe weather conditions according to claim 1, characterized in that, The detection results include: obstacle type, 3D spatial location, size, heading angle, and speed.

5. The adaptive obstacle detection method for severe weather conditions according to claim 3, characterized in that, The output decision results include: Output a binarized start / stop signal to control the start and stop of the image feature recovery module; Output a multimodal feature confidence weight matrix that matches the dimensions of the multimodal BEV feature set.

6. The adaptive obstacle detection method for severe weather conditions according to claim 1, characterized in that, include: The dual-branch adaptive decision network is trained based on a deep reinforcement learning framework, and its parameters are optimized using the real-time performance of obstacle detection as the core reward signal.

7. The adaptive obstacle detection method for severe weather conditions according to claim 5, characterized in that, include: Perform the corresponding operation based on the start / stop signal: If the start / stop signal is enabled, the image BEV feature is used as the query, and the multimodal guidance feature that integrates the millimeter-wave radar BEV feature is used as the key and value. Stable context information is extracted through a multi-head cross-attention mechanism to generate the restored image BEV feature. If the start / stop signal is off, the original image BEV features are directly used as the final image BEV features.

8. The adaptive obstacle detection method for severe weather conditions according to claim 1, characterized in that, include: The dual-branch adaptive decision network and image feature recovery module employ a two-stage alternating training strategy, specifically including: Phase 1: Pre-training of the basic perception model. The image feature recovery module and the dual-branch adaptive decision network are turned off. A real-world clear weather dataset is used, and the ground truth of obstacle 3D annotation is used as the supervision signal. End-to-end pre-training is performed on the feature extraction network, BEV projection module, fixed weight fusion module and detection head to obtain a converged baseline model and pre-trained weights. Phase Two: Core Module Optimization and Full Model Fine-Tuning, consisting of two alternating sub-steps: Sub-step 1: Freeze training. Freeze all weights of the baseline model and only open the parameters of the dual-branch adaptive decision network and the image feature recovery module. Use a multi-weather condition dataset generated by a high-fidelity simulation platform to optimize the image feature recovery module with mean squared error loss and optimize the dual-branch adaptive decision network with the goal of maximizing cumulative reward. Sub-step 2: Joint fine-tuning of the entire model. Unfreeze all weights of the entire model, and use a mixed reality severe weather dataset and a high-fidelity synthetic dataset to perform end-to-end fine-tuning with a multi-task joint loss function until the total model loss converges.

9. An adaptive obstacle detection device for adverse weather conditions, characterized in that, include: The system includes a data acquisition and preprocessing module, a feature extraction module, a feature projection module, a feature stitching module, a final feature generation module, a weighted processing module, a stitching and fusion module, and a result output module. The data acquisition and preprocessing module is used to acquire synchronous raw data from vehicle-mounted multi-source sensors, and preprocess the synchronous raw data in combination with pre-calibrated sensor parameters to generate spatiotemporally aligned multi-source sensor data. The feature extraction module is used to extract features from the spatiotemporally aligned multi-source sensor data and obtain feature maps of the corresponding modes. The feature projection module is used to project the feature maps of all modes onto the bird's-eye view BEV space of the same resolution, using the LiDAR coordinate system as the global reference, to complete the alignment of the spatial and channel dimensions, and output the aligned multimodal BEV feature set. The feature concatenation module is used to concatenate the multimodal BEV feature set along the channel dimension to obtain global state features, and input the global state features into a pre-trained dual-branch adaptive decision network to generate output decision results; The final feature generation module is used to generate the final image BEV features based on the output decision results; The weighted processing module is used to obtain the multimodal feature confidence weight matrix, and to perform weighted processing on the final image BEV features and the multimodal BEV feature set according to the weight matrix. The splicing and fusion module is used to perform channel splicing and fusion encoding on the weighted BEV features to obtain the global fused BEV features; The result output module is used to input the globally fused BEV features into the 3D obstacle detection head and decode and output the detection results.

10. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the adaptive obstacle detection method for adverse weather conditions as described in any one of claims 1-8.

Images