Adaptive routing target detection method for all-weather multi-sensor fusion
Patent Information
- Application Number
- CN202611119748.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-27
- Publication Date
- 2026-10-27
Smart Images
Figure CN122990169A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and in particular to an adaptive routing target detection method for all-weather multi-sensor fusion. Background Technology
[0002] 3D object detection is one of the core tasks of environmental perception in autonomous driving systems. Its goal is to accurately identify and locate the 3D spatial position, size, and category of traffic participants such as vehicles, pedestrians, and cyclists. In related technologies, 3D object detection methods typically employ fixed sensor combinations and fusion strategies, such as multi-sensor fusion schemes that combine LiDAR with cameras or LiDAR with millimeter-wave radar. These schemes jointly model data collected by different sensors to achieve 3D object detection.
[0003] However, the relevant technologies employ fixed sensor combinations and fusion strategies, making it difficult to dynamically adjust according to different weather conditions and fully leverage the complementary advantages of different sensors, thus resulting in a decrease in the accuracy and robustness of 3D target detection in complex scenarios. Summary of the Invention
[0004] This application provides an adaptive routing target detection method for all-weather multi-sensor fusion, in order to solve the problem in related technologies where the use of fixed sensor combinations and fusion strategies makes it difficult to dynamically adjust according to different weather conditions, resulting in a decrease in the accuracy and robustness of 3D target detection in complex scenes.
[0005] The first aspect of this application provides an adaptive routing target detection method for all-weather multi-sensor fusion, comprising the following steps: extracting image features, LiDAR (Light Detection and Ranging) point cloud features, and 4D (Dimensional) millimeter-wave radar point cloud features from different modal data; mapping the image features and LiDAR point cloud features to a first bird's-eye view fusion feature using a first fusion branch, and / or mapping the LiDAR point cloud features and 4D millimeter-wave radar point cloud features to a second bird's-eye view fusion feature using a second fusion branch; and predicting the category and / or location of a 3D target based on the first bird's-eye view fusion feature and / or the second bird's-eye view fusion feature and a pre-constructed 3D target detection head.
[0006] By using the above technical means, the bird's-eye view fusion features generated by the first fusion branch and / or the second fusion branch can be flexibly fused with multimodal information such as images, LiDAR point clouds and 4D millimeter-wave radar point clouds, giving full play to the complementary advantages of different sensors; thereby predicting the category and / or location of three-dimensional targets, which can improve the accuracy of target recognition and spatial positioning, and enhance the stability and robustness of three-dimensional target detection in complex weather and dynamic traffic scenarios.
[0007] Optionally, in one embodiment of this application, the adaptive routing target detection method for all-weather multi-sensor fusion further includes: extracting features representing the weather semantics of the current scene to determine the weather prior information of the current environment; generating a gating control signal based on the weather prior information to determine the probability of the fusion branch being selected in the previous environment; and activating the first fusion branch or the second fusion branch according to the probability.
[0008] By using the above technical means, a gating control signal is generated based on prior weather information to select and activate the first fusion branch or the second fusion branch. The fusion path of the multimodal sensors can be adaptively adjusted according to the weather conditions of the current scene, thereby making full use of effective sensor information for different weather environments, improving the effectiveness of multimodal feature fusion, as well as the accuracy and environmental adaptability of 3D target detection.
[0009] Optionally, in one embodiment of this application, extracting image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features from different modal data includes: extracting image features using a preset two-dimensional backbone network; voxelizing the LiDAR point cloud to convert it into first raster data that meets a first preset rule condition, and obtaining LiDAR point cloud features based on the first raster data using a first preset voxel feature encoder; voxelizing the 4D millimeter-wave radar point cloud to convert it into second raster data that meets a second preset rule condition, and obtaining 4D millimeter-wave radar point cloud features based on the second raster data using a preset second voxel feature encoder.
[0010] Using the above techniques, image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features are extracted from images, LiDAR point clouds, and 4D millimeter-wave radar point clouds, respectively. Corresponding feature extraction networks can be used to process the data according to the characteristics of different modalities, thereby obtaining multimodal features containing rich semantic information, accurate three-dimensional geometric information, and information such as target distance, velocity, and spatial location. This provides a data foundation for selecting different fusion branches based on the weather environment and generating bird's-eye view fusion features.
[0011] Optionally, in one embodiment of this application, the first fusion branch is used to map image features and LiDAR point cloud features into a first bird's-eye view fusion feature, and / or the second fusion branch is used to map LiDAR point cloud features and 4D millimeter-wave radar point cloud features into a second bird's-eye view fusion feature, including: based on the first fusion branch, mapping image features into high-level semantic features to convert high-level semantic features into image BEV features; mapping LiDAR point cloud features into sparse three-dimensional features, and flattening the sparse three-dimensional features into the target BEV space to obtain LiDAR point cloud BEV features; and obtaining the first bird's-eye view fusion feature based on the image BEV features and LiDAR point cloud BEV features.
[0012] By employing the above technical means, the first fusion branch maps image features and LiDAR point cloud features into a first bird's-eye view fusion feature. This can be achieved by converting image features and LiDAR point cloud features into a unified bird's-eye view space, and then performing spatial alignment, feature interaction, and feature fusion. This allows the first bird's-eye view fusion feature to simultaneously contain rich semantic information from the image and precise three-dimensional geometric information from the LiDAR point cloud, thereby improving the accuracy of target category recognition and three-dimensional spatial positioning, and providing a more comprehensive feature representation for subsequent three-dimensional target detection.
[0013] Optionally, in one embodiment of this application, the first fusion branch is used to map image features and LiDAR point cloud features into a first bird's-eye view fusion feature, and / or the second fusion branch is used to map LiDAR point cloud features and 4D millimeter-wave radar point cloud features into a second bird's-eye view fusion feature, including: based on the second fusion branch, using each non-empty LiDAR voxel as an index, querying multiple nearest-neighbor millimeter-wave radar voxel features that meet preset range conditions; obtaining the fusion feature of the non-empty LiDAR voxel based on the multiple nearest-neighbor millimeter-wave radar voxel features; aggregating all fusion features to obtain the fusion feature, and adding the fusion feature to the LiDAR point cloud features to obtain the enhanced LiDAR point cloud feature; generating the second bird's-eye view fusion feature based on the LiDAR point cloud feature.
[0014] By employing the above technical means, the second fusion branch maps LiDAR point cloud features and 4D millimeter-wave radar point cloud features into a second bird's-eye view fusion feature. This transforms the two types of point cloud features into a unified bird's-eye view space. Through spatial alignment, feature interaction, and information fusion, the high-precision three-dimensional geometric information of LiDAR point clouds and the distance, velocity, and spatial position information of 4D millimeter-wave radar point clouds are fully utilized. This achieves complementary advantages of information from different sensors, thereby enhancing the integrity of target features in adverse weather or complex environments and improving the accuracy, stability, and environmental adaptability of three-dimensional target detection.
[0015] A second aspect of this application provides an adaptive routing target detection device for all-weather multi-sensor fusion, comprising: an extraction module for extracting image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features from different modal data; a mapping module for mapping the image features and LiDAR point cloud features to a first bird's-eye view fusion feature using a first fusion branch, and / or mapping the LiDAR point cloud features and 4D millimeter-wave radar point cloud features to a second bird's-eye view fusion feature using a second fusion branch; and a detection module for predicting the category and / or location of a three-dimensional target based on the first bird's-eye view fusion feature and / or the second bird's-eye view fusion feature and a pre-built three-dimensional target detection head.
[0016] Optionally, in one embodiment of this application, the adaptive routing target detection device for all-weather multi-sensor fusion further includes: a determination module, used to extract features characterizing the weather semantics of the current scene to determine the weather prior information of the current environment; a generation module, used to generate a gating control signal based on the weather prior information to determine the probability of the fusion branch being selected in the previous environment; and an activation module, used to activate the first fusion branch or the second fusion branch according to the probability.
[0017] Optionally, in one embodiment of this application, the extraction module includes: an extraction unit for extracting image features using a preset two-dimensional backbone network; a first conversion unit for voxelizing the LiDAR point cloud to convert it into first raster data that meets first preset rule conditions, and obtaining LiDAR point cloud features based on the first raster data using a first preset voxel feature encoder; and a second conversion unit for voxelizing the 4D millimeter-wave radar point cloud to convert it into second raster data that meets second preset rule conditions, and obtaining 4D millimeter-wave radar point cloud features based on the second raster data using a preset second voxel feature encoder.
[0018] Optionally, in one embodiment of this application, the mapping module includes: a mapping unit, configured to map image features to high-level semantic features based on a first fusion branch, so as to convert the high-level semantic features into image BEV features; a flattening unit, configured to map LiDAR point cloud features to sparse three-dimensional features, and flatten the sparse three-dimensional features to the target BEV space to obtain LiDAR point cloud BEV features; and a determination unit, configured to obtain a first bird's-eye view fusion feature based on the image BEV features and the LiDAR point cloud BEV features.
[0019] Optionally, in one embodiment of this application, the mapping unit includes: a query unit, used to query multiple nearest-neighbor millimeter-wave radar voxel features that satisfy a preset range condition, based on the second fusion branch and using each non-empty LiDAR voxel as an index; a fusion unit, used to obtain fusion features of the non-empty LiDAR voxels based on the multiple nearest-neighbor millimeter-wave radar voxel features; an aggregation unit, used to aggregate all fusion features to obtain fusion features, and add the fusion features to the LiDAR point cloud features to obtain enhanced LiDAR point cloud features; and a generation unit, used to generate a second bird's-eye view fusion feature based on the LiDAR point cloud features.
[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the adaptive routing target detection method for all-weather multi-sensor fusion as described in the above embodiments.
[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described adaptive routing target detection method for all-weather multi-sensor fusion.
[0022] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, is used to implement the above-described adaptive routing target detection method for all-weather multi-sensor fusion.
[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of an adaptive routing target detection method based on all-weather multi-sensor fusion, according to an embodiment of this application. Figure 2 This is a block diagram of an adaptive routing target detection device for all-weather multi-sensor fusion according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.
[0025] Figure label: 10-Adaptive routing target detection device for all-weather multi-sensor fusion; 100-Extraction module, 200-Mapping module, 300-Detection module; 301-Memory, 302-Processor, 303-Communication interface. Detailed Implementation
[0026] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0027] The following description, with reference to the accompanying drawings, describes an adaptive routing target detection method for all-weather multi-sensor fusion based on embodiments of this application. Addressing the technical problem mentioned in the background art, where the use of fixed sensor combinations and fusion strategies makes dynamic adjustment difficult according to different weather conditions, resulting in decreased accuracy and robustness of 3D target detection in complex scenarios, this application provides an adaptive routing target detection method for all-weather multi-sensor fusion. This method aims to dynamically adjust the multi-modal sensor fusion strategy based on real-time weather conditions, adaptively select appropriate fusion branches for different weather conditions, and fully leverage the complementary advantages of images, LiDAR point clouds, and 4D millimeter-wave radar point clouds to improve the accuracy, stability, and environmental adaptability of 3D target detection in complex weather and dynamic traffic scenarios.
[0028] Specifically, Figure 1 This is a flowchart illustrating an adaptive routing target detection method based on all-weather multi-sensor fusion provided in an embodiment of this application.
[0029] like Figure 1 As shown, this adaptive routing target detection method for all-weather multi-sensor fusion includes the following steps: In step S101, image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features of different modal data are extracted.
[0030] Here, different modal data refers to heterogeneous sensing data collected by different types of sensors, including but not limited to images. LiDAR point cloud and 4D millimeter-wave radar point cloud Where H is the image height, W is the image width, and N is the number of point clouds.
[0031] It can be explained that image features refer to the features obtained after processing two-dimensional image data acquired by a camera through a feature extraction network. These features are beneficial for extracting rich semantic information such as the color, texture, shape, and category of the target. However, they are easily affected by environmental factors such as rain, fog, snow, low light at night, and strong light, which can lead to a decrease in target visibility and feature quality. LiDAR point cloud features refer to the features obtained after extracting features from three-dimensional point cloud data acquired by a LiDAR. These features are beneficial for providing accurate three-dimensional spatial location, target size, and geometric structure information. However, they are easily affected by sparse point clouds, noise interference, and reduced effective detection range under adverse weather conditions such as rain, fog, and snow. 4D millimeter-wave radar point cloud features refer to the features obtained after processing point clouds acquired by 4D millimeter-wave radar. These features are beneficial for providing information such as the target's distance, spatial location, speed, and direction of motion, and have strong adaptability to adverse weather conditions. However, their point clouds are usually sparse, and their spatial resolution and target semantic representation capabilities are relatively limited, making it difficult to achieve fine target classification and three-dimensional bounding box localization on their own.
[0032] As a specific example, extracting image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features from different modal data includes: extracting image features using a preset two-dimensional backbone network; voxelizing the LiDAR point cloud to convert it into first raster data that meets a first preset rule condition, and obtaining LiDAR point cloud features based on the first raster data using a first preset voxel feature encoder; voxelizing the 4D millimeter-wave radar point cloud to convert it into second raster data that meets a second preset rule condition, and obtaining 4D millimeter-wave radar point cloud features based on the second raster data using a preset second voxel feature encoder.
[0033] First, in this embodiment, a two-dimensional backbone network, Swing Transformer (Shifted Window Transformer), can be used to extract features from image I. Where S is the downsampling step size, The number of channels representing the feature. The default 2D backbone network can be a Swin Transformer, or a network such as VGG (Visual Geometry Group) or EfficientNet (Efficient Convolutional Neural Network) that can be used to convert irregular 3D point cloud data into voxel feature representations with a uniform spatial structure.
[0034] Secondly, the embodiments of this application can be used for LiDAR point clouds. Voxelization is performed to transform the irregular point set into regular raster data, which is then passed through a HardSimpleVFE (Hard Simple Voxel Feature Encoder) to obtain LiDAR point cloud features. The voxel feature encoder can be HardSimpleVFE, or Dynamic VFE (Dynamic Voxel Feature Encoder), Pillar Feature Net, etc., which can be used to extract and aggregate features from point cloud data within voxels or pillars to generate an encoder with voxel feature representations containing spatial structure information.
[0035] This application embodiment can also be used for 4D millimeter-wave radar point clouds. Voxelization is performed to transform the irregular point set into regular raster data, which is then processed by the voxel feature encoder HardSimpleVFE to obtain 4D millimeter-wave radar point cloud features. .
[0036] The embodiments of this application extract image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features to obtain multimodal features with different spatial, semantic, and motion information, providing feature inputs for subsequent multimodal feature fusion and 3D target detection.
[0037] In step S102, the first fusion branch is used to map the image features and LiDAR point cloud features into the first bird's-eye view fusion features, and / or the second fusion branch is used to map the LiDAR point cloud features and 4D millimeter-wave radar point cloud features into the second bird's-eye view fusion features.
[0038] In the embodiments of this application, the fusion branch refers to a network structure or processing path used to align, interact with, and fuse feature information from different modal sensors. In the embodiments of this application, the fusion branch can receive image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features, and fully utilize the complementary information between different modal features through feature mapping, spatial alignment, cross-modal feature interaction, and feature weighted fusion to generate fused multimodal features.
[0039] Bird's-eye view fusion features refer to the fused features obtained by mapping or transforming image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features to a unified BEV (Bird's Eye View) spatial coordinate system, and then processing them through feature alignment, feature interaction, and feature fusion. This feature can comprehensively represent the spatial location, geometric structure, semantic information, and motion state of a target from a unified top-down perspective, providing a unified multimodal feature representation for subsequent 3D target detection.
[0040] This application employs two fusion branches, which can perform multimodal feature fusion processing separately for different scene conditions or sensor data quality conditions. For example, the first fusion branch... It can be used to process image features under good weather conditions and LiDAR point cloud features; the second fusion branch It can be used to process 4D millimeter-wave radar point cloud features and LiDAR point cloud features under adverse weather conditions. First fusion branch. Image features and LiDAR point cloud features can be mapped to bird's-eye view (BEV) fusion features using cross-modal deep fusion methods. Second fusion branch By employing a cross-modal deep fusion method, LiDAR point cloud features and 4D millimeter-wave radar point cloud features are mapped into bird's-eye view (BEV) fusion features. .
[0041] This application's embodiments employ different feature fusion strategies, enabling the model to fully utilize effective multimodal information according to different situations, thereby improving the accuracy and robustness of 3D target detection in complex scenes.
[0042] As one possible approach, the image features and LiDAR point cloud features are mapped to a first bird's-eye view fusion feature using a first fusion branch, and / or the LiDAR point cloud features and 4D millimeter-wave radar point cloud features are mapped to a second bird's-eye view fusion feature using a second fusion branch. This includes: mapping image features to high-level semantic features based on the first fusion branch, thereby converting the high-level semantic features into image BEV features; mapping LiDAR point cloud features to sparse three-dimensional features, and flattening the sparse three-dimensional features into the target BEV space to obtain LiDAR point cloud BEV features; and obtaining the first bird's-eye view fusion feature based on the image BEV features and the LiDAR point cloud BEV features.
[0043] It can be explained that the first fusion branch This may include, but is not limited to: feature pyramid neck network View Transition Module 3D sparse encoder BEV encoders, etc.
[0044] In embodiments of this application, a feature pyramid neck network Image features can be Mapped to high-level semantic features : , View Transition Module It can be based on camera intrinsic and extrinsic parameters. High-level semantic features Convert to image BEV features .in, , These represent the camera's intrinsic and extrinsic parameters, respectively. (View transformation module) The LSS (Lift-Splat-Shoot) mechanism can be used to predict depth and project features into the BEV space. , 3D Sparse Encoder It can display LiDAR point cloud features Mapped to sparse 3D features : , The BEV encoder for LiDAR point clouds can encode sparse 3D features. The data is flattened into the BEV space and then sequentially passed through a 2D backbone network and a 2D neck network to output LiDAR point cloud BEV features. .
[0045] This application embodiment fuses image BEV features and LiDAR point cloud BEV features to achieve the desired image BEV features. BEV features of LiDAR point clouds The data is concatenated along the channel dimension and then fused through a convolutional layer to obtain the output of the first fusion branch. .
[0046] This application embodiment can utilize the first fusion branch to process image features and LiDAR point cloud features under good weather conditions. By fusing the rich semantic information in the image with the precise three-dimensional geometric information in the LiDAR point cloud, the accuracy of target category recognition and three-dimensional spatial positioning can be improved.
[0047] In some cases, the first fusion branch is used to map image features and LiDAR point cloud features into a first bird's-eye view fusion feature, and / or the second fusion branch is used to map LiDAR point cloud features and 4D millimeter-wave radar point cloud features into a second bird's-eye view fusion feature. This includes: based on the second fusion branch, using each non-empty LiDAR voxel as an index, querying multiple nearest-neighbor millimeter-wave radar voxel features that meet preset range conditions; obtaining the fusion feature of the non-empty LiDAR voxel based on the multiple nearest-neighbor millimeter-wave radar voxel features; aggregating all fusion features to obtain the fusion feature, and adding the fusion feature to the LiDAR point cloud features to obtain the enhanced LiDAR point cloud feature; and generating the second bird's-eye view fusion feature based on the LiDAR point cloud feature.
[0048] The second fusion branch of the embodiments of this application This may include, but is not limited to, the following steps: First, the embodiments of this application can convert each non-empty LiDAR voxel The feature is used as a query to find its radius. Within range The nearest neighbor millimeter-wave radar voxel feature is denoted as Activation relationships are calculated using attention mechanisms. ,in As a query As a key: , Here, softmax is the normalized exponential function.
[0049] This application embodiment can activate the relationship Multiplied by the voxel feature of the adjacent millimeter-wave radar After value function The transformed values yield the fusion features of non-empty LiDAR voxels. : , Furthermore, embodiments of this application aggregate the fusion features of all non-empty LiDAR voxels. , obtain fusion features .
[0050] Subsequently, this application embodiment adopts a direct fusion strategy to fuse the features. With LiDAR point cloud features Add them together: , in, To enhance LiDAR point cloud features.
[0051] Finally, in this embodiment, the enhanced LiDAR point cloud features are input into the BEV encoder to obtain features in the BEV space, which are then used as the output of the second fusion branch. .
[0052] This application embodiment can utilize the second fusion branch to process 4D millimeter-wave radar point cloud features and LiDAR point cloud features under adverse weather conditions. By fusing the target motion information of the 4D millimeter-wave radar point cloud with the three-dimensional geometric information of the LiDAR point cloud, the impact of adverse weather conditions such as rain, fog, and snow on image perception performance can be reduced, thereby improving the robustness and accuracy of three-dimensional target detection in complex environments.
[0053] Optionally, in one embodiment of this application, the adaptive routing target detection method for all-weather multi-sensor fusion further includes: extracting features representing the weather semantics of the current scene to determine the weather prior information of the current environment; generating a gating control signal based on the weather prior information to determine the probability of the fusion branch being selected in the previous environment; and activating the first fusion branch or the second fusion branch according to the probability.
[0054] This application embodiment can employ a pre-trained visual language model image encoder to extract features from images, generating features that characterize the weather semantics of the current scene, such as weather or environmental states like sunny, cloudy, rainy, foggy, or snowy, or scene features caused by weather, such as reduced visibility, image blurring, raindrop interference, snow cover, and insufficient lighting, as prior weather information for the current environment. The pre-trained visual language model image encoder can utilize general visual knowledge trained on massive amounts of data to map image I to prior weather information. : , in, Indicates an image encoder. It contains weather semantic information for the current scene.
[0055] Furthermore, embodiments of this application can obtain the gating control signal through logical reasoning calculation based on prior weather information. This represents the probability of each fusion branch being selected under the current weather conditions.
[0056] For example, embodiments of this application can construct a lightweight multilayer perceptron classifier. , weather prior information The input is fed into a multilayer perceptron classifier and activated by the sigmoid function. Perform nonlinear normalization and output gate signal : , Among them, gating signals ∈[0,1]; Gating signal Characterizing the probability of each fusion branch being selected under current weather conditions: when the weather is good, the gating signal... Approaching 0, it tends to activate the first fusion branch; when the weather is bad, the gating signal... Approaching 1, it tends to activate the second fusion branch.
[0057] The embodiments of this application can adaptively select the corresponding feature fusion path according to the weather conditions of the current scene, so that the sensor information under different weather conditions can be used in a targeted manner, thereby improving the accuracy and robustness of 3D target detection under different weather conditions.
[0058] In step S103, the category and / or location of the three-dimensional target are predicted based on the first bird's-eye view fusion feature and / or the second bird's-eye view fusion feature and the pre-built three-dimensional target detection head.
[0059] The pre-built 3D target detection head can adopt detection head structures such as CenterHead, AnchorHead, and PointHead to decode the fused features of the bird's-eye view and generate detection results such as the category, 3D spatial position, length, width, height, and orientation angle of the corresponding 3D target.
[0060] As a specific example, embodiments of this application construct a three-dimensional target detection head. Integrating BEV features As input, CenterHead is used to predict the category and location of 3D targets.
[0061] The adaptive routing target detection method for all-weather multi-sensor fusion proposed in this application utilizes a first fusion branch to map image features and LiDAR point cloud features into a first bird's-eye view fusion feature, and / or utilizes a second fusion branch to map LiDAR point cloud features and 4D millimeter-wave radar point cloud features into a second bird's-eye view fusion feature. This is then combined with a pre-built 3D target detection head to predict the category and / or location of 3D targets. The sensor fusion strategy can be dynamically adjusted according to real-time weather conditions, adaptively selecting or switching appropriate fusion branches for different environmental conditions such as good weather, rain / fog, and nighttime, effectively compensating for the decrease in single-sensor perception capability caused by weather changes. Under good weather conditions, the rich semantic information of images and the precise geometric information of LiDAR can be fully utilized; under adverse weather conditions, the strong environmental adaptability of 4D millimeter-wave radar and the 3D spatial information of LiDAR can be complementaryly fused, thereby improving the effective utilization rate of multimodal perception information and reducing the impact of adverse weather, low illumination, and reduced visibility on target detection performance. This achieves improved accuracy, stability, and robustness of 3D target detection in complex dynamic traffic environments.
[0062] Next, referring to the accompanying drawings, an adaptive routing target detection device for all-weather multi-sensor fusion proposed according to an embodiment of this application is described.
[0063] Figure 2 This is a block diagram of an adaptive routing target detection device for all-weather multi-sensor fusion according to an embodiment of this application.
[0064] like Figure 2 As shown, the adaptive routing target detection device 10 for all-weather multi-sensor fusion includes: an extraction module 100, a mapping module 200, and a detection module 300.
[0065] The extraction module 100 is used to extract image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features from different modal data.
[0066] The mapping module 200 is used to map image features and LiDAR point cloud features into first bird's-eye view fusion features using a first fusion branch, and / or to map LiDAR point cloud features and 4D millimeter-wave radar point cloud features into second bird's-eye view fusion features using a second fusion branch.
[0067] The detection module 300 is used to predict the category and / or location of a 3D target based on the first bird's-eye view fusion features and / or the second bird's-eye view fusion features and a pre-built 3D target detection head.
[0068] Optionally, in one embodiment of this application, the adaptive routing target detection device 10 for all-weather multi-sensor fusion further includes: a determination module, a generation module, and an activation module.
[0069] The determination module is used to extract features that characterize the weather semantics of the current scene in order to determine the prior weather information of the current environment.
[0070] The generation module is used to generate gating control signals based on prior weather information to determine the probability of the fusion branch being selected in the previous environment.
[0071] The activation module is used to activate either the first fusion branch or the second fusion branch based on probability.
[0072] Optionally, in one embodiment of this application, the extraction module 100 includes: an extraction unit, a first conversion unit, and a second conversion unit.
[0073] The extraction unit is used to extract image features using a preset two-dimensional backbone network.
[0074] The first conversion unit is used to voxelize the LiDAR point cloud to convert it into first raster data that meets the first preset rule conditions, and to obtain LiDAR point cloud features based on the first raster data using the first preset voxel feature encoder.
[0075] The second conversion unit is used to voxelize the 4D millimeter-wave radar point cloud to convert it into second grid data that meets the second preset rule conditions, and to obtain the 4D millimeter-wave radar point cloud features based on the second grid data using a preset second voxel feature encoder.
[0076] Optionally, in one embodiment of this application, the mapping module 200 includes: a mapping unit, a flattening unit, and a determining unit.
[0077] The mapping unit is used to map image features to high-level semantic features based on the first fusion branch, so as to convert the high-level semantic features into image BEV features.
[0078] The flattening unit is used to map LiDAR point cloud features into sparse 3D features and flatten the sparse 3D features into the target BEV space to obtain LiDAR point cloud BEV features.
[0079] The determination unit is used to obtain the first bird's-eye view fusion features based on the image BEV features and the LiDAR point cloud BEV features.
[0080] Optionally, in one embodiment of this application, the mapping unit 200 includes: a query unit, a fusion unit, an aggregation unit, and a generation unit.
[0081] The query unit is used to query the features of multiple nearest millimeter-wave radar voxels that meet the preset range conditions, based on the second fusion branch and using each non-empty LiDAR voxel as an index.
[0082] The fusion unit is used to obtain the fusion features of non-empty LiDAR voxels based on the features of multiple nearest-neighbor millimeter-wave radar voxels.
[0083] The aggregation unit is used to aggregate all fusion features to obtain fusion features, and add the fusion features to the LiDAR point cloud features to obtain enhanced LiDAR point cloud features.
[0084] The generation unit is used to generate a second bird's-eye view fusion feature based on LiDAR point cloud features.
[0085] It should be noted that the foregoing explanation of the adaptive routing target detection method embodiment for all-weather multi-sensor fusion also applies to the adaptive routing target detection device for all-weather multi-sensor fusion in this embodiment, and will not be repeated here.
[0086] The adaptive routing target detection device for all-weather multi-sensor fusion proposed in this application utilizes a first fusion branch to map image features and LiDAR point cloud features into a first bird's-eye view fusion feature, and / or utilizes a second fusion branch to map LiDAR point cloud features and 4D millimeter-wave radar point cloud features into a second bird's-eye view fusion feature. This is then combined with a pre-built 3D target detection head to predict the category and / or location of 3D targets. The sensor fusion strategy can be dynamically adjusted according to real-time weather conditions, adaptively selecting or switching corresponding fusion branches for different environmental conditions such as good weather, rain / fog, and nighttime. This effectively compensates for the decrease in the perception capability of a single sensor caused by weather changes, thereby improving the effective utilization rate of multimodal perception information and reducing the impact of adverse weather, low light, and reduced visibility on target detection performance. This achieves an improvement in the accuracy, stability, and robustness of 3D target detection in complex dynamic traffic environments.
[0087] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 301, the processor 302, and the computer program stored on the memory 301 and capable of running on the processor 302.
[0088] When the processor 302 executes the program, it implements the adaptive routing target detection method for all-weather multi-sensor fusion provided in the above embodiments.
[0089] Furthermore, electronic devices also include: Communication interface 303 is used for communication between memory 301 and processor 302.
[0090] The memory 301 is used to store computer programs that can run on the processor 302.
[0091] The memory 301 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0092] If the memory 301, processor 302, and communication interface 303 are implemented independently, then the communication interface 303, memory 301, and processor 302 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0093] Optionally, in a specific implementation, if the memory 301, processor 302, and communication interface 303 are integrated on a single chip, then the memory 301, processor 302, and communication interface 303 can communicate with each other through an internal interface.
[0094] Processor 302 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.
[0095] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described adaptive routing target detection method for all-weather multi-sensor fusion.
[0096] This application also provides a computer program product, including a computer program that can run computer instructions. When the computer instructions are executed by a processor, they implement the adaptive routing target detection method for all-weather multi-sensor fusion provided in this application.
[0097] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0098] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0099] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0100] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.
[0101] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0102] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0104] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. An adaptive routing target detection method for all-weather multi-sensor fusion, characterized in that, Includes the following steps: Extracting image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features from different modal data; The image features and the LiDAR point cloud features are mapped to a first bird's-eye view fusion feature using a first fusion branch, and / or the LiDAR point cloud features and the 4D millimeter-wave radar point cloud features are mapped to a second bird's-eye view fusion feature using a second fusion branch; The category and / or location of a 3D target are predicted based on the first bird's-eye view fusion features and / or the second bird's-eye view fusion features and a pre-built 3D target detection head.
2. The method according to claim 1, characterized in that, Also includes: Extract features that represent the semantics of weather in the current scene to determine prior weather information for the current environment; A gating control signal is generated based on the prior weather information to determine the probability that the fusion branch is selected in the prior environment. The first fusion branch or the second fusion branch is activated according to the probability.
3. The method according to claim 1, characterized in that, The extraction of image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features from different modal data includes: The image features are extracted using a pre-defined two-dimensional backbone network; The LiDAR point cloud is voxelized to convert it into first grid data that meets the first preset rule conditions, and the LiDAR point cloud features are obtained by using the first preset voxel feature encoder based on the first grid data. The 4D millimeter-wave radar point cloud is voxelized to convert it into second grid data that meets the second preset rule conditions, and the 4D millimeter-wave radar point cloud features are obtained by using a preset second voxel feature encoder based on the second grid data.
4. The method according to claim 1, characterized in that, The step of mapping the image features and LiDAR point cloud features into a first bird's-eye view fusion feature using a first fusion branch, and / or mapping the LiDAR point cloud features and 4D millimeter-wave radar point cloud features into a second bird's-eye view fusion feature using a second fusion branch, includes: Based on the first fusion branch, the image features are mapped to high-level semantic features, so as to convert the high-level semantic features into image BEV features; The LiDAR point cloud features are mapped to sparse three-dimensional features, and the sparse three-dimensional features are flattened into the target BEV space to obtain LiDAR point cloud BEV features. The first bird's-eye view fusion feature is obtained based on the image BEV features and the LiDAR point cloud BEV features.
5. The method according to claim 1, characterized in that, The step of mapping the image features and LiDAR point cloud features into a first bird's-eye view fusion feature using a first fusion branch, and / or mapping the LiDAR point cloud features and 4D millimeter-wave radar point cloud features into a second bird's-eye view fusion feature using a second fusion branch, includes: Based on the second fusion branch, using each non-empty LiDAR voxel as an index, query the features of multiple nearest millimeter-wave radar voxels that meet the preset range conditions. The fusion features of non-empty LiDAR voxels are obtained based on multiple nearest-neighbor millimeter-wave radar voxel features. All the fusion features are aggregated to obtain fusion features, and the fusion features are added to the LiDAR point cloud features to obtain enhanced LiDAR point cloud features; The second bird's-eye view fusion feature is generated based on the LiDAR point cloud features.
6. An adaptive routing target detection device for all-weather multi-sensor fusion, characterized in that, include: The extraction module is used to extract image features, LiDAR point cloud features, and 4D millimeter-wave radar point cloud features from different modal data. The mapping module is used to map the image features and the LiDAR point cloud features into a first bird's-eye view fusion feature using a first fusion branch, and / or to map the LiDAR point cloud features and the 4D millimeter-wave radar point cloud features into a second bird's-eye view fusion feature using a second fusion branch. The detection module is used to predict the category and / or location of a 3D target based on the first bird's-eye view fusion features and / or the second bird's-eye view fusion features and a pre-built 3D target detection head.
7. The apparatus according to claim 6, characterized in that, Also includes: The determination module is used to extract features that characterize the weather semantics of the current scene in order to determine the prior weather information of the current environment; The generation module is used to generate a gating control signal based on the weather prior information to determine the probability that the fusion branch is selected in the prior environment; An activation module is used to activate the first fusion branch or the second fusion branch according to the probability.
8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and capable of running on the processor, the processor executing the program to implement the adaptive routing target detection method for all-weather multi-sensor fusion as described in any one of claims 1-5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the adaptive routing target detection method for all-weather multi-sensor fusion as described in any one of claims 1-5.
10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the adaptive routing target detection method for all-weather multi-sensor fusion as described in any one of claims 1-5.