Obstacle detection system and method based on multi-sensor fusion in off-road scenarios

Abstract

The application discloses a target detection technical field and relates to an obstacle detection system and method based on multi-sensor fusion in a cross-country scene, which comprises the following steps: acquiring original data of a cross-country scene collected by a multi-sensor combination, wherein the multi-sensor combination comprises a monocular camera, an infrared sensor, a laser radar, a millimeter wave radar and an inertial measurement unit; and pre-processing the original data to eliminate the heterogeneity of multi-source data. The application uses a multi-sensor strategy of a monocular camera, an infrared sensor, a laser radar, a millimeter wave radar and an inertial measurement unit to acquire information such as an RGB image, infrared thermal imaging, a point cloud and a motion state parameter of an obstacle, and then fuses the multi-sensor data through an improved Transform network model, so that accurate obstacle detection information can be output in real time in a complex cross-country scene.

Description

Technical Field

[0001] This invention relates to an obstacle detection system and method based on multi-sensor fusion in off-road scenarios, belonging to the field of target detection technology. Background Technology

[0002] Obstacle detection is a core technology for autonomous driving to achieve environmental perception, navigation, and obstacle avoidance. Its fundamental goal is to identify obstacles in vehicles and paths in real time and accurately, and determine their position, size, and speed to provide a basis for subsequent decision-making and control. Unlike structured scenarios such as urban roads, the unstructured nature of off-road scenarios poses far more stringent challenges to obstacle detection technology than in general scenarios. Mature perception solutions in general scenarios often rely on prior information such as regular road markings and fixed obstacle shapes. However, the dynamic uncertainties of off-road environments, such as abrupt terrain changes, the appearance of temporary obstacles, and the complexity of environmental interference, make single-sensor perception solutions prone to performance degradation. Therefore, multi-sensor fusion has become the core technical path to overcome the perception bottleneck in off-road scenarios.

[0003] The core characteristics of off-road scenarios are a high proportion of unstructured surfaces, dramatic terrain undulations, complex weather conditions, and diverse and irregular obstacle types. Obstacle detection in these scenarios places extremely high demands on the environmental adaptability, data complementarity, and real-time performance of sensors. Currently, obstacle detection technology in this scenario has evolved from single-sensor perception to multi-sensor fusion. Existing multi-sensor fusion technologies mostly focus on the combined application of 2-3 sensors, and generally suffer from problems such as insufficient sensor compatibility, poor robustness of heterogeneous data fusion, and lack of targeted optimization for off-road scenarios. They do not fully consider the irregular shapes of off-road obstacles and complex background interference, resulting in low recognition rates for occluded targets, and a significantly increased false negative rate under adverse weather conditions such as rain, fog, and sandstorms. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an obstacle detection system and method based on multi-sensor fusion in off-road scenarios, which can output accurate obstacle detection information in real time in complex off-road scenarios.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution: In a first aspect, the present invention provides an obstacle detection method based on multi-sensor fusion in off-road scenarios, comprising: The system acquires raw off-road scene data collected by a combination of multiple sensors, including a monocular camera, an infrared sensor, a lidar, a millimeter-wave radar, and an inertial measurement unit. The raw data is preprocessed to eliminate the heterogeneity of multi-source data. The preprocessing includes time synchronization and spatial coordinate alignment of the data from each sensor, as well as denoising and standardization format conversion of the data. The preprocessed data is input into the improved Transformer fusion model, which processes the visual modal data and the geometric-kinematic modal data respectively through a bimodal branch encoder, and uses a 3D query-driven cross-modal fusion module to perform deep fusion of the bimodal features, and outputs a globally unified 3D feature containing obstacle information. Based on the globally unified 3D features, obstacle category identification, 3D bounding box regression, and motion state prediction are performed through a 3D iterative decoder to output the final obstacle detection information.

[0006] Furthermore, the preprocessing of the raw data includes: Using the high-precision timestamp of the inertial measurement unit as a reference, a strategy combining hardware triggering and software compensation is adopted to achieve time synchronization of data from cameras, lidar, millimeter-wave radar and infrared sensors; The rotation matrix and translation vector between the camera and the lidar are obtained by hand-eye calibration, and the spatial alignment of the millimeter-wave radar and lidar is completed by using a planar calibration plate. At the same time, the sensor coordinate system offset caused by the tilt of the carrier is compensated by the attitude angle output by the inertial measurement unit, so as to realize the spatial alignment of multi-source data in the world coordinate system.

[0007] Furthermore, the data denoising and standardization format conversion includes: Statistical filtering and voxel filtering are performed on the lidar point cloud to remove noise points and downsample the data. Bilateral filtering is applied to the image data from the camera and infrared sensor to remove Gaussian noise, and histogram equalization is applied to the infrared image to enhance contrast. Kalman filtering was used to smooth the ranging and velocity measurement results from millimeter-wave radar data. Complementary filtering is used to fuse accelerometer and gyroscope data from the inertial measurement unit to suppress drift error; The denoised image data is converted into normalized tensors, the point cloud data is converted into bird's-eye view feature maps, and the millimeter-wave radar and IMU data are converted into vector form.

[0008] Furthermore, the improved Transformer fusion model includes: The dual-modal branch encoder receives RGB images and infrared thermal images from a camera, pre-encodes them using ResNet50, extracts global features via a visual Transformer embedded with Deformable Attention, and fuses a cross-scale feature pyramid and a thermal source attention gating module to output a visual global feature map. The geometry-motion modal branch encoder receives bird's-eye view features generated by LiDAR point cloud, target vectors from millimeter-wave radar, and temporal motion vectors from IMU. After feature unification, it is input to another Transformer encoder embedded with DeformableAttention. Combined with temporal attention mechanism and millimeter-wave radar velocity correction, it outputs a geometry-motion global feature map.

[0009] Furthermore, the 3D query-driven cross-modal fusion module includes: Generate a set of learnable 3D query vectors, which are initialized based on the bird's-eye view features of the LiDAR and the average features of the off-road obstacles. The 2D feature map output by the visual modality branch, combined with camera parameters and depth information provided by LiDAR, is projected into 3D space. Through a cross-attention mechanism, the 3D query vector dynamically aggregates visual features from the 3D projection and bird's-eye view features from the geometry-motion modality branch, and embeds Deformable sampling logic in the attention calculation.

[0010] Furthermore, the 3D query-driven cross-modal fusion module also includes: The modality adaptive weight adjustment mechanism dynamically generates fusion weights for visual modal and geometric-motion modal based on an environmental state vector that includes camera image brightness, LiDAR point cloud efficiency, IMU angular velocity variance, image sharpness, millimeter-wave radar signal strength, and infrared heat source contrast. The attention output features of the two modalities are weighted and summed according to the fusion weights to generate the globally unified 3D features.

[0011] Furthermore, the 3D iterative decoder is composed of multiple stacked Transformer decoders, and during the decoding process of each layer: A motion compensation factor calculated based on inertial measurement unit data is introduced, which includes a rotation matrix for correcting feature space coordinates and a carrier linear velocity for correcting obstacle position prediction. Between cross-attention and self-attention operations, the motion compensation factor is used to iteratively correct the features and predicted obstacle positions to eliminate errors caused by carrier motion.

[0012] Secondly, the present invention provides an obstacle detection system based on multi-sensor fusion in off-road scenarios, comprising: The multi-sensor hardware module is used to collect raw data in off-road scenarios. It includes a monocular camera, an infrared sensor, a lidar, a millimeter-wave radar, and an inertial measurement unit. The data preprocessing module is used to perform time synchronization, spatial alignment, noise reduction, and standardized format conversion on the raw data collected by the multi-sensor hardware module. The multi-sensor fusion perception module has an improved Transformer fusion model built in any of the methods described above, which is used to receive preprocessed data, perform feature encoding and cross-modal fusion, and output globally unified 3D features. The 3D detection and output module is used to perform obstacle category identification, 3D spatial positioning and motion state prediction based on the global unified 3D features, and output the final obstacle detection information.

[0013] Thirdly, the present invention provides an obstacle detection device based on multi-sensor fusion in off-road scenarios, including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any of the foregoing.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0015] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: I. In view of the complexity of off-road scenarios and the limitations of existing detection technologies, this invention provides an obstacle detection method based on multi-sensor fusion in off-road scenarios. It uses a multi-sensor strategy of monocular camera, infrared sensor, lidar, millimeter-wave radar and inertial measurement unit to acquire information such as RGB image, infrared thermal image, point cloud and motion state parameters of obstacles. Then, the multi-sensor data is fused by an improved Transformer network model, and finally, accurate obstacle detection information can be output in real time in complex off-road scenarios. II. This invention improves the DualViT fusion architecture by embedding Deformable Attention and DETR3D 3D query mechanisms, and designs a dual-modal dedicated encoder to achieve precise alignment of visual and geometric-motion feature 3D space, solving the problems of low efficiency in multi-sensor fusion and loss of accuracy when upgrading from 2D to 3D. It integrates cameras, LiDAR, millimeter-wave radar, infrared sensors, and inertial measurement units, along with a modal adaptive weighting mechanism. Its effects include: strengthening sensor complementarity in extreme environments, improving detection accuracy in rainy and nighttime conditions, and significantly enhancing robustness; constructing dual dynamic modeling, using a temporal attention mechanism to capture IMU temporal dependencies, and millimeter-wave radar velocity correction to provide obstacle motion priors, adapting to vehicle turbulence and obstacle movement, and improving the accuracy and stability of dynamic target detection. Attached Figure Description

[0016] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the obstacle detection method based on multi-sensor fusion in an off-road scenario provided in Embodiment 1 of the present invention. Figure 2 This is a flowchart of the dual-modal branch coding fusion method for obstacle detection in off-road scenarios provided in Embodiment 1 of the present invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0018] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this invention is for describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0019] Example 1: like Figure 1 As shown, this embodiment provides an obstacle detection method based on multi-sensor fusion in off-road scenarios, including the following steps: The construction of a multi-sensor hardware system includes cameras, lidar, millimeter-wave radar, infrared sensors, and inertial measurement units. The inertial measurement unit can be a combination of multiple sensors, such as a 6-axis inertial measurement unit (6D IMU). Data preprocessing converts data from multiple sensors into a standardized format, eliminating the heterogeneity of multi-source data and providing high-quality input for subsequent fusion. Design an improved Transformer to achieve efficient deep fusion of multi-sensor data by embedding Deformable Attention and DETR3D 3D logic; 3D detection and post-processing accurately extracts the category, 3D spatial information and motion status of obstacles from globally unified 3D features; In this embodiment, the specific implementation steps for building the multi-sensor hardware system are as follows: Based on the core principles of environmental adaptability, performance complementarity, real-time performance, and integrability, the selection criteria for each sensor are determined to meet the detection requirements of complex terrain, extreme environments, and heterogeneous obstacles in off-road scenarios. After selection, the selected sensor is then put into actual verification to ensure that key indicators meet the standards. Based on the off-road vehicle's body structure, the installation positions of each sensor were determined to ensure no blind spots and no mutual interference; the camera was installed above the center of the windshield; the lidar was installed in the center of the roof; the millimeter-wave radar was installed on both sides of the front bumper and in the center of the rear bumper; the infrared sensor was installed below the center of the windshield; and the 6D IMU attitude sensor was placed at the vehicle's center of gravity. In this embodiment, the specific implementation steps of data preprocessing are as follows: A dual synchronization strategy of hardware triggering and software compensation is adopted. Based on the high-precision timestamp of the 6D IMU, the FPGA triggers the synchronous acquisition of the camera and LiDAR, and performs linear interpolation compensation on the timestamps of the millimeter-wave radar and infrared sensor to ensure the time consistency of multi-source data. The rotation matrix and translation vector of the camera and lidar are obtained by hand-eye calibration. The spatial alignment of the millimeter-wave radar and lidar is completed by using a planar calibration board. The attitude angle output by the 6D IMU is used to correct the sensor coordinate system offset caused by the tilt of the carrier, ensuring that the multi-source data are aligned in the same world coordinate system. The data acquired by each sensor is then denoised. For lidar point clouds, statistical filtering removes isolated noise points, a threshold for the number of neighboring points is set, and points below the threshold are removed. Voxel filtering is used to downsample and balance real-time performance and resolution. For image data, bilateral filtering removes Gaussian noise from camera and infrared sensor images, and histogram equalization enhances the contrast of infrared images. Data acquired by millimeter-wave radar is smoothed using Kalman filtering to achieve ranging and velocity measurement results, and abnormal jump data is removed. Complementary filtering of 6DIMU data is used to fuse accelerometer and gyroscope data to suppress drift errors. Then, the data were formatted and converted. Image data was converted into a 3×1080×1920 tensor and normalized to [0,1]. Point cloud data was converted into BEV feature maps with a resolution of 512×512. The millimeter-wave radar output target candidate boxes (x,y,z,r,v), where x,y,z are the target positions, r is the target radius, and v is the target velocity. IMU data was converted into a 3×3 attitude matrix and motion vector (vx,vy,vz,ax,ay,az), where v is the carrier velocity and a is the carrier acceleration.

[0020] This embodiment designs an improved Transformer to achieve efficient deep fusion of multi-sensor data. Its main innovative features include: bimodal branch coding, Deformable Attention efficient modeling, DETR3D 3D logic embedding, and cross-modal dynamic fusion. It solves key problems such as low efficiency of multi-sensor heterogeneous data fusion, insufficient 3D modeling accuracy, and poor robustness in extreme environments, and achieves efficient deep fusion of multi-source data in off-road scenarios. Please see Figure 2 In this embodiment, the dual-modal branch encoding divides the five sensors into two categories—visual modality and geometric-motion modality—based on the characteristics of the sensor data. The visual modality branch includes a camera and an infrared sensor, while the geometric-motion modality branch includes a lidar, a millimeter-wave radar, and a 6D IMU. Deep features are extracted using dedicated Deformable ViT encoders to avoid feature interference between different data types and improve encoding efficiency.

[0021] In the visual modality branch encoder module, a three-level structure of ResNet50 precoding + Deformable ViT encoding + multimodal visual fusion was designed for image data from cameras and infrared sensors: In the pre-encoding stage, feature dimensionality reduction and local extraction are performed. The RGB image (3×1920×1080) and infrared image (1×640×512) from the camera are input into ResNet50, and after four stages of convolution and pooling, local feature maps with dimensions of 2048×60×33 and 2048×20×16 are output respectively. After this operation, the high-resolution image is downsampled by 32 times, which significantly reduces the number of tokens in the subsequent Transformer. At the same time, the low-level features such as edges and textures are extracted through the local receptive field of the convolution kernel, which makes up for the lack of local capture capability of Transformer.

[0022] In the Deformable ViT encoding stage, efficient global feature extraction is performed. The local feature maps of RGB and infrared are unified to a size of 2048×60×33 through bilinear interpolation and stitched together in the channel dimension to form a fused feature map of 4096×60×33. Deformable Multi-Head Attention is embedded to replace the full attention of traditional ViT. Each encoder layer is equipped with 8 attention heads, and each attention head samples only 16 key feature points around the reference point. The computational complexity is reduced from O(N²) to O(N×K), and the computational cost of a single encoder layer is reduced by 70%. The encoder configuration is a 6-layer encoder with a feedforward network hidden layer dimension of 8192. The activation function is GELU. Each interpolation layer is normalized and connected to the residual to avoid gradient vanishing. Finally, a 2048×60×33 visual global feature map is output, containing fused information of RGB texture and infrared heat source.

[0023] By embedding a cross-scale feature pyramid (FPN), the features output by the Deformable ViT feature encoder are concatenated with the local features of each stage of ResNet50 to improve the feature response of small obstacles. Then, a thermal source attention gating module is introduced to dynamically weight and enhance the features fused by Deformable ViT and FPN and the infrared single-mode features, and finally generate enhanced features.

[0024] In this embodiment, the geometry-motion modal branch encoder is designed with a structure of multi-source feature unification + Deformable ViT encoding + temporal motion enhancement for 3D geometry and motion data from LiDAR, millimeter-wave radar, and 6D IMU. The LiDAR BEV feature map is obtained by voxelization and 3D convolution transformation of the preprocessed point cloud, containing 3D geometric contour information of obstacles; the target candidate box of the millimeter-wave radar is mapped to 2048 dimensions through 3 layers of MLP; the attitude matrix and motion vector of the IMU are concatenated in time sequence and encoded into 2048-dimensional temporal features through 2 layers of LSTM, containing information on vehicle attitude changes and motion trends; the millimeter-wave radar feature vector and IMU temporal features are expanded to 2048×512×512 through a broadcast mechanism, and concatenated with the LiDAR BEV feature map to form a unified geometry-motion feature map of 6144×512×512.

[0025] For 3D BEV feature maps, the reference points of Deformable Attention are set based on the world coordinate system. The coordinates of the reference points are initialized by the prior target position of the LiDAR, ensuring that the attention sampling is focused on the 3D spatial region of the obstacle. It includes a 6-layer encoder, 8 attention heads, each sampling 20 key feature points, an FFN hidden layer dimension of 8192, the activation function GELU, and layer normalization and residual connections. It can output a geometry-motion global feature map containing 3D geometry, distance, velocity and attitude information.

[0026] Embed a temporal attention mechanism to capture the temporal dependencies of IMU data;

[0027] In the formula: Temporal enhancement features for the geometry-motion mode; Static BEV characteristics for geometric-motion modes; For the temporal feature sequence of the IMU, the weights of geometric features are dynamically adjusted through temporal attention to adapt to dynamic scenarios such as vehicle turbulence and obstacle movement; the first The first part is responsible for telling the model which part of the geometric features needs IMU motion information; the second part... It is responsible for providing IMU motion information to the corresponding parts of the geometric features.

[0028] By introducing millimeter-wave radar velocity correction, the target velocity of the millimeter-wave radar is used as prior information to correct the motion state estimation in the BEV feature map, thereby improving the feature purity of dynamic obstacles.

[0029] In this embodiment, a 3D query-driven cross-modal fusion module is constructed, which is the core fusion unit for improving DualViT. The 3D query mechanism of DETR3D achieves 3D spatial alignment of bi-branch features, and combined with modal adaptive weights to dynamically aggregate complementary information, thus solving the problems of spatial misalignment and fixed modal weights in traditional fusion methods.

[0030] Referring to the design of DETR 3D, this invention generates 300 learnable 3D query vectors, corresponding to the maximum number of obstacles in the off-road scene. Of the 300 query vectors, 250 are used to cover areas with high probability of obstacles, and 50 are used as redundant queries specifically to match small obstacles and edge area obstacles. Each query vector has a dimension of 2048 and includes 3D position prior and obstacle type prior information. The 3D position prior is initialized based on the target candidate region of the LiDAR BEV feature map to ensure that the query vector focuses on the potential obstacle space. Then, it is initialized by the average feature vector of various off-road obstacles to improve the aggregation efficiency of category-related features.

[0031] The 2D feature map output by the vision branch is converted into a 3D spatial feature point cloud through camera intrinsic and extrinsic parameters; then the BEV feature map of the geometry branch is directly matched with the spatial position of the 3D query vector.

[0032] in,( ) represents the 3D feature point coordinates in the world coordinate system; R is the camera extrinsic rotation matrix; f x f y These are the camera's intrinsic focal lengths; c x c y These are the coordinates of the principal point of the camera's intrinsic parameters; t x t y These are the translation vector components of the camera's extrinsic parameters; , ) represents the pixel coordinates of the 2D feature map. Depth information provided for lidar; The BEV feature map of the geometric branch has been constructed based on the world coordinate system, and can be directly matched with the spatial position of the 3D query vector without additional transformation.

[0033] A cross-attention mechanism based on 3D query and dual-branch features is employed to dynamically aggregate appearance features of the visual modality and spatial-motion features of the geometric-motion modality, as shown in the following formula:

[0034] Among them, F fused This is a dual-modal cross-attention fusion feature; It is a 3D query vector of 300×2048; , It is the key and value features after the visual modality 3D projection, with a dimension of M×2048, where M is the number of visual 3D feature points; , The key and value features of the geometric-motion modality are P×2048, where P is the number of effective feature points in the BEV feature map. Deformable sampling logic is embedded in the attention calculation. The 3D query vector only calculates the attention weights for K spatially adjacent and feature-similar points (K=24) in the bi-branch features, avoiding the inefficiency caused by global calculation.

[0035] An environment-state-driven modal weight adjustment is introduced to dynamically balance the contributions of the visual modality and the geometric-motor modality, as shown in the following formula:

[0036]

[0037] Among them, F finalModality-adaptive weighted fusion features; , These are the attention output features for the visual modality and the geometric-motor modality, respectively. , For modal adaptive weights, derived from the environment state vector Env Dynamically generated using a single-layer MLP; environment state vector Env The data is divided into six dimensions, including average camera image brightness, LiDAR point cloud efficiency, IMU angular velocity variance, camera image sharpness, millimeter-wave radar signal strength, and infrared image thermal source contrast. The weight generation logic uses the sigmoid function to... Env Mapping to the [0.1, 0.9] interval ensures that the weights of a single modality are not too low, which could lead to information loss. After cross-modal attention fusion and weight adjustment, a global unified 3D feature vector of 300×2048 is output, and each vector corresponds to a global unified 3D feature fusion feature of a potential obstacle. In this embodiment, the 3D iterative decoder is responsible for gradually refining the obstacle category, 3D bounding box, and motion state information from the globally unified 3D features. It improves the prediction accuracy through multi-layer iterative optimization and deeply integrates 6D IMU data to achieve motion compensation. A 3-layer stacked Transformer decoder is adopted, with each layer consisting of cross-attention, self-attention, FFN, residual connection, and layer normalization. Using globally unified 3D features as key-value pairs and the query vector of the current layer as the query, multimodal information is further aggregated; self-attention between query vectors captures the spatial relationships between obstacles, such as the relative positions of multiple rocks, to avoid duplicate detection; the hidden layer has a dimension of 8192 and uses the GELU activation function to enhance the non-linear expressive power of features; each layer of the decoder outputs more refined query features, and the output of the previous layer is used as the input of the next layer to gradually reduce the prediction error; An IMU motion compensation factor is introduced and deeply integrated into each layer of the decoder's iteration process to achieve feature-level motion compensation; Construct a rotation matrix using the carrier attitude angles output by the IMU. The spatial coordinates of the globally unified 3D features are corrected using the following formula:

[0038] in, The corrected geometric-motion modal characteristics; It is the original geometric-motion modal feature before correction, which is a four-dimensional tensor; It is a 3×3 rotation matrix constructed based on the carrier attitude angle output by the 6D IMU, so that the feature coordinates shift with the carrier when the carrier tilts; yes The transpose matrix is ​​used to restore the features in the tilted or bumpy carrier coordinate system to the stable world coordinate system, thus eliminating attitude offset.

[0039] IMU-based carrier linear velocity ( , , ) and frame interval The formula for correcting the 3D position prediction of obstacles is as follows:

[0040] in, , These are the x-coordinates of the obstacles output by the decoders at layers i and (i+1), respectively, ensuring accurate position prediction in dynamic scenes; y pred,i y pred,i+1 The y-coordinates of the obstacles output by the decoders at layers i and (i+1) are respectively; pred,i z pred,i+1 These are the z-coordinates of the obstacles output by the decoders at layers i and (i+1), respectively. , , These represent the carrier linear velocities of the IMU along the x, y, and z axes, respectively. The frame interval.

[0041] Motion compensation is performed after cross-attention and before self-attention in each decoder layer to achieve a closed loop of iterative optimization and dynamic correction. Using the company's internal dataset and data acquired by various sensors as input data, the improved algorithm is trained, and finally a trained multi-sensor fusion perception model is obtained, which is then transformed into a deployable ONNX model. Then, the trained multi-sensor fusion model is designed to be lightweight, including knowledge distillation, channel pruning, and feature dimension optimization. The model is accelerated using TensorRT with INT8 quantization, and then the CUDA core is used to accelerate the high parallelism operations such as sampling and attention calculation of DeformableAttention. At the same time, the dynamic batch processing mechanism of TensorRT is adopted to adapt to the changes in the number of targets in different frames; finally, the optimized multi-sensor fusion model is obtained. The improved DualViT of this invention is compared with BEV Transformer v2, the original DualViT, CNN+Kalman filter and DETR3D model; The experimental environment was an NVIDIA Jetson AGX Orin platform, including an Orin NX 1000 GPU and an 8-core ARM Cortex-A78 CPU; The dataset uses a self-made multi-sensor dataset for off-road scenarios, mainly including mountainous, desert, snowy, rainy, and nighttime scenes, with over 100,000 labeled samples; including rocks, trees, man-made facilities, vegetation, pedestrians, vehicles, and positive and negative obstacles; The comparison results are shown in Table 1.

[0042]

[0043]

[0044] Table 1: Comparison of Results As shown in the table above, existing models support a maximum of four types of sensor data fusion, while the invention in this paper supports the fusion of five heterogeneous sensors, covering a more comprehensive range of off-road scene perception needs. The 3D positioning error of this invention is 0.25m, a 37.5% reduction compared to DETR3D and a 28.6% reduction compared to BEVFormer v2. This is attributed to 3D query-driven cross-modal space alignment and IMU motion compensation. The accuracy of this invention in rainy / nighttime scenes reaches 90%, an 8.4% improvement compared to BEVFormer v2, validating the effectiveness of environmental state vectors. Env The modal adaptive weighting mechanism driven by the system can effectively adapt to complex environments; the inference latency is 46ms, which meets the real-time requirement of ≤50ms in off-road scenarios. At the same time, the recall rate of small obstacles reaches 85%, which is 6%-23% higher than existing models, demonstrating the advantages of FPN multi-scale fusion and Deformable Attention accurate sampling. When a single sensor fails, the performance degradation rate of this invention is only 12%, which is much lower than existing models, indicating that dual-modal dedicated encoding and complementary fusion design can improve system reliability.

[0045] Example 2: The obstacle detection system based on multi-sensor fusion in off-road scenarios can realize the obstacle detection method based on multi-sensor fusion in off-road scenarios described in Example 1, including: The multi-sensor hardware module is used to collect raw data in off-road scenarios. It includes a monocular camera, an infrared sensor, a lidar, a millimeter-wave radar, and a 6D IMU. The data preprocessing module is used to perform time synchronization, spatial alignment, noise reduction, and standardized format conversion on the raw data collected by the multi-sensor hardware module. The multi-sensor fusion sensing module has an improved Transformer fusion model built in as described in Embodiment 1. It is used to receive preprocessed data, perform feature encoding and cross-modal fusion, and output globally unified 3D features. The 3D detection and output module is used to perform obstacle category identification, 3D spatial positioning and motion state prediction based on the global unified 3D features, and output the final obstacle detection information.

[0046] Example 3: This invention also provides an obstacle detection device based on multi-sensor fusion in off-road scenarios, which can realize the obstacle detection method based on multi-sensor fusion in off-road scenarios described in Embodiment 1, including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the following method: The system acquires raw off-road scene data collected by a combination of multiple sensors, including a monocular camera, an infrared sensor, a lidar, a millimeter-wave radar, and an inertial measurement unit. The raw data is preprocessed to eliminate the heterogeneity of multi-source data. The preprocessing includes time synchronization and spatial coordinate alignment of the data from each sensor, as well as denoising and standardization format conversion of the data. The preprocessed data is input into the improved Transformer fusion model, which processes the visual modal data and the geometric-kinematic modal data respectively through a bimodal branch encoder, and uses a 3D query-driven cross-modal fusion module to perform deep fusion of the bimodal features, and outputs a globally unified 3D feature containing obstacle information. Based on the globally unified 3D features, obstacle category identification, 3D bounding box regression, and motion state prediction are performed through a 3D iterative decoder to output the final obstacle detection information.

[0047] Example 4: This invention also provides a computer-readable storage medium that implements the obstacle detection method based on multi-sensor fusion in off-road scenarios described in Embodiment 1. The medium stores a computer program that, when executed by a processor, performs the steps of the following method: The system acquires raw off-road scene data collected by a combination of multiple sensors, including a monocular camera, an infrared sensor, a lidar, a millimeter-wave radar, and an inertial measurement unit. The raw data is preprocessed to eliminate the heterogeneity of multi-source data. The preprocessing includes time synchronization and spatial coordinate alignment of the data from each sensor, as well as denoising and standardization format conversion of the data. The preprocessed data is input into the improved Transformer fusion model, which processes the visual modal data and the geometric-kinematic modal data respectively through a bimodal branch encoder, and uses a 3D query-driven cross-modal fusion module to perform deep fusion of the bimodal features, and outputs a globally unified 3D feature containing obstacle information. Based on the globally unified 3D features, obstacle category identification, 3D bounding box regression, and motion state prediction are performed through a 3D iterative decoder to output the final obstacle detection information.

[0048] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative and not exhaustive. All modifications within the scope of this invention or its equivalents are included in this invention.

[0049] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0051] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0052] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An obstacle detection method based on multi-sensor fusion in off-road scenarios, characterized by: include: The system acquires raw off-road scene data collected by a combination of multiple sensors, including a monocular camera, an infrared sensor, a lidar, a millimeter-wave radar, and an inertial measurement unit. The raw data of the off-road scenario is preprocessed to eliminate the heterogeneity of multi-source data. The preprocessing includes time synchronization and spatial coordinate alignment of the data from each sensor, as well as noise reduction and standardization format conversion of the data. The preprocessed data is input into the improved Transformer fusion model, which processes the visual modal data and the geometric-kinematic modal data respectively through a bimodal branch encoder, and uses a 3D query-driven cross-modal fusion module to perform deep fusion of the bimodal features, and outputs a globally unified 3D feature containing obstacle information. Based on the globally unified 3D features, obstacle category identification, 3D bounding box regression, and motion state prediction are performed through a 3D iterative decoder to output the final obstacle detection information.

2. The obstacle detection method based on multi-sensor fusion in off-road scenarios according to claim 1, characterized in that, The preprocessing of the raw data includes: Using the high-precision timestamp of the inertial measurement unit as a reference, a strategy combining hardware triggering and software compensation is adopted to achieve time synchronization of data from cameras, lidar, millimeter-wave radar and infrared sensors; The rotation matrix and translation vector between the camera and the lidar are obtained by hand-eye calibration, and the spatial alignment of the millimeter-wave radar and lidar is completed by using a planar calibration plate. At the same time, the sensor coordinate system offset caused by the tilt of the carrier is compensated by the attitude angle output by the inertial measurement unit, so as to realize the spatial alignment of multi-source data in the world coordinate system.

3. The obstacle detection method based on multi-sensor fusion in off-road scenarios according to claim 2, characterized in that, The data denoising and standardization conversion includes: Statistical filtering and voxel filtering are performed on the lidar point cloud acquired by lidar to remove noise points and downsample the data; Bilateral filtering is performed on the image data acquired by the camera and infrared sensor to remove Gaussian noise, and histogram equalization is performed on the infrared image acquired by the infrared sensor to enhance contrast. The millimeter-wave radar data acquired by the millimeter-wave radar was smoothed using Kalman filtering to obtain ranging and velocity measurement results; Complementary filtering is used to fuse accelerometer and gyroscope data acquired by the inertial measurement unit to suppress drift error; The denoised image data is converted into normalized tensors, the point cloud data is converted into bird's-eye view feature maps, and the millimeter-wave radar and IMU data are converted into vector form.

4. The obstacle detection method based on multi-sensor fusion in off-road scenarios according to claim 3, characterized in that, The improved Transformer fusion model includes: The dual-modal branch encoder receives RGB images and infrared thermal images from a camera, pre-encodes them using ResNet50, extracts global features via a visual Transformer embedded with Deformable Attention, and fuses a cross-scale feature pyramid and a thermal source attention gating module to output a visual global feature map. The geometry-motion modal branch encoder receives bird's-eye view features generated by LiDAR point cloud, target vectors from millimeter-wave radar, and temporal motion vectors from IMU. After feature unification, it is input to another Transformer encoder embedded with Deformable Attention. Combined with temporal attention mechanism and millimeter-wave radar velocity correction, it outputs a geometry-motion global feature map.

5. The obstacle detection method based on multi-sensor fusion in off-road scenarios according to claim 4, characterized in that, The 3D query-driven cross-modal fusion module includes: Generate a set of learnable 3D query vectors, which are initialized based on the bird's-eye view features of the LiDAR and the average features of the off-road obstacles. The 2D feature map output by the visual modality branch, combined with camera parameters and depth information provided by LiDAR, is projected into 3D space. Through a cross-attention mechanism, the 3D query vector dynamically aggregates visual features from the 3D projection and bird's-eye view features from the geometry-motion modality branch, and embeds Deformable sampling logic in the attention calculation.

6. The obstacle detection method based on multi-sensor fusion in off-road scenarios according to claim 5, characterized in that, The 3D query-driven cross-modal fusion module also includes: The modality adaptive weight adjustment mechanism dynamically generates fusion weights for visual modal and geometric-motion modal based on an environmental state vector that includes camera image brightness, LiDAR point cloud efficiency, IMU angular velocity variance, image sharpness, millimeter-wave radar signal strength, and infrared heat source contrast. The attention output features of the two modalities are weighted and summed according to the fusion weights to generate the globally unified 3D features.

7. The obstacle detection method based on multi-sensor fusion in off-road scenarios according to claim 1, characterized in that, The 3D iterative decoder is composed of multiple stacked Transformer decoders, and during the decoding process of each layer: A motion compensation factor calculated based on inertial measurement unit data is introduced, which includes a rotation matrix for correcting feature space coordinates and a carrier linear velocity for correcting obstacle position prediction. Between cross-attention and self-attention operations, the motion compensation factor is used to iteratively correct the features and predicted obstacle positions to eliminate errors caused by carrier motion.

8. An obstacle detection system based on multi-sensor fusion in off-road scenarios, characterized by: include: The multi-sensor hardware module is used to collect raw data in off-road scenarios. It includes a monocular camera, an infrared sensor, a lidar, a millimeter-wave radar, and an inertial measurement unit. The data preprocessing module is used to perform time synchronization, spatial alignment, noise reduction, and standardized format conversion on the raw data collected by the multi-sensor hardware module. The multi-sensor fusion sensing module has a built-in improved Transformer fusion model as described in any one of claims 1-7, which is used to receive preprocessed data, perform feature encoding and cross-modal fusion, and output globally unified 3D features. The 3D detection and output module is used to perform obstacle category identification, 3D spatial positioning and motion state prediction based on the global unified 3D features, and output the final obstacle detection information.

9. An obstacle detection device based on multi-sensor fusion in off-road scenarios, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 7.

Images