Hybrid elevation map generation method based on multi-modal data fusion and reinforcement learning

Abstract

According to the hybrid elevation map generation method based on multi-modal data fusion and reinforcement learning, through vision-LiDAR deep fusion, direction perception feature coding, space-time consistency optimization and reinforcement learning dynamic parameter adjustment, elevation map generation with 15-meter long distance and 0.05-meter / pixel high resolution in a medium-speed unmanned driving scene is realized. According to the method, the elevation error is reduced to be lower than 15 cm, the path planning success rate is increased to 90%, the system cost is reduced by 70%, and robust navigation support is provided for a complex unstructured environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, sensor fusion and three-dimensional environment modeling, and particularly relates to a hybrid elevation map generation method based on multi-modal data fusion and reinforcement learning, which is used for generating high-precision semantic elevation maps to provide core data support for the navigation, obstacle avoidance and decision-making of agents in unstructured environments. BACKGROUND

[0002] Current three-dimensional environment modeling techniques mainly rely on multi-modal sensors such as LiDAR and depth cameras. Existing elevation map generation methods can be divided into the following three categories: (1) LiDAR dominant scheme: construct elevation map through voxel filtering and plane fitting, but limited by point cloud sparsity, it is difficult to capture complex terrain details (such as vegetation occluded areas); (2) Depth camera dominant scheme: use binocular / structured light to obtain depth images and combine with SLAM algorithm to generate semantic map, but its effective ranging is short (<10m), and it is sensitive to noise; (3) Traditional fusion scheme: adopt loose coupling strategy (such as ICP algorithm to register multiple frames of point cloud or SIFT feature matching), the representative technology such as "Multi-Sensor Fusion for Semantic 3D Mapping" improves the map accuracy through two-stage fusion (geometric layer and semantic layer), but still relies on high-precision odometer.

[0003] However, the existing elevation map generation methods have the following defects in the medium-speed unmanned driving scene (2m / s-5m / s): (1) Insufficient multi-modal data fusion accuracy: traditional loose coupling fusion (such as ICP) does not solve the contradiction between laser radar sparsity and depth image noise, resulting in loss of terrain details (such as vegetation occluded area error exceeding 20%), and lacks dynamic weight distribution and cross-modal feature enhancement mechanism, relies on high-precision odometer, and long-time walking affects the quality of elevation map generation; (2) Poor environmental adaptability: fixed fusion parameters (such as filtering strength) cannot cope with complex terrain (such as off-road scenes), and the path planning success rate is less than 85%; (3) Strong dependence on odometer: existing schemes require high-precision odometer to align multi-modal data, and long-time running leads to error accumulation (error growth rate exceeds 0.1% / m), which is difficult to meet the real-time and long-distance prediction needs of medium-speed scenes; (4) Lack of semantic information: traditional methods focus on geometric modeling and do not effectively fuse semantic information (such as passable area identification), resulting in insufficient robustness of navigation decision-making.

[0004] Therefore, the applicant proposed a hybrid elevation map generation method based on multimodal data fusion and reinforcement learning, which breaks through the traditional elevation map generation method's dependence on high-precision odometers in medium-speed unmanned driving scenarios (2m / s-5m / s) and effectively expands the effective detection distance of environmental perception. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a hybrid elevation map generation method based on multimodal data fusion and reinforcement learning to eliminate the chattering phenomenon in the permanent magnet synchronous motor (PMSM) system and achieve fast dynamic response and good anti-disturbance performance under the condition of limited input in engineering practice.

[0006] The object of the present invention is achieved like this: A hybrid elevation map generation method based on multimodal data fusion and reinforcement learning includes the following contents: S1. Multimodal sensor collaborative perception: S1.1, Vision-LiDAR Deep Fusion: Synchronously collect multi-camera RGB images, LiDAR point clouds, and IMU data, and then use the sparse depth information of the LiDAR point clouds to construct an initial elevation map; S1.2, Super-resolution Reconstruction: Based on the initial elevation map constructed in the above steps, a generative adversarial network is used to reconstruct the depth image through super-resolution, solving the problem of insufficient ranging of visual sensors at distances greater than 10m; S1.3, Spatiotemporal Alignment: The visual features in the camera coordinate system are converted to a global coordinate system consistent with the direction of gravity through the gravity alignment matrix G = R·K^−1, where R is the vehicle posture rotation matrix and K is the camera intrinsic parameter, to eliminate the interference of terrain undulation on posture estimation; S2. Direction-aware feature encoding S2.1, Direction vector deprojection: Convert the image pixel coordinates (ui, vi) into a three-dimensional direction vector di = P[ui, vi, 1]T^-1, where P = G -1 ×R −1 ×K −1 Fusion of camera parameters and vehicle posture information; S3, spatiotemporal consistency optimization S3.1, recursive prediction mechanism: the elevation map prediction result of the previous moment is transformed into the vehicle posture transformation matrix T t−1 →T t Align to the current coordinate system and generate historical enhanced query Q t =[Qlearned,CNN(M t ^−1⊙Ot ) ], wherein O t is a temporal overlap mask; S4, reinforcement learning optimization closed loop S4.1, policy network design: adopt proximal policy optimization algorithm, take the passability evaluation of path planning as the reward signal, and dynamically adjust the parameters of the filtering strength σ and the semantic weight λ; S5, laser radar secondary training S5.1, consistency regularization: by constraining the consistency of visual prediction and laser radar measurement through adversarial training, the reconstruction accuracy of terrain details at a distance greater than 15m is improved.

[0007] Further, in step S1.1, a multi-view camera array is first established: three front, left and right cameras are set to synchronously collect RGB images, and multi-scale features are extracted through a shared EfficientNet-B0 backbone network; then the LiDAR point cloud is denoised and voxelized to generate an initial elevation map.

[0008] Further, in step S1.2, a generative adversarial network is used to perform super-resolution reconstruction on the depth image, and the reconstructed resolution is 0.05m / pixel.

[0009] Further, in step S2, S2.2, position encoding enhancement is also included: a multi-layer perceptron is used to encode the direction vector generated by the IMU and the wheel encoder into a learnable position embedding, and a direction perception query matrix is constructed by combining visual features, thereby improving the feature positioning accuracy on uneven terrain.

[0010] Further, in step S3, S3.2, temporal consistency constraint is also included: a smooth L1 loss and a gradient matching loss are introduced to force the continuity of adjacent frame predictions and reduce high-frequency noise in dynamic environments.

[0011] Further, in step S4, S4.2, uncertainty quantification is also included: a Bayesian deep learning framework is used to estimate the confidence of the elevation value, and a laser radar resampling mechanism is triggered in the low-confidence area to realize data-driven dynamic optimization.

[0012] Further, in step S5, before the consistency regularization step, there is also a step of generating pseudo-labels: the elevation map generated by vision is used as the initial pseudo-label, and a weakly supervised training set is constructed by combining the LiDAR point cloud.

[0013] Compared with the prior art, the beneficial effects of the present application are: The application provides a hybrid elevation map generation method based on multi-modal data fusion and reinforcement learning. By constructing a multi-modal deep fusion framework driven by reinforcement learning, robust map generation without high-precision odometer is realized. The effective detection distance is improved to more than 10m by radar-camera collaborative enhancement technology, while maintaining a high resolution of 0.05m / pixel, providing a reliable environment foundation for medium-speed unmanned driving. Thus, the elevation map generation method of the application has the following advantages: (1) Precision improvement: in KITTI dataset verification, the true effective value (RMSE) of the elevation map is reduced to 15cm (30% lower than the prior art), and the semantic classification accuracy is 90%; (2) Environmental adaptability: dynamic parameter optimization improves the path planning success rate to 90% (compared with 85% of the traditional scheme), supporting off-road terrain fluctuation scenarios; (3) Cost reduction: fusion of vision and LiDAR collaborative enhancement greatly reduces the system cost compared with the pure LiDAR scheme; (4) Real-time and robustness: the effective detection distance is expanded to more than 15 meters, supporting real-time navigation at a speed of 2-5m / s, and the error accumulation rate is less than 0.05% / m. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The flow framework diagram of the hybrid elevation map generation method based on multi-modal data fusion and reinforcement learning of the application. DETAILED DESCRIPTION

[0015] In order to better understand the technical scheme of the application, the following will be described in detail in combination with related drawings. It should be understood that the following specific examples are not intended to limit the specific implementation of the technical scheme of the application, but only to adopt the implementation of the technical scheme of the application. It should be noted that the description of the position relationship of each component in this paper, such as A component above B component, is based on the relative position of each component in the drawing, and is not intended to limit the actual position relationship of each component.

[0016] Referring to Figure 1 , Figure 1 The flowchart of the hybrid elevation map generation method based on multi-modal data fusion and reinforcement learning of this embodiment 1 is drawn. As shown in the figure, the hybrid elevation map generation method based on multi-modal data fusion and reinforcement learning involved in this embodiment 1 includes the following contents: S1, multi-modal sensor collaborative perception S1.1, vision-laser radar deep fusion: First, a multi-view camera array is established: three front, left, and right cameras (FOV 120) synchronously capture RGB images, and multi-scale features (resolution 512x512) are extracted through a shared EfficientNet-B0 backbone network; then, an initial elevation map is constructed using the sparse depth information of LiDAR point clouds; S1.2, super-resolution reconstruction: based on the initial elevation map constructed in the above step, a generative adversarial network (GAN) is generated, and the depth image is super-resolution reconstructed (0.05m / pixel) through the generative adversarial network (GAN) to solve the problem of insufficient distance measurement (>10m) of the visual sensor; S1.3, spatio-temporal alignment: through the gravity alignment matrix G=R·K^−1 (R is the vehicle pose rotation matrix, and K is the camera intrinsic parameter), the visual features in the camera coordinate system are converted into the global coordinate system consistent with the direction of gravity, and the interference of the terrain undulation on the pose estimation is eliminated.

[0017] S2, direction perception feature encoding S2.1, direction vector de-projection: convert the image pixel coordinates (ui, vi) into a three-dimensional direction vector di=P[ui, vi, 1]T^-1, where P=G -1 ×R −1 ×K −1 fuse camera parameters and vehicle pose information; S2.2, position encoding enhancement: encode the direction vector generated by the IMU and the wheel encoder into a learnable position embedding through a multi-layer perception (MLP), combine the visual features to construct a direction perception query matrix, and improve the feature positioning accuracy on uneven terrain.

[0018] S3, spatio-temporal consistency optimization S3.1, recursive prediction mechanism: the elevation map prediction result of the previous time is aligned to the current coordinate system through the vehicle pose transformation matrix T t−1 →T t , and a history-enhanced query Q t =[Qlearned, CNN(M t −1⊙O t ) ] is generated, where O t is a time overlap mask; S3.2, temporal consistency constraint: introduce a smooth L1 loss and a gradient matching loss to force the continuity of adjacent frame predictions and reduce high-frequency noise in dynamic environments.

[0019] S4, reinforcement learning optimization closed loop S4.1, Policy network design: Adopt the proximal policy optimization (PPO) algorithm, with the path planning passability evaluation (DWA algorithm) as the reward signal, dynamically adjusting parameters such as filter strength (σ), semantic weight (λ), etc. S4.2, Uncertainty quantification: Estimate the confidence (mean μ and variance σ²) of the elevation value through the Bayesian deep learning framework, trigger the resampling mechanism of the laser radar in the low confidence area, and realize dynamic optimization driven by data.

[0020] S5, Laser radar secondary training S5.1, Pseudo-label generation: Use the visually generated elevation map as the initial pseudo-label, and combine it with the LiDAR point cloud to construct a weakly supervised training set. S5.2, Consistency regularization: Constrain the consistency of visual prediction and laser radar measurement through adversarial training (GAN), and improve the reconstruction accuracy of long-distance (15m+) terrain details. Specific embodiment 1

[0021] Step 1, data acquisition and preprocessing 1.1, Synchronously collect multi-camera RGB images, LiDAR point clouds, and IMU data; 1.2, Denoise and voxelize the LiDAR point cloud to generate an initial elevation map.

[0022] Step 2, feature extraction and fusion 2.1, Use EfficientNet-B0 to extract multi-scale visual features; 2.2, Super-resolution reconstruction of depth images through GAN, and spatiotemporal alignment with LiDAR point clouds; 2.3, Embed direction-aware position and concatenate with visual features, input into the Transformer encoder.

[0023] Step 3, spatiotemporal consistency modeling 3.1, Recursively fuse historical elevation map prediction results, and suppress redundant information through temporal overlap masks; 3.2, Calculate the smooth L1 loss and gradient matching loss to optimize the prediction continuity.

[0024] Step 4, reinforcement learning optimization 4.1, Deploy the PPO policy network to adjust the fusion parameters in real time; 4.2, Trigger LiDAR resampling based on Bayesian confidence to update the pseudo-label training set.

[0025] Step 5, model training and deployment 5.1, Stage training: First, pre-train the visual-LiDAR fusion module, and then jointly optimize the reinforcement learning strategy; 5.2, Deployed to embedded platforms, achieve 20Hz real-time inference.

[0026] Working principle: The application provides a height map prediction generation method, comprising: (1) learning the cross-view association of RGB cameras, depth cameras and three-dimensional radar data; (2) combining the vehicle attitude information in combination with these associations; (3) the time information encoded in the previous prediction history is realized by using a multi-scale transform encoder (Transformer Encoder) form.

[0027] Firstly, a backbone feature extractor is used to extract multi-scale visual features, then the features are connected with direction-aware position embedding and are transmitted to a transform encoder; the latter also receives a learned map view embedding connected with the predicted encoding history (the motivation behind this design is to allow the transformer to learn to associate features from different views in space and from multiple time steps to the map view perspective); finally, the output features of the transform encoder are transmitted to a decoder, which performs regression on the 2.5D height map.

[0028] The following is a description of several innovations of the application: Geometric-semantic joint representation: the visual features are associated with the map view through the cross-attention mechanism, breaking through the alignment error limitation of traditional loose coupling fusion.

[0029] Dynamic environment adaptation: the reinforcement learning framework establishes a direct mapping between the generated parameters and the navigation performance, solving the failure problem of fixed parameters in complex terrain.

[0030] Uncertainty perception: Bayesian deep learning realizes the quantitative evaluation of terrain risk, providing safety margin guarantee for medium-speed driving.

[0031] The application realizes that in the KITTI dataset verification, the true effective value of the height map is reduced to 15cm, the semantic classification accuracy rate reaches 90%, and the real-time navigation demand under the speed of 2-5m / s is effectively supported.

[0032] Through the multi-modal deep fusion architecture and the reinforcement learning optimization mechanism, the application breaks through the bottleneck of sparsity, fixed parameters and semantic loss of traditional height map generation, realizes the real-time height map generation of more than 15 meters in the medium-speed unmanned driving scene, 0.05m / pixel high resolution, reduces the height error to 15cm, and improves the path planning success rate to 90%. At the same time, through Bayesian deep learning, the terrain risk is quantitatively evaluated, an interpretable safety decision basis is provided for complex unstructured environment, the system cost is reduced by 70% compared with the pure laser radar scheme, and the navigation robustness and engineering practicability of the intelligent agent in the dynamic environment are significantly enhanced.

[0033] The above are only specific application examples of the present application, and do not constitute any limitation on the protection scope of the present application. Any technical solution formed by equivalent transformation or equivalent replacement falls within the protection scope of the present application.

Claims

1. A hybrid elevation map generation method based on multimodal data fusion and reinforcement learning, characterized in that: Includes the following: S1. Multimodal sensor collaborative perception: S1.1, Vision-LiDAR Deep Fusion: Synchronously collect multi-camera RGB images, LiDAR point clouds, and IMU data, and then use the sparse depth information of the LiDAR point clouds to construct an initial elevation map; S1.2, Super-resolution Reconstruction: Based on the initial elevation map constructed in the above steps, a generative adversarial network is used to reconstruct the depth image through super-resolution, solving the problem of insufficient ranging of visual sensors at distances greater than 10m; S1.3, Spatiotemporal Alignment: The visual features in the camera coordinate system are converted to a global coordinate system consistent with the direction of gravity through the gravity alignment matrix G = R·K^−1, where R is the vehicle posture rotation matrix and K is the camera intrinsic parameter, to eliminate the interference of terrain undulation on posture estimation; S2, direction-aware feature encoding: S2.1, Direction vector deprojection: Convert the image pixel coordinates (ui, vi) into a three-dimensional direction vector di = P[ui, vi, 1]T^-1, where P = G -1 ×R −1 ×K −1 Fusion of camera parameters and vehicle posture information; S3, spatiotemporal consistency optimization: S3.1, recursive prediction mechanism: the elevation map prediction result of the previous moment is transformed into the vehicle posture transformation matrix T t−1 →T t Align to the current coordinate system and generate historical enhanced query Q t =[Qlearned,CNN(M t −1⊙O t )], where O t is the temporal overlap mask; S4. Reinforcement learning optimization closed loop: S4.

1. Policy Network Design: Using a proximal policy optimization algorithm, the feasibility evaluation of the path plan is used as the reward signal, and the parameters of the filter strength σ and semantic weight λ are dynamically adjusted; S5. LiDAR secondary training S5.

1. Consistency Regularization: Improves the reconstruction accuracy of terrain details at distances greater than 15m by constraining the consistency between visual predictions and lidar measurements through adversarial training.

2. The hybrid elevation map generation method based on multimodal data fusion and reinforcement learning according to claim 1, characterized in that: In step S1.1, a multi-view camera array is first established: the front, left, and right cameras are set to synchronously capture RGB images and extract multi-scale features using a shared EfficientNet-B0 backbone network. The LiDAR point cloud is then denoised and voxelized to generate an initial elevation map.

3. The hybrid elevation map generation method based on multimodal data fusion and reinforcement learning according to claim 1 is characterized in that: In step S1.2, the depth image is super-reconstructed using a generative adversarial network, and the reconstructed resolution is 0.05m / pixel.

4. The hybrid elevation map generation method based on multimodal data fusion and reinforcement learning according to claim 1, characterized in that: Step S2 also includes S2.2, position encoding enhancement: encoding the direction vectors generated by the IMU and wheel encoder into a learnable position embedding through a multi-layer perceptron, and constructing a direction perception query matrix based on visual features to improve feature positioning accuracy on uneven terrain.

5. The hybrid elevation map generation method based on multimodal data fusion and reinforcement learning according to claim 1, characterized in that: Step S3 also includes S3.2, temporal consistency constraint: introducing smooth L1 loss and gradient matching loss to enforce the continuity of adjacent frame predictions and reduce high-frequency noise in dynamic environments.

6. The hybrid elevation map generation method based on multimodal data fusion and reinforcement learning according to claim 1, characterized in that: Step S4 also includes S4.2, uncertainty quantification: estimating the confidence of the elevation value through the Bayesian deep learning framework, triggering the resampling mechanism of the lidar in the low-confidence area, and realizing data-driven dynamic optimization.

7. The hybrid elevation map generation method based on multimodal data fusion and reinforcement learning according to claim 1, characterized in that: Before the consistency regularization step in step S5, the pseudo-label generation step is also included: the visually generated elevation map is used as the initial pseudo-label, and the LiDAR point cloud is combined to construct a weakly supervised training set.

Images