High-precision positioning method fusing reinforcement learning dynamic weight and cross-modal pre-completion
By combining modal health prediction and reinforcement learning dynamic weight fusion with a cross-modal pre-completion mechanism, the real-time performance and robustness issues of multimodal fusion positioning technology in complex scenarios are solved, achieving high-precision and stable autonomous driving positioning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUILIN UNIV OF ELECTRONIC TECH
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-19
AI Technical Summary
Existing multimodal fusion positioning technologies cannot intelligently adjust weights in complex scenarios, respond passively to modal failures, and lack real-time completion mechanisms, thus failing to meet the real-time and robustness requirements of autonomous driving.
Employing modal health prediction, pre-complete templates, reinforcement learning dynamic weight fusion, and closed-loop feedback mechanisms, this approach predicts modal failures through a time-series feature prediction model, completes data using a cross-modal generation model, and dynamically adjusts weights through reinforcement learning strategies to achieve proactive response and real-time optimization.
It achieves stable and uninterrupted positioning accuracy in complex scenarios, with positioning accuracy controlled within 0.05 meters and an interruption rate of less than 0.1%, meeting the real-time and robustness requirements of L4 autonomous driving.
Smart Images
Figure CN122064927A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, and in particular to a high-precision localization method that integrates reinforcement learning dynamic weights and cross-modal pre-completion. Background Technology
[0002] High-precision and robust positioning is a key technology for achieving high-level autonomous driving (Level 4 and above). Currently, mainstream positioning solutions are mainly divided into single-modal positioning and multi-modal fusion positioning. Single-modal positioning (such as pure laser SLAM or pure vision SLAM) is greatly affected by environmental interference and is prone to failure in complex scenarios such as tunnels, strong light, and heavy rain. Multi-modal fusion positioning, by fusing data from multiple sensors such as LiDAR, vision cameras, and inertial measurement units (IMUs), leverages their complementarity to improve the system's robustness and has become the mainstream approach.
[0003] However, existing multimodal fusion positioning technologies still have obvious defects. The closest existing technology is the "confidence-based multimodal positioning fusion system". This scheme allocates fusion weights according to the confidence of each modality's real-time output (based on a simple judgment of data quality), and directly discards the data of a modality when the confidence of a certain modality is lower than the threshold. This scheme has the following core problems: (1) The weight allocation strategy is fixed or simply adaptive, and cannot be intelligently adjusted according to scene characteristics (such as reducing visual weight in advance when entering a tunnel) and modality failure trend; (2) The response to modality failure is passive, and data is discarded only after failure occurs, resulting in a reduction in the effective data dimension and fluctuations in positioning results; (3) There is a lack of efficient completion mechanism, or the real-time generation of completion data in the event of sudden failure takes too long (15-20ms), which cannot meet the real-time requirements of autonomous driving (≤10ms); (4) The system lacks self-optimization capability and cannot iteratively optimize the fusion and prediction models based on long-term operating results.
[0004] Therefore, it is necessary to provide a new high-precision localization method that integrates reinforcement learning dynamic weights and cross-modal pre-completion to solve the above-mentioned technical problems. Summary of the Invention
[0005] The technical problem solved by this invention is to provide a high-precision positioning method that integrates reinforcement learning dynamic weights and cross-modal pre-completion, which can upgrade from passive response to active response by introducing mechanisms such as modal health prediction, pre-completion template, reinforcement learning dynamic weight fusion, and closed-loop feedback, thereby ensuring stable and uninterrupted positioning accuracy in various complex scenarios.
[0006] To address the aforementioned technical problems, the high-precision localization method merging reinforcement learning dynamic weights and cross-modal pre-completion provided by this invention includes the following steps: S1: Sensor data acquisition and preprocessing: Multi-source data is acquired through lidar, vision camera and inertial measurement unit (IMU), and time synchronization and preprocessing are performed to extract semantic features of each modality; S2: Multi-model parallel localization solution: Based on the preprocessed data, independent localization solutions are performed using the laser odometry LiO model, visual odometry ViO model, and IMU model, and the localization results and real-time confidence scores for each modality are output. S3: Modal health prediction: Based on the temporal confidence, scene features and sensor state features of each modality, a temporal feature prediction model is used to predict the failure probability of each modality within a set time in the future; S4: Pre-completion preparation: When the predicted failure probability of a certain mode reaches a preset threshold, the pre-completion mechanism is triggered to screen the health of each mode, select the mode currently in a high confidence state as the source mode, and preload and initialize the cross-modal generation model. S5: Reinforcement Learning Dynamic Fusion: Construct a unified feature vector containing localization results, confidence, failure probability and scene features of each modality, input it into the trained reinforcement learning decision model, combine multi-head attention mechanism, dynamically calculate the fusion weight of localization results of each modality, and generate intermediate localization results by weighting. S6: Modal Failure Completion: When a modal failure is detected, virtual completion data is generated in real time using a preloaded cross-modal generation model; S7: Consistency check: Perform trajectory continuity check, semantic and geometric consistency check on the completed positioning results to ensure the reliability of the completed data; S8: Final positioning output and closed-loop feedback: Output the verified final positioning result and feed back the positioning accuracy and failure prediction accuracy to the reinforcement learning decision model of the reinforcement learning dynamic fusion and the temporal feature prediction model of the modality health prediction for continuous optimization.
[0007] Preferably, in the modal health prediction, a time-series feature prediction model is used to predict the failure probability, specifically including: The input to the temporal feature prediction model is a feature vector after feature extraction or dimensionality reduction, including: the confidence sequence of LiO, ViO, and IMU within the last 5 seconds, the one-hot encoding of the current scene type, the density of the laser point cloud, the number of image feature points, and the zero bias value of the IMU; The structure of the temporal feature prediction model includes three temporal convolutional layers, two fully connected layers, and the final output layer uses the Sigmoid activation function to output the failure probability of LiO, ViO, and IMU in the next 2 seconds.
[0008] Preferably, in the pre-completion preparation, generating the virtual feature template specifically involves: A virtual ORB feature library for visual ViO is generated by mapping the static semantic point cloud features of the LiDAR LiO using a pre-trained point cloud feature extraction network; or, A combined model of a pre-trained image feature extraction network and a generative adversarial network (GAN) is used to map visual image features to generate virtual features of LiDAR point clouds.
[0009] Preferably, in the reinforcement learning dynamic fusion, the training process of the reinforcement learning decision model includes: The state space is all possible states corresponding to the unified feature vector; The action space is a weighted combination of the three modes of LiO, ViO, and IMU, with a weight sum of 1; The reward function is set as follows: a positive reward is given when the positioning accuracy is ≤0.05 meters, the continuous positioning is uninterrupted, and the pre-complete template matching degree is ≥90%; a negative penalty is given when the positioning accuracy is >0.3 meters, the positioning is interrupted after the mode fails, and the matching degree after completion is <80%.
[0010] Preferably, the consistency check further includes crowdsourced auxiliary check: Obtain de-identified location results of other vehicles in the same area through the Internet of Vehicles; Calculate the average deviation between the self-localization result and the crowdsourced results. If the deviation is greater than a set threshold, trigger the reinforcement learning dynamic fusion to re-optimize the fusion weights.
[0011] This invention also provides a high-precision localization system that integrates reinforcement learning dynamic weights and cross-modal pre-completion, comprising: The data acquisition and preprocessing module is used to acquire and preprocess multi-source data from LiDAR, visual cameras, and IMU. The multi-model parallel localization solution module is used to run LiO, ViO, and IMU models to output localization results and confidence scores for each mode. The modal health prediction module has a built-in time-series feature prediction model to predict the failure probability of each modality. The pre-completion management module is used to preload and initialize the cross-modal generation model when the predicted failure probability exceeds the threshold, and store the feature mapping parameters in the cache. The dynamic fusion decision module has a built-in reinforcement learning decision model, which is used to calculate dynamic fusion weights and generate intermediate localization results; The modal failure completion module is used to perform data completion using a preloaded cross-modal generative model when a mode actually fails. The consistency verification module is used to perform trajectory, semantic, and geometric consistency verification on the completed data. The closed-loop feedback optimization module is used to optimize the dynamic fusion decision module and the modal health prediction module based on the feedback of the final positioning effect.
[0012] Compared with related technologies, the high-precision localization method that integrates reinforcement learning dynamic weights and cross-modal pre-completion provided by this invention has the following advantages: This invention provides a high-precision positioning method that integrates reinforcement learning dynamic weights and cross-modal pre-completion. By using a TCN model to predict modal failure probability 2 seconds in advance and generating pre-completion templates, it avoids positioning fluctuations at the moment of failure, thus enabling proactive response to failures. Employing a reinforcement learning strategy that integrates DQN and attention mechanisms, the weight allocation considers not only the current confidence level but also the failure probability and scene features, improving scene adaptability. The pre-completion mechanism compresses the completion time from the existing 15-20ms to less than 2ms, and even real-time completion can be completed within 10ms, far exceeding the real-time requirements of autonomous driving. Through a dual closed-loop mechanism of "positioning-feedback-optimization," it continuously learns from actual operation, constantly optimizing the fusion and prediction models, improving long-term robustness. Through the synergistic effect of the above mechanisms, this invention can stably control the final positioning accuracy within 0.05 meters, with an interruption rate of less than 0.1%, fully meeting the requirements of L4 level autonomous driving. Attached Figure Description
[0013] Figure 1 A flowchart illustrating the high-precision localization method that integrates reinforcement learning dynamic weights and cross-modal pre-completion provided by this invention; Figure 2 for Figure 1 An enlarged schematic diagram of one of the parts shown; Figure 3 for Figure 1 An enlarged schematic diagram of another part is shown. Detailed Implementation
[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0015] Please refer to the following: Figure 1 , Figure 2 and Figure 3 ,in, Figure 1 A flowchart of the high-precision localization method that integrates reinforcement learning dynamic weights and cross-modal pre-completion provided by the present invention; Figure 2 for Figure 1 An enlarged schematic diagram of one of the parts shown; Figure 3 for Figure 1 The diagram shown is an enlarged version of another unsealed schematic. The high-precision localization method integrating reinforcement learning dynamic weights and cross-modal pre-completion includes the following steps: S1: Sensor data acquisition and preprocessing: Multi-source data is acquired through lidar, vision camera and inertial measurement unit (IMU), and time synchronization and preprocessing are performed to extract semantic features of each modality; S2: Multi-model parallel localization solution: Based on the preprocessed data, independent localization solutions are performed using the laser odometry LiO model, visual odometry ViO model, and IMU model, and the localization results and real-time confidence scores for each modality are output. S3: Modal health prediction: Based on the temporal confidence of each modality, scene features and sensor state features, the temporal convolutional neural network (TCN) model is used to predict the failure probability of each modality within a set time in the future. In the modal health prediction, the temporal convolutional neural network (TCN) model is used to predict the failure probability, specifically including: The input to the TCN model is a feature vector after feature extraction or dimensionality reduction, including: the confidence sequence of LiO, ViO, and IMU in the last 5 seconds, the one-hot encoding of the current scene type, the density of laser point cloud, the number of image feature points, and the zero bias value of IMU. The TCN model consists of three temporal convolutional layers, two fully connected layers, and the final output layer uses the Sigmoid activation function to output the failure probability of LiO, ViO, and IMU in the next 2 seconds. S4: Pre-completion preparation: When the predicted failure probability of a certain modality reaches a preset threshold, the pre-completion mechanism is triggered to screen the health of each modality, select the modality currently in a high confidence state as the source modality, preload and initialize the cross-modal generation model, and generate and cache virtual feature templates that match the failed modality based on the features of the current effective modality through the pre-trained cross-modal generation model. The specific steps for generating virtual feature templates are as follows: The static semantic point cloud features of the LiDAR LiO are mapped using a pre-trained PointNet++ network to generate a virtual ORB feature library for the Vision ViO; or, The model combines pre-trained ResNet and generative adversarial network GAN to generate virtual features of LiDAR point clouds by mapping visual image features. S5: Reinforcement learning dynamic fusion: Construct a unified feature vector containing localization results, confidence, failure probability and scene features of each modality, input it into a trained deep Q network (DQN), combine it with a multi-head attention mechanism, dynamically calculate the fusion weights of localization results of each modality, and generate intermediate localization results by weighting. In the aforementioned reinforcement learning dynamic fusion, the training process of the Deep Q-Network (DQN) includes: The state space is all possible states corresponding to the unified feature vector; The action space is a weighted combination of the three modes of LiO, ViO, and IMU, with a weight sum of 1; The reward function is set as follows: a positive reward is given when the positioning accuracy is ≤0.05 meters, continuous positioning is uninterrupted, and the pre-complete template matching degree is ≥90%; a negative penalty is given when the positioning accuracy is >0.3 meters, positioning is interrupted after modal failure, and the matching degree after completion is <80%. S6: Modal Failure Completion: When a modality is detected to be actually failed, the cached pre-completion template is called first to complete the data. If there is no valid template, virtual completion data is generated in real time. S7: Consistency check: Perform trajectory continuity check, semantic and geometric consistency check on the completed positioning results to ensure the reliability of the completed data; The consistency check also includes crowdsourced auxiliary verification: Obtain de-identified location results of other vehicles in the same area through the Internet of Vehicles; Calculate the average deviation between the self-localization result and the crowdsourced result. If the deviation is greater than a set threshold, trigger the reinforcement learning dynamic fusion to re-optimize the fusion weights. S8: Final positioning output and closed-loop feedback: Output the verified final positioning result and feed back the positioning accuracy and failure prediction accuracy to the DQN dynamic fusion model of reinforcement learning and the TCN model of modality health prediction for continuous optimization.
[0016] This invention also provides a high-precision localization system that integrates reinforcement learning dynamic weights and cross-modal pre-completion, comprising: The data acquisition and preprocessing module is used to acquire and preprocess multi-source data from LiDAR, visual cameras, and IMU. The multi-model parallel localization solution module is used to run LiO, ViO, and IMU models to output localization results and confidence scores for each mode. The modal health prediction module has a built-in temporal convolutional neural network (TCN) to predict the failure probability of each modality. The pre-completion management module is used to generate and cache virtual feature templates when the predicted failure probability exceeds a threshold. The dynamic fusion decision module has a built-in deep Q-network (DQN) for calculating dynamic fusion weights and generating intermediate localization results. The modal failure completion module is used to perform data completion when a mode actually fails. The consistency verification module is used to perform trajectory, semantic, and geometric consistency verification on the completed data. The closed-loop feedback optimization module is used to optimize the dynamic fusion decision module and the modal health prediction module based on the feedback of the final positioning effect.
[0017] Compared with related technologies, the high-precision localization method that integrates reinforcement learning dynamic weights and cross-modal pre-completion provided by this invention has the following advantages: This invention provides a high-precision positioning method that integrates reinforcement learning dynamic weights and cross-modal pre-completion. By using a TCN model to predict modal failure probability 2 seconds in advance and generating pre-completion templates, it avoids positioning fluctuations at the moment of failure, thus enabling proactive response to failures. Employing a reinforcement learning strategy that integrates DQN and attention mechanisms, the weight allocation considers not only the current confidence level but also the failure probability and scene features, improving scene adaptability. The pre-completion mechanism compresses the completion time from the existing 15-20ms to less than 2ms, and even real-time completion can be completed within 10ms, far exceeding the real-time requirements of autonomous driving. Through a dual closed-loop mechanism of "positioning-feedback-optimization," it continuously learns from actual operation, constantly optimizing the fusion and prediction models, improving long-term robustness. Through the synergistic effect of the above mechanisms, this invention can stably control the final positioning accuracy within 0.05 meters, with an interruption rate of less than 0.1%, fully meeting the requirements of L4 level autonomous driving.
[0018] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A high-precision localization method integrating reinforcement learning dynamic weights and cross-modal pre-completion, characterized in that, Includes the following steps: S1: Sensor data acquisition and preprocessing: Multi-source data is acquired through lidar, vision camera and inertial measurement unit (IMU), and time synchronization and preprocessing are performed to extract semantic features of each modality; S2: Multi-model parallel localization solution: Based on the preprocessed data, independent localization solutions are performed using the laser odometry LiO model, visual odometry ViO model, and IMU model, and the localization results and real-time confidence scores for each modality are output. S3: Modal health prediction: Based on the temporal confidence, scene features and sensor state features of each modality, a temporal feature prediction model is used to predict the failure probability of each modality within a set time in the future; S4: Pre-completion preparation: When the predicted failure probability of a certain mode reaches a preset threshold, the pre-completion mechanism is triggered to screen the health of each mode, select the mode currently in a high confidence state as the source mode, and preload and initialize the cross-modal generation model. S5: Reinforcement Learning Dynamic Fusion: Construct a unified feature vector containing localization results, confidence, failure probability and scene features of each modality, input it into the trained reinforcement learning decision model, combine multi-head attention mechanism, dynamically calculate the fusion weight of localization results of each modality, and generate intermediate localization results by weighting. S6: Modal Failure Completion: When a modal failure is detected, virtual completion data is generated in real time using a preloaded cross-modal generation model; S7: Consistency check: Perform trajectory continuity check, semantic and geometric consistency check on the completed positioning results to ensure the reliability of the completed data; S8: Final positioning output and closed-loop feedback: Output the verified final positioning result and feed back the positioning accuracy and failure prediction accuracy to the reinforcement learning decision model of the reinforcement learning dynamic fusion and the temporal feature prediction model of the modality health prediction for continuous optimization.
2. The high-precision localization method integrating reinforcement learning dynamic weights and cross-modal pre-completion as described in claim 1, characterized in that, In the modal health prediction, a time-series feature prediction model is used to predict the failure probability, specifically including: The input to the temporal feature prediction model is a feature vector after feature extraction or dimensionality reduction, including: the confidence sequence of LiO, ViO, and IMU within the last 5 seconds, the one-hot encoding of the current scene type, the density of the laser point cloud, the number of image feature points, and the zero bias value of the IMU; The structure of the temporal feature prediction model includes three temporal convolutional layers, two fully connected layers, and the final output layer uses the Sigmoid activation function to output the failure probability of LiO, ViO, and IMU in the next 2 seconds.
3. The high-precision localization method integrating reinforcement learning dynamic weights and cross-modal pre-completion as described in claim 1, characterized in that, In the pre-completion preparation, generating the virtual feature template specifically involves: A virtual ORB feature library for visual ViO is generated by mapping the static semantic point cloud features of the LiDAR LiO using a pre-trained point cloud feature extraction network; or, A combined model of a pre-trained image feature extraction network and a generative adversarial network (GAN) is used to map visual image features to generate virtual features of LiDAR point clouds.
4. The high-precision localization method integrating reinforcement learning dynamic weights and cross-modal pre-completion as described in claim 1, characterized in that, In the aforementioned reinforcement learning dynamic fusion, the training process of the reinforcement learning decision model includes: The state space is all possible states corresponding to the unified feature vector; The action space is a weighted combination of the three modes of LiO, ViO, and IMU, with a weight sum of 1; The reward function is set as follows: a positive reward is given when the positioning accuracy is ≤0.05 meters, the continuous positioning is uninterrupted, and the pre-complete template matching degree is ≥90%; a negative penalty is given when the positioning accuracy is >0.3 meters, the positioning is interrupted after the mode fails, and the matching degree after completion is <80%.
5. The high-precision localization method integrating reinforcement learning dynamic weights and cross-modal pre-completion as described in claim 1, characterized in that, The consistency check also includes crowdsourced auxiliary verification: Obtain de-identified location results of other vehicles in the same area through the Internet of Vehicles; Calculate the average deviation between the self-localization result and the crowdsourced results. If the deviation is greater than a set threshold, trigger the reinforcement learning dynamic fusion to re-optimize the fusion weights.
6. A high-precision localization system for implementing the method described in any one of claims 1-5, which integrates reinforcement learning dynamic weights and cross-modal pre-completion, characterized in that, include: The data acquisition and preprocessing module is used to acquire and preprocess multi-source data from LiDAR, visual cameras, and IMU. The multi-model parallel localization solution module is used to run LiO, ViO, and IMU models to output localization results and confidence scores for each mode. The modal health prediction module has a built-in time-series feature prediction model to predict the failure probability of each modality. The pre-completion management module is used to preload and initialize the cross-modal generation model when the predicted failure probability exceeds the threshold, and store the feature mapping parameters in the cache. The dynamic fusion decision module has a built-in reinforcement learning decision model, which is used to calculate dynamic fusion weights and generate intermediate localization results; The modal failure completion module is used to perform data completion using a preloaded cross-modal generative model when a mode actually fails. The consistency verification module is used to perform trajectory, semantic, and geometric consistency verification on the completed data. The closed-loop feedback optimization module is used to optimize the dynamic fusion decision module and the modal health prediction module based on the feedback of the final positioning effect.