Multi-sensor fusion slam parameter adjustment method and system, and related device

Abstract

The application provides a multi-sensor fusion SLAM parameter adjustment method and system and related equipment, the method comprises the following steps: obtaining and preprocessing the original observation data of each sensor to generate observation features of each sensor; multi-dimensional features representing the current SLAM system state, environmental characteristics and data quality of each sensor are extracted from the front end and rear end of each sensor to construct a multi-dimensional state vector; the multi-dimensional state vector is input into a deep reinforcement learning agent to output an action vector for the key parameters of the multi-sensor fusion SLAM system; the action vector is applied to the SLAM system, and a multi-objective reward is calculated according to the running effect of the system after parameter adjustment; the deep reinforcement learning agent is driven by the multi-objective reward to update and optimize the strategy. Through the deep learning collaborative optimization of the multi-dimensional features of the multi-sensor and the key parameters of the system, the positioning accuracy and robustness of the system in complex and variable and sensor degradation scenarios are improved.

Description

Technical Field

[0001] This invention relates to the field of robot navigation and positioning technology, and in particular to a method, system and related equipment for adjusting SLAM parameters using multi-sensor fusion. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology for autonomous navigation in mobile robots. Its goal is to simultaneously estimate the robot's pose and construct an environmental map in an unknown environment. 3D laser SLAM uses LiDAR (Light Detection and Ranging) as its primary sensor, emitting laser beams and receiving echoes to measure distance information from the surrounding environment, generating high-density 3D point cloud data. Compared to visual SLAM, 3D laser SLAM offers advantages such as higher accuracy, less susceptibility to lighting conditions, and more accurate ranging, leading to its widespread application in autonomous driving, drones, and service robots.

[0003] Multi-sensor fusion SLAM systems aim to leverage the complementary advantages of multiple sensors to achieve more robust and accurate localization and mapping than single-sensor systems. However, existing multi-sensor fusion SLAM systems still face the following technical challenges: (1) The difficulty of manually tuning parameters in existing multi-sensor fusion SLAM systems. Traditional multi-sensor fusion SLAM systems (such as LIO-SAM, VINS-Fusion, LVI-SAM, R2LIVE, etc.) involve front-end processing, data association, and fusion optimization modules of various sensors such as LiDAR, vision camera, IMU (inertial measurement unit), wheel speedometer, and RTK (Real-Time Kinematic). Each module relies on a large number of manually preset parameters (such as feature extraction threshold, ICP maximum corresponding point distance threshold, reprojection error threshold, IMU noise covariance, fusion weight coefficient, key frame selection threshold, RTK confidence threshold, etc.). These parameters require tedious manual tuning for different sensor models and application scenarios, and there are complex coupling relationships between the parameters of each sensor, which seriously restricts the deployment efficiency and generalization ability of multi-sensor fusion SLAM systems.

[0004] (2) The problem of sharp decline in positioning accuracy of multi-sensor fusion SLAM system in some sensor degradation scenarios. In multi-sensor fusion system, different sensors have their own degradation modes in different environments: visual sensors are prone to failure in scenarios with drastic changes in lighting, weak texture, and fast movement; lidar is prone to degradation in scenarios such as rainy and foggy weather, open squares, and long straight corridors; RTK loses signal in occluded environments (high-rise buildings, tunnels, dense forests); wheel speedometers drift in skidding scenarios. Traditional fusion SLAM system adopts fixed fusion weights and parameter strategies, which cannot adaptively adjust the fusion strategy in scenarios where one or more sensors are degraded, resulting in a sharp decline in the overall positioning accuracy of the system or even complete failure.

[0005] (3) Multi-sensor fusion systems lack online adaptive capability to the characteristics of heterogeneous sensor data. In practical applications, the data quality, noise characteristics, and sampling frequency of different sensors vary significantly and change with time and environment. Traditional methods use fixed noise covariance and fusion weights, which cannot dynamically adjust the fusion strategy according to the real-time data quality of the sensors, resulting in the fusion results being contaminated by low-quality sensor data.

[0006] (4) Issues related to the quality of time synchronization and extrinsic parameter errors of multi-sensor systems. Multi-sensor fusion SLAM systems rely on time synchronization and spatial extrinsic parameter calibration. In actual operation, temperature changes, mechanical vibrations, and communication jitter may cause synchronization errors or a decrease in the effectiveness of extrinsic parameters. Traditional methods usually lack online perception of synchronization quality, extrinsic parameter consistency, and the resulting fusion residuals, making it difficult to reduce the weight of constraints imposed by affected sensors in a timely manner.

[0007] (5) The computational complexity of multi-parameter joint optimization. Multi-sensor fusion SLAM systems involve 20-30 mutually coupled key parameters. Manual tuning not only relies on expert experience, but also makes it difficult to achieve joint optimization of multiple parameters. Exhaustive search or grid search methods have huge computational costs in high-dimensional parameter spaces and cannot meet real-time requirements.

[0008] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0009] This invention provides a method, system, and related equipment for adjusting SLAM parameters using multi-sensor fusion. The main objective of this invention is to solve the technical problems mentioned in the background section of the prior art.

[0010] The first aspect of this invention discloses a method for adjusting SLAM parameters through multi-sensor fusion, comprising: The raw observation data of each sensor is acquired and preprocessed to generate the observation features of each sensor. The preprocessing includes time synchronization, distortion removal and feature extraction. Multi-dimensional features characterizing the current SLAM system state, environmental characteristics, and data quality of each sensor are extracted from the front and back ends of each sensor to construct a multi-dimensional state vector. The multi-dimensional state vector is input into the deep reinforcement learning agent, and the deep reinforcement learning agent outputs action vectors for key parameters of the multi-sensor fusion SLAM system. The motion vectors are applied to the multi-sensor fusion SLAM system, and multi-objective rewards are calculated based on the system's performance after parameter adjustment. The multi-objective reward is used to drive the deep reinforcement learning agent to update and optimize its policies.

[0011] In an optional embodiment of the first aspect of the present invention, the multi-dimensional state vector includes at least two of the following: LiDAR point cloud statistical features, LiDAR registration quality features, visual feature tracking features, IMU state features, wheel speedometer state features, RTK state features, multi-sensor consistency features, environmental degradation detection features, and historical performance features.

[0012] In an optional embodiment of the first aspect of the present invention, the action vector includes at least two of the following: LiDAR feature extraction threshold, ICP maximum corresponding point distance threshold, visual feature matching threshold, visual reprojection error threshold, LiDAR-visual fusion weight, IMU noise covariance scaling factor, wheel velocity meter fusion weight, RTK availability determination threshold, RTK-SLAM fusion weight, and keyframe insertion threshold.

[0013] In an optional embodiment of the first aspect of the present invention, the multi-target reward includes a weighted sum of at least two of the following: positioning accuracy reward, multi-sensor consistency reward, computational efficiency reward, map quality reward, robustness reward, and parameter smoothness reward.

[0014] In an optional embodiment of the first aspect of the present invention, the deep reinforcement learning agent adopts an algorithm framework based on dual-delay deep deterministic policy gradient, including an Actor policy network and a dual Critic value network, and introduces a multi-head attention mechanism to handle and learn the complex coupled interaction relationships between multi-sensor state features.

[0015] In an optional embodiment of the first aspect of the present invention, the training of the deep reinforcement learning agent adopts a two-stage deployment strategy that combines simulation pre-training with online fine-tuning: in the first stage, pre-training is carried out in a multi-sensor degradation simulation environment in combination with a course learning mechanism to obtain basic parameter adjustment strategies; in the second stage, after deployment in a real environment, online fine-tuning is carried out by collecting actual operation experience data to achieve continuous learning.

[0016] In an optional embodiment of the first aspect of the present invention, the multi-sensor fusion SLAM system is provided with multiple adjustable parameter interfaces, and the deep reinforcement learning agent modifies the parameter values ​​of the multiple adjustable parameter interfaces in real time through parameter service or dynamic reconfiguration mechanism. Each of the adjustable parameter interfaces has safe upper and lower bound constraints, and the parameter adjustment of the deep reinforcement learning agent and the main loop of the multi-sensor fusion SLAM system are executed asynchronously at a fixed frequency. The degradation status of each sensor is monitored in real time. When degradation of any sensor is detected by the sliding window statistics, the deep reinforcement learning agent automatically reduces or blocks the fusion weight of the sensor and gradually restores it after the quality is restored. When the multi-sensor fusion SLAM system detects tracking loss or fusion failure, it automatically switches to a conservative mode using preset safety parameters.

[0017] A second aspect of the present invention provides a multi-sensor fusion SLAM parameter adjustment system, the SLAM parameter adjustment system comprising: A multi-sensor data preprocessing module is used to acquire the raw observation data of each sensor and preprocess it to generate the observation features of each sensor. The preprocessing includes time synchronization, distortion removal and feature extraction. The multi-sensor state observation module is used to extract multi-dimensional features from the front and back ends of each sensor to characterize the current SLAM system state, environmental characteristics, and data quality of each sensor, and to construct a multi-dimensional state vector. The deep reinforcement learning agent module is used to input the multi-dimensional state vector into the deep reinforcement learning agent, and output the action vector for the key parameters of the multi-sensor fusion SLAM system through the deep reinforcement learning agent. The parameter execution and effect evaluation module is used to apply the action vector to the multi-sensor fusion SLAM system and calculate multi-objective rewards based on the system operation effect after parameter adjustment. The experience replay and policy update module is used to drive the deep reinforcement learning agent to update and optimize its policies using the multi-objective reward.

[0018] A third aspect of the present invention provides a navigation device, the navigation device comprising: a memory and at least one processor, the memory storing instructions, and the memory and the at least one processor being interconnected via a circuit; The at least one processor invokes the instructions in the memory to cause the navigation device to perform the multi-sensor fusion SLAM parameter adjustment method as described in any one of the first aspects of the invention.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the SLAM parameter adjustment method for multi-sensor fusion as described in any one of the first aspects of the present invention.

[0020] Beneficial Effects: This invention provides a method, system, and related equipment for adjusting SLAM parameters using multi-sensor fusion. The method includes acquiring and preprocessing raw observation data from each sensor to generate observation features for each sensor; extracting multi-dimensional features representing the current SLAM system state, environmental characteristics, and data quality from the front and back ends of each sensor to construct a multi-dimensional state vector; inputting the multi-dimensional state vector into a deep reinforcement learning agent to output action vectors for key parameters of the multi-sensor fusion SLAM system; applying the action vectors to the SLAM system and calculating multi-objective rewards based on the system's performance after parameter adjustment; and using the multi-objective rewards to drive the deep reinforcement learning agent to update and optimize policies. This invention improves the positioning accuracy and robustness of the system in complex, variable, and sensor-degraded scenarios by extracting multi-dimensional features from multiple sensors and performing deep learning-based collaborative optimization with key system parameters. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of an embodiment of a multi-sensor fusion SLAM parameter adjustment method of the present invention; Figure 2 This is a schematic diagram of an embodiment of a multi-sensor fusion SLAM parameter adjustment system according to the present invention; Figure 3 This is a schematic diagram of an embodiment of a navigation device according to an example of the present invention. Detailed Implementation

[0022] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Multi-sensor fusion 3D SLAM systems aim to leverage the complementary advantages of multiple sensors to achieve more robust and accurate localization and mapping than single-sensor systems. Currently, mainstream multi-sensor fusion SLAM systems include: (1) LiDAR-Vision Fusion SLAM: Represented by LVI-SAM, R2LIVE, and R3LIVE. LVI-SAM tightly couples LIO-SAM and VINS-Mono, using visual information to supplement the shortcomings of LiDAR in degraded scenes, while using the depth information of LiDAR to assist visual feature tracking. R3LIVE further introduces photometric calibration and mapping capabilities based on RGB colors, achieving accurate geometric and color reconstruction.

[0024] (2) LiDAR-Vision-IMU Fusion SLAM: Systems such as FAST-LIVO and LVIO-Fusion tightly couple and fuse three sensors: LiDAR, vision and IMU. The IMU provides high-frequency motion priors, LiDAR provides accurate depth measurement, and vision provides rich environmental texture information, thus achieving robust localization in a variety of challenging scenarios.

[0025] (3) Multi-source fusion SLAM (including wheel speed measurement, RTK (Real-Time Kinematic) etc.): Represented by GVINS and multi-source fusion frameworks, wheel speed measurement provides stable motion constraints, which are particularly suitable for structured road scenarios; RTK-GNSS provides centimeter-level absolute position constraints in open environments, which can effectively eliminate long-term cumulative drift. The fusion framework of these systems is usually based on graph optimization or filtering methods, which unifies and optimizes the constraints of multiple sensors through factor graphs.

[0026] Through in-depth analysis of the existing technology, this invention has discovered the following key shortcomings of the existing technology: Disadvantage 1: Limited parameter adjustment range and complex coupling. Existing adaptive methods (such as adaptive thresholding in KISS-ICP and fault switching in LVI-SAM) only adjust single or a few parameters, which cannot cover the joint optimization of 20-30 mutually coupled key parameters in a multi-sensor fusion SLAM system (such as LiDAR feature extraction threshold, ICP distance threshold, visual feature matching threshold, reprojection error threshold, IMU noise covariance, wheel velocity meter weight coefficient, RTK confidence threshold, fusion weight coefficient, keyframe selection threshold, etc.). There are complex nonlinear coupling relationships between the parameters, and adjusting a single parameter may lead to a decrease in overall performance.

[0027] Disadvantage 2: The fusion strategy lacks adaptability to heterogeneous degradation modes of multiple sensors. Existing methods (such as fault handover of LVI-SAM and GNSS weight adjustment of GVINS) adopt handover or weighting strategies based on simple threshold judgment, which cannot handle more complex multi-sensor progressive degradation scenarios. For example, in scenarios where vision gradually degrades (light dims) and LiDAR partially degrades (rainy or foggy weather), existing methods struggle to find the optimal fusion strategy.

[0028] Disadvantage 3: The adjustment strategy relies on heuristic rules, resulting in poor scalability. The parameter adjustment rules of existing methods need to be manually designed by domain experts based on experience, and for specific sensor combinations, when adding a new sensor or changing the sensor model, the rules need to be redesigned and debugged, resulting in poor scalability.

[0029] Disadvantage 4: Lack of optimization for long-term cumulative rewards. Existing adaptive methods only consider the fusion effect of the current frame when adjusting parameters, without considering the impact of parameter selection on the positioning accuracy and grounding of subsequent frames. Figure 1 The long-term effects of consistency may lead to suboptimal local decisions.

[0030] Disadvantage 5: Inability to achieve continuous online learning. Once deployed, the existing methods have fixed parameter adjustment rules, making it impossible to continuously optimize and adjust strategies based on actual operating data. Consequently, system performance cannot gradually improve over time.

[0031] Disadvantage 6: Lack of online awareness of multi-sensor time synchronization quality and extrinsic parameter consistency. Existing multi-sensor fusion SLAM systems typically assume that spatiotemporal calibration parameters are known and fixed. In actual operation, synchronization jitter or decreased extrinsic parameter effectiveness can lead to abnormal fusion residuals. Without quality awareness and weight compensation mechanisms, fusion accuracy can easily be reduced.

[0032] To address the shortcomings of the existing technologies, the present invention aims to provide a multi-sensor fusion SLAM parameter adjustment method based on deep reinforcement learning, primarily to achieve the following objectives: (1) Establish a unified deep reinforcement learning framework to achieve joint online optimization of multiple key parameters of the multi-sensor fusion SLAM system, including LiDAR processing parameters, visual processing parameters, IMU processing parameters, wheel speed measurement weights, RTK confidence thresholds, fusion weight coefficients, etc.

[0033] (2) The agent automatically learns the optimal parameter adjustment strategy through a data-driven approach, eliminating the reliance on manually designed heuristic rules. The agent can automatically adjust the fusion strategy based on the current environmental characteristics and the real-time data quality of each sensor.

[0034] (3) Construct a parameter optimization objective that considers long-term cumulative rewards, so that parameter adjustment not only focuses on the fusion accuracy of the current frame, but also on the overall trajectory accuracy and ground accuracy. Figure 1 To the point of being responsive.

[0035] (4) To achieve automatic perception of the degradation level of multiple sensors and adaptive adjustment of the fusion strategy, thereby improving the robustness of the SLAM system in some sensor degradation scenarios. When a sensor degrades, the system can automatically reduce the fusion weight of that sensor and adjust the relevant parameters.

[0036] (5) Supports online continuous learning, enabling the multi-sensor fusion SLAM system to continuously optimize parameters and adjust strategies based on actual operating data after deployment.

[0037] (6) It has the ability to perceive the quality of multi-sensor time synchronization and the consistency of external parameters online. By using reinforcement learning agents to adjust the relevant fusion weights and confidence parameters according to the consistency characteristics, the impact of synchronization jitter or the decrease in the effectiveness of external parameters on the fusion results can be reduced.

[0038] This invention models a multi-sensor fusion SLAM system as a Markov decision process, and uses a deep reinforcement learning agent to learn the optimal parameter adjustment strategy online. (See also...) Figure 1 The multi-sensor fusion SLAM parameter adjustment method includes: S100: Acquire the raw observation data from each sensor and perform preprocessing to generate observation features for each sensor. The preprocessing includes time synchronization, distortion correction, and feature extraction. In this step of the present invention, the system acquires LiDAR point clouds, visual camera images, IMU data, wheel velocity meter pulses, and RTK (Real-Time Kinematic) positioning data in real time through various hardware interfaces. First, time synchronization of multiple sensors is performed (based on hardware triggering or software interpolation), and distortion correction is performed on each sensor. Subsequently, each front-end performs basic data processing, such as edge / plane feature extraction of LiDAR point clouds, corner point extraction and optical flow tracking of visual images, and pre-integration of the IMU. Finally, the initial observation features of each sensor are generated.

[0039] S200. Extract multi-dimensional features characterizing the current SLAM system state, environmental characteristics, and data quality of each sensor from the front and back ends of each sensor, and construct a multi-dimensional state vector. The multi-dimensional state vector includes at least two of the following: LiDAR point cloud statistical features, LiDAR registration quality features, visual feature tracking features, IMU state features, wheel speedometer state features, RTK state features, multi-sensor consistency features, environmental degradation detection features, and historical performance features.

[0040] For example, the present invention designs a high-dimensional state vector s_t The state vector is used to comprehensively characterize the operating state, environmental characteristics, and real-time data quality of the multi-sensor fusion SLAM system at time t. The state vector includes the following components (total dimensions are 52): (1) LiDAR point cloud statistical features (dimension 8): total number of points, effective number of points, number of edge feature points, number of planar feature points, point cloud density (number of points / volume), point cloud distribution entropy, average distance of nearest neighbor points, and point cloud height distribution variance.

[0041] (2) LiDAR registration quality features (dimensional 6): number of ICP iterations (actual use), ICP convergence residual, inlier ratio, registration time, continuity measure of pose change with the previous frame, and local map matching score.

[0042] (3) Visual feature tracking features (dimension 8): number of feature points tracked in the current frame, uniformity of feature point distribution, average optical flow, feature tracking success rate, image gradient richness, image exposure quality score, mean of visual reprojection error, and variance of visual reprojection error.

[0043] (4) IMU state characteristics (dimension 6): acceleration modulus, angular velocity modulus, motion intensity index, IMU pre-integration error, IMU zero bias estimation stability, IMU time synchronization deviation with LiDAR / camera.

[0044] (5) Wheel speed meter status characteristics (dimension 4): linear and angular velocities calculated by the wheel speed meter, consistency measurement of wheel speed meter and IMU speed, wheel slippage detection index, and wheel speed meter data quality score.

[0045] (6) RTK state characteristics (dimension 5): RTK positioning state (fixed solution / floating solution / single point solution), number of visible satellites, PDOP value (position accuracy attenuation factor), RTK positioning covariance, and difference between RTK and SLAM estimated position.

[0046] (7) Multi-sensor consistency features (dimension 6): LiDAR-visual depth consistency score, LiDAR-IMU pose consistency score, visual-wheel velocity consistency score, RTK-SLAM position consistency score, time synchronization deviation measurement between sensors, and fusion residual distribution entropy.

[0047] (8) Environmental degradation detection features (dimension 5): Hessian matrix eigenvalue spectrum (comprehensive observability of multiple sensors), LiDAR degradation factor, visual degradation factor, RTK usability factor, and overall environmental complexity score.

[0048] (9) Historical performance characteristics (dimension 4): average positioning variance, trajectory smoothness measure, map point cloud overlap rate, and loop closure detection success rate of the past N frames.

[0049] The total state dimension is 52-dimensional, which can comprehensively reflect the operating status, data quality, mutual consistency, environmental characteristics and historical performance of each sensor in the multi-sensor fusion SLAM system.

[0050] S300. Input the multi-dimensional state vector into the deep reinforcement learning agent, and output action vectors for key parameters of the multi-sensor fusion SLAM system through the deep reinforcement learning agent. The action vectors include at least two of the following: LiDAR feature extraction threshold, ICP maximum corresponding point distance threshold, visual feature matching threshold, visual reprojection error threshold, LiDAR-visual fusion weight, IMU noise covariance scaling factor, wheel velocity meter fusion weight, RTK availability determination threshold, RTK-SLAM fusion weight, and keyframe insertion threshold.

[0051] For example, in this invention, the 10 key parameters that have the greatest impact on performance in a multi-sensor fusion SLAM system are selected as the output dimensions of the action vector: (1) LiDAR feature extraction threshold a _1 ∈[0.1, 2.0]: Controls the curvature threshold for edge and planar features. In feature-rich environments, the threshold should be increased to reduce computation; in feature-sparse environments, the threshold should be decreased to retain more features.

[0052] (2) ICP maximum corresponding point distance threshold a _2 ∈[0.5, 5.0] (meters): Controls the maximum distance between valid corresponding points in scan-to-map registration. The threshold should be increased to tolerate larger errors when moving at high speeds or when LiDAR is partially degraded.

[0053] (3) Visual feature matching threshold a _3 ∈[5.0, 50.0] (pixels): Controls the size of the matching search window for visual feature tracking. The window should be increased during rapid motion or visual degradation; the window should be decreased in normal environments to improve accuracy.

[0054] (4) Visual reprojection error threshold a _4 ∈[1.0, 10.0] (pixels): Controls the threshold for outlier removal of visual features. The threshold should be lowered when visual data quality is high to preserve accurate features; the threshold should be increased when visual data deteriorates to preserve more features.

[0055] (5) LiDAR-visual fusion weight a _5∈[0.0, 1.0]: Controls the relative weights of LiDAR and vision in the fusion. a should be reduced when LiDAR degrades. _5 To increase visual weight; when visual deterioration occurs, a should be increased. _5 To increase LiDAR weights.

[0056] (6) IMU noise covariance scaling factor a _6 ∈[0.5, 2.0]: Scaling the IMU's preset noise covariance matrix. The scaling factor should be reduced when the IMU data quality is high, and increased when the IMU is subjected to vibration interference.

[0057] (7) Wheel speed measurement fusion weight a _7 ∈[0.0, 1.0]: Controls the weight of the wheel speed meter in the fusion. The weight should be increased on structured flat ground and decreased in scenarios where the wheels may slip.

[0058] (8) RTK availability threshold a _8 ∈[0.0, 1.0]: Controls the minimum quality requirement for RTK measurements to be included in the fusion optimization. Higher weights can be used when the RTK quality is good. When occlusion, multipath, or floating-point solutions cause quality degradation, the availability threshold is increased or the RTK fusion weights are reduced to avoid low-quality RTK constraints polluting the SLAM estimation.

[0059] (9) RTK-SLAM fusion weight a _9 ∈[0.0, 1.0]: Controls the weight of RTK absolute position constraints in fusion. When the RTK quality is good, the weight should be increased to suppress cumulative drift; when the RTK signal is unstable, the weight should be decreased.

[0060] (10) Keyframe insertion threshold a _10 ∈[0.05, 2.0] (dimensionless, determined by the translation threshold t) _kf ∈[0.05, 0.5]m and rotation threshold r _kf (Normalized weighted ∈ [0.5°, 5°]) controls the insertion frequency of new keyframes. The threshold should be lowered to increase keyframe density when the environment changes rapidly or the sensor degrades.

[0061] The 10-dimensional action vector output by the Actor network is scaled and translated to map to the physical value range of each parameter, and a smoothing filter (first-order low-pass filter) is used to avoid the impact of parameter abrupt changes on system stability: a _t_smooth =α×a _t +(1-α)×a _{t-1} , where α∈(0, 1] is the smoothing coefficient.

[0062] S400. Apply the action vector to the multi-sensor fusion SLAM system, and calculate the multi-target reward based on the system operation effect after parameter adjustment. The multi-target reward includes a weighted sum of at least two of the following: positioning accuracy reward, multi-sensor consistency reward, computational efficiency reward, map quality reward, robustness reward, and parameter smoothness reward.

[0063] In this invention, the reward function is crucial for the success of reinforcement learning. For example, this invention designs a multi-objective weighted reward function that comprehensively considers positioning accuracy, multi-sensor fusion consistency, computational efficiency, map quality, and system robustness. r _t =w _1 ×r _accuracy +w _2 ×r _consistency +w _3 ×r _efficiency +w _4 ×r _map +w _5 ×r _robustness +w _6 ×r _smoothness The weight of each reward is defined as follows: (1) Positioning accuracy bonus r _accuracy : Calculation of comprehensive positioning quality based on multi-sensor fusion. _accuracy =-log(1+λ _1 ·NIS _t +λ _2 ·ε _fusion +λ _3 ·trace(Σ _pose )+λ _4 ·ε_abs), where NIS _t To integrate and update the normalized innovation square, ε _fusion For multi-sensor fusion residuals, Σ _pose ε is the covariance matrix for pose estimation; when RTK or external ground truth is reliable, ε _abs This indicates the absolute position constraint error; otherwise, this item is set to zero or weighted less.

[0064] (2) Multi-sensor consistency reward r _consistency Encourage consistency among sensor outputs and penalize abnormal sensor data.

[0065] r _consistency =ξ _1 ×s _lidar_visual +ξ _2 ×s _lidar_imu +ξ _3 ×s _visual_wheel +ξ _4 ×s_rtk_slam +ξ _5 ×s _temporal_sync , where s _* Assign a consistency score (values ​​[0,1], where 1 indicates complete consistency) to each sensor pair. _1 To ξ _5 These are the weighting coefficients.

[0066] (3) Calculation efficiency reward r _efficiency Encourage reducing computational overhead while meeting accuracy requirements. _efficiency =β×max(0, 1-t _process / t _max ), where t _process t is the processing time per frame. _max This is the maximum allowed processing time.

[0067] (4) Map quality bonus r _map To evaluate the consistency of local maps and the quality of point cloud distribution. _map = γ×(map _overlap_ratio -θ _1 ) +δ×point _distribution_score .

[0068] (5) Robust reward r _robustness Penalize fusion failures, tracking loss, processing timeouts, and cases where abnormal sensors are over-trusted; reward continuous localization even with partial sensor degradation.

[0069] r _robustness =ζ·I _success +η·inlier _ratio -μ·I _failure -ν·I _timeout -ρ·w _bad_sensor , among which, I _success For the successful fusion indicator function, I _failure For fusion failure or tracking loss indicator functions, I _timeout To handle timeout indicator functions, w _bad_sensor =Σ_{k∈degraded} max(0, w _k w _k ^th), the sensor k that is judged to be degraded is still assigned an excessively high fusion weight w. _k (w) _k ^th represents the degree of weight reduction threshold.

[0070] (6) Parameter smoothness reward r _smoothness : Penalize drastic changes in parameters and encourage a smooth transition. _smoothness =-κ×||a_t - a _{t-1} ||^2, weighting coefficient w _1 to w _6 It can be adjusted according to the needs of specific application scenarios. For example, in scenarios with high real-time requirements, increase w. _3 In scenarios where the reliability of multiple sensors varies greatly, increase w _2 .

[0071] S500. The deep reinforcement learning agent is driven to perform policy updates and optimizations using the multi-objective reward. In an optional embodiment of the first aspect of the present invention, the deep reinforcement learning agent adopts an algorithm framework based on Twin Delayed Deep Deterministic Policy Gradient (TD3), including an Actor policy network and a dual Critic value network, and introduces a multi-head attention mechanism to handle and learn the complex coupled interaction relationships between multi-sensor state features.

[0072] Specifically, the network architecture used by the deep reinforcement learning agent of this invention can be as follows: Actor Network (Policy Network): The Actor Network receives a 52-dimensional state vector as input and outputs a 10-dimensional continuous action vector. Considering the higher dimensionality of the state space and the complex relationships between sensors in a multi-sensor fusion SLAM system, the network structure adopts a multilayer perceptron (MLP) with appropriate depth and width: Input layer (52-dimensional) → Hidden layer 1 (512 neurons, ReLU activation) → Hidden layer 2 (512 neurons, ReLU activation) → Hidden layer 3 (256 neurons, ReLU activation) → Hidden layer 4 (128 neurons, ReLU activation) → Output layer (10 neurons, Tanh activation). The Tanh activation function restricts the output to the range [-1, 1], and then maps it to the actual physical range of each parameter through a linear transformation. The network weights are initialized with Xavier to accelerate convergence, and batch normalization is added between hidden layers to stabilize the training process.

[0073] Critic Network (Value Network): The Critic Network receives the concatenation of the state vector s and action vector a as input and outputs a scalar Q-value. The network structure is as follows: Input layer (52+10=62 dimensions) → Hidden layer 1 (512 neurons, ReLU activation) → Hidden layer 2 (512 neurons, ReLU activation) → Hidden layer 3 (256 neurons, ReLU activation) → Hidden layer 4 (128 neurons, ReLU activation) → Output layer (1 neuron, linear activation); This invention employs a dual-Critic architecture based on the TD3 algorithm, maintaining two independent Critic networks (Q..._1 and Q _2 During policy updates, the smaller of the two values ​​is taken as the target Q value, effectively alleviating the Q-value overestimation problem: y = r + γ × min(Q _1 (s', a'),Q _2 (s', a')), where, a'~π_target(s') +ε, ε~clip(N(0, σ), -c, c).

[0074] Multi-sensor attention mechanism: To better handle the complex interactions between different sensors in a multi-sensor fusion system, this invention introduces a lightweight multi-head attention module (h=4 heads, d per head) after the input layer of the Actor and Critic networks. _k =13, dropout=0.1), this module divides the 52-dimensional state vector into 8 tokens according to sensor type (the dimension order corresponds to: LiDAR 14-dimensional = point cloud statistics 8 + registration quality 6; vision 8-dimensional; IMU 6-dimensional; wheel speed meter 4-dimensional; RTK 5-dimensional; multi-sensor consistency 6-dimensional; environmental degradation 5-dimensional; historical performance 4-dimensional; total 14+8+6+4+5+6+5+4=52-dimensional), calculates the attention weights between each sensor group, so that the network can adaptively focus on the most important sensor information at present. The output of the attention module is concatenated with the original state and then sent to the subsequent fully connected layer.

[0075] In this invention, when the deep reinforcement learning agent updates its policy, the system will use an experience tuple (s) consisting of the current state, action, reward, next state, and completion flag. _t , a _t , r _t , s _{t+1}The data (, done) is stored in a pre-built experience replay pool. During each network update, the agent randomly samples a micro-batch of experience data from the experience replay pool. To alleviate the problem of Q-value overestimation in traditional algorithms, this invention uses a dual-Critic network architecture of TD3 for value evaluation: First, the target action corresponding to the next state is calculated through the target Actor network, and truncated normal distribution noise (target policy smoothing technique) is added to the action to enhance the network's robustness to parameter perturbations; then, the next state and the noisy target action are input into two target Critic networks, and the smaller value of their outputs is taken. Combined with the reward and discount factor, the target Q-value is calculated using the Bellman equation; finally, the mean square error between the current Q-values ​​output by the two current Critic networks and the target Q-value is calculated, and the weight parameters of the two current Critic networks are updated through backpropagation using the gradient descent method. This process enables the agent to increasingly accurately evaluate the long-term cumulative SLAM performance gains brought by taking a certain set of parameter actions under specific sensor degradation states.

[0076] To ensure training stability and prevent drastic oscillations in the policy network before the value network has converged accurately, a delayed update mechanism is adopted for the Actor policy network (e.g., updating the Actor network only once every two updates to the Critic network). When updating the Actor network, the agent freezes the parameters of the Critic network. The current state is input into the current Actor network to obtain the action output. Subsequently, the current state and the action are input into the first Critic network to output the Q-value. The optimization objective of the Actor network is to maximize this Q-value. Therefore, the gradient of the Q-value with respect to the Actor network weights is calculated using the deterministic policy gradient theorem, and the Actor network weight parameters are updated using gradient ascent. Through this step, the agent automatically adjusts its internal multi-head attention weights and network parameters, making it tend to output a fusion parameter combination that yields higher multi-objective rewards.

[0077] After updating the current Actor and Critic networks, the system employs a soft update strategy, smoothly assigning the parameters of the current network to the corresponding target network θ with a very small update coefficient τ. _target ←τ×θ+ (1-τ) ×θ _target This soft update mechanism avoids sudden changes in the target network, further ensuring the smooth convergence of the multi-sensor fusion parameter adaptive adjustment strategy during online learning and offline fine-tuning.

[0078] In an optional embodiment of the first aspect of the present invention, the training of the deep reinforcement learning agent adopts a two-stage deployment strategy that combines simulation pre-training with online fine-tuning: in the first stage, pre-training is carried out in a multi-sensor degradation simulation environment in combination with a course learning mechanism to obtain basic parameter adjustment strategies; in the second stage, after deployment in a real environment, online fine-tuning is carried out by collecting actual operation experience data to achieve continuous learning.

[0079] Specifically, the training and inference process of the deep reinforcement learning agent described in this invention can be as follows: (1) Pre-training stage in simulation environment: In order to obtain the basic strategy before deployment, this invention first performs pre-training in a high-quality simulation environment: Step 1: Construct a multi-sensor simulation environment based on Gazebo / Isaac Sim, configuring virtual LiDAR (simulating 16 / 32 / 64 / 128 lines), binocular / monocular cameras, IMU, wheel speedometer, and RTK sensor models. The simulation environment includes various typical scenarios (indoor corridors, offices, outdoor city roads, underground tunnels, open squares, semi-obscured streets, etc.), and injects different levels of noise (Gaussian noise, motion distortion, dynamic obstacles, lighting changes, rain and fog effects, RTK occlusion, etc.).

[0080] Step 2: Design a multi-sensor degradation simulation course. The simulation environment supports progressive degradation simulation for single or multiple sensors: visual sensors can simulate dimming, motion blur, and occlusion; LiDAR can simulate rain, fog, noise, and point cloud sparsity; RTK can simulate satellite occlusion and multipath effects; and wheel speedometers can simulate skidding and sideslip.

[0081] Step 3: Use an automated curriculum learning strategy to start training from simple scenarios (all sensors are normal, structured environment) and gradually increase the complexity of the environment (slight degradation of a single sensor → simultaneous degradation of multiple sensors, unstructured environment, high dynamic obstacles).

[0082] Step 4: Run the multi-sensor fusion SLAM system in the simulation. The RL agent observes the 52-dimensional state, outputs 10-dimensional parameter adjustment actions, collects multi-objective rewards, and sets the experience tuple (s) as the result. _t , a _t , r _t , s _{t+1} , done) Store in the experience replay buffer.

[0083] Step 5: Every N steps, sample mini-batch data from the experience replay pool and update the gradients of the Actor and Critic networks.

[0084] Step 6: Regularly use the target network soft update strategy: θ _target ←τ×θ+ (1-τ) × θ _target , where τ is the soft update coefficient (usually taken as 0.005).

[0085] (2) Online fine-tuning stage in real environment: After the policy network pre-trained in the simulation is deployed to the real robot platform, online fine-tuning continues to adapt it to the real environment: Step 1: Load the pre-trained Actor and Critic network weights and initialize the experience replay buffer (retain a certain proportion of the pre-trained data, such as 30%-50%).

[0086] Step 2: In actual operation, the RL agent receives the status of the multi-sensor fusion SLAM system at a fixed frequency (such as 5Hz or 10Hz) and outputs parameter adjustment actions.

[0087] Step 3: Calculate the reward signal using the actual multi-sensor fusion SLAM operation results (such as fusion residual, multi-sensor consistency score, processing time, trajectory smoothness, and RTK constraint satisfaction); among which, RTK-related rewards are only enabled when the RTK quality meets the fixed solution, PDOP, and covariance threshold, otherwise they are downweighted or masked.

[0088] Step 4: Adopt a conservative strategy to update the strategy, reduce the learning rate and limit the range of changes in action output; in the online phase, low-frequency updates are only performed when the quality judgment of each sensor is reliable, the system has not failed and the safe rollback parameters are available.

[0089] Step 5: Set up a safety constraint mechanism so that when the performance of multi-sensor fusion SLAM decreases significantly after parameter adjustment (e.g., fusion residual exceeds the threshold, multi-sensor consistency score is lower than the threshold, or processing time exceeds the frame interval), it automatically reverts to a conservative parameter combination.

[0090] Step 6: Design a sensor degradation safety mode. Statistically detect severe degradation (e.g., visual effective features consistently below a threshold, insufficient LiDAR effective points, or inability to fix RTK for extended periods) using a sliding window, automatically reducing or disabling the sensor's constraints; once the quality recovers and passes hysteresis detection, gradually restore its fusion weights.

[0091] In an optional embodiment of the first aspect of the present invention, the multi-sensor fusion SLAM system is provided with multiple adjustable parameter interfaces, and the deep reinforcement learning agent modifies the parameter values ​​of the multiple adjustable parameter interfaces in real time through parameter service or dynamic reconfiguration mechanism. Each of the adjustable parameter interfaces has safe upper and lower bound constraints, and the parameter adjustment of the deep reinforcement learning agent and the main loop of the multi-sensor fusion SLAM system are executed asynchronously at a fixed frequency. The degradation status of each sensor is monitored in real time. When degradation of any sensor is detected by the sliding window statistics, the deep reinforcement learning agent automatically reduces or blocks the fusion weight of the sensor and gradually restores it after the quality is restored. When the multi-sensor fusion SLAM system detects tracking loss or fusion failure, it automatically switches to a conservative mode using preset safety parameters.

[0092] Specifically, this invention also designs a loosely coupled integration scheme with existing multi-sensor fusion SLAM systems, using R3LIVE / FAST-LIVO / LVI-SAM as an example to illustrate the integration method: (1) Data Flow: After processing each frame of data, the front end of the multi-sensor fusion SLAM system publishes the processing features of each sensor (LiDAR registration quality, visual tracking quality, IMU pre-integration error, wheel speedometer consistency, RTK state) and multi-sensor consistency features to the ROS topic / shared memory area. The RL (Reinforcement Learning) agent subscribes to these topics to obtain a 52-dimensional state vector.

[0093] (2) Parameter Interface: Ten adjustable parameter interfaces are set in the parameter server of the SLAM system. The RL agent modifies the parameter values ​​in real time through parameter service or dynamic reconfiguration mechanism. Each parameter interface has safe upper and lower bound constraints to prevent parameters from going out of bounds.

[0094] (3) Synchronization mechanism: The parameter adjustment of the RL agent is performed asynchronously at a fixed frequency (such as 5Hz or 10Hz) and runs asynchronously with the main loop of the SLAM system to avoid introducing additional delay. The parameter adjustment is thread-safe through atomic operations or mutex locks.

[0095] (4) Multi-sensor degradation detection and response: The state observation module monitors the degradation indicators of each sensor in real time. When degradation of a sensor is detected, the RL agent automatically adjusts the fusion weight parameters (such as a). _5 , a _7 , a _9 When sensor recovery is detected, the fusion weight of that sensor is gradually increased. Degradation detection uses a sliding window statistical method to avoid misjudgments caused by transient noise.

[0096] (5) RTK constraint management: When the RTK status is good (fixed solution, PDOP < 3, number of visible satellites > 8), increase a _8 and a _9 To fully utilize the high-precision constraints of RTK; when the RTK state decreases (floating-point solution, PDOP>5), reduce a. _8 and a _9To mitigate the impact of RTK noise on fusion; when RTK completely fails, a _9 When the value drops to 0, the system degenerates into a LiDAR-Visual-IMU-Wheel fusion.

[0097] (6) Anomaly handling: When the SLAM system detects tracking loss or fusion failure, the RL agent automatically switches to conservative mode (using preset safety parameters), and the learned strategy is activated after the system returns to normal. At the same time, the abnormal event is recorded for subsequent offline analysis and strategy improvement.

[0098] To better illustrate the technical effectiveness of the multi-sensor fusion SLAM parameter adjustment method of this invention, the positioning accuracy of the method of this invention and the traditional method were tested in the following scenarios. The test results are shown in Table 1 below (ATE (Absolute Trajectory Error) is the RMSE value of the absolute trajectory error; the traditional method, the degradation scenario settings, and the improvement should all be recorded under the same hardware, the same data partitioning, and the same evaluation tool): Table 1. Comparison of positioning accuracy between the method of the present invention and the traditional method in different scenarios.

[0099] In addition, sensor degradation experiments were conducted to verify the performance of the invention in multi-sensor degradation scenarios: Experiment 1: Progressive Degradation Experiment. In simulated or real-world environments, the progressive degradation of the visual sensor was simulated by gradually reducing illumination intensity, increasing motion blur, and decreasing texture richness. The positioning accuracy change curves of the present invention and LVI-SAM were recorded to verify the adaptive adjustment capability of the present invention.

[0100] Experiment 2: LiDAR Rain and Fog Degradation Experiment. Different intensities of rain and fog noise were injected into the simulation environment (simulating point cloud sparsification and loss of distant points), or tested under real rain and fog conditions to compare the localization robustness of the present invention and the comparative method.

[0101] Experiment 3: RTK Obstruction Experiment. The experiment started in an open environment (RTK functioning normally), gradually moving into an area obstructed by tall buildings (RTK signal quality deteriorated until it completely failed), verifying the drift suppression capability of this invention under RTK constraint loss conditions.

[0102] Experiment 4: Simultaneous Degradation Experiment of Multiple Sensors. This experiment simulates extreme scenarios where two or three sensors degrade simultaneously (e.g., entering a tunnel: RTK failure + LiDAR degradation due to dust + visual degradation due to insufficient lighting) to verify the basic positioning maintenance capability of this invention when only the wheel speedometer and IMU remain functional.

[0103] Experiment 5: Sensor Recovery Experiment. After the degradation experiment, each sensor was gradually restored to normal operation to verify that the present invention can detect sensor recovery in a timely manner and gradually improve its fusion weight.

[0104] Of course, in other embodiments of the present invention, the technical effects of the present invention can also be objectively verified through the following indicators: (1) Relative pose error (RPE): assesses the consistency and smoothness of local trajectories.

[0105] (2) Multi-sensor consistency index: LiDAR-visual depth consistency score, LiDAR-IMU pose consistency score, and distribution of fusion residuals of each sensor.

[0106] (3) Real-time performance indicators: single frame processing time, CPU / GPU utilization, and time consumption analysis of each module.

[0107] (4) Robustness indicators: tracking loss rate, loop closure detection success rate, positioning retention capability under various sensor degradation scenarios, and drift rate after RTK failure.

[0108] (5) Parameter adjustment effect: smoothness of parameter changes, rationality analysis of parameter selection in different scenarios, and impact of parameter adjustment on fusion consistency.

[0109] In summary, the key technical points of the multi-sensor fusion SLAM parameter adjustment method of this invention include at least the following: (1) Multi-sensor integrated state space design: The operating status of the multi-sensor fusion SLAM system, the data quality of each sensor, the consistency between sensors, environmental characteristics and historical performance are comprehensively encoded into a 52-dimensional state vector to fully reflect the operating status of the multi-sensor fusion system.

[0110] (2) Multi-sensor joint parameter action space: Ten key parameters that have the greatest impact on the performance of multi-sensor fusion SLAM are selected to form a joint action space, which covers LiDAR processing parameters, vision processing parameters, IMU processing parameters, wheel speed meter weight, RTK confidence and fusion weight coefficient, to achieve collaborative optimization of multi-sensor parameters.

[0111] (3) Multi-objective weighted reward function design: The reward function design takes into account positioning accuracy, multi-sensor fusion consistency, computational efficiency, map quality, robustness and parameter smoothness. In particular, multi-sensor consistency reward is introduced to encourage the coordination and consistency of the output of each sensor.

[0112] (4) TD3-based deep reinforcement learning framework: adopts dual Critic architecture and target policy smoothing technology, combined with multi-head attention mechanism to handle complex interaction relationships between multiple sensors, ensuring stable learning in high-dimensional continuous parameter space.

[0113] (5) Deployment strategy of simulation pre-training + online fine-tuning: the basic strategy is obtained in the simulation through multi-sensor degradation simulation and course learning, and then continuously optimized in the real environment through conservative updates.

[0114] (6) Multi-sensor degradation detection and adaptive fusion: Real-time monitoring of the degradation status of each sensor, automatic adjustment of fusion weights and parameters, and support for sensor degradation operation mode.

[0115] (7) Safety constraint mechanism: Set safety boundaries for parameter adjustment, performance rollback mechanism and sensor degradation safety mode to ensure the stable operation of multi-sensor fusion SLAM system during online learning.

[0116] Based on the above technical points, the multi-sensor fusion SLAM parameter adjustment method of the present invention has at least the following significant advantages: (1) Multi-sensor parameter adaptive optimization that reduces the workload of manual parameter tuning: By learning multi-sensor parameter adjustment strategies through deep reinforcement learning, the workload of repeated manual parameter tuning for different sensor combinations and scenarios is reduced while retaining safe upper and lower bounds and initial conservative parameters.

[0117] (2) Joint optimization of multi-sensor parameters with clear synergistic relationship: This invention achieves joint adjustment of 10 key parameters and can learn the complementary relationship between LiDAR, vision, IMU, wheel speed meter and RTK; compared with single sensor weight switching, it has a verifiable basis for further reducing error and failure rate in multiple degradation scenarios.

[0118] (3) Data-driven fusion strategy learning with strong generalization ability: The parameter adjustment strategy of this invention learns directly from the data and does not rely on manually designed heuristic rules, which has a stronger adaptability to new sensor models and new environments. New sensors can be quickly adapted by simply expanding the state space and action space.

[0119] (4) Multi-sensor consistency optimization and improved fusion robustness: By introducing multi-sensor consistency rewards, this invention not only focuses on the performance of a single sensor, but also emphasizes the coordination and consistency of the outputs of each sensor, effectively avoiding the contamination of the fusion results by low-quality sensor data.

[0120] (5) Sensor degradation adaptive, enhancing system reliability: This invention can detect the degradation status of each sensor in real time and automatically adjust the fusion strategy, maintaining high positioning accuracy even in scenarios where one or more sensors are degraded. Compared to the simple fault-switching mechanism of LVI-SAM, the progressive fusion adjustment strategy of this invention performs better in some degradation scenarios.

[0121] (6) Long-term reward optimization, better global performance: Through the discount cumulative reward mechanism of reinforcement learning, the parameter adjustment of the present invention takes into account the long-term impact and avoids short-sighted local optima, which is particularly suitable for long-distance navigation tasks.

[0122] (7) Constrained online fine-tuning and offline incremental updates: The present invention can improve the strategy using runtime data after deployment, but online updates must be constrained by reward credibility, security boundaries and rollback mechanisms; high-risk updates can be transferred to offline incremental training and then deployed.

[0123] (8) The computational overhead is controllable and does not affect real-time performance: the forward propagation of the Actor network only takes about 1-2ms (on the NVIDIA Jetson AGX Orin platform), and parameter adjustment is performed asynchronously at a low frequency (5-10Hz), which hardly increases the computational burden of the multi-sensor fusion SLAM system.

[0124] (9) Loosely coupled with existing multi-sensor fusion SLAM systems and easy to integrate: As a parameter adjustment module, this invention can be easily integrated into mainstream multi-sensor fusion SLAM systems such as R3LIVE, FAST-LIVO, LVI-SAM, and GVINS without making significant modifications to the original system.

[0125] Based on the above advantages, the multi-sensor fusion SLAM parameter adjustment method of the present invention can be applied to the following typical application scenarios: (1) Indoor service robot: Operating in structured indoor environments (corridors, lobbies, offices), vision and LiDAR usually function normally, and wheel velocity meter provides stable constraints. When the robot passes through visually degraded areas such as glass curtain walls and large white walls, this invention automatically increases the LiDAR weight and wheel velocity meter weight to maintain stable positioning.

[0126] (2) Autonomous vehicles: When driving in different scenarios such as urban roads, highways, and underground parking garages, RTK provides high-precision constraints on open roads, while LiDAR and vision perceive the surrounding environment. When entering a tunnel (RTK fails, and LiDAR may degrade due to dust), this invention automatically reduces the weight of RTK and increases the weight of vision and wheel speed measurement.

[0127] (3) Outdoor power inspection robot: In substations, transmission lines and other scenarios, it faces challenges such as strong electromagnetic interference (affecting IMU and RTK), drastic changes in lighting (affecting vision), and interference from metal structures with LiDAR. This invention can dynamically adjust the fusion strategy according to the real-time sensor status to ensure reliable positioning in various harsh environments.

[0128] (4) Unmanned Aerial Vehicles / Quadripods: On platforms with rapid movement or significant vibration, the IMU may saturate, vision may degrade due to motion blur, and LiDAR may be distorted due to vibration. This invention automatically detects the degradation state of each sensor and dynamically adjusts the fusion weights and parameters to maintain robust localization.

[0129] (5) Underground / Tunnel Detection: In feature-sparse and degraded environments, the available features of LiDAR decrease, RTK signals are blocked, and vision degrades due to insufficient illumination. This invention retains more features and maintains basic positioning capabilities by increasing the weight of the wheel speed meter and adjusting LiDAR and vision parameters.

[0130] It should also be noted that, for the technical solution of this invention, there are the following alternative implementations, all of which should fall within the protection scope of this invention: Alternative Solution 1: Discrete Parameter Selection Based on PPO: Discretize the continuous action space into several parameter combination levels (e.g., each parameter takes 3-5 discrete values), and use the PPO (Proximal Policy Optimization) algorithm for policy learning. The advantages of this approach are higher training stability, easier engineering implementation and debugging of the discrete action space, and natural suitability for scenarios involving parameter level switching. The disadvantages are that discretization sacrifices the finesse of parameter adjustment, and the high-dimensional discrete action space (10 parameters, 5 levels each, totaling 9,765,625 combinations) faces the curse of dimensionality. This can be mitigated by combining a hierarchical strategy (the upper layer selects a subset of parameters, and the lower layer selects specific levels).

[0131] Alternative Solution 2: SAC-based Entropy Maximization Strategy: Replacing TD3 with the SAC (Soft Actor-Critic) algorithm, this approach encourages exploration by maximizing policy entropy, potentially leading to more creative parameter combinations in multi-sensor fusion parameter tuning tasks. SAC's automatic temperature regulation mechanism also simplifies hyperparameter tuning. SAC excels in handling multimodal optimal strategies and is suitable for complex scenarios involving multiple equivalent fusion strategies in multi-sensor fusion.

[0132] Alternative Option 3: Parameter Optimization Based on Model Predictive Control (MPC): This approach utilizes the learned system dynamic model to perform finite-time rolling optimization (MPC) at each time step, instead of directly outputting fixed parameters. This method can explicitly consider the effects of parameter adjustments over multiple future steps, allowing for proactive adjustments to the fusion strategy in multi-sensor progressive degradation scenarios. However, it incurs significant computational overhead and may require GPU acceleration to meet real-time requirements.

[0133] Alternative Solution 4: Rapid Adaptation Based on Meta-Learning: Employing meta-learning algorithms such as MAML (Model-Agnostic Meta-Learning), meta-training is performed in simulation environments with multiple different sensor configurations and scenarios. This enables the agent to quickly adapt to new sensor configurations or environments with only a small number of gradient updates. This approach is particularly suitable for applications requiring frequent sensor changes or rapid switching between various heterogeneous environments.

[0134] Alternative Option 5: Hierarchical Reinforcement Learning Architecture: This approach employs a hierarchical reinforcement learning (RL) architecture. The upper-layer policy is responsible for identifying the current environment type (indoor / outdoor / degraded scene, etc.) and sensor status (normal / mildly degraded / severely degraded sensors), selecting the corresponding baseline parameter template. The lower-layer policy is responsible for fine-tuning around the baseline template. This approach reduces the learning difficulty, improves the interpretability of the policy, and facilitates the integration of domain expert knowledge.

[0135] Alternative Solution Six: Sensor Relationship Modeling Based on Graph Neural Networks: This approach explicitly models the topological relationships and interactions between multiple sensors using graph neural networks (GNNs). Each sensor is treated as a graph node, and the constraints between sensors are treated as edges. A graph attention mechanism is used to learn the dynamic importance weights between sensors. Compared to multi-head attention, this approach is more suitable for scenarios with fixed sensor topologies and offers greater interpretability.

[0136] See Figure 2 The second aspect of the present invention provides a multi-sensor fusion SLAM parameter adjustment system, the SLAM parameter adjustment system comprising: The multi-sensor data preprocessing module 10 is used to acquire the raw observation data of each sensor and perform preprocessing to generate the observation features of each sensor. The preprocessing includes time synchronization, distortion correction, and feature extraction. Specifically, it is responsible for receiving lidar point clouds, visual images, IMU data, wheel velocity meter pulses, and RTK positioning data, performing preprocessing operations such as time synchronization, distortion correction, and feature extraction to generate the observation features of each sensor.

[0137] The multi-sensor state observation module 20 is used to extract multi-dimensional features representing the current SLAM system state, environmental characteristics, and data quality of each sensor from the front end and back end of each sensor, and construct a multi-dimensional state vector. Specifically, it is responsible for extracting multi-dimensional feature vectors representing the current system state, environmental characteristics, and data quality of each sensor from the front end (point cloud preprocessing and feature extraction, visual feature tracking, IMU pre-integration, wheel speed measurement and odometer, RTK calculation) and back end (multi-sensor factor graph optimization, loop closure detection) of each sensor.

[0138] The deep reinforcement learning agent module 30 is used to input the multi-dimensional state vector into the deep reinforcement learning agent, and output the action vector for the key parameters of the multi-sensor fusion SLAM system through the deep reinforcement learning agent. Specifically, the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm is used as the basic learning framework, which includes an Actor network (policy network) and a Critic network (value network), which are responsible for adjusting the action according to the optimal multi-sensor fusion SLAM parameters output based on the observation state.

[0139] The parameter execution and effect evaluation module 40 is used to apply the action vector to the multi-sensor fusion SLAM system and calculate multi-objective rewards based on the system operation effect after parameter adjustment; specifically, it is used to apply the parameter adjustment action output by the agent to the multi-sensor fusion SLAM system and calculate reward signals based on the system operation effect after parameter adjustment.

[0140] The experience replay and policy update module 50 is used to drive the deep reinforcement learning agent to update and optimize its policies using the multi-objective reward. Specifically, it stores interaction experiences and performs offline policy updates to achieve stable and efficient deep learning training.

[0141] The data flow of the system of this invention can be as follows: After preprocessing, the raw data from multiple sensors are input to the front-end processing modules of each sensor respectively; the output features of the front-end processing modules are input to the multi-sensor state observation module to generate a comprehensive state vector; the RL agent receives the state vector and outputs parameter adjustment actions; after the parameter adjustment actions are checked by the safety constraint module, they are applied to each sensor processing module and the fusion backend through the parameter interface; the running results of the fusion backend (such as positioning accuracy, fusion consistency, processing time, etc.) are used to calculate the reward signal and drive the agent's policy update.

[0142] In an optional embodiment of the second aspect of the present invention, the multi-dimensional state vector includes at least two of the following: LiDAR point cloud statistical features, LiDAR registration quality features, visual feature tracking features, IMU state features, wheel speedometer state features, RTK state features, multi-sensor consistency features, environmental degradation detection features, and historical performance features.

[0143] In an optional embodiment of the second aspect of the present invention, the action vector includes at least two of the following: LiDAR feature extraction threshold, ICP maximum corresponding point distance threshold, visual feature matching threshold, visual reprojection error threshold, LiDAR-visual fusion weight, IMU noise covariance scaling factor, wheel speed meter fusion weight, RTK availability determination threshold, RTK-SLAM fusion weight, and keyframe insertion threshold.

[0144] In an optional embodiment of the second aspect of the present invention, the multi-target reward includes a weighted sum of at least two of the following: positioning accuracy reward, multi-sensor consistency reward, computational efficiency reward, map quality reward, robustness reward, and parameter smoothness reward.

[0145] In an optional embodiment of the second aspect of the present invention, the deep reinforcement learning agent adopts an algorithm framework based on dual-delay deep deterministic policy gradient, including an Actor policy network and a dual Critic value network, and introduces a multi-head attention mechanism to handle and learn the complex coupled interaction relationships between multi-sensor state features.

[0146] In an optional embodiment of the second aspect of the present invention, the training of the deep reinforcement learning agent adopts a two-stage deployment strategy that combines simulation pre-training with online fine-tuning: the first stage is to perform pre-training in a multi-sensor degradation simulation environment in conjunction with a course learning mechanism to obtain basic parameter adjustment strategies; the second stage is to perform online fine-tuning by collecting actual operation experience data after deployment in a real environment to achieve continuous learning.

[0147] In an optional embodiment of the second aspect of the present invention, the multi-sensor fusion SLAM system is provided with multiple adjustable parameter interfaces. The deep reinforcement learning agent modifies the parameter values ​​of the multiple adjustable parameter interfaces in real time through parameter service or dynamic reconfiguration mechanism. Each adjustable parameter interface has safety upper and lower bound constraints. The parameter adjustment of the deep reinforcement learning agent and the main loop of the multi-sensor fusion SLAM system are executed asynchronously at a fixed frequency. The degradation status of each sensor is monitored in real time. When degradation of any sensor is detected by sliding window statistics, the deep reinforcement learning agent automatically reduces or blocks the fusion weight of the sensor and gradually restores it after the quality is restored. When the multi-sensor fusion SLAM system detects tracking loss or fusion failure, it automatically switches to a conservative mode using preset safety parameters.

[0148] Figure 3This is a schematic diagram of a navigation device provided in an embodiment of the present invention. The navigation device can vary significantly due to differences in configuration or performance, and may include one or more central processing units (CPUs) 60 (e.g., one or more processors) and a memory 70, and one or more storage media 80 (e.g., one or more mass storage devices) for storing recording programs or data. The memory and storage media can be short-term or persistent storage. The program stored in the storage media may include one or more modules (not shown in the figure), each module including a series of instruction operations on the navigation device. Furthermore, the processor may be configured to communicate with the storage media and execute the series of instruction operations stored in the storage media on the navigation device.

[0149] The navigation device of this invention may further include one or more power supplies 90, one or more wired or wireless network interfaces 100, one or more input / output interfaces 110, and / or one or more operating systems, such as Windows Server, Mac OS, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The navigation device structure shown does not constitute a limitation on the navigation device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0150] This invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the multi-sensor fusion SLAM parameter adjustment method.

[0151] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system, module, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0152] If the integrated unit is implemented as a software functional unit or a combination of software and hardware and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0153] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for adjusting SLAM parameters through multi-sensor fusion, characterized in that, include: The raw observation data of each sensor is acquired and preprocessed to generate the observation features of each sensor. The preprocessing includes time synchronization, distortion removal and feature extraction. Multi-dimensional features characterizing the current SLAM system state, environmental characteristics, and data quality of each sensor are extracted from the front and back ends of each sensor to construct a multi-dimensional state vector. The multi-dimensional state vector is input into the deep reinforcement learning agent, and the deep reinforcement learning agent outputs action vectors for key parameters of the multi-sensor fusion SLAM system. The motion vectors are applied to the multi-sensor fusion SLAM system, and multi-objective rewards are calculated based on the system's performance after parameter adjustment. The multi-objective reward is used to drive the deep reinforcement learning agent to update and optimize its policies.

2. The SLAM parameter adjustment method for multi-sensor fusion according to claim 1, characterized in that, The multi-dimensional state vector includes at least two of the following: LiDAR point cloud statistical features, LiDAR registration quality features, visual feature tracking features, IMU state features, wheel speedometer state features, RTK state features, multi-sensor consistency features, environmental degradation detection features, and historical performance features.

3. The SLAM parameter adjustment method for multi-sensor fusion according to claim 1, characterized in that, The action vector includes at least two of the following: LiDAR feature extraction threshold, ICP maximum corresponding point distance threshold, visual feature matching threshold, visual reprojection error threshold, LiDAR-visual fusion weight, IMU noise covariance scaling factor, wheel speed meter fusion weight, RTK availability determination threshold, RTK-SLAM fusion weight, and keyframe insertion threshold.

4. The SLAM parameter adjustment method for multi-sensor fusion according to claim 1, characterized in that, The multi-target reward includes a weighted sum of at least two of the following: positioning accuracy reward, multi-sensor consistency reward, computational efficiency reward, map quality reward, robustness reward, and parameter smoothness reward.

5. The SLAM parameter adjustment method for multi-sensor fusion according to claim 1, characterized in that, The deep reinforcement learning agent adopts an algorithm framework based on dual-delay deep deterministic policy gradient, including an Actor policy network and a dual Critic value network, and introduces a multi-head attention mechanism to handle and learn the complex coupled interaction relationships between multi-sensor state features.

6. The SLAM parameter adjustment method for multi-sensor fusion according to claim 1, characterized in that, The training of the deep reinforcement learning agent adopts a two-stage deployment strategy that combines simulation pre-training with online fine-tuning: the first stage is to pre-train in a multi-sensor degradation simulation environment in combination with a course learning mechanism to obtain basic parameter adjustment strategies. The second phase involves online fine-tuning after deployment in a real-world environment by collecting data from actual operational experience, enabling continuous learning.

7. The SLAM parameter adjustment method for multi-sensor fusion according to claim 1, characterized in that, The multi-sensor fusion SLAM system is equipped with multiple adjustable parameter interfaces. The deep reinforcement learning agent modifies the parameter values ​​of the multiple adjustable parameter interfaces in real time through parameter service or dynamic reconfiguration mechanism. Each adjustable parameter interface has safe upper and lower bound constraints. The parameter adjustment of the deep reinforcement learning agent and the main loop of the multi-sensor fusion SLAM system are executed asynchronously at a fixed frequency. The degradation status of each sensor is monitored in real time. When degradation of any sensor is detected by the sliding window statistics, the deep reinforcement learning agent automatically reduces or blocks the fusion weight of the sensor and gradually restores it after the quality is restored. When the multi-sensor fusion SLAM system detects tracking loss or fusion failure, it automatically switches to a conservative mode using preset safety parameters.

8. A multi-sensor fusion SLAM parameter adjustment system, characterized in that, The SLAM parameter adjustment system includes: A multi-sensor data preprocessing module is used to acquire the raw observation data of each sensor and preprocess it to generate the observation features of each sensor. The preprocessing includes time synchronization, distortion removal and feature extraction. The multi-sensor state observation module is used to extract multi-dimensional features from the front and back ends of each sensor to characterize the current SLAM system state, environmental characteristics, and data quality of each sensor, and to construct a multi-dimensional state vector. The deep reinforcement learning agent module is used to input the multi-dimensional state vector into the deep reinforcement learning agent, and output the action vector for the key parameters of the multi-sensor fusion SLAM system through the deep reinforcement learning agent. The parameter execution and effect evaluation module is used to apply the action vector to the multi-sensor fusion SLAM system and calculate multi-objective rewards based on the system operation effect after parameter adjustment. The experience replay and policy update module is used to drive the deep reinforcement learning agent to update and optimize its policies using the multi-objective reward.

9. A navigation device, characterized in that, The navigation device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the navigation device to execute the multi-sensor fusion SLAM parameter adjustment method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the SLAM parameter adjustment method for multi-sensor fusion as described in any one of claims 1-7.

Images