Reinforcement learning adaptive multimodal SLAM method based on 4D Gaussian splashing

Abstract

The present application describes a reinforcement learning adaptive multimodal SLAM method based on 4D Gaussian splattering, which belongs to the field of simultaneous positioning and map construction. The present application proposes an overall solution that further combines 4D GS and reinforcement learning on the basis of the existing multi-sensor fusion SLAM system. Reinforcement learning is used to autonomously select sensor modes based on real-time environmental information. 4D GS is used to update and optimize the map and integrate it into the SLAM system, thereby achieving the purpose of improving the working efficiency, real-time performance, and resource saving of the entire real-time multimodal SLAM system. The method includes the following steps: step (1), multi-source sensor input and data preprocessing; step (2), selection of sensor combination type; step (3), multi-source sensor data fusion processing; and step (4), map update and optimization.

Description

Technical Field

[0001] The present application relates to the field of simultaneous localization and mapping, and specifically proposes an adaptive multimodal online SLAM method based on reinforcement learning and 4D Gaussian Splatting (4D GS) for dynamic environments. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) utilizes data from multiple sensors (such as cameras, LiDAR, and IMUs) to estimate sensor poses and simultaneously reconstruct a 3D map of the surrounding environment. SLAM technology can solve the problem of autonomous localization and navigation for robots or devices in unknown environments and is a key technology for achieving fully autonomous mobile robots.

[0003] Existing 3D Gaussian splattering is a technology used for 3D scene reconstruction and rendering. It uses Gaussian functions to represent points or surfaces in the scene and processes them in 3D space, but 3D GS focuses on static scenes. It can be extended to dynamic scenes as a 4D representation. The key technical issue is how to model complex point motion from sparse input. 4D Gaussian splattering can be seen as an extension of 3D Gaussian splattering, which adds a fourth dimension to the 3D space and is typically used to process dynamic scenes or data that changes over time. 4D Gaussian splattering has advantages when processing dynamic content, such as modeling and rendering of dynamic objects, and scene reconstruction under time-varying lighting conditions.

[0004] SLAM technology has so far developed into a SLAM system that integrates laser, vision, and IMU sensors. For example, the following previously published Chinese patent application, application number CN202410154510.8, entitled "A dense visual SLAM method and system using three-dimensional Gaussian backend representation," proposes a dense visual SLAM method using three-dimensional Gaussian backend representation. By constructing a three-dimensional Gaussian scene representation, adaptive three-dimensional Gaussian expansion is performed to reconstruct the scene geometry, and based on the reconstructed scene geometry, coarse-to-fine camera differentiable pose estimation is performed. It faces challenges when processing dynamic objects or rapidly changing scenes, and the computational cost is relatively high, which limits its application in resource-constrained environments.

[0005] Another example is the application number CN202411103239.1, entitled "A Laser-Enhanced Vision 3D Reconstruction Method and System Based on Gaussian Splashing." It is divided into two parts: pose calculation and scene reconstruction. During pose calculation, image feature matching is used to solve relative rotation, initialize the point cloud ICP pose and optimize the relative pose. During scene reconstruction, a 3D Gaussian sphere is used to represent the scene. The depth map generated by the high-precision pose image and point cloud projection is used to supervise the synthetic image of the differentiable rendering, and the parameters of the 3D Gaussian sphere are optimized to achieve scene reconstruction. The reliance on LiDAR data may affect the reconstruction effect when the LiDAR data is inaccurate or blocked, and the data fusion process faces problems such as data inconsistency and calibration errors. The overall complexity of the algorithm is high, and the computing resource requirements are also high.

[0006] Another example is application number CN202410277176.5, titled "A Method and System for Reconstructing Dense Gaussian Maps in Dynamic Environments." It uses instance segmentation to divide potential moving objects, obtaining a dynamic prior region. A dynamic point detection algorithm is used to detect and remove dynamic feature points within the segmented region. Camera pose estimation information is obtained based on static feature points. The system combines a visual SLAM system with a 3D Gaussian Splatting framework to reconstruct a dense map of the static region. The definition and detection of dynamic objects relies on instance segmentation and preset dynamic prior regions, making it unsuitable for all types of dynamic scenes, especially for scenes with rapid or complex dynamic changes. Furthermore, the system's reliance on optical flow and epipolar geometry results in inaccurate dynamic point detection in some cases.

[0007] The aforementioned prior art multimodal SLAM technologies suffer from the following major shortcomings and drawbacks: First, dynamic objects in the scene can cause SLAM systems to experience tracking loss and relocalization failures; in other words, they face severe degradation when operating in dynamic, complex, and changing environments. Pose estimation and map construction are particularly poor in low-texture areas, under limited viewing angle constraints, and in dynamic environments. Dynamic objects can cause the SLAM system to experience large trajectory drift, loop closure failures, and ghosting in the map. Second, the widespread use of fixed rules such as weighted fusion and static priority makes it difficult to cope with sudden environmental changes such as sensor occlusion or sudden illumination changes. Furthermore, the simultaneous operation of multiple sensors leads to redundant computations, increasing power consumption and latency on mobile devices. Third, existing NeRF methods incur significant training and rendering costs, resulting in significant latency during the rendering process. Fourth, the scene representation as a 3D Gaussian is unable to meet the real-time rendering speed requirements of dynamic scenes.

[0008] In view of this, this application is hereby filed. Summary of the Invention

[0009] The reinforcement learning adaptive multimodal SLAM method based on 4D Gaussian splattering described in this application aims to solve the problems existing in the above-mentioned prior art and proposes an overall solution that further combines 4D GS and reinforcement learning on the basis of the existing multi-sensor fusion SLAM system. Through reinforcement learning, the sensor mode is autonomously selected according to real-time environmental information. At the same time, 4D GS is used to update and optimize the map and integrate it into the SLAM system, thereby achieving the purpose of improving the working efficiency, real-time performance and resource saving of the entire real-time multimodal SLAM system.

[0010] To achieve the above design objectives, the reinforcement learning adaptive multimodal SLAM method based on 4D Gaussian splashing includes the following steps:

[0011] Step 1: Multi-source sensor input and data preprocessing;

[0012] Receive point cloud data from the lidar, RGB image data from the depth camera, and IMU data, and perform data preprocessing;

[0013] Step 2. Select the sensor combination type;

[0014] Based on the reinforcement learning sensor selection module, the reinforcement learning strategy network learns the environment status in real time, dynamically selects the most reliable sensor combination type and activates sensors on demand;

[0015] Step 3: Multi-source sensor data fusion processing;

[0016] Based on the data fusion processing module, the sensor combination performs multi-source sensor data fusion processing;

[0017] Step 4: Map update and optimization;

[0018] Based on the map update and back-end optimization module, the environment map is updated and optimized using the map data provided by the front-end data fusion processing module in combination with 4D GS, and error correction and map optimization are performed through loop closure detection.

[0019] Furthermore, the step 1 includes the following steps:

[0020] Step 1.1 Data input;

[0021] The SLAM system synchronizes the point cloud data received from the lidar, the RGB image data from the depth camera, and the IMU data in time;

[0022] Step 1.2: Incorporate time information;

[0023] Introducing timestamps during data preprocessing ,Each lidar point cloud and image frame is timestamped;

[0024] Step 1.3 Multi-sensor external parameter calibration;

[0025] Use the Kalibr framework to calibrate the external parameters between sensors;

[0026] Step 1.4: Initialize the reinforcement learning module.

[0027] The reinforcement learning-based sensor selection module is initialized in the data preprocessing stage. By analyzing historical data and environmental characteristics, it learns the strategy of selecting the optimal sensor combination under different scenario conditions. In the initial state, a random exploration strategy is adopted.

[0028] Furthermore, the step 2 includes the following steps:

[0029] Step 2.1 State space design;

[0030] Define the environmental information that the reinforcement learning system can observe at each decision moment;

[0031] Step 2.2 defines the action space formula;

[0032] This includes defining actions for the system to switch sensor modes to optimize data acquisition and system performance;

[0033] The SLAM system uses a hybrid discrete-continuous space to achieve refined control. The mode selection part is discrete and includes three sensor-dominated modes: depth camera-IMU mode, lidar-depth camera mode, and lidar-IMU-depth camera mode.

[0034] The weight distribution part is continuous and includes sensor gain coefficients and resource constraint parameters. The sensor gain coefficients are the weights of the lidar, depth camera, and IMU, ranging from 0 to 1. The expression is as follows:

[0035] (9)

[0036] The resource limiting parameter is the lidar sampling rate, which ranges from 5Hz to 40Hz; the expression is as follows:

[0037] (10)

[0038] Step 2.3 Reward function module;

[0039] Contains main rewards, auxiliary rewards and penalties;

[0040] Step 2.4 Network architecture design;

[0041] Including the state encoder, as the foundation of the entire network, which is responsible for converting multimodal sensor data into a unified feature representation so that the subsequent policy network and value network can make decisions and evaluations based on these features;

[0042] The policy network is responsible for outputting specific action selections based on the encoded state features, including discrete action patterns and continuous parameter adjustments;

[0043] The value network is used to evaluate the pros and cons of the current state and action combination, providing learning signals for the policy network to help it optimize the decision-making process;

[0044] Step 2.5 executes the training strategy;

[0045] Through efficient data collection mechanism, intelligent exploration strategy and strict safety learning constraints, a comprehensive training framework is provided for the reinforcement learning sensor adaptive selection module; including,

[0046] The data collection mechanism provides the system with experience data of interactions with the environment, which is the basis for learning and optimizing strategies;

[0047] Exploration strategy, which determines how the system effectively explores unknown or partially known environments to obtain more information and experience;

[0048] Safe learning constraints ensure that the system not only pursues high performance but also meets a series of safety and practicality requirements during the learning and decision-making process.

[0049] Furthermore, the step 2.1 includes:

[0050] Geometric dynamic feature design, including lidar point cloud density gradient and dynamic object coverage;

[0051] The density gradient of the lidar point cloud reflects the density change of the point cloud in space, helps the system perceive the geometric structure and potential obstacles of the environment, and quantifies the observation confidence of the lidar in the local area. The expression is as follows:

[0052] (1)

[0053] in, Indicates the number of lidar hit points within a unit voxel, represents the voxel volume, Indicates the normalized value range of reflection intensity;

[0054] Calculating the dynamic object coverage rate represents the proportion and distribution of dynamic objects in the scene, so that the system can timely understand the dynamic characteristics of the environment. The expression is as follows:

[0055] (2)

[0056] in, represents the estimated velocity vector of the dynamic object, Represents the projected area of the dynamic object detection frame, Indicates the field of view area of the sensor;

[0057] Sensory reliability, including IMU confidence and radar-visual depth consistency. IMU confidence reflects the credibility of inertial measurement unit data and is calculated based on its noise level and bias stability. The higher the result value, the more reliable the IMU data. The expression is as follows:

[0058] (3)

[0059] in, represents the gyroscope bias estimation error, represents the standard deviation of accelerometer noise (calculated over a sliding window);

[0060] Radar-visual depth consistency measures the degree of agreement between the visual sensor and radar depth data. It is calculated by comparing the depth measurements of the two sensors. The higher the consistency, the more coordinated the depth data from the two sensors. The expression is as follows:

[0061] (4)

[0062] in, represents the observed depth distribution histogram, represents the depth distribution histogram of the depth camera, It represents the intersection and union ratio of the effective areas of the two;

[0063] System resource status monitoring, including real-time computing load and remaining energy budget. Real-time computing load reflects the current computing pressure of the system and helps the module decide whether to reduce the amount of sensor data processing or simplify computing tasks. The expression is as follows:

[0064] (5)

[0065] in, Indicates the memory capacity currently used by the GPU. Indicates the total GPU memory capacity. Indicates the time taken to render a single frame. Indicates the expected frame period;

[0066] The remaining energy budget indicates the energy reserve of the system, enabling the module to select a sensor combination or adjust the working mode based on the current remaining energy budget. The expression is as follows:

[0067] (6)

[0068] in, Indicates the current remaining battery level. Indicates the initial total power, Indicates the system running time. Indicates the power attenuation factor;

[0069] Spatiotemporal context includes historical decision memory and scene category probability. Historical decision memory records the success rate and effect of past sensor selection decisions, helping the module learn from experience, avoid repeated errors, and improve decision efficiency. The expression is as follows:

[0070] (7)

[0071] in, Indicates the past Step action, represents the long short-term memory network;

[0072] The scene category probability is calculated based on the current environment characteristics, and the probability of belonging to low-texture scenes, limited-viewing-constrained scenes, and dynamic scenes is calculated, so that the module can adjust the sensor selection strategy according to the scene characteristics. The expression is as follows:

[0073] (8)

[0074] in, express CLIP The joint image-text embedding vector extracted by the model, represents the learnable weight matrix, To normalize the distribution, the sum is 1.

[0075] Furthermore, the step 2.3 includes:

[0076] The main rewards include positioning accuracy rewards and energy efficiency rewards, which are used to improve the accuracy of system positioning and energy efficiency respectively. The expressions are as follows:

[0077] (11)

[0078] (12)

[0079] Where ATE represents the absolute trajectory error, represents the total instantaneous power consumption, Indicates the reward compensation item when the remaining power exceeds the threshold;

[0080] Auxiliary rewards, including rendering quality rewards and policy stability rewards, are designed to improve the visual quality of scene reconstruction and ensure the continuity and reliability of the policy; the expressions are as follows:

[0081] (13)

[0082] (14)

[0083] Among them, SSIM represents the structural similarity index, LPIPS represents the learning-perceptual image block similarity, Indicates the penalty for action mutation, Represents the smoothness constraint of the policy network parameters;

[0084] Penalty terms, including fatal penalty errors and resource overload penalties, are used to avoid unreasonable sensor selection and excessive resource consumption, thereby guiding the system to learn the optimal sensor selection strategy and achieve comprehensive optimization of system performance. The expression is as follows:

[0085] (15)

[0086] (16)

[0087] The formula for calculating the comprehensive reward is as follows:

[0088] (17)

[0089] Furthermore, the step 3 includes the following steps:

[0090] The reinforcement learning sensor selection module adaptively selects the most suitable sensor mode based on the current environmental conditions and system status, considering the impact of various parameters on sensor mode selection. The weight function expression is as follows:

[0091] + + + + + (twenty one)

[0092] in, , , , , and Respectively represent the lidar confidence , dynamic object coverage , IMU confidence , radar vision consistency , computing power load and energy remaining budget High weight;

[0093] Step 3.1 Depth Camera-IMU Mode;

[0094] When the reinforcement learning sensor selection module is based on the current environment characteristics, according to the weight function S≥ , To determine the threshold, the SLAM system selects the sensor combination of depth camera and IMU, with the depth camera as the main and IMU as the auxiliary;

[0095] Step 3.2 LiDAR-Depth Camera Mode;

[0096] When the reinforcement learning sensor selection module is based on the current environment characteristics, according to the weight function , To determine the threshold, the SLAM system selects a sensor combination of lidar and depth camera, with the depth camera as the main sensor and the lidar as the auxiliary sensor.

[0097] Step 3.3 LiDAR-IMU-Depth Camera Multi-Sensor Balance Mode;

[0098] When the reinforcement learning sensor selection module is based on the current environment characteristics, the weight function is used according to the weight function , determine whether the SLAM system should use a sensor mode that combines lidar, depth camera and IMU;

[0099] Step 3.4 After the reinforcement learning sensor selection module executes the action and interacts with the environment, the reward function module records the historical decision memory, and the value network is evaluated to optimize the update of the entire policy network.

[0100] Furthermore, the step 3.1 includes,

[0101] Step 3.1.1 Data input and key frame selection;

[0102] Prioritize frames containing new scene structures or significant features as key frames to improve the integrity and efficiency of the map;

[0103] Step 3.1.2 uses generalized ICP tracking for point cloud registration;

[0104] Select RGB image of the frame and depth image , generate point cloud , where each point , and calculate the covariance matrix for each point Estimate the source point cloud of the current frame using the GICP algorithm and target map point cloud Relative pose transformation ;

[0105] Step 3.1.3. IMU pre-integration;

[0106] By fusing IMU high-frequency measurements, we can generate inter-frame motion pre-integration as the initial guess for GICP, which improves tracking efficiency and combines with the GICP algorithm to enhance the robustness and accuracy of the system.

[0107] Step 3.1.4: Update pre-integration;

[0108] The GICP optimization results are used to correct the drift of IMU integral caused by noise and bias over time.

[0109] Furthermore, the step 3.2 includes the following steps:

[0110] Step 3.2.1 Data input: images from depth camera and point cloud from lidar;

[0111] The time-aligned LiDAR point cloud is converted into a depth image using the calibrated external signal for integration; the expression is as follows:

[0112] (38)

[0113] in, is the lidar point cloud, and are the rotation matrix and translation vector from the laser radar to the camera coordinate system, is the intrinsic matrix of the camera;

[0114] Step 3.2.2 uses the incremental error minimization function;

[0115] To ensure the exact correspondence between the plane and the point, the formula is as follows:

[0116] (39)

[0117] in, Represents a lidar point cloud A point in It is based on the current posture estimation from the previous moment to the world coordinate system The result after iterations, is the closest The Gaussian center of yes The normal vector of . Yes The weight of is a regularization term used to enhance the stability and accuracy of the error function, taking into account the directional error between normal vectors;

[0118] Introducing regularization terms To enhance the stability and accuracy of the error function and take into account the error in the normal direction; the expression is as follows:

[0119] (40)

[0120] in, is the normal of the current Gaussian distribution;

[0121] Step 3.2.3 weight function calculation;

[0122] The calculation steps of the weight function are as follows:

[0123] a. Determine the Gaussian center within the local spherical region: Find all the nearest Gaussian distribution centers within The center of the ball. is the radius;

[0124] b. Calculate the density function of Gaussian points, density function Calculated by the following formula:

[0125] (41)

[0126] in, is the reconstructed covariance matrix, which is obtained by selecting the minimum variance along the normal direction and larger variance in the vertical direction to build;

[0127] c. Simplify the calculation of density function. In order to speed up the calculation, simplify the calculation of density function during the tracking process:

[0128] (42)

[0129] d. Consistency calculation, for each point , calculate the normal of the current Gaussian distribution and the local mean normal consistency ;

[0130] e. Complex texture calculation: For each image area corresponding to a radar point, calculate the local texture complexity. Project it to the camera's pixel coordinate system to get its position in the image , intercepted with this as the center ( = 16), convert the image block to grayscale, and calculate the variance of pixel intensity:

[0131] (43)

[0132] in, is the pixel intensity, Image block mean;

[0133] The variance is converted to a 0-1 texture weight through the sigmoid function:

[0134] (44)

[0135] in, is a scaling factor used to adjust the variance sensitivity;

[0136] f. Final weight function, define the final weight function is the product of normal consistency, density function and texture complexity, that is, .

[0137] Furthermore, the step 3.3 includes the following steps:

[0138] Step 3.3.1 Data input and hardware synchronization;

[0139] The LiDAR provides a sparse but highly accurate 3D point cloud, the depth camera captures the RGB texture of the scene, and the IMU outputs angular velocity and acceleration at high frequency for motion prediction. This ensures that the data timestamps of the three are strictly aligned to avoid timing drift.

[0140] Step 3.3.2: Select keyframes using depth camera input.

[0141] Select representative frames from the continuous data stream as key frames to reduce redundant calculations;

[0142] Step 3.3.3. IMU data is used for state propagation;

[0143] Based on the input IMU data and the state of the previous key frame, the current state is predicted through IMU pre-integration, which is used for state estimation and forward prediction of motion distortion removal;

[0144] Step 3.3.4 dedistorts the lidar input;

[0145] Using the continuous pose predicted by IMU, each lidar point Convert from the local coordinate system at the time of scanning to the global coordinate system; the expression is as follows:

[0146] (45)

[0147] in, Indicates a point The collection timestamp, Indicates the value obtained through IMU interpolation The posture of the moment.

[0148] Furthermore, the step (4) includes the following steps:

[0149] Step 4.1 Sliding window maintenance;

[0150] Step 4.2 4D Gaussian distribution;

[0151] Step 4.3 introduces optical flow to solve the 4D GS overfitting problem;

[0152] Step 4.4 Map update and optimization;

[0153] Step 4.5 loop detection;

[0154] By extracting lidar and visual features and generating feature descriptors, the module is able to detect potential loop closure candidates;

[0155] Confirm loop closure assumptions using geometric verification and consistency checks;

[0156] The verified loop constraints are fed back into the map optimization process to globally optimize the map structure.

[0157] In summary, the reinforcement learning adaptive multimodal SLAM method based on 4D Gaussian splashing described in this application has the following advantages and beneficial effects:

[0158] 1. Based on the existing multi-sensor fusion SLAM system, this application uses reinforcement learning to adaptively select appropriate sensor combinations according to specific environmental characteristics. It has significant environmental adaptability and resource optimization capabilities, improves positioning accuracy and map construction quality, while reducing power consumption and computational complexity, and improves the system's robustness, efficiency, and resource utilization.

[0159] 2. This application integrates 4D Gaussian splattering technology into a multimodal SLAM system, which expands the scene adaptability of the SLAM system and has significant advantages over traditional SLAM technology in terms of rendering speed, memory efficiency, dynamic adaptability, robustness, resource efficiency and global consistency.

[0160] 3. The 4D GS employed in this application adds a temporal dimension to the 3D GS, enabling efficient representation and processing of complex changes in dynamic scenes. Furthermore, optical flow is incorporated into 4D GS to address overfitting, enabling it to not only capture the motion and deformation of dynamic objects but also significantly reduce memory usage and computational complexity through a sparse Gaussian distribution representation, while also improving model stability and generalization capabilities.

[0161] 4. This application provides more prior information through optical flow to limit the overfitting of the 4D GS deformation field network to the noise in the training data, so that the deformation field network can learn a more reasonable and more physically consistent Gaussian point deformation method, thereby improving the stability and generalization ability of the model.

[0162] 5. This application is particularly suitable for resource-constrained mobile devices and complex and changing dynamic scenes, providing a more efficient and accurate solution for the application of SLAM systems in dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0163] The present application will now be further described with reference to the following drawings;

[0164] Figure 1 1 is a system framework diagram for applying the adaptive multimodal SLAM method described in this application;

[0165] Figure 2 It is a flow chart of the reinforcement learning sensor adaptive selection module;

[0166] Figure 3 This is the depth camera-IMU mode flow chart;

[0167] Figure 4 This is the LiDAR-Depth Camera mode flow chart;

[0168] Figure 5 This is the LiDAR-Depth Camera-IMU mode flow chart;

[0169] Figure 6 It is a 4D Gaussian splash flow chart;

[0170] Figure 7 It is a schematic diagram of Gaussian splashing; DETAILED DESCRIPTION

[0171] In order to further illustrate the technical means adopted by this application to achieve the predetermined design objectives, the following preferred implementation scheme is proposed in conjunction with the accompanying drawings.

[0172] The following description sets forth specific details to facilitate a thorough understanding of the present invention. However, the present invention can be implemented in a variety of other ways than those described herein, and those skilled in the art can make similar generalizations without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0173] like Figure 1 As shown, the system using the adaptive multimodal SLAM method described in this application includes a reinforcement learning-based sensor selection module, a data fusion processing module, a map update and backend optimization module, and a loop detection and global map optimization module. The SLAM system adopts three sensor combinations, namely depth camera-IMU, lidar-depth camera, and lidar-IMU-depth camera.

[0174] like Figures 2 to 7 The method of reinforcement learning adaptive multimodal SLAM based on 4D Gaussian splashing includes the following steps:

[0175] Step 1: Multi-source sensor input and data preprocessing;

[0176] Receive point cloud data from the lidar, RGB image data from the depth camera, and IMU data, and perform data preprocessing;

[0177] Step 1.1 Data input;

[0178] The SLAM system synchronizes the point cloud data received from the lidar, the RGB image data from the depth camera, and the IMU data in time to ensure that all data at the same timestamp can accurately correspond;

[0179] Step 1.2: Incorporate time information;

[0180] In order to handle dynamic scenes, timestamps are introduced in the data preprocessing stage. ,Each lidar point cloud and image frame is time-stamped so that the dynamic changes of the scene can be accurately tracked and compensated in subsequent processing, thus providing the necessary temporal context for the deformation field network of 4D GS;

[0181] Step 1.3 Multi-sensor external parameter calibration;

[0182] Use the Kalibr framework to calibrate the external parameters between sensors;

[0183] Step 1.4: Initialize the reinforcement learning module.

[0184] The reinforcement learning-based sensor selection module is initialized during the data preprocessing phase. By analyzing historical data and environmental characteristics, the module learns a strategy for selecting the optimal sensor combination under different scenario conditions. Initially, the module has no prior knowledge of sensor reliability and therefore adopts a random exploration strategy.

[0185] Step 2. Select the sensor combination type;

[0186] Based on the reinforcement learning sensor selection module, the reinforcement learning strategy network learns the environmental status in real time, including but not limited to data such as lighting, dynamic object density, and sensor noise, dynamically selects the most reliable sensor combination type and activates sensors on demand to adapt to highly dynamic, complex and changing environments;

[0187] Step 2.1 State space design;

[0188] Define the environmental information that the reinforcement learning system can observe at each decision moment to guide the sensor selection strategy; including the following aspects:

[0189] Geometric dynamic feature design, including lidar point cloud density gradient and dynamic object coverage;

[0190] The density gradient of the lidar point cloud reflects the density change of the point cloud in space, helps the system perceive the geometric structure and potential obstacles of the environment, and quantifies the observation confidence of the lidar in the local area. The expression is as follows:

[0191] (1)

[0192] in, Indicates the number of lidar hit points within a unit voxel, represents the voxel volume, Indicates the normalized value range of reflection intensity;

[0193] Calculating the dynamic object coverage rate represents the proportion and distribution of dynamic objects in the scene, so that the system can timely understand the dynamic characteristics of the environment. The expression is as follows:

[0194] (2)

[0195] in, represents the estimated velocity vector of the dynamic object, Represents the projected area of the dynamic object detection frame, Indicates the field of view area of the sensor;

[0196] These two features together provide key information for the reinforcement learning module, enabling it to optimize the sensor selection strategy based on the static and dynamic characteristics of the environment, thereby improving the adaptability and accuracy of the system in complex environments.

[0197] Sensory reliability, including IMU confidence and radar-visual depth consistency. IMU confidence reflects the credibility of inertial measurement unit data and is calculated based on its noise level and bias stability. The higher the result value, the more reliable the IMU data. The expression is as follows:

[0198] (3)

[0199] in, represents the gyroscope bias estimation error, represents the standard deviation of accelerometer noise (calculated over a sliding window);

[0200] Radar-visual depth consistency measures the degree of agreement between the visual sensor and radar depth data. It is calculated by comparing the depth measurements of the two sensors. The higher the consistency, the more coordinated the depth data from the two sensors. The expression is as follows:

[0201] (4)

[0202] in, represents the observed depth distribution histogram, represents the depth distribution histogram of the depth camera, It represents the intersection and union ratio of the effective areas of the two;

[0203] Together, they provide the reinforcement learning sensor module with key information on sensor data reliability, helping it optimize sensor selection and weight distribution, thereby improving the system's perception and decision-making accuracy in complex environments.

[0204] System resource status monitoring, including real-time computing load and remaining energy budget. Real-time computing load reflects the current computing pressure of the system and helps the module decide whether to reduce the amount of sensor data processing or simplify computing tasks. The expression is as follows:

[0205] (5)

[0206] in, Indicates the memory capacity currently used by the GPU. Indicates the total GPU memory capacity. Indicates the time taken to render a single frame. Indicates the expected frame period;

[0207] The remaining energy budget indicates the energy reserve of the system, enabling the module to prioritize the sensor combination with lower energy consumption or adjust the working mode when energy is limited. The expression is as follows:

[0208] (6)

[0209] in, Indicates the current remaining battery level. Indicates the initial total power, Indicates the system running time. Indicates the power attenuation factor;

[0210] The combined effect of the two ensures that the system can achieve efficient resource utilization and reasonable management of energy consumption while meeting performance requirements;

[0211] Spatiotemporal context includes historical decision memory and scene category probability. Historical decision memory records the success rate and effect of past sensor selection decisions, helping the module learn from experience, avoid repeated errors, and improve decision efficiency. The expression is as follows:

[0212] (7)

[0213] in, Indicates the past Step action, represents the long short-term memory network;

[0214] The scene category probability calculates the likelihood of the scene belonging to different categories based on the current environment characteristics, such as long corridors and weak texture areas, so that the module can adjust the sensor selection strategy according to the scene characteristics; the expression is as follows:

[0215] (8)

[0216] in, express CLIP The joint image-text embedding vector extracted by the model, represents the learnable weight matrix, To normalize the distribution, the sum is 1;

[0217] The combination of the two can make sensor selection more intelligent and adaptable;

[0218] Step 2.2 defines the action space formula;

[0219] This involves defining all possible actions that the system can perform, i.e., adjusting sensor parameters to optimize data acquisition and system performance;

[0220] The SLAM system uses a hybrid discrete-continuous space to achieve refined control. The mode selection part is discrete and includes three sensor-dominated modes: Mode 1 is the depth camera-IMU mode, which is suitable for situations with good lighting conditions and relatively stable environmental structures; Mode 2 is the lidar-depth camera mode, which is suitable for environments with many dynamic objects and low-light or no-light scenes; Mode 3 is the lidar-IMU-depth camera mode, which is suitable for complex and changing environments, long-running systems, and scenarios requiring high-precision positioning;

[0221] The weight distribution part is continuous and includes sensor gain coefficients and resource constraint parameters. The sensor gain coefficients are the weights of the lidar, depth camera, and IMU, ranging from 0 to 1. The expression is as follows

[0222] (9)

[0223] The resource limiting parameter is the lidar sampling rate, which ranges from 5Hz to 40Hz; the expression is as follows:

[0224] (10)

[0225] The action space formulation provides a mathematical framework for adjusting sensor configurations, enabling the system to make flexible decisions in dynamic environments, balancing data quality and resource consumption, thereby improving overall system performance.

[0226] Step 2.3 Reward function module;

[0227] This guides the SLAM system to learn the optimal sensor selection strategy to achieve comprehensive optimization of system performance. This improves positioning accuracy and mapping quality, balances resource utilization efficiency, and avoids excessive consumption. It also constrains system behavior through penalty terms to prevent unreasonable sensor selection, ensuring the entire SLAM system operates stably and efficiently in complex and changing environments.

[0228] Contains main rewards, auxiliary rewards and penalties;

[0229] The main rewards include positioning accuracy rewards and energy efficiency rewards, which are used to improve the accuracy of system positioning and energy efficiency respectively. The expressions are as follows:

[0230] (11)

[0231] (12)

[0232] Where ATE represents the absolute trajectory error, represents the total instantaneous power consumption, Indicates the reward compensation item when the remaining power exceeds the threshold;

[0233] Step 2.3.2 Auxiliary Rewards;

[0234] It includes rendering quality rewards and policy stability rewards, aiming to improve the visual effect of scene reconstruction and ensure the continuity and reliability of the policy; the expression is as follows:

[0235] (13)

[0236] (14)

[0237] Among them, SSIM represents the structural similarity index, LPIPS represents the learning-perceptual image block similarity, Indicates the penalty for action mutation, Represents the smoothness constraint of the policy network parameters;

[0238] Penalty terms, including fatal penalty errors and resource overload penalties, are used to avoid unreasonable sensor selection and excessive resource consumption, thereby guiding the system to learn the optimal sensor selection strategy and achieve comprehensive optimization of system performance. The expression is as follows:

[0239] (15)

[0240] (16)

[0241] The above structure can realize the reward function to fully guide the system to optimize sensor selection to improve overall performance;

[0242] The formula for calculating the comprehensive reward is as follows:

[0243] (17)

[0244] Step 2.4 Network architecture design;

[0245] As the core of the reinforcement learning sensor selection module, it determines how the system processes multimodal sensor data and makes intelligent decisions; including,

[0246] The state encoder, as the foundation of the entire network, is responsible for converting multimodal sensor data into a unified feature representation so that the subsequent policy network and value network can make decisions and evaluations based on these features; it includes,

[0247] a. Backbone network: The ResNet-18 architecture is used to process visual features. ResNet-18 is a classic deep residual network that effectively extracts high-level semantic information from images. Residual connections alleviate the degradation problem of deep networks, ensuring network training effectiveness and feature extraction capabilities.

[0248] b. Branch network: Utilizes the point cloud Transformer to extract geometric features. The Transformer architecture is advantageous in processing sequential data, capturing global dependencies in point cloud data and extracting more representative geometric features, which helps the system understand the 3D structure of the environment.

[0249] c. Time Series Network: Use Bi-LSTM to process IMU sequences. Bi-LSTM can simultaneously consider the past and future information of IMU data, capturing bidirectional dependencies in the time series. This allows for more accurate modeling of the dynamic characteristics of IMU data and provides a stable attitude estimate for the system.

[0250] The policy network is responsible for outputting specific action selections based on the encoded state features, including discrete action modes and continuous parameter adjustments.

[0251] a. Mode selection branch: Gumbel-Softmax is used to output discrete actions. Gumbel-Softmax is a technique that balances continuous relaxation and discrete sampling. This allows the network to effectively learn the selection of discrete actions during training while maintaining the propagation of gradients. This makes it suitable for discrete decision-making tasks such as selecting the dominant sensor mode.

[0252] b. Parameter adjustment branch: This branch uses the Tanh activation function to output continuous weights. The Tanh activation function constrains the output to the range [-1, 1]. Through appropriate scaling and offsetting, it can be mapped to the desired continuous parameter space, such as sensor gain coefficients and sampling rate parameters, enabling fine-tuning of sensor parameters.

[0253] The value network is used to evaluate the pros and cons of the current state and action combination, providing learning signals for the policy network to help it optimize the decision-making process; it includes:

[0254] Dueling DQN Architecture: Separating State Value and Advantage Functions. The Dueling DQN architecture divides the value network into two parts, estimating the state value function (V) and the advantage function (A) separately. The final Q value is then derived through a specific combination. This separation enables the network to more accurately assess the relative merits of different actions in the current state, improving learning efficiency and stability.

[0255] b. Multi-head attention: Dynamically weighting multimodal features. The multi-head attention mechanism can simultaneously focus on different aspects of different modal features and dynamically adjust the weights of each modal feature based on the current task requirements, achieving effective fusion of multimodal information and enhancing the network's adaptability to complex environments.

[0256] Step 2.5 executes the training strategy;

[0257] Through efficient data collection mechanism, intelligent exploration strategy and strict safety learning constraints, a comprehensive training framework is provided for the reinforcement learning sensor adaptive selection module; including,

[0258] The data collection mechanism provides the system with experience data of interactions with the environment and is the basis for learning and optimizing strategies. Specifically,

[0259] a. Deployment environment: Choose the Gazebo simulation environment, which provides high-fidelity scene simulation and flexible sensor configuration options for testing and training robotic algorithms.

[0260] b. Sampling frequency: Set to 120FPS real-time sampling. The high frame rate captures more detailed environmental dynamics and robot motion status, providing richer information for subsequent data processing and strategy learning.

[0261] Exploration strategy determines how the system can effectively explore in unknown or partially known environments to obtain more information and experience. Specifically,

[0262] a. Adaptive -greddy: As the number of training rounds increases, The value gradually decreases, and the system transitions from extensive exploration to utilizing learned knowledge, balancing exploration and utilization and improving policy performance; the expression is as follows:

[0263] (18)

[0264] b. Intrinsic curiosity: Add a state prediction error reward, that is, estimate the next state through the prediction model and use the prediction error as the intrinsic reward; the expression is as follows:

[0265] (19)

[0266] in, is the intrinsic reward, indicating that the system is in state Take action The intrinsic rewards gained when is a regulation coefficient that controls the strength of intrinsic rewards; is the system's prediction of the next state, estimated by a forward model. is the actual next state feature;

[0267] Safe learning constraints ensure that the system not only pursues high performance during learning and decision-making, but also meets a series of safety and practicality requirements. Specifically,

[0268] a. Action Masking: This prevents the selection of high-power actions that exceed battery capacity. This means that actions that would cause rapid battery depletion are masked from the action space. This ensures that the system considers energy constraints during decision-making, avoids unreasonable high-power behaviors, and improves the practicality and sustainability of the system.

[0269] b. Policy gradient correction, the expression is as follows:

[0270] (20)

[0271] Among them, represents the gradient of the probability distribution of the policy network output action with respect to the network parameters, is the action-value function, is the baseline function.

[0272] This application achieves a dynamic balance in sensor decision-making in the three dimensions of time, space and energy through the deep combination of the above-mentioned hierarchical state representation, hybrid action space and multi-objective reward function. Therefore, the SLAM system can achieve the advantages of dynamic environment adaptation, resource optimization management, improved positioning and mapping accuracy, enhanced system robustness and support for long-term stable operation.

[0273] Step 3: Multi-source sensor data fusion processing;

[0274] Based on the data fusion processing module, the sensor combination performs multi-source sensor data fusion processing, thereby providing rich and accurate data for the construction of Gaussian maps;

[0275] The reinforcement learning sensor selection module adaptively selects the most suitable sensor mode based on the current environmental conditions and system status, considering the impact of various parameters on sensor mode selection. The weight function expression is as follows:

[0276] + + + + + (twenty one)

[0277] in, , , , , and Respectively represent the lidar confidence , dynamic object coverage , IMU confidence , radar vision consistency , computing power load and energy remaining budget High weight;

[0278] The steps include:

[0279] Step 3.1 Depth Camera-IMU Mode;

[0280] When the reinforcement learning sensor selection module selects the laser radar based on the current environment characteristics, Low, dynamic object coverage Low, IMU confidence High, radar vision consistency Low computing load Moderate and energy surplus budget When the value is high, for example, in an environment with good lighting conditions, relatively stable environment structure, many mirror objects, or few dynamic objects with rich textures, the weight function S≥ , To determine the threshold, the SLAM system selects mode 1, which uses a sensor combination of depth camera and IMU, with the depth camera as the main sensor and the IMU as the auxiliary sensor.

[0281] Step 3.1.1 Data input and key frame selection;

[0282] Prioritize frames containing new scene structures or significant features as key frames to improve the integrity and efficiency of the map;

[0283] Step 3.1.2 uses generalized ICP tracking (GICP) for point cloud registration;

[0284] Select RGB image of the frame and depth image , generate point cloud , where each point , and calculate the covariance matrix for each point Estimate the source point cloud of the current frame using the GICP algorithm and target map point cloud Relative pose transformation ;include,

[0285] a. Distribution distance calculation: Model each point as a Gaussian distribution , the source point cloud is transformed Then, the distance to the target point cloud Defined as:

[0286] (twenty two)

[0287] Its distribution is:

[0288] (twenty three)

[0289] in, , represents the coordinates of corresponding points in the target map and the source point cloud, , Represents the covariance matrix of the target point cloud and the source point cloud;

[0290] b. Maximum likelihood estimation: Solve the optimal transformation by maximizing the log-likelihood of the probability density function :

[0291] (twenty four)

[0292] The optimization objective is simplified to minimizing the Mahalanobis distance, which is expressed as follows:

[0293] (25)

[0294] Step 3.1.3 IMU pre-integration;

[0295] Fusion of IMU high-frequency measurements to generate inter-frame motion pre-integration as the initial guess for GICP, improving tracking efficiency and combining with the GICP algorithm to enhance the robustness and accuracy of the system; including,

[0296] a. Construct the original measurement value of IMU measurement model:

[0297] (26)

[0298] (27)

[0299] in, , represents the raw acceleration and angular velocity measurements, Indicates the rotation of IMU to the world coordinate system, represents the gravity vector, / represents the sensor bias, / represents sensor noise;

[0300] b. Perform motion pattern recursion: position ,speed , rotate The recursive formula is as follows:

[0301] (28)

[0302] (29)

[0303] (30)

[0304] c. Calculate the IMU pre-integration quantity:

[0305] To avoid repeated integration, define Frame to The relative motion between frames is expressed as follows:

[0306] (31)

[0307] (32)

[0308] (33)

[0309] in, , , Represent the product of position, velocity and rotation respectively, ; Skew-symmetric matrix representing angular velocity .

[0310] d. Calculate the relative transformation between consecutive frames : Using external parameters between camera and IMU sensor Convert the relative transformation to camera coordinates. Get a good initial guess for GICP tracking from IMU pre-integration ;

[0311] (34)

[0312] (35)

[0313] Step 3.1.4: Update pre-integration;

[0314] Use GICP optimization results to correct the drift of IMU integral due to noise and bias over time; including,

[0315] a. GICP optimized camera pose Convert to the IMU coordinate system; the expression is as follows:

[0316] (36)

[0317] in, Represents the external parameters from the camera to the IMU, and Represent the results of GICP tracking position and rotation respectively.

[0318] b. Update IMU status: ， ， (37)

[0319] Step 3.2 LiDAR-Depth Camera Mode;

[0320] When the reinforcement learning sensor selection module selects the laser radar based on the current environment characteristics, High, dynamic object coverage High, IMU confidence Low, radar vision consistency Medium, computing load Larger and more energy-efficient budget Moderate time reinforcement learning sensor adaptive selection module according to the current environment, such as low light, low texture environment, according to the weight function , To determine the threshold, when the SLAM system selects mode 2, a combination of lidar and depth camera sensors is used, with the depth camera as the main sensor and the lidar as the auxiliary sensor; including:

[0321] Step 3.2.1 Data input: images from depth camera and point cloud from lidar;

[0322] The time-aligned LiDAR point cloud is converted into a depth image using the calibrated external signal for integration; the expression is as follows:

[0323] (38)

[0324] in, is the lidar point cloud, and are the rotation matrix and translation vector from the laser radar to the camera coordinate system, is the intrinsic matrix of the camera;

[0325] Step 3.2.2 uses the incremental error minimization function;

[0326] To ensure the exact correspondence between the plane and the point, the formula is as follows:

[0327] (39)

[0328] in, Represents a lidar point cloud A point in It is based on the current posture estimation from the previous moment to the world coordinate system The result after iterations, is the closest The Gaussian center of yes The normal vector of . Yes The weight of is a regularization term used to enhance the stability and accuracy of the error function, taking into account the directional error between normal vectors;

[0329] Introducing regularization terms To enhance the stability and accuracy of the error function and take into account the error in the normal direction; the expression is as follows:

[0330] (40)

[0331] in, is the normal of the current Gaussian distribution;

[0332] Step 3.2.3 weight function calculation;

[0333] In order to distinguish Gaussian points generated by color supervision alone from those generated by lidar depth, the system introduces a weight function. This weight function combines the consistency of the normal vector, density factors, and texture complexity to evaluate the reliability of different Gaussian points. The calculation steps of the weight function are as follows:

[0334] a. Determine the Gaussian center within the local spherical region: Find all the nearest Gaussian distribution centers within The center of the ball. is the radius;

[0335] b. Calculate the density function of Gaussian points, density function Calculated by the following formula:

[0336] (41)

[0337] in, is the reconstructed covariance matrix, which is obtained by selecting the minimum variance along the normal direction and larger variance in the vertical direction To build.

[0338] c. Simplify the calculation of density function. In order to speed up the calculation, simplify the calculation of density function during the tracking process:

[0339] (42)

[0340] d. Consistency calculation, for each point , calculate the normal of the current Gaussian distribution and the local mean normal consistency ;

[0341] e. Complex texture calculation: For each image area corresponding to a radar point, calculate the local texture complexity. Project it to the camera's pixel coordinate system to get its position in the image , intercepted with this as the center ( = 16), convert the image block to grayscale, and calculate the variance of pixel intensity:

[0342] (43)

[0343] in, is the pixel intensity, Image block mean;

[0344] The variance is converted to a 0-1 texture weight through the sigmoid function:

[0345] (44)

[0346] in, is a scaling factor used to adjust the variance sensitivity;

[0347] f. Final weight function, define the final weight function is the product of normal consistency, density function and texture complexity, that is, ;

[0348] Step 3.3 LiDAR-IMU-Depth Camera Multi-Sensor Balance Mode;

[0349] When the reinforcement learning sensor selection module selects the laser radar based on the current environment characteristics, Medium, dynamic object coverage High, IMU confidence High, radar vision consistency Low computing load Larger and more energy-efficient budget When the weight function is high, such as in a complex and changing dynamic environment, or in a long-term high-precision positioning environment, the weight function is used to determine the weight of the target. , when judging that the SLAM system selects mode 3, a sensor mode combining lidar, depth camera and IMU is used; including,

[0350] Step 3.3.1 Data input and hardware synchronization;

[0351] The LiDAR provides a sparse but highly accurate 3D point cloud, the depth camera captures the RGB texture of the scene, and the IMU outputs angular velocity and acceleration at high frequency for motion prediction. This ensures that the data timestamps of the three are strictly aligned to avoid timing drift.

[0352] Step 3.3.2: Select keyframes using depth camera input.

[0353] Select representative frames from the continuous data stream as key frames to reduce redundant calculations;

[0354] Step 3.3.3. IMU data is used for state propagation;

[0355] Based on the input IMU data and the state of the previous key frame, the current state is predicted through IMU pre-integration, which is used for state estimation and forward prediction of motion distortion removal;

[0356] Step 3.3.4 dedistorts the lidar input;

[0357] Using the continuous pose predicted by IMU, each lidar point Convert from the local coordinate system at the time of scanning to the global coordinate system; the expression is as follows:

[0358] (45)

[0359] in, Indicates a point The collection timestamp, Indicates the value obtained through IMU interpolation The posture of the moment.

[0360] Step 3.4 After the reinforcement learning sensor selection module executes the action and interacts with the environment, the reward function module records the historical decision memory, and the value network is evaluated to optimize the update of the entire policy network.

[0361] Step 4: Map update and optimization;

[0362] Based on the map update and back-end optimization module, the system uses the rich map data provided by the front-end multi-modal sensor data fusion processing module and combines it with 4D GS to perform Gaussian map update and optimization of the environment map. It also performs error correction and map optimization through loop closure detection.

[0363] The following steps are included:

[0364] Step 4.1 Sliding window maintenance;

[0365] In the data fusion processing module, the system maintains a sliding window that filters and selects point clouds from the most recent 10 time frames in the Gaussian map to construct Gaussian points, while masking the remaining Gaussian points. This selection process ensures that the Gaussian points are relevant in the sub-map of current interest;

[0366] Step 4.2 4D Gaussian distribution;

[0367] The Gaussian deformation field network is used to model the motion and shape changes of the Gaussian distribution of dynamic objects. The network consists of an efficient space-time structure encoder and a multi-head Gaussian deformation decoder. Its goal is to transform the standard 3D Gaussian distribution into new positions and shapes by learning the Gaussian deformation field, thereby achieving efficient representation and real-time rendering of dynamic scenes. Predict the deformation of the current frame point cloud ; The expression is as follows:

[0368] (46)

[0369] in, Represents the deformed three-dimensional Gaussian function.

[0370] Specifically, the space-time structure encoder The goal of

[15] is to efficiently encode the spatial and temporal features of a 3D Gaussian distribution. It consists of a multi-resolution HexPlane module and a small multi-layer perceptron (MLP).

[0371] The multi-resolution HexPlane module is used to efficiently encode the spatial and temporal features of 3D Gaussian distribution. It does this by decomposing 4D neural voxels into multiple 2D planes that can be sampled and encoded at different resolutions.t , extract the center coordinates of the 3D Gaussian distribution G and timestamp t , by querying the multi-resolution plane module Get voxel features and query using bilinear interpolation. Contains 6 multi-resolution plane modules ,That They are , Indicates the resolution level.

[0372] Each flat module The definition of ,in is the hidden dimension of the feature, is the base resolution of the voxel grid.

[0373] The voxel features are queried by bilinear interpolation. The formula is:

[0374] (47)

[0375] in, It is a characteristic of neurotoxins.

[0376] A small MLP , used to combine all features:

[0377] (48)

[0378] in, is the final feature representation.

[0379] Specifically, the multi-head Gaussian deformation decoder D It is used to decode the deformation of each 3D Gaussian distribution from the features obtained by the encoder. It consists of three independent MLPs to calculate the deformation of position, rotation and scale respectively.

[0380] Position deformation head, Used to calculate position deformation , the formula is:

[0381] (49)

[0382] Rotating deformation head, Used to calculate position deformation r , the formula is:

[0383] (50)

[0384] Scale the deformation head, Used to calculate scaling deformation , the formula is:

[0385] (51)

[0386] Apply these deformations to the original 3D Gaussian distribution to obtain the deformed Gaussian distribution , the formula is:

[0387] (52)

[0388] in, It's a new location. It's a new spin, It's the new zoom.

[0389] Step 4.3 introduces optical flow to solve the 4D GS overfitting problem;

[0390] During the real-time operation of the SLAM system, due to the large amount of newly emerging data and the influence of noise in the data, the 4D GS will have the problem of overfitting in constructing the environment map.

[0391] Specifically, the optical flow calculation method is used to capture the motion information of pixels in the time dimension, provide additional time consistency constraints for the model, and use the RAFT optical flow algorithm to calculate the pixel motion between adjacent timestamps. Including,

[0392] Step 4.3.1 Input image;

[0393] For each pair of images with adjacent timestamps and , use pre-trained RAFT to calculate optical flow , the pixel motion predicted by 4D GS is ;

[0394] Step 4.3.2: Deformation of 3D Gaussian points by optical flow constraint;

[0395] Ensure that the dynamic part predicted by the model is consistent with the optical flow and construct the loss function constrained by the optical flow; the expression is as follows:

[0396] (53)

[0397] in, represents the observed optical flow calculated from the RGB image by RAFT, Represents pixel-level motion prediction based on 4D GS deformation field rendering;

[0398] Step 4.3.3 Predict Gaussian parameter changes based on the deformation field network of the 4D GS;

[0399] Deformation field network using 4D GS Predict Gaussian parameter changes, including position changes , rotation change , scaling changes ; For the k Gaussian, its position after deformation is expressed as follows:

[0400] (54)

[0401] in, represents the position of the initial Gaussian, represents the position offset of the deformation field prediction;

[0402] Step 4.3.4: Project the Gaussian onto the image plane using a differentiable splatting process to calculate pixel-level motion.

[0403] The expression is as follows:

[0404] (55)

[0405] Step 4.3.5 introduces optical flow confidence to filter unreliable optical flow predictions;

[0406] To reduce the noise of the optical flow algorithm, the confidence map provided by the optical flow algorithm is used , optical flow constraint loss is applied only in areas with high confidence; the expression is as follows:

[0407] (56)

[0408] in, Indicates the optical flow confidence at position The value of

[0409] Step 4.4 Map update and optimization;

[0410] Use the Structure-from-Motion (SfM) method to initialize the static 3D Gaussian distribution. The first 3000 iterations only optimize the static 3D Gaussian, and then use the 3D Gaussian Instead of 4D Gaussian Perform image rendering, learn a reasonable initial 3D Gaussian distribution, separate the dynamic and static parts, reduce the pressure of large deformation learning, and avoid numerical instability problems when directly optimizing the deformation field network;

[0411] For the construction of loss function, use The training process is supervised by color loss, optical flow constraint loss and grid-based total score difference loss is also applied; the expression is as follows:

[0412] (57)

[0413] in, and The weight parameters representing the optical flow constraint loss and the grid-based total score difference loss are used to balance the effects of different losses.

[0414] Step 4.5 loop detection;

[0415] By extracting lidar and visual features and generating feature descriptors, the module is able to detect potential loop closure candidates;

[0416] Confirm loop closure assumptions using geometric verification and consistency checks;

[0417] The verified loop constraints are fed back into the map optimization process to globally optimize the map structure.

[0418] Through the above loop detection steps, the robustness and efficiency of the SLAM system can be significantly improved, ensuring the map quality and positioning accuracy during long-term operation.

[0419] As described above, similar technical solutions can be derived from the solutions described in the accompanying drawings and descriptions. However, any solution that does not deviate from the structure of the present invention still falls within the scope of the technical solution of this application.

Claims

1. A reinforcement learning adaptive multimodal SLAM method based on 4D Gaussian splatting, characterized by: The following steps are included: Step 1: Multi-source sensor input and data preprocessing; Receive point cloud data from the lidar, RGB image data from the depth camera, and IMU data, and perform data preprocessing; Step 2. Select the sensor combination type; Based on the reinforcement learning sensor selection module, the reinforcement learning strategy network learns the environment status in real time, dynamically selects the most reliable sensor combination type and activates sensors on demand; Step 2.1 State space design; Define the environmental information that the reinforcement learning system can observe at each decision moment; Step 2.2 defines the action space formula; This includes defining actions for the system to switch sensor modes to optimize data acquisition and system performance; The SLAM system uses a hybrid discrete-continuous space to achieve refined control. The mode selection part is discrete and includes three sensor-dominated modes: depth camera-IMU mode, lidar-depth camera mode, and lidar-IMU-depth camera mode. The weight distribution part is continuous and includes sensor gain coefficients and resource constraint parameters. The sensor gain coefficients are the weights of the lidar, depth camera, and IMU, ranging from 0 to 1. The expression is as follows: （9） The resource limiting parameter is the lidar sampling rate, which ranges from 5Hz to 40Hz; the expression is as follows: （10） Step 2.3 Reward function module; Contains main rewards, auxiliary rewards and penalties; Step 2.4 Network architecture design; Including the state encoder, as the foundation of the entire network, which is responsible for converting multimodal sensor data into a unified feature representation so that the subsequent policy network and value network can make decisions and evaluations based on these features; The policy network is responsible for outputting specific action selections based on the encoded state features, including discrete action patterns and continuous parameter adjustments; The value network is used to evaluate the pros and cons of the current state and action combination, providing learning signals for the policy network to help it optimize the decision-making process; Step 2.5 executes the training strategy; Through efficient data collection mechanisms, intelligent exploration strategies, and strict safety learning constraints, a comprehensive training framework is provided for the reinforcement learning sensor adaptive selection module. include, The data collection mechanism provides the system with experience data of interactions with the environment, which is the basis for learning and optimizing strategies; Exploration strategy, which determines how the system effectively explores unknown or partially known environments to obtain more information and experience; Safe learning constraints ensure that the system not only pursues high performance but also meets a series of safety and practicality requirements during the learning and decision-making process; Step 3: Multi-source sensor data fusion processing; Based on the data fusion processing module, the sensor combination performs multi-source sensor data fusion processing; Step 4: Map update and optimization; Based on the map update and back-end optimization module, the environment map is updated and optimized using the map data provided by the front-end data fusion processing module in combination with 4DGS, and error correction and map optimization are performed through loop closure detection.

2. The 4D Gaussian splattering-based adaptive multimodal SLAM method according to claim 1, wherein: The step 1 comprises the following steps, Step 1.1 Data input; The SLAM system synchronizes the point cloud data received from the lidar, the RGB image data from the depth camera, and the IMU data in time; Step 1.2: Incorporate time information; Introducing timestamps during data preprocessing ,Each lidar point cloud and image frame is timestamped; Step 1.3 Multi-sensor external parameter calibration; Use the Kalibr framework to calibrate the external parameters between sensors; Step 1.4: Initialize the reinforcement learning module. The reinforcement learning-based sensor selection module is initialized in the data preprocessing stage and learns the strategy of selecting the optimal sensor combination under different scenario conditions by analyzing historical data and environmental characteristics; In the initial state, a random exploration strategy is adopted.

3. The reinforcement learning adaptive multimodal SLAM method based on 4D Gaussian splattering according to claim 1, characterized in that: The step 2.1 includes: Geometric dynamic feature design, including lidar point cloud density gradient and dynamic object coverage; The density gradient of the lidar point cloud reflects the density change of the point cloud in space, helps the system perceive the geometric structure and potential obstacles of the environment, and quantifies the observation confidence of the lidar in the local area. The expression is as follows: （1） in, Indicates the number of lidar hit points within a unit voxel, represents the voxel volume, Indicates the normalized value range of reflection intensity; Calculating the dynamic object coverage rate represents the proportion and distribution of dynamic objects in the scene, so that the system can timely understand the dynamic characteristics of the environment. The expression is as follows: （2） in, represents the estimated velocity vector of the dynamic object, Represents the projected area of the dynamic object detection frame, Indicates the field of view area of the sensor; Sensory reliability, including IMU confidence and radar-visual depth consistency. IMU confidence reflects the credibility of inertial measurement unit data and is calculated based on its noise level and bias stability. The higher the result value, the more reliable the IMU data. The expression is as follows: （3） in, represents the gyroscope bias estimation error, represents the standard deviation of accelerometer noise; Radar-visual depth consistency measures the degree of agreement between the visual sensor and radar depth data. It is calculated by comparing the depth measurements of the two sensors. The higher the consistency, the more coordinated the depth data from the two sensors. The expression is as follows: （4） in, represents the observed depth distribution histogram, represents the depth distribution histogram of the depth camera, It represents the intersection and union ratio of the effective areas of the two; System resource status monitoring, including real-time computing load and remaining energy budget. Real-time computing load reflects the current computing pressure of the system and helps the module decide whether to reduce the amount of sensor data processing or simplify computing tasks. The expression is as follows: （5） in, Indicates the memory capacity currently used by the GPU. Indicates the total GPU memory capacity. Indicates the time taken to render a single frame. Indicates the expected frame period; The remaining energy budget indicates the energy reserve of the system, enabling the module to select a sensor combination or adjust the working mode based on the current remaining energy budget. The expression is as follows: （6） in, Indicates the current remaining battery level. Indicates the initial total power, Indicates the system running time. Indicates the power attenuation factor; Spatiotemporal context includes historical decision memory and scene category probability. Historical decision memory records the success rate and effect of past sensor selection decisions, helping the module learn from experience, avoid repeated errors, and improve decision efficiency. The expression is as follows: （7） in, Indicates the past Step action, represents the long short-term memory network; The scene category probability is calculated based on the current environment characteristics, and the probability of belonging to low-texture scenes, limited-viewing-constrained scenes, and dynamic scenes is calculated, so that the module can adjust the sensor selection strategy according to the scene characteristics. The expression is as follows: （8） in, represents the joint image-text embedding vector extracted by the CLIP model, represents the learnable weight matrix, To normalize the distribution, the sum is 1.

4. The method of claim 1 , wherein: The step 2.3 includes, The main rewards include positioning accuracy rewards and energy efficiency rewards, which are used to improve the accuracy of system positioning and energy efficiency respectively. The expressions are as follows: （11） （12） Where ATE represents the absolute trajectory error, represents the total instantaneous power consumption, Indicates the reward compensation item when the remaining power exceeds the threshold; Auxiliary rewards, including rendering quality rewards and policy stability rewards, are designed to improve the visual quality of scene reconstruction and ensure the continuity and reliability of the policy; the expressions are as follows: （13） （14） Among them, SSIM represents the structural similarity index, LPIPS represents the learning-perceptual image block similarity, Indicates the penalty for action mutation, Represents the smoothness constraint of the policy network parameters; Penalty terms, including fatal penalty errors and resource overload penalties, are used to avoid unreasonable sensor selection and excessive resource consumption, thereby guiding the system to learn the optimal sensor selection strategy and achieve comprehensive optimization of system performance. The expression is as follows: （15） （16） The formula for calculating the comprehensive reward is as follows: (17)。 5. The 4D Gaussian splattering-based adaptive multimodal SLAM method according to claim 1, wherein: The step 3 comprises the following steps, The reinforcement learning sensor selection module adaptively selects the most suitable sensor mode based on the current environmental conditions and system status, considering the impact of various parameters on sensor mode selection. The weight function expression is as follows: + + + + + (21) in, , , , , and Respectively represent the lidar confidence , dynamic object coverage , IMU confidence , radar vision consistency , computing power load and energy remaining budget High weight; Step 3.1 Depth Camera-IMU Mode; When the reinforcement learning sensor selection module is based on the current environment characteristics, according to the weight function S≥ , To determine the threshold, the SLAM system selects the sensor combination of depth camera and IMU, with the depth camera as the main and IMU as the auxiliary; Step 3.2 LiDAR-Depth Camera Mode; When the reinforcement learning sensor selection module is based on the current environment characteristics, according to the weight function , To determine the threshold, the SLAM system selects a sensor combination of lidar and depth camera, with the depth camera as the main sensor and the lidar as the auxiliary sensor. Step 3.3 LiDAR-IMU-Depth Camera Multi-Sensor Balance Mode; When the reinforcement learning sensor selection module is based on the current environment characteristics, the weight function is used according to the weight function , determine whether the SLAM system should use a sensor mode that combines lidar, depth camera and IMU; Step 3.4 After the reinforcement learning sensor selection module executes the action and interacts with the environment, the reward function module records the historical decision memory, and the value network is evaluated to optimize the update of the entire policy network.

6. The 4D Gaussian splattering-based adaptive multimodal SLAM method according to claim 5, characterized in that: The step 3.1 includes, Step 3.1.1 Data input and key frame selection; Prioritize frames containing new scene structures or significant features as key frames to improve the integrity and efficiency of the map; Step 3.1.2 uses generalized ICP tracking for point cloud registration; Select RGB image of the frame and depth image , generate point cloud , where each point , and calculate the covariance matrix for each point ; Estimate the source point cloud of the current frame through the GICP algorithm and target map point cloud Relative pose transformation ; Step 3.1.3 IMU pre-integration; By fusing IMU high-frequency measurements, we can generate inter-frame motion pre-integration as the initial guess for GICP, which improves tracking efficiency and combines with the GICP algorithm to enhance the robustness and accuracy of the system. Step 3.1.4: Update pre-integration; The GICP optimization results are used to correct the drift of IMU integral caused by noise and bias over time.

7. The method of claim 5 , wherein: Said step 3.2 comprises the following steps, Step 3.2.1 Data input: images from depth camera and point cloud from lidar; The time-aligned LiDAR point cloud is converted into a depth image using the calibrated external signal for integration; the expression is as follows: （38） in, is the lidar point cloud, and are the rotation matrix and translation vector from the laser radar to the camera coordinate system, is the intrinsic matrix of the camera; Step 3.2.2 uses the incremental error minimization function; To ensure the exact correspondence between the plane and the point, the formula is as follows: （39） in, Represents a lidar point cloud A point in It is based on the current posture estimation from the previous moment to the world coordinate system The result after iterations, is the closest The Gaussian center of yes The normal vector of Yes The weight of is a regularization term used to enhance the stability and accuracy of the error function, taking into account the directional error between normal vectors; Introducing regularization terms To enhance the stability and accuracy of the error function and take into account the error in the normal direction; the expression is as follows: (40) in, is the normal of the current Gaussian distribution; Step 3.2.3 weight function calculation; The calculation steps of the weight function are as follows: a. Determine the Gaussian center within the local spherical region: Find all the nearest Gaussian distribution centers within The center of the ball. is the radius; b. Calculate the density function of Gaussian points, density function Calculated by the following formula: （41） in, is the reconstructed covariance matrix, which is obtained by selecting the minimum variance along the normal direction and larger variance in the vertical direction to build; c. Simplify the calculation of density function. In order to speed up the calculation, simplify the calculation of density function during the tracking process: （42） d. Consistency calculation, for each point , calculate the normal of the current Gaussian distribution and the local mean normal consistency ; e. Complex texture calculation: for each radar point corresponding to the image area, calculate the local texture complexity; Project it to the camera's pixel coordinate system to get its position in the image , intercepted with this as the center image blocks, where =16, convert the image block to grayscale and calculate the variance of pixel intensity: （43） in, is the pixel intensity, Image block mean; The variance is converted to a 0-1 texture weight through the sigmoid function: （44） in, is a scaling factor used to adjust the variance sensitivity; f. Final weight function, define the final weight function is the product of normal consistency, density function and texture complexity, that is, .

8. The 4D Gaussian splattering-based reinforcement learning adaptive multimodal SLAM method according to claim 5, characterized in that: Said step 3.3 comprises the following steps, Step 3.3.1 Data input and hardware synchronization; The LiDAR provides a sparse but highly accurate 3D point cloud, the depth camera captures the RGB texture of the scene, and the IMU outputs angular velocity and acceleration at high frequency for motion prediction. This ensures that the data timestamps of the three are strictly aligned to avoid timing drift. Step 3.3.2: Select keyframes using depth camera input. Select representative frames from the continuous data stream as key frames to reduce redundant calculations; Step 3.3.3IMU data is used for state propagation; Based on the input IMU data and the state of the previous key frame, the current state is predicted through IMU pre-integration, which is used for state estimation and forward prediction of motion distortion removal; Step 3.3.4 dedistorts the lidar input; Using the continuous pose predicted by IMU, each lidar point Convert from the local coordinate system at the time of scanning to the global coordinate system; the expression is as follows: （45） in, Indicates a point The collection timestamp, Indicates the value obtained through IMU interpolation The posture of the moment.

9. The 4D Gaussian splattering-based adaptive multimodal SLAM method according to claim 1, wherein: The step 4 comprises the following steps, Step 4.1 Sliding window maintenance; Step 4.2 4D Gaussian distribution; Step 4.3 introduces optical flow to solve the 4D GS overfitting problem; Step 4.4 Map update and optimization; Step 4.5 loop detection; By extracting lidar and visual features and generating feature descriptors, the module is able to detect potential loop closure candidates; Confirm loop closure assumptions using geometric verification and consistency checks; The verified loop constraints are fed back into the map optimization process to globally optimize the map structure.

Images