Dynamic semantic SLAM (Simultaneous Localization and Mapping)-driven robot full-life-cycle navigation method

Abstract

The invention discloses a dynamic semantic SLAM (Simultaneous Localization and Mapping)-driven robot full life cycle navigation method, which comprises the following steps: synchronously acquiring multi-modal data through a multi-modal sensor array, and carrying out joint coding based on a cross-modal contrast learning framework to obtain embedded features. A dynamic three-dimensional Gaussian radiation field is constructed, a scene is represented as a parameterized Gaussian kernel set, and mixed Gaussian field representation is obtained through micro rendering and joint training optimization. A Gaussian mixture field is divided into a global static layer and a local dynamic layer, retention constraints are applied to the static layer, and an incremental online updating strategy is adopted for the dynamic layer. Based on the optimized Gaussian field parameters and the multi-modal data, a factor graph containing Gaussian rendering residual factors and inertia pre-integration factors is constructed and solved, the corrected robot pose is obtained, and the space occupancy probability is generated. Operation control parameters are adjusted on line according to the pose and the occupancy probability through a strategy network, and autonomous navigation is achieved in combination with path planning and trajectory tracking.

Description

Technical Field

[0001] This invention belongs to the field of robot autonomous navigation and scene understanding technology, specifically involving a dynamic semantic SLAM-driven robot full lifecycle navigation method. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, autonomous navigation and scene understanding have gradually become core capabilities for intelligent agents to achieve long-term reliable operation in complex and dynamic environments. Current mainstream Simultaneous Localization and Mapping (SLAM) methods are mostly based on static environment assumptions and rely on linear modeling techniques such as geometric feature matching and Kalman filtering. These methods have significant shortcomings in handling dynamic object interference, sensor noise accumulation, and model degradation caused by long-term operation.

[0003] Improved solutions for dynamic environments, such as DS-SLAM and ORB-SLAM3-Dynamic, typically combine semantic segmentation or optical flow tracing techniques to isolate dynamic regions. However, their semantic modeling largely relies on predefined categories and lacks the ability to incrementally learn semantics from unknown objects in open scenes. Furthermore, the motion state estimation of dynamic regions is susceptible to occlusion, false detections, and noise from multiple sensor sources, leading to… Figure 1 The reduced consistency and insufficient positioning robustness make it difficult to meet the operational requirements of robots for long-term tasks.

[0004] On the other hand, in recent years, dense scene reconstruction methods based on neural radiation fields (NeRF) have been able to generate high-fidelity 3D models, demonstrating superior detail representation capabilities compared to traditional sparse feature methods. However, these methods employ implicit representations, resulting in high computational costs for parameter optimization, making real-time performance difficult. Furthermore, they lack explicit modeling capabilities for semantics and motion in dynamic scenes, thus hindering the robot's ability to continuously optimize environmental cognition during long-term tasks.

[0005] 3D Gaussian Splatting technology models scene geometry and appearance using an explicit set of Gaussian distributions, offering advantages such as efficient parallel rendering and dynamic parameter updates, making it a research hotspot for real-time scene reconstruction. However, existing research largely focuses on rapid modeling of static scenes, failing to address the deep integration of dynamic semantic information and Gaussian parameters, and lacking a hierarchical storage and incremental update mechanism suitable for long-term tasks.

[0006] Meanwhile, lifelong learning methods have mitigated the catastrophic forgetting problem of models to some extent through strategies such as knowledge distillation and flexible weight solidification. However, their integration with dynamic SLAM still faces several challenges, such as spatiotemporal alignment of cross-modal data, continuous updating of dynamic semantic labels, and collaborative optimization of perception, mapping, and knowledge transfer under limited computing resources. Especially in application scenarios covering the entire robot lifecycle, the system must achieve continuous evolution of navigation, obstacle avoidance, and semantic understanding in dynamic environments while ensuring real-time performance. This places higher demands on the lightweight design of the algorithm architecture and its multi-objective optimization capabilities.

[0007] In the prior art, Chinese patent CN119780955A discloses an autonomous navigation method and related equipment based on laser and vision fusion. The method includes: performing multi-sensor spatiotemporal joint synchronization processing on laser point cloud data, binocular image data, and clock signals to generate a spatiotemporally aligned multimodal data stream; fusing the multimodal data stream with cross-modal feature pyramids and attention guidance to generate a dense environment representation and a spatiotemporal mask for dynamic objects; determining a multi-layer semantic map corresponding to the dense environment representation through hierarchical factor graph optimization; fusing the multi-layer semantic map with a neural radiation field to generate a hybrid map of explicit semantic map and implicit radiation field; and determining the motion trajectory of the target device through spatiotemporal joint constraints on the hybrid map and the spatiotemporal mask for dynamic objects.

[0008] However, this method has the following limitations: 1. The processing of dynamic semantic regions mainly relies on mask segmentation and spatiotemporal filtering, lacking deep integration with map representation, making it difficult to support robust localization in complex dynamic scenarios; 2. It has not established an incremental semantic learning and lifetime optimization mechanism, which cannot solve the problems of unknown object categories, semantic updates, and knowledge transfer encountered by robots in long-term tasks; 3. Its factor graph optimization and trajectory planning are mostly limited to static or quasi-static constraints, failing to achieve adaptive control based on dynamic trade-offs between positioning accuracy, reconstruction quality, and computing resources during runtime.

[0009] Therefore, existing technologies lack a unified framework that integrates dynamic semantic SLAM and lifelong learning capabilities to support robots in achieving long-term stable localization and mapping, adaptive environmental understanding, and autonomous navigation throughout their entire lifecycle in complex dynamic environments. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a dynamic semantic SLAM-driven robot full lifecycle navigation method.

[0011] The objective of this invention can be achieved through the following technical solutions:

[0012] This invention provides a dynamic semantic SLAM-driven robot lifecycle navigation method, comprising the following steps:

[0013] Deploy a multimodal sensor array to synchronously acquire multimodal data;

[0014] The multimodal data is jointly encoded based on a cross-modal contrastive learning framework to generate joint embedding features that represent the geometric, semantic, and motion relationships of the scene.

[0015] Based on the joint embedding features, a dynamic three-dimensional Gaussian radiation field representation is constructed, and the scene is represented as a parameterizable set of Gaussian kernels and the Gaussian kernel parameters are initialized.

[0016] The Gaussian kernel set is optimized using a differentiable rendering and joint training strategy to obtain a hybrid Gaussian field representation for localization and reconstruction.

[0017] The mixed Gaussian field is stored and parameters are separated into a global static layer and a local dynamic layer. Retention constraints are applied to the static layer, and an incremental online update strategy is adopted for the dynamic layer to adapt to newly emerging dynamic elements, so as to obtain Gaussian field parameters optimized by lifelong learning.

[0018] Based on optimized Gaussian field parameters and multimodal data, a factor graph containing Gaussian rendering residual factors and inertial pre-integration factors is constructed and solved to obtain the corrected robot pose, and space occupancy probability is generated based on the Gaussian field.

[0019] Based on the corrected robot pose and space occupancy probability, a policy network is used to adjust the operation control parameters online, and autonomous navigation is achieved based on the adjusted operation control parameters.

[0020] Furthermore, the multimodal sensor array includes an RGB-D camera, a lidar, and an inertial measurement unit;

[0021] The multimodal data includes RGB images acquired by the RGB-D camera, laser point clouds and their three-dimensional coordinates acquired by the lidar, and three-dimensional linear acceleration and angular velocity measurements acquired by the inertial measurement unit, and includes timestamp information for time synchronization of data from each sensor.

[0022] Furthermore, the joint encoding of the multimodal data based on the cross-modal contrastive learning framework to generate joint embedding features representing the geometric, semantic, and motion associations of the scene specifically includes:

[0023] Modality-specific feature extraction is performed on the multimodal data at time t, and the RGB image I is... t Laser point cloud P t and inertial measurement unit data ξ tThe modal feature vectors are fed into their respective modal encoders and then fused using a cross-modal attention mechanism to obtain the joint embedding feature F. multi , represented as:

[0024] F multi =CrossAttn(Enc RGB (I t ),Enc LiDAR (P t ),Enc IMU (ξ t ))

[0025] Among them, F multi For joint embedding features, CrossAttn(·) denotes an attention fusion operation based on the cross-modal similarity matrix; Enc RGB (·), Enc LiDAR (·), Enc IMU (·) represent the encoders for the corresponding modes.

[0026] Furthermore, the loss function of the modal encoder is:

[0027]

[0028] Among them, L align For the joint contrastive learning loss function, z t = CLIP(F multi ) represents the cross-modal contrastive embedding vector obtained from the joint embedding feature map; CLIP(·) represents the cross-modal embedding model, z CLIP (I t ,P t ) represents the reference cross-modal embedding vector generated from the RGB image and the laser point cloud, λ motion The weighting coefficients of the motion consistency loss term, v t v represents the dynamic target velocity vector estimated based on multimodal data. GT Indicates the reference speed of a dynamic target.

[0029] Furthermore, the construction of a dynamic three-dimensional Gaussian radiation field representation based on the joint embedding features, representing the scene as a parameterizable set of Gaussian kernels and initializing the Gaussian kernel parameters, specifically includes:

[0030] From the joint embedding feature F multi Extracting local sample sets Each sampling point s j ∈ At a spatial location, there is a local embedding vector e obtained from the joint embedding feature map. j =φ(F multi,s j ), where φ(·,·) is the mapping function that jointly embeds the mapping into the local representation of the sampling point; based on {(s j ,e j The sampling points are divided into clusters to obtain a cluster set. Among them, s j The coordinates of the j-th 3D point are obtained from the point cloud data corresponding to the joint embedding feature space; M represents the total number of local sampling points; e j The joint embedding feature F represents the characteristic of the joint embedding feature F. multi Spatial location s of the sampling point j The local embedding vectors obtained from the mapping are used to represent the geometric, semantic, and motion properties of the points; C i Let N represent the i-th cluster, and N represent the number of Gaussian kernels obtained through clustering.

[0031] According to cluster C i For the i-th Gaussian kernel Initialize the initial parameters.

[0032] Furthermore, the optimization of the Gaussian kernel set using a differentiable rendering and joint training strategy to obtain a mixture of Gaussian fields for localization and reconstruction specifically includes:

[0033] Represent the scene as a set of Gaussian kernels. And for each Gaussian kernel Define the rendering radiation field as:

[0034]

[0035] Where C(r) represents the rendered radiance value at point r for each ray, r represents the ray sampling point, and N represents the number of Gaussian kernels obtained through clustering; σ i μ represents the opacity of the i-th Gaussian kernel. i Σ i Let c represent the spatial mean and covariance matrix of the i-th Gaussian kernel, respectively. i Represents the semantic embedding vector of the i-th Gaussian kernel;

[0036] Based on the Gaussian rendering results, a joint optimization loss function is constructed:

[0037] L render =ω1L photo +ω2L geom +ω3L motion

[0038] Among them, L render This represents the joint optimization loss function, where ω1, ω2, and ω3 are preset weights, and L... photo L represents the photometric error loss function. geomL represents the geometric consistency error loss function. motion This represents the motion compensation error loss function;

[0039] Based on the joint optimization loss function for the Gaussian kernel parameter set Training and optimization are performed to obtain a mixed Gaussian field representation for robot localization and scene reconstruction.

[0040] Furthermore, the photometric error loss function is defined by the following formula:

[0041]

[0042] Among them, I input Represents the pixel values ​​of the input reference image; Represents the L2 norm;

[0043] The geometric consistency error loss function is formulated as follows:

[0044]

[0045] Where, n i Let p represent the surface normal vector of the i-th Gaussian kernel, and p represent the spatial sampling point;

[0046] The motion compensation error loss function is formulated as follows:

[0047]

[0048] Among them, v i Let v represent the velocity vector estimated from the dynamic elements corresponding to the i-th Gaussian kernel. GT The reference velocity vector represents the dynamic target.

[0049] Furthermore, the process of storing and separating parameters of the mixed Gaussian field into a global static layer and a local dynamic layer, applying retention constraints to the static layer, and employing an incremental online update strategy for the dynamic layer to adapt to newly emerging dynamic elements, thereby obtaining Gaussian field parameters optimized through lifelong learning, specifically includes:

[0050] The mixed Gaussian field parameter set is divided into a static parameter set Θ. static With dynamic parameter set Θ dyn Furthermore, an elastic weighting constraint is applied to the static parameter set, and its regularized form is expressed as:

[0051]

[0052] Where, θ i Let be the parameter vector of the i-th static Gaussian kernel. The corresponding parameter vector saved for the previous task; F iLet G be the Fisher information matrix for the i-th parameter. static Represents a static parameter index set, by using L EWC Incorporating overall objective constraints into static parameter updates suppresses forgetting;

[0053] An update strategy combining incremental online adaptation based on meta-learning and knowledge distillation is adopted for the dynamic parameter set, where the meta-learning loss is expressed as:

[0054]

[0055] in, T represents the distribution of meta-tasks. support With T query L represents the support set and query set, respectively. task (·,·) represents the dynamic semantic segmentation and motion prediction loss for a single task, U(Θ) dyn ,T support The inner fast-adaptive update operator is represented as:

[0056]

[0057] Where α is the inner layer update learning rate;

[0058] The rendering / reconstruction loss is combined with static preservation constraints and dynamic layer updates to form the overall online loss function:

[0059] L total =L render +βL EWC +γL MAML

[0060] Among them, L total Let L be the overall online loss function. render The loss function is used for joint optimization, where β and γ are weighting coefficients.

[0061] By analyzing L total Minimize the joint update of Θ static With Θ dyn Gaussian field parameters optimized through lifelong learning were obtained.

[0062] Furthermore, based on the optimized Gaussian field parameters and multimodal data, a factor map containing Gaussian rendering residual factors and inertial pre-integration factors is constructed and solved to obtain the corrected robot pose, and the space occupancy probability is generated based on the Gaussian field, specifically including:

[0063] The visual-inertial joint cost of the factor graph is represented as:

[0064]

[0065] Among them, L SLAM The joint cost of the factor graph is represented by M, where M represents the number of visual frames or factors involved in the optimization, and I represents the joint cost of the factor graph. j This represents the RGB image of the j-th frame in the multimodal data. Indicates pose T j With Gaussian field Differentiable rendering result; α represents the weight coefficient, T represents the odometry transformation obtained by IMU pre-integration. j This represents the pose transformation in the j-th frame. Denotes the norm on the SE(3) Lie algebra space;

[0066] By minimizing L SLAM Pose sequence By jointly solving with the necessary Gaussian field local parameters, the corrected set of robot poses is obtained.

[0067] A space occupancy probability field is constructed based on Gaussian kernel parameters, and the occupancy probability function is defined as follows:

[0068]

[0069] in, Let φ(·) represent the occupancy probability of spatial location x, φ(·) represent the encoder that maps spatial location x to a feature vector, w represent the linear weight vector of the occupancy probability mapping, and σ(·) represent the Sigmoid function.

[0070] Based on occupancy probability and corrected pose, define safety boundary constraints for dynamic obstacle avoidance:

[0071]

[0072] Among them, L safe Indicates safety boundary constraints. Let γ represent the set of sampled positions on the candidate trajectory, γ represent the weight vector, and v represent the robot's corresponding velocity vector. safe This indicates the preset safe speed threshold;

[0073] L SLAM With L safe By incorporating a nonlinear least squares solution process, the pose TTT and necessary local Gaussian field parameters are iteratively optimized to output a corrected robot pose T for navigation and obstacle avoidance. * And the real-time space occupancy probability based on Gaussian fields.

[0074] Furthermore, the process of adjusting operational control parameters online using a policy network based on the corrected robot pose and space occupancy probability, and achieving autonomous navigation based on the adjusted operational control parameters, specifically includes:

[0075] Based on the obtained corrected robot pose T * Spatial occupancy probability generated based on mixed Gaussian fields The change in positioning accuracy ΔATE was calculated using windowed statistics and rendering / occupancy assessment processing, respectively. t ΔChamfer, a change in reconstruction quality t Real-time performance metrics FPS t Construct the current state vector s t ={ΔATE t ,ΔChamfer t FPS t};

[0076] Define the action space:

[0077] a t ={ρ t ,τ t}

[0078] Where, ρ t τ represents the Gaussian rendering resolution. t Indicates the frequency of knowledge distillation execution;

[0079] Based on the current state vector s t The Gaussian kernel is output through the policy network to render the resolution ρ. t =π(s) t );

[0080] Based on the obtained Gaussian kernel rendering resolution ρ t Adjust the rendering resolution and sampling rate of the differentiable rendering module and the distillation scheduling of the online learning module, and based on The strategy output actions are subject to safety screening based on the updated rendering resolution and the corrected pose T. * Perform path planning and trajectory tracking to achieve adaptive autonomous navigation.

[0081] Compared with the prior art, the present invention has the following advantages:

[0082] (1) In the prior art, traditional SLAM methods are easily affected by moving targets in complex dynamic environments, leading to decreased positioning accuracy and trajectory drift. This invention acquires RGB-D images, laser point clouds and IMU data through a multimodal sensor array, and introduces Gaussian rendering residuals and inertial pre-integration factors in factor graph optimization to improve the joint constraint capability of multimodal information, thereby effectively suppressing the impact of dynamic interference on positioning and achieving high-precision and robust robot pose estimation in dynamic scenes.

[0083] (2) In the prior art, single-modal encoding is prone to separation of geometric, semantic and motion features, resulting in insufficient dynamic semantic understanding. This invention uses a cross-modal contrastive learning framework to encode RGB, point cloud and IMU features and then fuse them using a cross-modal attention mechanism to obtain joint embedded features that include geometric, semantic and motion associations. This solves the problem of semantic decoupling from geometry and motion, thereby improving the consistency and completeness of dynamic target detection and environment modeling, and realizing a higher level of dynamic semantic mapping capability.

[0084] (3) In the prior art, the 3D representation of a scene mostly adopts dense point clouds or voxels, which have large parameter dimensions and are difficult to update quickly, especially in dynamic environments, resulting in insufficient efficiency in modeling and rendering. This invention represents the scene as a parameterizable 3D Gaussian kernel set and initializes it based on joint embedding features. By optimizing the Gaussian kernel parameters through differentiable rendering and joint training, the scene reconstruction becomes more compact and efficient, which not only improves the rendering and reconstruction quality but also ensures real-time performance in dynamic environments.

[0085] (4) In the prior art, SLAM systems are prone to "catastrophic forgetting" during long-term operation, that is, losing the memory of the old environment when adapting to a new task or new environment, resulting in insufficient autonomous navigation capability throughout the entire life cycle. This invention solves the forgetting problem in long-term operation by dividing the mixed Gaussian field parameters into a static layer and a dynamic layer. The static layer uses elastic weight solidification constraints to maintain historical knowledge, while the dynamic layer adopts an optimization strategy that combines incremental online updates based on meta-learning with knowledge distillation. This effectively solves the forgetting problem in long-term operation, realizes lifelong learning and adaptive updates of the Gaussian field representation, and ensures the robot's continuous adaptability in complex dynamic environments.

[0086] (5) In the prior art, robot navigation relies on fixed control parameters and lacks adaptive adjustment to environmental changes and system operating states, which easily leads to decreased positioning accuracy, unstable reconstruction quality, or insufficient real-time performance. This invention constructs a state vector (positioning accuracy change, reconstruction quality change, and real-time performance index) and uses a policy network to output Gaussian kernel rendering resolution and knowledge distillation execution frequency to achieve online adaptive adjustment of operating control parameters. Combined with space occupancy probability for safety screening, the navigation strategy is dynamically optimized under the premise of ensuring safety, effectively solving the problems of control rigidity and performance fluctuation, and realizing autonomous navigation with high precision, high real-time performance, and high safety.

[0087] (6) In the prior art, obstacle avoidance in dynamic environments relies on local perception and lacks modeling of global spatiotemporal occupancy information, resulting in insufficient obstacle avoidance safety. This invention generates a spatial occupancy probability field based on a Gaussian field, defines an occupancy probability function and introduces safety boundary constraints, and jointly optimizes the occupancy information with the corrected pose, thereby achieving globally-locally consistent obstacle avoidance constraints in path planning and trajectory tracking. This effectively solves the problem of unstable dynamic obstacle avoidance and enables safe and reliable navigation of the robot in dynamic and complex environments. Attached Figure Description

[0088] Figure 1 This is a flowchart of the autonomous navigation method according to an embodiment of the present invention;

[0089] Figure 2 This is a system architecture diagram according to an embodiment of the present invention;

[0090] Figure 3 This is a flowchart of the dynamic 3D Gaussian radiation field modeling process according to an embodiment of the present invention;

[0091] Figure 4 This is a schematic diagram of the hierarchical incremental lifelong learning and knowledge transfer algorithm according to an embodiment of the present invention;

[0092] Figure 5 This is a schematic diagram of the closed-loop optimization and navigation decision-making mechanism driven by reverse rendering in an embodiment of the present invention. Detailed Implementation

[0093] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0094] Example 1:

[0095] This embodiment provides a dynamic semantic SLAM-driven method for robot full lifecycle navigation, such as... Figure 1 , Figure 2 As shown, it includes the following steps:

[0096] Step S1: Deploy a multimodal sensor array to synchronously acquire multimodal data; wherein, the multimodal sensor array includes an RGB-D camera, a LiDAR, and an inertial measurement unit; the multimodal data includes RGB images acquired by the RGB-D camera, laser point clouds and their three-dimensional coordinates acquired by the LiDAR, and three-dimensional linear acceleration and angular velocity measurements acquired by the inertial measurement unit, and includes timestamp information for time synchronization of data from each sensor.

[0097] Step S2: Based on the cross-modal contrastive learning framework, the multimodal data is jointly encoded to generate joint embedding features representing the geometric, semantic and motion associations of the scene;

[0098] Step S2 includes: deploying a multimodal sensor array and synchronously acquiring RGB images. LiDAR point cloud and IMU data The lightweight multi-task Transformer network DynamicViT-B0 is used to extract spatiotemporal features:

[0099] F multi =CrossAttn(Enc RGB (I t ),Enc LiDAR (P t ),Enc IMU (ξ t ))

[0100] Here, Enc. is a modality-specific encoder, CrossAttn is a cross-modal attention mechanism, and the fusion weights are the cross-modal similarity matrix generated by the CLIP encoder. Dynamic adjustment.

[0101] Introducing contrastive learning loss Constrain the consistency between the motion trajectory of dynamic objects and their semantic labels:

[0102]

[0103] Among them, z t = CLIP(F multi ) is a cross-modal embedding vector, v t λ is the velocity of the dynamic object estimated by Kalman filtering. motion =0.7 is the motion consistency weight.

[0104] Step S3: Construct a dynamic 3D Gaussian radiation field representation based on joint embedding features, represent the scene as a parameterizable set of Gaussian kernels and initialize the Gaussian kernel parameters;

[0105] Step S3 includes: the dynamic Gaussian field based on step S2. Design a hierarchical incremental lifelong learning framework, using a global static Gaussian kernel. With local dynamic Gaussian kernel Separate the model, use elastic weights to solidify static parameters, and update dynamic parameters through meta-learning strategies and course distillation mechanisms to suppress catastrophic forgetting;

[0106] Step S4: Optimize the Gaussian kernel set using a differentiable rendering and joint training strategy to obtain a mixed Gaussian field representation for localization and reconstruction;

[0107] Specifically, it includes:

[0108] Represent the scene as a set of Gaussian kernels. And for each Gaussian kernel Define the rendering radiation field as:

[0109]

[0110] Where C(r) represents the rendered radiance value at point r for each ray, r represents the ray sampling point, and N represents the number of Gaussian kernels obtained through clustering; σ i μ represents the opacity of the i-th Gaussian kernel. i Σ i Let c represent the spatial mean and covariance matrix of the i-th Gaussian kernel, respectively. i Represents the semantic embedding vector of the i-th Gaussian kernel;

[0111] Based on the Gaussian rendering results, a joint optimization loss function is constructed:

[0112] L render =ω1L photo +ω2L geom +ω3L motion

[0113] Among them, L render This represents the joint optimization loss function, where ω1, ω2, and ω3 are preset weights, and L... photo L represents the photometric error loss function. geom L represents the geometric consistency error loss function. motion This represents the motion compensation error loss function;

[0114] Based on the joint optimization loss function for the Gaussian kernel parameter set Training and optimization are performed to obtain a mixed Gaussian field representation for robot localization and scene reconstruction.

[0115] The photometric error loss function is expressed by the following formula:

[0116]

[0117] Among them, I input Represents the pixel values ​​of the input reference image; Represents the L2 norm;

[0118] The geometric consistency error loss function is given by the following formula:

[0119]

[0120] Where, n i Let p represent the surface normal vector of the i-th Gaussian kernel, and p represent the spatial sampling point;

[0121] The motion compensation error loss function is given by the following formula:

[0122]

[0123] Among them, v i Let v represent the velocity vector estimated from the dynamic elements corresponding to the i-th Gaussian kernel. GT The reference velocity vector represents the dynamic target.

[0124] Step S5: Store the mixed Gaussian field in layers according to the global static layer and the local dynamic layer, and separate the parameters. Apply retention constraints to the static layer and adopt an incremental online update strategy for the dynamic layer to adapt to newly emerging dynamic elements, so as to obtain the Gaussian field parameters optimized by lifelong learning.

[0125] The mixed Gaussian field parameter set is divided into a static parameter set Θ. static With dynamic parameter set Θ dyn Furthermore, an elastic weighting constraint is applied to the static parameter set, and its regularized form is expressed as:

[0126]

[0127] Where, θ i Let be the parameter vector of the i-th static Gaussian kernel. The corresponding parameter vector saved for the previous task; F i Let G be the Fisher information matrix for the i-th parameter. static Represents a static parameter index set, by using L EWC Incorporating overall objective constraints into static parameter updates suppresses forgetting;

[0128] An update strategy combining incremental online adaptation based on meta-learning and knowledge distillation is adopted for the dynamic parameter set, where the meta-learning loss is expressed as:

[0129]

[0130] in, T represents the distribution of meta-tasks. support With T query L represents the support set and query set, respectively. task (·,·) represents the dynamic semantic segmentation and motion prediction loss for a single task, U(Θ) dyn ,T support The inner fast-adaptive update operator is represented as:

[0131]

[0132] Where α is the inner layer update learning rate;

[0133] The rendering / reconstruction loss is combined with static preservation constraints and dynamic layer updates to form the overall online loss function:

[0134] L total =L render +βL EWC +γL MAML

[0135] Among them, L total Let L be the overall online loss function. render The loss function is used for joint optimization, where β and γ are weighting coefficients.

[0136] By analyzing L total Minimize the joint update of Θ static With Θ dyn Gaussian field parameters optimized through lifelong learning were obtained.

[0137] Step S6: Based on the optimized Gaussian field parameters and multimodal data, construct and solve a factor map containing Gaussian rendering residual factors and inertial pre-integration factors to obtain the corrected robot pose, and generate the space occupancy probability based on the Gaussian field, specifically including:

[0138] Constructing a factor graph optimization problem and jointly rendering residuals using Gaussian methods. With IMU pre-integration constraints

[0139]

[0140] Where α = 0.7 represents the IMU weights, and log represents the Lie group logarithmic mapping. The visual reprojection error and IMU motion constraints are unified into a nonlinear least squares problem, which is solved iteratively using the Levenberg-Marquardt algorithm. Compared to the traditional extended Kalman filter method, this improves positioning accuracy.

[0141] Space Occupancy Probability Map Based on Gaussian Kernel (φ is the MLP encoder), generate safety boundary constraints:

[0142]

[0143] Where γ = 0.5 is the velocity constraint weight, v safe The safe speed threshold is defined. Real-time obstacle avoidance in dynamic environments is achieved by combining probabilistic occupancy grids with model predictive control.

[0144] Step S7: Based on the corrected robot pose and space occupancy probability, the operation control parameters are adjusted online using a policy network, and autonomous navigation is achieved based on the adjusted operation control parameters.

[0145] Define the state space s t ={ΔATE,ΔChamfer,FPS}, Action space a t ={ρ t ,τ t}(where ρ t For rendering resolution, τ t (where the distillation frequency is ), the reward function is:

[0146] R(s t ,a t )=ω1·ΔATE -1 +ω2·ΔChamfer -1 -ω3·FPS -1

[0147] Where ω1, ω2, and ω3 are weighting coefficients. This reward function is based on the Pareto optimization idea of ​​multi-objective reinforcement learning, and balances positioning accuracy and real-time performance through the NSGA-II algorithm.

[0148] Based on the current state s t The Gaussian kernel is output through the policy network to render the resolution ρ. t =π(s) t ):

[0149]

[0150] This embodiment relates to a dynamic semantic SLAM-driven method for autonomous navigation and scene understanding throughout the robot's lifecycle. The method includes: collecting spatiotemporally synchronized geometric and dynamic semantic data using a multimodal sensor array; generating joint embedding features with motion semantic associations based on a cross-modal contrastive learning framework to provide multidimensional perceptual input for dynamic Gaussian modeling; constructing a dynamic 3D Gaussian radiation field based on the joint embedding features; optimizing Gaussian parameters through differentiable ray projection and adversarial training to achieve explicit geometric-semantic joint modeling of the dynamic scene; designing a hierarchical incremental lifelong learning framework based on the dynamic Gaussian field, separating the global static Gaussian kernel from the local dynamic Gaussian kernel; using elastic weights to solidify static parameters; and updating dynamic parameters through a meta-learning strategy and a course distillation mechanism to suppress catastrophic forgetting of the model; and constructing a factor graph optimization problem combined with Gaussian rendering residuals and IMU pre-integration constraints. The robot pose is iteratively corrected using a nonlinear optimization algorithm, and dynamic collision prediction constraints are generated based on the spatial occupancy probability map of the Gaussian kernel to achieve closed-loop pose estimation and dynamic obstacle avoidance. Based on the corrected pose and collision prediction results, a reinforcement learning policy network is designed to dynamically adjust the Gaussian rendering resolution and knowledge distillation frequency. An online evaluation module achieves Pareto-optimal allocation of computational resources and navigation accuracy, supporting adaptive optimization throughout the entire lifecycle. Compared with existing technologies, this invention can guarantee the accuracy and stability of long-term localization and mapping, as well as the adaptability of collision prediction and path replanning in dynamic and complex environments.

[0151] Example 2:

[0152] Traditional robot navigation and scene understanding methods are limited by static environment assumptions, making it difficult to distinguish moving objects from static backgrounds in dynamic scenes, leading to persistent discrepancies in semantic understanding and spatial modeling. Furthermore, in long-term tasks, models cannot continuously adapt to new environmental features due to catastrophic forgetting, and the correlation between semantic labels and geometric structures degrades over time. Especially in complex dynamic environments, the dynamic alignment problem between sensor data and spatiotemporal references further exacerbates the risk of localization drift and scene representation failure, restricting the robot's autonomous decision-making capabilities throughout its entire lifecycle.

[0153] This invention proposes a closed-loop optimization framework that deeply integrates dynamic semantic SLAM and lifelong learning. It extracts geometric depth, semantic category, and dynamic motion features in real time using multimodal sensors, constructing a differentiable 3D Gaussian radiation field to explicitly represent dynamic scenes. This method introduces a hierarchical incremental learning mechanism, separating the semantic-geometric association updates of the static background and dynamic foreground. Combined with inverse rendering optimization and knowledge distillation strategies, it dynamically compensates for environmental changes and model forgetting effects. Through the globally consistent modeling of dynamic semantic SLAM and the long-term adaptability of lifelong learning, this invention significantly improves the robot's autonomous understanding of unknown objects and scene changes, ensuring continuous and accurate navigation in complex dynamic environments. Its closed-loop optimization mechanism effectively suppresses model degradation, supporting autonomous operation throughout the entire lifecycle from initial exploration to long-term maintenance, providing a systematic solution for continuous learning and reliable decision-making in dynamic scenes.

[0154] Taking a home service robot as an example, the robot is equipped with an RGB-D camera, solid-state LiDAR, and IMU unit, and fuses multimodal data in real time through a lightweight Transformer network. When the robot moves autonomously indoors, the network uses a cross-modal attention mechanism to align the semantic features and depth information of the RGB image with the geometric contours of the LiDAR point cloud, while simultaneously compensating for the robot's own pose jitter by combining high-frequency motion data from the IMU. For example, in a living room environment, the dynamic optical flow module captures the movement trajectory of a pet that suddenly enters the room in real time, and distinguishes dynamic creatures from static background objects such as sofas by embedding vectors with spatiotemporal features, providing basic support for subsequent semantic mapping and obstacle avoidance decisions.

[0155] Taking home service robots as an example, the robot represents the home environment as a set of differentiable Gaussian distributions. Each Gaussian kernel is bound to a semantic label and a physical attribute. The dynamic 3D Gaussian radiation field modeling process is as follows: Figure 3 As shown, when a moving robotic vacuum cleaner is detected, the system applies motion compensation parameters to its corresponding Gaussian kernel and generates an RGB-D semantic probability map containing dynamic obstacles through a differentiable renderer. For example, the rotating cleaning path of the robotic vacuum cleaner is mapped in real time to the rotation transformation of the Gaussian kernel covariance matrix. Combined with spatial distribution optimization of the LiDAR point cloud, this ensures that the fusion scene representation of dynamic objects and static furniture maintains geometric consistency in the projected view.

[0156] Taking home service robots as an example, to address the layout changes in the home environment over long-term use, the system adopts a hierarchical incremental map update strategy. Its hierarchical incremental lifelong learning and knowledge transfer algorithm is as follows: Figure 4As shown, the Gaussian kernel parameters of the global static map are locked in historical memory through elastic weight solidification technology, while the semantic association topology of local dynamic areas is preserved through knowledge distillation. When the user rearranges the study, the robot automatically recognizes the change in the position of the bookshelves, only fine-tunes the Gaussian kernel of the relevant areas, and loads historical keyframe data through the memory playback module. After comparing the differences between the old and new point clouds, the semantic labels are progressively updated to avoid gaps in scene understanding caused by sudden environmental changes.

[0157] Taking a home service robot as an example, during autonomous navigation, the inverse rendering closed-loop correction function optimizes the robot's pose estimation in real time. Its inverse rendering-driven closed-loop optimization and navigation decision-making mechanism is as follows: Figure 5 As shown, when approaching the kitchen counter, the system constructs a probabilistic graphical model with the robot's motion state as a latent variable. By comparing the real-time observed distribution of tableware with the semantic-geometric layout predicted by the radiation field, it decouples dynamic disturbances from its own odometry cumulative error. For example, after iteratively correcting the pose offset through reverse rendering, the path planning module combines the space occupancy probability of the dynamic Gaussian kernel to generate a smooth trajectory that bypasses temporary obstacles in real time, and continuously updates the semantic position information of the obstacles during the bypass process.

[0158] Taking home service robots as an example, the full lifecycle adaptive optimization module is integrated throughout the long-term operation and maintenance of home services. The system dynamically adjusts the resolution of Gaussian splash modeling and the weights of multi-sensor fusion based on scene complexity and task requirements. For instance, it automatically increases the sampling density of the LiDAR point cloud to capture wall details when navigating narrow corridors, while reducing the priority of RGB image processing in open living rooms to save computing power. When multiple positioning deviations are detected, it autonomously enhances the weight coefficients of IMU data in factor graph optimization, achieving full lifecycle performance balance from daily cleaning to sudden obstacle avoidance, ensuring the robot gradually adapts to the evolution of the home environment during continuous use.

[0159] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A dynamic semantic SLAM-driven robot lifecycle navigation method, characterized in that, Includes the following steps: Deploy a multimodal sensor array to synchronously acquire multimodal data; The multimodal data is jointly encoded based on a cross-modal contrastive learning framework to generate joint embedding features that represent the geometric, semantic, and motion relationships of the scene. Based on the joint embedding features, a dynamic three-dimensional Gaussian radiation field representation is constructed, and the scene is represented as a parameterizable set of Gaussian kernels and the Gaussian kernel parameters are initialized. The Gaussian kernel set is optimized using a differentiable rendering and joint training strategy to obtain a hybrid Gaussian field representation for localization and reconstruction. The mixed Gaussian field is stored and parameters are separated into a global static layer and a local dynamic layer. Retention constraints are applied to the static layer, and an incremental online update strategy is adopted for the dynamic layer to adapt to newly emerging dynamic elements, so as to obtain Gaussian field parameters optimized by lifelong learning. Based on optimized Gaussian field parameters and multimodal data, a factor graph containing Gaussian rendering residual factors and inertial pre-integration factors is constructed and solved to obtain the corrected robot pose, and space occupancy probability is generated based on the Gaussian field. Based on the corrected robot pose and space occupancy probability, a policy network is used to adjust the operation control parameters online, and autonomous navigation is achieved based on the adjusted operation control parameters.

2. The dynamic semantic SLAM-driven robot lifecycle navigation method according to claim 1, characterized in that, The multimodal sensor array includes an RGB-D camera, a lidar, and an inertial measurement unit; The multimodal data includes RGB images acquired by the RGB-D camera, laser point clouds and their three-dimensional coordinates acquired by the lidar, and three-dimensional linear acceleration and angular velocity measurements acquired by the inertial measurement unit, and includes timestamp information for time synchronization of data from each sensor.

3. The dynamic semantic SLAM-driven robot lifecycle navigation method according to claim 1, characterized in that, The multimodal data is jointly encoded using a cross-modal contrastive learning framework to generate joint embedding features representing the geometric, semantic, and motion relationships of the scene, specifically including: Modality-specific feature extraction is performed on the multimodal data at time t, and the RGB image I is... t Laser point cloud P t and inertial measurement unit data ξ t The modal feature vectors are fed into their respective modal encoders and then fused using a cross-modal attention mechanism to obtain the joint embedding feature F. multi , represented as: F multi =CrossAttn(Enc RGB (I t ),Enc LiDAR (P t ),Enc IMU (ξ t )) Among them, F multi For joint embedding features, CrossAttn(·) denotes an attention fusion operation based on the cross-modal similarity matrix; Enc RGB (·), Enc LiDAR (·), Enc IMU (·) represent the encoders for the corresponding modes.

4. The dynamic semantic SLAM-driven robot lifecycle navigation method according to claim 3, characterized in that, The modal encoder has the following loss function: Among them, L align For the joint contrastive learning loss function, z t = CLIP(F multi ) represents the cross-modal contrastive embedding vector obtained from the joint embedding feature map; CLIP(·) represents the cross-modal embedding model, z CLIP (I t ,P t ) represents the reference cross-modal embedding vector generated from the RGB image and the laser point cloud, λ motion The weighting coefficients of the motion consistency loss term, v t v represents the dynamic target velocity vector estimated based on multimodal data. GT Indicates the reference speed of a dynamic target.

5. The dynamic semantic SLAM-driven robot lifecycle navigation method according to claim 1, characterized in that, The construction of a dynamic three-dimensional Gaussian radiation field representation based on the joint embedding features, which represents the scene as a parameterizable set of Gaussian kernels and initializes the Gaussian kernel parameters, specifically includes: From the joint embedding feature F multi Extracting local sample sets Each sampling point At a spatial location, there is a local embedding vector e obtained from the joint embedding feature map. j =φ(F multi ,s j ), where φ(·,·) is the mapping function that jointly embeds the mapping into the local representation of the sampling point; based on {(s j ,e j The sampling points are divided into clusters to obtain a cluster set. Among them, s j The coordinates of the j-th 3D point are obtained from the point cloud data corresponding to the joint embedding feature space; M represents the total number of local sampling points; e j The joint embedding feature F represents the characteristic of the joint embedding feature F. multi Spatial location s of the sampling point j The local embedding vectors obtained from the mapping are used to represent the geometric, semantic, and motion properties of the points; C i Let N represent the i-th cluster, and N represent the number of Gaussian kernels obtained through clustering. According to cluster C i For the i-th Gaussian kernel Initialize the initial parameters.

6. The method for robot lifecycle navigation driven by dynamic semantic SLAM according to claim 1, characterized in that, The optimization of the Gaussian kernel set using a differentiable rendering and joint training strategy to obtain a mixture of Gaussian fields for localization and reconstruction specifically includes: Represent the scene as a set of Gaussian kernels. And for each Gaussian kernel Define the rendering radiation field as: Where C(r) represents the rendered radiance value at point r for each ray, r represents the ray sampling point, and N represents the number of Gaussian kernels obtained through clustering; σ i μ represents the opacity of the i-th Gaussian kernel. i , Σ i Let c represent the spatial mean and covariance matrix of the i-th Gaussian kernel, respectively. i Represents the semantic embedding vector of the i-th Gaussian kernel; Based on the Gaussian rendering results, a joint optimization loss function is constructed: L render =ω1L photo +ω2L geom +ω3L motion Among them, L render This represents the joint optimization loss function, where ω1, ω2, and ω3 are preset weights, and L... photo L represents the photometric error loss function. geom L represents the geometric consistency error loss function. motion This represents the motion compensation error loss function; Based on the joint optimization loss function for the Gaussian kernel parameter set Training and optimization are performed to obtain a mixed Gaussian field representation for robot localization and scene reconstruction.

7. A dynamic semantic SLAM-driven robot lifecycle navigation method according to claim 6, characterized in that, The photometric error loss function is formulated as follows: Among them, I input Represents the pixel values ​​of the input reference image; Represents the L2 norm; The geometric consistency error loss function is formulated as follows: Where, n i Let p represent the surface normal vector of the i-th Gaussian kernel, and p represent the spatial sampling point; The motion compensation error loss function is formulated as follows: Among them, v i Let v represent the velocity vector estimated from the dynamic elements corresponding to the i-th Gaussian kernel. GT The reference velocity vector represents the dynamic target.

8. A dynamic semantic SLAM-driven robot lifecycle navigation method according to claim 1, characterized in that, The process of storing and separating parameters of the mixed Gaussian field into a global static layer and a local dynamic layer, applying retention constraints to the static layer, and employing an incremental online update strategy for the dynamic layer to adapt to newly emerging dynamic elements, thereby obtaining Gaussian field parameters optimized through lifelong learning, specifically includes: The mixed Gaussian field parameter set is divided into a static parameter set Θ. static With dynamic parameter set Θ dyn Furthermore, an elastic weighting constraint is applied to the static parameter set, and its regularized form is expressed as: Where, θ i Let be the parameter vector of the i-th static Gaussian kernel. The corresponding parameter vector saved for the previous task; F i Let G be the Fisher information matrix for the i-th parameter. static Represents a static parameter index set, by using L EWC Incorporating overall objective constraints into static parameter updates to suppress forgetting; An update strategy combining incremental online adaptation based on meta-learning and knowledge distillation is adopted for the dynamic parameter set, where the meta-learning loss is expressed as: in, T represents the distribution of meta-tasks. support With T query L represents the support set and query set, respectively. task (·,·) represents the dynamic semantic segmentation and motion prediction loss for a single task, U(Θ) dyn ,T support The inner fast-adaptive update operator is represented as: Where α is the inner layer update learning rate; The rendering / reconstruction loss is combined with static preservation constraints and dynamic layer updates to form the overall online loss function: L total =L render +βL EWC +γL MAML Among them, L total Let L be the overall online loss function. render The loss function is used for joint optimization, where β and γ are weighting coefficients. By analyzing L total Minimize the joint update of Θ static With Θ dyn Gaussian field parameters optimized through lifelong learning were obtained.

9. A dynamic semantic SLAM-driven robot lifecycle navigation method according to claim 1, characterized in that, Based on optimized Gaussian field parameters and multimodal data, a factor map containing Gaussian rendering residual factors and inertial pre-integration factors is constructed and solved to obtain the corrected robot pose. Furthermore, spatial occupancy probabilities are generated based on the Gaussian field, specifically including: The visual-inertial joint cost of the factor graph is represented as: Among them, L SLAM The joint cost of the factor graph is represented by M, which represents the number of visual frames or factors involved in the optimization, and I... j This represents the RGB image of the j-th frame in the multimodal data. Indicates pose T j With Gaussian field Differentiable rendering result; α represents the weight coefficient, T represents the odometry transformation obtained by IMU pre-integration. j This represents the pose transformation in the j-th frame. Denotes the norm on the SE(3) Lie algebra space; By minimizing L SLAM Pose sequence By jointly solving with the necessary Gaussian field local parameters, the corrected set of robot poses is obtained. A space occupancy probability field is constructed based on Gaussian kernel parameters, and the occupancy probability function is defined as follows: in, Let φ(·) represent the occupancy probability of spatial location x, φ(·) represent the encoder that maps spatial location x to a feature vector, w represent the linear weight vector of the occupancy probability mapping, and σ(·) represent the Sigmoid function. Based on occupancy probability and corrected pose, define safety boundary constraints for dynamic obstacle avoidance: Among them, L safe Indicates safety boundary constraints. Let γ represent the set of sampled positions on the candidate trajectory, γ represent the weight vector, and v represent the robot's corresponding velocity vector. safe This indicates the preset safe speed threshold; L SLAM With L safe By incorporating a nonlinear least squares solution process, the pose TTT and necessary local Gaussian field parameters are iteratively optimized to output a corrected robot pose T for navigation and obstacle avoidance. * And the real-time space occupancy probability based on Gaussian fields.

10. A dynamic semantic SLAM-driven robot lifecycle navigation method according to claim 1, characterized in that, The process of adjusting operational control parameters online using a policy network based on the corrected robot pose and space occupancy probability, and achieving autonomous navigation based on the adjusted operational control parameters, specifically includes: Based on the obtained corrected robot pose T * Spatial occupancy probability generated based on mixed Gaussian fields The change in positioning accuracy ΔATE was calculated using windowed statistics and rendering / occupancy assessment processing, respectively. t , Reconstruction quality change ΔChamfer t Real-time performance metrics FPS t Construct the current state vector s t ={ΔATE t ,ΔChamfer t FPS t }; Define the action space: a t ={ρ t ,t t } Where, ρ t τ represents the Gaussian rendering resolution. t Indicates the frequency of knowledge distillation execution; Based on the current state vector s t The Gaussian kernel is output through the policy network to render the resolution ρ. t =π(s) t ); Based on the obtained Gaussian kernel rendering resolution ρ t Adjust the rendering resolution and sampling rate of the differentiable rendering module and the distillation scheduling of the online learning module, and based on The strategy output actions are subject to safety screening based on the updated rendering resolution and the corrected pose T. * Perform path planning and trajectory tracking to achieve adaptive autonomous navigation.

Images