Graph structure robot offline exploration method and system based on latent space diffusion model

Abstract

The application discloses a graph structure robot offline exploration method and system based on a latent space diffusion model, and the method comprises the following steps: constructing an offline data set of a robot historical exploration trajectory and an end-to-end exploration network model; based on the end-to-end exploration network model, performing discrete action selection on the offline data set to obtain a discrete action probability distribution; and constructing a double graph critic network model to perform implicit supervision training on the discrete action probability distribution without explicit reconstruction of a target, so as to obtain an optimal exploration strategy of the robot. The application can improve the exploration efficiency and robustness of the robot in a complex unknown environment. The application can be widely applied to the technical field of robot offline exploration as a graph structure robot offline exploration method and system based on a latent space diffusion model.

Description

Technical Field

[0001] This invention relates to the field of offline robot exploration technology, and in particular to a graph structure robot offline exploration method and system based on a latent space diffusion model. Background Technology

[0002] Autonomous exploration is a crucial research direction in the field of intelligent mobile robotics. Its goal is for robots to autonomously construct environmental maps and plan their paths in unknown or partially observable environments using onboard sensors (such as LiDAR or depth cameras), thereby achieving autonomous environmental coverage and information collection. In complex scenarios, to improve decision-making efficiency and scalability, the environment is typically abstracted into a topological graph structure, where nodes represent candidate observation points or reachable locations, and edges represent feasible paths between nodes. This graph structure effectively reduces the dimensionality of continuous space, enabling robots to perform decision-making and planning with limited computational resources.

[0003] Early exploration methods often employed heuristic or planning algorithm-based strategies, such as frontier-based exploration methods and hierarchical or skeleton mapping-based exploration frameworks. These methods rely on explicit environment modeling and search mechanisms, possessing a certain degree of computational efficiency, but lacking adaptive policy optimization capabilities and easily getting trapped in local optima in dynamic or partially observable environments.

[0004] With the development of reinforcement learning technology, researchers have begun to introduce it into robotic exploration tasks, aiming to automatically learn optimal policies through interaction with the environment. Online reinforcement learning methods can optimize policies through continuous interaction, but their training process relies on a large amount of online sampling, resulting in low sample efficiency, unstable policies, and high deployment costs on real robots. To address this, researchers have proposed offline reinforcement learning, which utilizes pre-collected datasets for policy training, thereby avoiding high-risk online interactions. Representative methods include conservative Q-learning and implicit Q-learning. Although offline reinforcement learning improves training efficiency and safety, it still faces problems such as distribution shift and pattern collapse in discrete action or graph-structured tasks, leading to overly concentrated policy outputs and a lack of diversity, thus weakening robustness and generalization ability.

[0005] To overcome the aforementioned limitations, generative models, especially diffusion models, have been introduced into the field of reinforcement learning in recent years. Diffusion models generate diverse distributions through a process of progressive noise addition and reverse denoising, exhibiting good multimodal modeling capabilities. Existing research has applied diffusion models to offline reinforcement learning for continuous control tasks, achieving richer policy representations and stronger generalization capabilities. However, these methods are mainly geared towards continuous action spaces. When directly applied to discrete or graph-structured action spaces, the denoising process is prone to collapse and the distribution becomes unstable, making it difficult to integrate with pointer-based action selection mechanisms. Meanwhile, exploration strategies based on graph neural networks and attention mechanisms have made progress in modeling spatial structures and node relationships, but they generally rely on online training, resulting in low sample efficiency, unstable convergence, and an inability to effectively address the dynamic balance between exploration and exploitation. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention aims to provide an offline exploration method and system for graph-structured robots based on a latent space diffusion model, which can improve the exploration efficiency and robustness of robots in complex and unknown environments.

[0007] The first technical solution adopted in this invention is: an offline exploration method for graph-structured robots based on a latent space diffusion model, comprising the following steps:

[0008] Construct an offline dataset of the robot's historical exploration trajectories and an end-to-end detection network model;

[0009] Based on the end-to-end detection network model, discrete action selection for the next target exploration viewpoint is performed on the offline dataset to obtain a discrete action probability distribution.

[0010] A dual-graph critic network model is constructed, and the discrete action probability distribution is trained with implicit supervision without explicit target reconstruction to obtain the robot's optimal exploration strategy.

[0011] Furthermore, the offline dataset specifically includes the robot's current state, the robot's current action, the robot's reward value, and the robot's next state. The robot's current state and the robot's next state represent an environmental topology map constructed based on incremental sensor data. The robot's current action represents the next discrete target exploration viewpoint selected by the robot in the current topology map. The robot's reward value represents a numerical feedback calculated based on the movement path cost and exploration completion bonus. The end-to-end detection network model includes a graph encoder, a latent diffusion model, a feature linear modulation fusion module, and a pointer decoder.

[0012] Furthermore, the step of selecting discrete actions for the next target exploration viewpoint based on the end-to-end detection network model to obtain the discrete action probability distribution specifically includes:

[0013] The environmental topology map from the offline dataset is input into the end-to-end probe network model;

[0014] Based on the graph encoder of the end-to-end probe network model, feature extraction is performed on the environmental topology graph to obtain the node feature matrix and global embedded features.

[0015] Based on the potential diffusion model of the end-to-end probe network model, the global embedding features are iteratively denoised to obtain the denoised embedding of the multimodal policy distribution.

[0016] Based on the feature linear modulation fusion module of the end-to-end detection network model, predictive modulation parameters are generated and residual fusion is performed on the denoised embedding of the multimodal policy distribution to obtain fused features;

[0017] Based on the pointer decoder of the end-to-end probe network model, the node feature matrix is ​​queried based on the fused features to obtain the discrete action probability distribution.

[0018] Furthermore, the step of iteratively denoising the global embedding features based on the latent diffusion model of the end-to-end probe network model to obtain the denoised embedding of the multimodal policy distribution specifically includes:

[0019] The global embedded features are input into the latent diffusion model of the end-to-end probe network model;

[0020] Based on the forward-noise-adding network in the potential diffusion model, a noise embedding is generated given a time step;

[0021] Based on the inverse denoising network in the latent diffusion model, the diffusion step number is set, and the global embedding feature and the noise embedding are iteratively denoised to obtain the denoised embedding of the multimodal policy distribution.

[0022] Furthermore, the step of generating predicted modulation parameters and performing residual fusion on the denoised embedding of the multimodal policy distribution to obtain fused features based on the end-to-end probe network model specifically includes:

[0023] Based on the aforementioned characteristic linear modulation fusion module, the predicted modulation parameters are generated through MLP.

[0024] The denoised embedding of the multimodal policy distribution is modulated according to the predicted modulation parameters to obtain the modulated denoised embedding.

[0025] The modulated denoised embedding is residually fused with the global embedding feature to obtain the fused feature.

[0026] Furthermore, the step of constructing a dual-graph critic network model to perform implicit supervised training on the discrete action probability distribution without explicit target reconstruction, thereby obtaining the robot's optimal exploration strategy, specifically includes:

[0027] A dual-graph critic network model is constructed, comprising two parallel value networks and actor networks sharing a backbone structure;

[0028] Based on the parallel value network, the value of each action in the current state is evaluated based on the discrete action probability distribution, and the minimum Q value is calculated.

[0029] Based on the minimum Q value, the parallel value network is updated by minimizing the temporal difference error function, and the advantage function value of the current data action relative to the average level is calculated.

[0030] Based on the actor network, a comprehensive loss function is constructed by introducing a behavior cloning term, an advantage weighting term, and a regularization term. The advantage function value is implicitly guided to generate the robot's optimal exploration strategy.

[0031] Furthermore, the expression for minimizing the timing difference error function is as follows:

[0032]

[0033] In the above formula, This represents minimizing the timing difference error function. Indicates the robot's current state. This indicates the robot's current action. This represents the robot's reward value. This indicates the robot's state at the next moment. Indicates the data from the offline dataset Calculate the mathematical expectation of the sampled state transition data. The parameter is The first value network assessment in the current state Take action below value, This indicates that the bi-graph critic network is considering the state at the next moment. and actions The minimum Q value was evaluated. This represents the discount factor.

[0034] Furthermore, the expression for the comprehensive loss function is as follows:

[0035]

[0036] In the above formula, Represents the comprehensive loss function. Indicates a clone of behavior. Indicates the advantage-weighted term. Represents the regularization term, This represents the weighting coefficient of the advantage-weighted term. This represents the weight coefficient of the regularization term.

[0037] Furthermore, the advantage weighting term is equivalent to minimizing the KL divergence between the policy distribution and the optimal Boltzmann distribution, and its expression is as follows:

[0038]

[0039] In the above formula, Represents a diffusion model. Indicates the pointer head. Represents the regularization term, Indicates the parameter and gradient operator, Indicates the data from the offline dataset Calculate the mathematical expectation of the sampled state-action pairs. The parameter is The advantage function value evaluated by the critics' network. Indicates the temperature coefficient. Indicates the current state of the network. Next generation action The strategy probability distribution.

[0040] The second technical solution adopted in this invention is: an offline exploration system for graph-structured robots based on a latent space diffusion model, comprising:

[0041] The first module is used to construct an offline dataset of the robot's historical exploration trajectories and an end-to-end detection network model;

[0042] The second module is used to perform discrete action selection for the next target exploration viewpoint on the offline dataset based on the end-to-end detection network model, and obtain a discrete action probability distribution.

[0043] The third module is used to construct a dual-graph critic network model, which performs implicit supervised training on the discrete action probability distribution without explicit target reconstruction, and obtains the robot's optimal exploration strategy.

[0044] The beneficial effects of the method and system of this invention are as follows: This invention constructs an offline dataset of the robot's historical exploration trajectory and an end-to-end probe network model; then, based on the end-to-end probe network model, it selects discrete actions for the next target exploration viewpoint on the offline dataset, obtaining a discrete action probability distribution. This can automatically adjust the exploration intensity and utilization weights according to the environmental complexity, enhancing exploration capabilities in unknown areas and strengthening target accuracy in known areas, thereby significantly improving overall exploration efficiency; finally, it constructs a dual-graph critic network model to perform implicit supervised training on the discrete action probability distribution without explicit target reconstruction, obtaining the robot's optimal exploration strategy. By introducing a diffusion modeling mechanism in the latent space and generating a multimodal latent feature distribution through multi-step noise addition and denoising, it can effectively avoid the problem of policy pattern collapse in traditional offline reinforcement learning, enabling the robot to maintain policy diversity and decision stability in complex and unknown environments, thereby improving the robot's exploration efficiency and robustness in complex and unknown environments. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the steps of the offline exploration method for graph-structured robots based on the latent space diffusion model of this invention.

[0046] Figure 2 This is a structural block diagram of the graph structure robot offline exploration system based on the latent space diffusion model of the present invention;

[0047] Figure 3 This is a schematic diagram of an exploration based on a latent space diffusion model provided in a specific embodiment of the present invention;

[0048] Figure 4 This is a simulation diagram of the experimental results provided in a specific embodiment of the present invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0050] First, the technical terms used in the embodiments of this invention will be explained:

[0051] 1) Autonomous Robot Exploration Tasks: This refers to the process by which a robot, in an unknown or partially known environment, gradually completes environmental coverage through sensor perception, map building, and path planning. Its goal is to explore unknown spaces quickly and efficiently with the least possible distance traveled.

[0052] 2) Topology graph: A sparse graph structure used to represent the robot's environment, where nodes represent candidate viewpoints or reachable locations, and edges represent feasible paths between nodes. This embodiment performs path planning and action decision-making on this graph structure.

[0053] 3) Latent space: refers to the low-dimensional feature space extracted by the neural network encoder, used to represent compressed information about the environment or state. This invention performs diffusion and denoising processes in the latent space to obtain stable and diverse policy representations.

[0054] 4) Diffusion Model: A generative model that generates diverse outputs by progressively adding noise to the data and learning a reverse denoising process. This embodiment utilizes this mechanism to generate a multimodal policy distribution in the latent space to improve the robustness and generalization ability of the exploration policy.

[0055] 5) Characteristic Linear Modulation Module: A learnable fusion structure used to adaptively weight the latent variables generated by diffusion with the original latent variables, achieving a dynamic balance between exploration and exploitation. In this embodiment, this module automatically adjusts the policy weights in different regions.

[0056] 6) Offline Advantage-Weighted Reinforcement Learning: A reinforcement learning method that utilizes a pre-collected dataset for training. During the update process, higher-value actions are assigned higher weights to achieve more stable and efficient policy optimization. This embodiment combines this mechanism with the latent space diffusion process to form an implicit energy minimization training objective.

[0057] The related technologies also have the following problems, such as:

[0058] 1) Lack of strategy diversity, prone to pattern collapse. Existing offline learning methods tend to be deterministic in the policy generation stage, making it difficult to maintain multimodal output, resulting in poor generalization ability of robots in unknown or complex scenarios.

[0059] 2) Diffusion models are difficult to apply stably to discrete or graph-structured tasks. Traditional diffusion models are mainly designed for continuous action spaces. When directly applied to node decision-making or graph-structured environments, they are prone to collapse during the denoising process, resulting in unstable generation results and making it difficult to effectively guide the selection of discrete actions.

[0060] 3) Insufficient utilization of graph structure information and limited feature representation capabilities. Existing exploration methods based on graph neural networks still have limitations in node relationship modeling and global information integration, making it difficult to simultaneously consider local structure and global topological features, thus limiting the effectiveness of policy learning.

[0061] 4) Lack of an adaptive balance mechanism between exploration and utilization. Most methods use a fixed strategy for updates during training or decision-making, and cannot automatically adjust the ratio of "continuing to explore unknown areas" and "utilizing known information" according to the environmental coverage, thus affecting the overall exploration efficiency.

[0062] 5) Offline training is prone to distribution shift and overfitting problems. The limited coverage of offline datasets causes the learned policies to deviate from the optimal distribution in unseen environments, resulting in performance degradation and instability.

[0063] 6) Policy optimization lacks global energy consistency constraints. Existing reinforcement learning optimization objectives mostly rely on explicit numerical estimation and fail to model from the perspective of energy minimization or probability consistency, making it difficult for training to converge to the globally optimal policy.

[0064] Based on this, the embodiments of the present invention introduce latent space diffusion modeling and offline reinforcement learning optimization mechanisms in a graph structure environment, forming an autonomous exploration method that combines multimodal generation capabilities, stable convergence characteristics, and adaptive decision-making balance, which can significantly improve the exploration efficiency and robustness of robots in complex and unknown environments.

[0065] Reference Figure 1 This invention provides an offline exploration method for graph-structured robots based on a latent space diffusion model, the method comprising the following steps:

[0066] S100, Construct an offline dataset and end-to-end detection network model of the robot's historical exploration trajectory;

[0067] First, it should be noted that this invention models the robot's autonomous exploration task as a partially observable Markov decision process (POMDP), using tuples. express.

[0068] In terms of scene representation, to address the gap between simulation and reality based on raw sensor data, this invention uses a topology graph. To represent the environment. Nodes It encapsulates key attributes, including boundary values ​​and drivability metrics; edges This represents the feasible paths between nodes. At each decision step... The robot can only observe a portion of the topology. This reflects the current incomplete environmental knowledge.

[0069] Explanation of state, observation, and action space:

[0070] 1) State Space : Represents the global state containing complete environmental information, each state It corresponds to a completely known topological graph.

[0071] 2) Observation space Observed values Corresponding to a partial graph constructed from incremental sensor data. Observation model A mapping from the global state to partial observations is defined.

[0072] 3) Action Space : Using a discrete action space, actions Defined as selecting the next exploration viewpoint from the current topology. Compared to dense map planning, this node-based selection significantly simplifies planning complexity.

[0073] To balance exploration costs and task completion, the following reward function is designed: Its expression is:

[0074]

[0075] in This represents the path cost incurred by the distance traveled. This represents the reward bonus upon completion of the exploration; As a discount factor, Let be the weight parameters, and satisfy ? This ensures that the algorithm prioritizes completing the exploration task rather than simply optimizing the path length.

[0076] Furthermore, collect offline datasets containing the robot's historical exploration trajectories. Each data entry contains tuples The dataset can originate from expert policies (such as the TSP algorithm), randomized policies, or a mixture of policies. Offline datasets. Tuples from the robot's historical exploration trajectory Composition, in which state and the state at the next moment Represents an environmental topology map constructed based on incremental sensor data. ,action The reward refers to the next discrete target exploration viewpoint selected by the robot in the current topology graph. It is a numerical feedback calculated based on the cost of the movement path and the exploration completion bonus. This dataset can come from expert policies, random policies, or a mixture of both generated by the TSP algorithm, and is used to provide implicit supervision signals to the network without requiring the truth value of the latent space.

[0077] like Figure 3As shown, the end-to-end detection network model includes a graph encoder, a latent diffusion model, a feature linear modulation fusion module, and a pointer decoder. First, the graph encoder extracts the node feature matrix and global embedding features of the local topology graph. Then, the global embedding features are injected into the latent diffusion model as conditional input. After iterative denoising, a denoised embedding representing the multimodal policy distribution is generated. Next, the feature linear modulation module predicts the modulation parameters, performs a linear transformation on the denoised embedding, and performs residual fusion with the original global features to output a fused feature that can adaptively balance exploration and utilization. Finally, the pointer decoder queries the node-level features based on the fused feature, calculates and outputs the selection probability of candidate nodes, thereby realizing the robot's discrete action selection for the next target exploration viewpoint.

[0078] S200. Based on the end-to-end detection network model, discrete action selection for the next target exploration viewpoint is performed on the offline dataset to obtain the discrete action probability distribution.

[0079] It should be noted that in some embodiments, step S200 may include steps S210 to S250.

[0080] S210. Input the environmental topology map from the offline dataset into the end-to-end probe network model;

[0081] S220, a graph encoder based on an end-to-end probe network model, extracts features from the environmental topology graph to obtain node feature matrices and global embedded features;

[0082] Specifically, a graph neural network encoder is used to process the current topological graph observations. Extracting node-level features and global graph features global features This will serve as a conditional input for the subsequent diffusion process.

[0083] In this embodiment, for the graph encoder, the SGFormer architecture is used to extract topological graph features, and its expression is:

[0084]

[0085]

[0086] in The node feature matrix, This is the global context embedding obtained after pooling operations.

[0087] S230. Based on the end-to-end probe network model, the potential diffusion model is used to iteratively denoise the global embedding features to obtain the denoised embedding of the multimodal policy distribution.

[0088] Specifically, the global embedding features are input into the latent diffusion model of the end-to-end probe network model; based on the forward denoising network in the latent diffusion model, a noise embedding is generated given a time step; based on the reverse denoising network in the latent diffusion model, the diffusion step number is set, and the global embedding features and the noise embedding are iteratively denoised to obtain the denoised embedding of the multimodal policy distribution.

[0089] Furthermore, the diffusion process is performed in the latent embedding space rather than the original action space. Forward process: Injects Gaussian noise into the latent features. Backward process: Uses global features... As a condition, through a denoising network By progressively removing noise, a denoised embedding that reflects the multimodal exploration strategy is generated. .

[0090] In this embodiment, for the potential diffusion model, forward noise addition is included: given a time step Generate noise embedding ,in .

[0091] Reverse Denoising Network: Designing a Denoising Network The input is the current noise embedding. diffusion steps and conditions , output prediction denoising embedding .

[0092] S240, a feature linear modulation fusion module based on an end-to-end probe network model, generates predicted modulation parameters and performs residual fusion on the denoised embedding of the multimodal policy distribution to obtain fused features;

[0093] Specifically, based on the feature linear modulation fusion module, predicted modulation parameters are generated through MLP; the denoised embedding of the multimodal policy distribution is modulated according to the predicted modulation parameters to obtain the modulated denoised embedding; the modulated denoised embedding is residually fused with the global embedding features to obtain the fused features.

[0094] Furthermore, by introducing a FiLM-based fusion module, the original deterministic global features are... With the randomness of diffusion generation Perform adaptive fusion to generate the final fused features. This step aims to dynamically balance "utilization" and "exploration".

[0095] In this embodiment, to address the issue of mode collapse in pure diffusion models on discrete graph structures, a FiLM-based ResNet architecture is introduced. First, modulation parameters are calculated using an MLP. Its expression is:

[0096]

[0097] The modulation of the denoised embedding is expressed as follows:

[0098]

[0099] Finally, the residual connections are used to fuse the original global features, and the expression is as follows:

[0100]

[0101] It allows the network to adaptively switch between a "deterministic greedy strategy" and a "multimodal diffusion strategy".

[0102] S250, a pointer decoder based on an end-to-end probe network model, queries the node feature matrix based on fused features to obtain the discrete action probability distribution.

[0103] Specifically, pointer networks are used to process fused features. and node features Calculate the selection probability of each candidate node, and generate discrete actions based on the probability distribution. (i.e., the next target viewpoint).

[0104] In this embodiment, the pointer decoder is a decoder. Utilizing attention mechanisms, based on fusion features Node features Perform a query and output the probability distribution of the actions:

[0105]

[0106] in, This represents the probability distribution of discrete actions.

[0107] S300. Construct a dual-graph critic network model to perform implicit supervised training on discrete action probability distributions without explicit target reconstruction, thereby obtaining the robot's optimal exploration strategy.

[0108] First, it should be noted that this specific embodiment adopts an implicit supervised training framework that does not require explicit reconstruction of the target, combining advantage-weighted offline RL and behavior cloning (BC).

[0109] Furthermore, it should be noted that in some embodiments, step S300 may include steps S310 to S340.

[0110] S310. Construct a dual-graph critic network model, which includes two parallel value networks and actor networks with a shared backbone structure.

[0111] S320. Based on a parallel value network, evaluate the value of each action in the current state based on the discrete action probability distribution, and calculate the minimum Q value.

[0112] In this embodiment, an implicit supervised training framework that does not require explicit reconstruction of the target is adopted. Policy optimization is achieved by combining advantage-weighted offline reinforcement learning and behavior cloning mechanism: First, a dual-graph critic network is constructed. The value networks with two shared backbone structures are used to evaluate the value of each action in the current state in parallel, and the minimum value is taken to alleviate the problem of value overestimation.

[0113] S330. Based on the minimum Q value, the parallel value network is updated by minimizing the temporal difference error function, and the advantage function value of the current data action relative to the average level is calculated.

[0114] In this embodiment, the value network is iteratively updated by minimizing the temporal difference error, while the advantage function value of the current data action relative to the average level is calculated.

[0115] Specifically, for the dual-graph critic network, in order to alleviate the Q-value overestimation problem, two Q-networks sharing the SGFormer backbone are constructed. , And take its minimum value:

[0116]

[0117] Training is performed by minimizing the temporal difference (TD) error:

[0118]

[0119] In the above formula, This represents minimizing the timing difference error function. Indicates the robot's current state. This indicates the robot's current action. This represents the robot's reward value. This indicates the robot's state at the next moment. Indicates the data from the offline dataset Calculate the mathematical expectation of the sampled state transition data. The parameter is The first value network assessment in the current state Take action below value, This indicates that the bi-graph critic network is considering the state at the next moment. and actions The minimum Q value was evaluated. This represents the discount factor.

[0120] S340. Based on the actor network, a comprehensive loss function is constructed by introducing a behavior cloning term, an advantage weighting term, and a regularization term. The advantage function value is implicitly guided to generate the robot's optimal exploration strategy.

[0121] In this embodiment, during actor network training, the behavior cloning term, the advantage weighting term, and the regularization term are combined to form a comprehensive loss function. The advantage weighting term implicitly guides the diffusion model using the calculated advantage value, driving the latent space diffusion process to automatically generate a probability distribution that conforms to the optimal exploration strategy, without needing to preset the ground truth target of the latent space. This mechanism is theoretically equivalent to minimizing the divergence between the current strategy and the optimal Boltzmann distribution, thereby achieving stable optimization of the robot's exploration behavior and global energy consistency constraints through gradient backpropagation under the constraint of offline dataset.

[0122] Specifically, for the actor network objective function, the actor network (including the diffusion model) and pointer head The loss function consists of three parts, and its expression is as follows:

[0123]

[0124] The meanings of each part are as follows:

[0125] 1) Behavioral clones ( ): Ensure that the strategy does not deviate from the behavior distribution in the offline dataset, and guarantee the stability of the training basis.

[0126] 2) Advantage-weighted items : Utilizing the dominant function Weighting the data drives the diffusion model to generate high-value actions.

[0127] 3) KL regularization term ( The constrained diffusion strategy ensures that the output distribution does not deviate excessively from the encoder's prior distribution, thus preventing gradient explosion.

[0128] For the implicit energy minimization mechanism, the advantage-weighted objective function is theoretically equivalent to minimizing the KL divergence between the policy distribution and the optimal Boltzmann distribution, and its expression is:

[0129]

[0130] In the above formula, Represents a diffusion model. Indicates the pointer head. Represents the regularization term, Indicates the parameter and gradient operator, Indicates the data from the offline dataset Calculate the mathematical expectation of the sampled state-action pairs. The parameter is The advantage function value evaluated by the critics' network. Indicates the temperature coefficient. Indicates the current state of the network. Next generation action The strategy probability distribution.

[0131] This means that even without explicit ground truth values ​​in the latent space, the mechanism can automatically generate embedding representations that conform to the optimal exploration strategy by driving the latent diffusion process through gradient backpropagation.

[0132] Finally, the following explanation is provided in conjunction with the accompanying drawings of the embodiments of the present invention:

[0133] First, both experimental training and evaluation tasks were performed on a computing platform equipped with an AMD EPYC 7773X CPU and two NVIDIA RTX4090 GPUs. Simulation testing was conducted using the MARSIM simulator, an environment capable of providing high-fidelity, lightweight point cloud data for LiDAR systems. In terms of model design, this approach constructs a unified heterogeneous graph general encoder for both the policy and value networks. The input layer transforms 6-dimensional node features into 64-dimensional latent representations through two MLP layers. To address the policy failure problem in large-scale environments, this embodiment introduces a dual normalization preprocessing technique: Min-Max scaling is used to limit coordinates to a standard range, followed by robot center coordinate transformation to localize the absolute position. This approach not only meets the smooth training requirements of course learning but also significantly improves the model's transferability across heterogeneous scenarios by addressing the challenges of out-of-distribution data.

[0134] like Figure 4 As shown in the experimental results, the trajectory performance demonstrates that the path generated based on latent space diffusion exhibits extremely high spatial continuity and smoothness while satisfying topological constraints. When facing maze corners or multi-branch intersections such as partition doors, the algorithm demonstrates excellent multimodal decision-making capabilities, accurately predicting and selecting the optimal branch pointing to unexplored areas. Furthermore, the robot exhibits efficient local-global backtracking logic, quickly locating the boundaries to be explored and planning the shortest backtracking path using global spatial representation, without any dead loops or decision oscillations throughout the process. This efficient and stable exploration behavior fully verifies that the invention possesses strong generalization stability and sample utilization when handling unstructured and complex constrained environments.

[0135] Therefore, compared with the prior art, the embodiments of the present invention have the following advantages:

[0136] 1) To achieve an exploration strategy that combines multimodality and robustness, this invention introduces a diffusion modeling mechanism in the latent space. Through multi-step noise addition and denoising, a multimodal latent feature distribution is generated, which can effectively avoid the problem of policy pattern collapse in traditional offline reinforcement learning, enabling the robot to maintain policy diversity and decision stability in complex and unknown environments.

[0137] 2) Achieving stable application of latent space diffusion in graph structure tasks. This invention combines graph structure encoding with latent variable conditional diffusion to achieve a stable denoising generation process in the discrete node action space, fundamentally solving the problem that existing diffusion models are difficult to converge in discrete or structured environments.

[0138] 3) Achieving an adaptive dynamic balance between exploration and utilization. Through a learnable feature linear modulation module (FiLM module), this invention can automatically adjust the exploration intensity and utilization weight according to the complexity of the environment, enhancing exploration capabilities in unknown areas and strengthening target accuracy in known areas, thereby significantly improving overall exploration efficiency.

[0139] 4) Improve policy generalization ability and offline training stability. This invention employs advantage-weighted optimization on offline datasets and introduces an energy consistency regularization term to ensure that latent diffusion features and encoded features maintain consistent distribution, effectively suppressing distribution shift and overfitting, and achieving stable convergence and cross-environment generalization.

[0140] 5) Enhance environmental modeling and spatial understanding capabilities. By introducing a graph encoder structure, it is possible to simultaneously capture local topological relationships and global environmental structure, enabling robots to have stronger structural perception and path planning capabilities in large-scale, sparse, or complex scenarios.

[0141] 6) High scalability and deployment efficiency. The network structure of this invention is modular and lightweight, enabling real-time inference on general-purpose computing hardware (such as embedded GPU platforms). It has good computational efficiency and engineering scalability, and is suitable for various robot platforms and task scenarios.

[0142] Reference Figure 2 An offline exploration system for graph-structured robots based on a latent space diffusion model includes:

[0143] The first module 201 is used to construct an offline dataset of the robot's historical exploration trajectories and an end-to-end detection network model;

[0144] The second module 202 is used to select discrete actions for the next target exploration viewpoint on the offline dataset based on the end-to-end detection network model, and obtain the discrete action probability distribution.

[0145] The third module 203 is used to construct a dual-graph critic network model, which performs implicit supervised training on discrete action probability distributions without explicit target reconstruction, and obtains the robot's optimal exploration strategy.

[0146] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0147] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An offline exploration method for graph-structured robots based on a latent space diffusion model, characterized in that, Includes the following steps: Construct an offline dataset of the robot's historical exploration trajectories and an end-to-end detection network model; Based on the end-to-end detection network model, discrete action selection for the next target exploration viewpoint is performed on the offline dataset to obtain a discrete action probability distribution. A dual-graph critic network model is constructed, and the discrete action probability distribution is trained with implicit supervision without explicit target reconstruction to obtain the robot's optimal exploration strategy.

2. The offline exploration method for graph-structured robots based on a latent space diffusion model according to claim 1, characterized in that, The offline dataset specifically includes the robot's current state, current action, reward value, and next state. The robot's current state and next state represent an environmental topology map constructed based on incremental sensor data. The robot's current action represents the next discrete target exploration viewpoint selected by the robot in the current topology map. The robot's reward value represents a numerical feedback calculated based on the movement path cost and exploration completion bonus. The end-to-end detection network model includes a graph encoder, a latent diffusion model, a feature linear modulation fusion module, and a pointer decoder.

3. The offline exploration method for graph-structured robots based on the latent space diffusion model according to claim 1, characterized in that, The step of selecting discrete actions for the next target exploration viewpoint based on the end-to-end detection network model and obtaining the discrete action probability distribution specifically includes: The environmental topology map from the offline dataset is input into the end-to-end probe network model; Based on the graph encoder of the end-to-end probe network model, feature extraction is performed on the environmental topology graph to obtain the node feature matrix and global embedded features. Based on the potential diffusion model of the end-to-end probe network model, the global embedding features are iteratively denoised to obtain the denoised embedding of the multimodal policy distribution. Based on the feature linear modulation fusion module of the end-to-end detection network model, predictive modulation parameters are generated and residual fusion is performed on the denoised embedding of the multimodal policy distribution to obtain fused features; Based on the pointer decoder of the end-to-end probe network model, the node feature matrix is ​​queried based on the fused features to obtain the discrete action probability distribution.

4. The offline exploration method for graph-structured robots based on the latent space diffusion model according to claim 3, characterized in that, The step of iteratively denoising the global embedding features based on the latent diffusion model of the end-to-end probe network model to obtain the denoised embedding of the multimodal policy distribution specifically includes: The global embedded features are input into the latent diffusion model of the end-to-end probe network model; Based on the forward-noising network in the potential diffusion model, a noise embedding is generated given a time step; Based on the inverse denoising network in the latent diffusion model, the diffusion step number is set, and the global embedding feature and the noise embedding are iteratively denoised to obtain the denoised embedding of the multimodal policy distribution.

5. The offline exploration method for graph-structured robots based on the latent space diffusion model according to claim 3, characterized in that, The step of the feature linear modulation fusion module based on the end-to-end probe network model, which generates predicted modulation parameters and performs residual fusion on the denoised embedding of the multimodal policy distribution to obtain fused features, specifically includes: Based on the aforementioned characteristic linear modulation fusion module, the predicted modulation parameters are generated through MLP. The denoised embedding of the multimodal policy distribution is modulated according to the predicted modulation parameters to obtain the modulated denoised embedding. The modulated denoised embedding is residually fused with the global embedding feature to obtain the fused feature.

6. The offline exploration method for graph-structured robots based on the latent space diffusion model according to claim 1, characterized in that, The step of constructing a dual-graph critic network model and performing implicit supervised training on the discrete action probability distribution without explicit target reconstruction to obtain the robot's optimal exploration strategy specifically includes: A dual-graph critic network model is constructed, comprising two parallel value networks and actor networks sharing a backbone structure; Based on the parallel value network, the value of each action in the current state is evaluated based on the discrete action probability distribution, and the minimum Q value is calculated. Based on the minimum Q value, the parallel value network is updated by minimizing the temporal difference error function, and the advantage function value of the current data action relative to the average level is calculated. Based on the actor network, a comprehensive loss function is constructed by introducing a behavior cloning term, an advantage weighting term, and a regularization term. The advantage function value is implicitly guided to generate the robot's optimal exploration strategy.

7. The offline exploration method for graph-structured robots based on the latent space diffusion model according to claim 6, characterized in that, The specific expression for the function that minimizes the time-series difference error is as follows: In the above formula, This represents minimizing the timing difference error function. Indicates the robot's current state. Indicates the robot's current action. This represents the robot's reward value. This indicates the robot's state at the next moment. Indicates the data from the offline dataset Calculate the mathematical expectation of the sampled state transition data. The parameter is The first value network assessment in the current state Take action below value, This indicates that the bi-graph critic network is considering the state at the next moment. and actions The minimum Q value was evaluated. This represents the discount factor.

8. The offline exploration method for graph-structured robots based on the latent space diffusion model according to claim 6, characterized in that, The expression for the comprehensive loss function is as follows: In the above formula, Represents the comprehensive loss function. Indicates a cloned behavior item. Indicates the advantage-weighted term. Represents the regularization term. This represents the weighting coefficient of the advantage-weighted term. This represents the weight coefficient of the regularization term.

9. The offline exploration method for graph-structured robots based on the latent space diffusion model according to claim 6, characterized in that, The advantage weighting term is equivalent to minimizing the KL divergence between the policy distribution and the optimal Boltzmann distribution, and its expression is as follows: In the above formula, Represents a diffusion model. Indicates the pointer head. Represents the regularization term. Indicates the parameter and gradient operator, Indicates the data from the offline dataset Calculate the mathematical expectation of the sampled state-action pairs. The parameter is The advantage function value evaluated by the critics' network. Indicates the temperature coefficient. Indicates the current state of the network. Next generation action The strategy probability distribution.

10. An offline exploration system for graph-structured robots based on a latent space diffusion model, characterized in that, Includes the following modules: The first module is used to construct an offline dataset of the robot's historical exploration trajectories and an end-to-end detection network model; The second module is used to perform discrete action selection for the next target exploration viewpoint on the offline dataset based on the end-to-end detection network model, and obtain a discrete action probability distribution. The third module is used to construct a dual-graph critic network model, which performs implicit supervised training on the discrete action probability distribution without explicit target reconstruction, and obtains the robot's optimal exploration strategy.

Images