Robot obstacle avoidance method based on WMSRL model

Abstract

The present invention discloses a robot obstacle avoidance method based on the WMSRL model, which runs in the robot motion coordination control module. The WMSRL model includes two parts. One part is to use safety reinforcement learning means in the process of robot safe exploration, and the other part is to use an improved world model to perform strategy learning of the robot's posture information. The robot's motion trajectory processed by the obstacle avoidance method of the present invention will be simulated and executed in the world model, so as to predict the reward, cost, reward return value and cost return value corresponding to the robot's state at each moment. The present invention adopts a model-free safety reinforcement learning method combined with an improved world model. This combination aims to improve the utilization rate of posture information, thereby promoting faster strategy convergence in the feasible domain, and using online planning in the world model to achieve cost minimization during training and exploration.

Description

Technical Field

[0001] In this paper, the WMSRL model combines a world model with safe reinforcement learning (SafeRL). The improved world model employed in the WMSRL model effectively utilizes the robot's pose information during motion, ensuring the safety of the entire reinforcement learning training and deployment process. Background Art

[0002] A mobile robot is a robotic system composed of sensors, a remote operator, and an automatically controlled mobile carrier. Mobile robots possess mobility capabilities, offering greater maneuverability and flexibility than conventional robots, enabling them to replace humans in dangerous, harsh, and otherwise inaccessible environments. Robotic obstacle avoidance is a fundamental and critical function in robotics, ensuring collision-free robot movement. See "Mobile Robot Systems and Their Coordinated Control," by Cheng Lei, published by Huazhong University of Science and Technology Press, March 2014, pages 59-60, for an introduction to coordinated robot motion control. The motion coordination algorithm estimates the robot's control output for the next cycle based on the robot's current position information, thereby avoiding collisions between robots or with obstacles in the environment.

[0003] Traditional reinforcement learning algorithms require exploration during the learning process, exploring the environment without constraints to generate an optimal control policy. However, this unconstrained exploration inherently carries significant safety risks, as the agent could potentially learn dangerous policies. Traditional reinforcement learning involves learning across the entire parameter space, which doesn't guarantee safety. This means the policy could potentially exceed safety boundaries. Therefore, when developing reinforcement learning algorithms, it's important to be mindful of these safety risks and take necessary measures to ensure policy safety. For example, in coordinated robot motion control, if safety constraints are not considered, the robot could take dangerous actions, potentially damaging components, surrounding objects, or even personnel. Therefore, when using deep reinforcement learning algorithms for control, safety constraints must be considered to ensure policy safety.

[0004] Reinforcement learning is a machine learning paradigm that maximizes numerical reward signals through autonomous exploration. Unlike supervised learning, which relies on task-driven learning, and unsupervised learning, which relies on finding implicit structures in data, reinforcement learning strikes a balance between exploration and exploitation. During the exploration phase, the agent discovers and learns more valuable action sequences through interaction with the unknown environment; during the exploitation phase, the agent fully utilizes existing experience data and uses this sampled data to learn strategies.

[0005] Safety reinforcement learning (SRL) utilizes reinforcement learning to address system safety issues. It combines technologies from reinforcement learning and security to ensure that the behavior of intelligent agents during the learning process does not render the system unsafe. In SRL, intelligent agents, which can be single or multiple, learn how to make correct and safe decisions through interaction with the environment. During reinforcement learning, agents optimize their strategies by performing actions and receiving reward signals. SRL considers the impact of the agent's strategy on system safety and employs different strategies to achieve an optimal balance between safety and utility. SRL also requires consideration of security metrics and modeling of security threats to ensure that the system can respond and make appropriate decisions in the face of diverse security threats. Summary of the Invention

[0006] This paper presents a robot obstacle avoidance method based on the WMSRL model. Due to the inefficiency of model-free safe reinforcement learning (SafeRL) in utilizing pose information during robot motion, this method struggles to achieve safe exploration. This paper combines the SafeRL method with an improved world model. This combination aims to improve the utilization of pose information, thereby accelerating policy convergence within the feasible domain. Furthermore, the world model is incorporated with online planning to minimize cost during training and exploration.

[0007] The present invention is a robot obstacle avoidance method based on the WMSRL model. Figure 2 The WMSRL model has been added to the robot motion coordination control module in

[15] . This WMSRL model uses an improved world model to fully utilize the robot's motion pose information, ensuring the safety of each robot's motion pose throughout the entire reinforcement learning training and deployment process.

[0008] A robot obstacle avoidance method based on the WMSRL model of the present invention includes the following steps:

[0009] Step 1: robot motion trajectory expansion and value estimation;

[0010] Step 2: The trade-off between rewards and costs of a safe trajectory;

[0011] Step 3: Safe online planning of robot motion trajectory based on Lagrangian method;

[0012] Step 4: Construct the components of the world model of the robot's motion trajectory;

[0013] Step 5: Calculate the loss function of the world model;

[0014] Step 6: Strategy learning based on the actor-critic model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the robot obstacle avoidance method based on the WMSRL model of the present invention.

[0016] Figure 2 It is a structural diagram of the coordinated control of robot motion.

[0017] Figure 3 These are the experimental results of SafeDreamer and baseline algorithms in visual input-based tasks.

[0018] Figure 4 SafeDreamer is compared with baseline algorithms in a visual roaming-based security task.

[0019] Figure 5 SafeDreamer is compared with baseline algorithms in security tasks based on low-dimensional vector inputs. DETAILED DESCRIPTION

[0020] The present invention will be described in further detail below with reference to the accompanying drawings.

[0021] See also Figure 1 、 Figure 2 As shown in the figure, a robot obstacle avoidance method based on the WMSRL model of the present invention is Figure 2 The WMSRL model has been added to the robot motion coordination control module in [1]. This WMSRL model utilizes an improved world model to maximize the utilization of the robot's pose information during motion, ensuring the safety of reinforcement learning training and deployment throughout the entire process. The WMSRL model consists of two components: one employs secure reinforcement learning to safely explore the robot's pose information; the other utilizes the improved world model to strategically learn the robot's pose information.

[0022] See also Figure 1 As shown, a robot obstacle avoidance method based on the WMSRL model of the present invention includes the following steps:

[0023] 1. Safe reinforcement learning of robot posture information

[0024] Step 1: Robot motion trajectory expansion and reward value estimation and cost value estimation;

[0025] This paper combines online planning and temporal difference learning to achieve safe exploration of robot motion during obstacle avoidance. To strike a balance between safe exploration and exploration performance, the present invention employs a Lagrangian approach. Furthermore, by leveraging the Dreamer V3 (world model) architecture, the accuracy of the world model's cost model is improved, thereby enhancing the robot's safe obstacle avoidance planning performance. Finally, the safe exploration algorithm simultaneously processes vector and image inputs, providing the versatility to handle diverse data types. The proposed robot's obstacle avoidance motion achieves cost minimization by planning within a separately trained world model trained on the Safety-Gymnasium vision task benchmark.

[0026] Compared with other algorithms (such as CPO and PPO-Lag), it has the following advantages:

[0027] (1) Our method is an algorithm that can effectively process low-dimensional input and visual input, and performs well in the Safety-Gymnasium benchmark test, achieving near-zero cost.

[0028] (2) The architecture of the proposed method is highly scalable, as reflected in its compatibility with the original Lagrangian method and its seamless integration with the planning mechanism, which fully leverages the world model’s ability to solve the robot’s safe obstacle avoidance problems.

[0029] (3) The method of the present invention is an algorithm that achieves near-zero-cost deployment based on visual input.

[0030] In this paper, the safety reinforcement learning of robot obstacle avoidance based on the WMSRL model is modeled as a constrained Markov decision process (CMDP), namely , then at the current moment The state transition probability function under and the reward function .

[0031] Indicates taking action After that, from the current state Transition to the next state The transition probability.

[0032] Indicates the current state Execute an action and transfer to the next state Rewards obtained when .

[0033] In the present invention, let the state space Any state in , the initial state is recorded as .

[0034] In the present invention, let the action space Any action in .

[0035] Indicates the state transition probability identifier.

[0036] Represents the reward set identifier.

[0037] represents the cost set, and ,in is the cost function, is the cost threshold, subscript Indicates the cost threshold identification number, Indicates the total number of cost thresholds. In the present invention, let the cost set Any price is recorded as .

[0038] is the initial state distribution, and μ ( ⋅ ) : S → [ 0 , 1 ] .

[0039] is the discount factor, and .

[0040] is the current moment of the robot's motion.

[0041] is the moment before the robot moves.

[0042] is the last moment of the robot's movement.

[0043] In this invention, a parameterized steady-state strategy It is defined in On the probability distribution of Also known as being in state Take action The probability of using Represents the set of all steady-state strategies, using Represents the state transition probability matrix. Any element in is represented by its state as P π θ [ s t , s t + 1 ] ,and P π θ [ s t , s t + 1 ] = ∑ a ∈ A π θ ( a t ∣ s t ) ℙ ( s t + 1 ∣ s t , a t ) .

[0044] Represents the probability identifier under the steady-state strategy.

[0045] are the network parameters that need to be learned.

[0046] Indicates the real number space, the superscript Indicates the dimension.

[0047] P π θ [ s t , s t + 1 ] Represents the state element of the matrix.

[0048] Indicates the actions taken under the steady-state strategy After that, from the current state Transition to the next state The state transition probability.

[0049] In the present invention, any element in the state transition probability matrix P π θ [ s t , s t + 1 ] By implementing a steady-state strategy From the current state Transition to the next state The probability of one-step state transition. When all elements complete one-step state transition, finally, As the initial state under the steady-state strategy The steady-state distribution of the Markov chain at the beginning, however,  s 0 ∼ μ ( ⋅ ) [ d π θ μ ( s ) ] As the initial distribution under the steady-state strategy Discount status on access distribution.

[0050] represents the steady-state state of the Markov chain under the steady-state strategy.

[0051] Indicates the current time The discount factor below.

[0052] Indicates the current state under the steady-state strategy The transition probability.

[0053] is the initial state distribution, and μ ( ⋅ ) : S → [ 0 , 1 ] .

[0054] represents the initial distribution under the steady-state strategy Discount status on .

[0055] Among them, the infinite time discounted reward function is defined as:

[0056] J R ( π θ ) =  [ ∑ t = 0 ∞ γ t R ( s t + 1 ∣ s t , a t ) ∣ s 0 ∼ μ , a t ∼ π θ ] ( 1 ) ;

[0057] represents the reward function.

[0058]  [ ⋅ ] Indicates expected value.

[0059] Indicates the current time The discount factor below.

[0060] Indicates that the status Execute an action and transfer to the next state Rewards obtained when .

[0061] is the initial state distribution, and μ ( ⋅ ) : S → [ 0 , 1 ] .

[0062] Represents the probability sign under the steady-state strategy.

[0063] Among them, the infinite time discounted cost function is defined as:

[0064] J i C ( π θ ) =  [ ∑ t = 0 ∞ γ t c i ( s t + 1 ∣ s t , a t ) ∣ s 0 ∼ μ , a t ∼ π θ ] ( 2 ) ;

[0065] Subscript Indicates the cost threshold identification number.

[0066] Indicates that the status Execute an action and transfer to the next state The discount price obtained when .

[0067] Defining a set of feasible strategies for:

[0068] ;

[0069] is the cost threshold.

[0070] Among them, the goal of the constrained Markov model CMDP is to The optimal strategy for searching is:

[0071] ;

[0072] Represents the probability sign under the steady-state strategy.

[0073] Represents a set of feasible strategies.

[0074] represents the reward function.

[0075] In the present invention, according to the reward function in the constrained Markov model CMDP , combined with the coordinated control of multi-robot motion, Dreamer V3 (world model) is used to modify the state transition probability function of the robot's motion obstacle avoidance process, and the safety reinforcement learning problem based on the robot world model is defined as follows:

[0076] ;

[0077] J ϕ R ( π θ ) =  [ ∑ t = 0 ∞ γ t R ( s t + 1 ∣ s t , a t ) ∣ s 0 ∼ μ , s t + 1 ∼ ℙ ϕ ( ⋅ ∣ s t , a t ) , a t ∼ π θ ] ( 6 ) ;

[0078] J ϕ C ( π θ ) =  [ ∑ t = 0 ∞ γ t c Fixed ( s t + 1 ∣ s t , a t ) ∣ s 0 ∼ μ , s t + 1 ∼ ℙ ϕ ( ⋅ ∣ s t , a t ) , a t ∼ π θ ] ( 7 ) ;

[0079] Represents the probability sign under the steady-state strategy.

[0080] represents the set of all steady-state strategies, and .

[0081] Represents the reward function in the secure reinforcement learning model.

[0082] Represents the cost function in the secure reinforcement learning model.

[0083] Represents a specified obstacle avoidance safety constraint threshold.

[0084] Indicates that the status Execute an action and transfer to the next state Rewards obtained when .

[0085] Indicates the initial state.

[0086] is the initial state distribution, and μ ( ⋅ ) : S → [ 0 , 1 ] .

[0087] It is a parameterized world model.

[0088] Represents a specified cost function.

[0089] In the present invention, is a Parameterized world model, assuming the initial state From the true initial state distribution Sampling, then The world model will be used To sample trajectories, the sampled trajectories are used to estimate the rewards and costs required by the policy optimization algorithm. At the same time, the world model is used for online planning to ensure safer exploration.

[0090] In this invention, the safety planning algorithm refers to the process of using the world model for online or offline planning to balance the robot's reward return and cost reporting, where OSRP stands for Online Safety-Reward Planning and BSRP stands for Background Safety-Reward Planning. The design of OSRP is to combine model planning with safety constraints in safety tasks that rely solely on vision. Specifically, the SR-MPC algorithm of this invention uses the Constrained Cross-Entropy Method (CCEM) for planning and uses a reinforcement learning model to 、 、 and In addition, the reward valuation is also used and price valuation and a reward-driven policy network π_θ to guide the planning process.

[0091] Represents the predicted state sequence.

[0092] Represents the predicted dynamic state.

[0093] Represents the predicted reward value.

[0094] Represents the predicted cost value.

[0095] First, initialize the step length to The independent parameters of each action within the time range, that is, the action parameters ,in ; Next, use the world model to independently sample trajectories and estimate their infinite-time discounted rewards outside the planning horizon and infinite time discounted cost , this estimate is made by introducing a bootstrap-based returns to achieve:

[0096] ;

[0097] ;

[0098] Indicates the current status.

[0099] Indicates the current status Reward value returns.

[0100] Superscript yes The value of returns.

[0101] Indicates the next state Reward value returns.

[0102] Indicates the planning step length.

[0103] Indicates the current status The predicted reward value.

[0104] is the discount factor, and .

[0105] Indicates the next state The estimated reward return value.

[0106] Indicates the The estimated reward value for the state at this step.

[0107] In the present invention, the infinite time discount cost Can be used and To calculate.

[0108] In the present invention, the prediction of the cost model is used to To evaluate the situation The total cost within the step:

[0109] J ϕ C , H ( π θ ) =  [ ∑ t t + H γ t C ϕ ( s t ) ∣ s t + 1 ∼ ( f ϕ , q ϕ ) , a t ∼ N ( μ J − 1 , ( σ J − 1 ) 2 × I ) ] ( 1 0 ) ;

[0110] Indicates the planning step length.

[0111] Indicates the current time The discount factor below.

[0112] Represents a sequence model.

[0113] Indicates the current status The predicted cost value.

[0114] Represents the observation encoder.

[0115] Represents a normal distribution.

[0116] Represents the identity matrix.

[0117] is the action mean.

[0118] Superscript Indicates the total number of iterations, the number of iterations .

[0119] represents the action mean of the previous iteration.

[0120] represents the action variance of the previous iteration.

[0121] Use the cost value of all trajectories As An approximate estimate of is the round length. As a criterion for evaluating trajectory safety.

[0122] In the present invention, the reward values ​​of all trajectories calculated in step 1 are and cost value It will be used in step 2.

[0123] Step 2: The trade-off between the reward value and cost value of the safe trajectory;

[0124] In the present invention, the trade-off between reward value and cost value is achieved in the following way: after calculating the cost value of all trajectories Afterwards, the number of safe trajectories is expressed as .if Lower than expected number of safe trajectories , it means that it is difficult to find a safe trajectory in the current state. Therefore, it becomes crucial to prioritize safety, and The value of becomes the sorting criterion, recorded as , and the entire set of sampled robot motion trajectories becomes the candidate action set On the contrary, if , it means that the trajectory that meets the safety constraints is relatively easy to find. Therefore, the reward value Used as sorting criteria , and the safe action trajectory The actions within are selected as candidate actions From the candidate action set , select the sorting criteria The corresponding maximum value, and the action sequence corresponding to the maximum value is used as the elite action ,in accordance with In the Get the first parameter in the iteration and the second parameter :

[0125] ;

[0126] ;

[0127] Indicates the number of predefined elite actions.

[0128] Represents any predefined elite action.

[0129] After reaching the predetermined number of iterations After that, the robot motion planning process ends. At the current moment of each decision robot motion The mean of the first action of the planned action trajectory will be used as the action to be executed in the real environment.

[0130] Step 3: Safe online planning of robot motion trajectory based on Lagrangian method;

[0131] In the present invention, the criteria for evaluating the cost constraint in step 2 are , when the hazard detection field of view required during the robot's obstacle avoidance process exceeds the planning field of view, the planning process may not be able to ensure zero cost violations in complex vision-only tasks. To solve this problem, the present invention uses a cost critic to estimate the expected cost in the later stages of the planning process. In order to fully utilize the potential of the cost critic, the present invention integrates the PID Lagrangian method into the world model planning framework, thereby obtaining the Online Safety-Reward Planning (OSRP-Lag) algorithm of the present invention, which satisfies the requirements in step 2. Sorting criteria when Adjust to ,in This means that when there are many safe trajectories in the robot's motion planning process, the LAG-MPC algorithm of the present invention will dynamically adjust the Lagrange multiplier based on the robot's historical constraint violations, and continue to be cautious during safe exploration.

[0132] The robot motion data obtained using safe online planning in step 3 will be used to construct a world model of the robot's motion trajectory in step 4.

[0133] (2) Using the improved world model to analyze robot pose information

[0134] Step 4: Construct the components of the world model of the robot's motion trajectory;

[0135] The improved world model is used to train the historical data of the robot's interaction with the environment from step 3. This data includes observations, actions, rewards, costs, and round continuation indicators, and the time length is , expressed as .

[0136] Indicates the current status.

[0137] Indicates the current action.

[0138] Indicates the next state.

[0139] Indicates the next reward.

[0140] Indicates the next generation price.

[0141] Indicates the continuation indicator of the next round.

[0142] In the present invention, the improved world model includes a sequential model denoted as , the observation encoder is denoted as The dynamic model is recorded as , the reward decoder is denoted as , the cost decoder is denoted as , the round continuation indicator decoder is recorded as and the observation decoder is denoted as .

[0143] represents the posterior distribution of the observation encoder predictions.

[0144] represents the prior distribution of the kinetic model predictions.

[0145] Represents the hidden state predicted by the sequence model.

[0146] Indicates the previous action.

[0147] Represents the actual observation state received in the environment.

[0148] Indicates the current reward.

[0149] Indicates the current price.

[0150] Indicates the continuation indicator of the current round.

[0151] Gated Recurrent Unit (GRU) is used. For visual input, Convolutional Neural Network (CNN) is used as the observation encoder. and observation decoder , while for low-dimensional inputs, a multi-layer perceptron (MLP) is used. The reward, cost, round continuation marker decoder and dynamics model are implemented using MLP. This world model can be understood as a sequence variational autoencoder (VAE), where the output of the observation encoder acts as an approximate posterior, which first transforms the observation Encoded as the posterior distribution, i.e. Output , given the initial state and , and the previous action , loop state and , use the sequence model to predict the next cycle state during the world model training process However, during policy training or online planning, the sequence model is trained with its own previous output As input instead of . Dynamic utilization Predict the prior distribution, i.e. Output By and Connection is model training status , which the present invention uses to predict rewards, costs, and round continuation indicators.

[0152] Step 5: Calculate the loss function of the world model;

[0153] In this paper, all modules of the improved world model are jointly optimized. The loss function consists of two parts: KL divergence loss and decoder loss. Given a batch size of sequence , the model can be optimized end-to-end by minimizing the following loss:

[0154] ;

[0155] Loss Function It is composed of the log-likelihood between the distributions generated by the decoder for observations, rewards, costs, and episode continuation markers and their corresponding labels:

[0156] ;

[0157] represents the loss function of the observation decoder.

[0158] Represents the weight coefficient of the loss function of the observation decoder.

[0159] represents the loss function of the reward decoder.

[0160] Represents the weight coefficient of the loss function of the reward decoder.

[0161] represents the loss function of the cost decoder.

[0162] Represents the weight coefficient of the loss function of the cost decoder.

[0163] represents the loss function of the episode continuation marker decoder.

[0164] Represents the weight coefficient of the loss function of the round continuation marker decoder.

[0165] When processing visual input, each weight coefficient 、 、 and are all set to 1.0. However, in the case of low-dimensional vector inputs, since the prediction cost always depends on the accurate reconstruction of the observations, is set to 5.0. Loss function It is calculated by measuring the KL divergence between the dynamics predictor and the observation encoder. The main goal is to train the sequence model and the dynamics predictor using The previously available information is used to generate a prior distribution that is as close as possible to the posterior distribution obtained using the encoder, which is achieved by minimizing At the same time, the observation encoder is trained to extract relevant information from the observations and enhance the predictive power of the dynamics predictor, which is achieved by minimizing The loss function is implemented by the first term in The calculation is as follows:

[0166] L K L ( ϕ ) = ν × K L [ q ϕ ( z t ∣ h t , o t ) ∥ s g ( p ϕ ( z t ∣ h t ) ) ] + δ × KL [ s g ( q ϕ ( z t ∣ h t , o t ) ) ∥ p ϕ ( z t ∣ h t ) ] (15) ;

[0167] K L [ · ] Represents the KL divergence between the two elements of the distribution.

[0168] sg stands for stop gradient operator.

[0169] is the first constant coefficient.

[0170] is the second constant coefficient.

[0171] In the present invention, and They are set to 0.5 and 0.1 respectively in all tasks.

[0172] In addition, the present invention uses free bits to limit the KL divergence to 1.0. This technique effectively alleviates the problem that the encoder output may lack sufficient information, which will cause the dynamic model and sequence model to collapse and make meaningful predictions impossible.

[0173] Step 6: Strategy learning based on the actor-critic model;

[0174] In this invention, in the strategy learning process based on the actor-critic model, Tagged as actor, Mark as reward critics, Marked as cost critic. Use the improved world model to train the model state Make predictions and get potential states Due to the sparsity and uneven distribution of costs in the environment, directly regressing the cost value will make the cost critic learning very difficult. This paper follows the approach of the world model framework DreamerV3 and adopts three techniques, namely discrete regression, twohot encoding and logarithmic symlog smoothing. This paper discretizes the target value that the critic needs to regress, and then trains the critic network to predict each discrete value Twohot softmax probability , the reward and cost critic networks are trained using the following loss functions:

[0175] ;

[0176] ;

[0177] ;

[0178] Indicates the number of predefined discrete values, using 255 discrete values. , B = [ − 20 ,..., 20 ] Represents an array of discrete values ​​of length 255, .

[0179] Two-hot encoding extends the concept of one-hot encoding to continuous values. Therefore, the target value can be predicted and recovered by the following formula:

[0180] V ψ ( s t ) = s y m e x p ( E p ( b b i ∣ s t ) [ b b g ] ) = s y m e x p ( p ψ ( ⋅ ∣ s t ) T B ) ( 1 9 ) ;

[0181] in , the symexp function is the inverse function of symlog.

[0182] In the OSRP-Lag and OSRP algorithms of the present invention, an unsafe actor model is trained to guide the planning process, thereby achieving faster reward convergence. The objective function of the actor model is to maximize the expected reward and the entropy of the action. The present invention estimates the gradient of the first term using stochastic back-propagation, while the gradient of the second term is obtained by analytical calculation.

[0183] L ( ψ ) = ∑ t = 1 T s g ( R θ ( s t ) ) − λ H η [ π Entropy ( a t ∣ s t ) ] ( 2 0 ) ;

[0184] is a fixed coefficient.

[0185] In addition, since online planning is often very time-consuming, in order to meet certain scenarios that require rapid response, this paper also proposes an offline safety planning algorithm combined with the Lagrangian method (BSRP-Lag). This algorithm does not use the online planning process and directly generates exploration actions through a safe actor strategy. The actor strategy is updated using the augmented Lagrangian method. The main goal is to maximize the expected reward while minimizing the expected cost. Its loss function is as follows:

[0186] L ( θ ) = ∑ t = 1 T s g ( R θ ( s t ) ) − λ ( s g ( C Ψ ( s t ) ) , λ p p , λ , ω η ) − ( 2 1 ) ω H η [ π Entropy ( a t ∣ s t ) ] ;

[0187] ;

[0188] ;

[0189] express Reward value in the state returns.

[0190] express The cost value of the state returns.

[0191] Indicates the current gradient update round.

[0192] Represents the Lagrange multiplier for the current gradient update round.

[0193] Represents the Lagrangian penalty factor for the current gradient update round.

[0194] The difference between the method of the present invention and the similar algorithm LAMBDA

[0195] (1) World Model: In LAMBDA experiments, planning methods failed to achieve satisfactory results. In contrast, our world model excels in leveraging online information for accurate prediction and enhanced safety.

[0196] In addition, the world model of the present invention is based on an improved version of DreamerV3, while the world model of LAMBDA is based on DreamerV1. Therefore, the algorithm of the present invention shows greater stability in terms of vector and visual input, achieving low-cost violations.

[0197] (2) Cost Model: In LAMBDA, a binary classifier is directly trained as the cost model. However, when the cost obtained from the environment feedback is sparse, the cost model may take longer to converge. On the other hand, in the BSRP-Lag algorithm of the present invention, more robust techniques such as logarithmic sign transformation, discrete regression, and two-hot encoding are used to train the cost model, which can improve its performance even when the cost feedback is sparse.

[0198] (3) Reward-cost trade-off: Unlike LAMBDA, which utilizes optimistic and pessimistic estimates, we have found that adopting these estimates in vision-only safety tasks prevents the world model from accurately predicting policy behavior. Therefore, more data and additional training overhead are required. Therefore, BSRP-Lag does not adopt this technique.

[0199] Effects obtained by the method of the present invention

[0200] The proposed method evaluated the algorithm using different robotic agents in Safety-Gymnasium1. The task in each environment is to navigate the robot to a specified location while avoiding collisions with obstacles. Safety-Gymnasium is an environment library specifically designed for SafeRL. This library extends the basic Gymnasium API and utilizes the high-performance MuJoCo engine for physics simulation. Experiments were conducted in five different environments: SafetyPointGoal1, SafetyPointGoal2, SafetyPointPush1, SafetyPointButton1, and SafetyCarGoal1. The safety evaluation of the algorithm was based on three key metrics: average reward return, average round cost return, and cost rate. These metrics have been widely recognized in the SafeRL community as comparison standards for evaluating agents: average undiscounted reward return over E rounds, and average undiscounted cost return over E rounds.

[0201] The method of the present invention is calculated by averaging the cost and reward over 10 rounds, with the length of each round being 1000, and the network is not updated during the evaluation.

[0202] (1). PPO-Lag, TRPO-Lag (no model): Lagrangian versions of PPO and TRPO.

[0203] (2). CPO (model-free): a policy search algorithm with near constraint satisfaction guarantees.

[0204] (3). Sauté PPO, TRPO (model-free): train safety policies by eliminating safety constraints into the state space and reshaping the objective function.

[0205] (4).Safe SLAC (model-based): a combination of SLAC and Lagrangian methods.

[0206] (5). LAMBDA (model-based): Based on the implementation in DreamerV1, it combines Bayesian and Lagrangian methods.

[0207] (6). MPC:sim (model-based): MPC is performed using a real simulator and the CCEM algorithm, called MPC:sim.

[0208] (7). MBPPO-Lag (model-based): trains security policies by integrating Gaussian models and Lagrangian methods.

[0209] (8). DreamerV3 (model-based): A world-model-based reinforcement learning algorithm that performs well in various control areas.

[0210] like θ As shown, the evaluation is performed after the agent completes 2M training steps. Ours (OSRP-Lag), Ours (OSRP), and Ours (BSRP-Lag) are the algorithms proposed by the present invention. The present invention follows the standardized indicators proposed by Safety-Gym and uses the rliable library to calculate the median (Median), interquartile mean (IQM), and mean (Mean) of the normalized reward performance (Normalized Reward Performance atConvergence) and the normalized cost performance (Normalized Reward Performance atConvergence) after convergence of the algorithm. SafeDreamer (BSRP-Lag) obtains rewards similar to those of the model-based safety reinforcement learning method LAMBDA, and improves safety by 94.3%. The training curves of the experiment are shown in Figure 3 and Figure 4 Each experiment was run with 5 different random seeds. SafeDreamer achieves near-zero-cost performance and outperforms existing model-free and model-based algorithms.

[0211] The dual optimization objective is to achieve a balance between maximizing rewards and minimizing costs. Figure 5As shown, SafeDreamer achieves higher rewards while achieving the minimum cost. In contrast, due to the inaccuracy of the world model, model-based algorithms such as Safe SLAC reach a point where the cost stops decreasing after it drops to a certain level. On the other hand, in environments with denser or more dynamic obstacles, such as SafetyPointButton1, MPC:sim has difficulty ensuring the long-term safety of the agent due to the lack of a cost critic. From the beginning of training, the algorithm of the present invention exhibits safe behavior and ensures safe exploration. Specifically, in the SafetyPointGoal1 and SafetyPointPush1 environments, SafeDreamer achieves reward performance similar to DreamerV3 while maintaining a near-zero cost.

[0212] Supports vision and low-dimensional vector input tasks. Evaluation is conducted in two low-dimensional input environments, Safety-PointGoal1 (vector, vector input) and SafetyCarGoal1 (vector, vector input) (see Figure 4 Figure 5 Although model-free algorithms can reduce costs, they struggle to achieve higher rewards. This is because they rely entirely on trial-and-error learning to achieve a safe policy, lacking the assistance of a world model. This makes it difficult for them to search for optimal solutions within a limited data sample. Furthermore, MBPPO-Lag's reward stops increasing once the cost reaches a relatively low level. SafeDreamer surpasses these algorithms in both reward and cost.

Claims

1. A robot obstacle avoidance method based on the WMSRL model is to add the WMSRL model to the robot motion coordination control module; it is characterized by Robot obstacle avoidance includes the following steps: Step 1: Robot motion trajectory expansion and reward value estimation and cost value estimation; Setting the current moment in the WMSRL model The state transition probability function under and the reward function ; Indicates taking action After that, from the current state Transition to the next state The transition probability of Indicates the current state Execute an action and transfer to the next state Rewards received when represents the state space; Let the state space Any state in , the initial state is recorded as ; represents the action space; Action Space Any action in ; Indicates the state transition probability identifier; Represents the reward set identifier; represents the cost set, and ,in is the cost function, is the cost threshold, subscript Indicates the cost threshold identification number, Indicates the total number of cost thresholds; Order cost set Any price is recorded as ; is the initial state distribution, and ; is the discount factor, and ; is the current moment of the robot's motion; is the moment before the robot moves; is the last moment of the robot's movement; use represents the state transition probability matrix; Any element in is represented by its state as ,and ; Represents the probability identifier under the steady-state strategy; are the network parameters that need to be learned; Indicates the real number space, the superscript Represents the dimension; Represents the state element of the matrix; Indicates the actions taken under the steady-state strategy After that, from the current state Transition to the next state The state transition probability of Any element in the state transition probability matrix By implementing a steady-state strategy From the current state Transition to the next state One-step state transition probability; when all elements complete one-step state transition, finally, As the initial state under the steady-state strategy The steady-state distribution of the Markov chain at the beginning, however, As the initial distribution under the steady-state strategy The distribution of discount status visits on ; represents the steady-state state of the Markov chain under the steady-state strategy; Indicates the current time Discount factor under ; Indicates the current state under the steady-state strategy The transition probability of is the initial state distribution, and ; represents the initial distribution under the steady-state strategy Discount status on; Among them, the infinite time discounted reward function is defined as: ； represents the reward function; Indicates expected value; Indicates the current time Discount factor under ; Indicates that the status Execute an action and transfer to the next state Rewards received when is the initial state distribution, and ; represents the probability sign under the steady-state strategy; Among them, the infinite time discounted cost function is defined as: ； Subscript Indicates the cost threshold identification number; Indicates that the status Execute an action and transfer to the next state Discount price obtained when Defining a set of feasible strategies for: ； is the cost threshold; Among them, the goal of the constrained Markov model CMDP is to The optimal strategy for searching is: ； represents the probability sign under the steady-state strategy; represents the set of feasible strategies; represents the reward function; Modify the state transition probability function of the robot's obstacle avoidance process and define the safety reinforcement learning problem based on the robot world model as follows: ； ； ； represents the probability sign under the steady-state strategy; represents the set of all steady-state strategies, and ; Represents the reward function in the secure reinforcement learning model; Represents the cost function in the secure reinforcement learning model; Represents a specified obstacle avoidance safety constraint threshold; Indicates that the status Execute an action and transfer to the next state Rewards received when Indicates the initial state; is the initial state distribution, and ; It is a parameterized world model; Represents a specified cost function; The design of the safety planning algorithm OSRP is to combine model planning with safety constraints in safety tasks that rely only on vision; the SR-MPC algorithm uses the constrained cross entropy method CCEM for planning and uses the reinforcement learning model to 、 、 and predictions are made separately; in addition, the reward valuation is also used and price valuation and a reward-driven policy network π_θ to guide the planning process; represents the predicted state sequence; represents the predicted dynamic state; Represents the predicted reward value; represents the predicted cost value; First, initialize the step length to The independent parameters of each action within the time range, that is, the action parameters ,in ; Next, use the world model to independently sample trajectories and estimate their infinite-time discounted rewards outside the planning horizon and infinite time discounted cost , this estimate is made by introducing a bootstrap-based returns to achieve: ； ； Indicates the current state; Indicates the current status Reward value returns; Superscript yes The value of returns; Indicates the next state Reward value returns; represents the planning step length; Indicates the current status The predicted reward value of is the discount factor, and ; Indicates the next state The estimated reward return value; Indicates the The estimated reward return value of the step state; Using the prediction of the cost model, given the state To evaluate the situation The total cost within the step: ； represents the planning step length; Indicates the current time Discount factor under ; Represents a sequence model; Indicates the current status The predicted cost value of represents the observation encoder; represents a normal distribution; represents the identity matrix; is the action mean; Superscript Indicates the total number of iterations, the number of iterations ; represents the action mean of the previous iteration; represents the action variance of the previous iteration; Use the cost value of all trajectories As An approximate estimate of is the round length; at the same time, as a criterion for evaluating trajectory safety; Step 2: The trade-off between the reward value and cost value of the safe trajectory; The trade-off between reward and cost is achieved by calculating the cost of all trajectories. Afterwards, the number of safe trajectories is expressed as ;if Lower than expected number of safe trajectories , it indicates that it is difficult to find a safe trajectory in the current state; therefore, it becomes crucial to prioritize safety, and The value of becomes the sorting criterion, recorded as , and the entire set of sampled robot motion trajectories becomes the candidate action set On the contrary, if , it means that the trajectory that meets the safety constraints is relatively easy to find; therefore, the reward value Used as sorting criteria , and the safe action trajectory The actions within are selected as candidate actions ; From the candidate action set , select the sorting criteria The corresponding maximum value, and the action sequence corresponding to the maximum value is used as the elite action ,in accordance with In the Get the first parameter in the iteration and the second parameter : ； ； represents the number of predefined elite actions; Represents any predefined elite action; After reaching the predetermined number of iterations After that, the robot motion planning process ends; at the current moment of each decision robot motion The mean of the first action of the planned action trajectory will be used as the action to be executed in the real environment; Step 3: Safe online planning of robot motion trajectory based on Lagrangian method; When the hazard detection field of view required for obstacle avoidance exceeds the planning field of view, the planning process may not be able to ensure zero-cost violations in complex vision-only tasks. To address this issue, a cost critic is used to estimate the expected cost in the later stages of the planning process. To fully utilize the potential of the cost critic, the PID Lagrangian method is integrated into the world model planning framework to obtain the OSRP-Lag algorithm, which satisfies the requirements in step 2. Sorting criteria when Adjust to ,in is the Lagrange multiplier; this means that when there are many safe trajectories during the robot's motion planning process, the LAG-MPC algorithm will dynamically adjust the Lagrange multiplier based on the robot's historical constraint violations, and continue to be cautious in safe exploration; Step 4: Construct the components of the world model of the robot's motion trajectory; The improved world model is used to train the historical data of the robot's interaction with the environment from step 3; this data includes observations, actions, rewards, costs, and round continuation indicators, and the time length is , expressed as ; Indicates the current state; Indicates the current action; Indicates the next state; Indicates the next reward; Indicates the next generation valence; Indicates the next moment round continuation indicator; The improved world model includes a sequential model denoted as , the observation encoder is denoted as The dynamic model is recorded as , the reward decoder is denoted as , the cost decoder is denoted as , the round continuation indicator decoder is recorded as and the observation decoder is denoted as ; represents the posterior distribution of the observation encoder prediction; represents the prior distribution of the kinetic model predictions; represents the hidden state predicted by the sequence model; Indicates the previous action; Indicates the observation state actually received in the environment; Indicates the current reward; Indicates the current price; Indicates the continuation indicator of the current round; Adopt Gated Recurrent Unit (GRU); for visual input, use Convolutional Neural Network (CNN) as observation encoder and observation decoder , and for low-dimensional input, a multi-layer perceptron (MLP) is used; the reward, cost, round continuation marker decoder and dynamics model are implemented using MLP; this world model can be understood as a sequence variational autoencoder VAE, where the output of the observation encoder acts as an approximate posterior, which first transforms the observation Encoded as the posterior distribution, i.e. Output , given the initial state and , and the previous action , loop state and , use the sequence model to predict the next cycle state during the world model training process However, during policy training or online planning, since there is no access to true future observations, the sequence model uses its last output As input instead of Dynamic utilization Predict the prior distribution, i.e. Output ; By and Connection is model training status , using it to predict rewards, costs, and round continuation indicators; Step 5: Calculate the loss function of the world model; All modules of the improved world model are jointly optimized; the loss function consists of two parts: KL divergence loss and decoder loss; given a batch size of sequence , the model can be optimized end-to-end by minimizing the following loss: ； Loss Function It is composed of the log-likelihood between the distributions generated by the decoder for observations, rewards, costs, and episode continuation markers and their corresponding labels: ； represents the loss function of the observation decoder; Represents the weight coefficient of the loss function of the observation decoder; represents the loss function of the reward decoder; Represents the weight coefficient of the loss function of the reward decoder; represents the loss function of the cost decoder; Represents the weight coefficient of the loss function of the cost decoder; represents the loss function of the round continuation marker decoder; The weight coefficient representing the loss function of the round continuation marker decoder; When processing visual input, each weight coefficient 、 、 and are all set to 1.0; however, in the case of low-dimensional vector inputs, since the prediction cost always depends on the accurate reconstruction of the observations, is set to 5.0; the loss function It is calculated by measuring the KL divergence between the dynamics predictor and the observation encoder; the main goal is to train the sequence model and the dynamics predictor using The previously available information is used to generate a prior distribution that is as close as possible to the posterior distribution obtained using the encoder, which is achieved by minimizing At the same time, the observation encoder is trained to extract relevant information from the observations and enhance the predictive power of the dynamics predictor, which is achieved by minimizing The first item in is implemented; the loss function The calculation is as follows: ； Represents the KL divergence between the two distributed elements; sg represents the stop gradient operator; is the first constant coefficient; is the second constant coefficient; Step 6: Strategy learning based on the actor-critic model; In the strategy learning process based on the actor-critic model, Tagged as actor, Mark as reward critics, Mark as cost critic; use the improved world model to train the model state Make predictions and get potential states ; Discretize the target value that the critic needs to regress, and then train the critic network to predict each discrete value Twohot softmax probability , the reward and cost critic networks are trained using the following loss functions: ； ； ； Indicates the number of predefined discrete values, using 255 discrete values. , Represents an array of discrete values ​​of length 255, ; Two-hot encoding extends the concept of one-hot encoding to continuous values; therefore, the target value can be predicted and recovered by the following formula: ； where symexp . The symexp function is the inverse function of symlog; The objective function of the actor model is to maximize the expected reward and the entropy of the action The gradient of the first term is estimated using stochastic backpropagation, while the gradient of the second term is calculated analytically. ； is a fixed coefficient; The actor strategy is updated using the augmented Lagrangian method. The main goal is to minimize the expected cost while maximizing the expected reward. The loss function is as follows: ； ； ； express Reward value in the state returns; express The cost value of the state returns; Indicates the current gradient update round; Represents the Lagrange multiplier of the current gradient update round; Represents the Lagrangian penalty factor for the current gradient update round.

2. The robot obstacle avoidance method based on the WMSRL model according to claim 1 is characterized in that: The WMSRL model consists of two parts: one is the use of safe reinforcement learning in the safe exploration of robot posture information, and the other is the application of an improved world model to strategically learn the robot posture information.

3. The robot obstacle avoidance method based on the WMSRL model according to claim 1, characterized in that: and They are set to 0.5 and 0.1 respectively in all tasks.

4. The robot obstacle avoidance method based on the WMSRL model according to claim 1, characterized in that: The improved world model used in the WMSRL model fully improves the utilization efficiency of the robot's posture information during motion, thereby ensuring the security of the entire reinforcement learning training and deployment process.

Images