Robot navigation and obstacle avoidance task implementation method and system based on reinforcement learning

Abstract

The present invention provides a method and system for implementing a robot navigation obstacle avoidance task based on reinforcement learning, including: modeling the navigation task as a Markov process, designing a state space, action space and reward function for the reinforcement learning method, and determining the index for completing the task; analyzing the constraints of the control barrier function, adjusting the parameters, and training to obtain a parameterized control barrier function; using the state space, action space and reward function designed in the modeling step, and adjusting the hyperparameters in the reinforcement learning algorithm, training to obtain a policy network; modifying the output of the policy network according to the learned control barrier function, while collecting data and updating the control barrier function; applying noise to dangerous actions through predictive shielding control, so that the intelligent agent deviates from the predetermined trajectory and makes the training process safe. Based on the framework of reinforcement learning, the present invention integrates a provably effective control barrier function to ensure safe exploration during the learning process.

Description

Technical Field

[0001] The present invention relates to the technical field of safe learning, and in particular to a method and system for realizing a robot navigation and obstacle avoidance task based on reinforcement learning. Background Art

[0002] Reinforcement Learning (RL) is a sequential decision-making process in which an agent learns effective policies through interaction with the environment. To search for the optimal policy, RL allows the agent to explore unvisited states and try new actions to find the most rewarding policy. However, unconstrained exploration may lead to unsafe actions that endanger system safety. For example, when training a drone to perform an autonomous navigation task, the drone may collide with obstacles or other drones during exploration. Therefore, safe exploration plays a crucial role in RL to ensure system safety.

[0003] The main problem of navigation and obstacle avoidance research is to deploy the intelligent agent in the starting area, hoping that it can reach the target location and complete the transportation task. In safe learning, it is necessary not only to obtain a strategy that can quickly reach the target location, but also to ensure safety during the entire learning period.

[0004] The Chinese invention patent document with publication number CN113359717A discloses a mobile robot navigation and obstacle avoidance method based on deep reinforcement learning. A neural network is used to extract the features of pairwise interactions between the robot and each person, and the interactions between humans are captured through local maps. A self-attention mechanism is used to aggregate interaction features to infer the relative importance of neighboring humans relative to their future states, and a reinforcement learning method is used to pre-train the value network. A safe action space is added during the training process, ultimately realizing the mobile robot navigation and obstacle avoidance function.

[0005] Regarding the above-mentioned related technologies, the inventor believes that most of the existing methods have problems such as design difficulties, or are heuristic, and the superiority of the method is proved by experimental results. In addition, the movement of the agent needs to be restricted to keep its underlying communication network always connected. It is very promising to develop a control framework that enables the agent to learn strategies to safely complete the navigation task. Summary of the invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for realizing robot navigation and obstacle avoidance tasks based on reinforcement learning.

[0007] A method for implementing a robot navigation obstacle avoidance task based on reinforcement learning provided by the present invention comprises the following steps:

[0008] Modeling steps: Model the navigation task as a Markov process, design the state space, action space and reward function for the reinforcement learning algorithm, and determine the indicators for completing the task;

[0009] Control barrier function training steps: analyze the constraints of the control barrier function, adjust the parameters, and train to obtain a parameterized control barrier function;

[0010] Policy network training step: Use the state space, action space, and reward function designed in the modeling step, and adjust the hyperparameters in the reinforcement learning algorithm to train the policy network;

[0011] Control barrier function update step: According to the learned control barrier function, modify the output of the policy network, collect data and update the control barrier function;

[0012] Ensure safety steps: Through predictive masking control, noise is added to dangerous actions to make the agent deviate from the intended trajectory, making the training process safe.

[0013] Preferably, in the modeling step, the navigation model of the agent is measured by the geometric relationship between the target point and the agent position, and the definition is

[0014]

[0015] Among them, d i (t) represents the two-dimensional space distance between agent i and the corresponding target point at time t; q i (t) is the current position of the agent; p i is the target position corresponding to the agent; Represents a definition;

[0016] The agent is represented as a circular area with a radius of R, and the agents exchange information within a given maximum communication distance R, that is,

[0017] R ij =||q i -q j ||≤R;

[0018] Among them, R ij represents the distance between agent i and agent j; q i represents the location information of agent i; q j Represents the location information of agent j;

[0019] The transport navigation task is modeled by a Markov process as follows:

[0020] State space S: agent state Contains the two-dimensional coordinates of agent i in space at time t And the direction angle The state space is set to the state set of all N agents at time T, that is,

[0021] Where i = 1, 2, ..., N, t = 1, 2, ..., T;

[0022] in, represents the horizontal coordinate of the ith agent at time t; represents the ordinate of the i-th agent at time t;

[0023] Action Space Assume that each agent selects a direction at a time and moves a specified distance in the selected direction; thus the action of agent i at time t is The speed from the upward direction and angular velocity Composition, that is

[0024]

[0025] and then,

[0026]

[0027] Reward function: The reward function includes navigation task reward and virtual boundary penalty;

[0028] Navigation mission reward m t :When the agent approaches the target position, it gets the navigation task reward:

[0029]

[0030] Among them, c p and c t is a positive constant; r goal is the reward value; d i is the distance to the destination; δ is the allowable arrival error; t step is the number of steps to complete the task; B represents the given exploration boundary value; Indicates all; Indicates existence;

[0031] Virtual boundary penalty n t : Introduce virtual boundaries in the training environment and limit the exploration range of the agent by setting penalties;

[0032] When the distance between the agent and the target position is greater than the given value B, it will be penalized with a negative value.

[0033] n t =c n ,

[0034] Among them, c n is a negative constant;

[0035] Then, construct the reward function r of the entire task t :

[0036]

[0037] Preferably, in the control barrier function training step,

[0038] In order to obtain the control barrier function, we first define the expert trajectory P exp :

[0039]

[0040] Among them, ∶= means that it is defined as; x i is the state of the agent, x i ∈S;u i is the action information of the intelligent agent;

[0041] pass

[0042]

[0043] in, is an n-dimensional real number space; x represents an n-dimensional state; ||·|| p represents the p-norm;

[0044] Get defined in x i The p-normal ball B with a surrounding radius ∈ ∈,p (x i );

[0045] pass

[0046]

[0047] D∶=D′\bd(D′)

[0048] Get area D;

[0049] Where bd(D′) represents the boundary of D′; D and D′ are both regions consisting of points; ∪ represents union; D′ is the union of N paradigm balls; \ represents removal; D is an open set that does not contain the boundary of D′;

[0050] Define the Minkowski sum as

[0051]

[0052] in, represents the Minkowski sum; C1 and C2 are two sets;

[0053] pass

[0054]

[0055] Define a set L, where L represents a layer with a width of σ around D; B σ,p (0) represents the p-normal ball with radius σ around 0; bd(D) represents the boundary of D;

[0056] Sample the dataset X from L L

[0057]

[0058] Define the security state set X safe

[0059] X safe ={x i :(x i ,u i )∈P exp},

[0060] Among them,: represents the trajectory from the expert P exp The state-action pair (x i ,u i ) to extract the status information x i ;

[0061] Define safe set X safe Subset of for

[0062]

[0063] Among them, inf means taking the lower limit; the hyperparameter γ safe , γ unsafe is a positive constant, L h represents the local Lipschitz bound;

[0064] Local Lipschitz bound L h (x) is defined as

[0065]

[0066] Among them, sup represents the upper limit, B ∈,p (x) represents the p-normal ball region with radius ∈ around x; h(x) represents the control barrier function, h(x1) represents the control barrier function value of x1; h(x2) represents the control barrier function value of x2;

[0067] An effective control barrier function h(x) on D is defined as: there exists a local Lipschitz continuous extension K-type function α such that the following holds for all x∈D;

[0068]

[0069]

[0070] Where u(x) is the action taken by the agent when the state is x; f(x) and q(x) represent the nonlinear system The dynamic model of is the symbol of partial derivative;

[0071] In order to ensure the effectiveness of the obtained control barrier function, ∈ has constraints;

[0072] Define the function g(x):

[0073] q(x): = L f h(x)+L g h(x)u i +α(h(x))

[0074] Solve the following optimization problem to obtain the required control barrier function h:

[0075] min||h||

[0076]

[0077] h(x i )≤-γ unsafe

[0078]

[0079] q(x i ,u i )∶=L f h(x i )+L g h(x i ) i +α(h(x i ))≥γ exp

[0080] Lip(q(x i ,u i ),∈)≤L q ,

[0081] Among them, the hyperparameter γ exp , L h and L q is a positive constant; h(x i ) means that in x i When controlling the barrier function value, is a subset of the safe set, Lip(·,∈) represents the upper bound of the Lipschitz constant of the parameter in the neighborhood of ∈, Indicates that the parameter is The upper bound of the Lipschitz constant in the neighborhood; st represents the constraint condition;

[0082] For all x i ∈X L , Should meet For all x i ∈X safe ,∈ should satisfy∈≤γ exp / L q (x i );

[0083] Among them, L h (x i ) and L q (x i ) represents the local Lipschitz bound;

[0084] at this time,

[0085]

[0086]

[0087] Among them, q(x i ) means when x=x i The function value of q(x) when .

[0088] Preferably, in the strategy network training step,

[0089] Considering the continuous state and action space, x is the state and a is the action taken in state x; For the action state value, the Actor-Critic framework is adopted to define two critic networks, namely and Defining the actor network The parameters are θ1, θ2 and The target network parameters θ1′, θ2′ and The critic network is updated as follows:

[0090]

[0091]

[0092] Among them, y is the update target; Representative Experience Pool On expectations; Stores the agent's N-step experience; r is the reward after taking action a in the current state; γ is the discount factor for the cumulative reward; x′ represents the state at the next moment; a′ represents the action value at the next moment; θ i represents the critic network parameters; θ i ′ represents the target network parameters; represents the critic network; represents the target network; a N represents the action of step N, N Indicates the reward obtained in step N;

[0093] Actor updates through policy gradients:

[0094]

[0095] in, is the objective function; is the objective function The gradient of is the action chosen by the actor network in state x; and is the gradient of the network.

[0096] Preferably, in the control barrier function updating step, the controller output action obtained in the policy network training step is modified in combination with the control barrier function obtained in the control barrier function training step to obtain a safe action, and new data X generated by the interaction with the environment is collected. RL , select data away from the unsafe set and update the control barrier function.

[0097] Preferably, in the step of ensuring safety, predictive shielding control is used to shield the output of the learned control barrier function in dangerous situations, and the subsequent H-step states are predicted through the known system dynamics model; if no safety constraints are violated, the action selected by the control barrier function is executed; otherwise, random Gaussian noise will be added to the action until no collision occurs within a finite number of steps H.

[0098] A robot navigation obstacle avoidance task realization system based on reinforcement learning provided by the present invention includes the following modules:

[0099] Modeling module: Model the navigation task as a Markov process, design the state space, action space and reward function for the reinforcement learning algorithm, and determine the indicators for completing the task;

[0100] Control barrier function training module: analyzes the constraints of the control barrier function, adjusts the parameters, and trains to obtain a parameterized control barrier function;

[0101] Policy network training module: Use the state space, action space and reward function designed in the modeling module, and adjust the hyperparameters in the reinforcement learning algorithm to train the policy network;

[0102] Control barrier function update module: Modify the output of the policy network according to the learned control barrier function, collect data and update the control barrier function;

[0103] Safety ensuring module: Through predictive shielding control, noise is applied to dangerous actions to make the agent deviate from the predetermined trajectory, making the training process safe.

[0104] Preferably, in the modeling module, the navigation model of the agent is measured by the geometric relationship between the target point and the agent position, and the definition is

[0105]

[0106] Among them, d i (t) represents the two-dimensional space distance between agent i and the corresponding target point at time t; q i (t) is the current position of the agent; p i is the target position corresponding to the agent; Represents a definition;

[0107] The agent is represented as a circular area with a radius of R, and the agents exchange information within a given maximum communication distance R, that is,

[0108] R ij =||q i -q j ||≤R;

[0109] Among them, R ij represents the distance between agent i and agent j; q i represents the location information of agent i; q j Represents the location information of agent j;

[0110] The transport navigation task is modeled by a Markov process as follows:

[0111] State space S: agent state Contains the two-dimensional coordinates of agent i in space at time t And the direction angle The state space is set to the state set of all N agents at time T, that is,

[0112] Where i = 1, 2, ..., N, t = 1, 2, ..., T;

[0113] in, represents the horizontal coordinate of the ith agent at time t; represents the ordinate of the i-th agent at time t;

[0114] Action Space Assume that each agent selects a direction at a time and moves a specified distance in the selected direction; thus the action of agent i at time t is The speed from the upward direction and angular velocity Composition, that is

[0115]

[0116] and then,

[0117] i=1,2,…,N, t=1,2,…,T;

[0118] Reward function: The reward function includes navigation task reward and virtual boundary penalty;

[0119] Navigation mission reward m t :When the agent approaches the target position, it gets the navigation task reward:

[0120]

[0121] Among them, c p and c t is a positive constant; r goal is the reward value; d i is the distance to the destination; δ is the allowable arrival error; t step is the number of steps to complete the task; B represents the given exploration boundary value; Indicates all; Indicates existence;

[0122] Virtual boundary penalty n t : Introduce virtual boundaries in the training environment and limit the exploration range of the agent by setting penalties;

[0123] When the distance between the agent and the target position is greater than the given value B, it will be penalized with a negative value.

[0124] n t =c n ,

[0125] Among them, c n is a negative constant;

[0126] Then, construct the reward function r of the entire task t :

[0127]

[0128] Preferably, in the control barrier function training module,

[0129] In order to obtain the control barrier function, we first define the expert trajectory P exp :

[0130]

[0131] Among them, ∶= means that it is defined as; x i is the state of the agent, x i ∈S;u i is the action information of the intelligent agent;

[0132] pass

[0133]

[0134] in, is an n-dimensional real number space; x represents an n-dimensional state; ||·|| p represents the p-norm;

[0135] Get defined in x i The p-normal ball B with a surrounding radius ∈ ∈,p (x i );

[0136] pass

[0137]

[0138] D∶=D′\bd(D′)

[0139] Get area D;

[0140] Where bd(D′) represents the boundary of D′; D and D′ are both regions consisting of points; ∪ represents union; D′ is the union of N paradigm balls; \ represents removal; D is an open set that does not contain the boundary of D′;

[0141] Define the Minkowski sum as

[0142]

[0143] in, represents the Minkowski sum; C1 and C2 are two sets;

[0144] pass

[0145]

[0146] Define a set L, where L represents a layer with a width of σ around D; B σ,p(0) represents the p-normal ball with radius σ around 0; bd(D) represents the boundary of D;

[0147] Sample data set X from L L

[0148]

[0149] Define the security state set X safe

[0150] X safe ={x i :(x i ,u i )∈P exp},

[0151] Among them,: represents the trajectory from the expert P exp The state-action pair (x i ,u i ) to extract the status information x i ;

[0152] Define safe set X safe Subset of for

[0153]

[0154] Among them, inf means taking the lower limit; the hyperparameter γ safe , γ unsafe is a positive constant, L h represents the local Lipschitz bound;

[0155] Local Lipschitz bound L h (x) is defined as

[0156]

[0157] Among them, sup represents the upper limit, B ∈,p (x) represents the p-normal ball region with radius ∈ around x; h(x) represents the control barrier function, h(x1) represents the control barrier function value of x1; h(x2) represents the control barrier function value of x2;

[0158] An effective control barrier function h(x) on D is defined as: there exists a local Lipschitz continuous extension K-type function α such that the following holds for all x∈D;

[0159]

[0160]

[0161] Where u(x) is the action taken by the agent when the state is x; f(x) and g(x) represent the nonlinear system The dynamic model of is the symbol of partial derivative;

[0162] In order to ensure the effectiveness of the obtained control barrier function, ∈ has constraints;

[0163] Define the function q(x):

[0164] q(x): = L f h(x)+L g h(x)u i +α(h(x))

[0165] Solve the following optimization problem to obtain the required control barrier function h:

[0166] min||h||

[0167] sth(x i )≥γ safe ,

[0168] h(x i )≤-γ unsafe

[0169]

[0170] q(x i ,u i )∶=L f h(x i )+L g h(x i ) i +α(h(x i ))≥γ exp

[0171] Lip(q(x i ,u i ),∈)≤L q ,

[0172] Among them, the hyperparameter γ exp , L h and L q is a positive constant; h(x i ) means that in x i When controlling the barrier function value, is a subset of the safe set, Lip(·,∈) represents the upper bound of the Lipschitz constant of the parameter in the neighborhood of ∈, Indicates that the parameter is The upper bound of the Lipschitz constant in the neighborhood; st represents the constraint condition;

[0173] For all x i ∈X L , Should meet For all x i ∈X safe ,∈ should satisfy∈≤γ exp / L q (x i );

[0174] Among them, L h (x i ) and L q (x i ) represents the local Lipschitz bound;

[0175] at this time,

[0176]

[0177]

[0178] Among them, q(x i ) means when x=x i The function value of q(x) when .

[0179] Preferably, in the strategy network training module,

[0180] Considering the continuous state and action space, x is the state and a is the action taken in state x; For the action state value, the Actor-Critic framework is adopted to define two critic networks, namely and Defining the actor network The parameters are θ1, θ2 and The target network parameters θ1′, θ2′ and The critic network is updated as follows:

[0181]

[0182]

[0183] Among them, y is the update target; Representative Experience Pool On expectations; Stores the agent's N-step experience; r is the reward after taking action a in the current state; γ is the discount factor for the cumulative reward; x′ represents the state at the next moment; a′ represents the action value at the next moment; θ irepresents the critic network parameters; θ i ′ represents the target network parameters; represents the critic network; represents the target network; a N represents the action of step N, N Indicates the reward obtained in step N;

[0184] Actor updates are done through policy gradients:

[0185]

[0186] in, is the objective function; is the objective function The gradient of is the action chosen by the actor network in state x; and is the gradient of the network.

[0187] Compared with the prior art, the present invention has the following beneficial effects:

[0188] 1. Based on the framework of reinforcement learning, this invention integrates a provably effective control barrier function to ensure safe exploration during the learning process;

[0189] 2. Since it is very challenging to manually synthesize the control barrier function of a complex dynamic system, the present invention uses a data-driven approach to learn the function from the collected demonstrations of safe and ideal behaviors, and considers online learning of the control barrier function, learning and updating from the online generated data during reinforcement learning training, which ensures safety while improving the efficiency of strategy exploration and the adaptability to complex environments;

[0190] 3. The present invention proposes a predictive shielding control method to cope with the inaccuracy of the neural network and further ensure the safety of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0191] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0192] Figure 1 A task diagram of a robot navigation and obstacle avoidance method based on reinforcement learning provided by the present invention;

[0193] Figure 2 A pseudo code diagram of a data collection algorithm for training a control barrier function provided by the present invention;

[0194] Figure 3 This is a pseudo code diagram of the model prediction shielding control algorithm provided by the present invention. DETAILED DESCRIPTION

[0195] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0196] The embodiment of the present invention discloses a method for realizing a robot navigation obstacle avoidance task based on reinforcement learning, such as Figure 1 As shown in the figure, a control framework based on TD3 (Double Delayed Deep Deterministic Policy Gradient) is developed. The state space, action space and reward function of the algorithm are designed by modeling the navigation problem. The actions are modified by a dynamically updateable control barrier function to ensure safety during learning while the agent is able to complete the set navigation task. The required control barrier function is obtained by neural network training, and predictive masking control is used to ensure the safety of the agent during exploration.

[0197] Predictive shielding control is adopted. If within the predicted number of steps, the system takes the controller output action that will lead the intelligent agent to an unsafe area, noise is added to the dangerous action to make it deviate from the output value until the intelligent agent is always in the safe area. At this time, the intelligent agent runs forward one step according to the modified action sequence, and this cycle is repeated to ensure the safety of the system.

[0198] The reinforcement learning-based framework (framework of a robot navigation and obstacle avoidance method based on reinforcement learning) includes the following steps:

[0199] Modeling steps: Model the navigation task as a Markov decision process (MDP), design the state space, action space and reward function for the reinforcement learning algorithm, and determine the indicators for completing the task.

[0200] Specifically, the navigation model of the agent is measured by the geometric relationship between the target point and the agent's position, and is defined as

[0201]

[0202] Among them, d i (t) represents the two-dimensional space distance between agent i and the corresponding target point at time t; q i (t) is the current position of the agent; p i is the target position corresponding to the agent; Represents a definition.

[0203] The agent has limited communication capabilities, represented as a circular area with a radius of R. Agents can only exchange information within this distance. That is, the agent is represented as a circular area with a radius of R. Agents exchange information within a given maximum communication distance R.

[0204] R ij =||q i -q j ||≤R;

[0205] Among them, R ij represents the distance between agent i and agent j; q i represents the location information of agent i; q j Represents the location information of agent j. i and q j represents the location information of agents i and j, and R is the given maximum communication distance.

[0206] The transportation navigation task is modeled by Markov process as follows (the transportation navigation task can be modeled by MDP as follows):

[0207] State space S: agent state Contains the two-dimensional coordinates of agent i in space at time t And the direction angle The state space is set to the state set of all N agents at time T, that is,

[0208] Where i=1,2,…,N, t=1,2,…,T.

[0209] in, represents the horizontal coordinate of the ith agent at time t; Represents the vertical coordinate of the i-th agent at time t.

[0210] Action Space Assume that each agent selects a direction at a time and moves a specified distance in the selected direction; thus the action of agent i at time t is The speed from the upward direction and angular velocity Composition, that is

[0211]

[0212] and then,

[0213]

[0214] Reward function: The reward function includes navigation task reward and virtual boundary penalty.

[0215] Navigation mission reward mt :When the agent approaches the target location, it will get a navigation task reward. When the agent approaches the target location, it will get this reward, which is in the form of:

[0216]

[0217] Among them, m t represents navigation task reward; c p and c t is a positive constant, r goal is a large reward value, d i is the distance to the destination, δ is the allowable arrival error, t step is the number of steps to complete the task, and B represents the given exploration boundary value; Indicates all, Indicates existence.

[0218] Virtual boundary penalty n t : Introduce virtual boundaries in the training environment and set penalties to limit the exploration range of the agent, improve data utilization, and thus improve training efficiency. The specific implementation is: when the distance between the agent and the target position is greater than the given value B, it will be punished with a negative value, the specific form is:

[0219] n t =c n ,

[0220] Among them, n t represents the virtual boundary penalty; c n A negative constant.

[0221] Then, construct the reward function r of the entire task t :

[0222]

[0223] Among them, r t Represents the reward function for the entire task.

[0224] Control barrier function training steps: analyze the constraints of the control barrier function, adjust the parameters, and train to obtain a parameterized control barrier function.

[0225] Specifically, in order to obtain the control barrier function, we first define the expert trajectory P exp :

[0226]

[0227] Among them, ∶= means that it is defined as; x i is the state of the agent, x i ∈S;u iis the action information of the agent, x i ,u i They are the state and action information of the agent respectively.

[0228] pass

[0229]

[0230] in, is an n-dimensional real number space; x represents an n-dimensional state; ||·|| p represents the p-norm.

[0231] Get defined in x i The p-normal ball B with a surrounding radius ∈ ∈,p (x i ).

[0232] pass

[0233]

[0234] D∶=D′\bd(D′)

[0235] Get area D;

[0236] Among them, bd(D′) represents the boundary of D′; D and D′ are both regions composed of points; ∪ represents union; D′ is the union of N paradigm balls; \ represents removal; D is an open set that does not contain the boundary of D′.

[0237] Define the Minkowski sum as

[0238]

[0239] in, represents the Minkowski sum; C1 and C2 are two sets. x1,x2 represent any point in the p-normal ball with radius ∈ around x.

[0240] pass

[0241]

[0242] Define a set L, where L represents a layer with a width of σ around D; B σ,p (0) represents the p-normal ball with radius σ around 0; bd(D) represents the boundary of D.

[0243] Sample the dataset X from L L

[0244]

[0245] Define the security state set X safe

[0246] X safe ={x i :(x i ,u i )∈P exp}

[0247] Among them,: represents the trajectory from the expert P exp The state-action pair (x i ,u i ) to extract the status information x i .

[0248] Define safe set X safe Subset of for

[0249]

[0250] Among them, inf means taking the lower limit; the hyperparameter γ safe , γ unsafe is a positive constant, L h represents the local Lipschitz bound. x represents the data set X L The point in i Represents the dataset X L The point in ||·|| p represents the p-norm.

[0251] Local Lipschitz bound L h (x) is defined as

[0252]

[0253] Among them, sup represents the upper limit, B ∈,p (x) represents the p-normal ball region with radius ∈ around x; x1, x2 represent any point in the p-normal ball with radius ∈ around x. h(x) represents the control barrier function, h(x1) represents the control barrier function value of x1; h(x2) represents the control barrier function value of x2. h(x1), h(x2) represent the control barrier function values ​​of x1, x2.

[0254] An effective control barrier function h(x) on D is defined as: there exists a local Lipschitz continuous extension K-type function α such that the following holds for all x∈D.

[0255]

[0256]

[0257] Where u(x) is the action taken by the agent when the state is x; u is the action taken by the agent, and U is all the control inputs that can be taken. f(x) and g(x) represent the nonlinear system The dynamic model of is the symbol of the partial derivative. f h(x) and L g h(x) represents substitution and has no meaning.

[0258] In order to ensure the effectiveness of the obtained control barrier function, ∈ has certain constraints.

[0259] Define the function q(x):

[0260] q(x): = L f h(x)+L g h(x)u i +α(h(x))

[0261] Solve the following optimization problem to obtain the required control barrier function h:

[0262] min||h||

[0263] sth(x i )≥γ safe ,

[0264] h(x i )≤-γ unsafe

[0265]

[0266] q(x i ,u i )∶=L f h(x i )+L g h(x i ) i +α(h(x i ))≥γ exp

[0267] Lip(q(x i ,u i ),∈)≤L q , Among them, the hyperparameter γ exp , L h and L q is a positive constant; h(x i ) means that in x i When controlling the barrier function value, is a subset of the safe set, Lip(·,∈) represents the upper bound of the Lipschitz constant of the parameter in the neighborhood of ∈, Indicates that the parameter is The upper bound of the Lipschitz constant in the neighborhood; st represents the constraint. Here is the constraint of min||h||. Lip(·,∈), Respectively represent the first parameter (i.e. the omitted parameter) in the ∈ neighborhood and The upper bound of the Lipschitz constant in the neighborhood is defined at x i The p-normal ball B with a surrounding radius ∈ ∈,p (x i ) and defined in x i The p-normal ball B with a surrounding radius ∈ ∈,p (x i ) on the Lipschitz bound. L q represents the local Lipschitz bound.

[0268] For all x i ∈X L , Should meet For all x i ∈X safe ,∈ should satisfy∈≤γ exp / L q (x i ).

[0269] Among them, L h (x i ) and L q (x i ) represents the local Lipschitz bound.

[0270] at this time,

[0271]

[0272]

[0273] Among them, q(x i ) means when x=x i The function value of q(x) when .

[0274] Policy network training step: Use the state space, action space, and reward function designed in the modeling step, and adjust the hyperparameters in the reinforcement learning algorithm to train a reasonable policy network.

[0275] Specifically, considering the continuous state and action space, x is the state, a is the action taken under state x; x and a are the state and the action taken under the state, respectively. For the action state value, the Actor-Critic (AC) framework is adopted to define the critic network Define two critic networks, namely and Defining the actor network The parameters are θ1, θ2 and The target network parameters θ1′, θ2′ and The critic network is updated as follows:

[0276]

[0277]

[0278] Among them, y is the update target; Representative Experience Pool On expectations; Stores the agent's N-step experience; r is the reward after taking action a in the current state; γ is the discount factor for the cumulative reward; x' represents the state at the next moment; a' represents the action value at the next moment; x', a' represent the state and action value at the next moment. θ i represents the critic network parameters; θ i ′ represents the target network parameters; represents the critic network; represents the target network; a N represents the action of step N, and N represents the reward obtained in step N.

[0279] Actor updates through policy gradients:

[0280]

[0281] in, is the objective function; is the objective function The gradient of is the action chosen by the actor network in state x; and is the gradient of the network. Stores the agent’s N-step experience.

[0282] Control barrier function update step: According to the learned control barrier function, modify the output of the policy network and collect data for updating the control barrier function.

[0283] Specifically, the control barrier function obtained in the control barrier function training step is combined to modify the controller output action obtained in the policy network training step to obtain a safe action and collect new data X generated by the interaction with the environment. RL , select data away from the unsafe set and update the control barrier function. Figure 2 The pseudo code of the data collection algorithm used to train the control barrier function is shown below. d represents the update frequency. demis the initial expert dataset given at the beginning of training, (P RL It is a collection of state-action pairs that meet the conditions during RL training).

[0284] Ensure safety steps: Ensure safety during training by predicting shielding control and applying noise to dangerous actions to make the agent deviate from the predetermined trajectory.

[0285] Specifically, in order to improve the safety of the system during learning, especially in the early stages of training, the present invention develops a predictive shielding control module. The predictive shielding control module improves the safety of the system during learning, such as Figure 1 As shown in the overall framework diagram, in dangerous situations, the output (i.e., action) of the learned control barrier function is shielded to keep the system safe. The specific process is as follows Figure 3 The pseudo code of the model predictive shielding control algorithm is shown in Figure 1. With the known system dynamics model, we can predict the subsequent H-step state. If no safety constraints are violated, the action selected by the barrier function is executed. Otherwise, random Gaussian noise will be added to the action until no collision occurs within a finite number of steps H. πpsc (x) indicates the strategy π of predictive masking control output in state x psc , so that the system transfers to the next state.

[0286] The Twin Delayed Deep Deterministic Policy Gradient algorithm is based on the Actor-Critic (AC) framework. It uses two sets of networks to estimate the action state value Q and uses the smaller one as the update target to solve the problem of overestimation of Q value.

[0287] Predictive shielding control, in dangerous situations, shields the output of the learned control barrier function and predicts the subsequent H-step state through the known system dynamics model; if no safety constraints are violated, the action selected by the control barrier function is executed; otherwise, random Gaussian noise will be added to the action until no collision occurs within a finite number of steps H.

[0288] The embodiment of the present invention also discloses a robot navigation obstacle avoidance task implementation system based on reinforcement learning, such as Figure 1 As shown, it includes the following modules:

[0289] Modeling module: Model the navigation task as a Markov process, design the state space, action space and reward function for the reinforcement learning algorithm, and determine the indicators for completing the task.

[0290] The navigation model of the agent is measured by the geometric relationship between the target point and the agent's position, defined as

[0291]

[0292] Among them, d i (t) represents the two-dimensional space distance between agent i and the corresponding target point at time t; q i (t) is the current position of the agent; p i is the target position corresponding to the agent; Represents a definition.

[0293] The agent is represented as a circular area with a radius of R, and the agents exchange information within a given maximum communication distance R, that is,

[0294] R ij =||q i -q j ||≤R;

[0295] Among them, R ij represents the distance between agent i and agent j; q i represents the location information of agent i; q j Represents the location information of agent j.

[0296] The transport navigation task is modeled by a Markov process as follows:

[0297] State space S: agent state Contains the two-dimensional coordinates of agent i in space at time t And the direction angle The state space is set to the state set of all N agents at time T, that is,

[0298] Where i=1,2,…,N, t=1,2,…,T.

[0299] in, represents the horizontal coordinate of the ith agent at time t; Represents the vertical coordinate of the i-th agent at time t.

[0300] Action Space Assume that each agent selects a direction at a time and moves a specified distance in the selected direction; thus the action of agent i at time t is The speed from the upward direction and angular velocity Composition, that is

[0301]

[0302] and then,

[0303]

[0304] Reward function: The reward function includes navigation task reward and virtual boundary penalty.

[0305] Navigation mission reward m t :When the agent approaches the target position, it gets the navigation task reward:

[0306]

[0307] Among them, c p and c t is a positive constant; r goal is the reward value; d i is the distance to the destination; δ is the allowable arrival error; t step is the number of steps to complete the task; B represents the given exploration boundary value; Indicates all; Indicates existence.

[0308] Virtual boundary penalty n t : Introduce virtual boundaries in the training environment and limit the exploration range of the agent by setting penalties;

[0309] When the distance between the agent and the target position is greater than the given value B, it will be penalized with a negative value.

[0310] n t =c n ,

[0311] Among them, c n A negative constant.

[0312] Then, construct the reward function r of the entire task t :

[0313]

[0314] Control barrier function training module: analyzes the constraints of the control barrier function, adjusts the parameters, and trains to obtain a parameterized control barrier function.

[0315] In order to obtain the control barrier function, we first define the expert trajectory P exp :

[0316]

[0317] Among them, ∶= means that it is defined as; x i is the state of the agent, x i ∈S;u i is the action information of the agent.

[0318] pass

[0319]

[0320] in, is an n-dimensional real number space; x represents an n-dimensional state; ||·|| p represents the p-norm.

[0321] Get defined in x i The p-normal ball B with a surrounding radius ∈ ∈,p (x i ).

[0322] pass

[0323]

[0324] D: =D′\bd(D′)

[0325] Get area D.

[0326] Among them, bd(D′) represents the boundary of D′; D and D′ are both regions composed of points; ∪ represents union; D′ is the union of N paradigm balls; \ represents removal; D is an open set that does not contain the boundary of D′.

[0327] Define the Minkowski sum as

[0328]

[0329] in, represents the Minkowski sum; C1 and C2 are two sets.

[0330] pass

[0331]

[0332] Define a set L, where L represents a layer with a width of σ around D; B σ,p (0) represents the p-normal ball with radius σ around 0; bd(D) represents the boundary of D.

[0333] Sample data set X from L L

[0334]

[0335] Define the security state set X safe

[0336] X safe ={x i :(x i ,u i )∈P exp}

[0337] Among them,: represents the trajectory from the expert P exp The state-action pair (x i ,u i ) to extract the status information x i .

[0338] Define safe set X safe Subset of for

[0339]

[0340] Among them, inf means taking the lower limit; the hyperparameter γ safe , γ unsafe is a positive constant, L h represents the local Lipschitz bound.

[0341] Local Lipschitz bound L h (x) is defined as

[0342]

[0343] Among them, sup represents the upper limit, B ∈,p (x) represents the p-normal ball region with radius ∈ around x; h(x) represents the control barrier function, h(x1) represents the control barrier function value of x1; h(x2) represents the control barrier function value of x2.

[0344] An effective control barrier function h(x) on D is defined as: there exists a local Lipschitz continuous extension K-type function α such that the following holds for all x∈D.

[0345]

[0346]

[0347] Where u(x) is the action taken by the agent when the state is x; f(x) and g(x) represent the nonlinear system The dynamic model of Represents partial derivative.

[0348] In order to ensure the effectiveness of the obtained control barrier function, ∈ has constraints.

[0349] Define the function q(x):

[0350] q(x): = L f h(x)+L g h(x)u i +α(h(x))

[0351] Solve the following optimization problem to obtain the required control barrier function h:

[0352] min||h||

[0353] sth(x i )≥γ safe ,

[0354] h(x i )≤-γ unsafe

[0355]

[0356] q(x i ,u i )∶=L f h(x i )+L g h(x i ) i +α(h(x i ))≥γ exp

[0357] Lip(q(x i ,u i ),∈)≤L q ,

[0358] Among them, the hyperparameter γ exp , L h and L q is a positive constant; h(x i ) means that in x i When controlling the barrier function value, is a subset of the safe set, Lip(·,∈) represents the upper bound of the Lipschitz constant of the parameter in the neighborhood of ∈, Indicates that the parameter is The upper bound of the Lipschitz constant in the neighborhood; st represents the constraint condition.

[0359] For all x i ∈X L , Should meet For all x i ∈X safe ,∈ should satisfy∈≤γ exp / L q (x i ).

[0360] Among them, L h (x i ) and L q (x i ) represents the local Lipschitz bound.

[0361] at this time,

[0362]

[0363]

[0364] Among them, q(x i ) means when x=x i The function value of q(x) when .

[0365] Policy network training module: Use the state space, action space and reward function designed in the modeling module, and adjust the hyperparameters in the reinforcement learning algorithm to train the policy network.

[0366] Considering the continuous state and action space, x is the state and a is the action taken in state x; For the action state value, the Actor-Critic framework is adopted to define two critic networks, namely and Defining the actor network The parameters are θ1, θ2 and The target network parameters θ1′, θ2′ and The critic network is updated as follows:

[0367]

[0368]

[0369] Among them, y is the update target; Representative Experience Pool On expectations; Stores the agent's N-step experience; r is the reward after taking action a in the current state; γ is the discount factor for the cumulative reward; x′ represents the state at the next moment; a′ represents the action value at the next moment; θ i represents the critic network parameters; θ i ′ represents the target network parameters; represents the critic network; represents the target network; a N represents the action of step N, N Represents the reward obtained in step N.

[0370] Actor updates through policy gradients:

[0371]

[0372] in, is the objective function; is the objective function The gradient of is the action chosen by the actor network in state x; and is the gradient of the network.

[0373] Control barrier function update module: Modify the output of the policy network according to the learned control barrier function, collect data and update the control barrier function.

[0374] Combined with the control barrier function obtained by the control barrier function training module, the controller output action obtained in the policy network training module is modified to obtain a safe action and collect new data X generated by the interaction with the environment. RL , select data away from the unsafe set and update the control barrier function.

[0375] Safety ensuring module: Through predictive shielding control, noise is applied to dangerous actions to make the agent deviate from the predetermined trajectory, making the training process safe.

[0376] Predictive shielding control, in dangerous situations, shields the output of the learned control barrier function and predicts the subsequent H-step state through the known system dynamics model; if no safety constraints are violated, the action selected by the control barrier function is executed; otherwise, random Gaussian noise will be added to the action until no collision occurs within a finite number of steps H.

[0377] The present invention provides a method for enabling an intelligent agent to autonomously learn navigation and obstacle avoidance strategies, and completes a transportation task while ensuring the underlying network connectivity of the intelligent agent group. The task content is as follows: Figure 1 As shown, the present invention develops a control framework based on TD3. By modeling the navigation problem, the state space, action space and reward function in the algorithm are designed on this basis. Through the dynamically updateable control barrier function, the action is modified when necessary to achieve the purpose of maintaining safety during learning, and the agent can finally complete the set navigation task. The required effective control barrier function is obtained by neural network training, so we add an additional link, namely predictive shielding control to ensure the safety of the agent during exploration.

[0378] Considering online learning of control barrier functions, unlike training control barrier functions limited to expert demonstrations collected offline, learning and updating from online generated data during reinforcement learning training improves adaptability to complex environments. Using predictive shielding control, if the system takes controller output actions that will lead the agent to an unsafe area within the predicted number of steps, noise is added to the dangerous action to deviate from the output value until the agent is always in the safe area. At this time, the agent moves forward one step according to the modified action sequence, and this cycle is repeated to ensure the safety of the system.

[0379] The robot navigation and obstacle avoidance method based on reinforcement learning of the present invention is used for autonomous and safe learning of navigation task control strategy. The robot navigation and obstacle avoidance task implementation method based on reinforcement learning provided by the present invention models the navigation problem, and designs the state space, action space and reward function in TD3 on this basis. By giving constraints, the control barrier function is continuously optimized to achieve better safety assurance performance; combined with the additional predictive shielding control module, the present invention realizes the completion of the transportation obstacle avoidance task while maintaining the network connectivity of the intelligent group.

[0380] The present invention discloses a method for implementing a robot navigation and transportation task based on reinforcement learning, which integrates a model-based control barrier function with provable effectiveness to ensure safety during the reinforcement learning process. A data-driven approach is adopted to learn the control barrier function from the collected demonstrations of safe and ideal behaviors. The ability of reinforcement learning algorithms in learning high-performance controllers is utilized, and the control barrier function is used as a safety guarantee by maximizing the reward function without requiring complete knowledge about the dynamics of the system, and the learning process is guided by constraining the set of explorable strategies. In addition, the present invention also discloses a forward-looking and proactive method, namely predictive shielding control, to cope with the inaccuracy of neural networks and further ensure the safety of the system.

[0381] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0382] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for implementing a robot navigation obstacle avoidance task based on reinforcement learning, characterized in that: The steps include: Modeling steps: Model the navigation task as a Markov process, design the state space, action space and reward function for the reinforcement learning algorithm, and determine the indicators for completing the task; Control barrier function training steps: analyze the constraints of the control barrier function, adjust the parameters, and train to obtain a parameterized control barrier function; Policy network training step: Use the state space, action space, and reward function designed in the modeling step, and adjust the hyperparameters in the reinforcement learning algorithm to train the policy network; Control barrier function update step: According to the learned control barrier function, modify the output of the policy network, collect data and update the control barrier function; Ensure safety steps: Through predictive shielding control, noise is applied to dangerous actions to make the agent deviate from the predetermined trajectory, making the training process safe; In the control barrier function training step, In order to obtain the control barrier function, we first define the expert trajectory P exp : Among them, ∶= means that it is defined as; x i is the state of the agent, x i ∈S;μ i is the action information of the intelligent agent; pass in, is an n-dimensional real number space; x represents an n-dimensional state; ||·|| p represents the p-norm; Get defined in x i The p-normal ball B with a surrounding radius ∈ ∈,p (x i ); pass D∶=D′\bd(D′) Get area D; Where bd(D′) represents the boundary of D′; D and D′ are both regions consisting of points; ∪ represents union; D′ is the union of N paradigm balls; \ represents removal; D is an open set that does not contain the boundary of D′; Define the Minkowski sum as Among them, ⊕ represents the Minkowski sum; C1 and C2 are two sets; pass Define a set L, where L represents a layer with a width of σ around D; B σ,p (0) represents the p-normal ball with radius σ around 0; bd(D) represents the boundary of D; Sample data set X from L L Define the security state set X safe X safe ={x i :(x i ,u i )∈P exp }, Among them,: represents the trajectory from the expert P exp The state-action pair (x i ,u i ) to extract the status information x i ; Define safe set X safe Subset of for Among them, inf means taking the lower limit; the hyperparameter γ safe , γ unsafe is a positive constant, L h represents the local Lipschitz bound; Local Lipschitz bound L h (x) is defined as Among them, sup represents the upper limit, B ∈,p (x) represents the p-normal ball region with radius ∈ around x; h(x) represents the control barrier function, h(x1) represents the control barrier function value of x1; h(x2) represents the control barrier function value of x2; An effective control barrier function h(x) on D is defined as: there exists a local Lipschitz continuous extension K-type function α such that the following holds for all x∈D; Where u(x) is the action taken by the agent when the state is x; f(x) and g(x) represent the nonlinear system The dynamic model of is the symbol of partial derivative; In order to ensure the effectiveness of the obtained control barrier function, ∈ has constraints; Define the function q(x): q(x):=L f h(x)+L g h(x)u i +α(h(x)) Solve the following optimization problem to obtain the required control barrier function h: min||h|| h(x i )≤-γ unsafe q(x i ,u i )∶=L f h(x i )+L g h(x i )u i +α(h(x i ))≥γ exp Among them, the hyperparameter γ exp , L h and L q is a positive constant; h(x i ) means that in x i When controlling the barrier function value, is a subset of the safe set, Lip(·,∈) represents the upper bound of the Lipschitz constant of the parameter in the neighborhood of ∈, Indicates that the parameter is The upper bound of the Lipschitz constant in the neighborhood; st represents the constraint condition; For all x i ∈X L , Should meet For all x i ∈X safe ,∈ should satisfy∈≤γ exp / L q (x i ); Among them, L h (x i ) and L q (x i ) represents the local Lipschitz bound; at this time, Among them, q(x i ) means when x=x i When , the function value of q(x); In the step of ensuring safety, predictive shielding control is used to shield the output of the learned control barrier function in dangerous situations, and the subsequent H-step states are predicted through the known system dynamics model; if no safety constraints are violated, the action selected by the control barrier function is executed; otherwise, random Gaussian noise will be added to the action until no collision occurs within a finite number of steps H.

2. The method for realizing robot navigation obstacle avoidance task based on reinforcement learning according to claim 1, characterized in that: In the modeling step, the navigation model of the agent is measured by the geometric relationship between the target point and the agent's position, and is defined as Among them, d i (t) represents the two-dimensional space distance between agent i and the corresponding target point at time t; q i (t) is the current position of the agent; p i is the target position corresponding to the agent; Represents a definition; The agent is represented as a circular area with a radius of R, and the agents exchange information within a given maximum communication distance R, that is, R ij =||q i -q j ||≤R; Among them, R ij represents the distance between agent i and agent j; q i represents the location information of agent i; q j Represents the location information of agent j; The transport navigation task is modeled by a Markov process as follows: State space S: agent state Contains the two-dimensional coordinates of agent i in space at time t And the direction angle The state space is set to the state set of all N agents at time T, that is, Where i = 1, 2, ..., N, t = 1, 2, ..., T; in, represents the horizontal coordinate of the ith agent at time t; represents the ordinate of the i-th agent at time t; Action Space Assume that each agent selects a direction at a time and moves a specified distance in the selected direction; thus the action of agent i at time t is The speed from the upward direction and angular velocity Composition, that is and then, Reward function: The reward function includes navigation task reward and virtual boundary penalty; Navigation mission reward m t :When the agent approaches the target position, it gets the navigation task reward: Among them, c p and c t is a positive constant; r goal is the reward value; d i is the distance to the destination; δ is the allowable arrival error; t step is the number of steps to complete the task; B represents the given exploration boundary value; Indicates all; Indicates existence; Virtual boundary penalty n t : Introduce virtual boundaries in the training environment and limit the exploration range of the agent by setting penalties; When the distance between the agent and the target position is greater than the given value B, it will be penalized with a negative value. Among them, c n is a negative constant; Then, construct the reward function r of the entire task t :

3. The method for realizing robot navigation obstacle avoidance task based on reinforcement learning according to claim 1, characterized in that: In the policy network training step, Considering the continuous state and action space, x is the state and a is the action taken in state x; For the action state value, the Actor-Critic framework is adopted to define two critic networks, namely and Defining the actor network The parameters are θ1, θ2 and The target network parameters θ1′, θ2′ and The critic network is updated as follows: Among them, y is the update target; Representative Experience Pool On expectations; Stores the agent's N-step experience; r is the reward after taking action a in the current state; γ is the discount factor for the cumulative reward; x′ represents the state at the next moment; a′ represents the action value at the next moment; θ i represents the critic network parameters; θ i ′ represents the target network parameters; represents the critic network; represents the target network; a N represents the action of step N, N Indicates the reward obtained in step N; Actor updates through policy gradients: in, is the objective function; is the objective function The gradient of is the action chosen by the actor network in state x; and is the gradient of the network.

4. The method for realizing robot navigation obstacle avoidance task based on reinforcement learning according to claim 1, characterized in that: In the control barrier function updating step, the controller output action obtained in the policy network training step is modified by combining the control barrier function obtained in the control barrier function training step to obtain a safe action and collect new data X generated by the interaction with the environment. RL , select data away from the unsafe set and update the control barrier function.

5. A robot navigation and obstacle avoidance task implementation system based on reinforcement learning, characterized in that: Includes the following modules: Modeling module: Model the navigation task as a Markov process, design the state space, action space and reward function for the reinforcement learning algorithm, and determine the indicators for completing the task; Control barrier function training module: analyzes the constraints of the control barrier function, adjusts the parameters, and trains to obtain a parameterized control barrier function; Policy network training module: Use the state space, action space and reward function designed in the modeling module, and adjust the hyperparameters in the reinforcement learning algorithm to train the policy network; Control barrier function update module: Modify the output of the policy network according to the learned control barrier function, collect data and update the control barrier function; Safety ensuring module: through predictive shielding control, noise is applied to dangerous actions to make the agent deviate from the predetermined trajectory, making the training process safe; In the control barrier function training module, In order to obtain the control barrier function, we first define the expert trajectory P exp : Among them, ∶= means that it is defined as; x i is the state of the agent, x i ∈S;u i is the action information of the intelligent agent; pass in, is an n-dimensional real number space; x represents an n-dimensional state; ||·|| p represents the p-norm; Get defined in x i The p-normal ball B with a surrounding radius ∈ ∈,p (x i ); pass D∶=D′\bd(D′) Get area D; Where bd(D′) represents the boundary of D′; D and D′ are both regions consisting of points; ∪ represents union; D′ is the union of N paradigm balls; \ represents removal; D is an open set that does not contain the boundary of D′; Define the Minkowski sum as Among them, ⊕ represents the Minkowski sum; C1 and C2 are two sets; pass Define a set L, where L represents a layer with a width of σ around D; B σ,p (0) represents the p-normal ball with radius σ around 0; bd(D) represents the boundary of D; Sample data set X from L L Define the security state set X safe X safe ={x i :(x i ,u i )∈P exp }, Among them,: represents the trajectory from the expert P exp The state-action pair (x i ,u i ) to extract the status information x i ; Define safe set X safe Subset of for Among them, inf means taking the lower limit; the hyperparameter γ safe , γ unsafe is a positive constant, L b represents the local Lipschitz bound; Local Lipschitz bound L h (x) is defined as Among them, sup represents the upper limit, B ∈,p (x) represents the p-normal ball region with radius ∈ around x; h(x) represents the control barrier function, h(x1) represents the control barrier function value of x1; h(x2) represents the control barrier function value of x2; An effective control barrier function h(x) on D is defined as: there exists a local Lipschitz continuous extension K-type function α such that the following holds for all x∈D; Where u(x) is the action taken by the agent when the state is x; f(x) and g(x) represent the nonlinear system The dynamic model of is the symbol of partial derivative; In order to ensure the effectiveness of the obtained control barrier function, ∈ has constraints; Define the function q(x): q(x):=L f h(x)+L g h(x)u i +α(h(x)) Solve the following optimization problem to obtain the required control barrier function h: min||h|| h(x i )≤-γ unsafe q(x i ,u i )∶=L f h(x i )+L g h(x i )u i +α(h(x i ))≥γ exp Among them, the hyperparameter γ exp , L h and L q is a positive constant; h(x i ) means that in x i When controlling the barrier function value, is a subset of the safe set, Lip(·,∈) represents the upper bound of the Lipschitz constant of the parameter in the neighborhood of ∈, Indicates that the parameter is The upper bound of the Lipschitz constant in the neighborhood; st represents the constraint condition; For all x i ∈X L , Should meet For all x i ∈X safe ,∈ should satisfy∈≤γ exp / L q (x i ); Among them, L h (x i ) and L q (x i ) represents the local Lipschitz bound; at this time, Among them, q(x i ) means when x=x i When , the function value of q(x); In the safety ensuring module, predictive shielding control is used to shield the output of the learned control barrier function in dangerous situations, and predict the subsequent H-step states through the known system dynamics model; if no safety constraints are violated, the action selected by the control barrier function is executed; otherwise, random Gaussian noise will be added to the action until no collision occurs within a finite number of steps H.

6. The robot navigation obstacle avoidance task realization system based on reinforcement learning according to claim 5 is characterized in that: In the modeling module, the navigation model of the agent is measured by the geometric relationship between the target point and the agent's position, and the definition is Among them, d i (t) represents the two-dimensional space distance between agent i and the corresponding target point at time t; q i (t) is the current position of the agent; p i is the target position corresponding to the agent; Represents a definition; The agent is represented as a circular area with a radius of R, and the agents exchange information within a given maximum communication distance R, that is, R ij =||q i -q j ||≤R; Among them, R ij represents the distance between agent i and agent j; q i represents the location information of agent i; q j Represents the location information of agent j; The transport navigation task is modeled by a Markov process as follows: State space S: agent state Contains the two-dimensional coordinates of agent i in space at time t And the direction angle The state space is set to the state set of all N agents at time T, that is, Where i = 1, 2, ..., N, t = 1, 2, ..., T; in, represents the horizontal coordinate of the ith agent at time t; represents the ordinate of the i-th agent at time t; Action Space Assume that each agent selects a direction at a time and moves a specified distance in the selected direction; thus the action of agent i at time t is The speed from the upward direction and angular velocity Composition, that is and then, Reward function: The reward function includes navigation task reward and virtual boundary penalty; Navigation mission reward m t :When the agent approaches the target position, it gets the navigation task reward: Among them, c p and c t is a positive constant; r goal is the reward value; d i is the distance to the destination; δ is the allowable arrival error; t step is the number of steps to complete the task; B represents the given exploration boundary value; Indicates all; Indicates existence; Virtual boundary penalty n t : Introduce virtual boundaries in the training environment and limit the exploration range of the agent by setting penalties; When the distance between the agent and the target position is greater than the given value B, it will be penalized with a negative value. Among them, c n is a negative constant; Then, construct the reward function r of the entire task t :

7. The robot navigation obstacle avoidance task realization system based on reinforcement learning according to claim 5 is characterized in that: In the policy network training module, Considering the continuous state and action space, x is the state and a is the action taken in state x; For the action state value, the Actor-Critic framework is adopted to define two critic networks, namely and Defining the actor network The parameters are θ1, θ2 and The target network parameters θ1′, θ2′ and The critic network is updated as follows: Among them, y is the update target; Representative Experience Pool On expectations; Stores the agent's N-step experience; r is the reward after taking action a in the current state; γ is the discount factor for the cumulative reward; x′ represents the state at the next moment; a′ represents the action value at the next moment; θ i represents the critic network parameters; θ i ′ represents the target network parameters; represents the critic network; represents the target network; a N represents the action of step N, N Indicates the reward obtained in step N; Actor updates through policy gradients: in, is the objective function; is the objective function The gradient of is the action chosen by the actor network in state x; and is the gradient of the network.

Images