Reinforcement learning method for unmanned vehicle path planning based on simplified environment and dynamics

Abstract

The present invention proposes a reinforcement learning method for unmanned boat path planning based on simplified environment and dynamics, the steps are as follows: build a real environment, set the unmanned boat state s as the unmanned boat position, the reward function r is the negative value of the length of the unmanned boat planning path, the unmanned boat has a high-order dynamic model with uncertain parameters, and the control signal is u; build a simple environment, the unmanned boat state s s , reward function r s The setting of environmental obstacles is the same as that of the real environment. The unmanned boat has a second-order integral series dynamic model, and the control signal is u s ; Design the first strategy network to interact with the simple environment, with the input and output being the state of the unmanned boat in the simple environment s s and control signal u s ; Design the second strategy network to interact with the first strategy network and the real environment, with the input and output being the control signal u of the simple environment respectively s and the control signal u of the real environment; and complete the training of the two strategy networks. The present invention simplifies the path planning of the unmanned boat system in a complex environment.

Description

Technical Field

[0001] The present invention belongs to the field of control and information technology, and in particular relates to a reinforcement learning method for unmanned boat path planning based on simplified environment and dynamics. Background Art

[0002] Over the past decade, reinforcement learning-based path planning methods for intelligent agent systems have rapidly developed in a variety of application scenarios, particularly in the areas of obstacle avoidance and path optimization in complex environments. Examples include path planning and obstacle avoidance, dynamic obstacle avoidance, and multi-objective path planning. As a special type of intelligent agent, unmanned aerial vehicle (UAV) systems have garnered significant attention due to their widespread application in marine environments. In path planning, the goal of an UAV is to reach a target location from an initial position while avoiding collisions with obstacles. However, the obstacles and complex dynamics of the environment make path planning a highly challenging task.

[0003] It is worth noting that most existing methods are trained in the environment set by the problem and are targeted at the corresponding unmanned vehicle model. However, this framework leads to training difficulties, slow convergence, or even no convergence.

[0004] Against this backdrop, we propose a reinforcement learning approach for path planning for unmanned aerial vehicles (UAVs) based on simplified environments and dynamics. This approach first considers a simple dynamics model in a simple simulation environment and uses MLP+PPO to train a single UAV for path planning. Secondly, an additional policy network is designed within the real environment and dynamics model to translate control information from the simplified environment into real-world control information.

[0005] The difficulties in designing a reinforcement learning method for unmanned boat path planning based on simplified environments and dynamics are:

[0006] First: How to design a simple environment so that the core of the basic path planning problem is still preserved;

[0007] Second: How to design a simple dynamics model so that its state evolution can be synchronized with the state evolution of a high-order dynamics model in the real environment through an additional policy network;

[0008] Third: How to train in a simple environment, grasp the main core issues, and then expand to a real environment for additional training. Summary of the Invention

[0009] The purpose of this invention is to propose a reinforcement learning method for unmanned vehicle path planning based on simplified environment and dynamics, which is used to train the unmanned vehicle system to perform path planning in a complex environment, plan the path from the initial position and run to the target point while avoiding collision with obstacles.

[0010] To achieve the above objectives, the technical solution of the present invention is: a reinforcement learning method for unmanned vehicle path planning based on simplified environment and dynamics, which specifically includes the following steps:

[0011] Step 1: Build a real environment E , set the UAV state s including the UAV position, the reward function r is the negative value of the length of the UAV planning path, the UAV has a high-order dynamic model with uncertain parameters, and the input control signal is u;

[0012] Step 2: Build a simple environment E s , where the unmanned boat status s s , reward function r s , the setting of environmental obstacles and the real environment E The same, and the unmanned boat has a second-order integral series dynamic model, the input control signal is u s ;

[0013] Step 3: Design the first policy network based on the multi-layer perceptron; the first policy network and the simple environment E s interaction, and the first policy network input is the state of the unmanned boat in a simple environment s s , the output is the control signal u for the simple environment s ;

[0014] Step 4: Design a second policy network based on the multi-layer perceptron; the second policy network is combined with the first policy network and the real environment E interaction, and the second policy network input is the control signal u of the simple environment s , the output is the control signal u of the real environment;

[0015] Step 5: Train the first policy network based on the PPO algorithm. The training goal is to maximize the simple environment reward function r s Corresponding cumulative rewards;

[0016] Step 6: Train the second policy network based on the PPO algorithm. The training goal is to minimize the error between the states and the error between the rewards in the two environments.

[0017] Preferably, the step 1 specifically includes:

[0018] Said real environment includes constantly changing sea conditions and obstacles;

[0019] The state of the unmanned boat s includes the position of the unmanned boat. When the state of the unmanned boat s adopts the two-dimensional position of the unmanned boat, it is expressed as , when the UAV state s is expressed as the three-dimensional UAV position ,in, x 、 y 、z Indicates the coordinates of the unmanned boat;

[0020] Depend on M The intermediate path points will be the initial position s 0 and end target position s f The planning path between M +1 segment, the path point position in the planned path is recorded as s i , Among them, when i Take 0 time s i Indicates the initial position s 0, when i Pick M +1 hour s i Indicates the end target location s f ,when hour s i Indicates the i Intermediate waypoint locations;

[0021] Set the reward function r to the negative path length, path length The reward function r is obtained by summing the Euclidean distances between all adjacent waypoint positions:

[0022]

[0023] in, For the i +1 and i The Euclidean distance between the positions of the path points is calculated as follows:

[0024]

[0025] in s ij and s i+1,j They are i Hedi i +1 waypoint location j dimensional coordinates, when using the maximum dimension of the two-dimensional unmanned boat position J The maximum dimension is 2 when the three-dimensional unmanned boat position is used. J is 3;

[0026] The high-order dynamic model with uncertain parameters of the unmanned boat is set as:

[0027]

[0028] in, s is the position of the unmanned boat; the · above the parameter represents the first-order derivative; for s Function expression of the derivative; u is the input control signal; θ is the uncertain parameters of the dynamic model, including environmental factors and internal system parameters.

[0029] Preferably, the step 2 specifically includes:

[0030] The simple environment only includes simplified obstacles;

[0031] The second-order integrated series dynamics model is set as:

[0032]

[0033] Among them, s s It is the state of the unmanned boat in a simple environment, including the position and speed of the unmanned boat; for a two-dimensional environment, the state of the unmanned boat ; For three-dimensional environment, the state of the unmanned boat ;m s Indicates the intermediate state, ,in Representing the unmanned boat n The state vector of the intermediate state; u s Indicates the control signal of simple environment input, ,in The control signal representing the input of the unmanned boat is k Control quantity in each direction; represents the scale factor in the kinetic model;

[0034] The reward function r s for:

[0035]

[0036] in, Indicates the path length in a simple environment; Indicates that the path from the starting point to the end point in a simple environment is divided into M +1 paragraph obtained i The status of the unmanned boat at each waypoint.

[0037] Preferably, the first strategy network is used to implement path planning in a simple environment and a simple model; the multi-layer perceptron includes an input layer, L There are three hidden layers and an output layer. Between the input layer and the output layer, the network processes data through a fully connected layer.

[0038] The input layer receives the state of the unmanned boat in a simple environments s ; The output of each hidden layer Expressed as:

[0039]

[0040] in, l When 1 is taken, For the input layer , indicating the status of the unmanned boat s s ; W l and b l They are l The layer's weight matrix and bias terms, is the activation function;

[0041] The calculation of the output layer is:

[0042]

[0043] in, is the output of the last hidden layer L; and are the weight matrix and bias term of the output layer respectively; the output of the output layer This is the control signal for a simple environment u s .

[0044] Preferably, the second strategy network is used to convert the path planning control signal obtained from the simple environment and the simple model into the control signal in the real environment and the real model; the second strategy network includes an input layer, L Layer hidden layer and output layer; the network input layer receives the control signal of the simple environment u s , the output layer outputs the control signal u of the real environment.

[0045] Preferably, the loss function of the first strategy network is Specifically:

[0046]

[0047] in, Indicates the expected calculation of the empirical data at time step t; Is the new and old strategies in a simple environment state Down control signal The probability ratio of Indicates that the parameter is strategy, Indicates that the current policy is in state Take control signal probability; Indicates that the old policy is in the state Take control signal probability; is the advantage function, which indicates the superiority of the current action relative to the average action; It is a policy clipping parameter used to limit the policy update range;

[0048] The control signal At the same time, it serves as the output action of the policy network ,Right now , the advantage function The expression is as follows:

[0049]

[0050] in, Indicates that the status Take action The expected cumulative rewards obtained, Indicates that the status The average policy value of the current action is not considered.

[0051] Preferably, the advantage function The generalized advantage estimate is used for calculation, and the expression is:

[0052]

[0053] in, d From 0 to Tt An integer representing the current time step t After a series of delayed time steps, T Indicates the maximum time steps of the current trajectory; is the time difference error; Indicates immediate reward; is a discount factor used to control the importance of future rewards; is a hyperparameter that balances the variance and bias of the estimate.

[0054] Preferably, the training goal of the second strategy network is to minimize the total loss function , to optimize the parameters of the second policy network , used to achieve effective mapping between the simple environment and the real environment through the optimized second strategy network; the total loss function of the second strategy network Specifically:

[0055]

[0056] in, Represents the error between the states in two environments; Represents the error between the rewards in the two environments; It is the strategy optimization loss based on the PPO algorithm; the details are as follows:

[0057]

[0058] in, s I It is I The state of the unmanned boat in the real environment E of the training samples, It is I training samples in a simple environment The state of the unmanned boat in N Indicates the number of training samples;

[0059]

[0060] in, r I It is I The reward of training samples in the real environment E, It is I training samples in a simple environment Rewards in

[0061]

[0062] in, is the strategy ratio, which represents the probability ratio between the current strategy and the old strategy.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) By simplifying the environment settings and dynamics model, the training efficiency of the path planning strategy is greatly improved. The difficulty of convergence of the first strategy network is reduced, and the number of network layers required is also less than that of the normal path planning strategy network.

[0065] (2) By designing a second policy network, the control signal in the simplified environment can be converted into the control signal in the real environment, and the state and reward errors corresponding to the two environments will converge;

[0066] (3) The first policy network and the second policy network are trained in a decoupled manner, which makes the framework have strong generalization ability. For example, the same first policy network can be used for different high-order dynamics models, and only the second policy network needs to be retrained, which reduces the training time and difficulty of the generalization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0068] The following is combined with Figure 1 , the technical solution of the present invention is described in detail.

[0069] The present invention proposes a reinforcement learning method for unmanned vehicle path planning based on simplified environment and dynamics, which specifically includes the following steps:

[0070] Step 1: Build a real environment E , set the UAV state s including the UAV position, the reward function r is the negative value of the length of the UAV planning path, the UAV has a high-order dynamic model with uncertain parameters, and the input control signal is u;

[0071] Step 2: Build a simple environment E s , where the unmanned boat status s s , reward function r s , the setting of environmental obstacles and the real environment E The same, and the unmanned boat has a second-order integral series dynamic model, the input control signal is u s ;

[0072] Step 3: Design the first policy network based on the multi-layer perceptron; the first policy network and the simple environment E s interaction, and the first policy network input is the state of the unmanned boat in a simple environment s s , the output is the control signal u for the simple environment s ;

[0073] Step 4: Design a second policy network based on the multi-layer perceptron; the second policy network is combined with the first policy network and the real environment E interaction, and the second policy network input is the control signal u of the simple environment s , the output is the control signal u of the real environment;

[0074] Step 5: Train the first policy network based on the PPO algorithm. The training goal is to maximize the simple environment reward function r s The corresponding cumulative reward is used to optimize the parameters of the first policy network. The learning rate of the PPO algorithm can be set to 0.01;

[0075] Step 6: Train the second policy network based on the PPO algorithm. The training goal is to minimize the error between the states and the error between the rewards in the two environments, that is, to minimize s and s s and r and r s The mean square error is calculated to ensure that the control strategies between the real environment E and the simplified environment Es are consistent.

[0076] In this embodiment, step 1 specifically includes:

[0077] The real environment includes constantly changing sea conditions and obstacles, such as various winds, waves, currents, tides, and islands and reefs;

[0078] The state of the unmanned boat s includes the position of the unmanned boat. When the state of the unmanned boat s adopts the two-dimensional position of the unmanned boat, it is expressed as , when the UAV state s is expressed as the three-dimensional UAV position ,in, x 、 y 、 z Indicates the coordinates of the unmanned boat;

[0079] Depend on M The intermediate path points will be the initial position s 0 and end target position s f The planning path between M +1 segment, the path point position in the planned path is recorded as s i , ; Among them, when i Take 0 time s i Indicates the initial position s 0, when i Pick M +1 hour s i Indicates the end target location s f ,when hour s i Indicates the i The intermediate path point position is expressed as:

[0080]

[0081] in, Indicates the i The proportion of the position of the intermediate points in the total path;

[0082] Set the reward function r to the negative path length, path length The reward function r is obtained by summing the Euclidean distances between all adjacent waypoint positions:

[0083]

[0084] in, For the i +1 and i The Euclidean distance between the positions of the path points is calculated as follows:

[0085]

[0086] in s ij and s i+1,j They are i Hedi i +1 waypoint location j dimensional coordinates, when using the maximum dimension of the two-dimensional unmanned boat position J The maximum dimension is 2 when the three-dimensional unmanned boat position is used. J is 3;

[0087] The high-order dynamic model with uncertain parameters of the unmanned boat is set as:

[0088]

[0089] in, s is the position of the unmanned boat; the · above the parameter represents the first-order derivative; for s Function expression of the derivative; u is the input control signal; θ is the uncertain parameters of the dynamic model, including environmental factors and internal system parameters.

[0090] The high-order unmanned vehicle dynamics model with uncertain parameters mentioned above selects different models and parameters according to the unmanned vehicle actually used. For example, the size parameters in the model will be different depending on the size of the unmanned vehicle.

[0091] In this embodiment, step 2 specifically includes:

[0092] The simple environment only includes simplified obstacles, such as islands and reefs;

[0093] The second-order integrated series dynamics model is set as:

[0094]

[0095] Among them, s s It is the state of the unmanned boat in a simple environment, including the position and speed of the unmanned boat; for a two-dimensional environment, the state of the unmanned boat ; For three-dimensional environment, the state of the unmanned boat ;m s Indicates the intermediate state, ,in Representing the unmanned boat n The state vector of the intermediate state; u s Indicates the control signal of simple environment input, ,in The control signal representing the input of the unmanned boat is k Control quantity in each direction; represents the scale factor in the kinetic model;

[0096] The reward function r s for:

[0097]

[0098] in, Indicates the path length in a simple environment; Indicates that the path from the starting point to the end point in a simple environment is divided into M +1 paragraph obtained i The status of the unmanned boat at each waypoint.

[0099] Among them, the dynamic model in step 2 is uniformly set to a second-order integral string type, that is, the choice of the simple model is irrelevant to the actual unmanned boat model and parameters used.

[0100] In this embodiment, the first strategy network is used to implement path planning in a simple environment and a simple model, so only a simple multi-layer perceptron architecture is needed to design the network; the multi-layer perceptron includes an input layer, L Layer (for example, 24 layers) hidden layer and output layer; between the input layer and the output layer, the network processes data through the fully connected layer;

[0101] The input layer receives the state of the unmanned boat in a simple environment s s ; The output of each hidden layer Expressed as:

[0102]

[0103] in, l When 1 is taken, For the input layer , indicating the status of the unmanned boat s s ; W l and b l They are l The layer's weight matrix and bias terms, is the activation function;

[0104] The calculation of the output layer is:

[0105]

[0106] in, is the output of the last hidden layer L; and are the weight matrix and bias term of the output layer respectively; the output of the output layer This is the control signal for a simple environment u s .

[0107] In this embodiment, the second strategy network is used to convert the path planning control signal obtained from the simple environment and the simple model into the control signal in the real environment and the real model; the second strategy network includes an input layer, L Layer (for example, 24 layers) hidden layer and output layer; the network input layer receives the control signal of the simple environment u s , the output layer outputs the control signal u of the real environment.

[0108] While the second policy network can't fully translate the control signals, it can ensure the error between the two control signals is limited after training, resulting in a smaller error in the final path planning. In many practical unmanned vehicle applications, a small error in path planning is acceptable.

[0109] In this embodiment, the cumulative reward Expressed as:

[0110]

[0111] in, Indicates that the parameters are Strategy The complete trajectory generated, Indicates the path, Indicates that the parameter is strategy, T represents the maximum number of time steps for each trajectory, t represents the current time step, The unmanned boat is in a simple environment Take control signal The instant rewards you receive, Indicates the t The simple environment state of time steps, Indicates the t Simple environmental control actions for time steps, is the discount factor.

[0112] The reward maximization in PPO is to indirectly optimize the strategy through the advantage function. Therefore, the loss function of the first strategy network is Specifically:

[0113]

[0114] in, Indicates the expected calculation of the empirical data at time step t; Is the new and old strategies in a simple environment state Down control signal The probability ratio of Indicates that the parameter is strategy, Indicates that the current policy is in state Take control signal probability; Indicates that the old policy is in the state Take control signal probability; is the advantage function, which indicates the superiority of the current action relative to the average action; It is a policy clipping parameter used to limit the policy update range;

[0115] The control signal At the same time, it serves as the output action of the policy network ,Right now , the advantage function The expression is as follows:

[0116]

[0117] in, Indicates that the status Take action The expected cumulative rewards obtained, Indicates that the status The average policy value of the current action is not considered.

[0118] In this embodiment, the advantage function The generalized advantage estimate is used for calculation, and the expression is:

[0119]

[0120] in, d From 0 to Tt An integer representing the current time step t After a series of delayed time steps, T Indicates the maximum time steps of the current trajectory; is the time difference error; Indicates immediate reward; is a discount factor used to control the importance of future rewards; is a hyperparameter that balances the variance and bias of the estimate.

[0121] In this embodiment, the total loss function of the second strategy network is Specifically:

[0122]

[0123] in, Represents the error between the states in two environments; Represents the error between the rewards in the two environments; It is the strategy optimization loss based on the PPO algorithm; the details are as follows:

[0124]

[0125] in, s I It is I The state of the unmanned boat in the real environment E of the training samples, It is I training samples in a simple environment The state of the unmanned boat in N Indicates the number of training samples;

[0126]

[0127] in, r I It is I The reward of training samples in the real environment E, It is I training samples in a simple environment Rewards in

[0128]

[0129] in, is the strategy ratio, which represents the probability ratio between the current strategy and the old strategy.

[0130] The training goal is to minimize , through the parameters of the second optimization strategy network , thereby achieving effective mapping between the simple environment and the real environment;

[0131]

[0132] in, Represents the optimal parameters of the second optimization strategy network obtained by optimization.

[0133]

[0134] At this point, all steps are completed.

[0135] The present invention studies how to design a reinforcement learning method for unmanned boat path planning based on simplified environment and dynamics. This method can be used to train unmanned boat systems to perform path planning in complex environments, and to plan the trajectory from the initial position and run to the target point under the premise of avoiding collisions with obstacles. For the path planning problem of unmanned boats, the mainstream existing algorithms are based on optimization methods or learning methods, which are solved in a real environment. This method is improved on the basis of existing theories, and the training process becomes simple and efficient by simplifying the real environment and dynamic model. The control signal in the simplified environment can be converted into the control signal in the real environment through the linked second strategy network, and the state and reward errors corresponding to the two environments will converge. This feature shows that this method has strong generalization for different dynamic models.

[0136] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions and effects do not exceed the scope of the technical solution of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A reinforcement learning method for unmanned boat path planning based on simplified environment and dynamics, characterized by: The specific steps include: Step 1: Build a real environment E, set the UAV state s including the UAV position, the reward function r is the negative value of the length of the UAV's planned path, the UAV has a high-order dynamic model with uncertain parameters, and the input control signal is u; Step 2: Build a simple environment E s , where the unmanned boat status s s , reward function r s The setting of environmental obstacles is the same as the real environment E, and the unmanned boat has a second-order integral series dynamic model, and the input control signal is u s ; Step 3: Design the first policy network based on the multi-layer perceptron; the first policy network and the simple environment E s interaction, and the first policy network input is the state of the unmanned boat in a simple environment s s , the output is the control signal u for the simple environment s ; The first policy network is used to implement path planning in a simple environment and a simple model; the multilayer perceptron includes an input layer, L hidden layers, and an output layer; between the input layer and the output layer, the network performs data processing through a fully connected layer; The input layer receives the state of the simple environment unmanned boat s s ; The output h of each hidden layer l Expressed as: h l =σ(W l h l-1 +b l ),l=1,2...,L When l is 1, h l-1 is the input layer h0, which represents the state s of the unmanned boat s ;W l and b l are the weight matrix and bias term of the lth layer, and σ is the activation function; The calculation of the output layer is: y output =W out h L +b out Among them, h L is the output of the last hidden layer L; W out and b out are the weight matrix and bias term of the output layer respectively; the output y of the output layer output This is the control signal u for the simple environment s ; Step 4: Design a second policy network based on a multilayer perceptron; the second policy network interacts with the first policy network and the real environment E, and the input of the second policy network is the control signal u of the simple environment s , the output is the control signal u of the real environment; The second strategy network is used to convert the path planning control signal obtained from the simple environment and the simple model into the control signal in the real environment and the real model; the second strategy network includes an input layer, L hidden layers and an output layer; the network input layer receives the control signal u of the simple environment s , the output layer outputs the control signal u of the real environment; Step 5: Train the first policy network based on the PPO algorithm. The training goal is to maximize the simple environment reward function r s Corresponding cumulative rewards; Step 6: Train the second policy network based on the PPO algorithm. The training goal is to minimize the error between the states and the error between the rewards in the two environments.

2. The unmanned vehicle path planning reinforcement learning method based on simplified environment and dynamics according to claim 1 is characterized in that: The step 1 specifically includes: Said real environment includes constantly changing sea conditions and obstacles; The UAV state s includes the UAV position. When the UAV state s adopts the two-dimensional UAV position, it is expressed as s = (x, y). When the UAV state s adopts the three-dimensional UAV position, it is expressed as s = (x, y, z), where x, y, and z represent the coordinates of the UAV. The initial position s0 and the final target position s are connected by M intermediate path points. f The planned path between is divided into M+1 segments, and the path point position in the planned path is recorded as s i , i∈{0, 1, 2, ..., M, M+1}; when i is 0, s i Indicates the initial position s0, when i is M+1, s i Indicates the end target position s f , when i∈{1,2,...,M} i Indicates the position of the i-th intermediate path point; Set the reward function r to the negative path length, path length L path The reward function r is obtained by summing the Euclidean distances between all adjacent waypoint positions: Among them, ||s i -s i+1 || is the Euclidean distance between the i+1th and ith path points, and the specific calculation formula is: where s ij and s i+1,j are the j-th dimension coordinates of the i-th and i+1-th path points, respectively. When the two-dimensional unmanned boat position is used, the maximum dimension J is 2; when the three-dimensional unmanned boat position is used, the maximum dimension J is 3; The high-order dynamic model with uncertain parameters of the unmanned boat is set as: Where s is the position of the USV; · above the parameter represents the first-order derivative; f(s, θ, u) is the functional expression of the derivative of s; u is the input control signal; and θ is the uncertain parameter of the dynamic model, including environmental factors and internal system parameters.

3. The unmanned vehicle path planning reinforcement learning method based on simplified environment and dynamics according to claim 2 is characterized in that: The step 2 specifically includes: The simple environment only includes simplified obstacles; The second-order integrated series dynamics model is set as: Among them, s s It is the state of the unmanned boat in a simple environment, including the position and speed of the unmanned boat; for a two-dimensional environment, the state of the unmanned boat For a three-dimensional environment, the state of the unmanned boat m s Indicates the intermediate state, m s =(m s1 , m s2 ,...,m sn ), where m sn Represents the state vector of the unmanned boat in the nth intermediate state; u s Indicates the control signal of the simple environment input, u s =(u s1 ,u s2 ,...,u sk ), where u sk represents the control signal input by the unmanned boat in the kth direction; g s represents the scale factor in the kinetic model; The reward function r s for: in, Indicates the path length in a simple environment; Indicates the state of the unmanned boat at the i-th path point obtained by dividing the path from the starting point to the end point in a simple environment into M+1 segments.

4. The unmanned vehicle path planning reinforcement learning method based on simplified environment and dynamics according to claim 1 is characterized in that: The loss function L of the first policy network CLIP (θ1) is specifically: in, Indicates the expected calculation of the empirical data at time step t; r t (θ1) is the new and old strategies in the simple environment state Down control signal The probability ratio of represents a policy with parameter θ1, Indicates that the current policy is in state Take control signal probability; Indicates that the old policy is in the state Take control signal The probability of A t is the advantage function, which indicates the pros and cons of the current action relative to the average action; ε is the policy clipping parameter, which is used to limit the policy update amplitude; The control signal At the same time, it serves as the output action of the policy network Right now The advantage function A t The expression is as follows: in, Indicates that the status Take action The expected cumulative rewards obtained, Indicates that the status The average policy value of the current action is not considered.

5. The unmanned vehicle path planning reinforcement learning method based on simplified environment and dynamics according to claim 4 is characterized in that: The advantage function A t The generalized advantage estimate is used for calculation, and the expression is: Where d is an integer from 0 to Tt, representing a series of delayed time steps after the current time step t, and T represents the maximum number of time steps of the current trajectory; δ t is the time difference error; represents the immediate reward; γ∈[0,1] is the discount factor used to control the importance of future rewards; λ∈[0,1] is a hyperparameter that balances the estimation variance and bias.

6. The unmanned vehicle path planning reinforcement learning method based on simplified environment and dynamics according to claim 4 is characterized in that: The training goal of the second policy network is to minimize the total loss function L total (θ2) is used to optimize the parameter θ2 of the second strategy network, so as to achieve an effective mapping between the simple environment and the real environment through the optimized second strategy network; the total loss function L of the second strategy network is total (θ2) is specifically: L total (θ2)=L state (θ2)+L reward (θ2)+L CLIP (θ2) Among them, L state (θ2) represents the error between the states in the two environments; L reward (θ2) represents the error between the rewards in the two environments; L CLIP (θ2) is the policy optimization loss based on the PPO algorithm; specifically: Among them, s I is the state of the unmanned boat of the I-th training sample in the real environment E, S sI is the I-th training sample in the simple environment E s The state of the unmanned boat in , N represents the number of training samples; Among them, r I is the reward of the I-th training sample in the real environment E, r sI is the I-th training sample in the simple environment E s Rewards in Among them, r t (θ2) is the strategy ratio, which represents the probability ratio between the current strategy and the old strategy.

Images