Route planning method based on reinforcement learning
By converting the coordinates of multiple obstacles into repulsion vectors, the problem of high computational complexity in traditional reinforcement learning methods in multi-obstacle environments is solved, achieving efficient trajectory planning and improved safety.
Patent Information
- Application Number
- CN202511804933.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-02-27
AI Technical Summary
Traditional reinforcement learning methods face multiple obstacles in trajectory planning, which leads to increased computational complexity and reduced planning efficiency due to excessive observations.
The coordinates of multiple obstacles are converted into repulsion vectors. The magnitude and direction of the repulsion vectors are used as input observations to construct a state space vector. A reasonable reward function and reinforcement learning algorithm are designed to guide the moving object to avoid obstacles.
It effectively reduces the number of states, lowers computational complexity, improves planning efficiency, reduces collision risk, and enhances motion safety.
Smart Images
Figure CN121577041A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of path planning, in particular to a path planning method based on reinforcement learning. BACKGROUND
[0002] With the rapid development of low-altitude economy, path planning technology is increasingly widely used. In many tasks, moving objects need to autonomously plan paths in complex environments to avoid obstacles and efficiently complete tasks. Traditional path planning methods such as A* algorithm, artificial potential field method, RRT algorithm, etc. have many limitations when facing complex and variable environments and a large number of obstacles. For example, the artificial potential field method is prone to local optimization. Search algorithms such as A* algorithm have high time and space complexity. In recent years, reinforcement learning as a new artificial intelligence technology has been gradually applied to the field of path planning. Reinforcement learning is an important branch of machine learning, unlike supervised learning and unsupervised learning, reinforcement learning places more emphasis on the interaction between the agent and the environment. The agent observes the state quantity in the environment and performs actions, and receives feedback rewards according to the results of the actions. These reward signals will guide the agent to adjust its strategy to maximize the long-term cumulative reward. The core of reinforcement learning is the reward function, which estimates the expected cumulative reward that can be obtained by starting from a given state and following a specific strategy. Reinforcement learning has many unique advantages: 1. It has advantages in solving nonlinear, non-stationary, high-dimensional and complex tasks, and can handle complex problems in areas such as autonomous driving, financial trading, game control, and robot control. 2. The agent can learn through interaction with the environment, without relying on manually labeled data, and has the characteristics of autonomous learning, which can adapt to new environments and new tasks. 3. It can consider long-term rewards to make current decisions and has long-term planning capabilities. 4. It can handle uncertain and dynamic environments and adapt to complex real-world environments. However, reinforcement learning also has some challenges: 1. It usually requires a large amount of training data and time to learn to adapt to complex environments. In some complex tasks, the training time and sample requirements can be very high, which increases the training cost. 2. During training, exploration and trial-and-error can lead to unstable behavior of the agent. At the same time, some algorithms may have convergence problems, which increases the uncertainty of the algorithm. 3. When dealing with high-dimensional state space, reinforcement learning faces high computational complexity and low sample efficiency. These problems make it difficult to apply reinforcement learning in some cases.
[0003] In the path planning, the reinforcement learning method is based on the observation quantity to predict the motion object action. The observation quantity is usually the motion object position, target position, obstacle position and the like, and the action quantity is usually the motion object speed, acceleration and the like. The increase in the number of obstacles will cause the dimension of the observation quantity to increase. When facing multiple obstacles, the traditional reinforcement learning method has too much observation quantity, which causes the calculation complexity to increase and the planning efficiency to decrease. SUMMARY
[0004] The purpose of the present application is to propose a path planning method based on reinforcement learning, which converts the coordinates of multiple obstacles into repulsive force vectors of the motion object, and takes the size and direction of the repulsive force vector as the input observation quantity of the multiple obstacles, so as to solve the problem that the traditional reinforcement learning method has too much observation quantity when facing multiple obstacles in the motion object path planning, and the calculation complexity increases.
[0005] The technical scheme of the present application is: a path planning method based on reinforcement learning, comprising: Step 1: constructing a two-dimensional motion environment and obtaining the state parameter of the motion object; Step 2: constructing a state space vector S; Step 3: constructing the action vector A of the motion object; wherein A has 2 elements, which are the acceleration a of the motion object and the angular velocity omega of the speed direction angle respectively; Step 4: constructing the action function action of the motion object according to the action vector A of the motion object and the state space vector S, wherein the input quantity of the action function is the action vector A, and the output quantity is the state space vector S; Step 5: using a preset reward function f to guide the motion object to move towards the target point and avoid obstacles; Step 6: training the motion object by using a reinforcement learning algorithm to obtain a trained motion object path planning strategy network B; Step 7: evaluating the motion object strategy network B; Step 8: using the motion object strategy network B to plan the motion object path in a time advancing manner.
[0006] Specifically, step 1 comprises: Taking the plane where the motion object, the target point and the obstacles are located as the coordinate plane, taking the initial position of the motion object as the coordinate origin O, taking the right as the positive direction of the Ox axis and taking the above as the positive direction of the Oy axis, a plane map coordinate system Oxy is defined; the position coordinates uav of the motion object, the center position coordinates obs of the obstacles and the position coordinates target of the target point in the coordinate system Oxy are obtained, as well as the speed speed, the speed direction angle yaw, the maximum angular velocity omega_max and the maximum speed speed_max of the motion object.
[0007] Specifically, step 2 comprises: Step 21: According to the moving object position coordinates uav and the target point position coordinates target, the attractive force vector F_a=target-uav of the moving object is calculated; Step 22: The size target_dist and the direction angle target_theta of the attractive force vector F_a are calculated; Step 23: The center position coordinates obs of the plurality of obstacles are converted into the repulsive force vector F_r of the moving object, and the repulsive force size and direction generated by each obstacle to the moving object are calculated according to the distance and relative position between the moving object and each obstacle; the repulsive forces are synthesized to obtain the total repulsive force vector F_r; Step 24: The size obs_dist and the direction angle obs_theta of the repulsive force vector F_r are calculated; Step 25: S includes 8 elements, and each element in S is: the moving object horizontal coordinate uav_x, the moving object vertical coordinate uav_y, the moving object speed speed, the moving object speed direction angle yaw, the attractive force vector size target_dist, the attractive force vector direction angle target_theta, the repulsive force vector size obs_dist, and the repulsive force vector direction angle obs_theta.
[0008] Specifically, step 4 comprises: Step 41: After a time step dt, the size target_dist and the direction angle target_theta of the attractive force vector F_a are updated according to step 2, and the size obs_dist and the direction angle obs_theta of the repulsive force vector F_r are updated; Step 42: According to the updated moving object horizontal coordinate uav_x, the moving object vertical coordinate uav_y, the moving object speed speed, the moving object speed direction angle yaw, the attractive force vector size target_dist, the attractive force vector direction angle target_theta, the repulsive force vector size obs_dist, and the repulsive force vector direction angle obs_theta, the state space vector S is reconstructed.
[0009] Specifically, step 41 comprises: Step 411: Obtain the obstacle center position coordinates and the target point position coordinates target; Step 412: update the moving object speed speed, the moving object speed direction angle yaw and the moving object position coordinate uav; wherein, according to the moving object speed speed, the moving object speed direction angle yaw and the moving object position coordinate uav at the current time, the moving object speed speed', the moving object speed direction angle yaw' and the moving object position coordinate uav' after the time dt are updated; Step 413: according to the updated moving object position coordinate uav', the size target_dist and the direction angle target_theta of the attractive force vector F_a are updated according to step 2, and the size obs_dist and the direction angle obs_theta of the repulsive force vector F_r are updated.
[0010] Specifically, step 5 includes: Step 51: according to target_theta, yaw, speed, speed_max, construct angle reward function f_near: f_near = cos(target_theta - yaw) * speed / speed_max; When the moving object speed direction angle is towards the target point, a positive reward is given to encourage the moving object to move towards the target as soon as possible; Step 52: according to obs_theta, yaw, speed and speed_max, construct obstacle avoidance reward function f_obs: f_obs = cos(obs_theta - yaw) * speed / speed_max, When the moving object approaches the obstacle or collides with the obstacle, a larger negative reward is given to punish the collision behavior of the moving object; Step 53: according to target_dist, construct distance reward function f_dist: f_dist = 1 / target_dist; The shorter the distance from the moving object to the target point, the greater the reward, to encourage the moving object to reach the target as soon as possible; Step 54: according to the current time step of the moving object step, construct step reward function f_step: f_step = -step; The more steps the moving object takes to reach the target point, the greater the negative reward, to minimize the moving object's track; Step 55: Set the target reward f_finish, which gives a larger reward when the moving object reaches the target; if target_dist is less than the set threshold e, it means that the moving object has reached the target point, and f_finish is 20, otherwise, f_finish is 0; Step 56: According to the angle reward function f_near, the obstacle avoidance reward function f_obs, the distance reward function f_dist, the step reward function f_step and the target reward f_finish, the reward function f is constructed: f = k1 * f_near + k2 * f_obs + k3 * f_dist + k4 * f_step + k5 * f_finish; Wherein, k1, k2, k3, k4 and k5 are the weight coefficients of each reward, which are positive numbers.
[0011] Specifically, step 6 includes: Step 61: Construct the policy network B and the value network V based on the multi-layer perception mechanism, and initialize the network parameters; wherein the input of the policy network is the state space vector S, and the output is the action vector A; the input of the value network is the state space vector S and the action vector A, and the output is the value V_a; Step 62: The moving object executes the action function action according to the action vector A output by the current policy network B in the manner described in step 4, and collects the state vector S, action vector A and reward function f; Step 63: Based on the value network V and the collected state vector S, action vector A and reward function f, calculate V_a at each time step to evaluate the relative goodness of the action; Step 64: According to V_a, update the parameters of the policy network B by gradient ascent method to increase the probability of generating high reward action vector A; update the parameters of the value network V so that it can more accurately predict the value of state space vector S and action vector A; Step 65: Repeat steps 61-65 until the policy network B converges, i.e. the moving object can stably and efficiently reach the target point from the starting point; get the trained moving object policy network B.
[0012] Specifically, step 7 includes: Step 71: According to the two-dimensional motion environment, use the moving object policy network B to perform an iterative process on the moving object horizontal coordinate uav_x and the moving object vertical coordinate uav_y; Specifically, step 71 includes: inputting the state space vector S into the policy network B and outputting the action vector A; updating the state space vector S according to step 4; if the distance from the moving object to the target target_dist is less than the set threshold e or the maximum number of iterations is reached, then the iteration is terminated. Step 72: If the distance from the moving object to the target, target_dist, is less than the set threshold e after the iteration is completed, the trained model is considered to be usable; otherwise, the policy network B is retrained according to the method described in step 6.
[0013] Specifically, step 8 includes: Input the current state space vector S into the policy network B of the moving object, and output the action vector A. Update the state space vector S for the next moment according to step 4. The updated position coordinates uav' of the moving object are the planned trajectory points for the next moment. If the distance from the moving object to the target target_dist is less than the set threshold e, the trajectory planning is completed.
[0014] In summary, this invention proposes a motion object trajectory planning method based on reinforcement learning. By transforming the coordinates of multiple obstacles into repulsion vectors, the complex obstacle information is simplified to the magnitude and direction of the repulsion force, effectively reducing the number of state variables, lowering the complexity of the reinforcement learning algorithm, and improving planning efficiency. Through reasonable design of the reward function and the learning process of the reinforcement learning algorithm, the method can guide the moving object to avoid obstacles, reduce the risk of collision, and improve the safety of motion. Attached Figure Description
[0015] Figure 1 A trajectory planning diagram of a trajectory planning method based on reinforcement learning. Detailed Implementation
[0016] Example 1 like Figure 1 As shown, this invention proposes a trajectory planning method based on reinforcement learning, the specific steps of which include: Step 1: Construct a two-dimensional motion environment and obtain the state parameters of the moving object; Specifically, step 1 includes: defining a planar map coordinate system Oxy with the plane containing the moving object, the target point, and the obstacle as the coordinate plane, the initial position of the moving object as the origin O, the right as the positive direction of the Ox axis, and the top as the positive direction of the Oy axis; obtaining the moving object's position coordinates uav, the obstacle's center position coordinates obs, and the target point's position coordinates target, as well as the moving object's speed, velocity direction angle yaw, maximum angular velocity omega_max, and maximum speed_max in the Oxy coordinate system; Step 2: Construct the state space vector S; Preferably, step 2 comprises: Step 21: Calculate the attractive force vector F_a of the moving object according to the position coordinates uav of the moving object and the position coordinates target of the target point: F_a=target-uav Step 22: Calculate the size target_dist and the direction angle target_theta of the attractive force vector F_a.
[0017] Step 23: Convert the center position coordinates obs of the plurality of obstacles into the repulsive force vector F_r of the moving object. According to the distance and relative position between the moving object and each obstacle, the size and direction of the repulsive force generated by each obstacle on the moving object are calculated. Then, the repulsive forces are synthesized to obtain the total repulsive force vector F_r, and the calculation formula of F_r is:
[0018] Wherein, k is the repulsive force coefficient, n is the number of obstacles, obs i is the i-th obstacle coordinate.
[0019] Step 24: Calculate the size obs_dist and the direction angle obs_theta of the repulsive force vector F_r.
[0020] Step 25: S includes 8 elements, and each element in S is: the horizontal coordinate uav_x of the moving object, the vertical coordinate uav_y of the moving object, the speed speed of the moving object, the direction angle yaw of the moving object, the size target_dist of the attractive force vector, the direction angle target_theta of the attractive force vector, the size obs_dist of the repulsive force vector, and the direction angle obs_theta of the repulsive force vector.
[0021] Step 3: Construct the action vector A of the moving object; Wherein, A has 2 elements, which are the acceleration a of the moving object and the angular velocity omega of the speed direction angle.
[0022] It should be noted that the acceleration a is used to change the speed speed of the moving object, and the angular velocity omega is used to change the speed direction angle yaw of the moving object. By adjusting the speed and direction angle, the moving object can move in the environment and avoid obstacles.
[0023] Step 4: According to the action vector A and the state space vector S of the moving object, construct the action function action of the moving object, and the input quantity of the action function is the action vector A, and the output quantity is the state space vector S.
[0024] The specific implementation of the action function action is: Step 41: After time step dt, update the size target_dist and the direction angle target_theta of the attractive force vector F_a, and update the size obs_dist and the direction angle obs_theta of the repulsive force vector F_r according to step 2; Preferably, step 41 comprises: Step 411: Obtain the obstacle center position coordinates and the target point position coordinates target; Step 412: Update the moving object speed speed, the moving object speed direction angle yaw, and the moving object position coordinates uav; Wherein, according to the current time moving object speed speed, the moving object speed direction angle yaw, and the moving object position coordinates uav, update the moving object speed speed', the moving object speed direction angle yaw', and the moving object position coordinates uav' after time step dt. The update method is: speed' = speed +dt*a(2) yaw' = yaw +dt*omega(3) uav' = uav + speed' * dt * [cos(yaw'), sin(yaw')](4) Step 413: According to the updated moving object position coordinates uav', update the size target_dist and the direction angle target_theta of the attractive force vector F_a, and update the size obs_dist and the direction angle obs_theta of the repulsive force vector F_r according to step 2.
[0025] Step 42: According to the updated moving object horizontal coordinates uav_x, the moving object vertical coordinates uav_y, the moving object speed speed, the moving object speed direction angle yaw, the attractive force vector size target_dist, the attractive force vector direction angle target_theta, the repulsive force vector size obs_dist, and the repulsive force vector direction angle obs_theta, reconstruct the state space vector S.
[0026] Step 5: Use the preset reward function f to guide the moving object to move towards the target point and avoid obstacles.
[0027] Specifically, step 5 comprises: Step 51: According to target_theta, yaw, speed, and speed_max, construct an angle reward function f_near: f_near = cos(target_theta - yaw) * speed / speed_max; A positive reward is given when the speed direction angle of the moving object is towards the target point, to encourage the moving object to move towards the target as soon as possible.
[0028] Step 52: According to obs_theta, yaw, speed and speed_max, construct the obstacle avoidance reward function f_obs: f_obs = cos(obs_theta - yaw) * speed / speed_max A larger negative reward is given when the moving object approaches the obstacle or collides with the obstacle, to punish the collision behavior of the moving object.
[0029] Step 53: According to target_dist, construct the distance reward function f_dist: f_dist = 1 / target_dist The shorter the distance from the moving object to the target point, the greater the reward, to encourage the moving object to reach the target as soon as possible.
[0030] Step 54: According to the current time step of the moving object step, construct the step reward function f_step: f_step = -step The more steps the moving object takes to reach the target point, the greater the negative reward, to minimize the trajectory of the moving object.
[0031] Step 55: Set the target arrival reward f_finish, which gives a larger reward when the moving object reaches the target. If target_dist is less than the set threshold e, it means that the moving object has reached the target point, and f_finish is 20, otherwise, f_finish is 0.
[0032] Step 56: According to the angle reward function f_near, the obstacle avoidance reward function f_obs, the distance reward function f_dist, the step reward function f_step and the target arrival reward f_finish, construct the reward function f: f = k1 * f_near + k2 * f_obs + k3 * f_dist + k4 * f_step + k5 * f_finish Where k1, k2, k3, k4 and k5 are the weighting coefficients of each reward, all of which are positive numbers.
[0033] In practical applications, the value range of k1, k2, k3, k4 and k5 is [0~10].
[0034] Step 6: Use reinforcement learning algorithm to train the moving object, and get the trained moving object trajectory planning strategy network B.
[0035] Specifically, step 6 includes: Step 61: build a policy network B and a value network V based on multi-layer perception, and initialize network parameters. The input of the policy network is the state space vector S, and the output is the action vector A. The input of the value network is the state space vector S and the action vector A, and the output is the value V_a.
[0036] Step 62: the moving object executes the action function action according to the action vector A output by the current policy network B in the manner described in step 4, and collects the state vector S, the action vector A, and the reward function f.
[0037] Step 63: based on the value network V and the collected state vector S, action vector A, and reward function f, calculate V_a at each time step to evaluate the relative goodness of the action.
[0038] Step 64: update the parameters of the policy network B by the gradient ascent method according to V_a, so that the probability of generating a high-reward action vector A increases. Update the parameters of the value network V so that it can more accurately predict the value of the state space vector S and the action vector A.
[0039] Step 65: loop steps 61-65 until the policy network B converges, i.e., the moving object can stably and efficiently reach the target point from the starting point. Get the trained moving object policy network B.
[0040] Step 7: evaluate the moving object policy network B.
[0041] Specifically, step 7 includes: Step 71: according to the two-dimensional motion environment, use the moving object policy network B to perform an iterative process on the moving object horizontal coordinate uav_x and the moving object vertical coordinate uav_y. Specifically, step 71 includes: inputting the state space vector S into the policy network B to output the action vector A, and updating the state space vector S according to step 4. If the distance from the moving object to the target target_dist is less than the set threshold e or the maximum number of iterations is reached, terminate the iteration.
[0042] Step 72: if the distance from the moving object to the target target_dist is less than the set threshold e after the iteration is completed, it is considered that the trained model is usable, otherwise retrain the policy network B according to the method described in step 6.
[0043] Step 8: use the moving object policy network B to perform moving object trajectory planning in a time advancing manner.
[0044] Specifically: The current state space vector S is input into the policy network B of the moving object, which outputs the action vector A. The state space vector S is then updated according to step 4 for the next time step. The updated position coordinates uav' of the moving object are the planned trajectory points for the next time step. If the distance from the moving object to the target, target_dist, is less than the set threshold e, trajectory planning is complete.
[0045] Example 2 like Figure 1 As shown, this application provides a path planning method based on reinforcement learning, including the following steps: Step 1: Construct a two-dimensional motion environment and obtain the state parameters of the moving object; Specifically, step 1 includes: defining a planar map coordinate system Oxy with the plane containing the moving object, the target point, and the obstacle as the coordinate plane, the initial position of the moving object as the origin O, the right as the positive direction of the Ox axis, and the top as the positive direction of the Oy axis; obtaining the moving object's position coordinates uav, the obstacle's center position coordinates obs, and the target point's position coordinates target in the Oxy coordinate system, as well as the moving object's speed, speed direction angle yaw, maximum angular velocity omega_max of 1 rad / s, and maximum speed speed_max of 100m / s; Step 2: Construct the state space vector S; Preferably, step 2 includes: Step 21: Calculate the gravitational vector F_a of the moving object based on the position coordinates uav of the moving object and the position coordinates target of the target point. F_a=target-uav Step 22: Calculate the magnitude target_dist and direction angle target_theta of the gravitational vector F_a.
[0046] Step 23: Convert the center position coordinates obs of multiple obstacles into the repulsive force vector F_r of the moving object. Based on the distance and relative position between the moving object and each obstacle, calculate the magnitude and direction of the repulsive force exerted by each obstacle on the moving object. Then, synthesize these repulsive forces to obtain the total repulsive force vector F_r. The formula for calculating F_r is:
[0047] Where k is the repulsion coefficient, k=1, n is the number of obstacles, n=12, and obs i It is the coordinate of the i-th obstacle.
[0048] Step 24: Calculate the magnitude obs_dist and direction angle obs_theta of the repulsive force vector F_r.
[0049] Step 25: S includes 8 elements, each element in S is: the horizontal coordinate of the moving object uav_x, the vertical coordinate of the moving object uav_y, the speed of the moving object speed, the speed direction angle of the moving object yaw, the attractive force vector size target_dist, the attractive force vector direction angle target_theta, the repulsive force vector size obs_dist, and the repulsive force vector direction angle obs_theta.
[0050] Step 3: build the action vector A of the moving object; Wherein, A has a total of 2 elements, which are the acceleration a of the moving object and the angular velocity omega of the speed direction angle.
[0051] It should be noted that the acceleration a is used to change the speed speed of the moving object, and the angular velocity omega is used to change the speed direction angle yaw of the moving object. By adjusting the speed and direction angle, the moving object can move in the environment and avoid obstacles.
[0052] Step 4: according to the action vector A and the state space vector S of the moving object, build the action function action of the moving object, and the input quantity of the action function is the action vector A and the output quantity is the state space vector S.
[0053] The specific implementation of the action function action is: Step 41: after time step dt, update the size target_dist and direction angle target_theta of the attractive force vector F_a, and update the size obs_dist and direction angle obs_theta of the repulsive force vector F_r according to step 2; Preferably, step 41 includes: Step 411: obtain the obstacle center position coordinates and target point position coordinates target; Step 412: update the moving object speed speed, the moving object speed direction angle yaw, and the moving object position coordinates uav; Wherein, according to the current time moving object speed speed, the moving object speed direction angle yaw, and the moving object position coordinates uav, update the moving object speed speed', the moving object speed direction angle yaw', and the moving object position coordinates uav' after time step dt. The update method is: speed' = speed +dt*a(2) yaw' = yaw +dt*omega(3) uav' = uav + speed' *dt * [cos(yaw'), sin(yaw')](4) Step 413: According to the updated motion object position coordinates uav', update the size target_dist and the direction angle target_theta of the attractive force vector F_a, and update the size obs_dist and the direction angle obs_theta of the repulsive force vector F_r according to step 2.
[0054] Step 42: According to the updated motion object horizontal coordinate uav_x, motion object vertical coordinate uav_y, motion object speed speed, motion object speed direction angle yaw, attractive force vector size target_dist, attractive force vector direction angle target_theta, repulsive force vector size obs_dist, and repulsive force vector direction angle obs_theta, reconstruct the state space vector S.
[0055] Step 5: Use the preset reward function f to guide the motion object to move towards the target point and avoid obstacles.
[0056] Specifically, step 5 includes: Step 51: According to target_theta, yaw, speed, and speed_max, construct an angle reward function f_near: f_near = cos(target_theta - yaw) * speed / speed_max; When the motion object speed direction angle is towards the target point, a positive reward is given to encourage the motion object to move towards the target as soon as possible.
[0057] Step 52: According to obs_theta, yaw, speed, and speed_max, construct an obstacle avoidance reward function f_obs: f_obs = cos(obs_theta - yaw) * speed / speed_max When the motion object approaches the obstacle or collides with the obstacle, a larger negative reward is given to punish the collision behavior of the motion object.
[0058] Step 53: According to target_dist, construct a distance reward function f_dist: f_dist = 1 / target_dist The shorter the distance between the motion object and the target point, the greater the reward, to encourage the motion object to reach the target as soon as possible.
[0059] Step 54: According to the current time step number step of the motion object, construct a step reward function f_step: f_step = -step The more steps the moving object takes to reach the target point, the greater the negative reward is, so as to shorten the moving object's trajectory.
[0060] Step 55: Set the reaching target reward f_finish, which gives a greater reward when the moving object reaches the target. If target_dist is less than the set threshold e, which means the moving object reaches the target point, f_finish is 20, otherwise, f_finish is 0.
[0061] Step 56: According to the angle reward function f_near, the obstacle avoidance reward function f_obs, the distance reward function f_dist, the step reward function f_step and the reaching target reward f_finish, construct the reward function f: f = k1 * f_near + k2 * f_obs + k3 * f_dist + k4 * f_step + k5 * f_finish Wherein, k1, k2, k3, k4 and k5 are the weighting coefficients of each reward, which are positive numbers. k1 = 2, k2 = 1, k3 = 0.5, k4 = 1 / 200 and k5 = 1.
[0062] Step 6: Train the moving object using reinforcement learning algorithm to obtain the trained moving object trajectory planning strategy network B.
[0063] Specifically, step 6 includes: Step 61: Build the strategy network B and the value network V based on the multilayer perception mechanism, and initialize the network parameters. The input of the strategy network is the state space vector S, and the output is the action vector A. The input of the value network is the state space vector S and the action vector A, and the output is the value V_a.
[0064] Step 62: The moving object executes the action function action according to the action vector A output by the current strategy network B in the manner described in step 4, and collects the state vector S, the action vector A and the reward function f.
[0065] Step 63: Based on the value network V and the collected state vector S, action vector A and reward function f, calculate V_a of each time step to evaluate the relative goodness of the action.
[0066] Step 64: According to V_a, update the parameters of the strategy network B by gradient ascent method, so as to increase the probability of generating high reward action vector A. Update the parameters of the value network V, so that it can more accurately predict the value of the state space vector S and the action vector A.
[0067] Step 65: Cycle steps 61-65 until the policy network B converges, i.e. the moving object can stably and efficiently reach the target point from the starting point. The trained moving object policy network B is obtained.
[0068] Step 7: Evaluate the moving object policy network B.
[0069] Specifically, step 7 includes: Step 71: According to the two-dimensional motion environment, use the moving object policy network B to perform an iterative process on the moving object horizontal coordinate uav_x and the moving object vertical coordinate uav_y. Specifically, step 71 includes: inputting the state space vector S into the policy network B to output the action vector A, and updating the state space vector S according to step 4. If the distance from the moving object to the target target_dist is less than the set threshold e, e = 20m, then terminate the iteration.
[0070] Step 72: If the distance from the moving object to the target target_dist is less than the set threshold e after the iteration is completed, it is considered that the trained model is usable, otherwise, retrain the policy network B according to the method described in step 6.
[0071] Step 8: Use the moving object policy network B to perform moving object trajectory planning in a time advancing manner.
[0072] Specifically: input the state space vector S of the current time into the moving object policy network B to output the action vector A, and update the state space vector S of the next time according to step 4. The updated moving object position coordinate uav' is the next time trajectory point planned by the moving object. If the distance from the moving object to the target target_dist is less than the set threshold e, the trajectory planning is completed.
[0073] The above steps are simulated using Python, and the trajectory planning results are shown in FIG. 8. Figure 1
[0074] The above describes specific embodiments of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, some improvements and refinements can be made. These improvements and refinements should also be within the scope of protection of the present application.
Claims
1. A trajectory planning method based on reinforcement learning, characterized in that, include: Step 1: Construct a two-dimensional motion environment and obtain the state parameters of the moving object; Step 2: Construct the state space vector S; Step 3: Construct the motion vector A of the moving object; where A has two elements, namely the acceleration a and the angular velocity omega of the velocity direction angle of the moving object; Step 4: Based on the motion vector A and state space vector S of the moving object, construct the motion function action of the moving object. The input of the motion function is the motion vector A, and the output is the state space vector S. Step 5: Use the preset reward function f to guide the moving object toward the target point and avoid obstacles; Step 6: Train the moving object using a reinforcement learning algorithm to obtain the trained moving object trajectory planning policy network B; Step 7: Evaluate the motion object policy network B; Step 8: Use the moving object policy network B to plan the trajectory of the moving object in a time-progression manner.
2. The method according to claim 1, characterized in that, Step 1 includes: Using the plane containing the moving object, target point, and obstacle as the coordinate plane, and the initial position of the moving object as the origin O, with the right as the positive direction of the Ox axis and the top as the positive direction of the Oy axis, define a planar map coordinate system Oxy; obtain the position coordinates uav of the moving object, the center position coordinates obs of the obstacle, and the position coordinates target of the target point in the coordinate system Oxy, as well as the speed, velocity direction angle yaw, maximum angular velocity omega_max, and maximum speed_max of the moving object.
3. The method according to claim 1, characterized in that, Step 2 includes: Step 21: Calculate the gravitational vector F_a = target - uav of the moving object and the position coordinates of the target point. Step 22: Calculate the magnitude target_dist and direction angle target_theta of the gravitational vector F_a; Step 23: Convert the center position coordinates obs of multiple obstacles into the repulsive force vector F_r of the moving object. Calculate the magnitude and direction of the repulsive force exerted by each obstacle on the moving object based on the distance and relative position between the moving object and each obstacle. Combine these repulsive forces to obtain the total repulsive force vector F_r. Step 24: Calculate the magnitude obs_dist and direction angle obs_theta of the repulsive force vector F_r; Step 25: S consists of 8 elements, each of which is: the horizontal coordinate of the moving object uav_x, the vertical coordinate of the moving object uav_y, the speed of the moving object speed, the direction angle of the moving object speed yaw, the magnitude of the gravity vector target_dist, the direction angle of the gravity vector target_theta, the magnitude of the repulsion vector obs_dist, and the direction angle of the repulsion vector obs_theta.
4. The method according to claim 1, characterized in that, Step 4 includes: Step 41: After time step dt, update the magnitude target_dist and direction angle target_theta of the gravity vector F_a according to step 2, and update the magnitude obs_dist and direction angle obs_theta of the repulsion vector F_r. Step 42: Based on the updated x-coordinate uav_x, y-coordinate uav_y, velocity speed, velocity direction angle yaw, gravitational vector size target_dist, gravitational vector direction angle target_theta, repulsive vector size obs_dist, and repulsive vector direction angle obs_theta, reconstruct the state space vector S.
5. The method according to claim 4, characterized in that, Step 41 includes: Step 411: Obtain the coordinates of the obstacle's center and the target point's position. Step 412: Update the moving object's speed, moving object's velocity direction angle yaw, and moving object's position coordinates uav; wherein, based on the moving object's speed, moving object's velocity direction angle yaw, and moving object's position coordinates uav at the current moment, update the moving object's speed', moving object's velocity direction angle yaw', and moving object's position coordinates uav' after time dt. Step 413: Based on the updated position coordinates uav' of the moving object, update the magnitude target_dist and direction angle target_theta of the gravity vector F_a according to Step 2, and update the magnitude obs_dist and direction angle obs_theta of the repulsion vector F_r.
6. The method according to claim 1, characterized in that, Step 5 includes: Step 51: Construct the angle reward function f_near based on target_theta, yaw, speed, and speed_max. f_near = cos(target_theta – yaw) * speed / speed_max; A positive reward is given when the velocity direction angle of a moving object is toward the target point, in order to encourage the moving object to move toward the target as quickly as possible; Step 52: Construct the obstacle avoidance reward function f_obs based on obs_theta, yaw, speed, and speed_max: f_obs = cos(obs_theta – yaw) * speed / speed_max, A large negative reward is given when the moving object approaches or collides with an obstacle, in order to punish the moving object's collision behavior; Step 53: Construct the distance reward function f_dist based on target_dist: f_dist = 1 / target_dist; The shorter the distance between the moving object and the target point, the greater the reward, in order to encourage the moving object to reach the target as quickly as possible; Step 54: Construct the step reward function f_step based on the current step count of the moving object: f_step = -step; The more steps a moving object takes to reach the target point, the greater the negative reward, in order to minimize the moving object's trajectory; Step 55: Set the target reward f_finish. When the moving object reaches the target, a larger reward is given. If target_dist is less than the set threshold e, it means that the moving object has reached the target point, so f_finish is 20; otherwise, f_finish is 0. Step 56: Construct the reward function f based on the angle reward function f_near, obstacle avoidance reward function f_obs, distance reward function f_dist, step count reward function f_step, and goal completion reward f_finish: f = k1 * f_near + k2 * f_obs + k3 * f_dist + k4 * f_step + k5 * f_finish; Among them, k1, k2, k3, k4 and k5 are the weighting coefficients for each reward, and all are positive numbers.
7. The method according to claim 1, characterized in that, Step 6 includes: Step 61: Construct a policy network B and a value network V based on a multilayer perceptron, and initialize the network parameters; wherein, the input of the policy network is the state space vector S, and the output is the action vector A; the input of the value network is the state space vector S and the action vector A, and the output is the value V_a; Step 62: The moving object executes the action function action according to the method described in Step 4, based on the action vector A output by the current policy network B, and collects the state vector S, action vector A, and reward function f. Step 63: Based on the value network V and the collected state vector S, action vector A, and reward function f, calculate V_a for each time step and evaluate the relative quality of the action; Step 64: Based on V_a, update the parameters of the policy network B using the gradient ascent method to increase the probability of generating a high-reward action vector A; update the parameters of the value network V so that it can more accurately predict the value of the state space vector S and the action vector A. Step 65: Repeat steps 61 to 65 until policy network B converges, that is, the moving object can stably and efficiently reach the target point from the starting point; thus obtaining the trained moving object policy network B.
8. The method according to claim 1, characterized in that, Step 7 includes: Step 71: Based on the two-dimensional motion environment, use the motion object policy network B to iterate over the horizontal coordinate uav_x and the vertical coordinate uav_y of the motion object. Specifically, step 71 includes: inputting the state space vector S into the policy network B and outputting the action vector A; updating the state space vector S according to step 4; if the distance from the moving object to the target target_dist is less than the set threshold e or the maximum number of iterations is reached, then the iteration is terminated. Step 72: If the distance from the moving object to the target, target_dist, is less than the set threshold e after the iteration is completed, the trained model is considered to be usable; otherwise, the policy network B is retrained according to the method described in step 6.
9. The method according to claim 1, characterized in that, Step 8 includes: Input the current state space vector S into the policy network B of the moving object, and output the action vector A. Update the state space vector S for the next moment according to step 4. The updated position coordinates uav' of the moving object are the planned trajectory points for the next moment. If the distance from the moving object to the target target_dist is less than the set threshold e, the trajectory planning is completed.