Deep reinforcement learning path planning method and system based on reward function improvement

Abstract

The present invention proposes a deep reinforcement learning path planning method and system based on reward function improvement, which relates to the field of path planning technology. The method includes constructing a deep reinforcement learning training environment model; collecting feasible state information and position state information of an autonomous mobile robot set in the deep reinforcement learning training environment model; constructing a state space model corresponding to a deep deterministic policy gradient algorithm and a policy learning algorithm, and making the state space model output a state space function and an action space function; obtaining a reward function corresponding to all optional state actions of the autonomous mobile robot; storing the state space function, the action space function and the reward function in an experience pool, and selecting samples from the experience pool to update the deep deterministic policy gradient algorithm and the policy learning algorithm; repeating the above steps until the autonomous mobile robot moves to the target point to complete the path planning. The present invention helps to improve the convergence speed of path planning and the efficiency of data exploration.

Description

Technical Field

[0001] The present invention relates to the field of path planning technology, and in particular to a deep reinforcement learning path planning method and system based on improved reward function. Background Art

[0002] AMR is widely used in logistics, military, health and other industries due to its operational capabilities in manufacturing scenarios. With the transformation and upgrading of the industry, the complex process flow and flexible production requirements in advanced manufacturing have posed major challenges to the efficiency and reliability of AMR. Compared with traditional path planning technology, deep reinforcement learning based on algorithms such as DQN, DDRG, and DIAYN has strong adaptability and modeling capabilities in dealing with path planning problems in dynamic environments, and has been widely used in the field of AMR navigation.

[0003] A Chinese patent with publication number CN110977967A discloses a robot path planning method based on deep reinforcement learning. This method replaces the action value function in Q-learning with a value function approximation method, designs a deep convolutional neural network with a 4-layer structure, and replaces the traditional Q value table with the output of the network, solving the dimensionality curse problem caused by Q-learning when the state space is large. However, the above scheme has insufficient learning efficiency due to the low utilization rate of input data, and the lack of multi-level judgment of rewards for input parameters will seriously reduce the update efficiency of the neural network, thereby affecting the convergence of the algorithm. The single reward function selection principle will have the problem of sparse rewards. Therefore, it is very necessary to provide a deep reinforcement learning path planning method and system based on reward function improvement to improve the convergence speed of path planning and data exploration efficiency. Summary of the invention

[0004] In view of this, the present invention proposes a deep reinforcement learning path planning method and system based on reward function improvement. By building a simulation training environment that simulates the movement of an autonomous mobile robot in a complex environment to optimize the robot's mobile environment, and based on this, a corresponding state space model is constructed, so that the output state space function and action space function can accurately describe the robot's motion characteristics and decision-making process, improve the algorithm's convergence speed and performance, and thus achieve the purpose of path planning convergence speed and data exploration efficiency.

[0005] The present invention provides a deep reinforcement learning path planning method based on reward function improvement, the method comprising:

[0006] S1, constructing a deep reinforcement learning training environment model, and initializing the network of the deep reinforcement learning training environment model based on a deep deterministic policy gradient algorithm and a policy learning algorithm;

[0007] S2, collecting feasible state information and position state information of an autonomous mobile robot set in a deep reinforcement learning training environment model, wherein the position state information includes an initial position of the autonomous mobile robot and an initial position of a target point;

[0008] S3, based on the feasible state information and the position state information, construct a state space model corresponding to the deep deterministic policy gradient algorithm and the policy learning algorithm, and make the state space model output a state space function and an action space function;

[0009] S4, inputting the state space function and the action space function into the deep reinforcement learning training environment model to obtain reward functions corresponding to all optional state actions of the autonomous mobile robot;

[0010] S5, storing the state space function, the action space function and the reward function into an experience pool, and selecting samples from the experience pool to update the deep deterministic policy gradient algorithm and the policy learning algorithm;

[0011] S6, repeating steps S3-S5 until the autonomous mobile robot moves to the target point to complete the path planning.

[0012] On the basis of the above technical solution, preferably, the collecting of feasible state information and position state information of the autonomous mobile robot set in the deep reinforcement learning training environment model specifically includes:

[0013] Measuring the autonomous mobile robot's own position and posture through an odometer inside the autonomous mobile robot, and setting the autonomous mobile robot's initial position in the world coordinate system;

[0014] Measuring the distance and deflection angle of the autonomous mobile robot relative to the initial position and posture to obtain the position and posture of the autonomous mobile robot relative to the autonomous mobile robot coordinate system at the next moment;

[0015] The estimated position and posture of the autonomous mobile robot in the world coordinate system at any time is calculated to obtain the feasible state information and position state information corresponding to the autonomous mobile robot.

[0016] On the basis of the above technical solution, preferably, the state space function includes a mapping function between the robot orientation and the obstacle and a mapping function between the radar measurement information and the real measurement information.

[0017] More preferably, the expression of the mapping function between the robot orientation and the obstacle is:

[0018]

[0019] in, represents the distance between the robot and the target point at time t, y g Indicates the distance between the target point and the y-axis. represents the estimated distance of the robot from the y-axis, x g Indicates the distance between the target point and the x-axis. represents the estimated distance between the robot and the x-axis, math.atan2() represents the inverse tangent function, It represents the angle between the autonomous mobile robot and the line connecting the target point. Indicates the angle between the robot and the obstacle. Indicates the deviation between the robot and the target point in the x-axis direction. Indicates the deviation between the robot and the target point in the y-axis direction. Indicates the deviation value of the angle between the robot and the y-axis. Indicates the actual angle between the robot and the y-axis, Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, Represents the estimated angle of the robot with the y-axis.

[0020] More preferably, the expression of the mapping function between the radar measurement information and the real measurement information is:

[0021]

[0022] in, represents the estimated distance between the robot and the obstacle at time t, represents the angle between the robot and the nearest endpoint of the obstacle. math.atan2() represents the inverse tangent function. Indicates the actual distance between the robot and the x-axis, represents the estimated distance of the robot from the x-axis, Indicates the actual distance between the robot and the y-axis, represents the estimated distance of the robot from the y-axis, Represents the difference between radar measurement information and real information, Indicates the actual distance between the robot and the obstacle. Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, Indicates the actual coordinate of the obstacle endpoint on the x-axis at the current moment. Indicates the actual coordinate of the obstacle endpoint on the y-axis at the current moment.

[0023] More preferably, the expressions of the state space function of the deep deterministic policy gradient algorithm and the state space function of the policy learning algorithm are:

[0024]

[0025] Among them, S DDPG represents the state space function of the deep deterministic policy gradient algorithm, S DIAYN represents the state space function of the policy learning algorithm, w t-1 represents the angular velocity of the robot at time t-1, v t-1 represents the linear velocity of the robot at time t-1, Represents the difference between the robot’s estimated pose and its true pose. Represents the difference between radar measurement information and true information, w t-2 represents the angular velocity at time t-2, v t-2 represents the linear velocity of the robot at time t-2, X o represents the obstacle coordinate equation.

[0026] More preferably, the reward function is expressed as:

[0027]

[0028] Among them, r(s t ,a t ) represents the reward function, represents the stage-by-stage reward function, represents the continuous reward function, T represents the total movement time of the autonomous mobile robot, represents the distance between the robot and the target point at time t, represents the angle between the autonomous mobile robot and the line connecting the target point, d ig Represents the distance between the initial position of the autonomous mobile robot relative to the world coordinate system and the target point position, d g represents the threshold distance of the target endpoint, represents the distance between the robot and the nearest endpoint of the obstacle at time t, It represents the angle between the robot and the nearest endpoint of the obstacle. It represents the difference between the estimated pose and the real pose of the autonomous mobile robot obtained by simulation. represents the difference between the simulated radar measurement information and the real information, η represents the reward factor set, η∈(η1,η2,η3,η4,η5,η6) and η i >0.

[0029] In a second aspect of the present application, a deep reinforcement learning path planning system based on reward function improvement is provided, wherein the deep reinforcement learning path planning system comprises a data acquisition module, a strategy optimization module and a path planning module, wherein:

[0030] The data acquisition module is used to construct a deep reinforcement learning training environment model, perform network initialization on the deep reinforcement learning training environment model based on a deep deterministic policy gradient algorithm and a policy learning algorithm, and collect feasible state information and position state information of an autonomous mobile robot set in the deep reinforcement learning training environment model, wherein the position state information includes an initial position of the autonomous mobile robot and an initial position of a target point;

[0031] The policy optimization module is used to construct a state space model corresponding to the deep deterministic policy gradient algorithm and the policy learning algorithm based on the feasible state information and the position state information, and make the state space model output a state space function and an action space function, input the state space function and the action space function into the deep reinforcement learning training environment model, obtain the reward function corresponding to all optional state actions of the autonomous mobile robot, store the state space function, the action space function and the reward function into an experience pool, and select samples from the experience pool to update the deep deterministic policy gradient algorithm and the policy learning algorithm;

[0032] The path planning module is used to repeat the optimization steps in the strategy optimization module until the autonomous mobile robot moves to the target point to complete the path planning.

[0033] In a third aspect of the present application, an electronic device is provided, comprising a processor, a memory, a user interface and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory.

[0034] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of a deep reinforcement learning path planning method based on reward function improvement.

[0035] The deep reinforcement learning path planning method and system based on reward function improvement provided by the present invention have the following beneficial effects compared with the prior art:

[0036] (1) By constructing a deep reinforcement learning training environment model and initializing the model based on the deep deterministic policy gradient algorithm and policy learning algorithm, a simulation training environment is built to simulate the movement of an autonomous mobile robot in a complex environment to optimize the robot's mobile environment. At the same time, the feasible state information and position state information of the autonomous mobile robot are collected, and the corresponding state space model is constructed based on this, so that the output state space function and action space function can accurately describe the robot's motion characteristics and decision-making process, and can be trained and optimized more targetedly, thereby improving the convergence speed and performance of the algorithm. By continuously iterating the above process, the deep deterministic policy gradient algorithm and policy learning algorithm can continuously optimize their own performance based on accumulated experience, thereby achieving the purpose of convergence speed of path planning and data exploration efficiency;

[0037] (2) The problem of sparse rewards is overcome by combining staged rewards with continuous rewards during the operation of the autonomous mobile robot, and the deviation factor is used to update the deep deterministic policy gradient algorithm and policy learning algorithm in real time, which ensures the convergence of the deep deterministic policy gradient algorithm and policy learning algorithm, enabling the autonomous mobile robot to complete the optimal real-time path planning problem faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of the process flow of the deep reinforcement learning path planning method based on the improvement of the reward function provided by the present invention;

[0040] Figure 2 This is a graph showing experimental comparison results between the optimization algorithm combining the deep deterministic policy gradient algorithm and the policy learning algorithm provided by the present invention and the traditional method;

[0041] Figure 3 A comparison chart of the changes in motion parameters of an autonomous mobile robot when it moves between an optimization algorithm combining a deep deterministic policy gradient algorithm and a policy learning algorithm provided by the present invention and a traditional method;

[0042] Figure 4 A schematic diagram of the structure of the deep reinforcement learning path planning system provided by the present invention;

[0043] Figure 5 This is a schematic structural diagram of an electronic device provided by the present invention.

[0044] Explanation of the accompanying drawings: 1. Deep reinforcement learning path planning system; 11. Data acquisition module; 12. Strategy optimization module; 13. Path planning module; 2. Electronic device; 21. Processor; 22. Communication bus; 23. User interface; 24. Network interface; 25. Memory. DETAILED DESCRIPTION

[0045] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Before introducing the embodiments of the present invention, some terms and their abbreviations involved in the embodiments of the present invention are first defined and explained.

[0047] Deep Deterministic Policy Gradient Algorithm (DDPG): DDPG is an Actor-Critic based deep reinforcement learning algorithm that can be used to solve problems in continuous action space. It consists of two networks: the Actor network and the Critic network. The Actor network is responsible for outputting actions, and the Critic network is responsible for evaluating the value of actions. The Actor network uses a deterministic strategy, that is, it directly outputs continuous actions without sampling, and the Critic network uses the Q function to evaluate the value of actions. DDPG updates the parameters of the Actor network and the Critic network simultaneously by gradient descent, so that the output action can maximize the expected return. The DDPG algorithm has good convergence and stability, and performs well in continuous action space problems.

[0048] Strategy Learning Algorithm (DIAYN): DIAYN is an unsupervised strategy learning algorithm based on mutual information maximization. Its goal is to learn a set of different skills. Each skill corresponds to a different strategy. DIAYN contains two networks: skill recognition network and skill generation network. The skill recognition network is used to determine which skill the current state belongs to, and the skill generation network is used to output the action of the corresponding skill. DIAYN learns a set of different skills by maximizing the mutual information between the state and the skill. These skills can be combined to complete complex tasks. The DIAYN algorithm does not require manual design of reward functions, can automatically learn meaningful skills, and has strong transfer learning capabilities.

[0049] The present invention discloses a method for calibrating the phase center change of a navigation satellite antenna, referring to Figure 1 The method includes steps S1 to S6.

[0050] Step S1, construct a deep reinforcement learning training environment model, and initialize the network of the deep reinforcement learning training environment model based on the deep deterministic policy gradient algorithm and the policy learning algorithm.

[0051] In this step, you can choose to use common robot simulation environments such as Gazebo or PyBullet. These simulation environments can simulate real physical environments, including robots, obstacles, terrain, etc. In the simulation environment, build a simulation model that includes the autonomous mobile robot and its surrounding environment, set the initial position and target position of the robot, and add some obstacles in the environment. And configure sensors for the robot, such as odometers, radars, etc., to perceive environmental information.

[0052] Furthermore, two deep reinforcement learning algorithms are selected: the Deep Deterministic Policy Gradient (DDPG) algorithm and the Policy Learning Algorithm (DIAYN). Actor networks and Critic networks are constructed for these two algorithms respectively. The Actor network is used to output actions, and the Critic network is used to evaluate the value of actions. These neural networks are initialized, such as randomly initializing weight parameters. The constructed deep reinforcement learning algorithm is integrated with the simulation environment so that the algorithm can interact with the simulation environment. At each time step, the algorithm can output actions based on the current state, and feed the actions back to the simulation environment to obtain the next state and reward.

[0053] Step S2, collecting feasible state information and position state information of the autonomous mobile robot set in the deep reinforcement learning training environment model, the position state information includes the initial position of the autonomous mobile robot and the initial position of the target point.

[0054] This step also includes steps S21 to S23.

[0055] Step S21, measuring the posture of the autonomous mobile robot through the odometer inside the autonomous mobile robot, and setting the initial position of the autonomous mobile robot in the world coordinate system.

[0056] Step S22, measuring the distance and deflection angle of the autonomous mobile robot relative to the initial position and posture, so as to obtain the position and posture of the autonomous mobile robot relative to the autonomous mobile robot coordinate system at the next moment.

[0057] Step S23, calculating the estimated position and posture of the autonomous mobile robot in the world coordinate system at any time, so as to obtain feasible state information and position state information corresponding to the autonomous mobile robot.

[0058] Step S3, based on the feasible state information and the position state information, construct a state space model corresponding to the deep deterministic policy gradient algorithm and the policy learning algorithm, and make the state space model output the state space function and the action space function.

[0059] In this embodiment, the state space function includes a mapping function between the robot orientation and obstacles and a mapping function between radar measurement information and real measurement information.

[0060] When an autonomous mobile robot is navigating autonomously, the robot measures its own position and posture through its internal odometer. First, the robot's initial position in the world coordinate system is set, and then the distance and angle of movement of the robot relative to the initial position and posture are measured to obtain the robot's position and posture relative to the robot coordinate system at the next moment. Finally, the coordinate system transformation formula (4-1) is used to calculate the estimated position and posture of the robot in the world coordinate system at any moment. The set target point can be expressed relative to the world coordinate system as (x g ,y g ), the expression of the mapping function between the robot orientation and the obstacle is:

[0061]

[0062] in, represents the distance between the robot and the target point at time t, y g Indicates the distance between the target point and the y-axis. represents the estimated distance of the robot from the y-axis, x g Indicates the distance between the target point and the x-axis. represents the estimated distance between the robot and the x-axis, math.atan2() represents the inverse tangent function, It represents the angle between the autonomous mobile robot and the line connecting the target point. Indicates the angle between the robot and the obstacle. Indicates the deviation between the robot and the target point in the x-axis direction. Indicates the deviation between the robot and the target point in the y-axis direction. Indicates the deviation value of the angle between the robot and the y-axis. Indicates the actual angle between the robot and the y-axis, Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, Represents the estimated angle of the robot with the y-axis.

[0063] The information measured by the radar is actually the distance between the radar and the surrounding environment. When measuring the estimated distance between the robot and the obstacle, the estimated distance between the radar and the obstacle is measured. and estimation error In order to simplify the experimental model and facilitate the installation of the laser radar and the measurement of the surrounding environment information, the laser radar is set to be placed in front of the autonomous mobile robot, and the distance Δy between the geometric center of the radar and the geometric center of the robot is measured. The radar can be expressed as (0, Δy, 0) in the robot coordinate system. T , the estimated position of the radar relative to the world coordinate system is Similarly, the actual position (simulation) of the radar is: The estimated distance to the obstacle measured by the radar is The position of the obstacle is approximated by establishing a coordinate equation in the world coordinate system, the estimated coordinates of the robot and the lidar and the estimated distance to the obstacle measured by the radar Determine the type of obstacle, so as to obtain the obstacle endpoint of the robot perpendicular to the obstacle approximate equation at the current moment The expression of the mapping function between radar measurement information and real measurement information is:

[0064]

[0065] in, represents the estimated distance between the robot and the obstacle at time t, represents the angle between the robot and the nearest endpoint of the obstacle. math.atan2() represents the inverse tangent function. Indicates the actual distance between the robot and the x-axis, represents the estimated distance of the robot from the x-axis, Indicates the actual distance between the robot and the y-axis, represents the estimated distance of the robot from the y-axis, Represents the difference between radar measurement information and real information, Indicates the actual distance between the robot and the obstacle. Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, Indicates the actual coordinate of the obstacle endpoint on the x-axis at the current moment. Indicates the actual coordinate of the obstacle endpoint on the y-axis at the current moment.

[0066] The expression of the mapping function between radar measurement information and real measurement information is:

[0067]

[0068] in, represents the estimated distance between the robot and the obstacle at time t, represents the angle between the robot and the nearest endpoint of the obstacle. math.atan2() represents the inverse tangent function. Indicates the actual distance between the robot and the x-axis, represents the estimated distance of the robot from the x-axis, Indicates the actual distance between the robot and the y-axis, represents the estimated distance of the robot from the y-axis, Represents the difference between radar measurement information and real information, Indicates the actual distance between the robot and the obstacle. Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, Indicates the actual coordinate of the obstacle endpoint on the x-axis at the current moment. Indicates the actual coordinate of the obstacle endpoint on the y-axis at the current moment.

[0069] If the DDPG and DQN algorithms add navigation environment states that are not related to rewards to the state space, the state value function will become bloated, which will lead to low learning efficiency of neural network training. Therefore, the reasonable selection of state space is critical for performing navigation tasks. However, the DIAYN algorithm needs as large a state space as possible to learn useful skills in order to learn more unpredictable skills and experience, so as to better use the skill network to initialize the policy network parameters of DDPG. Based on the above ideas, the following states are newly added to the state space of the DIAYN algorithm based on the state space of the DDPG and DQN algorithms. Then the expression of the state space function of the deep deterministic policy gradient algorithm is:

[0070]

[0071] The parameter w t-2 and v t-2 Added to the state space, in the hope that the skill network can learn the robot's acceleration and angular acceleration information, the addition of other parameters injects new vitality into the state space of the strategy learning algorithm, and provides more possibilities for obtaining useful skills. DIAYN The expression is:

[0072]

[0073] Among them, S DDPG represents the state space function of the deep deterministic policy gradient algorithm, S DIAYN represents the state space function of the policy learning algorithm, w t-1 represents the angular velocity of the robot at time t-1, v t-1 represents the linear velocity of the robot at time t-1, Represents the difference between the robot’s estimated pose and its true pose. Represents the difference between radar measurement information and true information, w t-2 represents the angular velocity at time t-2, v t-2 represents the linear velocity of the robot at time t-2, X o represents the obstacle coordinate equation.

[0074] Furthermore, in the autonomous navigation task of the autonomous mobile robot, the purpose of designing the action space is to control the robot to move smoothly to the target point. The differential autonomous mobile robot uses the left and right wheels to move. The linear velocity and angular velocity information of the robot can be obtained through the coordinated control of the left and right wheels. Therefore, the motion control of the robot can be achieved by setting the action space of the linear velocity v and the angular velocity w. The action space function A can be expressed as:

[0075] A=(v,w)

[0076] Set the linear velocity v to the range of [0, 0.05] (m / s), and the angular velocity w to the range of [-0.8, 0.8] (rad / s). In DQN, the action space is discrete, and the action space is discretized into 6 actions, with the linear velocity separation unit of 0.01 and the angular velocity separation unit of 0.4. In DDPG, since it can be a continuous action space, it has continuous linear velocity and angular velocity.

[0077] Step S4: input the state space function and the action space function into the deep reinforcement learning training environment model to obtain the reward function corresponding to all optional state actions of the autonomous mobile robot.

[0078] In one example, the obtained state space function can also be input into the instantiated neural network, and the value function of all optional state-action pairs is obtained through neural network calculation. The action A to be executed is selected based on maximizing the return of the value function. After the execution is completed, the state space and action are input into the Critic network, and an estimated maximum Q value is output to obtain the reward function and the objective function based on the next state space.

[0079] The instantiated neural network consists of 4 layers, namely input layer, hidden layer, dropout layer, and output layer. The following are the specific parameters and selections of the deep neural network structure of DQN, DDPG and DIAYN in the 4-layer network structure:

[0080] In the input layer, the input is the state space dimension of the algorithm, the input of DQN and DDPG is 8, and the input of DIAYN is 16; in the hidden layer, in order to control the experimental variables, the hidden layer parameters of DQN, DDPG and DIAYN networks are all consistent; the Dropout network is due to too few experience samples in the experience pool, and the method of stopping the update of neurons with a certain probability avoids the overfitting phenomenon of the neural network model, and the Dropout layer parameters are consistent. In the output layer, the Sigmoid and Tanh activation functions are used to output the linear velocity v and angular velocity w in the continuous action space respectively, and the linear function is used to output the value function Q. That is, the value network and target value network of DQN, the value problem and target value network of DDOG and the discriminator network of DIAYN use linear algebra to output the value function, and the policy network of DDPG and DIAYN uses Sigmoid and Tanh activation functions to output the robot action respectively.

[0081] Furthermore, the discriminator network of DIAYN uses the state obtained after the action output from skill z interacts with the environment as the input of the network layer, takes the maximization of the reward function as the objective function, updates the initial parameters through the SGD algorithm, and then inputs the probability function of skill z in the current state into the skill network. DIAYN's skill network is a potential strategy, which is determined by the distribution p(z) satisfied by skill z. It can change the state space by changing z, solve the maximization objective function and select the optimal state space, which can be used to select the optimal initial parameters of the DDPG policy network and find the maximum reward function of DDPG. After the DIAYN algorithm iteratively converges, the optimal skill network parameters are used as the initial parameters of the DDPG policy network.

[0082] In DDPG, the goal of the Actor network is to maximize the Q value of the Critic network for the current state and action. The goal of the Critic network is to evaluate the quality of the strategy by learning a value function. It receives the state and action as input and outputs an estimated maximum Q value. After a certain number of iterative learning, the Critic network finds the optimal action value function Q * (s,a), and after inputting it into the Actor network, the optimal action a is obtained. * (s) a * (s) is the optimal controller of the mobile robot.

[0083] For the specific implementation of the above neural network, a total of six networks are used: skill network, discriminator network, policy network, value network, target policy network and target value network. Each network has three hidden layers, consisting of 384, 128 and 32 neurons respectively. The activation function used is the RELU function. The SGD optimizer is used to train the network and optimize the network parameters.

[0084] During the training process of DIAYN, the skill and discriminator network parameters in the DIAYN algorithm and the robot posture are initialized first, and then the state space s of DIAYN is obtained by obtaining the robot sensor data. t , input the state space parameters into the network to obtain the current action space a t , using a t Control the robot to move, and the next state s will be obtained in the next step t+1 and reward r t Add to experience sample (s t ,a t ,r t ,s t+1 ) and the samples are put into the playback experience pool, and the DIAYN network parameters are updated uniformly in the experience pool. Finally, it is determined whether the robot has reached the target position. If it has reached the target position, it is determined whether the training number of DIAYN has been reached. If it has reached the training number, the DDPG parameters are trained. If it has not reached the training number, the next training is started. If it has not reached the target position, the state space of DIAYN is obtained and the above process continues.

[0085] The training process of DDPG is similar to that of DIAYN. The difference is that the initial policy network parameters of DDPG are initialized by the skill network parameters of DIAYN. In addition, in order to maximize the use of samples, DIAYN's training samples are put into DDPG. DDPG and DIAYN experience samples are added to the experience pool after being screened by the TD-error method. This ensures that the sample volume in the experience pool is sufficient and more useful samples are selected. The completion of DDPG training marks the establishment of the DIAYN-DDPG algorithm model.

[0086] In this step, in the autonomous mobile robot navigation task, the robot observes the system environment and moves based on its own posture. After reaching the next state, the environment will feedback a reward information to the robot, and the robot will move in the direction of maximizing the reward based on the reward. Therefore, the reward function plays a guiding role in controlling the movement of the robot. Since the robot has only one navigation goal, that is, to reach the target point, if only considering whether the robot moves to the target point will result in too sparse rewards, and the robot will not learn useful information. Therefore, it is very important to design a reasonable reward function. This article combines the state space to divide the reward function into two cases. One is to design a stage reward. This includes whether the robot is approaching the target point stably and whether it has reached the target point; second, designing continuous rewards It includes the distance and angle deviation between the robot and the target point, the distance and angle deviation between the robot and the obstacle, and the error between the robot's posture and radar measurement.

[0087] Therefore, the expression of the reward function is:

[0088]

[0089] Among them, r(s t ,a t ) represents the reward function, represents the stage-by-stage reward function, represents the continuous reward function, T represents the total movement time of the autonomous mobile robot, represents the distance between the robot and the target point at time t, represents the angle between the autonomous mobile robot and the line connecting the target point, d ig Represents the distance between the initial position of the autonomous mobile robot relative to the world coordinate system and the target point position, d g represents the threshold distance of the target endpoint, represents the distance between the robot and the nearest endpoint of the obstacle at time t, It represents the angle between the robot and the nearest endpoint of the obstacle. It represents the difference between the estimated pose and the real pose of the autonomous mobile robot obtained by simulation. represents the difference between the simulated radar measurement information and the real information, η represents the reward factor set, η∈(η1,η2,η3,η4,η5,η6) and η i >0.

[0090] In phased rewards In the navigation task, whether the robot can reach the target point is the most important evaluation indicator, so the reward difference is set to the maximum. In order to make the robot more directional in the process of reaching the target point, a phased reward is set. Specifically, the total movement time T of the robot is evenly divided into 10 important moments, and d ig The distance is evenly divided into 10 sections according to important moments. At each moment, the position of the robot and the target point must be determined. Whether the specified distance is reached, if so, a reward will be given.

[0091] In continuity reward The current distance between the robot and the target point The larger the value, the smaller the reward. The robot faces the angle between the target point and the target point. The range is [-π,π], use To determine whether the robot is moving towards the target point, if it moves towards the target point, it will get a reward value. The distance and angle deviation between the robot and the obstacle are just the opposite.

[0092] During robot movement, direction is more sensitive than distance and encountering obstacles is devastating for training. Therefore, the reward factor for robot direction is generally 10 to 20 times larger than the proportional coefficient for distance, and the reward factor for obstacles is generally set larger. The error evaluation index is added to the reward function to eliminate experience samples with large errors by preferential experience playback, so that the robot can learn more useful knowledge. Among them, η can balance the sample size in the experience pool and the error degree of the experience samples.

[0093] Step S5, storing the state space function, the action space function and the reward function into the experience pool, and selecting samples from the experience pool to update the deep deterministic policy gradient algorithm and the policy learning algorithm.

[0094] In this step, a priority experience replay mechanism is added to the deep deterministic policy gradient algorithm and policy learning algorithm fused deep reinforcement learning algorithm applied to robot autonomous navigation. TD-error (temporal difference error) is used as the sample evaluation criterion. The larger the TD-error, the more special the experience sample is, that is, the sample is more likely to be put into the experience pool. The sampling probability of the experience sample is given by the following formula:

[0095]

[0096] Where i is the experience sample obtained by the interaction between the robot and the environment, j is the experience sample that meets the TD-error judgment standard, and P j and P i It is a priority index based on the TD-error standard. α is a priority adjustment parameter with a value range of [0,1]. The smaller α is, the greater the probability of randomly selecting experience samples. When α is 0, uniform random sampling is used for sampling. The smaller α is, the higher the TD-error judgment standard is, and the more experience samples are restricted. When α is 1, only TD-error is used. When the number of experience samples is limited, only using the TD-error method will result in too few experience samples that meet the conditions, resulting in underfitting of neural network training. Therefore, a reasonable selection of α can ensure that the sample size in the experience pool is sufficient and more useful samples can be selected. Therefore, the use of the above-mentioned priority experience replay mechanism can ensure that samples with important learning value are replayed at a high frequency, avoid the loss of sample diversity, reduce network training time, and ensure the learning quality of the network model.

[0097] The deviation factor is calculated through the objective function of the action space function A and the reward function output by the neural network, and the state, action, and reward function of this stage are obtained and stored in the priority experience replay pool to update the current strategy; after a certain number of iterative updates, the optimal action value function is output.

[0098] Step S6, repeating steps S3-S5 until the autonomous mobile robot moves to the target point to complete the path planning.

[0099] By constructing a deep reinforcement learning training environment model and initializing the model based on the deep deterministic policy gradient algorithm and policy learning algorithm, a simulation training environment is built to simulate the movement of an autonomous mobile robot in a complex environment to optimize the robot's mobile environment. At the same time, the feasible state information and position state information of the autonomous mobile robot are collected, and the corresponding state space model is constructed based on this, so that the output state space function and action space function can accurately describe the robot's motion characteristics and decision-making process, and can be trained and optimized more specifically, thereby improving the convergence speed and performance of the algorithm. By continuously iterating the above process, the deep deterministic policy gradient algorithm and policy learning algorithm can continuously optimize their own performance based on accumulated experience, thereby achieving the purpose of convergence speed of path planning and data exploration efficiency.

[0100] See also Figure 2 , Figure 2 The figure shows the robot trajectory comparison between the optimization algorithm based on the combination of the deep deterministic policy gradient algorithm and the policy learning algorithm and the previous deep reinforcement learning algorithm. The initial position of the robot is represented by the black circle at (0, -0.4), and the target point is represented by the red circle at (0, 0.8). The experiment shows that the optimized algorithm based on the combination of the deep deterministic policy gradient algorithm and the policy learning algorithm has the best path distance in this scenario.

[0101] See also Figure 3 , Figure 3 A comparison chart showing the motion parameters of the optimization algorithm based on the combination of the deep deterministic policy gradient algorithm and the policy learning algorithm with the previous deep reinforcement learning algorithm in terms of the angular velocity, linear velocity, distance target error, and angle error motion parameters of the autonomous mobile robot. Figure 3 It can be seen that the angular velocity changes smoothly when the DIAYN-DDPG optimization algorithm is used, which reflects that the AMR motion control is relatively stable and has significant advantages in operation reliability.

[0102] Based on the above method, the embodiment of the present application discloses a deep reinforcement learning path planning system based on reward function improvement, referring to Figure 4 The deep reinforcement learning path planning system 1 includes a data acquisition module 11, a strategy optimization module 12 and a path planning module 13, wherein:

[0103] The data acquisition module 11 is used to construct a deep reinforcement learning training environment model, initialize the network of the deep reinforcement learning training environment model based on the deep deterministic policy gradient algorithm and the policy learning algorithm, and collect the feasible state information and position state information of the autonomous mobile robot set in the deep reinforcement learning training environment model, and the position state information includes the initial position of the autonomous mobile robot and the initial position of the target point;

[0104] The policy optimization module 12 is used to construct a state space model corresponding to the deep deterministic policy gradient algorithm and the policy learning algorithm based on the feasible state information and the position state information, and make the state space model output a state space function and an action space function, input the state space function and the action space function into the deep reinforcement learning training environment model, obtain the reward function corresponding to all optional state actions of the autonomous mobile robot, store the state space function, the action space function and the reward function into the experience pool, and select samples from the experience pool to update the deep deterministic policy gradient algorithm and the policy learning algorithm;

[0105] The path planning module 13 is used to repeat the optimization steps in the strategy optimization module 12 until the autonomous mobile robot moves to the target point to complete the path planning.

[0106] In one example, the data acquisition module 11 is used to measure the autonomous mobile robot's own posture through the odometer inside the autonomous mobile robot, set the initial position of the autonomous mobile robot in the world coordinate system; measure the distance and deflection angle moved by the autonomous mobile robot relative to the initial posture to obtain the posture of the autonomous mobile robot relative to the autonomous mobile robot coordinate system at the next moment; calculate the estimated posture of the autonomous mobile robot in the world coordinate system at any moment to obtain the feasible state information and position state information corresponding to the autonomous mobile robot.

[0107] In one example, the state space function includes a mapping function between the robot orientation and obstacles and a mapping function between radar measurement information and real measurement information.

[0108] In one example, the expression of the mapping function between the robot orientation and the obstacle is:

[0109]

[0110] in, represents the distance between the robot and the target point at time t, y g Indicates the distance between the target point and the y-axis. represents the estimated distance of the robot from the y-axis, x g Indicates the distance between the target point and the x-axis. represents the estimated distance between the robot and the x-axis, math.atan2() represents the inverse tangent function, It represents the angle between the autonomous mobile robot and the line connecting the target point. Indicates the angle between the robot and the obstacle. Indicates the deviation between the robot and the target point in the x-axis direction. Indicates the deviation between the robot and the target point in the y-axis direction. Indicates the deviation value of the angle between the robot and the y-axis. Indicates the actual angle between the robot and the y-axis, Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, Represents the estimated angle of the robot with the y-axis.

[0111] In one example, the expression of the mapping function between radar measurement information and real measurement information is:

[0112]

[0113] in, represents the estimated distance between the robot and the obstacle at time t, represents the angle between the robot and the nearest endpoint of the obstacle. math.atan2() represents the inverse tangent function. Indicates the actual distance between the robot and the x-axis, represents the estimated distance of the robot from the x-axis, Indicates the actual distance between the robot and the y-axis, represents the estimated distance of the robot from the y-axis, Represents the difference between radar measurement information and real information, Indicates the actual distance between the robot and the obstacle. Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, Indicates the actual coordinate of the obstacle endpoint on the x-axis at the current moment. Indicates the actual coordinate of the obstacle endpoint on the y-axis at the current moment.

[0114] In one example, the expressions of the state space function of the deep deterministic policy gradient algorithm and the state space function of the policy learning algorithm are:

[0115]

[0116] Among them, S DDPG represents the state space function of the deep deterministic policy gradient algorithm, S DIAYN represents the state space function of the policy learning algorithm, w t-1represents the angular velocity of the robot at time t-1, v t-1 represents the linear velocity of the robot at time t-1, Represents the difference between the robot’s estimated pose and its true pose. Represents the difference between radar measurement information and true information, w t-2 represents the angular velocity at time t-2, v t-2 represents the linear velocity of the robot at time t-2, X o represents the obstacle coordinate equation.

[0117] In one example, the base reward function is expressed as:

[0118]

[0119] Among them, r(s t ,a t ) represents the reward function, represents the stage-by-stage reward function, represents the continuous reward function, T represents the total movement time of the autonomous mobile robot, represents the distance between the robot and the target point at time t, represents the angle between the autonomous mobile robot and the line connecting the target point, d ig Represents the distance between the initial position of the autonomous mobile robot relative to the world coordinate system and the target point position, d g represents the threshold distance of the target endpoint, represents the distance between the robot and the nearest endpoint of the obstacle at time t, It represents the angle between the robot and the nearest endpoint of the obstacle. It represents the difference between the estimated pose and the real pose of the autonomous mobile robot obtained by simulation. represents the difference between the simulated radar measurement information and the real information, η represents the reward factor set, η∈(η1,η2,η3,η4,η5,η6) and η i >0.

[0120] See also Figure 5 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 2 may include: at least one processor 21 , at least one network interface 24 , a user interface 23 , a memory 25 , and at least one communication bus 22 .

[0121] The communication bus 22 is used to realize the connection and communication between these components.

[0122] The user interface 23 may include a display screen (Display) and a camera (Camera), and the optional user interface 23 may also include a standard wired interface and a wireless interface.

[0123] The network interface 24 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0124] Among them, the processor 21 may include one or more processing cores. The processor 21 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 25, and calling data stored in the memory 25. Optionally, the processor 21 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 21 can integrate one or more combinations of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 21, and it can be implemented separately through a chip.

[0125] Among them, the memory 25 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 25 includes a non-transitory computer-readable storage medium. The memory 25 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 25 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 25 may also be optionally at least one storage device located away from the aforementioned processor 21. As Figure 5 As shown, the memory 25 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of a deep reinforcement learning path planning method based on an improvement of a reward function.

[0126] exist Figure 5 In the electronic device 2 shown, the user interface 23 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 21 can be used to call the application program of the deep reinforcement learning path planning method improved based on the reward function stored in the memory 25. When executed by one or more processors, the electronic device executes one or more methods in the above-mentioned embodiments.

[0127] A computer-readable storage medium stores instructions, which, when executed by one or more processors, enable the computer to execute one or more methods in the above-mentioned embodiments.

[0128] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0129] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A deep reinforcement learning path planning method based on reward function improvement, characterized in that: The method comprises: S1, constructing a deep reinforcement learning training environment model, and initializing the network of the deep reinforcement learning training environment model based on a deep deterministic policy gradient algorithm and a policy learning algorithm; S2, collecting feasible state information and position state information of an autonomous mobile robot set in a deep reinforcement learning training environment model, wherein the position state information includes an initial position of the autonomous mobile robot and an initial position of a target point; S3, based on the feasible state information and the position state information, construct a state space model corresponding to the deep deterministic policy gradient algorithm and the policy learning algorithm, and make the state space model output a state space function and an action space function; The state space function includes a mapping function between the robot orientation and obstacles and a mapping function between radar measurement information and real measurement information; The expression of the mapping function between the robot orientation and the obstacle is: in, represents the distance between the robot and the target point at time t, y g Indicates the distance between the target point and the y-axis. represents the estimated distance of the robot from the y-axis, x g Indicates the distance between the target point and the x-axis. represents the estimated distance between the robot and the x-axis, math.atan2() represents the inverse tangent function, It represents the angle between the autonomous mobile robot and the line connecting the target point. Indicates the angle between the robot and the obstacle. Indicates the deviation between the robot and the target point in the x-axis direction. Indicates the deviation between the robot and the target point in the y-axis direction. Indicates the deviation value of the angle between the robot and the y-axis. Indicates the actual angle between the robot and the y-axis, Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, represents the estimated angle between the robot and the y-axis; S4, inputting the state space function and the action space function into the deep reinforcement learning training environment model to obtain reward functions corresponding to all optional state actions of the autonomous mobile robot; S5, storing the state space function, the action space function and the reward function into an experience pool, and selecting samples from the experience pool to update the deep deterministic policy gradient algorithm and the policy learning algorithm; S6, repeating steps S3-S5 until the autonomous mobile robot moves to the target point to complete the path planning.

2. The method according to claim 1, characterized in that The collection of feasible state information and position state information of the autonomous mobile robot set in the deep reinforcement learning training environment model specifically includes: Measuring the autonomous mobile robot's own position and posture through an odometer inside the autonomous mobile robot, and setting the autonomous mobile robot's initial position in the world coordinate system; Measuring the distance and deflection angle of the autonomous mobile robot relative to the initial position and posture to obtain the position and posture of the autonomous mobile robot relative to the autonomous mobile robot coordinate system at the next moment; The estimated position and posture of the autonomous mobile robot in the world coordinate system at any time is calculated to obtain the feasible state information and position state information corresponding to the autonomous mobile robot.

3. The method according to claim 1, characterized in that The expression of the mapping function between the radar measurement information and the real measurement information is: in, represents the estimated distance between the robot and the obstacle at time t, represents the angle between the robot and the nearest endpoint of the obstacle. math.atan2() represents the inverse tangent function. Indicates the actual distance between the robot and the x-axis, represents the estimated distance of the robot from the x-axis, Indicates the actual distance between the robot and the y-axis, represents the estimated distance of the robot from the y-axis, Represents the difference between radar measurement information and real information, Indicates the actual distance between the robot and the obstacle. Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, Indicates the actual coordinate of the obstacle endpoint on the x-axis at the current moment. Indicates the actual coordinate of the obstacle endpoint on the y-axis at the current moment.

4. The method according to claim 3, characterized in that The expressions of the state space function of the deep deterministic policy gradient algorithm and the state space function of the policy learning algorithm are: Among them, S DDPG represents the state space function of the deep deterministic policy gradient algorithm, S DIAYN represents the state space function of the policy learning algorithm, w t-1 represents the angular velocity of the robot at time t-1, v t-1 represents the linear velocity of the robot at time t-1, Represents the difference between the robot’s estimated pose and its true pose. Represents the difference between radar measurement information and real information, w t-2 represents the angular velocity at time t-2, v t-2 represents the linear velocity of the robot at time t-2, X o represents the obstacle coordinate equation.

5. The method according to claim 1, characterized in that The expression of the reward function is: Among them, r(s t ,a t ) represents the reward function, represents the stage-by-stage reward function, represents the continuous reward function, T represents the total movement time of the autonomous mobile robot, represents the distance between the robot and the target point at time t, represents the angle between the autonomous mobile robot and the line connecting the target point, d ig Represents the distance between the initial position of the autonomous mobile robot relative to the world coordinate system and the target point position, d g represents the threshold distance of the target endpoint, represents the distance between the robot and the nearest endpoint of the obstacle at time t, It represents the angle between the robot and the nearest endpoint of the obstacle. It represents the difference between the estimated pose and the real pose of the autonomous mobile robot obtained by simulation. represents the difference between the simulated radar measurement information and the real information, η represents the reward factor set, η∈(η1,η2,η3,η4,η5,η6) and η i >0.

6. A deep reinforcement learning path planning system based on reward function improvement, characterized in that: The deep reinforcement learning path planning system (1) comprises a data acquisition module (11), a strategy optimization module (12) and a path planning module (13), wherein: The data acquisition module (11) is used to construct a deep reinforcement learning training environment model, perform network initialization on the deep reinforcement learning training environment model based on a deep deterministic policy gradient algorithm and a policy learning algorithm, and collect feasible state information and position state information of an autonomous mobile robot set in the deep reinforcement learning training environment model, wherein the position state information includes an initial position of the autonomous mobile robot and an initial position of a target point; The policy optimization module (12) is used to construct a state space model corresponding to the deep deterministic policy gradient algorithm and the policy learning algorithm based on the feasible state information and the position state information, and make the state space model output a state space function and an action space function, wherein the state space function includes a mapping function between the robot orientation and the obstacle and a mapping function between the radar measurement information and the real measurement information; The expression of the mapping function between the robot orientation and the obstacle is: in, represents the distance between the robot and the target point at time t, y g Indicates the distance between the target point and the y-axis. represents the estimated distance of the robot from the y-axis, x g Indicates the distance between the target point and the x-axis. represents the estimated distance between the robot and the x-axis, math.atan2() represents the inverse tangent function, It represents the angle between the autonomous mobile robot and the line connecting the target point. Indicates the angle between the robot and the obstacle. Indicates the deviation between the robot and the target point in the x-axis direction. Indicates the deviation between the robot and the target point in the y-axis direction. Indicates the deviation value of the angle between the robot and the y-axis. Indicates the actual angle between the robot and the y-axis, Indicates the estimated distance between the robot and the target point in the x-axis direction. Indicates the estimated distance between the robot and the target point in the y-axis direction, Represents the estimated angle between the robot and the y-axis; inputs the state space function and the action space function into the deep reinforcement learning training environment model, obtains the reward function corresponding to all optional state actions of the autonomous mobile robot, stores the state space function, the action space function and the reward function into the experience pool, and selects samples from the experience pool to update the deep deterministic policy gradient algorithm and the policy learning algorithm; The path planning module (13) is used to repeat the optimization steps in the strategy optimization module (12) until the autonomous mobile robot moves to the target point, thereby completing the path planning.

7. An electronic device, characterized in that: The electronic device (2) comprises a processor (21), a memory (25), a user interface (23) and a network interface (24), wherein the memory (25) is used to store instructions, the user interface (23) and the network interface (24) are used to communicate with other devices, and the processor (21) is used to execute the instructions stored in the memory (25) so that the electronic device (2) executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the deep reinforcement learning path planning method based on reward function improvement according to any one of claims 1 to 5 are implemented.

Images