Multi-objective reinforcement learning navigation method for crowd-aware robots

Abstract

The application discloses a multi-target reinforcement learning navigation method for crowd-aware robots. Firstly, multi-target reinforcement learning parameters are initialized. Then, state information of an agent under a current environment is collected, and data is correspondingly preprocessed. Secondly, the processed state information of the agent is sent into a value network to output a decision action of the agent, the agent is evaluated according to cumulative rewards calculated according to different target reward functions, and value network parameter updating is performed by using a multi-target reinforcement learning algorithm. Finally, whether the maximum training number is reached is judged, and good navigation performance is obtained through continuous trial and error learning. The application proposes an end-to-end robot navigation framework, realizes trade-off among multiple targets under dynamic weights, and improves navigation performance of the robot in a dense crowd place.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mobile robot path planning and trajectory tracking, in particular to a multi-objective reinforcement learning navigation method for crowd-aware robots. BACKGROUND

[0002] In recent years, with the development of science and technology, the application scenarios of mobile robots are becoming more and more extensive and complex. For example, during the COVID-19 pandemic, mobile robots played an important role in sampling, disinfection and clinical care, significantly improving efficiency and preventing disease transmission. In addition to this, mobile robots are also applied in shopping malls, airport terminals or supermarkets, etc. These are all crowded public places, which pose higher challenges to the navigation performance of mobile robots.

[0003] So far, there are many reinforcement learning-based robot path planning and trajectory tracking methods. When considering dense crowds, crowd-aware navigation tasks usually involve different conflicting objectives, but most methods convert multiple objectives into one objective for optimization, without considering the trade-off problem for different objectives, which cannot better optimize the navigation strategy. There are few existing crowd-aware robot navigation methods for multi-objective reinforcement learning. SUMMARY

[0004] Technical problem: In order to overcome the above technical problems, a multi-objective reinforcement learning navigation method for crowd-aware robots is proposed. This method is an end-to-end robot navigation framework that realizes the trade-off between multiple objectives under dynamic weights, improving the navigation performance of robots in crowded places.

[0005] Technical solution: The technical solution adopted by the present application is: a multi-objective reinforcement learning navigation method for crowd-aware robots, comprising the following steps:

[0006] (1) initialize multi-objective reinforcement learning parameters;

[0007] (2) collect the state information of the agent under the current environment, and perform corresponding preprocessing on the data;

[0008] (3) send the processed state information of the agent to the value network to output the decision action of the agent, calculate the cumulative reward according to different objective reward functions to evaluate the agent, and update the value network parameters using a multi-objective reinforcement learning algorithm;

[0009] (4) determine whether the maximum number of training times is reached, if not, return to step (2)

[0010] Further, the initialization parameters of the multi-objective reinforcement learning in step (1) are: value evaluation network Q and value target network Q', experience replay pool D τ and weight sampling distribution D ω ;

[0011] Further, the state information and data preprocessing method of the agent in step (2) is as follows:

[0012] The state information and data preprocessing method of the agent in step (2) is as follows:

[0013] (21) The state information of the agent includes spatial information, time information and information related to the target position, wherein the spatial information of the agent is measured by dense radar beams, and there are 200 components from 0 to 2π, represented as The time information of the agent is obtained by subtracting the spatial information of the adjacent two radar acquisitions, represented as s z = [s' o -s o ]; For the path planning method, the information related to the target position of the agent is represented as s g = [d x ,d y ], which is the distance vector between the agent and the target position; for the trajectory tracking method, the information related to the target position of the agent is represented as d e is the deviation distance of the robot relative to the guide trajectory, is the derivative of the deviation distance of the robot relative to the guide trajectory, is the angle difference between the heading angle of the robot and the guide trajectory, is the derivative of the angle difference between the heading angle of the robot and the guide trajectory, v a is the effective speed of the robot, v c is the vertical speed of the robot, d g is the Euclidean distance between the robot and the target position;

[0014] (22) The data preprocessing method of normalization converts all state information data to the value range of [0, 1], that is,

[0015] In step (3), the decision action of the robot output by the value network, the design of the reward function and the update method of the multi-objective reinforcement learning are as follows:

[0016] (31) The decision action of the robot output by the value network is represented as:

[0017] a = arg max α ωT Q(s, a, ω)

[0018] where ω is the dynamic weight for different objectives, and T denotes the transpose of ω, ω is denoted as ω for path planning method pp ω is denoted as ω for trajectory tracking method pt Q is the value network of reinforcement learning method, s is the current state information of the agent, and a is the current decision action of the agent

[0019] (32) For path planning method, in order to meet the two objectives of avoiding obstacles and reaching the end point, the reward function r of path planning method pp is divided into three parts: pedestrian collision penalty r p , static obstacle collision penalty r s and completion reward r g , that is ω pp is the dynamic weight for different objectives in path planning method, the pedestrian collision penalty is represented as the closer to the pedestrian, the greater the penalty, which is specifically:

[0020]

[0021] wherein and are negative constants, d p is the distance between the robot and the nearest pedestrian, and the pedestrian collision penalty is only considered when the distance between the robot and the pedestrian is within d p,min and d p,max , d p,min and d p,max are set constants

[0022] The static collision penalty is represented as when the distance between the robot and the static obstacle such as wall, shelf, etc. is less than d s,min , it is considered to collide, and the static collision penalty at this time is:

[0023]

[0024] wherein is a negative constant, d s is the distance between the robot and the nearest static obstacle, and d s,min is the distance threshold considering the static collision penalty

[0025] The completion reward is represented as the closer to the target position, the greater the reward, which is specifically:

[0026]

[0027] wherein is a normal number, and is a negative constant, d g is the Euclidean distance between the robot and the target position, d g is the difference of the Euclidean distance between the robot and the target position between two adjacent detections, d g,r is a threshold value considering the robot reaching the target position;

[0028] (33) For the trajectory tracking method, in order to meet the two goals of obstacle avoidance and trajectory following, the reward function r pt of the trajectory tracking method is divided into three parts: the pedestrian collision penalty r p , the static obstacle collision penalty r s and the trajectory deviation penalty r f , i.e. ω pt is the dynamic weight for different targets in the trajectory tracking method, where the pedestrian collision penalty and the static obstacle collision penalty are the same as in the path planning method; the trajectory deviation penalty is represented as the farther the distance from the guide trajectory, the greater the penalty, which is specifically:

[0029]

[0030] wherein and are negative constants, d e,r is the maximum distance allowed for the robot to deviate from the guide trajectory.

[0031] (34) The loss function L used for parameter updating of the multi-objective reinforcement learning method is:

[0032]

[0033] wherein K is the number of experience samples for updating, s j is the state information of the robot in the sample j, a j is the decision action of the robot in the sample, y j is the target update value of the Q network in the sample j, y j is calculated in the following manner:

[0034]

[0035] wherein γ is the discount factor, for the path planning method, r j = r pp , for the trajectory tracking method, r j = r pt .

[0036] Further, in step (4), a termination judgment is performed on the algorithm, if the maximum number of training is reached, the training is stopped, otherwise, the step (2) is returned.

[0037] Beneficial effects: compared with the prior art, the technical scheme of the application has the following beneficial technical effects:

[0038] (1) The robot collects spatial information, time information and information related to the target position, perceives the surrounding dynamics through the fusion of spatial information and time information, can effectively detect the adjustment of crowd movement and avoid collision.

[0039] (2) In order to avoid the robot falling into local optimum, the application designs a heuristic reward function to provide intensive reward and ensure the continuity of the learning process, which can make the robot more safely avoid obstacles and more efficiently complete the target.

[0040] (3) In order to meet multiple conflicting goals, the application proposes a navigation strategy based on multi-objective reinforcement learning, which can output the optimal strategy corresponding to different target weights by weighing multiple targets, effectively improving the universality of the navigation method.

[0041] (4) The application fits the multi-objective reinforcement learning algorithm through the neural network, learns through the neural network, obtains the correction value, improves the accuracy, reliability and operation speed of the correction value and the method of the application. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is the flow chart of the method of the application.

[0043] Figure 2 is the state representation diagram of the agent related to the target position in the trajectory tracking method. DETAILED DESCRIPTION

[0044] The application will be further described below in combination with the drawings.

[0045] The application proposes a crowd perception robot path planning and trajectory tracking method based on multi-objective reinforcement learning, and its flow chart is shown in Figure 1 The specific steps of the algorithm are as follows:

[0046] 1、Initialize multi-objective reinforcement learning parameters, including value evaluation network Q and value target network Q', experience replay pool D τ And weight sampling distribution D ω ;

[0047] 2、Collect the state information of the agent in the current environment, including spatial information, time information and information related to the target position. Among them, the spatial information of the agent is measured by dense radar beams, and there are 200 components from 0 to 2π, which are represented as The time information of the agent is obtained by subtracting the spatial information of two adjacent radar acquisitions, denoted as s z o o For the path planning method, the information related to the target position of the agent is denoted as s g x y , i.e., the distance vector of the agent and the target position; for the trajectory tracking method, the information related to the target position of the agent is denoted as e is the deviation distance of the robot relative to the guide trajectory, is the angle difference between the heading angle of the robot and the angle of the guide trajectory, v a is the effective sailing speed of the robot, v c is the vertical sailing speed of the robot, d g is the Euclidean distance between the robot and the target position. The data is normalized and preprocessed accordingly, and all state data is converted to a value range of [0, 1].

[0048] 3. The processed state information of the agent is sent to the value network to output the decision action of the agent, and the reward function for different targets is calculated.

[0049] The decision action of the robot output by the value network is denoted as:

[0050] a = arg max α ω T Q(s, a, ω)

[0051] where ω is the dynamic weight for different targets, denoted as ω pp for the path planning method, and ω pt for the trajectory tracking method, and Q is the value network of the reinforcement learning method.

[0052] For the path planning method, in order to meet the two goals of avoiding obstacles and reaching the end point, the reward function is divided into three parts: pedestrian collision penalty, static obstacle collision penalty and completion reward, i.e. The pedestrian collision penalty is represented as the closer to the pedestrian, the greater the penalty, which is specifically:

[0053]

[0054] where and are negative constants, d p is the distance between the robot and the nearest pedestrian, and the pedestrian collision penalty is only considered when the distance between the robot and the pedestrian is within d p,min and d p,max , d p,min ​​​​​and d p,max is a constant set;

[0055] Static collision penalty is represented as when the distance between robot and static obstacle such as wall, shelf, etc. is less than a certain range, it is considered to have collision, at this time the collision penalty is:

[0056]

[0057] wherein, is a negative constant, d p is the distance between robot and the nearest static obstacle, d s,min is the distance threshold considering static collision penalty;

[0058] The completion reward is represented as the closer to the target position, the greater the reward, specifically:

[0059]

[0060] wherein, is a normal number, and is a negative constant, d g is the Euclidean distance between robot and target position, Δd g is the difference between the Euclidean distance of adjacent two detections and target position, d g,r is the threshold considering robot reaching the target position.

[0061] For the trajectory tracking method, in order to meet the two goals of avoiding obstacles and trajectory following, the reward function is divided into three parts: pedestrian collision penalty, static obstacle collision penalty and trajectory deviation penalty, that is, ω pt is the dynamic weight for different targets. The pedestrian collision penalty and the static obstacle collision penalty are the same as in the path planning method. The trajectory deviation penalty is represented as the farther from the guide trajectory, the greater the penalty, specifically:

[0062]

[0063] wherein and is a negative constant, d e,r is the maximum distance allowed for robot to deviate from the guide trajectory.

[0064] 1. According to the cumulative reward calculated by the reward function of different targets, the agent is evaluated, and the value network parameter is updated by using the multi-objective reinforcement learning algorithm, and the loss function for parameter update is:

[0065]

[0066] wherein, K is the number of experience samples for updating, yij The calculation method is as follows:

[0067]

[0068] Wherein, γ is a discount factor, for the path planning method, r j = r pp , for the trajectory tracking method, r j = r pt

[0069] 2, termination judgment is performed on the algorithm, if the maximum training number is reached, the training is stopped, otherwise, step (2) is returned. Thus, the optimal navigation strategy is obtained through continuous trial and error.

[0070] The above only describes the preferred embodiments of the present application, and it should be noted that for ordinary skilled persons in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A multi-object reinforcement learning navigation method for crowd-aware robots, characterized in that, The method comprises the following steps: (1) initializing multi-objective reinforcement learning parameters; (2) collecting state information of an agent under a current environment, and performing corresponding preprocessing on the data; (3) sending the preprocessed state information of the agent into a value network to output a decision action of the agent, evaluating the agent according to different objective reward functions to calculate cumulative rewards, and updating parameters of the value network by using a multi-objective reinforcement learning algorithm; (4) determining whether the maximum number of training times is reached, and returning to step (2) if not; The initialization parameters of the multi-objective reinforcement learning in step (1) are: a value evaluation network Q and a value target network Q', an experience replay pool D τ and a weight sampling distribution D ω ; In step (2), the state information of the agent and the data preprocessing method are as follows: (21) The state information of the agent includes spatial information, time information and information related to the target position, wherein the spatial information of the agent is measured by dense radar beams, and there are 200 components from 0 to 2π, represented as The time information of the agent is obtained by subtracting the spatial information of two adjacent radar acquisitions, represented as s z = [s' o -s o ]; for the path planning method, the information related to the target position of the agent is represented as s g = [d x ,d y ], that is, the distance vector of the agent and the target position; for the trajectory tracking method, the information related to the target position of the agent is represented as d e is the deviation distance of the robot relative to the guide trajectory, is the derivative of the deviation distance of the robot relative to the guide trajectory, is the angle difference between the heading angle of the robot and the guide trajectory, is the derivative of the angle difference between the heading angle of the robot and the guide trajectory, v a is the effective sailing speed of the robot, v c is the vertical sailing speed of the robot, d g is the Euclidean distance between the robot and the target position; (22) The preprocessing method of normalizing data, converting all state information data to the value range of [0, 1], that is In step (3), the decision action of the robot output by the value network, the design of the reward function, and the update mode of the multi-objective reinforcement learning are as follows: (31) the decision action of the robot output by the value network is represented as: wherein ω is a dynamic weight for different objectives, the superscript T represents the transpose of ω, and ω is denoted as ω for the path planning method pp ω is denoted as ω for the trajectory tracking method pt Q is a value network of the reinforcement learning method, s is the current state information of the agent, and a is the current decision action of the agent (32) For the path planning method, in order to meet the two goals of avoiding obstacles and reaching the end point, the reward function r pp is divided into three parts: pedestrian collision penalty r p , static obstacle collision penalty r s and completion reward r g , that is ω pp is the dynamic weight for different goals in the path planning method, and the pedestrian collision penalty is represented as the closer to the pedestrian, the greater the penalty, which is specifically: wherein, and is a negative constant, d p is the distance between the robot and the closest pedestrian, the pedestrian collision penalty is only considered when the distance between the robot and the pedestrian is within d p,min and d p,max are set constants; p,min and d p,max are set constants; The static collision penalty is expressed as a collision is considered to have occurred when the distance of the robot to a static obstacle is less than d s,min The static collision penalty is expressed as a collision is considered to have occurred when the distance of the robot to a static obstacle is less than d wherein, is a negative constant, d s is the distance of the robot to the closest static obstacle, d s,min is a distance threshold value taking into account the static collision penalty; The reward for completing the task is represented as the closer to the target position, the greater the reward, and specifically: wherein, is a positive constant, and is a negative constant, d g is the Euclidean distance of the robot from the target position, Δd g is the difference in the Euclidean distance of the robot from the target position between two consecutive detections, d g,r is a threshold value considering the robot to reach the target position; (33)For the trajectory tracking method, in order to meet the two goals of obstacle avoidance and trajectory following, the reward function r pt is divided into three parts: pedestrian collision penalty r p , static obstacle collision penalty r s and trajectory deviation penalty r f , i.e. ω pt is the dynamic weight for different goals in the trajectory tracking method, where the pedestrian collision penalty and the static obstacle collision penalty are the same as in the path planning method; the trajectory deviation penalty is represented as the farther the distance from the guide trajectory, the greater the penalty, which is specifically: wherein and is a negative constant, d e,r is the maximum distance allowed for the robot to deviate from the guiding trajectory; (34) the loss function L for parameter updating of the multi-objective reinforcement learning method is: where K is the number of experience samples for updating, s j is the state information of the robot in sample j, a j is the decision action of the robot in sample, y j is the target update value of the Q network in sample j, y j The calculation method of is as follows: where γ is a discount factor, r j = r pp for the trajectory tracking method, r j = r pt .

2. The multi-objective reinforcement learning based crowd-aware robot path planning and trajectory tracking method according to claim 1, characterized in that, In step (4), the preset stopping condition is whether the number of cycles reaches a preset cycle value, if yes, the training is stopped, otherwise, the step (2) is returned.

3. The multi-objective reinforcement learning based crowd-aware robot path planning and trajectory tracking method according to claim 1, characterized in that, The neural network is used to fit the multi-objective reinforcement learning algorithm, and the gradient update value is obtained by training the neural network.

Images