Self-adaptive correction reward shaping method for reinforcement learning scheduling of multiple storage robots

Abstract

The invention discloses a self-adaptive correction reward shaping method for reinforcement learning scheduling of multiple storage robots, and the method comprises the steps: S1, designing a task environment, setting parameters of the storage robots, a target object and an obstacle, building a robot kinematic model, and setting hyper-parameters; s2, collecting training data through interaction between the storage robot and the environment, and processing the training data to obtain a reward function; s3, defining the utilization of the reward function as a double-layer optimization problem, and obtaining a self-adaptive correction reward function through calculation; s4, a unique self-adaptive correction reward function is provided for each robot, a reward value is obtained, and the value and training data are stored in an experience playback pool; and S5, collecting data from the experience playback pool, and training the robot to avoid the obstacle and reach the optimal strategy of the target object by using a multi-agent reinforcement learning algorithm. The credit distribution problem of the multi-robot reinforcement learning algorithm in a complex environment can be solved, and the learning efficiency and the convergence speed of the reinforcement learning algorithm are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of multi-robot collaborative control, and in particular relates to an adaptive correction reward shaping method for reinforcement learning scheduling of multiple storage robots. Background Art

[0002] The rapid development of the express delivery industry has put forward requirements for the efficiency and cost control of picking, handling, and sorting of goods. Intelligent warehouse technology uses robots to replace manual labor, which can not only reduce labor costs, but also increase warehouse logistics efficiency, improve warehouse safety, and increase warehouse throughput. Intelligent warehouse systems require multiple robots to coordinate and cooperate to complete tasks, and group collaborative control is the technical core of intelligent robot group research and application. The research on intelligent collaborative control technology based on group robots has extremely important practical significance and has broad application prospects in many labor-intensive and high-risk scenarios closely related to humans, including various land and air military activities in the military field, robot group warehousing and freight in the civilian field, joint manufacturing robots in the industrial field, and robot group type applications for disaster relief.

[0003] As the complexity of robot applications continues to increase and task requirements continue to update, robots are required to adaptively collaborate in groups to complete given tasks in complex scenarios, which requires group collaboration strategies to ensure this. The complexity of group collaboration strategies in complex environments continues to increase with the dynamic collaboration model of robot groups and adaptive collaborative control methods. Due to factors such as the autonomy of individual intelligent robots, the complexity of the task domain environment, and the sophistication of the robots themselves, the collaborative intelligent control of intelligent robot groups faces many challenges in theoretical methods and implementation technologies.

[0004] Among the methods for solving robot group tasks, reinforcement learning is a major research trend at present. In multi-robot collaborative tasks, there is usually only one total team reward. Each robot trains its own strategy to maximize this team reward, but the robot cannot know its contribution to the team reward. At the same time, since the robot cannot observe the global environment, the information obtained is different, which leads to the possibility that the reward obtained by the robot is a false reward brought by the behavior of teammates. Through the reward shaping method, a reward function can be designed for each robot in the multi-robot, which can make the robot strategy training more efficient and achieve better results in maximizing the total reward return. The earliest reward shaping appeared in the robot's action adjustment and the forward movement of the bicycle. Later, potential-based reward shaping appeared. These methods focus on ensuring that the optimal strategy remains unchanged, but they are not very helpful for robot training. The later intrinsic incentive reward shaping uses the information gain in the learning action as an intrinsic reward signal. The intrinsic incentive method can encourage robots to explore, but it cannot guarantee the consistency of the strategy. At the same time, this reward method does not take into account the particularity of the multi-robot scenario, and it itself also requires more training time for calculation.

[0005] In view of this, there is a need to develop an improved method to overcome these problems. Summary of the invention

[0006] In order to overcome the above technical problems, the present invention provides an adaptive modified reward shaping method for reinforcement learning scheduling of multiple warehouse robots. First, taking the two-layer optimization model as the framework, the reward given by the environment is set as an external reward, and an auxiliary reward function is added. The auxiliary reward function is composed of an intrinsic reward function, an intrinsic reward weight and a modified reward function; secondly, the incremental meta-gradient learning method is used in the calculation of gradient updates, combined with an adaptive reward function, so as to accelerate robot training; thirdly, an independent critic is set for each robot in a multi-robot environment, which guides the robot differently at each time step; finally, a multi-robot reinforcement learning training method based on reward shaping is adopted, and intelligent warehouse robot scheduling is finally realized through simulation training. In summary, the present invention solves the problems that multi-robot reinforcement learning research is difficult to train in a sparse reward environment, and that independent robots cannot obtain effective rewards in complex collaborative tasks, thereby improving the performance of warehouse robot scheduling technology.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] An adaptive modified reward shaping method for multi-warehouse robot reinforcement learning scheduling, characterized by comprising the following steps:

[0009] S1. Design the task environment, set the relevant parameters of the warehouse robot, target object and obstacle, establish the robot kinematic model, and set the hyperparameters of the multi-robot reinforcement learning algorithm;

[0010] S2, collect training data through the interaction between each warehouse robot and the environment, and obtain the reward function after standardization and normalization;

[0011] S3. Define the use of the reward function as a two-layer optimization problem, where the lower layer is the optimization strategy to maximize the overall reward, and the upper layer is the optimization parameter to maximize the expected cumulative external reward, and the adaptive modified reward function is calculated;

[0012] S4. Combine the multi-agent reinforcement learning algorithm to provide a unique adaptive modified reward function for each robot, calculate the reward value of each robot, and store the reward value and training data together in the experience replay pool;

[0013] S5. Collect a batch of data from the experience replay pool, and use a multi-agent reinforcement learning algorithm based on adaptive modified reward shaping to train the robot to obtain the optimal strategy network for multi-warehouse robot scheduling.

[0014] Furthermore, the hyperparameters include formal design parameters of the state space, action space, and reward function.

[0015] Furthermore, the step S1 includes the following steps:

[0016] Step S1.1: Design the task environment, set the map length and width, the length and width of the internal walls and the center coordinates, and the center coordinates of the shelves;

[0017] Step S1.2: Establish the robot kinematic model and set the hyperparameters of the multi-robot reinforcement learning algorithm, including:

[0018] a. Kinematic model of wheeled differential drive of storage robot;

[0019] b. The state space observed by the warehouse robot, provided by the laser radar, including the relative coordinates and movement speed of the robot;

[0020] c. The action space of the storage robot is determined by the linear velocity and angular velocity;

[0021] d. Setting of external rewards for the team in warehouse management tasks;

[0022] Step S1.3: Set the basic hyperparameters of the reinforcement learning algorithm, including at least: number of neural network layers, number of network neurons, activation function, training set size, replay pool size, learner, policy network learning rate, critic network learning rate, optimizer, optimizer parameters, discount factor, gradient intercept norm, auxiliary function network, auxiliary function network learning rate, auxiliary function network gradient intercept norm, initial shaping weight, shaping weight range, correction function weight, reward signal r reach , reward signal r crash .

[0023] Preferably, the wheel differential drive kinematic model describes the speed, steering and position of the vehicle, and is expressed as:

[0024]

[0025] Where v represents the linear velocity of the robot during movement, v l and v r They represent the speed of the left and right wheels of the robot respectively, ω represents the angular velocity, r represents the turning radius, and d represents the distance between the left and right wheels.

[0026] Preferably, the team of the warehouse management task is externally rewarded Set to:

[0027]

[0028] Where t represents time, represents the reward obtained by the robot when it arrives, It indicates the reward the robot gets when it encounters an obstacle, in the following format:

[0029]

[0030] In order to prevent the robot from hitting obstacles, a large negative reward is set when the robot collides with an obstacle. A collision occurs when the distance between the center point of the robot and the center point of the obstacle is less than or equal to the set value.

[0031] Furthermore, the step S2 comprises the following steps:

[0032] Step S2.1: At the beginning of each round, initialize the environment data, randomly generate target points, and randomly generate noise;

[0033] Step S2.2: Each robot obtains observations, inputs the observations into the policy network and adjusts the policy according to the output of the critic network in the system;

[0034] Step S2.3: Each robot performs actions according to the output of the policy function, obtains environmental rewards and the observed value of the state at the next time step, records the obtained information in the experience replay pool, and updates the network parameters in the system;

[0035] Step S2.4: Repeat steps S2.2 to S2.3 until the following events occur and the round ends: Event 1: A robot collides with an obstacle or other robots; Event 2: Each robot successfully picks up the goods and delivers them to the target point; Event 3: The time step exceeds the maximum number of round steps; After the round ends, the obtained data is output to the specified directory.

[0036] Furthermore, step S3 includes the following steps:

[0037] Step S3.1: Establish a two-level optimization framework; the state environment variables at time step t are represented by s t ∈S means that after the robot obtains the state value, it will choose an action according to its current strategy function, that is, a t , where the policy function is denoted by π; the policy π θ The value of J(π θ ) or J(θ), which represents the robot following the strategy π θ The sum of the expected cumulative discounted rewards obtained when performing the action, i.e.

[0038]

[0039] Among them, γ is the discount inductor, and T means the process of reaching another state after the robot takes a certain action through the policy function in a certain state; for the calculation of the value function, the policy gradient method is used in the two-layer optimization framework, and the specific update formula is as follows:

[0040]

[0041] in is the return value before termination;

[0042] There are two modules inside the robot, one is the policy function parameterized by θ, and the other is the intrinsic reward function parameterized by η. The robot gets a state s from the environment, updates the intrinsic reward parameter η by maximizing the extrinsic reward, and then gets the intrinsic reward r. in , and then by maximizing the intrinsic reward r in With extrinsic rewards ex The sum of is used to update the policy parameters, thus obtaining action a;

[0043] Step S3.2: Set the adaptive reward function: Set the intrinsic reward to r in , the auxiliary reward is parameterized as The two-level optimization objective is defined as follows:

[0044]

[0045] The intrinsic reward parameters are updated by maximizing the extrinsic reward, and the policy parameters are updated by maximizing the overall reward; the reward function is expressed as follows:

[0046]

[0047] Among them, r ex (s,a) is the external reward, i.e. the original reward; r in (s,a) is intrinsic reward; It is a parameterized weight function that can adaptively adjust and optimize the reward function; is the correction function, is a hyperparameter, It is a parameter function, which is used as a correction function to punish the sparse reward function.

[0048] Furthermore, the step S4 comprises the following steps:

[0049] Step S4.1: Take the real team reward as the external reward of robot i, denoted as r ex (s i ,a i ) ; Referring to the adaptive reward function form above, the auxiliary reward function is defined as The parameters of the intrinsic reward function are The state-action pair (s i ,a i ) is the input, assigning agent reward r to robot i proxy as follows:

[0050]

[0051] Among them, r in (s,a) is the intrinsic reward, is a parameterized weight function, is the correction function, given by parameterization, where λ is a hyperparameter;

[0052] Step S4.2: The external environment provides extrinsic rewards And the current state value s t , each robot chooses the action a to take according to the state and strategy parameters i,t , the action value is passed to the external environment, the state of the next step is obtained and the trajectory is formed and stored in the experience pool.

[0053] Furthermore, the step S5 comprises the following steps:

[0054] Step S5.1: After all robots have completed their actions in each round, a batch of data is randomly collected from the experience replay pool for training;

[0055] Step S5.2: Set the update method of the strategy parameters and adaptive reward parameters; create a proxy reward for each robot at time step t, and after t, the discounted proxy reward of each robot is correspondingly Defined as:

[0056]

[0057] Agent value function of robot i as follows:

[0058]

[0059] Since there are two parameters, namely the strategy parameter θ and the auxiliary reward parameter Use the two-layer optimization method to fix one value and optimize the other value. The specific two-layer optimization update target is as follows:

[0060]

[0061] Among them, the agent expected discounted return of robot i is Depends on θ i and Φ represents the auxiliary reward parameter set Θ represents the policy parameter set {θ1,θ2,...,θ n};

[0062] Step S5.3: Calculate the gradient for updating the policy network parameters; by optimizing the auxiliary function parameters To maximize By maximizing the expected discounted return J of robot i’s agent proxy To optimize the strategy parameter θ i ; By giving a strategy The generated trajectory, θ i Update via policy gradient:

[0063]

[0064] in, Is the acting critic, As the advantage function, is i Parameterized proxy value, s′ i is the next state of robot i in the trajectory; the updated policy parameter θ′ iIt can be expressed as:

[0065]

[0066] Step S5.4: Calculate the gradient for updating the adaptive reward parameters; use the two-layer alternating optimization method to calculate the adaptive reward function gradient, fix the policy parameter θ to optimize the auxiliary reward parameter Fixed auxiliary reward parameters To optimize the policy parameters θ; the expected cumulative modified reward J ex+in The gradient with respect to the policy parameters θ is of the following form:

[0067]

[0068] Among them G ex+in (s,a) is the state action value under the current strategy;

[0069] Step S5.5: Repeat steps S5.1 to S5.4 for a total of K times to end this round; repeat steps S2 to S5 until the algorithm fully converges and the optimal strategy network for multi-warehouse robot scheduling is obtained.

[0070] The beneficial effects of the present invention are as follows: The present invention proposes an adaptive corrected reward shaping method for multi-warehouse robot reinforcement learning scheduling, which can help robots learn better when rewards are sparse. The method is then integrated with multi-robot reinforcement learning to solve the credit allocation problem of multi-robot reinforcement learning algorithms in complex environments, and effectively improve the learning efficiency and convergence speed of reinforcement learning algorithms in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 The figure is a task scenario diagram of an embodiment of the present invention in a simulation environment.

[0072] Figure 2 It is the overall framework diagram of the algorithm of the present invention.

[0073] Figure 3 This is the wheel differential drive kinematic model of the robot in the present invention.

[0074] Figure 4 This is the algorithm network framework diagram of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be described more clearly and completely in conjunction with the drawings of the specification below. It should be noted that the described embodiments are part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0076] Example

[0077] like Figure 1 As shown, this embodiment uses the smart warehouse environment built in the Gazebo simulation platform to conduct multi-robot joint navigation and obstacle avoidance task experiments. The goal of the entire task is for the robot to pick up the corresponding goods and reach the corresponding target point as quickly as possible without collision. The shape of the robot can be regarded as a cylinder, and the coordinate origin is the center of the circle. Therefore, if the distance between other objects and the robot coordinate point is less than 0.02m, it is considered that a collision has occurred, and the positions of each target point and obstacle in each round change randomly. The design framework of an adaptive correction reward shaping method for reinforcement learning scheduling of multi-warehouse robots described in this embodiment is as follows Figure 2 As shown, at least the following steps are included:

[0078] Step S1 designs the task environment, sets the relevant parameters of the storage robot, target object and obstacle, establishes the robot kinematic model, sets the state space, action space, formal design parameters of the reward function and other hyper parameters of the multi-robot reinforcement learning algorithm, including:

[0079] Step S1.1: Design the task environment, set the map length and width, the length and width of the internal walls and the center coordinates, and the center coordinates of the shelves;

[0080] Step S1.2: Establish the robot kinematic model and set the hyper parameters of the multi-robot reinforcement learning algorithm, such as the state space, action space, and reward function, including:

[0081] a. The kinematic model of the wheeled differential drive of the storage robot is as follows Figure 3 As shown, the speed, steering and position of the vehicle described can be expressed as:

[0082]

[0083] Where v represents the linear velocity of the robot during movement, v l and v r Respectively represent the speed of the robot's left and right wheels, ω represents the angular velocity, r represents the turning radius, and d represents the distance between the left and right wheels;

[0084] b. The state space observed by the warehouse robot, provided by the laser radar, including the relative coordinates and movement speed of the robot;

[0085] c. The action space of the storage robot is determined by the linear velocity and angular velocity, which is discretized into 8 actions as shown in Table 1;

[0086] Table 1 Corresponding linear velocity and angular velocity of each action

[0087]

[0088] d. Team extrinsic rewards for warehouse management tasks The setup is as follows:

[0089]

[0090] Where t represents time, represents the reward obtained by the robot when it arrives, It indicates the reward the robot gets when it hits an obstacle. The specific form is as follows:

[0091]

[0092] In order to prevent the robot from hitting obstacles, a large negative reward is set when the robot collides with an obstacle. A collision occurs when the distance between the center point of the robot and the center point of the obstacle is less than or equal to the set value.

[0093] Step S1.3: As shown in Table 2 below, set the basic hyperparameters of the reinforcement learning algorithm, including: number of neural network layers, number of network neurons, activation function, training set size, replay pool size, learner, policy network learning rate, critic network learning rate, optimizer, optimizer parameters, discount factor, gradient intercept norm, auxiliary function network, auxiliary function network learning rate, auxiliary function network gradient intercept norm, initial shaping weight, shaping weight range, correction function weight, reward signal r reach , reward signal r crash wait.

[0094] Table 2 Basic hyperparameters of reinforcement learning algorithms

[0095]

[0096] In step S2, each warehouse robot interacts with the dynamic environment to collect training data, and obtains the reward function after standardization and normalization, including:

[0097] Step S2.1: At the beginning of each round, initialize the environment data, randomly generate target points, and randomly generate noise;

[0098] Step S2.2: Each robot obtains observations, inputs the observations into the policy network and adjusts the policy according to the output of the critic network in the system;

[0099] Step S2.3: Each robot performs actions according to the output of the policy function, obtains environmental rewards and the observed value of the state at the next time step, records the obtained information in the experience replay pool, and updates the network parameters in the system;

[0100] Step S2.4: Repeat steps S2.2 to S2.3 until the following events occur and the round ends: Event 1: a robot collides with an obstacle or other robots; Event 2: each robot successfully picks up the goods and delivers them to the target point; Event 3: the time step exceeds the maximum round step number. After the round ends, the obtained data is output to the specified directory.

[0101] like Figure 4 As shown, step S3 defines the use of the reward function as a two-layer optimization problem, where the lower layer is an optimization strategy to maximize the overall reward, and the upper layer is an optimization parameter to maximize the expected cumulative external reward. The adaptive modified reward function is calculated, including:

[0102] Step S3.1: Establish a two-level optimization framework. The state environment variables at time step t are represented by s t ∈S means that after the robot obtains the state value, it will choose an action according to its current strategy function, that is, a t , where the policy function is denoted by π. Policy π θ The value of J(π θ ) or J(θ), which represents the robot following the strategy π θ The sum of the expected cumulative discounted rewards obtained when performing the action, i.e.

[0103]

[0104] Among them, γ is the discount factor, and T means the process of reaching another state after the robot takes a certain action through the policy function in a certain state. For the calculation of the value function, the policy gradient method is used in the two-layer optimization framework. The specific update formula is as follows:

[0105]

[0106] in Is the return value before termination.

[0107] There are two modules inside the robot, one is the policy function parameterized by θ, and the other is the intrinsic reward function parameterized by η. The robot gets a state s from the environment, updates the intrinsic reward parameter η by maximizing the extrinsic reward, and then gets the intrinsic reward rin , and then by maximizing the intrinsic reward r in With extrinsic rewards ex The sum of is used to update the policy parameters to obtain action a.

[0108] Step S3.2: Set the adaptive reward function. Set the intrinsic reward to r in , the auxiliary reward is parameterized as The two-level optimization objective is defined as follows.

[0109]

[0110] The intrinsic reward parameters are updated by maximizing the extrinsic reward, and the policy parameters are updated by maximizing the overall reward. The reward function is expressed as follows:

[0111]

[0112] Among them, r ex (s,a) is the external reward, i.e. the original reward; r in (s,a) is intrinsic reward; It is a parameterized weight function that can adaptively adjust and optimize the reward function; is the correction function, is a hyperparameter, It is a parameter function, which is used as a correction function to punish the sparse reward function.

[0113] Step S4 combines the multi-agent reinforcement learning algorithm to provide a unique adaptive correction reward function for each robot. The reward value of each robot is calculated and stored in the experience replay pool together with the training data, including:

[0114] Step S4.1: Take the real team reward as the external reward of robot i, denoted as r ex (s i ,a i ). Referring to the adaptive reward function form above, the auxiliary reward function is defined as The parameters of the intrinsic reward function are The state-action pair (s i ,a i ) is the input, assigning agent reward r to robot i proxy as follows:

[0115]

[0116] Among them, r in (s,a) is the intrinsic reward, is a parameterized weight function, is the correction function, given by parameterized, where λ is a hyperparameter.

[0117] Step S4.2: The external environment provides extrinsic rewards And the current state value s t , each robot chooses the action a to take according to the state and strategy parameters i,t , the action value is passed to the external environment, the state of the next step is obtained and the trajectory is formed and stored in the experience pool.

[0118] Step S5 collects a batch of data from the experience replay pool, as shown in Table 3. The optimal strategy network for intelligent warehouse robot scheduling is trained using a multi-agent reinforcement learning algorithm based on adaptive modified reward shaping, including:

[0119] Step S5.1: After all robots have completed their actions in each round, a batch of data is randomly collected from the experience replay pool for training;

[0120] Step S5.2: Set the update method of the policy parameters and adaptive reward parameters. Create a proxy reward for each robot at time step t, and after t, the discounted proxy reward of each robot is Defined as:

[0121]

[0122] Agent value function of robot i as follows:

[0123]

[0124] Since there are two parameters, namely the strategy parameter θ and the auxiliary reward parameter Use the two-layer optimization method to fix one value and optimize the other value. The specific two-layer optimization update target is as follows:

[0125]

[0126] Among them, the agent expected discounted return of robot i is Depends on θ i and Φ represents the auxiliary reward parameter set Θ represents the policy parameter set {θ1,θ2,...,θ n}.

[0127] Step S5.3: Calculate the gradient for updating the policy network parameters. By optimizing the auxiliary function parameters To maximize By maximizing the expected discounted return J of robot i’s agent proxy To optimize the strategy parameter θi . By giving a strategy The generated trajectory, θ i Update via policy gradient:

[0128]

[0129] in, Is the acting critic, As the advantage function, is i Parameterized proxy value, s i ′ is the next state of robot i in the trajectory. The updated policy parameter θ i ′ can be expressed as:

[0130]

[0131] Step S5.4: Calculate the gradient for updating the adaptive reward parameters. Use the two-layer alternating optimization method to calculate the adaptive reward function gradient, fix the policy parameter θ to optimize the auxiliary reward parameter Fixed auxiliary reward parameters To optimize the policy parameters θ. The expected cumulative modified reward J ex+in The gradient with respect to the policy parameters θ is of the following form:

[0132]

[0133] Among them G ex+in (s,a) is the state-action value under the current strategy.

[0134] Step S5.5: Repeat steps S5.1 to S5.4 for K times to end this round. Repeat steps S2 to S5 until the algorithm fully converges and the optimal strategy network for multi-warehouse robot scheduling is obtained.

[0135] Table 3 Algorithm 4.1 Multi-agent reinforcement learning algorithm flow based on adaptive correction reward shaping

[0136]

[0137]

[0138] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0139] The above description is only a preferred embodiment of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. An adaptively modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots, Features The following steps are involved: S1. Design the task environment, set the relevant parameters of the warehouse robot, target object and obstacle, establish the robot kinematic model, and set the hyperparameters of the multi-robot reinforcement learning algorithm; S2, collect training data through the interaction between each warehouse robot and the environment, and obtain the reward function after standardization and normalization; S3. Define the use of the reward function as a two-layer optimization problem, where the lower layer is the optimization strategy to maximize the overall reward, and the upper layer is the optimization parameter to maximize the expected cumulative external reward, and the adaptive modified reward function is calculated; S4. Combine the multi-agent reinforcement learning algorithm to provide a unique adaptive modified reward function for each robot, calculate the reward value of each robot, and store the reward value and training data together in the experience replay pool; S5. Collect a batch of data from the experience replay pool, and use a multi-agent reinforcement learning algorithm based on adaptive modified reward shaping to train the robot to obtain the optimal strategy network for multi-warehouse robot scheduling.

2. The adaptive modified reward shaping method for multi-warehouse robot reinforcement learning scheduling according to claim 1, It is characterized in that The hyperparameters include state space, action space, and formal design parameters of the reward function.

3. The adaptive modified reward shaping method for multi-warehouse robot reinforcement learning scheduling according to claim 1, It is characterized in that The step S1 comprises the following steps: Step S1.1: Design the task environment, set the map length and width, the length and width of the internal walls and the center coordinates, and the center coordinates of the shelves; Step S1.2: Establish the robot kinematic model and set the hyperparameters of the multi-robot reinforcement learning algorithm, including: a. Kinematic model of wheeled differential drive of storage robot; b. The state space observed by the warehouse robot, provided by the laser radar, including the relative coordinates and movement speed of the robot; c. The action space of the storage robot is determined by the linear velocity and angular velocity; d. Setting of external rewards for the team in warehouse management tasks; Step S1.3: Set the basic hyperparameters of the reinforcement learning algorithm, including at least: number of neural network layers, number of network neurons, activation function, training set size, replay pool size, learner, policy network learning rate, critic network learning rate, optimizer, optimizer parameters, discount factor, gradient intercept norm, auxiliary function network, auxiliary function network learning rate, auxiliary function network gradient intercept norm, initial shaping weight, shaping weight range, correction function weight, reward signal r reach , reward signal r crash .

4. The adaptive modified reward shaping method for multi-warehouse robot reinforcement learning scheduling according to claim 3, It is characterized in that The wheel differential drive kinematic model describes the speed, steering and position of the vehicle and is expressed as: Where v represents the linear velocity of the robot during movement, v l and v r They represent the speed of the left and right wheels of the robot respectively, ω represents the angular velocity, r represents the turning radius, and d represents the distance between the left and right wheels.

5. The adaptive modified reward shaping method for multi-warehouse robot reinforcement learning scheduling according to claim 3, It is characterized in that External rewards for the team that performs the warehouse management task Set to: Where t represents time, represents the reward obtained by the robot when it arrives, It indicates the reward the robot gets when it encounters an obstacle, in the following format: In order to prevent the robot from hitting obstacles, a large negative reward is set when the robot collides with an obstacle. A collision occurs when the distance between the center point of the robot and the center point of the obstacle is less than or equal to the set value.

6. The adaptive modified reward shaping method for multi-warehouse robot reinforcement learning scheduling according to claim 1, It is characterized in that The step S2 comprises the following steps: Step S2.1: At the beginning of each round, initialize the environment data, randomly generate target points, and randomly generate noise; Step S2.2: Each robot obtains observations, inputs the observations into the policy network and adjusts the policy according to the output of the critic network in the system; Step S2.3: Each robot performs actions according to the output of the policy function, obtains environmental rewards and the observed value of the state at the next time step, records the obtained information in the experience replay pool, and updates the network parameters in the system; Step S2.4: Repeat steps S2.2 to S2.3 until the following events occur and the round ends: Event 1: A robot collides with an obstacle or other robots; Event 2: Each robot successfully picks up the goods and delivers them to the target point; Event 3: The time step exceeds the maximum number of round steps; After the round ends, the obtained data is output to the specified directory.

7. The adaptive modified reward shaping method for multi-warehouse robot reinforcement learning scheduling according to claim 1, It is characterized in that The step S3 comprises the following steps: Step S3.1: Establish a two-level optimization framework; the state environment variables at time step t are represented by s t ∈S means that after the robot obtains the state value, it will choose an action according to its current strategy function, that is, a t , where the policy function is denoted by π; the policy π θ The value of J(π θ ) or J(θ), which represents the robot following the strategy π θ The sum of the expected cumulative discounted rewards obtained when performing the action, i.e. Among them, γ is the discount inductor, and T means the process of reaching another state after the robot takes a certain action through the policy function in a certain state; for the calculation of the value function, the policy gradient method is used in the two-layer optimization framework, and the specific update formula is as follows: in is the return value before termination; There are two modules inside the robot, one is the policy function parameterized by θ, and the other is the intrinsic reward function parameterized by η. The robot gets a state s from the environment, updates the intrinsic reward parameter η by maximizing the extrinsic reward, and then gets the intrinsic reward r. in , and then by maximizing the intrinsic reward r in With extrinsic rewards ex The sum of is used to update the policy parameters, thus obtaining action a; Step S3.2: Set the adaptive reward function: Set the intrinsic reward to r in , the auxiliary reward is parameterized as The two-level optimization objective is defined as follows: The intrinsic reward parameters are updated by maximizing the extrinsic reward, and the policy parameters are updated by maximizing the overall reward; the reward function is expressed as follows: Among them, r ex (s,a) is the external reward, i.e. the original reward; r in (s,a) is intrinsic reward; It is a parameterized weight function that can adaptively adjust and optimize the reward function; is the correction function, is a hyperparameter, It is a parameter function, which is used as a correction function to punish the sparse reward function.

8. The adaptive modified reward shaping method for multi-warehouse robot reinforcement learning scheduling according to claim 1, It is characterized in that The step S4 comprises the following steps: Step S4.1: Take the real team reward as the external reward of robot i, denoted as r ex (s i ,a i ) ; Referring to the adaptive reward function form above, the auxiliary reward function is defined as The parameters of the intrinsic reward function are The state-action pair (s i ,a i ) is the input, assigning agent reward r to robot i proxy as follows: Among them, r in (s,a) is the intrinsic reward, is a parameterized weight function, is the correction function, given by parameterization, where λ is a hyperparameter; Step S4.2: The external environment provides extrinsic rewards And the current state value s t , each robot chooses the action a to take according to the state and strategy parameters i,t , the action value is passed to the external environment, the state of the next step is obtained and the trajectory is formed and stored in the experience pool.

9. The adaptive modified reward shaping method for multi-warehouse robot reinforcement learning scheduling according to claim 1, It is characterized in that The step S5 comprises the following steps: Step S5.1: After all robots have completed their actions in each round, a batch of data is randomly collected from the experience replay pool for training; Step S5.2: Set the update method of the strategy parameters and adaptive reward parameters; create a proxy reward for each robot at time step t, and after t, the discounted proxy reward of each robot is correspondingly Defined as: Agent value function of robot i as follows: Since there are two parameters, namely the strategy parameter θ and the auxiliary reward parameter Use the two-layer optimization method to fix one value and optimize the other value. The specific two-layer optimization update target is as follows: Among them, the agent expected discounted return of robot i is Depends on θ i and Φ represents the auxiliary reward parameter set Θ represents the policy parameter set {θ 1 ,θ 2 ,...,θ n }; Step S5.3: Calculate the gradient for updating the policy network parameters; by optimizing the auxiliary function parameters To maximize By maximizing the expected discounted return J of robot i’s agent proxy To optimize the strategy parameter θ i ; By giving a strategy The generated trajectory, θ i Update via policy gradient: in, Is a proxy critic, As the advantage function, is i Parameterized proxy value, s i ′ is the next state of robot i in the trajectory; the updated policy parameter θ i ′ It can be expressed as: Step S5.4: Calculate the gradient for updating the adaptive reward parameters; use the two-layer alternating optimization method to calculate the adaptive reward function gradient, fix the policy parameter θ to optimize the auxiliary reward parameter Fixed auxiliary reward parameters To optimize the policy parameters θ; the expected cumulative modified reward J ex+in The gradient with respect to the policy parameter θ is of the following form: Among them G ex+in (s,a) is the state action value under the current strategy; Step S5.5: Repeat steps S5.1 to S5.4 for a total of K times to end this round; repeat steps S2 to S5 until the algorithm fully converges and the optimal strategy network for multi-warehouse robot scheduling is obtained.

Images