Adaptive correction reward shaping method for multi-warehouse robot reinforcement learning scheduling

Abstract

This invention discloses an adaptive modified reward shaping method for reinforcement learning scheduling of multiple warehouse robots, comprising: S1: designing the task environment, setting parameters for the warehouse robots, target objects, and obstacles, establishing a robot kinematic model, and setting hyperparameters; S2: collecting training data through interaction between the warehouse robots and the environment, and obtaining a reward function after processing; S3: defining the utilization of the reward function as a bi-level optimization problem, and calculating an adaptive modified reward function; S4: providing a unique adaptive modified reward function for each robot, obtaining a reward value, which, along with the training data, is stored in an experience replay pool; S5: collecting data from the experience replay pool, and using a multi-agent reinforcement learning algorithm to train the robots to avoid obstacles and reach the target object using the optimal strategy. This invention can solve the credit allocation problem of multi-robot reinforcement learning algorithms in complex environments, effectively improving the learning efficiency and convergence speed of reinforcement learning algorithms.

Description

Technical Field

[0001] This invention belongs to the field of multi-robot collaborative control, specifically relating to an adaptive correction reward shaping method for reinforcement learning scheduling of multiple warehouse robots. Background Technology

[0002] The rapid development of the express delivery industry has placed demands on the efficiency and cost control of cargo picking, handling, and sorting. Smart warehouse technology uses robots to replace manual labor, which not only reduces labor costs but also increases warehouse logistics efficiency, improves warehouse security, and increases throughput. Smart warehouse systems require multiple robots to coordinate and cooperate to complete tasks; swarm collaborative control is the core technology for the research and application of intelligent robot swarms. Research on intelligent collaborative control technology based on swarm robots has extremely important practical significance and broad application prospects in many labor-intensive and high-risk scenarios closely related to human life, including various land and air military activities in the military field, robot swarm warehousing and freight transportation in the civilian field, collaborative manufacturing robots in the industrial field, and robot swarm applications in disaster relief and other areas.

[0003] As the complexity of robot applications continues to increase and task requirements constantly evolve, complex scenarios demand that robots adaptively collaborate in swarms to complete predetermined tasks, thus necessitating robust swarm collaboration strategies. The complexity of swarm collaboration strategies in complex environments increases with the dynamic collaboration model and adaptive collaborative control methods of the robot swarm. Due to factors such as the autonomy of individual intelligent robots, the complexity of the task domain environment, and the inherent sophistication of the robots themselves, the collaborative intelligent control of intelligent robot swarms faces numerous challenges in both theoretical methods and implementation technologies.

[0004] Reinforcement learning is a major research trend in solving swarm robot tasks. In multi-robot collaborative tasks, there is usually only one overall team reward. Each robot trains its own policy to maximize this reward, but the robots cannot know their own contribution to the team reward. Furthermore, because robots cannot observe the global environment, the information they receive is inconsistent, leading to the possibility that a robot's reward is a false reward resulting from the actions of its teammates. Reward shaping allows for the design of reward functions for each robot in a multi-robot scenario, resulting in faster policy training and better performance in maximizing the overall reward. The earliest reward shaping appeared in robot motion adjustments and bicycle movement. Later, potential energy-based reward shaping emerged. These methods focus on maintaining the optimal policy, but they don't significantly aid in robot training. Intrinsic incentive reward shaping, which uses the information gain from learned actions as an intrinsic reward signal, is the next step. Intrinsic incentive methods encourage robots to explore, but they cannot guarantee policy consistency. Moreover, this reward method does not consider the specificities of multi-robot scenarios and requires more training time for computation.

[0005] Therefore, there is a need to develop an improved method to overcome these problems. Summary of the Invention

[0006] To overcome the aforementioned technical problems, this invention provides an adaptive modified reward shaping method for reinforcement learning scheduling of multiple warehouse robots. First, using a two-layer optimization model as a framework, the reward provided by the environment is set as the extrinsic reward, and an auxiliary reward function is added. This auxiliary reward function consists of an intrinsic reward function, intrinsic reward weights, and a modified reward function. Second, incremental meta-gradient learning is used in the gradient update calculation, combined with the adaptive reward function, thereby accelerating robot training. Third, for multi-robot environments, an independent commentator is assigned to each robot, providing different guidance at each time step. Finally, a multi-robot reinforcement learning training method based on reward shaping is employed, and intelligent warehouse robot scheduling is ultimately achieved through simulation training. In summary, this invention solves the problems of training difficulties in sparse reward environments and the inability of independent robots to obtain effective rewards in complex collaborative tasks in multi-robot reinforcement learning research, thus improving the performance of warehouse robot scheduling technology.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] An adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots is characterized by the following steps:

[0009] S1. Design the task environment, set the relevant parameters of the warehouse robot, target object and obstacles, establish the robot kinematic model, and set the hyperparameters of the multi-robot reinforcement learning algorithm;

[0010] S2. Through the interaction between each warehouse robot and the environment, training data is collected, and the reward function is obtained after standardization and normalization.

[0011] S3. Define the utilization of the reward function as a two-level optimization problem, where the lower level is the optimization strategy to maximize the overall reward, and the upper level is the optimization parameters to maximize the expected cumulative external reward. The adaptive correction reward function is obtained by calculation.

[0012] S4. Combining a multi-agent reinforcement learning algorithm, a unique adaptive correction reward function is provided for each robot. The reward value of each robot is calculated and stored together with the training data in the experience replay pool.

[0013] S5. Collect a batch of data from the experience replay pool, and use a multi-agent reinforcement learning algorithm based on adaptive correction reward shaping to train the robot and obtain the optimal policy network for scheduling multi-warehouse robots.

[0014] Furthermore, the hyperparameters include formal design parameters for the state space, action space, and reward function.

[0015] Further, step S1 includes the following steps:

[0016] Step S1.1: Design the task environment, set the map length and width, the length and width of the internal walls and their center coordinates, and the center coordinates of the shelves;

[0017] Step S1.2: Establish the robot's kinematic model and set the hyperparameters of the multi-robot reinforcement learning algorithm, including:

[0018] a. Wheeled differential drive kinematic model of a warehouse robot;

[0019] b. The state space observed by the warehouse robot is provided by the LiDAR, which includes the robot's relative coordinates and movement speed;

[0020] c. The motion space of a warehouse robot is determined by its linear velocity and angular velocity;

[0021] d. Setting up external team rewards for warehouse management tasks;

[0022] Step S1.3: Set the basic hyperparameters of the reinforcement learning algorithm, including at least: number of neural network layers, number of network neurons, activation function, training set size, replay pool size, learner, policy network learning rate, critic network learning rate, optimizer, optimizer parameters, discount factor, gradient intercept norm, auxiliary function network, auxiliary function network learning rate, auxiliary function network gradient intercept norm, initial shaping weights, shaping weight range, correction function weights, and reward signal r. reach Reward signal r crash .

[0023] Preferably, the wheeled differential drive kinematic model describes the vehicle's speed, steering, and position, and is expressed as follows:

[0024]

[0025] Where v represents the linear velocity of the robot during its movement, v l and v r ω represents the speed of the robot's left and right wheels, r represents the turning radius, and d represents the distance between the left and right wheels.

[0026] Preferably, the team's external rewards for the warehouse management task are... Set to:

[0027]

[0028] Where t represents time. This indicates the reward the robot receives upon arrival. The reward a robot receives when it encounters an obstacle is represented in the following format:

[0029]

[0030] To prevent the robot from hitting obstacles, a larger negative reward is set when the robot collides with an obstacle. A collision is defined as the distance between the robot's center point and the center point of the obstacle being less than or equal to a set value.

[0031] Further, step S2 includes the following steps:

[0032] Step S2.1: At the start of each round, initialize environmental data, randomly generate target points, and randomly generate noise;

[0033] Step S2.2: Each robot obtains observations, inputs the observations into the policy network, and adjusts the policy based on the output of the critic network in the system;

[0034] Step S2.3: Each robot executes an action according to the policy function output, obtains environmental rewards and observations of the state at the next time step, records the obtained information in the experience replay pool, and updates the network parameters in the system;

[0035] Step S2.4: Repeat steps S2.2 to S2.3 until the following events occur to end the round: Event 1: A robot collides with an obstacle or another robot; Event 2: Each robot successfully retrieves the goods and delivers them to the target point; Event 3: The time step exceeds the maximum number of rounds; After the round ends, output the obtained data to the specified directory.

[0036] Further, step S3 includes the following steps:

[0037] Step S3.1: Establish a two-layer optimization framework; at time step t, the state environment variables are determined by s t ∈S indicates that after the robot obtains the state value, it will choose an action based on its current policy function, i.e., a t The policy function is represented by π; policy π θ The value is determined by J(π) θ J(θ) or J(θ) represents the robot following the strategy π. θ The sum of the expected cumulative discount rewards obtained when performing an action, i.e.

[0038]

[0039] Here, γ represents the discount factor, and T signifies the process by which the robot, after taking a certain action through the policy function in a given state, reaches another state. For the calculation of the value function, the policy gradient method is used in the bi-layer optimization framework, and the specific update formula is as follows:

[0040]

[0041] in It is the return value before termination;

[0042] The robot has two internal modules: a policy function parameterized by θ and an intrinsic reward function parameterized by η. The robot receives a state s from the environment, updates the intrinsic reward parameter η by maximizing the extrinsic reward, and then obtains the intrinsic reward r. in Then, by maximizing the intrinsic reward r in External rewards r ex The sum is used to update the policy parameters, thereby obtaining action a;

[0043] Step S3.2: Set the adaptive reward function: Set the intrinsic reward to r in The auxiliary reward parameterization is The two-level optimization objective is defined as follows:

[0044]

[0045] The intrinsic reward parameter is updated by maximizing the extrinsic reward, and the policy parameter is updated by maximizing the overall reward; the reward function is expressed as follows:

[0046]

[0047] Where, r ex (s,a) represents the extrinsic reward, i.e., the original reward; r in (s,a) represents the intrinsic reward; It is a parameterized weight function that can adaptively adjust and optimize the reward function; It's a correction function, it's a hyperparameter. It is a parameterized function, used as a correction function to penalize sparse reward functions.

[0048] Further, step S4 includes the following steps:

[0049] Step S4.1: Treat the real team reward as the external reward of robot i, denoted as r. ex (s i ,a i Referring to the adaptive reward function form above, the auxiliary reward function is defined as follows: The parameters of the intrinsic reward function are State-action pair (s) of a single robot i i ,a i () is the input, and the agent reward r is assigned to robot i. proxy as follows:

[0050]

[0051] Where, r in (s,a) represents intrinsic reward. For parameterized weight functions, It is a correction function, derived from Parameterization, where λ is a hyperparameter;

[0052] Step S4.2: External environment provides external rewards and the current state value s t Each robot selects its action a based on its state and strategy parameters. i,t The action value is transmitted to the external environment, the next state is obtained, and the trajectory is stored in the experience pool.

[0053] Further, step S5 includes the following steps:

[0054] Step S5.1: After all robots have completed their actions in each round, randomly collect a batch of data from the experience replay pool for training;

[0055] Step S5.2: Set the update method for strategy parameters and adaptive reward parameters; create an agent reward for each robot at time step t, and after t, accordingly reduce the discounted agent reward for each robot. Defined as:

[0056]

[0057] Agent value function of robot i as follows:

[0058]

[0059] Since there are two parameters, namely the policy parameter θ and the auxiliary reward parameter... A two-level optimization method is used, fixing one value while optimizing the other; the specific two-level optimization update objective is shown below:

[0060]

[0061] Among them, the expected discount return for robot i's agent Depends on θ i and Φ represents the auxiliary reward parameter set. Θ represents the policy parameter set {θ1, θ2, ..., θ n};

[0062] Step S5.3: Calculate the gradient used to update the policy network parameters; optimize the auxiliary function parameters. To maximize By maximizing the agent's expected discount return J for robot i proxy To optimize the strategy parameter θ i ; through a given strategy The generated trajectory, θ i Update using policy gradient:

[0063]

[0064] in, He is a proxy critic, making As the advantage function, where It is composed of φ i Parameterized surrogate value, s′ i This is the next state of robot i in the trajectory; the updated policy parameter θ′ iIt can be represented as:

[0065]

[0066] Step S5.4: Calculate the gradient used to update the adaptive reward parameters; use a two-level alternating optimization method to calculate the gradient of the adaptive reward function, and fix the policy parameter θ to optimize the auxiliary reward parameters. Fixed auxiliary reward parameters To optimize the policy parameters θ; the expected cumulative correction reward J ex+in The gradient with respect to the policy parameter θ is of the following form:

[0067]

[0068] Among them G ex+in (s,a) is the state action value under the current policy;

[0069] Step S5.5: Repeat steps S5.1 to S5.4 for a total of K times to end this round; repeat steps S2 to S5 until the algorithm fully converges and the optimal strategy network for multi-warehouse robot scheduling is obtained.

[0070] The beneficial effects of this invention are as follows: This invention proposes an adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots. This method can help robots learn better when rewards are sparse. Then, this method is integrated with multi-robot reinforcement learning to solve the credit allocation problem of multi-robot reinforcement learning algorithms in complex environments, and effectively improve the learning efficiency and convergence speed of reinforcement learning algorithms in complex dynamic environments. Attached Figure Description

[0071] Figure 1 This is a task scenario diagram in a simulation environment for an embodiment of the present invention.

[0072] Figure 2 This is a diagram illustrating the overall algorithm framework of the present invention.

[0073] Figure 3 This is the wheeled differential drive kinematic model of the robot in this invention.

[0074] Figure 4 This is a diagram of the algorithm network framework in this invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described more clearly and completely below in conjunction with the accompanying drawings. It should be noted that the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0076] Example

[0077] like Figure 1 As shown, this embodiment uses a smart warehouse environment built in the Gazebo simulation platform to conduct a multi-robot joint navigation and obstacle avoidance task experiment. The goal of the entire task is for the robot to retrieve the corresponding goods and reach the corresponding target point as quickly as possible without collisions. The robot's shape can be regarded as a cylinder, with the origin of the coordinate system as the center. Therefore, if the distance between other objects and the robot's coordinate points is less than 0.02m, a collision is considered to have occurred. The positions of each target point and obstacle change randomly in each round. The adaptive correction reward shaping method design framework for reinforcement learning scheduling of multi-warehouse robots described in this embodiment is as follows: Figure 2 As shown, it includes at least the following steps:

[0078] Step S1: Design the task environment, set the relevant parameters of the warehouse robot, target object, and obstacles, establish the robot's kinematic model, and set various hyperparameters of the multi-robot reinforcement learning algorithm, including the formal design parameters of the state space, action space, and reward function.

[0079] Step S1.1: Design the task environment, set the map's length and width, the length and width of the internal walls and their center coordinates, and the center coordinates of the shelves;

[0080] Step S1.2: Establish the robot's kinematic model, and set various hyperparameters for the multi-robot reinforcement learning algorithm, including the state space, action space, and reward function form.

[0081] a. The wheeled differential drive kinematic model of the warehouse robot is as follows: Figure 3 As shown, the vehicle's speed, steering, and position can be represented as follows:

[0082]

[0083] Where v represents the linear velocity of the robot during its movement, v l and v r Let ω represent the speeds of the robot's left and right wheels, respectively, where ω represents the angular velocity, r represents the turning radius, and d represents the distance between the left and right wheels.

[0084] b. The state space observed by the warehouse robot is provided by the LiDAR, which includes the robot's relative coordinates and movement speed;

[0085] c. The motion space of the warehouse robot is determined by its linear velocity and angular velocity. Table 1 shows the specific discretization into 8 types of motion.

[0086] Table 1 shows the corresponding linear and angular velocities for each action.

[0087]

[0088] d. External team rewards for warehouse management tasks The settings are as follows:

[0089]

[0090] Where t represents time. This indicates the reward the robot receives upon arrival. This represents the reward the robot receives when it encounters an obstacle. The specific format is as follows:

[0091]

[0092] To prevent the robot from hitting obstacles, a larger negative reward is set when the robot collides with an obstacle. A collision is defined as the distance between the center point of the robot and the center point of the obstacle being less than or equal to a set value.

[0093] Step S1.3: As shown in Table 2 below, set the basic hyperparameters of the reinforcement learning algorithm, including: number of neural network layers, number of network neurons, activation function, training set size, replay pool size, learner, policy network learning rate, critic network learning rate, optimizer, optimizer parameters, discount factor, gradient intercept norm, auxiliary function network, auxiliary function network learning rate, auxiliary function network gradient intercept norm, initial shaping weights, shaping weight range, correction function weights, and reward signal r. reach Reward signal r crash wait.

[0094] Table 2. Basic hyperparameters of reinforcement learning algorithms

[0095]

[0096] Step S2 involves each warehouse robot interacting with the dynamic environment to collect training data. After standardization and normalization, a reward function is obtained, including:

[0097] Step S2.1: At the start of each round, initialize environmental data, randomly generate target points, and randomly generate noise;

[0098] Step S2.2: Each robot obtains observations, inputs the observations into the policy network, and adjusts the policy based on the output of the critic network in the system;

[0099] Step S2.3: Each robot executes an action according to the policy function output, obtains environmental rewards and observations of the state at the next time step, records the obtained information in the experience replay pool, and updates the network parameters in the system;

[0100] Step S2.4: Repeat steps S2.2 to S2.3 until the round ends due to the following events: Event 1: A robot collides with an obstacle or another robot; Event 2: Each robot successfully retrieves the goods and delivers them to the target point; Event 3: The time step exceeds the maximum number of rounds. After the round ends, output the obtained data to the specified directory.

[0101] like Figure 4 As shown, step S3 defines the utilization of the reward function as a two-level optimization problem, where the lower level is the optimization strategy to maximize the overall reward, and the upper level is the optimization parameters to maximize the expected cumulative extrinsic reward. The adaptive correction reward function is calculated, including:

[0102] Step S3.1: Establish a two-layer optimization framework. At time step t, the state environment variables are determined by s. t ∈S indicates that after the robot obtains the state value, it will choose an action based on its current policy function, i.e., a t The policy function is represented by π. Policy π θ The value is determined by J(π) θ J(θ) or J(θ) represents the robot following the strategy π. θ The sum of the expected cumulative discount rewards obtained when performing an action, i.e.

[0103]

[0104] Here, γ represents the discount factor, and T signifies the process by which the robot, after taking a certain action through the policy function in a given state, reaches another state. For the calculation of the value function, the policy gradient method is used in the bi-layer optimization framework, and the specific update formula is as follows:

[0105]

[0106] in It is the return value before termination.

[0107] The robot has two internal modules: a policy function parameterized by θ and an intrinsic reward function parameterized by η. The robot receives a state s from the environment, updates the intrinsic reward parameter η by maximizing the extrinsic reward, and then obtains the intrinsic reward r.in Then, by maximizing the intrinsic reward r in External rewards r ex The sum is used to update the policy parameters, thereby obtaining action a.

[0108] Step S3.2: Set the adaptive reward function. Set the intrinsic reward to r. in The auxiliary reward parameterization is The two-level optimization objective is defined as follows.

[0109]

[0110] The intrinsic reward parameter is updated by maximizing the extrinsic reward, and the policy parameter is updated by maximizing the overall reward. The reward function is expressed as follows:

[0111]

[0112] Where, r ex (s,a) represents the extrinsic reward, i.e., the original reward; r in (s,a) represents the intrinsic reward; It is a parameterized weight function that can adaptively adjust and optimize the reward function; It's a correction function, it's a hyperparameter. It is a parameterized function, used as a correction function to penalize sparse reward functions.

[0113] Step S4 combines a multi-agent reinforcement learning algorithm to provide each robot with a unique adaptive correction reward function. The reward value for each robot is calculated and stored in the experience replay pool along with the training data, including:

[0114] Step S4.1: Treat the real team reward as the external reward of robot i, denoted as r. ex (s i ,a i Referring to the adaptive reward function form above, the auxiliary reward function is defined as follows: The parameters of the intrinsic reward function are State-action pair (s) of a single robot i i ,a i () is the input, and the agent reward r is assigned to robot i. proxy as follows:

[0115]

[0116] Where, r in (s,a) represents intrinsic reward. For parameterized weight functions, It is a correction function, derived from Parameterization, where λ is a hyperparameter.

[0117] Step S4.2: External environment provides external rewards and the current state value s t Each robot selects its action a based on its state and strategy parameters. i,t The action value is transmitted to the external environment, the next state is obtained, and the trajectory is stored in the experience pool.

[0118] Step S5 involves collecting a batch of data from the experience replay pool, as shown in Table 3 below. The optimal policy network for scheduling the intelligent warehouse robot is trained using a multi-agent reinforcement learning algorithm based on adaptive corrected reward shaping, including:

[0119] Step S5.1: After all robots have completed their actions in each round, randomly collect a batch of data from the experience replay pool for training;

[0120] Step S5.2: Set the update method for the strategy parameters and adaptive reward parameters. At time step t, create an agent reward for each robot. After t, accordingly reduce the discounted agent reward for each robot. Defined as:

[0121]

[0122] Agent value function of robot i as follows:

[0123]

[0124] Since there are two parameters, namely the policy parameter θ and the auxiliary reward parameter... A two-level optimization method is used, fixing one value while optimizing the other. The specific update objective for the two-level optimization is shown below:

[0125]

[0126] Among them, the expected discount return for robot i's agent Depends on θ i and Φ represents the auxiliary reward parameter set. Θ represents the policy parameter set {θ1, θ2, ..., θ n}

[0127] Step S5.3: Calculate the gradient used to update the policy network parameters. This is done by optimizing the auxiliary function parameters. To maximize By maximizing the agent's expected discount return J for robot i proxy To optimize the strategy parameter θi By a given strategy The generated trajectory, θ i Update using policy gradient:

[0128]

[0129] in, He is a proxy critic, making As the advantage function, where It is composed of φ i Parameterized proxy value, s i θ' is the next state of robot i on the trajectory. The updated policy parameters θ i ′ can be represented as:

[0130]

[0131] Step S5.4: Calculate the gradient used to update the adaptive reward parameters. A two-layer alternating optimization method is used to calculate the gradient of the adaptive reward function, while the auxiliary reward parameters are optimized with the policy parameters θ fixed. Fixed auxiliary reward parameters Optimize the policy parameters θ. The expected cumulative correction reward J. ex+in The gradient with respect to the policy parameter θ is of the following form:

[0132]

[0133] Among them G ex+in (s,a) is the state action value under the current policy.

[0134] Step S5.5: Repeat steps S5.1 to S5.4 for a total of K times to end this round. Repeat steps S2 to S5 until the algorithm fully converges, obtaining the optimal strategy network for multi-warehouse robot scheduling.

[0135] Table 3. Algorithm flow of Algorithm 4.1: Multi-agent Reinforcement Learning Based on Adaptive Reward Shaping

[0136]

[0137]

[0138] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The above description is merely a preferred embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots, characterized in that... Includes the following steps: S1. Design the task environment, set the relevant parameters of the warehouse robot, target object and obstacles, establish the robot kinematic model, and set the hyperparameters of the multi-robot reinforcement learning algorithm; S2. Through the interaction between each warehouse robot and the environment, training data is collected, and the reward function is obtained after standardization and normalization. S3. Define the utilization of the reward function as a two-level optimization problem, where the lower level is the optimization strategy to maximize the overall reward, and the upper level is the optimization parameters to maximize the expected cumulative external reward. The adaptive correction reward function is obtained by calculation. Step S3 includes the following steps: Step S3.1: Establish a two-layer optimization framework; at the time step State environment variables are This means that after the robot receives the state value, it will select an action based on its current policy function. The policy function is used to Representation; Strategy The value is determined by or This indicates that it represents the robot following the strategy. The sum of the expected cumulative discount rewards obtained when performing an action, i.e. in Using discounts as a starting point, and The meaning is that, in a certain state, the robot takes a certain action through the policy function and then reaches another state; for the calculation of the value function, the policy gradient method is used in the two-layer optimization framework, and the specific update formula is as follows: in It is the return value before termination; The robot has two modules inside, one of which is composed of The parameterized strategy function, the other is... Parameterized intrinsic reward function; the robot obtains a certain state from the environment. Update intrinsic reward parameters by maximizing extrinsic rewards. Afterwards, one receives an intrinsic reward. Then by maximizing intrinsic rewards External rewards The sum is used to update the policy parameters, thereby obtaining the action. ; Step S3.2: Set the adaptive reward function: Set the intrinsic reward to... The auxiliary reward parameterization is The two-level optimization objective is defined as follows: The intrinsic reward parameter is updated by maximizing the extrinsic reward, and the policy parameter is updated by maximizing the overall reward; the reward function is expressed as follows: in, External rewards, i.e., the original rewards; As an intrinsic reward; It is a parameterized weight function that can adaptively adjust and optimize the reward function; It's a correction function, it's a hyperparameter. It is a parameterized function, used as a correction function to penalize sparse reward functions; S4. Combining a multi-agent reinforcement learning algorithm, a unique adaptive correction reward function is provided for each robot. The reward value of each robot is calculated and stored together with the training data in the experience replay pool. S5. Collect a batch of data from the experience replay pool, and use a multi-agent reinforcement learning algorithm based on adaptive correction reward shaping to train the robot and obtain the optimal policy network for scheduling multi-warehouse robots.

2. The adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots according to claim 1, characterized in that, The hyperparameters include the formal design parameters of the state space, action space, and reward function.

3. The adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots according to claim 1, characterized in that, Step S1 includes the following steps: Step S1.1: Design the task environment, set the map length and width, the length and width of the internal walls and their center coordinates, and the center coordinates of the shelves; Step S1.2: Establish the robot's kinematic model and set the hyperparameters of the multi-robot reinforcement learning algorithm, including: a. Wheeled differential drive kinematic model of a warehouse robot; b. The state space observed by the warehouse robot is provided by the LiDAR, which includes the robot's relative coordinates and movement speed; c. The motion space of a warehouse robot is determined by its linear velocity and angular velocity; d. Setting up external team rewards for warehouse management tasks; Step S1.3: Set the basic hyperparameters of the reinforcement learning algorithm, including at least: number of neural network layers, number of network neurons, activation function, training set size, replay pool size, learner, policy network learning rate, critic network learning rate, optimizer, optimizer parameters, discount factor, gradient intercept norm, 5-auxiliary function network, auxiliary function network learning rate, auxiliary function network gradient intercept norm, initial shaping weights, shaping weight range, correction function weights, reward signal ssℎ, reward signal ssℎ.

4. The adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots according to claim 3, characterized in that, The wheeled differential drive kinematic model describes the vehicle's speed, steering, and position, expressed as: ， ， in Represents the linear velocity of the robot during its movement. and These represent the speeds of the robot's left and right wheels, respectively. Represents angular velocity. Represents the turning radius. This represents the distance between the left and right wheels.

5. The adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots according to claim 3, characterized in that, Set the team extrinsic reward 𝑟𝑡𝑒𝑥 for the aforementioned warehouse management task to: Where t represents time. This indicates the reward the robot receives upon arrival. The reward a robot receives when it encounters an obstacle is represented in the following format: To prevent the robot from hitting obstacles, a larger negative reward is set when the robot collides with an obstacle. A collision is defined as the distance between the robot's center point and the obstacle's center point being less than or equal to a set value.

6. The adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots according to claim 1, characterized in that, Step S2 includes the following steps: Step S2.1: At the start of each round, initialize environmental data, randomly generate target points, and randomly generate noise; Step S2.2: Each robot obtains observations, inputs the observations into the policy network, and adjusts the policy based on the output of the critic network in the system; Step S2.3: Each robot executes an action according to the policy function output, obtains environmental rewards and observations of the state at the next time step, records the obtained information in the experience replay pool, and updates the network parameters in the system; Step S2.4: Repeat steps S2.2 to S2.3 until the following events occur to end the round: Event 1: A robot collides with an obstacle or another robot; Event 2: Each robot successfully retrieves the goods and delivers them to the target point; Event 3: The time step exceeds the maximum number of rounds; After the round ends, output the obtained data to the specified directory.

7. The adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots according to claim 1, characterized in that, Step S4 includes the following steps: Step S4.1: Use real team rewards as bots External rewards, denoted as Referring to the adaptive reward function form above, the auxiliary reward function is defined as follows: The parameters of the intrinsic reward function are: With a single robot State Action For input, for robot Distribute agent rewards as follows: in, As an intrinsic reward, For parameterized weight functions, It is a correction function, derived from Parameterization, where It's a hyperparameter; Step S4.2: External environment provides external rewards and the current state value Each robot selects its action based on its state and strategy parameters. The action value is transmitted to the external environment, the next state is obtained, and the trajectory is stored in the experience pool.

8. The adaptive modified reward shaping method for reinforcement learning scheduling of multi-warehouse robots according to claim 1, characterized in that, Step S5 includes the following steps: Step S5.1: After all robots have completed their actions in each round, randomly collect a batch of data from the experience replay pool for training; Step S5.2: Set the update method for strategy parameters and adaptive reward parameters; create an agent reward for each robot at time step t, and after t, accordingly reduce the discounted agent reward for each robot. Defined as: robot Proxy value function as follows: Since there are two parameters, namely the strategy parameter And auxiliary reward parameters A two-level optimization method is used, fixing one value and optimizing the other; the specific two-level optimization update objective is shown below: Among them, robots Expected Discount Returns for Agents depending on and , Indicates the auxiliary reward parameter set , Represents the policy parameter set ; Step S5.3: Calculate the gradient used to update the policy network parameters; optimize the auxiliary function parameters. To maximize By maximizing the robot Expected Discount Returns for Agents To optimize strategy parameters ; through a given strategy The generated trajectory, Update using policy gradient: in, He is a proxy critic, making As the advantage function, where It is by Parameterized proxy value, This is the next state of robot i in the trajectory; the updated policy parameters. Represented as: Step S5.4: Calculate the gradient used to update the adaptive reward parameters; use a two-level alternating optimization method to calculate the gradient of the adaptive reward function, keeping the policy parameters fixed. Optimize auxiliary reward parameters Fixed auxiliary reward parameters Optimize strategy parameters Expected cumulative correction reward Relative to policy parameters The gradient is of the following form: in It is the state action value under the current policy; Step S5.5: Repeat steps S5.1 to S5.4 for a total of K times to end this round; repeat steps S2 to S5 until the algorithm fully converges and the optimal strategy network for multi-warehouse robot scheduling is obtained.

Images