A multi-machine shelf warehouse scheduling method based on deep reinforcement learning

Abstract

The application discloses a kind of multi-machine shelf warehousing scheduling methods based on deep reinforcement learning, can improve the execution efficiency of multi-machine warehousing scheduling logistics system;Using deep reinforcement learning algorithm realizes the autonomous decision of multi-machine shelf warehousing scheduling problem, artificial design cost is low, solution efficiency is high, real-time is good, and self-adapting ability is good;Multi-machine warehousing scheduling network mask mechanism is designed, and the hard constraint problem of variable action space allowed to execute is solved in combination with multi-head attention neural network;Multi-machine shelf warehousing scheduling action space structure is designed, while ensuring that the number of actions is small, the multi-agent learning problem is converted into a single-body Markov decision process model that can be centrally trained, avoiding the common non-stationary environment problem and action space explosion problem in multi-agent reinforcement learning, which is conducive to ensuring the convergence of reinforcement learning algorithm and improving the training speed of algorithm;Sequential execution operation is designed to avoid task conflict caused by multi-agent simultaneous decision.

Description

TECHNICAL FIELD

[0001] The present application relates to multi-robot warehouse scheduling planning technology, belonging to the field of combinatorial optimization and artificial intelligence, and in particular to a multi-robot shelf warehouse scheduling method based on deep reinforcement learning. BACKGROUND

[0002] In recent years, with the rapid development of e-commerce economy, autonomous intelligent technology has been widely applied in warehouse systems. In order to reduce labor costs and achieve higher order picking efficiency, an unmanned intelligent warehouse scheduling system composed of mobile shelves and multiple mobile robots is developed and applied. In this unmanned intelligent warehouse scheduling system, multi-robot shelf warehouse scheduling is an important problem: each robot needs to start from the initial position to a selected shelf retrieval position; then the shelf is transported to the required picking station; then the robot transports the picked shelf to an empty storage position, puts it down and sequentially executes the above process until all the required shelves are transported to the corresponding picking station, and each robot returns to its initial position. In order to achieve reasonable resource deployment under certain constraints and achieve the best performance indicators as much as possible, an efficient optimization scheduling method is urgently needed. For the multi-robot warehouse scheduling system, the existing scheduling methods still have the following problems: the algorithm based on intelligent swarm optimization often faces the problem of long solving time and difficult real-time guarantee; some heuristic construction methods are proposed to solve the online multi-robot warehouse scheduling planning problem. Although the search efficiency of the heuristic method is high, this method usually needs to be carefully designed by humans according to specific problems, and the designed method usually lacks universality, and in some cases the algorithm may even degenerate into a traversal algorithm. At the same time, the solution of the heuristic construction method is easy to fall into a local optimal solution, and the solution quality is unstable. In recent years, deep reinforcement learning has shown great application potential in solving multi-robot planning problems, providing ideas for efficiently solving multi-robot shelf warehouse scheduling problems. The deep reinforcement learning method has the characteristics of fast solving speed, high efficiency, and good self-adaptive ability, but when solving the multi-robot shelf warehouse scheduling problem, it also faces challenges such as complex action and state space of learning tasks, difficulty in model convergence, sparse rewards, and action conflicts between individuals. SUMMARY

[0003] The technical problem to be solved by the present application is that in view of the problems in the prior art, the present application provides a multi-robot shelf warehouse scheduling method based on deep reinforcement learning, which has low labor cost, does not require task conflict resolution, high solving efficiency, and good generalization.

[0004] A multi-robot shelf warehouse scheduling method based on deep reinforcement learning, comprising the following steps:

[0005] Step 1: Model the centralized multi-machine warehouse scheduling problem as a Markov Decision Process (MDP) model; define the various elements in the MDP model: state, action, reward, state transition function;

[0006] Step 2: Based on the MDP model established in Step 1, design a multi-machine warehouse scheduling neural network model based on attention mechanism; define the encoder mask rule and decoder mask rule in the attention mechanism neural network; define the multi-robot warehouse scheduling interaction environment;

[0007] Step 3: According to the MDP model established in Step 2, randomly generate a batch of problem instances and their initial states; initialize the neural network parameters and the encoder mask and decoder mask according to the multi-robot warehouse scheduling neural network model established in Step 2; initialize the environment parameters according to the interaction environment established in Step 2; initialize the number of training steps and the number of rounds.

[0008] Step 4: Input the current state observation of the robot in Step 3 into the multi-robot warehouse scheduling neural network model in Step 3 to obtain the strategy distribution output by the neural network; randomly select a warehouse scheduling action according to the strategy distribution;

[0009] Step 5: According to the action output in Step 4 and the environment interaction in Step 3, realize the state transition, obtain the new observation state after the interaction is completed, and obtain the immediate reward value from the environment interaction; the number of training steps in Step 4 is incremented by one;

[0010] Step 6: Execute Step 4 until all the warehouse moving tasks to be processed are completed, i.e. a round is completed, and the cumulative reward value is calculated;

[0011] Step 7: Calculate the loss function according to the cumulative reward value in Step 6 and the strategy distribution output by the neural network in Step 4, and update the parameters of the multi-robot warehouse scheduling neural network model according to the loss function through the Adam algorithm; the number of training rounds is incremented by one;

[0012] Step 8: Execute Steps 3 to 7 until the maximum number of training rounds is reached, and end the training; save the neural network model file and the process file;

[0013] Step 9: Apply the neural network model trained in Step 8 to solve the multi-robot warehouse scheduling problem; input the map model and environment parameters, and the multi-robot warehouse scheduling neural network will output the scheduling action sequence result; each robot executes its corresponding action in sequence according to the output scheduling planning action sequence.

[0014] Preferably, in Step 1:

[0015] The system state s is designed as the observation information of all robots, including the unprocessed shelf coordinates, all available storage location coordinates, the action type of all robots at the previous time step, and the position coordinates of the robot currently assigned with a task and the position coordinates of other robots waiting for task assignment.

[0016] The action a contains four action types: selecting a shelf, selecting a storage location, selecting a robot number that needs to be optimized, and selecting a robot currently working to return to the initial position.

[0017] The reward R uses the round reward instead of the immediate reward, which is specifically defined as: the negative of the maximum value of the total Manhattan distance of all robots completing all carrying shelf tasks and returning empty completed shelves to storage locations and then returning to the starting position, i.e. R = max(R1, R2, …, R i ,…,R n ), where R i is the total Manhattan distance walked by the i-th robot, which is specifically defined as:

[0018]

[0019] where φ(p i ,p j ) represents the Manhattan distance between two points p i ,p j in the Cartesian coordinate system, i.e. represents the initial position coordinates of the i-th robot, represents the j-th visited shelf location, represents the j-th visited picking station location, represents the j-th visited storage location. After N i iterations, all shelves that need to be transported complete the scheduling task.

[0020] The state transition function T indicates that the state s is transferred to the state s′, and the corresponding reward is obtained after the action is executed.

[0021] Preferably, in step 2, the encoder and decoder mask rules are: each mask bit corresponds to an action, and the meanings of the elements from left to right are: return the first robot to the initial position, return the second robot to the initial position,..., return the mth robot to the initial position, select the first robot optimization, select the second robot optimization,..., select the nth robot optimization, carry the first shelf, carry the second shelf,..., carry the wth shelf, put back into the first storage position, put back into the second storage position,..., put back into the cth storage position; if the corresponding position of the mask is 1, it means that the corresponding action is not allowed to be executed, and if the corresponding position of the mask is 0, it means that the corresponding action is allowed to be executed.

[0022] Preferably, in step 2, the multi-machine warehouse scheduling neural network model structure is composed of an encoder and a decoder, wherein the encoder is composed of a linear layer, an N-layer multi-head attention neural network, and an encoder mask, and the decoder is composed of a linear aggregation layer, a multi-head attention network, a linear layer, a SoftMax layer, and a decoder mask; after the state input is transmitted to the encoder, the hidden variable is obtained through the linear layer first, and then the hidden variable data is transmitted to the N-layer multi-head attention neural network; the encoder mask is used to control which hidden variables in the multi-head attention network can be observed, and the encoder mask is initially all observable, and then the encoder mask is determined by the output action state to obtain the encoding value; the hidden variable obtained by the encoder is extracted by the decoder according to the decoder mask signal, and the extracted signal is re-aggregated, and then the aggregated hidden variable is transmitted to a multi-head attention neural network controlled by a decoder mask and a linear fully connected layer network; wherein the decoder mask is initially all observable, and then the decoder mask is determined by the output action state to obtain the encoding value. Finally, the output data is mapped to 0-1 through the SoftMax layer to obtain the final output data, i.e. the probability of each action being selected; wherein the state input format of the multi-machine warehouse scheduling neural network is aligned with the action and mask format, and the hidden variable in the neural network is also aligned with the meaning of each element in the action space.

[0023] Preferably, in step 2, designing the multi-robot warehouse shelving scheduling interaction environment comprises: designing a warehouse scheduling scene map, which contains discrete distributed shelving positions, storage position, robot initial position and picking station position; designing the interaction rule as follows: the robot needs to access all the shelves requiring handling, then deliver the goods to the designated picking station, each shelf corresponds to a required access picking station, then deliver the completed picking shelves to an empty storage position, and then deliver the next shelf until all the shelving tasks are completed, and the robot returns to the starting position; setting the goal as: multiple robots simultaneously cooperate to complete all shelving tasks with the shortest possible distance; and generating the corresponding warehouse scheduling environment given the number of robots, the number of shelves and the number of storage positions.

[0024] Preferably, in step 5, the immediate reward value is the Manhattan distance required to access the necessary nodes between the position of the last action performed by the robot and the position after performing the new action:

[0025]

[0026] wherein, wherein φ (p i ,p j ) represents the Manhattan distance between two points p i ,p j in the Cartesian coordinate system, that is, p a is the initial position coordinate of the robot, is the jth accessed shelf position, is the jth accessed picking station position, is the jth accessed storage position, and N is the number of times of accessing the shelves, storage positions or picking stations to end all shelving tasks.

[0027] Preferably, in step 7, the calculation method of the loss function is:

[0028]

[0029] wherein i represents the problem instance number, R (π i ) represents the cumulative reward value obtained under the policy π i , r (s ) represents the episode reward value obtained under the baseline policy , p θ (π i |s i ) represents the probability value of the selected action output by the neural network when the neural network parameter is θ, the state s i and the action is selected by the policy π i . a baseline policy by freezing the policy pi every certain number of steps i corresponding neural network output is obtained.

[0030] Preferably, in the step, each robot sequentially executes the corresponding action according to the output scheduling action sequence, and the specific conversion operation is:

[0031] (1) According to the action sequence output by the neural network, record the action sequence from the selection of the i th robot to the end of the action of returning to the initial position of the current robot, and save the action sequence as the execution action sequence of the i th robot;

[0032] (2) According to the corresponding action after the end of the initial position action of the robot selected in step (1), obtain the next robot number k, and repeat step (1) until the end of the action of returning to the initial position of the current robot is encountered again, and save the action sequence as the execution action sequence of the k th robot;

[0033] (3) Repeat step (2) until the entire action sequence is completed, and obtain the execution action sequence assigned to each robot.

[0034] The present application has the following beneficial effects:

[0035] (1) The present application adopts a deep reinforcement learning method, which can efficiently solve the common multi-robot warehouse scheduling problem in the logistics warehouse environment in an end-to-end manner. The proposed method has low artificial design cost and high solving efficiency, which not only reduces the artificial cost of manually designing heuristic operators in traditional heuristic methods, but also avoids the problem of long solving time of population algorithms and solvers, which is beneficial to improve the operation efficiency of real logistics warehouse systems. Based on the good adaptability and generalization ability of the deep neural network itself, when the training sample is sufficient, the trained network can well adapt to different environmental parameters and different maps corresponding to the problem.

[0036] (2) The present application designs a mask mechanism for the multi-robot warehouse scheduling network, which solves the problem of variable action space allowed to be executed by combining a multi-head attention neural network: such as the storage position of the returned cabinet is not empty and cannot be accessed. Common reinforcement learning algorithms usually handle such potential constraint problems by adding a penalty term in the reward for violating the constraint, but for such hard constraints, there is still a possibility of violating the constraint. In the present application, the mask mechanism is directly used to handle the hard constraint problem of variable action space allowed to be executed, and the baseline network is frozen to improve the convergence ability of the algorithm, which is beneficial to the algorithm to converge well in different situations.

[0037] (3) The action space structure is ingeniously designed, the multi-agent learning problem is converted into a single MDP model which can be centrally trained under the premise of almost no increase in the number of actions, the non-stationary environment problem and the action space explosion problem commonly encountered in multi-agent reinforcement learning are avoided, and the convergence of the reinforcement learning algorithm is beneficial. The task conflict problem caused by the simultaneous decision of multiple agents is also avoided by the centrally designed action space structure, and there is no need to specially design a conflict resolution module like the auction algorithm. The designed action space structure can easily decouple the centralized action space to the distributed action space in the test stage, and each agent can easily obtain the action it should perform, so the multiple machines can operate in parallel, which is beneficial to the deployment and application of the algorithm to actual scheduling problems. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The flowchart of the present application.

[0039] Figure 2 The mask / action space structure diagram of the present application.

[0040] Figure 3 The neural network structure diagram of the present application. DETAILED DESCRIPTION

[0041] The present application will be described in detail below with reference to the drawings and examples.

[0042] As Figure 1 shown, the present application provides a multi-machine shelf warehouse scheduling method based on deep reinforcement learning, which specifically includes the following steps:

[0043] Step 1: centrally model the multi-machine shelf warehouse scheduling problem as a Markov Decision Process (MDP) model: define various elements in the MDP model: state, action, reward, state transition function. Specifically, it includes:

[0044] (1) The system state s is designed as the observation information of all robots, including unprocessed shelf coordinates, all available storage location coordinates, the action type of all robots at the previous time step, and the position coordinates of the robot currently assigned tasks and the position coordinates of other robots waiting for task assignment;

[0045] (2) The action a includes four types of actions: selecting a shelf, selecting a storage location, selecting a robot number to be optimized, and selecting a robot currently working to return to the initial position. The specific action space elements and their meanings are as shown in Figure 2 ;

[0046] (3) Reward R uses round reward instead of immediate reward, specifically defined as: the negative of the maximum value of the total Manhattan distance of all robots to complete all pallet tasks and put back empty completed pallets to storage locations and then return to the starting position, i.e. R = max(R1, R2, …, R i N n ), where R i is the total Manhattan distance walked by the i-th robot, specifically defined as:

[0047]

[0048] where φ(p i , p j ) refers to the Manhattan distance between two points p i , p j in the Cartesian coordinate system, i.e. refers to the initial position coordinates of the i-th robot, refers to the j-th visited pallet location, refers to the j-th visited picking station location, refers to the j-th visited storage location. After N i iterations, all pallets that need to be transported complete the scheduling task.

[0049] (4) The state transition function T refers to the state s moving to state s' and obtaining the corresponding reward after action execution. The transition sequence is fixed because the application designs an action execution cycle. During action execution, the system first selects an idle robot to work. Then, the selected robot performs the actions of selecting a retrieval pallet and selecting a storage location in turn. Once the storage location selection action is completed, the system allows another idle robot to be selected, which will perform the above operations in turn. When all the required tasks are completed or another robot is selected to work in the previous step, the previously working robot should return to the initial position.

[0050] Step 2: According to the MDP model established in step 1, design a multi-machine warehouse scheduling neural network model, a mask rule and an interactive environment, specifically including:

[0051] (1) Design a multi-machine warehouse scheduling neural network model based on attention mechanism. For example, Figure 3As shown, the designed multi-machine warehouse scheduling neural network model structure is composed of an encoder and a decoder structure, wherein the encoder is composed of a linear layer, an N-layer multi-head attention neural network, and an encoder mask, and the decoder is composed of a linear aggregation layer, a multi-head attention network, a linear layer, a SoftMax layer, and a decoder mask. After the state input is transmitted to the encoder, the hidden variable is obtained through the linear layer first, and then the hidden variable data is transmitted to the N-layer multi-head attention neural network. The encoder mask is used to control which hidden variable in the multi-head attention network can be observed. The encoder mask is initially fully observable, and then the encoder mask value is determined by the output action state. The decoder extracts the unshielded signal according to the decoder mask signal, and the signal is re-aggregated after being processed by the linear fully connected neural network to become a new aggregation hidden variable. The aggregation hidden variable is then transmitted to a multi-head attention neural network controlled by a layer of decoder mask and a layer of linear fully connected network. Among them, the decoder mask is initially fully observable, and then the decoder mask value is determined by the output action state. Finally, the output data is mapped to 0-1 through the SoftMax layer to obtain the final output data, that is, the probability of each action being selected.

[0052] (2) Define the encoder and decoder mask rules in the attention mechanism neural network. The encoder and decoder mask elements in the designed attention mechanism neural network have the following meanings: Figure 2 As shown, each mask bit corresponds to an action, and the meanings of the elements from left to right are as follows: return the initial position of the first robot, return the initial position of the second robot,..., return the initial position of the mth robot, select the first robot optimization, select the second robot optimization,..., select the nth robot optimization, carry the first shelf, carry the second shelf,..., carry the wth shelf, put back into the first storage position, put back into the second storage position,..., put back into the cth storage position. If the mask at the corresponding position is 1, it means that the corresponding action is not allowed to be executed, and if the mask at the corresponding position is 0, it means that the corresponding action is allowed to be executed. The state input format of the multi-machine warehouse scheduling neural network is aligned with the action and mask format of Figure 2 The hidden variables in the neural network are also aligned with the meanings of the elements in the action space.

[0053] (3) Design a multi-robot warehouse rack scheduling interaction environment, specifically including: design a warehouse scheduling scene map, which contains discrete distributed rack positions, storage position, robot initial position and picking station position; design interaction rules: the robot needs to access all racks requiring handling, then deliver the goods to the designated picking station, each rack corresponds to a required access picking station, then deliver the completed picking rack to an empty storage position, and then deliver the next rack until all rack handling tasks are completed, and the robot returns to the starting position; set the goal: multiple robots simultaneously cooperate to complete all rack handling tasks with the shortest possible distance. Given the number of robots, the number of racks, and the number of storage positions, the corresponding warehouse scheduling environment will be automatically generated.

[0054] Step 3: According to the MDP model established in step 2, given the map file designed in step 2, initialize the environment parameters: input the number of robots, the number of racks, and the number of storage positions, initialize the model training hyperparameters, and randomly generate a batch of (M) problem instances and their initial states. According to the multi-warehouse scheduling neural network model established in step 2, initialize the neural network parameters and mask state, and initialize the training step number and round number to 0.

[0055] Step 4: Input the current state observation results of the robots corresponding to the M problem instances in step 3 into the multi-warehouse scheduling neural network model in step 3. The state observation result refers to: when the training step number is 0, the state observation result is the initial state in step 3; when the training step number is not 0, it is the state observed after the last action and environment interaction. The neural network model outputs the strategy distribution of the M problem instances, i.e. the probability distribution of each action being selected: π(s|θ) = SoftMax(DNN(s)). According to the strategy distribution probability, a warehouse scheduling action is randomly selected, i.e. the larger the network output value, the higher the probability of being selected. The action corresponding to the network output probability value masked by the encoder mask is 0, so it cannot be selected. Then according to the warehouse scheduling action, assign the action to be executed next for each robot, as shown in Figure 2 The action number 0 represents returning to the initial position of the first robot, the action number 1 represents returning to the initial position of the second robot, and the meanings of the remaining action numbers correspond to the meanings of the positions corresponding to the action numbers in the mask in step 2.

[0056] Step 5: According to the action output in step 4 and the environment in step 3, realize state transition, obtain the new observation state s' after interaction, and obtain the immediate reward value r obtained by the current optimized robot and the environment interaction, i.e. the Manhattan distance required to access the necessary nodes between the position when the last action is executed and the initial position to the position after the new action is executed:

[0057]

[0058] where p a is the initial position coordinate of the robot, is the jth visited rack position, is the jth visited picking station position, is the jth visited storage position. N is the number of rack / storage / picking station visits to end all rack handling tasks.

[0059] The number of training steps in step 4 is increased by one.

[0060] Step 6: Perform step 4 until all rack handling tasks that need to be processed are completed, i.e. a round ends, and calculate the cumulative reward value.

[0061] Step 7: According to the cumulative reward value in step 6 and the neural network output policy distribution corresponding to each problem instance in step 4, calculate the loss function, and the calculation method of the loss function is:

[0062]

[0063] where i is the problem instance number, R(π i ) represents the cumulative reward value obtained under the policy π i , R represents the round reward value obtained under the baseline policy , p θ (π i |s i ) represents the probability value of the selected action output by the neural network when the action is selected by the policy π i under the state s i with the neural network parameter θ. is the gradient of the neural network with the parameter θ, and the baseline policy is obtained by freezing the neural network output corresponding to the policy π i every certain number of steps.

[0064] Update the neural network parameters according to the loss function using the Adam algorithm.

[0065] The number of training rounds is increased by one.

[0066] Step 8: Perform steps 3 to 7 until the maximum number of training rounds is reached, and end the training. Save the neural network model file and process file.

[0067] Step 9: Apply the trained neural network model of step 8 to solve the multi-robot warehouse scheduling problem. Input the map model and environmental parameters, and the multi-robot warehouse scheduling neural network will output a sequence of scheduling actions. Each robot will execute its corresponding actions in sequence according to the output scheduling action sequence. The specific conversion operation is as follows:

[0068] (1) According to the action sequence output by the neural network, record the action sequence from the action selection of the i-th robot until the action of returning to the initial position of the current robot is encountered, and save the action sequence as the execution action sequence of the i-th robot;

[0069] (2) According to the corresponding action after the initial position action of the robot selected in step (1) ends, obtain the next robot number k, and repeat step (1) until the action of returning to the initial position of the current robot is encountered again, and save the action sequence as the execution action sequence of the k-th robot;

[0070] (3) Repeat step (2) until the entire action sequence is completed, and obtain the execution action sequence assigned to each robot.

[0071] According to the above, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1.A multi-machine rack warehouse scheduling method based on deep reinforcement learning, characterized in that, The method comprises the following steps: Step 1: centrally model the multi-machine shelf storage scheduling problem as a Markov decision process (MDP) model; define various elements in the MDP model: state, action, reward, and state transition function; Step 2: design a multi-machine storage scheduling neural network model based on an attention mechanism according to the MDP model established in step 1; define the encoder mask rule and the decoder mask rule in the attention mechanism neural network; define the multi-robot storage shelf scheduling interactive environment; Step 3: randomly generate a batch of problem instances and their initial states according to the MDP model established in step 2; initialize the neural network parameters and the encoder mask and the decoder mask according to the multi-robot storage scheduling neural network model established in step 2; initialize the environment parameters according to the interactive environment established in step 2; initialize the number of training steps and the number of rounds; Step 4: input the current state observation of the robot in step 3 into the multi-robot storage scheduling neural network model in step 3 to obtain the strategy distribution output by the neural network; randomly select a storage scheduling action according to the strategy distribution; Step 5: interact with the environment according to the action output in step 4 and the environment in step 3 to realize state transition, obtain the new observation state after the interaction is completed, and obtain the immediate reward value through environment interaction; the number of training steps in step 4 is increased by one; Step 6: execute step 4 until all shelf moving tasks to be processed are completed, i.e., one round is completed, and the cumulative reward value is calculated; Step 7: calculate the loss function according to the cumulative reward value in step 6 and the strategy distribution output by the neural network in step 4, and update the multi-robot storage scheduling neural network model parameters through the Adam algorithm according to the loss function; the number of training rounds is increased by one; Step 8: execute steps 3 to 7 until the maximum number of training rounds is reached, and the training is ended; save the neural network model file and the process file; Step 9: apply the neural network model trained in step 8 to solve the multi-robot storage scheduling problem; input the map model and the environment parameters, and the multi-robot storage scheduling neural network will output a scheduling action sequence result; each robot executes its corresponding action in sequence according to the output scheduling planning action sequence. 2.The multi-robot warehouse scheduling method based on deep reinforcement learning of claim 1, wherein, In step 1: The system state s is designed as the observation information of all robots, including the coordinates of the unprocessed shelves, the coordinates of all available storage locations, the action types of all robots at the previous time step, and the position coordinates of the robot currently assigned tasks and other robots waiting for task assignment; The action a includes four types of actions: selecting a shelf, selecting a storage location, selecting a robot number to be optimized, and selecting a robot currently working to return to the initial position; The reward R uses round rewards instead of immediate rewards, defined as the negative of the maximum of the total Manhattan distance of all robots to complete all the picking tasks and put back the empty completed shelves to the storage locations and then return to the start locations, i.e., R = max(R1, R2,..., R i , ..., R n ), where R i is the total Manhattan distance of the i-th robot, defined as: where φ(p i , p j ) denotes the Manhattan distance between two points p i , p j in a Cartesian coordinate system, i.e. denotes the initial position coordinate of the i-th robot, denotes the j-th visited shelf position, denotes the j-th visited picking station position, denotes the j-th visited storage position; after N i iterations, all shelves that need to be shipped complete the scheduling task; The state transition function T indicates that the state s is transferred to the state s' and the corresponding reward is obtained after the action is executed. 3.The method of claim 1, wherein, In step 2, the encoder and decoder mask rules are: each mask bit corresponds to an action, and the meanings of the elements from left to right are: return the initial position of the first robot, return the initial position of the second robot,..., return the initial position of the mth robot, select the first robot optimization, select the second robot optimization,..., select the nth robot optimization, carry the first shelf, carry the second shelf,..., carry the wth shelf, return to the first storage position, return to the second storage position,..., return to the cth storage position; if the corresponding position mask is 1, it means that the corresponding action is not allowed to be executed, and if the corresponding position mask is 0, it means that the corresponding action is allowed to be executed. 4.The method of claim 3, wherein, In step 2, the multi-machine warehouse scheduling neural network model structure is composed of an encoder and a decoder, wherein the encoder is composed of a linear layer, an N-layer multi-head attention neural network and an encoder mask, and the decoder is composed of a linear aggregation layer, a multi-head attention network, a linear layer, a SoftMax layer and a decoder mask; after the state input is transmitted to the encoder, the hidden variable is obtained through the linear layer first, and then the hidden variable data is transmitted to the N-layer multi-head attention neural network, and the encoder mask is used to control which hidden variables in the multi-head attention network can be observed; the encoder mask is initially all observable, and then the encoder mask value is determined by the output action state; the hidden variable obtained after the encoder processing is extracted by the decoder according to the decoder mask signal, and the unshielded signal is re-aggregated to become a new aggregated hidden variable after being processed by a linear fully connected neural network; then the aggregated hidden variable is transmitted into a multi-head attention neural network controlled by a decoder mask and a linear fully connected layer network; wherein the decoder mask is initially all observable, and then the decoder mask value is determined by the output action state; finally, the output data is mapped to 0-1 through the SoftMax layer to obtain the final output data, i.e. the probability of each action being selected; wherein the state input format of the multi-machine warehouse scheduling neural network is aligned with the action and mask format, and the hidden variable in the neural network is also aligned with the meaning of each element in the action space. 5.The method of claim 1, wherein, In step 2, the multi-robot warehouse shelf scheduling interaction environment is designed, including: designing a warehouse scheduling scene map, which contains discrete distributed shelf positions, storage position, robot initial position and picking station position; the interaction rule is designed as follows: the robot needs to access all the required shelves to be carried, and then deliver the goods to the specified picking station; each shelf corresponds to a required picking station, and then the picked shelf is delivered to an empty storage position, and the next shelf is delivered until all the shelf carrying tasks are completed, and the robot returns to the starting position; the target is set as: multiple robots cooperate with each other to complete all the shelf carrying tasks with the shortest possible distance; the given robot number, shelf number and storage position number parameters generate the corresponding warehouse scheduling environment. 6.The method of claim 1, wherein, In step 5, the immediate reward value is the Manhattan distance required to access the necessary nodes between the position of the last action performed by the robot or the initial position and the position after the new action is performed: wherein, wherein φ(p i , p j ) denotes the Manhattan distance between two points p i , p j in a Cartesian coordinate system, i.e. p a is the initial position coordinate of the robot, denotes the jth visited rack position, denotes the jth visited picking station position, denotes the jth visited storage position, and N is the number of rack, storage or picking station visits to end all rack handling tasks. 7.The method of claim 1, wherein, In step 7, the loss function is calculated as: where i denotes the problem instance number, R(π i ) denotes the cumulative reward value obtained under policy π i , denotes the episode reward value obtained under baseline policy , θ (π i |s i ) denotes the probability value of the selected action output by the neural network when the action is selected by policy π i under state s i with the neural network parameter being θ, is the gradient of the neural network with the parameter being θ, and the baseline policy is obtained by freezing the corresponding neural network output of policy π i every certain number of steps. 8.The method of claim 1, wherein, In the step, each robot executes its corresponding action in sequence according to the output scheduling plan, and the specific conversion operation is: (1) According to the action sequence output by the neural network, record the action sequence from the selection of the i-th robot until the selection of the action to return to the initial position of the current robot ends, and save the action sequence as the execution action sequence of the i-th robot; (2) According to the corresponding action after the selection of the action to return to the initial position of the robot in step (1), obtain the next robot number k, and repeat step (1) until the selection of the action to return to the initial position of the current robot ends again, and save the action sequence as the execution action sequence of the k-th robot; (3) Repeat step (2) until the entire action sequence is completed, and obtain the execution action sequence assigned to each robot.

Images