Material picking method based on mobile robot

Abstract

Disclosed in the present invention is a material picking method based on a mobile robot, comprising the following steps: step S1, acquiring a real-time production plan and a bill of materials, wherein the bill of materials comprises material codes, material names, material quantity and lead time; step S2, on the basis of the material codes, dividing the bill of materials into a plurality of task units suitable for single operation of the mobile robot, and assigning corresponding picking stations; step S3, performing picking state labeling on the task units on the basis of the lead time of materials in the task units, and assigning corresponding mobile robots on the basis of the labeled picking states; and step S4, the mobile robots performing path planning on the basis of the assigned task units, and executing the assigned task units on the basis of the planned path, so that racks to which the materials in the assigned task units belong are transported to specified picking stations for operators to pick the required materials.

Description

A material picking method based on mobile robots Field of the invention

[0001] The present invention relates to the field of material management, in particular to a material picking method based on mobile robots. BACKGROUND

[0002] With the rapid development of intelligent manufacturing, each link in modern manufacturing has made great progress. Material picking, as an important link in production and manufacturing, is crucial to production efficiency and accuracy. Traditional material picking methods usually rely on manual operation, which has the problems of low efficiency, high labor cost and easy errors. Therefore, in order to improve the efficiency and accuracy of material picking, in recent years, a material picking method based on RMFS (Robotic Mobile Fulfillment Systems) has gradually attracted attention.

[0003] RMFS is an autonomous mobile robot system with task allocation, path planning and execution capabilities. It is composed of a group of mobile robots and supporting computer terminals, which can automatically execute material picking tasks in manufacturing environments. In the traditional RMFS material picking method, robots complete picking tasks by carrying sensors and actuators according to pre-set rules or paths. However, this method has some problems:

[0004] 1. Traditional RMFS lacks systematic thinking, only considering the completion of task execution, without considering the changes in warehouse environment according to production plans.

[0005] 2. The path planning and task allocation of the robot may not be efficient, leading to redundancy and resource waste in the picking process.

[0006] SUMMARY

[0007] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0008] The purpose of the present invention is to solve the above problems, and a material picking method based on mobile robots is provided, which adopts an improved path planning method based on counterfactual multi-agent policy gradient algorithm to plan the path for the mobile robot carrying materials, and uses a time series model to predict future material plans, thereby optimizing the production warehouse.

[0009] The technical scheme of the present invention is as follows:

[0010] The application provides a material picking method based on a mobile robot, comprising the following steps:

[0011] Step S1: obtaining a real-time production plan and a bill of materials; wherein the bill of materials comprises a material code, a material name, a material quantity and a material preparation time limit;

[0012] Step S2: dividing the bill of materials into a plurality of task units suitable for a single operation of the mobile robot according to the material code, and assigning corresponding picking stations;

[0013] Step S3: marking the task units according to the material preparation time limit of the materials in the task units, and assigning corresponding mobile robots according to the marked picking states;

[0014] Step S4: the mobile robot plans a path according to the assigned task unit, and executes the assigned task unit according to the planned path, so as to carry the materials in the assigned task unit to the designated picking station for the staff to pick the required materials.

[0015] According to an embodiment of the material picking method based on the mobile robot, the picking state comprises immediate picking and pending picking; wherein after the material picking method based on the mobile robot completes the marking of the picking state of the task unit, the task unit marked as immediate picking is assigned to the corresponding mobile robot, and the mobile robot carries the required materials in the task unit to the designated picking station for the staff to pick.

[0016] According to an embodiment of the material picking method based on the mobile robot, when the material picking method based on the mobile robot executes the task unit marked as immediate picking, it simultaneously searches whether there are materials located in the same shelf in the task unit marked as pending picking; if so, the task units corresponding to all the pending picking materials located in the same shelf are merged into a new task unit, and a corresponding mobile robot and a picking station are assigned.

[0017] According to an embodiment of the material picking method based on the mobile robot, after the mobile robot obtains the task unit, it plans a path according to the shelf position corresponding to the assigned task unit, the position of the picking station and the predetermined route of other mobile robots currently running; wherein the mobile robot plans a path by using an improved path planning method based on the counterfactual multi-agent policy gradient algorithm, and then executes the assigned task unit according to the best moving path obtained by planning.

[0018] According to an embodiment of the material picking method based on the mobile robot provided in the present application, when the mobile robot plans a path by using the improved counterfactual multi-agent policy gradient algorithm, first, the improved counterfactual multi-agent policy gradient algorithm is used to perform reinforcement learning training on the moving path of each mobile robot, so as to obtain a policy network, and then the learned policy network is applied to the path planning of each mobile robot, so as to obtain the corresponding optimal moving path.

[0019] According to an embodiment of the material picking method based on the mobile robot provided in the present application, the material picking method based on the mobile robot is obtained by the following steps of reinforcement learning training:

[0020] Step C1: environment modeling is performed according to an actual material picking site, so as to obtain a two-dimensional space grid model;

[0021] Step C2: the corresponding state space and action space are defined according to the constructed two-dimensional space grid model, and a policy network is constructed for each mobile robot according to the defined state space and action space;

[0022] Step C3: the moving path of the mobile robot is sampled according to the pre-defined action space, and the positioning information and state information of the mobile robot are obtained in real time according to the pre-defined state space;

[0023] Step C4: the incentive value of the real moving action taken by the mobile robot under the current state is calculated according to the pre-defined optimization incentive function;

[0024] Step C5: the advantage function value of the real moving action taken by the mobile robot under the current state is calculated according to the pre-defined advantage function and counterfactual baseline;

[0025] Step C6: the policy gradient value is calculated according to the calculated advantage function value, and the corresponding policy network parameters are updated and optimized according to the calculated policy gradient value;

[0026] Step C7: it is judged whether the current state reaches the pre-defined reinforcement learning training termination condition; if yes, the reinforcement learning training is ended, and the learned policy network is output; if not, steps C3-C6 are repeated, and the reinforcement learning training is continued.

[0027] According to an embodiment of the material picking method based on the mobile robot, in step C2, when defining the action space of the mobile robot, the material picking method based on the mobile robot further defines the conflict state rule of the mobile robot according to the constructed two-dimensional space grid model, and determines whether the mobile robot in the two-dimensional space grid model has collision conflict through the conflict state rule; wherein the material picking method based on the mobile robot measures whether the mobile robot has collision according to the following defined conflict state rule:

[0028] Conflict state rule one: two mobile robots occupy the same grid in the two-dimensional space grid model at the same time;

[0029] Conflict state rule two: the moving paths of two mobile robots are located on the same straight line, moving towards each other and both carrying shelves at the same time.

[0030] According to an embodiment of the material picking method based on the mobile robot, after the material picking method based on the mobile robot completes the definition of the conflict state rule of the mobile robot, a corresponding conflict warning mechanism is set according to the defined conflict state rule, so as to avoid the conflict and collision of the mobile robot in the two-dimensional space grid model, including the following steps:

[0031] Step D1: coordinate coding is performed on each grid in the two-dimensional space grid model, and the state code of the mobile robot is defined;

[0032] Step D2: the coordinate coding and the state code of any pair of mobile robots in the two-dimensional space grid model are monitored in real time;

[0033] Step D3: the conflict warning score between any pair of mobile robots is calculated based on the real-time monitored coordinate coding and state code;

[0034] Step D4: the collision conflict expectation of the corresponding two mobile robots is measured according to the calculated conflict warning score.

[0035] According to an embodiment of the material picking method based on the mobile robot, in step D3, when the material picking method based on the mobile robot calculates the conflict warning score between any pair of mobile robots, a conflict warning score calculation formula is defined according to the predefined conflict state rule of the mobile robot, and then the conflict warning score calculation formula is used to calculate the conflict warning score between any pair of mobile robots in the two-dimensional space grid model; wherein the conflict warning score calculation formula is as follows:

[0036] wherein S ij represents the conflict warning score between any pair of mobile robots i and j in the two-dimensional space grid.

[0037] and xi and yi represent the horizontal coordinate and the vertical coordinate of the mobile robot i respectively,

[0038] and xj and yj represent the horizontal coordinate and the vertical coordinate of the mobile robot j respectively,

[0039] and si and sj represent the state code of the mobile robots i and j respectively, wherein the value is 0 when the mobile robot is empty, and the value is 1 when the mobile robot carries a shelf.

[0040] According to an embodiment of the material picking method based on the mobile robot, after calculating the conflict warning score of any pair of mobile robots in the two-dimensional space grid model, the material picking method based on the mobile robot can also use the conflict warning score of any pair of mobile robots in the two-dimensional space grid model to calculate the global conflict warning value, and use the calculated global conflict warning value to measure the collision expectation of all mobile robots in the entire two-dimensional space grid model. The calculation formula is as follows:

[0041] wherein, fw represents the global conflict warning value,

[0042] S ij represents the conflict warning score between any pair of mobile robots i and j in the two-dimensional space grid,

[0043] N represents the number of mobile robots in the two-dimensional space grid.

[0044] According to an embodiment of the material picking method based on the mobile robot, in step C4, the material picking method based on the mobile robot constructs an optimization incentive function according to the vector angle between the moving direction and the path direction corresponding to the real moving action of the mobile robot, and uses the constructed optimization incentive function to give the real moving action of the mobile robot a corresponding reward value.

[0045] According to an embodiment of the material picking method based on the mobile robot, when constructing the optimization incentive function, the material picking method based on the mobile robot divides the complete path of the mobile robot carrying the shelf into a first moving path and a second moving path, calculates the vector angle between the moving direction and the path direction corresponding to the real moving action of the mobile robot in real time by monitoring the real-time path vectors corresponding to the first moving path and the second moving path, and then uses the calculated vector angle to calculate the reward value of the real moving action of the mobile robot; wherein,

[0046] The real-time path vector corresponding to the first moving path is a real-time path vector from the position of the mobile robot carrying the goods shelf to the target goods shelf,

[0047] The real-time path vector corresponding to the second moving path is a real-time path vector from the position of the mobile robot carrying the goods shelf to the target picking station.

[0048] According to an embodiment of the material picking method based on the mobile robot, in step C5, when constructing the advantage function of the mobile robot, the material picking method based on the mobile robot takes the global reward value brought by the default moving action in the given state as the counterfactual baseline cb, and then uses the difference between the global reward value brought by the real moving action taken by the mobile robot in the given state and the counterfactual baseline cb to construct the advantage function of the mobile robot, and the calculation formula is as follows:

[0049] Wherein, a represents the mobile robot a i The real moving action taken in the given state s The advantage function value of the mobile robot a is contained in the joint moving action u taken by all mobile robots;

[0050] Q(s, u) represents the global reward value brought by the joint moving action u taken by all mobile robots in the given state s;

[0051] a represents the mobile robot a i The case of selecting the default moving action u';

[0052] a represents the mobile robot a i The observed historical data The probability of selecting the default moving action u' after

[0053] a represents the mobile robot a i The joint action space of all mobile robots except the mobile robot a is unchanged, and the mobile robot a i The global reward value when the mobile robot a selects the default moving action u'.

[0054] According to an embodiment of the material picking method based on the mobile robot, in step C6, after the material picking method based on the mobile robot calculates the advantage function value of the real moving action taken by the mobile robot in the current state through step C5, the policy gradient value is calculated according to the calculated advantage function value, and then the corresponding policy network parameters are updated and optimized according to the calculated policy gradient value; wherein the calculation formula of the policy gradient value is as follows:

[0055] where g k denotes the policy gradient of the kth iteration,

[0056] E π denotes the expected value under different policies π,

[0057] denotes the action selected by the mobile robot a i the gradient operator of the kth policy network parameter θ k ,

[0058] denotes the action selected by the mobile robot a i according to the policy the log probability of the action u' given the observation data .

[0059] According to an embodiment of the mobile robot-based material picking method, the mobile robot-based material picking method further trains a time series model, predicts future production plans and material lists through the trained time series model, and then adjusts the positions of shelves with different moving frequencies according to the prediction results, thereby improving the material picking efficiency.

[0060] According to an embodiment of the mobile robot-based material picking method, the mobile robot-based material picking method trains a time series model for predicting future production plans and material lists by the following steps:

[0061] Step E1: Collect historical task unit data and clean the collected historical task unit data;

[0062] Step E2: Extract the required feature values from the cleaned historical task unit data; wherein the feature values include material code, logistics name, material quantity, production quarter, industry sentiment, and remaining amount;

[0063] Step E3: Divide the extracted feature values into a data set, including a training set and a test set;

[0064] Step E4: Select a time series model to be trained, and train the selected time series model using the training set to obtain a time series model for predicting future production plans and material lists;

[0065] Step E5: Evaluate the accuracy of the time series model using the test set, and adjust the training time series model according to the evaluation results to obtain a high-performance time series model;

[0066] Step E6: Forecasting future production plans and bill of materials using the trained time series model.

[0067] According to an embodiment of the mobile robot-based material picking method of the present application, in step E4, the mobile robot-based material picking method selects a Transformer-XL and LSTM-based time series model as the time series model for production planning and bill of materials, and obtains the prediction distribution result at time step t by the Transformer-XL and LSTM-based time series model, including the following steps:

[0068] Step E41: Establish a conditional distribution model as the input of the time step t encoder, encode the input training set by the conditional distribution model, and obtain the time series data, as follows: P(Z i,t：T |Z i,1：t-1 , h i,1：t-1 )

[0069] where Z i,t：T =[Z i,t , Z i,t+1 ,..., Z i,T ] represents the future situation of time series data i at time step t,

[0070] Z i，1：t-1 =[Z i，1 , Z i,2 ,..., Z i,t-2 , Z i,t-1 ] represents the past situation of time series data i at time step t,

[0071] h i,1：t-1 represents the hidden state of time series data i before time step t;

[0072] Step E42: Model the time dependency in the time series data using an LSTM neural network, and calculate the time dependency by the LSTM neural network, thereby obtaining the hidden state h i，t of time series data i at time step t;

[0073] Step E43: Model the periodic and seasonal features in the time series data using a Transformer-XL with an adaptive learning rate mechanism, and extract the periodic and seasonal features by the Transformer-XL model, thereby obtaining the output feature C i,t of time series data i at time step t;

[0074] Step E44: Concatenate the hidden state h i，tand the output feature C of the Transformer-XL model i，t , to obtain the output feature R of the time step t encoder i，t , as follows: R i，t = (h i,t , C i,t )

[0075] where (h i,t , C i,t ) represents a concatenation operation for connecting the hidden state h i，t of the LSTM neural network and the output feature C i，t of the Transformer-XL model in the feature dimension;

[0076] Step E45: decoding the output feature R i，t using the time step t decoder to obtain the prediction distribution result of the time series data i at the time step t, as follows: fc = W·(R i,1:t , y i,1:t-1 ) + b

[0077] where P(y i，t |R i,1:t , y i,1:t-1 ) represents the prediction distribution result of the time series data i at the time step t,

[0078] R i,1:t = [R i，1 , R i,2 , …, R i,t ] represents the historical information of the time series data i at the time step t,

[0079] y i,1:t-1 = [Y i，1 , Y i,2 , …, Y i，t-2 , Y i，t-1 ] represents the past prediction sequence of the time series data i at the time step t,

[0080] fc represents the output of the full connection layer,

[0081] W represents the weight matrix of the full connection layer,

[0082] b represents the bias vector of the full connection layer,

[0083] exp represents the exponential function,

[0084] sum represents the summation of all elements in the vector.

[0085] According to an embodiment of the material picking method based on a mobile robot of the present invention, in step E42, the hidden state h of time series data i at time step t is calculated through the following steps. i，t :

[0086] Step 1: Calculate the output f of the forget gate i，t The formula is as follows: f i，t =σ(W f ·(h i，t-1 Z i，t )+b f )

[0087] Among them, W f The weight matrix of the forget gate, b f Let σ represent the bias vector of the forget gate, and let σ represent a sigmoid activation function.

[0088] Step 2: Calculate the output i of the input gate. i，t The formula is as follows: i i,t =σ(W i ·(h i，t-1 Z i,t )+b i )

[0089] Among them, W i Let b represent the weight matrix of the input gate. i σ represents the input gate bias vector, and σ represents a sigmoid activation function.

[0090] Step 3: Calculate the state g of the candidate memory cell i,t The formula is as follows: g i，t =tanh(W c ·(h i，t-1 Z i,t )+b c )

[0091] Among them, W c The weight matrix representing the memory unit, b c represents the bias vector of the memory unit, and tanh represents a hyperbolic tangent activation function;

[0092] Step 4: Output f based on the forget gate i，t and the output i of the input gate i,t To update the memory cell state c i，t The formula is as follows: c i,t =f i，t ⊙c i,t-1 +i i,t ⊙g i,t

[0093] Where ⊙ represents element-wise multiplication, ci,t-1 represents the past memory cell state of the time series data i at the time step t;

[0094] Fifth step: calculating the output o of the output gate i，t , the formula is as follows: o i，t =σ(W o ·(h i，t-1 ,Z i,t )+b o )

[0095] Wherein, W o represents the weight matrix of the output gate, b o represents the bias vector of the output gate, and sigma represents a sigmoid activation function.

[0096] Sixth step: updating the hidden state h i，t of the time series data i at the time step t based on the updated memory cell state c i，t and the output o i，t of the output gate, the formula is as follows: h i，t =o i，t ⊙tanh(c i,t )

[0097] Wherein, represents element-wise multiplication, and tanh represents a hyperbolic tangent activation function.

[0098] According to an embodiment of the material sorting method based on the mobile robot, in step E43, the material sorting method based on the mobile robot extracts periodic and seasonal features and adaptively updates the Transformer-XL model parameters by the following steps:

[0099] Step E431: dividing the time series data into multiple fixed-length subsequences, and inputting the divided subsequences into the Transformer-XL encoder, so as to obtain the hidden state of the time series sequence data at each time step and the relative position encoding of the corresponding position; wherein, the relative position encoding formula is as follows: p j =f(j-t)

[0100] Wherein, the time step t represents a given time step,

[0101] The time step j represents any time step that needs to calculate the relative position of the time step t,

[0102] p j represents the relative position information of the time step t and the time step j,

[0103] f represents an encoding function for encoding the relative position between time steps.

[0104] Step E432: The attention score between the current time step and the previous time step is calculated using the self-attention mechanism, and the relative importance vector between the current time step and the previous time step is calculated according to the calculated attention score, as follows:

[0105] where o t represents the relative importance vector between time step t and time step j,

[0106] a tj represents the attention score between the current time step t and time step j,

[0107] V j represents the value vector of time step j,

[0108] Q t represents the query vector of the current time step t,

[0109] K j represents the key vector of time step j;

[0110] Step E433: The hidden state of the current time step and the relative position encoding of the corresponding position are extracted using the Transformer-XL encoder, and the output feature of the current time step is calculated as the periodic and seasonal feature according to the extracted hidden state and the relative position encoding of the corresponding position, as follows:

[0111] where C i,t represents the output feature of time series data i at the current time step t,

[0112] represents the hidden state of time series data i at the current time step t,

[0113] P i，t =(p1,p2,...,p T ) represents the relative position encoding of time series data i at the current time step t,

[0114] [,] represents a concatenation operation;

[0115] Step E434: The model parameters are updated using the adaptive learning rate mechanism, and the updated Transformer-XL model is applied to the feature extraction of the next time step, as follows:

[0116] where p represents the Transformer-XL model parameters, such as weights and biases,

[0117] α represents the current learning rate,

[0118] denotes the modified first moment estimate, representing the average of past gradients,

[0119] denotes the modified second moment estimate, representing the average of past squared gradients,

[0120] denotes a small constant to prevent the denominator from being zero;

[0121] Step E435: judging whether the output feature extraction of all time steps is completed; if yes, ending the feature extraction; if no, repeating steps E432-step E434 to extract the output feature of each time step and update the model parameters using the adaptive learning rate, time step by time step.

[0122] The application also provides a computer readable medium storing computer program codes, which, when executed by a processor, implement the method as described above.

[0123] The application also provides a material picking device based on a mobile robot, comprising:

[0124] a memory for storing instructions executable by the processor; and

[0125] a processor for executing the instructions to implement the method as described above.

[0126] The application has the following beneficial effects compared with the prior art: the application is aimed at material picking work in the context of intelligent manufacturing, and adopts an improved path planning method based on counterfactual multi-agent policy gradient algorithm to plan a path for a mobile robot carrying materials, so that the mobile robot carries the material belonging to the assigned task unit to the designated picking station according to the planned moving path to allow a worker to pick the required material. Compared with the traditional RMFS material picking method, the mobile robot carrying materials in the application can systematically plan a path, plan a more efficient moving path for material carrying according to changes in the warehouse environment and production plan, and greatly improve the intelligence and automation of logistics carrying. Meanwhile, the application also collects historical task unit data to train a time series model to predict future production plans and required materials, so as to optimize the deployment of shelves in the production warehouse and improve the efficiency of robot path planning and task allocation, greatly reducing redundancy and resource waste in the picking process. BRIEF DESCRIPTION OF DRAWINGS

[0127] The above features and advantages of the present application will be better understood by reading the detailed description of the embodiments of the present application in conjunction with the drawings, in which: In the drawings, the components are not necessarily drawn to scale, and like reference numbers designate like components or steps throughout the various figures.

[0128] FIG. 1 is a flow chart illustrating an embodiment of a mobile robot-based material picking method of the present application.

[0129] FIG. 2 is a flow chart illustrating steps of an embodiment of the present application for reinforcement learning training of a policy network for path planning by a mobile robot.

[0130] FIG. 3 is a flow chart illustrating steps of an embodiment of the present application for setting a collision warning mechanism to measure a mobile robot’s expected collision.

[0131] FIG. 4 is a schematic diagram illustrating an embodiment of a complete movement path of a mobile robot of the present application.

[0132] FIG. 5 is a flow chart illustrating steps of training a time series model for future production plan and bill of material prediction of the present application.

[0133] FIG. 6 is a flow chart illustrating steps of prediction using a Transformer-XL and LSTM-based time series model of the present application.

[0134] FIG. 7 is a flow chart illustrating steps of feature extraction by a Transformer-XL model of the present application.

[0135] Detailed description of the invention

[0136] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some examples or embodiments of the present application, and for those skilled in the art, the present application can also be applied to other similar scenarios without creative labor. Unless the context clearly indicates otherwise or otherwise stated, the same reference numbers in the drawings represent the same structures or operations.

[0137] As shown in the present application and claims, unless the context clearly indicates otherwise or otherwise stated, “one”, “a”, “an”, and / or “the” do not necessarily mean a single number, but can also include a plurality. Generally speaking, the terms “comprise” and “include” only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0138] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and practice described. Accordingly, all such variations are intended to be included within the scope of the present application as defined in the following claims, along with full equivalents thereof.

[0139] In describing and claiming the present application, it will be understood that the terms "above", "below", "up", "down", "top", "bottom", "over", "under" and "beneath" are intended to be relative terms that are understood only in the context of the particular orientation of the device being described and are not intended to be limiting.

[0140] In the description of the present application, it will be understood that, when a relative term such as "front", "back", "up", "down", "left", "right", "horizontal", "vertical", and "top", "bottom" is used, the term is used with respect to the orientation shown in the drawings and not necessarily with respect to the orientation of the device or element in use or in operation. Such relative terms are used in this description only to orient the reader with respect to the position of the device or element being described and not to denote proper or required orientation in use or operation. It will be further understood that the terms "in", "on" and "under" are intended to encompass both a position directly adjacent to something and a position indirectly adjacent to something, such that an element is not necessarily directly on or under another element, but can be on or under the other element with one or more elements between.

[0141] For purposes of the description hereinafter, the terms "upper", "lower", "right", "left", and the like shall relate to the device as it is shown in the drawing figures. The terms "front" and "back" shall relate to the device as it is shown in the drawing figures. The terms "over", "under", and the like shall relate to the device as it is shown in the drawing figures. The terms "first", "second", and the like shall relate to the elements as they are initially shown in the drawing figures. The terms "top", "bottom", and the like shall relate to the device as it is shown in the drawing figures. The terms "up", "down", and the like shall relate to the device as it is shown in the drawing figures. The terms "end" and "side" shall relate to the device as it is shown in the drawing figures. The terms "inner" and "outer" shall relate to the device as it is shown in the drawing figures. The terms "front", "back", "up", "down", "top", "bottom", "over", "under", "left", "right", "inner", and "outer" shall not be interpreted to limit the scope of the present application to only those orientations described in the specification. The terms "front", "back", "up", "down", "top", "bottom", "over", "under", "left", "right", "inner", and "outer" shall not be interpreted to limit the scope of the present application to only those orientations described in the specification. The terms "front", "back", "up", "down", "top", "bottom", "over", "under", "left", "right", "inner", and "outer" shall not be interpreted to limit the scope of the present application to only those orientations described in the specification. The terms "front", "back", "up", "down", "top", "bottom", "over", "under", "left", "right", "inner", and "outer" shall not be interpreted to limit the scope of the present application to only those orientations described in the specification.

[0142] In the context of this application, structures described as being "on" one feature relative to another feature can comprise embodiments where the first and second features are formed in direct contact, or can comprise embodiments where additional features are formed between the first and second features such that the first and second features can not be in direct contact.

[0143] It will be understood that when a component is referred to as being "on" another component, "connected to" another component, "coupled to" another component, or "in contact with" another component, it can be directly on, connected to, coupled to, or in contact with the other component, or one or more intervening components can also be present. In contrast, when a component is referred to as being "directly on", "directly connected to", "directly coupled to", or "directly in contact with" another component, there are no intervening components present. It will also be understood that, when a first component is referred to as being "electrically contacted" or "electrically coupled to" a second component, there is an electrical path between the first component and the second component that allows current to flow. The electrical path can include capacitors, coupled inductors, and / or other components that allow current to flow, even if there is no direct contact between electrically conductive components.

[0144] In addition, it should be pointed out that the use of the terms "first", "second", and so on to qualify components is merely intended to facilitate the distinction between the respective components, and such terms do not have a special meaning unless otherwise stated, and therefore should not be interpreted as limiting the scope of protection of the present application. Furthermore, although the terms used in the present application are selected from commonly known terms, some of the terms mentioned in the description of the present application can be selected by the applicant according to his or her judgment, and the detailed meaning of each term is described in the relevant part of the description. Furthermore, the present application is to be understood not only through the actual terms used, but also through the meaning implied by each term.

[0145] Disclosed herein is an embodiment of a mobile robot (AMR) based material picking method. FIG. 1 is a flowchart illustrating an embodiment of a mobile robot based material picking method of the present application. Please refer to FIG. 1, and the following is a detailed description of each step of the mobile robot based material picking method.

[0146] Step S1: Obtain real-time production plan and bill of materials; wherein the bill of materials includes material code, material name, material quantity, and material preparation timeliness.

[0147] Step S2: Divide the bill of materials into multiple task units suitable for single operation of the mobile robot according to the material code, and assign corresponding picking stations.

[0148] In this embodiment, before using the mobile robot to transport materials, the current required production technology and the corresponding material list need to be obtained first to perform subsequent mobile robot and sorting station allocation.

[0149] The material list contains material code, material name, material quantity, and information such as material preparation timeliness. When allocating mobile robots and sorting stations, the material list is first divided into multiple task units suitable for single operation of the mobile robot according to the material code, and the corresponding sorting station is allocated to these task units.

[0150] Step S3: According to the material preparation timeliness of each task unit, the sorting state of each task unit is marked, and the corresponding mobile robot is allocated according to the marked sorting state.

[0151] In this embodiment, before allocating the mobile robot to the task unit, the pre-labeling operation is performed first, the sorting state of each task unit is marked according to the material preparation timeliness of each task unit, and the corresponding mobile robot is allocated according to the marked sorting state. The sorting state includes immediate sorting and pending sorting. After completing the sorting state marking of the task unit, the task unit marked as immediate sorting is first allocated to the corresponding mobile robot, and then the required materials in the task unit are transported to the designated sorting station by the mobile robot for the worker to pick.

[0152] At the same time, for the task unit marked as pending sorting, whether there are materials located in the same shelf in the task unit marked as pending sorting is searched while executing the task unit marked as immediate sorting. If there are, all the task units corresponding to the pending sorting materials located in the same shelf are merged into a new task unit, and the new task unit is allocated to the mobile robot and the sorting station closest to the task shelf, and the task unit is executed in this way.

[0153] Step S4: The mobile robot plans a path according to the allocated task unit, and executes the allocated task unit according to the planned path, so as to transport the shelf of the material in the allocated task unit to the designated sorting station for the worker to pick the required material.

[0154] In this embodiment, after the mobile robot obtains the task unit, the shelf position corresponding to the allocated task unit, the sorting station position, and the predetermined route of other mobile robots in the current operation are used to plan a path, and then the shelf of the material in the task unit is transported to the designated sorting station for the worker to pick the required material according to the planned path.

[0155] In an embodiment, the mobile robots adopt a path planning method of an improved counterfactual multi-agent policy gradients (hereinafter sometimes referred to as COMA algorithm) to plan paths, and then execute the assigned task units according to the best movement paths planned.

[0156] In the path planning method of the improved COMA algorithm, first, the improved COMA algorithm is used to train the movement paths of the mobile robots, a policy network is obtained, and then the learned policy network is applied to the path planning of the mobile robots. The mobile robots select the best movement action in each state according to the learned policy network, thereby obtaining the best movement path. FIG. 2 is a flow chart showing steps of an embodiment of the policy network trained by reinforcement learning for path planning of the mobile robots according to the present application. Please refer to FIG. 2, and the following is a detailed description of each step of the policy network trained by reinforcement learning for path planning of the mobile robots.

[0157] Step C1: Environment modeling is performed according to the actual material picking site to obtain a two-dimensional space grid model;

[0158] In this embodiment, a two-dimensional space grid model is constructed as a production warehouse model according to the actual material picking site, and the mobile robots in the material picking site are modeled as agents in the two-dimensional space grid model. The two-dimensional space grid model includes movable shelves, shelf inventory units (SKUs), and picking stations. The mobile robots simulate the actual material picking site in the two-dimensional space grid model to plan paths.

[0159] Specifically, in this embodiment, the constructed two-dimensional space grid model is a general-scale production warehouse model, including 200 movable shelves, 5 layers of shelves, and 1-5 SKUs on each layer, totaling 1000-5000 SKUs. The number of picking stations and mobile robots needs to be set according to the production scale and material preparation time efficiency. Define A = {a1,..., an} to represent a set of n agents in the two-dimensional space grid model, representing n mobile robots in the two-dimensional space grid model. n} represents a set of n agents in the two-dimensional space grid model, representing n mobile robots in the two-dimensional space grid model.

[0160] Step C2: Define the corresponding state space and action space according to the constructed two-dimensional space grid model, and construct a policy network for each mobile robot according to the defined state space and action space.

[0161] In this embodiment, to enable the mobile robot to perform real-time localization and acquire state information from other mobile robots, a corresponding state space is defined based on a constructed two-dimensional spatial grid model. The state information includes the mobile robot's localization based on the state space and the acquisition of state information from other mobile robots, including the mobile robot's current position, target shelf position, target picking station position, and surrounding environmental information, etc. The acquired localization and state information are used to construct and train a path planning strategy network.

[0162] Specifically, in this embodiment, each mobile robot a is assigned a unit task instruction. i They will all be assigned to their initial position s i Initial position of the target shelf. i and the picking station location specified in the task e i Assume all agents begin executing their tasks at time t = 0. For any AMR, each step can be either moving or waiting, requiring one unit of time. Let P = {p1, ..., p...} n Let} represent the set of all AMR paths, where p i (i∈{1,...,n}) represents the relationship between the AMR robot a i Associated paths, and path p i It consists of two sets of xy path points, i.e., p i =(s i x1, ..., x n o i y1, ...,y n e i ).

[0163] Regarding the motion space, each AMR is allowed to move at a constant speed in four directions: forward, backward, left, and right. Furthermore, in this embodiment, when defining the motion space of the mobile robot, conflict state rules for the mobile robot are also defined based on the constructed two-dimensional spatial mesh model. The following two conflict state rules are used to measure whether collisions occur between the mobile robots in the two-dimensional spatial mesh model:

[0164] Conflict Rule 1: At the same time, two mobile robots occupy the same grid in the two-dimensional space grid model;

[0165] Conflict Rule 2: At the same time, the movement paths of two mobile robots are on the same straight line, moving towards each other and both carrying shelves.

[0166] Further, in this embodiment, after the definition of the conflict state rule of the mobile robot is completed, a corresponding conflict warning mechanism is also set according to the defined conflict state rule, so as to avoid the conflict collision of the mobile robot in the two-dimensional space grid model. FIG. 3 is a flow chart illustrating the steps of an embodiment of the present application for setting the conflict warning mechanism to measure the collision conflict expectation of the mobile robot. Please refer to FIG. 3, and the following is a detailed description of each step of setting the conflict warning mechanism to measure the collision conflict expectation of the mobile robot.

[0167] Step D1: Coordinate coding is performed on each grid in the two-dimensional space grid model, and the state code of the mobile robot is defined.

[0168] In this embodiment, the two-dimensional map of the two-dimensional space grid model is composed of uniform grids. When performing coordinate coding on each grid, the horizontal coordinate of each grid is marked from left to right as (1, 2, 3, …, m-1, m), and the vertical coordinate is marked from top to bottom as (1, 2, 3, …, n-1, n), that is, the coordinate coding of each grid is (x, y), where x∈(1, m) and y∈(1, n). As for the state code s of the mobile robot, it is specified that the value is 0 when it is in the empty state, and the value is 1 when it carries a shelf.

[0169] Step D2: Real-time monitoring of the coordinate coding and state code of any pair of mobile robots in the two-dimensional space grid model.

[0170] Step D3: Based on the real-time monitored coordinate coding and state code, the conflict warning score between any pair of mobile robots is calculated.

[0171] Step D4: According to the calculated conflict warning score, the collision conflict expectation of the corresponding two mobile robots is measured.

[0172] In this embodiment, for any pair of mobile robots a i and a j , the coordinate coding and the state code of the mobile robots in the two-dimensional space grid model are monitored in real time. Then, the conflict warning score S ij is calculated according to the above defined two conflict state rules, and the expression is as follows:

[0173] Wherein, S ij represents the conflict warning score between any pair of mobile robots i and j in the two-dimensional space grid, and represent the horizontal coordinate and the vertical coordinate of the mobile robot i, respectively, and represent the horizontal coordinate and the vertical coordinate of the mobile robot j, respectively. and respectively represent the state code s of the mobile robots i and j, wherein the value is 0 when the mobile robot is empty, and the value is 1 when the mobile robot carries the shelf. Thus, the first term on the right side of the expression measures the distance between any pair of mobile robots, and the second term measures the situation that the mobile robots carry the shelf and are located in the same row or column. The conflict warning score calculated by the expression indicates whether the pair of mobile robots will collide.

[0174] In addition, in the embodiment, after the conflict warning score of any pair of mobile robots in the two-dimensional space grid model is calculated by the conflict warning score expression, the global conflict warning value can also be calculated by using the conflict warning score of any pair of mobile robots in the two-dimensional space grid model. The global conflict warning value calculated by the expression measures the collision expectation of all mobile robots in the two-dimensional space grid model, and the calculation formula is as follows:

[0175] wherein fw represents the global conflict warning value, S ij represents the conflict warning score between any pair of mobile robots i and j in the two-dimensional space grid, and N represents the number of mobile robots in the two-dimensional space grid.

[0176] Step C3: sampling the moving path of the mobile robot according to the predefined action space, and obtaining the positioning information and state information of the mobile robot in real time according to the predefined state space.

[0177] Step C4: calculating the incentive value of the real moving action of the mobile robot in the current state according to the predefined optimization incentive function.

[0178] In the embodiment, before the incentive value of the real moving action of the mobile robot in the current state is calculated, the moving path of the mobile robot is first sampled according to the predefined action space, and the positioning information and state information of the mobile robot are obtained in real time according to the predefined state space, including the current position of the mobile robot, the target shelf position, the target picking station position, and the surrounding environment information, etc. Then, the optimization incentive function is constructed according to the vector angle between the moving direction and the path direction corresponding to the real moving action of the mobile robot, and the real moving action of the mobile robot is given a corresponding reward value through the constructed optimization incentive function. FIG. 4 is a schematic diagram showing an embodiment of the complete moving path of the mobile robot of the application. The embodiment will be further described in detail below in combination with FIG. 4.

[0179] As shown in FIG. 4, according to the defined motion space, the mobile robot a i which can run in four directions, the direction vector of the moving direction of the mobile robot in the process from one path point to the next path point is set as Based on the defined state space, the mobile robot a in the two-dimensional spatial mesh model i The complete path for shelving handling is divided into a first movement path and a second movement path, and the real-time path vectors of these two movement paths are marked as follows: and Wherein, the real-time path vector corresponding to the first movement path The first is the real-time path vector from the mobile robot's current position to the target shelf before it moves the shelf, and the second is the real-time path vector corresponding to the second movement path. This refers to the real-time path vector from the mobile robot's current position to the target picking station after it has moved the shelf. Both the path vector and the real-time path vector are updated in real time as the mobile robot's path points change. Therefore, by monitoring the real-time path vector of the first moving path... Real-time path vector of the second movement path The algorithm calculates the vector angle between the movement direction and the path direction corresponding to the actual movement action taken by the mobile robot in real time, and then uses the calculated vector angle to calculate the reward value of the actual movement action taken by the mobile robot. The vector angles θ1 and θ2 between the movement direction and the path direction in the first and second movement paths are respectively:

[0180] In this embodiment, the closer the values ​​of θ1 and θ2 are to 0, that is, the smaller the angle between the movement direction of the mobile robot and the path direction vector in each step, the larger the reward value is assigned to the mobile robot. Simultaneously, the reward value for the j-th step is recorded as Q. j The specific expression is as follows:

[0181] First movement path (s) i o i )middle,

[0182] In the second movement path (o i e i )middle,

[0183] Step C5: Calculate the dominance function value of the actual movement action taken by the mobile robot in the current state based on the predefined dominance function and counterfactual baseline.

[0184] In this embodiment, using the counterfactual baseline and dominance function means replacing the mobile robot a with a default movement action. i Selected real movement action Compare the global reward values ​​brought by the two to calculate the mobile robot a. iThe improvement of the overall performance of the multi-agent path planning system by actual movement actions is investigated. Specifically, when constructing the mobile robot's advantage function, the global reward value from the default movement action in a given state is used as the counterfactual baseline cb. Then, the difference between the global reward value from the actual movement action taken by the mobile robot in the given state and the counterfactual baseline cb is used to construct the mobile robot's advantage function. The calculation formula is as follows:

[0185] in, Represents mobile robot a i The actual movement action taken in a given state s The dominant function value, The joint movement action u is included in all mobile robots. Q(s, u) represents the global reward value brought by the joint movement action u taken by all mobile robots in a given state s. Represents mobile robot a i The case where the default movement action u′ is selected. Represents mobile robot a i Historical data observation Then, the probability of choosing the default movement action u′. Indicates that, given state s, except for mobile robot a i Joint motion space of all mobile robots except Unchanged, mobile robot a i The global reward value when the default movement action u′ is selected.

[0186] Step C6: Calculate the policy gradient value based on the calculated advantage function value, and optimize and update the corresponding policy network parameters based on the calculated policy gradient value.

[0187] In this embodiment, after calculating the advantage function value of the actual movement action taken by the mobile robot in the current state through step C5, the policy gradient value is calculated based on the calculated advantage function value, and then the corresponding policy network parameters are optimized and updated based on the calculated policy gradient value. The formula for calculating the policy gradient value is as follows:

[0188] Among them, g k This represents the policy gradient in the k-th iteration.

[0189] E π This represents the expected value under different strategies π.

[0190] This indicates that for mobile robot a i The k-th policy network parameter θ kthe gradient operator,

[0191] a mobile robot a i according to the policy the log probability of the action u' chosen given the observation data the log probability of the action u' chosen given the observation data

[0192] Step C7: determining whether the current state reaches a predefined reinforcement learning training termination condition; if yes, ending the reinforcement learning training and outputting the learned policy network; if no, repeating steps C3-C6 to continue the reinforcement learning training.

[0193] In this embodiment, if the current state reaches the predefined reinforcement learning training termination condition, for example, reaches the maximum number of iterations or the policy network converges to a stable state, the reinforcement learning training is ended and the learned policy network is output. Otherwise, steps C3-C6 are repeated to train and evaluate the policy network until the preset stopping condition is reached, and the policy network is output. The mobile robot performs path planning in the two-dimensional space grid model according to the learned policy network, thereby efficiently transporting the materials belonging to the assigned task unit to the designated picking station for the staff to pick the required materials.

[0194] In addition, in this embodiment, the past shelf movement information is also regularly integrated, and the feature information such as material information, production cycle, and inventory level related to the task unit is extracted to train a time series model. The trained time series model is used to predict future production plans and material lists, and then according to the prediction results, the shelves with high potential movement frequency are adjusted to a position closer to the picking station, and the shelves with low movement frequency are adjusted to a position farther away from the picking station, so as to improve the material picking efficiency. FIG. 5 is a flowchart showing the steps of training the time series model for predicting future production plans and material lists of the present application. Please refer to FIG. 5, and the following is a detailed description of each step of training the time series model for predicting future production plans and material lists.

[0195] Step E1: collecting historical task unit data and cleaning the collected historical task unit data.

[0196] Step E2: extracting the required feature data from the cleaned historical task unit data; wherein the feature data includes material code, logistics name, material quantity, production quarter, industry sentiment, and remaining amount.

[0197] In this embodiment, before the time series model training is performed, first, the material code, name, quantity, current production season, industry climate in the current season, and the inventory of the current material and other data involved in the past mobile shelf task unit are collected as historical task unit data, and the collected historical task unit data is cleaned to eliminate abnormal values, missing values and repeated values, and then the required feature data is extracted from the cleaned historical task unit, including material code, logistics name, material quantity, production season, industry climate and remaining amount, for subsequent time series model training.

[0198] Step E3: The extracted feature data is divided into a data set, including a training set and a test set.

[0199] In this embodiment, the feature data extracted in the above steps is integrated into a data set to be trained, wherein 80% of the data is randomly divided into a training set, and the remaining 20% is used as a test set.

[0200] Step E4: Select a time series model to be trained, and use the training set to train the selected time series model.

[0201] In this embodiment, a time series model based on Transformer-XL and LSTM is selected as the time series model for production planning and bill of materials, and the prediction distribution result of time step t is obtained through the time series model based on Transformer-XL and LSTM. FIG. 6 is a flowchart showing the steps of prediction using the time series model based on Transformer-XL and LSTM, please refer to FIG. 6, and the following is a detailed description of each step of prediction using the time series model based on Transformer-XL and LSTM:

[0202] Step E41: Establish a conditional distribution model as the input of the time step t encoder, encode the input training set through the conditional distribution model to obtain time series data, the expression is as follows: P(Z i,t：T |Z i,1：t-1 , h i,1：t-1 )

[0203] Wherein, Z i,t：T =[Z i,t , Z i,t+1 , …, Z i，T ] represents the future situation of time series data i at time step t, Z i,1:t-1 =[Z i，1 , Z i,2 , …, Z i,t-2 , Z i,t-1 ] represents the past situation of time series data i at time step t, h i,1：t-1denotes the hidden state of the time series data i at time step t-1.

[0204] Step E42: modeling the time dependency in the time series data using an LSTM neural network and calculating the time dependency by the LSTM neural network, thereby obtaining the hidden state h i，t .

[0205] Specifically, in the embodiment, the hidden state h i，t of the time series data i at time step t is calculated by the following steps.

[0206] First step: calculating the output f i，t of the forget gate, the formula is as follows: f i，t = σ(W f ·(h i，t-1 , Z i，t )+b f )

[0207] Wherein, W f denotes the weight matrix of the forget gate, b f denotes the bias vector of the forget gate, and σ denotes a sigmoid activation function.

[0208] Second step: calculating the output i i，t of the input gate, the formula is as follows: i i,t = σ(W i ·(h i，t-1 , Z i,t )+b i )

[0209] Wherein, W i denotes the weight matrix of the input gate, b i denotes the bias vector of the input gate, and σ denotes a sigmoid activation function.

[0210] Third step: calculating the candidate memory cell state g i,t , the formula is as follows: g i，t = tanh(W c ·(h i，t-1 , Z i,t )+b c )

[0211] Wherein, W c denotes the weight matrix of the memory cell, b c denotes the bias vector of the memory cell, and tanh denotes a hyperbolic tangent activation function.

[0212] Fourth step: based on the output f i，t of the forget gate and the output i i,tTo update the memory cell state c i，t The formula is as follows: c i,t =f i，t ⊙c i,t-1 +i i,t ⊙g i,t

[0213] Where ⊙ represents element-wise multiplication, c i，t-1 This represents the past memory unit state of time series data i at time step t.

[0214] Step 5: Calculate the output o of the output gate i，t The formula is as follows: i，t =σ(W o ·(h i，t-1 Z i,t )+b o )

[0215] Among them, W o Let b represent the weight matrix of the output gate. o Let σ represent the bias vector of the output gate, and let σ represent a sigmoid activation function.

[0216] Step 6: Based on the updated memory cell state c i，t The output of the output gate is calculated. i，t To update the hidden state h of time series data i at time step t. i，t The formula is as follows: h i，t =o i，t ⊙tanh(c i,t )

[0217] Where ⊙ represents element-wise multiplication, and tanh represents a hyperbolic tangent activation function.

[0218] Step E43: Use Transformer-XL with an adaptive learning rate mechanism to model the periodic and seasonal features in the time series data, and extract the periodic and seasonal features using the Transformer-XL model to obtain the output feature C of the time series data i at time step t. i，t .

[0219] In this embodiment, a Transformer-XL model is constructed, and the features of the time series data are extracted through the encoder part of the Transformer-XL model. Each sub-sequence after the division of the time series data is input into the Transformer-XL, and the output is the hidden state and the relative position encoding of the corresponding position. At the same time, in the encoder of the Transformer-XL, the relative importance between different positions is calculated using the self-attention mechanism to capture the context relationship. FIG. 7 is a flow chart showing the steps of feature extraction by the Transformer-XL model of the present application. Please refer to FIG. 7, and the following detailed description of each step of feature extraction by the Transformer-XL model.

[0220] Step E431: dividing the time series data into a plurality of sub-sequences of fixed length, and inputting the divided sub-sequences into the Transformer-XL encoder to obtain the hidden state of the time series data at each time step and the relative position encoding of the corresponding position; wherein the relative position encoding formula is as follows: j = f(j-t)

[0221] wherein time step t represents a given time step, time step j represents any time step for which the relative position with time step t needs to be calculated, p j represents the position information of time step t and time step j, and f represents an encoding function for encoding the relative position between time steps.

[0222] Step E432: calculating the attention score between the current time step and a certain time step using the self-attention mechanism, and calculating the relative importance vector between the current time step and a certain time step according to the calculated attention score, the formula is as follows:

[0223] wherein time step t is a given time step (default / current time step), time step j is any time step for which the relative importance vector with time step t needs to be calculated, o t represents the relative importance vector between time step t and time step j, a tj represents the attention score between the current time step t and time step j, V j represents the value vector of time step j Q t represents the query vector of the current time step t, and K j represents the key vector of time step j. Through this step, the self-attention mechanism is introduced to obtain the relative importance vector between the current time step and a certain time step, which is one of the important parameters for updating the time series model.

[0224] Step E433: Use the Transformer-XL encoder to extract the hidden state and the relative position encoding of the corresponding position at the current time step, and calculate the output features of the current time step as periodic and seasonal features based on the extracted hidden state and the relative position encoding of the corresponding position, as shown in the following formula:

[0225] Among them, C i,t This represents the output feature of time series data i at the current time step t. P represents the hidden state of time series data i at the current time step t. i，t = (p1, p2, ..., p T ) represents the relative position encoding of time series data i at the current time step t, and [,] represents a join operation.

[0226] Step E434: Update the model parameters using an adaptive learning rate mechanism, and apply the updated Transformer-XL model to the feature extraction of the next time step, as shown in the following formula:

[0227] Where p represents the Transformer-XL model parameters, such as weights and biases, and α represents the current learning rate. This represents the corrected first-moment estimate, and represents the average value of past gradients. The corrected second-moment estimate is represented by , which is the average of the squared gradients over time. ∈ represents a small constant to prevent the denominator from being zero. Thus, the initial learning rate α, and the hyperparameters β1, β2, and ∈ of the Adam algorithm can be updated. In each training iteration, the gradient of the loss function with respect to the model parameters is calculated, and the first-moment estimate m and the second-moment estimate v are calculated using the Adam algorithm, corrected, and then the Transformer-XL model with updated model parameters is used for feature extraction in the next time step.

[0228] Step E435: Repeat steps E432 to E434, extracting the output features at each time step, and using an adaptive learning rate to update the model parameters.

[0229] In this embodiment, through this step, the output features of the time series data at each time step are obtained sequentially as periodic and seasonal features C. i，t The model parameters are then updated adaptively to obtain the optimized Transformer-XL model.

[0230] Step E44: Connect the hidden states h of the LSTM neural network i，t and the output features C of the Transformer-XL model i，t, to obtain the output feature R of the time step t encoder i，t , as follows: R i，t = (h i,t , C i,t )

[0231] wherein (h i,t , C i,t ) represents a concatenation operation for connecting the hidden state h i，t of the LSTM neural network and the output feature C i，t of the Transformer-XL model in the feature dimension.

[0232] Specifically, in the decoder, the input of the time step t in the embodiment is composed of the predicted sequence y i，1：t-1 of the previous time step and the historical information R i，1：t output by the encoder. The historical information R i，1：t output by the encoder is represented by splicing the features from the first time step to the current time step, that is, R i，1：t = [R i，1 , R i,2 , …, R i,t ]. The output of the decoder is the predicted distribution R i，t of the time step t.

[0233] Step E45: decoding the output feature R i，t using the time step t decoder to obtain the predicted distribution result of the time series data i at the time step t, as follows: fc = W·(R i，1：t , y i,1：t-1 ) + b

[0234] wherein P(y i，t |R i,1:t , y i,1:t-1 ) represents the predicted distribution result of the time series data i at the time step t, R i，1：t = [R i，1 , R i,2 , …, R i,t ] represents the historical information of the time series data i at the time step t, y i,1:t-1 = [Y i，1 , Y i,2 , …, Y i，t-2 , Y i，t-1 ] represents the past predicted sequence of the time series data i at the time step t, fc represents the output of the full connection layer, W represents the weight matrix of the full connection layer, and b represents the bias vector of the full connection layer.

[0235] Specifically, in this embodiment, the prediction distribution result is obtained by using a fully connected layer to map the input to the dimension of the prediction distribution, and converting the original prediction score into a probability distribution by a softmax function. When making a prediction, first, the output fc of the fully connected layer is calculated, and then the prediction distribution is calculated using the output fc of the fully connected layer to obtain the prediction distribution result, P(y i，t |R i,1:t , y i,1:t-1 ).

[0236] Step E5: Evaluate the accuracy of the time series model using the test set, and adjust the training of the time series model according to the evaluation results to obtain a high-performance time series model.

[0237] In this embodiment, the accuracy of the model is evaluated using the above-mentioned test set, and the root mean square error is used for evaluation, which is expressed as follows:

[0238] Where err is the error value of the model, the smaller the better the performance of the model, y i,j is the predicted value of time series i at time step j, Z i,j is the true value of time series i at time step j, and T is the number of time steps in time series i.

[0239] Step E6: Use the trained time series model to predict future production plans and bill of materials.

[0240] In this embodiment, after obtaining the time series model through the above steps, the trained time series model is used to predict future production plans and bill of materials, and then the prediction results are used to organize mobile robots to move shelves carrying high-frequency materials to areas closer to the sorting station, thereby improving the efficiency of path planning and task allocation of mobile robots and reducing redundancy and resource waste in the sorting process.

[0241] The present specification also provides a computer readable medium storing computer program code, which, when executed by a processor, implements the mobile robot-based material sorting method as described above.

[0242] The present specification also provides a mobile robot-based material sorting device, comprising a memory storing instructions executable by a processor, and a processor for executing the instructions in the instruction memory to implement the mobile robot-based material sorting method as described above.

[0243] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0244] Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality, without reference to a specific

[0245] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0246] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0247] In one or more exemplary embodiments, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Claims

1. A material picking method based on a mobile robot, characterized in that, Includes the following steps: Step S1: Obtain the real-time production plan and bill of materials; the bill of materials includes the material code, material name, material quantity, and preparation time. Step S2: Divide the bill of materials into multiple task units suitable for a single operation of the mobile robot according to the material code, and assign corresponding picking stations; Step S3: Mark the sorting status of each task unit according to the material preparation time of each task unit, and assign the corresponding mobile robot according to the marked sorting status. Step S4: The mobile robot plans its path according to the assigned task unit and executes the assigned task unit according to the planned path, thereby moving the shelf to which the material in the assigned task unit belongs to the designated picking station for the staff to pick the required material.

2. The material picking method based on a mobile robot according to claim 1, characterized in that, The sorting status includes immediate sorting and pending sorting; wherein, after the material picking method based on mobile robots completes the sorting status labeling of the task unit, the task unit labeled as immediate sorting is assigned to the corresponding mobile robot, and the mobile robot transports the required materials in the task unit to the designated picking station for workers to pick.

3. The material picking method based on a mobile robot according to claim 2, characterized in that, When the material picking method based on mobile robots executes a task unit marked for immediate picking, it simultaneously checks whether there are materials located on the same shelf in the task unit marked for pending picking. If so, the task units corresponding to all the materials to be picked on the same shelf are merged into a new task unit, and the corresponding mobile robot and picking station are assigned.

4. The material picking method based on a mobile robot according to claim 1, characterized in that, After acquiring a task unit, the mobile robot performs path planning based on the shelf location, picking station location, and the established routes of other mobile robots currently in operation corresponding to the assigned task unit. The mobile robot uses an improved path planning method based on a counterfactual multi-agent policy gradient algorithm to plan the path, and then executes the assigned task unit according to the optimal movement path obtained from the planning.

5. The material picking method based on a mobile robot according to claim 4, characterized in that, When a mobile robot performs path planning using an improved path planning method based on a counterfactual multi-agent policy gradient algorithm, it first uses the improved counterfactual multi-agent policy gradient algorithm to perform reinforcement learning training on the movement paths of each mobile robot to obtain a policy network. Then, the learned policy network is applied to the path planning of each mobile robot to obtain the corresponding optimal movement path.

6. The material picking method based on a mobile robot according to claim 4, characterized in that, The material picking method based on mobile robots obtains a policy network for path planning by the mobile robot through reinforcement learning training through the following steps: Step C1: Perform environmental modeling based on the actual material picking site to obtain a one- or two-dimensional spatial grid model; Step C2: Define the corresponding state space and action space based on the constructed two-dimensional spatial mesh model, and construct a policy network for each mobile robot based on the defined state space and action space; Step C3: Sample the movement path of the mobile robot according to the predefined motion space, and obtain the positioning and status information of the mobile robot in real time according to the predefined state space; Step C4: Calculate the excitation value of the actual movement action taken by the mobile robot in the current state according to the predefined optimization excitation function; Step C5: Calculate the dominance function value of the actual movement action taken by the mobile robot in the current state based on the predefined dominance function and counterfactual baseline; Step C6: Calculate the policy gradient value based on the calculated advantage function value, and optimize and update the corresponding policy network parameters based on the calculated policy gradient value; Step C7: Determine whether the current state meets the predefined termination condition for reinforcement learning training; if yes, end the reinforcement learning training and output the learned policy network; if no, repeat steps C3 to C6 to continue reinforcement learning training.

7. The material picking method based on a mobile robot according to claim 6, characterized in that, In step C2, the material picking method based on the mobile robot, when defining the motion space of the mobile robot, also defines conflict state rules for the mobile robot based on the constructed two-dimensional spatial grid model. The conflict state rules are used to determine whether the mobile robot in the two-dimensional spatial grid model experiences a collision. Specifically, the material picking method based on the mobile robot measures whether a collision occurs according to the following defined conflict state rules: Conflict Rule 1: At the same time, two mobile robots occupy the same grid in the two-dimensional space grid model; Conflict Rule 2: At the same time, the movement paths of two mobile robots are on the same straight line, moving towards each other and both carrying shelves.

8. The material picking method based on a mobile robot according to claim 7, characterized in that, After defining the conflict state rules for the mobile robot, the material picking method based on the mobile robot sets up a corresponding conflict warning mechanism according to the defined conflict state rules, thereby avoiding collisions between the mobile robots in the two-dimensional spatial mesh model. This includes the following steps: Step D1: Encode the coordinates of each grid in the two-dimensional spatial grid model and define the status code of the mobile robot; Step D2: Monitor the coordinate codes and status codes of any pair of mobile robots in the two-dimensional spatial mesh model in real time; Step D3: Calculate the conflict warning score between any pair of mobile robots based on the coordinate codes and status codes monitored in real time; Step D4: Measure the expected collision conflict between the two mobile robots based on the calculated conflict warning score.

9. The material picking method based on a mobile robot according to claim 8, characterized in that, In step D3, when the material picking method based on mobile robots calculates the conflict warning score between any pair of mobile robots, a conflict warning score calculation formula is defined according to predefined mobile robot conflict state rules. Then, this formula is used to calculate the conflict warning score between any pair of mobile robots in the two-dimensional spatial mesh model. The conflict warning score calculation formula is as follows: Among them, S ij This represents the conflict warning score between any pair of mobile robots i and j in a two-dimensional spatial grid. and Let x and y represent the x and y coordinates of the mobile robot i, respectively. and Let x and y represent the x and y coordinates of the mobile robot j, respectively. and Let s represent the status codes of mobile robots i and j, respectively, where the value is 0 when the robot is unloaded and 1 when it is loaded onto a shelf.

10. The material picking method based on a mobile robot according to claim 9, characterized in that, After calculating the conflict warning score of any pair of mobile robots in the two-dimensional spatial grid model, the material picking method based on mobile robots can also use the conflict warning score of any pair of mobile robots in the two-dimensional spatial grid model to calculate the global conflict warning value. The calculated global conflict warning value is used to measure the collision expectation of all mobile robots in the entire two-dimensional spatial grid model. The calculation formula is as follows: Where fw represents the global conflict warning value. S ij This represents the conflict warning score between any pair of mobile robots i and j in a two-dimensional spatial grid. N represents the number of mobile robots in a two-dimensional spatial grid.

11. The material picking method based on a mobile robot according to claim 6, characterized in that, In step C4, the material picking method based on the mobile robot constructs an optimized incentive function based on the vector angle between the moving direction and the path direction corresponding to the actual moving action taken by the mobile robot, and assigns a corresponding reward value to the actual moving action taken by the mobile robot through the constructed optimized incentive function.

12. The material picking method based on a mobile robot according to claim 4, characterized in that, The mobile robot-based material picking method, when constructing the optimized incentive function, divides the complete path of the mobile robot to transport goods on the shelf into a first movement path and a second movement path. It calculates the vector angle between the movement direction and the path direction corresponding to the actual movement action taken by the mobile robot in real time by monitoring the real-time path vectors corresponding to the first and second movement paths. Then, it uses the calculated vector angle to calculate the reward value of the actual movement action taken by the mobile robot. The real-time path vector corresponding to the first movement path is the real-time path vector from the mobile robot's current position in front of the shelf to the target shelf. The real-time path vector corresponding to the second movement path is the real-time path vector from the mobile robot's own position to the target picking station after it has moved the shelf.

13. The material picking method based on a mobile robot according to claim 4, characterized in that, In step C5, when constructing the mobile robot's dominance function using the mobile robot-based material picking method, the global reward value brought by the default movement action in a given state is used as the counterfactual baseline cb. Then, the difference between the global reward value brought by the actual movement action taken by the mobile robot in the given state and the counterfactual baseline cb is used to construct the mobile robot's dominance function. The calculation formula is as follows: in, Represents mobile robot a i The actual movement action taken in a given state s The dominant function value, Included in the combined movement actions u taken by all mobile robots; Q(s, u) represents the global reward value brought about by the joint movement action u taken by all mobile robots in a given state s; Represents mobile robot a i The case where the default movement action u′ is selected; Represents mobile robot a i Historical data observation The probability that it will choose the default movement action u′; Indicates that, given state s, except for mobile robot a i Joint motion space of all mobile robots except Unchanged, mobile robot a i The global reward value when the default movement action u′ is selected.

14. The material picking method based on a mobile robot according to claim 6, characterized in that, In step C6, the material picking method based on the mobile robot calculates the advantage function value of the actual movement action taken by the mobile robot in the current state through step C5, then calculates the policy gradient value based on the calculated advantage function value, and then optimizes and updates the corresponding policy network parameters based on the calculated policy gradient value; wherein, the formula for calculating the policy gradient value is as follows: Among them, g k This represents the policy gradient in the k-th iteration. E π This represents the expected value under different strategies π. This indicates that for mobile robot a i The k-th policy network parameter θ k gradient operator, Represents mobile robot a i According to strategy Given observation data The logarithmic probability of the action u′ chosen under certain circumstances.

15. The material picking method based on a mobile robot according to claim 1, characterized in that, The material picking method based on mobile robots also trains a time series model. The trained time series model is used to predict future production plans and bills of materials. Then, based on the prediction results, the shelf positions with different movement frequencies are adjusted to improve material picking efficiency.

16. The material picking method based on a mobile robot according to claim 15, characterized in that, The mobile robot-based material picking method is trained to obtain a time series model for future production planning and bill of materials forecasting through the following steps: Step E1: Collect historical task unit data and clean the collected historical task unit data; Step E2: Extract the required feature values ​​from the cleaned historical task unit data; the feature values ​​include material code, logistics name, material quantity, production quarter, industry prosperity level, and remaining quantity; Step E3: Divide the extracted feature values ​​into a dataset, including a training set and a test set; Step E4: Select a time series model to be trained, and use the training set to train the selected time series model to obtain a time series model for predicting future production plans and bills of materials. Step E5: Use the test set to evaluate the accuracy of the time series model, and adjust the training time series model according to the evaluation results to obtain a high-performance time series model; Step E6: Use the trained time series model to predict future production plans and bills of materials.

17. The material picking method based on a mobile robot according to claim 16, characterized in that, In step E4, the mobile robot-based material picking method selects a time series model based on Transformer-XL and LSTM as the time series model for production planning and bill of materials. The method obtains the predicted distribution results for time step t using the Transformer-XL and LSTM-based time series model, including the following steps: Step E41: Establish a conditional distribution model as the input to the time-step t encoder. Encode the input training set using the conditional distribution model to obtain time series data. The expression is as follows: P(Z i，t：T |Z i,1:t-1 ,h i,1:t-1 ) Among them, Z i,t：T =[Z i,t Z i,t+1 , ..., Z i,T ] represents the future state of time series data i at time step t. Z i,1:t-1 =[Z i，1 Z i,2 ,…,Z i,t-2 Z i,t-1 ] represents the past state of time series data i at time step t. h i,1：t-1 This represents the hidden state of time series data i before time step t; Step E42: Use an LSTM neural network to model the temporal dependencies in the time series data, and calculate the temporal dependencies using the LSTM neural network to obtain the hidden state h of the time series data i at time step t. i，t ; Step E43: Use Transformer-XL with an adaptive learning rate mechanism to model the periodic and seasonal features in the time series data, and extract the periodic and seasonal features using the Transformer-XL model to obtain the output feature C of the time series data i at time step t. i,t ; Step E44: Connect the hidden states h of the LSTM neural network i，t and the output features C of the Transformer-XL model i，t The output characteristics R of the encoder at time step t are obtained. i，t The formula is as follows: R i，t =(h i,t ,C i,t ) Among them, (h i,t C i,t The ) indicates a connection operation, used to connect the hidden states h of the LSTM neural network. i，t and the output features C of the Transformer-XL model i，t Connect along the feature dimension; Step E45: Use the time-step t decoder to process the output feature R i，t Decoding is performed to obtain the predicted distribution of time series data i at time step t, as shown in the following formula: fc=W·(R i,1:t ,y i,1:t-1 )+b Wherein, P(y i，t |R i,1:t y i,1:t-1 () represents the predicted distribution of time series data i at time step t. R i,1:t =[R i，1 R i,2 ,…,R i,t ] represents the historical information of time series data i at time step t. y i,1:t-1 =[Y i，1 Y i,2 ,…,Y i，t-2 Y i，t-1 ] represents the past predicted sequence of time series data i at time step t. fc represents the output of the fully connected layer. W represents the weight matrix of the fully connected layer. b represents the bias vector of the fully connected layer. exp represents the exponential function. `sum` represents the summation of all elements in a vector.

18. The material picking method based on a mobile robot according to claim 17, characterized in that, In step E42, the hidden state h of time series data i at time step t is calculated through the following steps. i，t : Step 1: Calculate the output f of the forget gate i，t The formula is as follows: f i，t =σ(W f ·(h i，t-1 ,Z i，t )+b f ) Among them, W f The weight matrix of the forget gate, b f Let σ represent the bias vector of the forget gate, and let σ represent a sigmoid activation function. Step 2: Calculate the output i of the input gate. i，t The formula is as follows: i i,t =σ(W i ·(h i，t-1 ,Z i,t )+b i ) Among them, W i Let b represent the weight matrix of the input gate. i σ represents the input gate bias vector, and σ represents a sigmoid activation function. Step 3: Calculate the state g of the candidate memory cell i,t The formula is as follows: rice i，t = tanh(W c (h) i，t-1 Z i,t )+b c ) Among them, W c The weight matrix representing the memory unit, b c represents the bias vector of the memory unit, and tanh represents a hyperbolic tangent activation function; Step 4: Output f based on the forget gate i，t and the output i of the input gate i,t To update the memory cell state c i，t The formula is as follows: c i,t =f i，t ⊙c i,t-1 +i i,t ⊙g i,t Where ⊙ represents element-wise multiplication, c i,t-1 This represents the past memory unit state of time series data i at time step t; Step 5: Calculate the output o of the output gate i，t The formula is as follows: about i，t =σ(W o ·(h i，t-1 ,WITH i,t )+b o ) Among them, W o Let b represent the weight matrix of the output gate. o σ represents the bias vector of the output gate, and σ represents a sigmoid activation function. Step 6: Based on the updated memory cell state c i，t The output of the output gate is calculated. i，t To update the hidden state h of time series data i at time step t. i，t The formula is as follows: h i,t =o i,t ⊙tanh(c i,t ) Where ⊙ represents element-wise multiplication, and tanh represents a hyperbolic tangent activation function.

19. The material picking method based on a mobile robot according to claim 17, characterized in that, In step E43, the mobile robot-based material picking method extracts periodic and seasonal features through the following steps and adaptively updates the Transformer-XL model parameters: Step E431: Divide the time series data into multiple fixed-length subsequences and input the subsequences into the Transformer-XL encoder to obtain the hidden state of the time series data at each time step and the relative position encoding of the corresponding position; the relative position encoding formula is as follows: p j =f(j-t) Where time step t represents a given time step. Time step j represents any time step whose position is relative to time step t that needs to be calculated. p j This indicates the relative position information of time step t and time step j. f represents an encoding function used to encode the relative positions between time steps; Step E432: Calculate the attention score between the current time step and the previous time step using a self-attention mechanism, and calculate the relative importance vector between the current time step and the previous time step based on the calculated attention score, as shown in the following formula: Among them, o t This represents a vector indicating the relative importance between time step t and time step j. a tj This represents the attention score between the current time step t and time step j. V j This represents the value vector of time step j. Q t This represents the query vector at the current time step t. K j The key vector representing time step j; Step E433: Use the Transformer-XL encoder to extract the hidden state and the relative position encoding of the corresponding position at the current time step, and calculate the output features of the current time step as periodic and seasonal features based on the extracted hidden state and the relative position encoding of the corresponding position, as shown in the following formula: Among them, C i,t This represents the output feature of time series data i at the current time step t. This represents the hidden state of time series data i at the current time step t. P i，t = (p1, p2, ..., p T This represents the encoding of the relative position of time series data i at the current time step t. [,] indicates a join operation; Step E434: Update the model parameters using an adaptive learning rate mechanism, and apply the updated Transformer-XL model to the feature extraction of the next time step, as shown in the following formula: Where p represents the Transformer-XL model parameters, such as weights and biases. α represents the current learning rate. This represents the corrected first-moment estimate, where represents the average value of past gradients. This represents the corrected second-moment estimate, where represents the average of the squared gradients over time. ∈ represents a small constant used to prevent the denominator from being zero; Step E435: Determine whether the output features of all time steps have been extracted; if yes, end the feature extraction; if no, repeat steps E432 to E434 to extract the output features of each time step and use the adaptive learning rate to update the model parameters.

20. A computer-readable medium storing computer program code, characterized in that, The computer program code, when executed by a processor, implements the method as described in any one of claims 1-19.

21. A material picking device based on a mobile robot, characterized in that, include: Memory is used to store instructions that can be executed by the processor; as well as A processor for executing the instructions to implement the method as claimed in any one of claims 1-19.

Images