A target allocation and decision-making algorithm based on AGV collaborative handling

Abstract

The present invention discloses a target allocation and decision-making algorithm based on AGV collaborative handling. The key points of the technical solution include establishing a dominance matrix and a target decision matrix based on the number of AGVs and the number of targets to be handled, and establishing a target allocation optimization function, analyzing the AGV handling environment and establishing a probability matrix and a penetration action matrix, and establishing an action decision objective function and action decision constraints; linear weighted integration, establishing a unified objective function and performing reinforcement learning and designing a reward function to form a multi-stage decision problem model; designing a joint reward function for action rewards; setting a selection strategy and a backtracking update formula and improving them, and solving the decision problem model. The present invention has fast calculation speed, short training time, and good convergence effect, thereby achieving the effect of accurate intelligent calculation allocation and action decision-making.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent handling equipment, and more specifically to an AGV-based collaborative handling target allocation and decision-making algorithm. Background Art

[0002] AGV is an unmanned transport vehicle that detects the surrounding environment by installing a rotatable laser scanner to achieve vehicle positioning, orientation, obstacle avoidance and path planning. As a key transportation equipment in cargo handling, all AGVs form a transportation network and complete the transportation activities between the starting point and the target unit under the command of the control system. Therefore, how to use the control system to perform AGV target allocation and action decision-making will be an important issue that needs to be solved urgently.

[0003] The AGV target allocation and action decision problem is an extension of the vehicle routing problem (VRP), which is an NP-hard problem. Therefore, large-scale problems are difficult to solve using traditional exact solution algorithms. This problem is much more complicated than traditional VRP problems and scheduling problems due to the complex scenarios and the characteristics of multiple constraints, multiple objectives, and uncertainty. At present, the scheduling method mainly based on experience and rules leads to problems such as low on-time delivery rate, low AGV utilization rate, and long delivery time. Therefore, it is urgent to combine the optimization scheduling theory with the characteristics of AGV transportation and design an effective optimization theory and method to solve this problem.

[0004] In summary, the study of AGV scheduling problem has very important practical significance. This paper will focus on the AGV scheduling problem, integrate target allocation and action decision-making under multiple constraints and maneuvers, and design an effective intelligent algorithm to solve the problem. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an AGV collaborative handling target allocation and decision-making algorithm, which has fast calculation speed, short training time, and good convergence effect, thereby achieving the effect of intelligent calculation, accurate allocation and action decision-making.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An AGV collaborative handling target allocation and decision-making algorithm includes the following steps:

[0008] S1: Based on the number of AGVs and the number of targets to be transported, a dominance matrix and a target decision matrix are established respectively, and a target allocation optimization function is established based on the dominance matrix and the target decision matrix;

[0009] S2: Analyze the AGV handling environment and establish a probability matrix, optimize the penetration action matrix based on the probability matrix, and establish the action decision objective function based on the probability matrix and the penetration action matrix;

[0010] S3: Establish action decision constraints based on the action decision objective function;

[0011] S4: linearly weighted integration of the target allocation optimization function and the action decision objective function to establish a unified objective function;

[0012] S5: Perform reinforcement learning, model the Markov decision process for target allocation and decision-making in stages to construct state space and action space, and design the reward function based on the unified objective function to form a multi-stage decision-making problem model;

[0013] S6: Design a joint reward function based on a unified objective function;

[0014] S7: Establish a corresponding Monte Carlo tree for each AGV, search each Monte Carlo tree, form a joint action set for the search results, and substitute them into the joint reward function for evaluation and obtain the reward value;

[0015] S8: Set the selection strategy and backtracking update formula, return the reward value to each Monte Carlo tree, improve the algorithm's selection strategy and backtracking update formula, and solve the decision problem model.

[0016] As a further improvement of the present invention, the step S1 is specifically as follows:

[0017] In the AGV collaboration process, a total of N U AGV and N Tar Targets to be transported;

[0018] The established dominance matrix is ​​A ij , i in the matrix (i=1,2,....,N u ) row j (j=1,2,....,N Tar ) column represents the comprehensive advantage of the i-th AGV to the target j;

[0019] The target decision matrix established is X ij , the element value in the matrix is ​​1, which means that the i-th AGV is assigned to the target j, otherwise the matrix element value is 0;

[0020] The target allocation optimization function established is:

[0021]

[0022] Among them: J dis,i Assign optimization function to the target of the i-th AGV.

[0023] As a further improvement of the present invention, step S1 also includes:

[0024] In the process of target allocation, each AGV can only be allocated to one target, and each target is allocated at least one AGV. A constraint model is established for this:

[0025]

[0026] As a further improvement of the present invention, in step S2, due to the AGV handling process, different targets make different maneuvers to pass through obstacles based on different environments, and different numbers of air defense areas are set for different targets, that is, target j has a total of N d,j There are N obstacle areas in total for AGV. A actions to choose from, and select an action to penetrate each obstacle area, and establish the probability matrix P mk , the matrix represents the execution action m (m=1,2,....,N A ) in the obstacle area k (k=1,2,....N d,j ) can be reduced by the probability percentage, and the probability matrix is ​​optimized to obtain the penetration matrix M km This matrix indicates that if the AGV selects the penetration action m in the obstacle area k, the matrix element is 1, otherwise it is 0;

[0027] The action decision objective function of AGV is established as:

[0028]

[0029] Among them: J pene,i Represents the action decision objective function of the i-th AGV.

[0030] As a further improvement of the present invention, step S2 further includes constraining the number of action selections during the AGV handling process according to the physical characteristics of the AGV, and defining that each action selection shall not exceed b1 times, and establishing an action decision constraint:

[0031]

[0032] Optimal selection of decision variables is performed in discrete decision space based on action decision constraints.

[0033] As a further improvement of the present invention, the step S4 is specifically as follows:

[0034] The i-th AGV establishes a unified objective function based on the target allocation function and the action decision objective function:

[0035]

[0036] The multi-objective optimization of AGV cluster is transformed into single-objective optimization using linear weighted method:

[0037]

[0038] in: is the weight factor, and

[0039] As a further improvement of the present invention, step S4 further includes establishing an objective function constraint:

[0040]

[0041] The objective function constraint is a combination of the constraint model and the action decision constraint.

[0042] As a further improvement of the present invention, the step S5 is specifically as follows:

[0043] The state space is established as:

[0044]

[0045] Where: S dis Assign a state to the target, S r is the obstacle 1 area, is the obstacle 2 area passed in sequence, is the obstacle 3 area passed in sequence, is the obstacle 4 area passed in sequence. Combining the state space of all targets, the total state space can be expressed as:

[0046]

[0047] AGV has different actions according to the status of different stages. In the target allocation stage, the action is target selection. AGV can only select one target. The result is represented by a discretized vector: For the existence of N Tar When there are N targets, there is a target allocation stage with N Tar Actions to choose from;

[0048] There are five kinds of breakthrough actions in the obstacle passing stage. AGV selects an action in a certain obstacle area state and represents the five kinds of breakthrough actions with vectors. The elements in the vector represent action 1, action 2, action 3, action 4 and action 5 respectively. The reward function is established:

[0049]

[0050] As a further improvement of the present invention, the joint reward function is:

[0051]

[0052] Among them, when the target decision matrix and the penetration matrix do not meet the constraints, a penalty of -1 is given, otherwise the reward value is the unified objective function value.

[0053] As a further improvement of the present invention, the selection decision is:

[0054]

[0055] Where: v father is the parent node of the node v being calculated, C p is a constant used to weigh the orientation relationship between search and exploitation, Q(v) is the result based on all AGV allocation and penetration;

[0056] The backtracking update formula is:

[0057]

[0058] Where: N new (v) is the new node, N old (v) is the old node, Q new (v) is the new penetration result, Q old (v) is the old penetration result, and ΔQ is the penetration result difference.

[0059] The beneficial effects of the present invention are as follows: by considering the advantages of AGV itself, the target value and the action probability, the target allocation optimization function and the action decision objective function are established respectively, and the two are integrated to form a unified objective function for multi-AGV collaborative task planning, and the state space and action space are constructed in stages in the reinforcement learning framework, and the reward function is designed according to the unified objective function. Finally, an improved Monte Carlo tree search reinforcement learning algorithm is proposed to solve the AGV target allocation and decision-making problems under the maximum benefit of the unified objective function, and the switching and maintenance of the AGV formation configuration are realized, so as to avoid collisions between AGVs and external obstacles during the formation process, and ensure the communication connectivity between adjacent AGV individuals. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figures 1 to 5 Search depth graph for partial AGV search tree;

[0061] Figures 6 to 10 J for some AGVs i value. DETAILED DESCRIPTION

[0062] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The same parts are represented by the same reference numerals. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "bottom surface" and "top surface", "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0063] refer to Figure 1 As shown in the figure, a specific implementation of the present invention is a target allocation and decision-making algorithm based on AGV collaborative transportation, including the following steps:

[0064] S1: Based on the number of AGVs and the number of targets to be transported, a dominance matrix and a target decision matrix are established respectively, and a target allocation optimization function is established based on the dominance matrix and the target decision matrix;

[0065] S2: Analyze the AGV handling environment and establish a probability matrix, optimize the penetration action matrix based on the probability matrix, and establish the action decision objective function based on the probability matrix and the penetration action matrix;

[0066] S3: Establish action decision constraints based on the action decision objective function;

[0067] S4: linearly weighted integration of the target allocation optimization function and the action decision objective function to establish a unified objective function;

[0068] S5: Perform reinforcement learning, model the Markov decision process for target allocation and decision-making in stages to construct state space and action space, and design the reward function based on the unified objective function to form a multi-stage decision-making problem model;

[0069] S6: Design a joint reward function based on a unified objective function;

[0070] S7: Establish a corresponding Monte Carlo tree for each AGV, search each Monte Carlo tree, form a joint action set for the search results, and substitute them into the joint reward function for evaluation and obtain the reward value;

[0071] S8: Set the selection strategy and backtracking update formula, return the reward value to each Monte Carlo tree, improve the algorithm's selection strategy and backtracking update formula, and solve the decision problem model.

[0072] The step S1 is specifically as follows:

[0073] In the AGV collaboration process, a total of N U AGV and N Tar Targets to be transported;

[0074] The established dominance matrix is ​​A ij, i in the matrix (i=1,2,....,N u ) row j (j=1,2,....,N Tar ) column represents the comprehensive advantage of the i-th AGV to the target j;

[0075] The target decision matrix established is X ij , the element value in the matrix is ​​1, which means that the i-th AGV is assigned to the target j, otherwise the matrix element value is 0;

[0076] The target allocation optimization function established is:

[0077]

[0078] Among them: J dis,i The target allocation optimization function for the i-th AGV is used. During the target allocation process, each AGV can only be allocated to one target, and each target must be allocated at least one AGV. A constraint model is established for this:

[0079]

[0080] Each AGV is constrained and controlled through the constraint model.

[0081] During the AGV handling process, different targets pass through different environments, which are mainly composed of different obstacles. The AGV can make different maneuvers to pass through the obstacles. The original blocking probability of obstacles and the maneuvering probability are shown in the following table:

[0082] Table 1 Original blocking probability table of each obstacle

[0083]

[0084]

[0085] Table 2 AGV action to obstacle blocking reduction ratio table

[0086]

[0087] Due to the AGV handling process, different targets make different maneuvers based on different environments to pass obstacles. Different targets are set to deploy different numbers of air defense areas, that is, target j has a total of N d,j There are N obstacle areas in total for AGV. A actions to choose from, and select an action to penetrate each obstacle area, and establish the probability matrix P mk , the matrix represents the execution action m (m=1,2,....,N A ) in the obstacle area k (k=1,2,....N d,j) can be reduced by the probability percentage, and the probability matrix is ​​optimized to obtain the penetration matrix M km This matrix indicates that if the AGV selects the penetration action m in the obstacle area k, the matrix element is 1, otherwise it is 0;

[0088] The action decision objective function of AGV is established as:

[0089]

[0090] Among them: J pene,i Represents the action decision objective function of the i-th AGV.

[0091] According to the physical characteristics of AGV, the number of action selections during AGV handling is constrained, and each action selection is defined to be no more than b1 times, and the action decision constraints are established:

[0092]

[0093] Based on the action decision constraints, the optimal selection of decision variables is carried out in the discrete decision space. The solution form is consistent with the reinforcement learning solution process, and the Monte Carlo tree search algorithm is used to uniformly solve the multi-stage decision problem.

[0094] The i-th AGV establishes a unified objective function based on the target allocation function and the action decision objective function:

[0095]

[0096] The multi-objective optimization of AGV cluster is transformed into single-objective optimization using linear weighted method:

[0097]

[0098] in: is the weight factor, and

[0099] Establish objective function constraints:

[0100]

[0101] The objective function constraint is a combination of the constraint model and the action decision constraint.

[0102] The problem to be solved by the present invention is modeled using the Markov decision process in order to improve the algorithm solution process. The state is the amount of information that the agent uses to represent its own characteristics. In the iterative process, the agent selects actions based on the state, so the selection of the state has a very important impact on the quality of the training results. The problem to be solved by the present invention consists of two stages: target allocation and action decision. Target allocation is the prerequisite stage of action decision. Only after the AGV has selected the target can the action selection be performed in sequence according to the obstacle area of ​​the target. Therefore, for this problem, the target allocation is used as the prerequisite state and combined with the subsequent obstacle area state to perform unified state space modeling. According to different targets j, the state space is established as:

[0103]

[0104] Where: S dis Assign a state to the target, S r is the obstacle 1 area, is the obstacle 2 area passed in sequence, is the obstacle 3 area passed in sequence, is the obstacle 4 area passed in sequence. Combining the state space of all targets, the total state space can be expressed as:

[0105]

[0106] AGV has different actions according to the status of different stages. In the target allocation stage, the action is target selection. AGV can only select one target. The result is represented by a discretized vector: For the existence of N Tar When there are N targets, there is a target allocation stage with N Tar Actions to choose from;

[0107] There are five kinds of breakthrough actions in the obstacle passing stage. AGV selects an action in a certain obstacle area state and represents the five kinds of breakthrough actions with vectors. The elements in the vector represent action 1, action 2, action 3, action 4 and action 5 respectively. The reward function is established:

[0108]

[0109] The reward function is the core part of reinforcement learning, which guides the agent to learn. For this problem, only after the agent reaches the final state, that is, after deciding the complete target allocation matrix X and action matrix M, can the above decision results be evaluated according to the unified objective function.

[0110] Based on the unified objective function and objective function constraints, the joint reward function is designed as follows:

[0111]

[0112] Among them, when the target decision matrix and the penetration matrix do not meet the constraints, a penalty of -1 is given, otherwise the reward value is the unified objective function value.

[0113] A Monte Carlo tree is established for each AGV. Each tree performs an independent search. The search results form a joint action set and are substituted into the joint reward function for evaluation. Finally, the reward value is returned to each tree, and the result path node of each tree is backtracked and updated.

[0114] In order to avoid the influence of the average simulation results on the solution of the problem, the selection decision is established as:

[0115]

[0116] Where: v father is the parent node of the node v being calculated, C p is a constant used to weigh the orientation relationship between search and exploitation, Q(v) is the result based on all AGV allocation and penetration;

[0117] The backtracking update formula is:

[0118]

[0119] Where: N new (v) is the new node, N old (v) is the old node, Q new (v) is the new penetration result, Q old (v) is the old penetration result, and ΔQ is the penetration result difference.

[0120] The present invention takes 20 AGVs and 5 targets as an example to verify the effectiveness of the proposed unified objective function and improved algorithm. Considering the different target values ​​and obstacle areas, the target value advantage and situation information table are given. At the same time, each AGV has a different initial heading angle and handling time for each target. Each action selection is set to no more than b1=2 times, and the weight factors ρ1, ρ2,...., ρ 50 All are taken as 1 / 50, in the algorithm The algorithm runs on an i5-9400F processor, 2.90 GHz, the simulation environment is python 3.6, and the total number of iterations is set to 20,000.

[0121] After 20,000 steps of learning and training, the algorithm converged and the total training time was 162.1854 seconds. The target allocation matrix obtained from the training and the action matrix of each AGV are represented in the following table. In order to facilitate the representation of the action results, numbers 1 to 5 are used to represent actions 1 to 5 respectively, and a one-dimensional vector is used to represent the action sequence made by the AGV when passing through the obstacle area in sequence.

[0122] Table 2 Results of the proposed algorithm

[0123]

[0124]

[0125] During the training process, each AGV performs independent search based on its own heuristic factor. Due to limited space, this paper takes AGV U1, U5, U10, U15 and U20 as examples to give the search depth and single AGV objective function value j during the training process. i data.

[0126] Figures 1 to 5 The depth data of tree search of some AGVs during training is given. It can be seen from the figure that after 10,000 steps of training, the tree search can reach the maximum depth, indicating that the AGV strategy can obtain an optimal solution range based on the reward function and rollout random simulation in the early training. In the subsequent training, the search strategy mainly searches for the optimal solution in this range.

[0127] Figures 6 to 10 The following are some examples of AGVs during training. i In the early stage of training, the search depth is shallow, and Monte Carlo is mainly used for target shooting. The search tree relies on random simulation to obtain the reward value. However, as the search depth increases, the tree stores the better solution obtained in the node information and relies on heuristic factors to continuously search for the optimal solution. In the later stage of training, the search process oscillates. This is because the exploration part of the selection strategy is playing a role, avoiding the optimal solution recorded by the current node and selecting other nodes with fewer visits, so as to avoid falling into the local optimum.

[0128] Working principle and effect:

[0129] By considering the advantages of AGV itself, the target value and the action probability, the target allocation optimization function and the action decision objective function are established respectively, and the two are integrated to form a unified objective function for multi-AGV collaborative task planning. The state space and action space are constructed in stages in the reinforcement learning framework, and the reward function is designed according to the unified objective function. Finally, an improved Monte Carlo tree search reinforcement learning algorithm is proposed to solve the AGV target allocation and decision-making problems under the maximum benefit of the unified objective function, and the switching and maintenance of the AGV formation configuration are realized, avoiding collisions between AGVs and external obstacles during the formation process, and ensuring the communication connectivity between adjacent AGV individuals. The results show that the proposed algorithm has better convergence effect and short training time.

[0130] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An AGV collaborative transport target allocation and decision-making algorithm, characterized in that: The following steps are involved: S1: Based on the number of AGVs and the number of targets to be transported, a dominance matrix and a target decision matrix are established respectively, and a target allocation optimization function is established based on the dominance matrix and the target decision matrix; S2: Analyze the AGV handling environment and establish a probability matrix, optimize the penetration action matrix based on the probability matrix, and establish the action decision objective function based on the probability matrix and the penetration action matrix; S3: Establish action decision constraints based on the action decision objective function; S4: linearly weighted integration of the target allocation optimization function and the action decision objective function to establish a unified objective function; The step S4 is specifically as follows: The i-th AGV establishes a unified objective function based on the target allocation function and the action decision objective function: The multi-objective optimization of AGV cluster is transformed into single-objective optimization using linear weighted method: in: is the weight factor, and ; S5: Perform reinforcement learning, model the Markov decision process for target allocation and decision-making in stages to construct state space and action space, and design the reward function based on the unified objective function to form a multi-stage decision-making problem model; S6: Design a joint reward function based on a unified objective function; S7: Establish a corresponding Monte Carlo tree for each AGV, search each Monte Carlo tree, form a joint action set for the search results, and substitute them into the joint reward function for evaluation and obtain the reward value; S8: Set the selection strategy and backtracking update formula, return the reward value to each Monte Carlo tree, improve the algorithm's selection strategy and backtracking update formula, and solve the decision problem model.

2. The AGV collaborative transport target allocation and decision-making algorithm according to claim 1 is characterized by: The step S1 is specifically as follows: In the AGV collaboration process, a total of AGV and Targets to be transported; The established dominance matrix is ​​A ij , i(i=1,2,....,N in the matrix u ) row j(j=1,2,....,N Tar ) column represents the comprehensive advantage of the i-th AGV to the target j; The target decision matrix established is X ij , the element value in the matrix is ​​1, which means that the i-th AGV is assigned to the target j, otherwise the matrix element value is 0; The target allocation optimization function established is: in: Assign optimization function to the target of the i-th AGV.

3. The AGV collaborative transport target allocation and decision-making algorithm according to claim 2 is characterized by: Step S1 also includes: In the process of target allocation, each AGV can only be allocated to one target, and each target is allocated at least one AGV. A constraint model is established for this: 。 4. The AGV collaborative transport target allocation and decision-making algorithm according to claim 1 is characterized by: In step S2, due to the AGV handling process, different targets make different maneuvers to pass through obstacles based on different environments, and different targets are set to deploy different numbers of air defense areas, that is, target j has a total of N d,j There are N obstacle areas in total for AGV. A actions to choose from, and choose an action to break through each obstacle area, and establish a probability matrix , the matrix represents the execution action m (m=1,2,....,N A ) in the obstacle area k(k=1,2,....N d,j ) can reduce the probability percentage, and optimize the probability matrix to obtain the penetration matrix M km This matrix indicates that if the AGV selects the penetration action m in the obstacle area k, the matrix element is 1, otherwise it is 0; The action decision objective function of AGV is established as: in: Represents the action decision objective function of the i-th AGV.

5. The AGV collaborative transport target allocation and decision-making algorithm according to claim 3 is characterized by: The step S2 also includes constraining the number of action selections during the AGV handling process according to the physical characteristics of the AGV, and defining that each action selection shall not exceed b1 times, and establishing action decision constraints: Optimal selection of decision variables is performed in discrete decision space based on action decision constraints.

6. The AGV collaborative transport target allocation and decision-making algorithm according to claim 5 is characterized by: The step S4 also includes establishing objective function constraints: The objective function constraint is a combination of the constraint model and the action decision constraint.

7. The AGV collaborative transport target allocation and decision-making algorithm according to claim 6 is characterized by: The step S5 is specifically as follows: The state space is established as: in: Assign a state to a target, is the obstacle 1 area, is the obstacle 2 area passed in sequence, is the obstacle 3 area passed in sequence, For the obstacle 4 areas passed in sequence, the state space of all targets is integrated, and the total state space can be expressed as: AGV has different actions according to the status of different stages. In the target allocation stage, the action is target selection. AGV can only select one target. The result is represented by a discretized vector: , for the existence of N Tar When there are N targets, there is a target allocation stage with N Tar Actions to choose from; There are five kinds of breakthrough actions in the obstacle passing stage. AGV selects an action in a certain obstacle area state and represents the five kinds of breakthrough actions with vectors. The elements in the vector represent action 1, action 2, action 3, action 4 and action 5 respectively. The reward function is established: 。 8. The AGV collaborative transport target allocation and decision-making algorithm according to claim 7 is characterized by: The joint reward function is: Among them, when the target decision matrix and the penetration matrix do not meet the constraints, a penalty of -1 is given, otherwise the reward value is the unified objective function value.

9. The AGV collaborative transport target allocation and decision-making algorithm according to claim 1 is characterized by: The selection strategy is: in: is the parent node of the node v being calculated, C p is a constant used to weigh the orientation relationship between search and utilization, It is based on the results of all AGV allocation and penetration; The backtracking update formula is: in: For the new node, For old nodes, For the new breakthrough results, For the old breakthrough results, The penetration results were poor.

Images