A multi-robot joint scheduling method for flexible manufacturing system
Through reinforcement learning to adjust genetic algorithm parameters, combined with multi-layer coding and cross-mutation strategies, the joint scheduling problem of machines and AGV in flexible manufacturing systems is solved, and the production efficiency and flexibility of flexible manufacturing systems are improved.
Patent Information
- Application Number
- CN202111175163.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-10-09
AI Technical Summary
Traditional workshop scheduling algorithms are difficult to deal with the joint scheduling problem of machines and AGV in flexible manufacturing systems, resulting in limited system flexibility and productivity, and there is no research on solving using reinforcement learning and genetic algorithms.
The reinforcement learning method is used to dynamically adjust the key parameters of the genetic algorithm, and the iterative search process of the genetic algorithm is controlled through the Q-Learning algorithm, combining multi-layer coding and cross-mutation strategies to optimize the multi-robot joint scheduling algorithm.
It significantly improves the algorithm solution efficiency, realizes efficient joint scheduling in the workshop, and improves the production efficiency and flexibility of the flexible manufacturing system.
Smart Images

Figure CN113935610B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of flexible manufacturing, and in particular to a multi-robot joint scheduling method for a flexible manufacturing system. Background Art
[0002] With the continuous refinement of user needs and the flexibility of manufacturing, the manufacturing model has gradually changed from a product-driven model to a user-driven model. While gradually improving production efficiency, controlling production costs, shortening production cycles and improving production quality, flexible manufacturing systems (FMS) can quickly respond to the user-driven global manufacturing market. With the development of production automation and information integration, modern industrial production has put forward higher requirements for logistics systems. Automated guided vehicles (AGVs) are automatic transportation equipment that can complete specified transportation tasks under a given path and scene information layout. The AGV transportation management system is an important part of the AGV system and plays an important role in the management and scheduling of AGVs. Due to the close coupling relationship between processing and transportation, the simultaneous scheduling of machines and AGVs in FMS is of great significance to improving system flexibility and production efficiency. Then, compared with the traditional workshop scheduling problem, this joint scheduling problem adds a new type of scheduling subject. The traditional workshop scheduling algorithm is no longer competent for this type of scheduling problem because it only schedules the processing machines in the workshop. From the literature search, it is found that there are currently few studies on the solution to this problem, and no research on using reinforcement learning and genetic algorithms to solve this problem has been found. Summary of the invention
[0003] In view of this, the present invention provides a multi-robot joint scheduling method for a flexible manufacturing system, which can dynamically adjust the key parameters of the genetic algorithm through a reinforcement learning method, control the iterative search process of the genetic algorithm, thereby improving the solution efficiency of the algorithm and realizing efficient joint scheduling of workshops.
[0004] To achieve the above object, the technical solution of the present invention comprises the following steps:
[0005] First, a three-layer encoding method is used to match the machine sequence, process sequence and AGV sequence: each gene in the process sequence represents the number of the processed workpiece, and the number of times the number appears from the left side of the process sequence to the position represents the number of the processing process; each gene in the machine sequence represents the number of the machine that processes the corresponding process; each gene in the AGV sequence represents the AGV number that transports the corresponding workpiece to the corresponding machine;
[0006] Secondly, in terms of decoding method, the process is decoded strictly in the order of appearance of the processes from front to back in the encoding, and the process constraints and processing constraints are met;
[0007] Step 1: Algorithm initialization:
[0008] Step 1.1: Genetic algorithm initialization includes: population size N, establishment of random initial population, number of iterations g_max, selection ratio Ps, crossover probability Pc, and mutation probability Pm;
[0009] Step 1.2: The initialization content of the Q-Learning algorithm includes: Q-Table, state set S, action set A, Q-table update coefficient β and action selection coefficient ε; the elements in the state set S are selected by extracting state data from the genetic algorithm population. The state data includes population diversity sa, population convergence speed sv and population convergence trend sc. The combination of discretized state data is sd =<sda,sdv,sdc> , where sda is the discrete population diversity value, sdv is the discrete population convergence speed value, sdc is the discrete population convergence trend, and the discretized state data is used as the element of the state set S;
[0010] The elements in the action set A are selected by increasing or decreasing the selection ratio Ps, crossover probability Pc, and mutation probability Pm of the genetic algorithm; the Q-Table uses the state set S as the row vector and the action set A as the column vector to establish an empty initial Q-Table with all elements being 0;
[0011] Step 2: Calculate the fitness values of all individuals in the current population and record the best individual that has been searched so far;
[0012] Step 3: Calculate the state data of the current generation population according to the fitness value, including population diversity sa, population convergence speed sv, and population convergence trend sc, compare it with the state data of the previous generation population to obtain the reward value R, update the Q-Table, and then discretize the state data of the current generation population;
[0013] Step 4: According to sd=<sda,sdv,sdc> The discrete state value, the element value of the corresponding row of the Q-Table and the action value a selected by the ε-greedy strategy;
[0014] Step 5: According to the selected action value a, execute the corresponding genetic algorithm parameter adjustment action;
[0015] Step 6: Execute the selection operator, crossover operator and mutation operator of the genetic algorithm according to the adjusted parameters to form a new population;
[0016] Step 7: Determine whether the population iteration has reached the maximum number of iterations. If the conditions are met, output the best individual and fitness value; otherwise, jump to step 2.
[0017] Furthermore, the calculation process of population diversity sa, population convergence speed sv, and population convergence trend sc in step 1.2 includes the following steps:
[0018] Step 201: Calculate the average fitness value of all individuals in the k-th generation population as shown in formula (2), where Y(k) represents the average fitness value of individuals in the k-th generation population, and f(xjk) represents the fitness value of the j-th individual in the k-th generation population;
[0019]
[0020] Step 202: Calculate the average fitness value of all individuals in the population for 10 consecutive generations as shown in formula (3), where ave(i) represents the average fitness value of individuals in the population from the 10i-9th generation to the 10th generation.
[0021]
[0022] Where i is the learning cycle of the Q-Learning algorithm, i belongs to [1, iter_max / N], iter_max is the set maximum number of iterations; Q-Learning learns once every 10 iterations of the population.
[0023] Step 203: Calculate the average value of population diversity for 10 consecutive generations as shown in formula (4), where div(i) represents the average value of population diversity from the 10i-9th generation to the 10th generation.
[0024]
[0025] Step 204: Calculate the slope value con(i) of the linear fit of the average fitness value of the population for 10 consecutive generations as shown in formula (5):
[0026]
[0027] Step 205: Calculate the population diversity sa, population convergence speed sv, and population convergence trend sc of the genetic algorithm as shown in formula (6):
[0028]
[0029] where s a (i) is the population diversity of the i-th learning cycle, s v (i) is the population convergence speed of the i-th learning cycle, s c (i) is the population convergence trend of the i-th learning cycle.
[0030] step206: Calculate the state set sd after the various parameters of the genetic algorithm population are discretized.<sda,sdv,sdc> As shown in formula (7) and formula (8);
[0031]
[0032]
[0033]
[0034] sd a (i) is the discretized value of population diversity in the i-th learning cycle; sd v (i) is the discretized value of the population convergence speed in the i-th learning cycle; sd c (i) is the discretized value of the population convergence trend in the i-th learning cycle.
[0035] Furthermore, the calculation method of the return value R in step 3 is shown in formula (10):
[0036]
[0037] R(i) is the return value of the i-th learning cycle.
[0038] Further, in step 5, according to the selected action value a, the corresponding genetic algorithm parameter adjustment action is performed, where Pa is the general term for the selection ratio Ps, crossover probability Pc, and mutation probability Pm in the genetic algorithm, and the three parameters are adjusted using the following formula;
[0039]
[0040] The value of i here is
[0041] Further, in step 6, the selection operator, crossover operator and mutation operator of the genetic algorithm are executed according to the adjusted parameters to form a new population, and the specific steps include the following:
[0042] Selection operator: According to the selection ratio Ps and the elite retention strategy, use the selection crossover mutation strategy to generate N×Ps individuals; use the random generation method to generate the remaining N×(1-Ps) individuals; use the tournament algorithm to select individuals as the parent individuals of the crossover operator;
[0043] Crossover operator: Each time the crossover operator is executed, any of the following three crossover schemes is randomly adopted;
[0044] The first crossover scheme: only cross the chromosome position information of a specific workpiece of two individuals, without changing the combination relationship of <workpiece, machine, AGV>;
[0045] The second crossover scheme: cross <workpiece, machine> and the positional relationship between the two parent chromosomes corresponding to the workpiece, without changing the gene position of the AGV;
[0046] The third crossover scheme 3: crossover <workpiece, machine, AGV> and the positional relationship between the two chromosomes corresponding to the workpieces.
[0047] Mutation operator: Each time the mutation operator is executed, one of the following three mutation strategies is selected to perform different mutation operations on each gene according to the mutation probability:
[0048] The first mutation strategy: cross two adjacent <workpiece, machine, AGV>;
[0049] The second mutation strategy: randomly select other machines that can perform the corresponding process;
[0050] The third variation strategy: randomly reselect other AGVs that can perform the corresponding handling tasks.
[0051] Beneficial effects:
[0052] 1. The present invention provides a multi-robot joint scheduling method for a flexible manufacturing system, which is aimed at the joint scheduling problem of processing machines and AGVs in a flexible manufacturing system. Since this problem adds a new type of scheduling subject compared to the traditional flexible workshop scheduling problem, the traditional workshop scheduling algorithm is difficult to cope with this new problem. The present invention proposes an evolutionary optimization method based on reinforcement learning and genetic algorithm for the joint scheduling problem of production machines and AGVs in a flexible manufacturing system. This method uses a reinforcement learning method to dynamically adjust the key parameters of the genetic algorithm, control the iterative search process of the genetic algorithm, and use the upper-level Q-Learning algorithm to dynamically adjust the genetic algorithm parameters and the lower-level genetic algorithm to iteratively evolve and optimize the multi-robot joint scheduling algorithm. Compared with the traditional genetic algorithm, the solution quality is significantly improved.
[0053] 2. The present invention provides a multi-robot joint scheduling method for a flexible manufacturing system, which proposes a variety of crossover strategies and mutation strategies for the multi-layer coding scheme of the problem, realizes a full coverage search of the problem coding solution space, improves the algorithm's solution efficiency, and realizes efficient joint scheduling of workshops. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 A flow chart of a multi-robot joint scheduling method for a flexible manufacturing system provided by the present invention;
[0055] Figure 2 A schematic diagram of a coding scheme in a multi-robot joint scheduling method for a flexible manufacturing system provided by the present invention. DETAILED DESCRIPTION
[0056] The present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0057] The flexible workshop joint scheduling problem targeted by the present invention can be described as follows: There are a variety of processing machines with different capabilities in the workshop, the set is M = {M1, ..., M m ,...,M N_M}, the 1st to N_Mth processing machines; there are multiple isomorphic transfer AGVs, the set is R = {R1,...,R r ,...,R N_R}1st to N_Rth transfer AGVs; the set of workpieces to be processed is J = {J1,...,J j ,...,J N_J}, where N_M, N_R, and N_J represent the total number of processing machines, transfer AGVs, and workpieces to be processed, respectively. j With different processing technology ji , and in different processing machines M m With different processing time P jim . The problem of joint workshop scheduling needs to solve the problem of finding an efficient processing and transfer process for each processing technology of all the workpieces to be processed, minimizing the maximum completion time of all workpieces, that is, minimizing the makespan. Many assumptions and constraints need to be considered in the problem, including: 1) The storage capacity of the input / output buffer of each processing machine is unlimited; 2) Each machine is only allowed to process one workpiece at a time; 3) The processing process cannot be stopped after it starts; 4) All AGVs are isomorphic and are allowed to complete the handling task at the same speed; 5) The energy consumption of AGV is not considered; 6) When the AGV completes a transfer task, the AGV will move to the next task point to wait for the transfer task.
[0058] A multi-robot joint scheduling method for a flexible manufacturing system is proposed. An upper-level Q-Learning algorithm dynamically adjusts the genetic algorithm parameters and a lower-level genetic algorithm is iteratively evolved to optimize the multi-robot joint scheduling algorithm. Compared with the traditional genetic algorithm, the solution quality is significantly improved. The process of this method is as follows Figure 1 The steps shown are as follows:
[0059] First, a three-layer coding method is used to match the machine sequence, process sequence, and AGV sequence. The coding scheme is as follows: Figure 2 As shown in the figure: each gene in the process sequence represents the number of the workpiece to be processed, and the number of times the number appears from the left side of the process sequence to this position represents the number of the processing process; each gene in the machine sequence represents the machine number of the corresponding process; each gene in the AGV sequence represents the AGV number that transports the corresponding workpiece to the corresponding machine.
[0060] Secondly, in terms of decoding method, the order of appearance of the processes in the encoding from front to back is strictly followed, and the processing processes are decoded under the premise of satisfying the process constraints and processing constraints.
[0061] Step 1: Algorithm initialization:
[0062] Step 1.1: Genetic algorithm initialization includes: population size N, establishment of random initial population, number of iterations g_max, selection ratio Ps = 0.8, crossover probability Pc = 0.6, mutation probability Pm = 0.05, etc.;
[0063] Step 1.2: The initialization content of the Q-Learning algorithm includes: Q-Table, state set S, action set A, α, β, ε, etc. Among them, the elements in the state set S are selected by extracting key information such as population diversity s from the genetic algorithm population a , population convergence speed s v , population convergence trend s c After discretization, various information data are combined to form sd= <sd a ,sd v ,sd c >, and use it as an element of the state set. The selection method of the elements in the action set A is: increase or decrease the selection ratio Ps, crossover probability Pc, and mutation probability Pm of the genetic algorithm; Q-Table uses the state set S as the row vector and the action set A as the column vector to establish an empty Q-Table with all elements 0;
[0064] Table 1 Q-Table initialization example
[0065] Ps+ Ps- PC+ Pc- Pm+ Pm- s0 0 0 0 0 0 0 s1 0 0 0 0 0 0 s2 0 0 0 0 0 0 … 0 0 0 0 0 0 s14 0 0 0 0 0 0 s15 0 0 0 0 0 0
[0066] Step 2: Calculate the fitness values of all individuals in the current population and record the best individual that has been searched so far;
[0067] Step 3: Calculate population diversity s based on fitness value a , population convergence speed s v , population convergence trend s c The data is compared with the state data of the previous generation population to obtain the reward value R, update the Q-Table, and then discretize it;
[0068] Step 3.1: Calculate the average fitness value of all individuals in the k-th generation population as shown below, where Y(k) represents the average fitness value of individuals in the k-th generation population, f(x j k ) represents the fitness value of the jth individual in the kth generation population.
[0069]
[0070] Step 3.2: Calculate the average fitness value of all individuals in the population for 10 consecutive generations as shown in the following formula, where ave(i) represents the average fitness value of individuals in the population from the 10i-9th generation to the 10th generation.
[0071]
[0072] Step 3.3: Calculate the average population diversity of 10 consecutive generations as shown in the following formula, where div(i) represents the average population diversity from the 10i-9th generation to the 10th generation.
[0073]
[0074] Step 3.4: Calculate the slope of the linear fit of the average population fitness value for 10 consecutive generations as shown below.
[0075]
[0076] Step 3.5: Calculate the population diversity s of the genetic algorithm a , population convergence speed s v , population convergence trend s c As shown in the following formula.
[0077]
[0078] Step 3.6: Calculate the state set of the genetic algorithm population after discretization of various parameters <sd a ,sd v ,sd c >The formula is shown below.
[0079]
[0080]
[0081]
[0082] Step 3.7: The calculation method of the return value R is as follows.
[0083]
[0084] Step 3.8: The calculation formula of Q-Table is as follows. Where Q is the Q-Table vector; s i +1 indicates the state value of the next generation learning process; s i Represents the state value of this generation of learning process; a i +1 indicates the action value obtained by the next generation according to the Q-Table; a iIndicates the action value obtained by this generation according to the Q-Table;
[0085] Q(s i ,a i )←(1-α)·Q(s i ,a i )+α·(R t+1 +γQ(s i+1 ,a i+1 )) (10)
[0086] Step 4: According to sd= <sd a ,sd v ,sd c >Discrete state value, element value of the corresponding row of Q-Table and action value a selected by ε-greedy strategy;
[0087] Step 4.2: The ε-greedy action selection strategy is as follows:
[0088]
[0089] Step 5: According to the selected action value a, execute the corresponding genetic algorithm parameter adjustment action; the genetic algorithm parameter adjustment strategy is shown in the following formula, where Pa is the general term for the selection ratio Ps, crossover probability Pc, and mutation probability Pm in the genetic algorithm, and all three parameters are adjusted using the following formula.
[0090]
[0091] Step 6: Execute the selection operator, crossover operator and mutation operator of the genetic algorithm according to the adjusted parameters to form a new population; various strategies of the selection operator, crossover operator and mutation operator in the present invention are as follows:
[0092] Selection operator: According to the selection ratio Ps and the elite retention strategy, use the selection crossover mutation strategy to generate N*Ps individuals; use the random generation method to generate the remaining N*(1-Ps) individuals; use the tournament algorithm to select individuals as the parent individuals of the crossover operator.
[0093] Crossover operator: Due to the adoption of a three-layer coding scheme, the present invention designs three crossover schemes, and a crossover scheme is randomly adopted each time the crossover operator is executed.
[0094] Crossover scheme 1: Only the chromosome position information of a specific workpiece of two individuals is crossed, without changing the combination relationship of <workpiece, machine, AGV>;
[0095] Crossover scheme 2: Crossover <workpiece, machine> and the positional relationship between the two parent chromosomes corresponding to the workpiece, without changing the gene position of the AGV;
[0096] Crossover scheme 3: Crossover <workpiece, machine, AGV> and the positional relationship between the two chromosomes corresponding to the workpieces.
[0097] Mutation operator: Since a three-layer coding scheme is adopted, the actual meaning of each layer of coding is different. The present invention designs three mutation strategies. Each time the mutation operator is executed, different mutation operations are performed on each gene according to the mutation probability.
[0098] Mutation strategy 1: cross two adjacent <workpiece, machine, AGV>;
[0099] Mutation strategy 2: Randomly select other machines that can perform the corresponding process;
[0100] Mutation strategy 3: Randomly select other AGVs that can perform the corresponding handling tasks;
[0101] Step 7: Determine whether the population iteration has reached the maximum number of iterations. If the conditions are met, output the best individual and fitness value; otherwise, jump to step 2.
[0102] Example: Taking EX11 as an example, the problem example data is shown in the following table. There are 5 workpieces to be processed, 4 processing machines and 2 transfer AGVs in the example:
[0103] Table 2 Workshop node transfer time
[0104]
[0105] Table 3 Workpiece process information
[0106]
[0107] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A multi-robot joint scheduling method for a flexible manufacturing system, characterized in that: The steps include: First, a three-layer encoding method is used to match the machine sequence, process sequence and AGV sequence: each gene in the process sequence represents the number of the processed workpiece, and the number of times the number appears from the left side of the process sequence to the position represents the number of the processing process; each gene in the machine sequence represents the number of the machine that processes the corresponding process; each gene in the AGV sequence represents the AGV number that transports the corresponding workpiece to the corresponding machine; Secondly, in terms of decoding method, the process is decoded strictly in the order of appearance of the processes from front to back in the encoding, and the process constraints and processing constraints are met; Step 1: Algorithm initialization: Step 1.1: Genetic algorithm initialization includes: population size N, establishment of random initial population, number of iterations g_max, selection ratio Ps, crossover probability Pc, and mutation probability Pm; Step 1.2: The initialization content of the Q-Learning algorithm includes: Q-Table, state set S, action set A, Q-table update coefficient β and action selection coefficient ε; the elements in the state set S are selected by extracting state data from the genetic algorithm population, the state data includes population diversity sa, population convergence speed sv and population convergence trend sc, and the combination of discretized state data is sd=<sda,sdv,sdc> , where sda is the discrete population diversity value, sdv is the discrete population convergence speed value, sdc is the discrete population convergence trend, and the discretized state data is used as the element of the state set S; The elements in the action set A are selected by increasing or decreasing the selection ratio Ps, crossover probability Pc, and mutation probability Pm of the genetic algorithm; the Q-Table uses the state set S as the row vector and the action set A as the column vector to establish an empty initial Q-Table with all elements being 0; The calculation process of the population diversity sa, population convergence speed sv, and population convergence trend sc includes the following steps: Step 201: Calculate the average fitness value of all individuals in the k-th generation population as shown in formula (2), where Y(k) represents the average fitness value of individuals in the k-th generation population, and f(xjk) represents the fitness value of the j-th individual in the k-th generation population; Step 202: Calculate the average fitness value of all individuals in the population for 10 consecutive generations as shown in formula (3), where ave(i) represents the average fitness value of individuals in the population from the 10i-9th generation to the 10th generation; Where i is the learning cycle of the Q-Learning algorithm, i belongs to [1, iter_max / N], iter_max is the maximum number of iterations set; Q-Learning learns once every 10 iterations of the population; Step 203: Calculate the average value of population diversity for 10 consecutive generations as shown in formula (4), where div(i) represents the average value of population diversity from the 10i-9th generation to the 10th generation; Step 204: Calculate the slope value con(i) of the linear fit of the average fitness value of the population for 10 consecutive generations as shown in formula (5): Step 205: Calculate the population diversity sa, population convergence speed sv, and population convergence trend sc of the genetic algorithm as shown in formula (6): where s a (i) is the population diversity of the i-th learning cycle, s v (i) is the population convergence speed of the i-th learning cycle, s c (i) is the population convergence trend of the i-th learning cycle; step206: Calculate the state set sd after the various parameters of the genetic algorithm population are discretized.<sda,sdv,sdc> As shown in formula (7) and formula (8); sd a (i) is the discretized value of population diversity in the i-th learning cycle; sd v (i) is the discretized value of the population convergence speed in the i-th learning cycle; sd c (i) is the discretized value of the population convergence trend in the i-th learning cycle; Step 2: Calculate the fitness values of all individuals in the current population and record the best individual that has been searched so far; Step 3: Calculate the state data of the current generation population according to the fitness value, including population diversity sa, population convergence speed sv, and population convergence trend sc, compare it with the state data of the previous generation population to obtain the reward value R, update the Q-Table, and then discretize the state data of the current generation population; The calculation method of the return value R is shown in formula (10): R(i) is the return value of the i-th learning cycle; Step 4: According to sd=<sda,sdv,sdc> The discrete state value, the element value of the corresponding row of the Q-Table and the action value a selected by the ε-greedy strategy; Step 5: According to the selected action value a, execute the corresponding genetic algorithm parameter adjustment action; According to the selected action value a, the corresponding genetic algorithm parameter adjustment action is executed, where Pa is the general term for the selection ratio Ps, crossover probability Pc, and mutation probability Pm in the genetic algorithm. The three parameters are adjusted using the following formula; Step 6: Execute the selection operator, crossover operator and mutation operator of the genetic algorithm according to the adjusted parameters to form a new population; the specific steps include the following; Selection operator: According to the selection ratio Ps and the elite retention strategy, use the selection crossover mutation strategy to generate N×Ps individuals; use the random generation method to generate the remaining N×(1-Ps) individuals; use the tournament algorithm to select individuals as the parent individuals of the crossover operator; Crossover operator: Each time the crossover operator is executed, any of the following three crossover schemes is randomly adopted; The first crossover scheme: only cross the chromosome position information of a specific workpiece of two individuals, without changing the combination relationship of <workpiece, machine, AGV>; The second crossover scheme: cross <workpiece, machine> and the positional relationship between the two parent chromosomes corresponding to the workpiece, without changing the gene position of the AGV; The third crossover scheme 3: crossover <workpiece, machine, AGV> and the position relationship between the two chromosomes corresponding to the workpiece; Mutation operator: Each time the mutation operator is executed, one of the following three mutation strategies is selected to perform different mutation operations on each gene according to the mutation probability: The first mutation strategy: cross two adjacent <workpiece, machine, AGV>; The second mutation strategy: randomly select other machines that can perform the corresponding process; The third variation strategy: randomly reselect other AGVs that can perform the corresponding handling tasks; Step 7: Determine whether the population iteration has reached the maximum number of iterations. If the conditions are met, output the best individual and fitness value; otherwise, jump to step 2.
