A disassembly workshop AGV scheduling method based on Q-learning distribution estimation algorithm

Abstract

The application provides a disassembling workshop AGV scheduling method based on a Q learning distribution estimation algorithm, constructs a disassembling workshop AGV scheduling calculation model to optimize the completion time of a task, and designs coding and decoding rules to effectively represent a scheduling scheme; the application provides a Q learning-based distribution estimation algorithm, updates a probability model through an elite solution and a non-elite solution; a local search strategy based on Q learning is used to optimize the elite solution, so that an optimal scheduling scheme of the disassembling workshop AGV is obtained, and the AGV scheduling of the disassembling workshop is realized. Through the establishment of the disassembling workshop AGV scheduling model, the application realizes the rapid calculation of a scheduling target, and the Q learning-based distribution estimation algorithm can effectively solve the large-scale disassembling workshop AGV scheduling problem.

Description

TECHNICAL FIELD

[0001] The application proposes a Q-learning-based distribution estimation algorithm for solving the automatic guided vehicle (AGV) scheduling problem in a disassembly workshop. The AGV scheduling in a disassembly workshop involves industrial disassembly processes and optimization scheduling techniques, and belongs to the field of industrial manufacturing and optimization scheduling. BACKGROUND

[0002] Currently, green and low-carbon circular development has become a global consensus, and the country is vigorously promoting green manufacturing and resource recycling at the policy level. At the same time, manufacturing is in a critical period of transformation and upgrading, and intelligent and automated technology has become an important driving force for industrial upgrading. Under this background, the disassembly workshop, as an important link of resource recycling, is crucial to the development of circular economy. The introduction and application of automatic guided vehicles (AGVs) gradually become an important means to improve the efficiency of workshop operation, reduce costs, and achieve intelligent and automated production. Therefore, optimizing AGV scheduling in a disassembly workshop has broad application prospects.

[0003] With the increasing variety of disassembled materials in a disassembly workshop and the increasing complexity of the process, the complexity of material handling is also increasing. Traditional manual handling not only has low efficiency, but also is prone to operational errors, which cannot meet the high efficiency requirements of modern production. Therefore, to improve the efficiency and accuracy of handling, automatic guided vehicles (AGVs) are gradually used in disassembly workshops to effectively reduce the number of operating personnel and operating costs, and achieve efficient and automated operation of the disassembly workshop.

[0004] Common AGV scheduling methods include exact methods, heuristic algorithms, and intelligent optimization algorithms, which have certain effectiveness in solving AGV scheduling problems. However, due to the particularity of the disassembly workshop scene, such as complex environment and multiple constraints, related research is very limited. AGV scheduling in a disassembly workshop involves multiple complex factors, including path planning, task allocation, and workpiece disassembly sequencing, etc. At the same time, when the disassembly tasks and disassembly machines increase in large scale, traditional algorithms are difficult to effectively solve, showing obvious limitations. Therefore, there is an urgent need for effective and efficient optimization scheduling algorithms to improve the efficiency and reduce the cost of the disassembly workshop, and to ensure the efficient and stable operation of the disassembly process.

[0005] The application designs a Q-learning-based distribution estimation algorithm (QEDA), which updates the probability matrix in EDA and performs local search on elite solutions with the aid of Q-learning, to obtain the optimal AGV scheduling scheme for the disassembly workshop, and realizes the optimization of AGV scheduling in the disassembly workshop. SUMMARY

[0006] A Q-learning-based AGV scheduling method for a dismantling workshop is characterized by: constructing an AGV scheduling model for the dismantling workshop, designing a Q-learning-based AGV scheduling method for the dismantling workshop, and obtaining an AGV scheduling scheme for the dismantling workshop.

[0007] (1) Construct a calculation model for AGV scheduling in the dismantling workshop with transportation efficiency as the objective, specifically as follows:

[0008] Calculate the start time (es) of AGV empty transport. ij Before the AGV begins all transportation tasks, the start time of the AGV's empty transportation is 0; otherwise, the start time of the AGV's empty transportation is equal to the end time of the AGV's loaded transportation. ij As shown in formula (1):

[0009]

[0010] Calculate the end time ee of AGV empty transport. ij This includes the start time of AGV empty transport. ij and the time t for AGV empty transport ij As shown in formula (2):

[0011] ee ij =es ij +t ij (2)

[0012] Among them, the time t for AGV empty transport ij For the AGV to start from the current position m A To target location m g The required time is shown in formula (3). This indicates that the AGV has started from its current position m. A To target location m g The distance is v, where v represents the speed of the AGV;

[0013]

[0014] Calculate the start time (ls) of AGV carrying transport. ij That is, the previous disassembly task O. i(j-1) Disassembly completion time c i(j-1) and AGV empty transport end time ee ij The larger of the values ​​in the range, if the AGV is performing its first transport task, then c i(j-1) =0, as shown in formula (4):

[0015] ls ij =max(c i(j-1) ,ee ij (4)

[0016] calculating the end time le of the AGV's carrying transportation ij , the start time ls of the AGV's carrying transportation ij , and the carrying time T of the AGV ij , as shown in equation (5):

[0017] le ij = ls ij + T ij (5)

[0018] wherein the carrying time T of the AGV ij is the time required for the AGV to move from the location m l where the AGV is carrying to the target location m g , as shown in equation (6), denotes the distance from the location m l where the AGV is carrying to the target location m g ;

[0019]

[0020] Considering the power consumption of the AGV during transportation, the power consumption is positively correlated with the transportation time. The power change of the AGV when moving to the next location is shown in equation (7), E t denotes the power of the AGV at the current time, E t+1 denotes the power of the AGV after moving from the current location to the next location, E cost denotes the power consumption per unit distance of the AGV, and d' denotes the distance traveled by the AGV between two locations. If the AGV is performing the first transportation task, the power E t of the AGV is the full power E full ;

[0021] E t+1 = E t - E cost × d' (7)

[0022] Before the AGV moves to the next location, it is checked whether the remaining power of the AGV meets the minimum power requirement E min . If the remaining power of the AGV can meet the power consumption of the AGV from the current location to the target location and then to the charging station, i.e., the condition shown in equation (8), the AGV continues to perform the current disassembly task; otherwise, the AGV goes to the charging station for charging, wherein denotes the distance from the target location to the charging station;

[0023]

[0024] If the AGV is in an empty state and needs to be charged, the transportation time is updated according to equation (9), tcha denotes the charging time, denotes the distance from the current position m A of the AGV to the charging station m1, t′ ij denotes the updated empty transportation time; if the AGV is in a loaded state and needs to be charged, the transportation time is updated according to formula (10), denotes the distance from the current position m l of the AGV carrying a load to the charging station m1, T′ ij denotes the updated loaded transportation time, which is increased by the time required for the AGV to travel from the current position to the charging station, the charging time, and the transportation time of the AGV from the charging station to the target position; after the AGV completes charging, the power E′ is updated according to formula (11), t denotes the updated power;

[0025]

[0026] E′ t = E full (11)

[0027] The time s ij at which the disassembly machine starts processing the disassembly task O ij is calculated, i.e., the maximum of the end time le ij of the AGV loaded transportation and the completion time ms k of the previous disassembly task of the machine, as shown in formula (12); if the AGV is the first transportation task and all machines are in an idle state, ms k = 0.

[0028] s ij = max(le ij , ms k ) (12)

[0029] The time c ij at which the disassembly machine completes the disassembly task O ij is calculated, including the time s ij at which the machine starts processing the disassembly task O ij and the time p ij at which the machine processes the disassembly task O ij , as shown in formula (13).

[0030] c ij = s ij + p ij (13)

[0031] A target function for AGV scheduling in a disassembly workshop considering charging is established, and the minimum maximum completion time of all disassembly tasks is taken as the optimization objective of scheduling, and the function expression is shown in formula (14):

[0032]

[0033] (2) Design a disassembly workshop AGV scheduling method based on Q-learning, including designing coding and decoding rules, initializing probability matrix, sampling operation of probability matrix, updating rules of probability matrix based on Q-learning, local search operation based on Q-learning, specifically:

[0034] (a) Design coding and decoding rules to represent the mapping relationship between solution and scheduling scheme through coding and decoding:

[0035] Coding: Encode the disassembly tasks of n workpieces, each workpiece contains m disassembly tasks, use r = (i-1)·m+j to number the jth disassembly task of workpiece i, where i = 1, 2, …, n and j = 1, 2, …, m, number all disassembly tasks from 1 to n×m, and represent the coding solution by permutation π = (1, 2, …, r, …, n×m) of length n×m task number, the order of permutation is the scheduling order of tasks;

[0036] Decoding: Divide all elements in the coding solution by the number of disassembly tasks m, and the quotient plus 1 is the workpiece number i, and the remainder j is the disassembly task number of workpiece i. AGV transports all workpieces to the corresponding disassembly machine position in coding order for disassembly. Before AGV executes the transportation task, it needs to determine whether the power is sufficient. If not, it needs to charge and then calculate the transportation time according to the speed and distance of AGV. After AGV arrives at the target disassembly machine, calculate the start and completion time of each disassembly task according to formulas (12) and (13), and after the completion of the disassembly task, the workpiece can be transported to the next disassembly machine. Calculate the maximum completion time of all AGV transportation tasks and disassembly tasks;

[0037] (b) Initialize the probability matrix, sample the probability matrix, and update the probability model through elite individuals and non-elite individuals:

[0038] Construct an n×m row and n×m column probability matrix P(l), as shown in formula (15):

[0039]

[0040] P(l) represents the probability matrix of the lth generation population, and each matrix element represents the probability of the disassembly task numbered r in the lth generation located at the c k (k∈[1,n×m]) position of the individual, uniformly initialize the probability matrix, i.e.

[0041] Generate new individuals through the sampling operation of the probability matrix, and construct an n row and m column candidate set P' representing m disassembly tasks of n workpieces, as shown in formula (16):

[0042]

[0043] Considering the disassembly task constraints, determine the column vector P' of the set of optional processes P' that meet the process constraints from the set of alternative processes P' o , that is, select the first non-zero element of each row in the set P' as P' o , if all elements in a row in P' are 0, skip this row and continue to select the first non-zero element of the next row as P' o , and determine P' according to the probability matrix P(l) o The selection probability of each task at the current position c k (k∈[1,n×m]) is determined using the roulette method to select a task to be assigned to the current position c k , set the element in P' corresponding to the position of the task to 0, and update P' o , until all tasks are sampled, that is, all elements in P' are 0, and a new individual is generated.

[0044] According to the maximum completion time, sort all individuals from small to large, take the top 20% as elite individuals and the last 30% as non-elite individuals, update the probability matrix according to the population elite individuals and non-elite individuals, and define As shown in equation (17):

[0045]

[0046] In equation (17), s=a represents an elite individual, s=b represents a non-elite individual, and each element of the probability matrix is updated as shown in equation (18), , which represents the probability that the disassembly task numbered r in the l+1 generation is located at the c k position in the individual, α(0<α<1) represents the learning rate, A size represents the number of all elite individuals, and B size represents the number of all non-elite individuals.

[0047]

[0048] (c) Design a local search method based on Q learning for the above elite individuals:

[0049] Q table is established: three key time indicators are selected, i.e. total time of all disassembly tasks, AGV transportation time and machine idle time; discrete operation is performed on the three indicators: all solutions are arranged in descending order according to the indicators, and three different states are obtained by three equal divisions, and a state space with a total of 3x3x3=27 state combinations can be constructed by combining the three levels; the action space is composed of three local search operators, which are exchange, insertion and reversal; exchange refers to exchanging two disassembly tasks in different positions in the task sequence; insertion refers to taking a task from the task sequence and inserting it into another position; reversal refers to selecting a continuous disassembly task in the task sequence and reversing the order; a Q table with a size of 27x3 is constructed, each element represents a state-action pair, and the initial value of all elements of the Q table is 0;

[0050] Action selection: the state of the elite solution is determined according to the three time indicators, the action with the maximum Q value in the state is selected from the Q table, and the action is executed to generate a new individual;

[0051] Updating Q table: the state of the new individual is determined and the reward is calculated, the reward function is composed of the improvement of the total time of all disassembly tasks C max , AGV transportation time T AGV and machine idle time T ms , for each time indicator X (X∈{C max , T AGV , T ms}), the improvement ΔX is calculated by formula (19), wherein X last represents the value of time indicator X in the last state, and X new represents the value of time indicator X in the current state;

[0052]

[0053] The weighted sum r′ t+1 of the improvement of the three time indicators is obtained by formula (20), wherein ΔC max , ΔT AGV , ΔT ms are the improvements of the three time indicators calculated by formula (19), the weight ω c of the total time of all disassembly tasks is set to 0.6, the weight ω A of AGV transportation time is set to 0.2, and the weight ω m of machine idle time is set to 0.2; the reward function r t+1 is shown in formula (21), if any time indicator is improved, a positive reward is given, otherwise, a negative reward is given;

[0054] r′ t+1 = ω c · ΔCmax +ω A ·ΔT AGV +ω m ·ΔT ms (20)

[0055]

[0056] updating the Q table using the Q-learning update formula shown in formula (22);

[0057] Q'(s t ,a t )=Q(s t ,a t )+σ[r t+1 +γmax a Q(s t+1 ,a)-Q(s t ,a t )] (22)

[0058] wherein σ (0<σ<0.7) represents a learning rate, γ (0.5<γ<1) is a discount factor, Q(s t ,a t ) represents a Q value corresponding to the state s t and action a t before updating, Q'(s t ,a t ) represents a Q value corresponding to the new state s t and action a t , and max a Q(s t+1 ,a) is the maximum Q value of all possible actions in the new state s t+1 ;

[0059] storing the optimal solution: after updating the Q table, if the new solution is better than the old solution, the old solution is replaced by the new solution and the new solution information is stored;

[0060] (3) obtaining the disassembly workshop AGV scheduling scheme: according to the steps in (2), after initializing the probability matrix, the optimal solution is obtained by repeating the iteration steps of sampling to generate the population operation and updating the probability matrix operation, and selecting different actions based on the Q table for the state of the elite solution, updating the Q table and storing the optimal solution operation, until the iteration number reaches l max =100, the optimal solution is obtained, and the optimal AGV scheduling scheme is output through the decoding rule, realizing the optimization scheduling of the disassembly workshop AGV.

[0061] The creativity of the present application mainly lies in:

[0062] (1) the present application provides a calculation model for disassembly workshop AGV scheduling with the maximum completion time of tasks as the target,

[0063] (2) The encoding method provided by the application can effectively represent workpiece sequencing and AGV scheduling route, and the decoding method provided by the application can quickly convert the encoding into a feasible scheduling scheme;

[0064] (3) The probability model sampling and updating mechanism provided by the application can accurately find an effective scheduling scheme, and the local search based on Q learning is used to optimize the elite solution, the scheduling method has clear structure, is easy to implement, and can effectively solve large-scale disassembly workshop AGV scheduling problems. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is the actual disassembly flowchart provided by the embodiment of the application;

[0066] Figure 2 is the algorithm detailed flowchart provided by the embodiment of the application;

[0067] Figure 3 is the scheduling Gantt chart obtained by processing a simple test set by the method of the application DETAILED DESCRIPTION

[0068] The application provides a disassembly workshop AGV scheduling method, establishes a disassembly workshop AGV scheduling calculation model, effectively finds an elite solution by improving the updating mechanism of the probability model in the distribution estimation algorithm, further optimizes the elite solution by using the proposed local search method, obtains a scheduling scheme with the least time consumption, and effectively solves large-scale disassembly workshop AGV scheduling problems;

[0069] Figure 1 is the disassembly flowchart of the actual disassembly workshop, and from the feeding of the disassembly workpiece, AGV needs to be transported in each subsequent disassembly step until the last disassembly step is completed, and finally the truck transports the disassembled articles out of the warehouse;

[0070] Figure 2 is the algorithm detailed flowchart, after initializing the probability matrix, the updating and sampling operations of the probability matrix are repeated to obtain a scheduling solution, and the local search method based on Q learning is used to iteratively optimize the elite solution, and finally the optimal scheduling scheme is obtained;

[0071] The application adopts the following technical solutions and implementation steps:

[0072] (1) A disassembly workshop AGV scheduling calculation model is constructed with transportation efficiency as the target, and specifically:

[0073] The starting time es of AGV empty transportation is calculated ij Before the AGV starts all transportation tasks, the starting time of AGV empty transportation is 0; otherwise, the starting time of AGV empty transportation is equal to the ending time le of AGV loaded transportation ijAs shown in formula (23):

[0074]

[0075] Calculate the end time ee of AGV empty transport. ij This includes the start time of AGV empty transport. ij And the time t for AGV empty transport ij As shown in formula (24):

[0076] ee ij =es ij +t ij (twenty four)

[0077] Among them, the time t for AGV empty transport ij For the AGV to start from the current position m A To target location m g The required time is shown in formula (25). This indicates the distance the AGV transports when unloaded, and v represents the speed of the AGV.

[0078]

[0079] Calculate the start time (ls) of AGV carrying transport. ij That is, the previous disassembly task O. i(j-1) Disassembly completion time c i(j-1) and AGV empty transport end time ee ij The larger of the values ​​in the range, if the AGV is performing its first transport task, then c i(j-1) =0, as shown in formula (26):

[0080] ls ij =max(c i(j-1) ,ee ij (26)

[0081] Calculate the end time of AGV transport. ij ls represents the start time of AGV transport. ij and the transportation time T carried by the AGV ij The sum is shown in formula (27):

[0082] le ij =ls ij +T ij (27)

[0083] Among them, the AGV carrying and transporting time T ij For the AGV to move from the disassembly task location m l Head to target location m gThe required time is shown in equation (28), m represents the position where the AGV is located with load l to the target position m g ;

[0084]

[0085] Considering the power consumption of the AGV transportation process, the power consumption is positively correlated with the transportation time. The power change of the AGV moving to the next location is shown in equation (29), E t represents the power of the AGV at the current time, t+1 represents the power of the AGV after moving from the current location to the next location, cost represents the power consumption per unit distance of the AGV, d' represents the distance traveled by the AGV between two locations, if the AGV is the first transportation task, the power E t of the AGV is full power E full ;

[0086] E t+1 = E t - E cost × d' (29)

[0087] Before the AGV moves to the next location, check whether the remaining power of the AGV meets the minimum power requirement E min , if the remaining power of the AGV can meet the power consumption of the AGV from the current location to the target location and then to the charging station, that is, the condition shown in equation (30), the AGV continues to perform the current task; otherwise, the AGV goes to the charging station to charge, wherein represents the distance from the target location to the charging station;

[0088]

[0089] If the AGV is in an empty state and needs to be charged, update the transportation time according to equation (31), t cha represents the charging time, represents the distance from the current location m A of the AGV to the charging station m1, t' ij represents the updated empty transportation time; if the AGV is in a loaded state and needs to be charged, update the transportation time according to equation (32), represents the distance from the current location m l of the AGV with load to the charging station m1, T' ij represents the updated loaded transportation time, which increases the required time of the AGV from the current location to the charging station, the charging time and the transportation time of the AGV from the charging station to the target location compared to the previous update; after the AGV completes charging, update the power according to equation (33), E't represents the updated power;

[0090]

[0091] E′ t = E full (33)

[0092] Calculate the time s ij at which the disassembly machine starts to process the disassembly task O ij , i.e. the end time le ij of AGV carrying and the completion time ms k of the pre-sequencing disassembly task of the machine, the maximum value of which is shown in formula (34), if the AGV is the first transportation task and all machines are in idle state, then ms k = 0.

[0093] s ij = max(le ij , ms k ) (34)

[0094] Calculate the time c ij at which the disassembly machine completes the disassembly task O ij , including the time s ij at which the machine starts to process the disassembly task O ij and the time p ij at which the machine processes the disassembly task O ij , as shown in formula (35).

[0095] c ij = s ij +p ij (35)

[0096] Establish the objective function of the disassembly workshop AGV scheduling considering charging, and take the minimum maximum completion time of all disassembly tasks as the scheduling optimization goal, and the function expression is shown in formula (36):

[0097]

[0098] In the above calculation model, the inherent parameters of AGV, such as speed v, minimum power requirement E min , charging time t cha , full power E full , AGV unit distance power consumption E cost , the total number of disassembly tasks, the disassembly time p ij of each disassembly task, and the distance between each two places are known. According to the above calculation model, the target can be quickly calculated through the scheduling scheme.

[0099] (2) Design a Q-learning-based disassembly workshop AGV scheduling method, including the design of encoding and decoding rules, the initialization of the probability matrix, the sampling operation of the probability matrix, the Q-learning-based probability matrix update rule, and the Q-learning-based local search operation. Specifically:

[0100] (a) Design the encoding and decoding rules to represent the mapping relationship between the solution and the scheduling scheme through encoding and decoding:

[0101] Encoding: Encode the disassembly tasks of n workpieces, each workpiece containing m disassembly tasks, where n and m are obtained from the data set. Use r = (i-1)·m+j to number the jth disassembly task of workpiece i, where i = 1, 2, …, n and j = 1, 2, …, m. Number all disassembly tasks from 1 to n×m. Represent the encoding solution by arranging the task numbers of length n×m, π = (1, 2, …, r, …, n×m). The order of the arrangement is the scheduling order of the tasks.

[0102] Decoding: Divide all elements in the encoding solution by the number of disassembly tasks m, and the quotient plus 1 is the workpiece number i, and the remainder j is the disassembly task number of workpiece i. AGV transports all workpieces to the corresponding disassembly machine location in the encoding order for disassembly. Before AGV performs the transportation task, it needs to determine whether the power is sufficient. If not, it needs to charge and then calculate the transportation time based on the speed and distance of the AGV. After the AGV arrives at the target disassembly machine, the start and completion times of each disassembly task are calculated. After the disassembly task is completed, the workpiece can be transported to the next disassembly machine. The maximum completion time of all AGV transportation tasks and disassembly tasks is calculated.

[0103] (b) Initialize the probability matrix, sample the probability matrix, and update the probability model through elite individuals and non-elite individuals:

[0104] Construct an n×m row and n×m column probability matrix P, as shown in equation (37):

[0105]

[0106] P(l) represents the probability matrix of the lth generation population, and each matrix element represents the probability of the lth generation disassembly task numbered r being located at the c k position of the individual. Uniformly initialize the probability matrix, i.e., let

[0107] Generate new individuals through the probability matrix sampling operation, and construct an n-row and m-column candidate set P' representing the m disassembly tasks of n workpieces, as shown in equation (38):

[0108]

[0109] Considering the disassembly task constraints, determine the column vector P' of the set of optional processes P' that meet the process constraints from the set of alternative processes P' o , that is, select the first non-zero element of each row in the set P' as P' o , and the elements of the corresponding row, if all the elements in a row in P' are 0, skip this row and continue to select the first non-zero element of the next row as P' o ; determine P' according to the probability matrix P(l) o The selection probability of each task at the current position c k (k∈[1,n×m]) is determined by using the roulette method to select a task to be assigned to the current position c k , set the element in P' corresponding to the position of the task to 0, and update P' o , until all tasks are sampled, that is, all elements in P' are 0, and a new individual is generated.

[0110] According to the maximum completion time, sort all individuals from small to large, take the top 20% as elite individuals and the last 30% as non-elite individuals, update the probability matrix according to the population elite individuals and non-elite individuals, and define As shown in equation (39):

[0111]

[0112] In equation (39), s=a represents elite individuals, s=b represents non-elite individuals, and each element of the probability matrix is updated as shown in equation (40), , which represents the probability that the disassembly task numbered r in the l+1 generation is located at the c k position in the individual, set the learning rate α=0.5, A size , which represents the number of all elite individuals, B size , which represents the number of all non-elite individuals, in the experiment, set the population size to 100, A size =20, B size =30;

[0113]

[0114] (c) Design a local search method based on Q learning for the above elite individuals:

[0115] Q table is established: three key time indicators are selected, i.e. total time of all disassembly tasks, AGV transportation time and machine idle time; discrete operation is performed on the three indicators: all solutions are arranged in descending order according to the indicators, and three different states are obtained by three equal divisions, and a state space with a total of 3*3*3=27 state combinations can be constructed by combining the three levels; the action space is composed of three local search operators, which are exchange, insertion and reversal; exchange refers to exchanging two disassembly tasks in different positions in the task sequence; insertion refers to taking a task from the task sequence and inserting it into another position; reversal refers to selecting a continuous disassembly task in the task sequence and reversing the order; a Q table with a size of 27*3 is constructed, each element represents a state-action pair, and the initial value of all elements of the Q table is 0;

[0116] Action selection: the state of the elite solution is determined according to the three time indicators, the action with the maximum Q value in the state is selected from the Q table, and the action is executed to generate a new individual;

[0117] Updating Q table: the state of the new individual is determined and the reward is calculated, the reward function is composed of the improvement of the total time of all disassembly tasks C max , AGV transportation time T AGV and machine idle time T ms , for each time indicator X (X∈{C max , T AGV , T ms}), the improvement ΔX is calculated by formula (41), wherein X last represents the value of time indicator X in the last state, and X new represents the value of time indicator X in the current state;

[0118]

[0119] The weighted sum r′ t+1 of the improvement of the three time indicators is obtained by formula (42), wherein ΔC max , ΔT AGV , ΔT ms are the improvements of the three time indicators calculated by formula (41), the weight ω c of the total time of all disassembly tasks is set to 0.6, the weight ω A of AGV transportation time is set to 0.2, and the weight ω m of machine idle time is set to 0.2; the reward function r t+1 is shown in formula (43), if any time indicator is improved, a positive reward is given, otherwise, a negative reward is given;

[0120] r′ t+1 = ω c · ΔCmax +ω A ·ΔT AGV +ω m ·ΔT ms (42)

[0121]

[0122] updating the Q table using the Q-learning updating formula shown in formula (44);

[0123] Q'(s t ,a t )=Q(s t ,a t )+σ[r t+1 +γmax a Q(s t+1 ,a)-Q(s t ,a t )] (44)

[0124] wherein the learning rate σ is set to 0.2, the discount factor γ is set to 0.95, Q(s t ,a t ) represents the Q value corresponding to the state s t and the action a t before updating, Q'(s t ,a t ) represents the Q value corresponding to the new state s t and the action a t , and max a Q(s t+1 ,a) is the maximum Q value of all possible actions in the new state s t+1 ;

[0125] storing the optimal solution: after updating the Q table, if the new solution is better than the old solution, the old solution is replaced by the new solution and the new solution information is stored;

[0126] (3) obtaining the disassembly workshop AGV scheduling scheme: according to the steps in (2), after initializing the probability matrix, the optimal solution is obtained by repeating the iteration steps of sampling to generate the population operation and updating the probability matrix operation, and selecting different actions based on the Q table for the state of the elite solution, updating the Q table and storing the optimal solution operation, until the iteration number reaches l max =100, the optimal solution is obtained, and the optimal AGV scheduling scheme is output through the decoding rule, so as to realize the optimization scheduling of the disassembly workshop AGV.

[0127] (4) Figure 3 is a scheduling result diagram of the method of the application on a 3-machine, 3-workpiece test set, and the processing machine matrix is shown in formula (45), wherein the element j ikJT represents the machine corresponding to the kth task of workpiece i, the processing time matrix is shown as formula (46), the element jt in JT ik represents the processing time of the kth task of workpiece i on the corresponding machine, the unit is second (s), each position coordinate in Map is located on a 10x10 grid, the unit is meter (m), the first element represents the coordinate position of the warehouse, the middle three elements represent the positions of the three machines, and the last element represents the position of the charging station, assuming that the running speed of the AGV is 1 m / s;

[0128]

[0129] Map = [(0, 4), (4, 9), (4, 5), (6, 0), (9, 5)] (47)

[0130] The learning rate of the update probability formula in the scheduling method is set to α = 0.5, the learning rate of Q learning is set to σ = 0.2, the discount factor γ = 0.95, the population size is 100, the number of elite individuals A size = 20, the number of non-elite individuals B size = 30, the running speed of the AGV v = 1 m / s, the battery capacity E full = 30, the power consumption of the AGV per unit distance E cost = 1, the charging time t cha = 20 s, the minimum power requirement E min = 1;

[0131] The scheduling method proposed by the application obtains the results of the above test set, and the time for completing all tasks is 129 s, Figure 3 is a scheduling scheme diagram, the horizontal coordinate is the scheduling time, the vertical coordinate is the place, the label in the middle of the workpiece block represents the workpiece-task, for example, 1-1 represents the first disassembly task of the first workpiece, and the blue line represents the moving route of the AGV, and the test results prove the effectiveness of the method.

Claims

1. A Q-learning-based disassembly workshop AGV scheduling method, characterized by: constructing a disassembly workshop AGV scheduling model, designing a Q-learning-based disassembly workshop AGV scheduling method, and obtaining a disassembly workshop AGV scheduling scheme; (1) Constructing a disassembly workshop AGV scheduling model with transportation efficiency as the target, specifically: es = 0 ij The start time of the empty transportation of the AGV is 0 before the AGV starts all transportation tasks; otherwise, the start time of the empty transportation of the AGV is equal to the end time le of the loaded transportation of the AGV ij As shown in formula (1): calculating an end time ee of the empty transport of the AGV ij , comprising a start time es of the empty transport of the AGV ij and a time t of the empty transport of the AGV ij as shown in equation (2): ee ij = es ij + t ij (2) wherein AGV empty transport time t ij The AGV from the current position m A to the target position m g required time, as shown in equation (3), The AGV from the current position m A to the target position m g distance, v represents the speed of the AGV; Calculate the start time (ls) of AGV carrying transport. ij That is, the previous disassembly task O. i(j-1) Disassembly completion time c i(j-1) and AGV empty transport end time ee ij The larger of the values ​​in the range, if the AGV is performing its first transport task, then c i(j-1) =0, as shown in formula (4): ls ij = max(c i(j-1) , ee ij ) (4) the end time le of the AGV with load transport ij the start time ls of the AGV with load transport ij and the transport time T of the AGV with load ij the sum, as shown in equation (5): le ij = ls ij +T ij (5) wherein T is the time taken by the AGV to transport the load ij from the position m l where the load is located to the target position m g , as shown in equation (6), denotes the distance from the position m l where the load is located to the target position m g . Considering the power consumption of AGV transportation process, the power consumption is positively correlated with the transportation time. The power change of AGV moving to the next location is shown in formula (7), E t represents the power of AGV at the current time, E t+1 represents the power of AGV after moving from the current location to the next location, E cost represents the power consumption per unit distance of AGV, d' represents the driving distance of AGV between two locations, if AGV is the first transportation task, the power E t of AGV is full power E full ; E t+1 = E t - E cost x d' (7) Before the AGV moves to the next location, it is checked whether the remaining power of the AGV meets the minimum power requirement E min If the remaining power of the AGV can meet the power consumption of the AGV from the current location to the target location and then to the charging station, i.e., the condition shown in formula (8), the AGV continues to perform the current task; otherwise, the AGV goes to the charging station for charging, wherein represents the distance from the target location to the charging station. If the AGV is in an empty state and needs to be charged, the transport time is updated according to formula (9), t cha represents the charging time, represents the distance from the current position m A of the AGV in an empty state to the charging station m1, t' ij represents the updated empty transport time; if the AGV is in a loaded state and needs to be charged, the transport time is updated according to formula (10), represents the distance from the current position m l of the AGV in a loaded state to the charging station m1, T' ij represents the updated loaded transport time, which is increased by the time required for the AGV to travel from its current position to the charging station, the charging time, and the transport time for the AGV to travel from the charging station to the target position; after the AGV completes charging, the power E' is updated according to formula (11), t represents the updated power; E′ t = E full (11) The computing disassembling machine starts to process the disassembling task O ij The maximum value of the time s ij , i.e. the end time le of the AGV carrying transportation ij , and the completion time ms of the pre-sequence disassembling task of the machine k , as shown in formula (12), if the AGV is the first transportation task and all machines are in idle state, then ms k = 0; s ij = max(le ij , ms k ) (12) The calculation and disassembly machine completes the disassembly task. ij Time c ij This includes the machine starting to process the disassembly task. ij time s ij And machine processing disassembly tasks O ij Time p ij As shown in formula (13); c ij = s ij + p ij (13) Establishing a target function for disassembly workshop AGV scheduling considering charging, taking the minimization of the maximum completion time of all disassembly tasks as the scheduling optimization goal, and the function expression is shown in formula (14): (2) Designing a Q-learning-based disassembly workshop AGV scheduling method, including designing coding and decoding rules, initializing a probability matrix, sampling operation of the probability matrix, Q-learning-based probability matrix update rule, and Q-learning-based local search operation, specifically: (a) Designing coding and decoding rules to represent the mapping relationship between solutions and scheduling schemes through coding and decoding: Coding: Coding the disassembly tasks of n workpieces, each workpiece containing m disassembly tasks, using r = (i-1)·m+j to number the jth disassembly task of workpiece i, where i = 1, 2, …, n and j = 1, 2, …, m, all disassembly tasks are numbered from 1 to n×m, and the permutation π = (1, 2, …, r, …, n×m) of length n×m task numbers represents the coded solution, and the order of the permutation is the scheduling order of the tasks; Decoding: Divide all elements in the coded solution by the number of disassembly tasks m, and the quotient plus 1 is the workpiece number i, and the remainder j is the disassembly task number of workpiece i. AGV transports all workpieces to the corresponding disassembly machine position for disassembly in the coding order. Before AGV performs the transportation task, it needs to determine whether the power is sufficient. If not, it needs to charge first and then calculate the transportation time according to the speed and distance of the AGV. After the AGV arrives at the target disassembly machine, the start and completion times of each disassembly task are calculated according to formulas (12) and (13). After the disassembly task is completed, the workpiece can be transported to the next disassembly machine. The maximum completion time of all AGV transportation tasks and disassembly tasks is calculated; (b) Initializing the probability matrix, sampling the probability matrix, and updating the probability model through elite individuals and non-elite individuals: Construct a probability matrix P(l) of n×m rows and n×m columns as shown in formula (15): P(l) denotes the probability matrix of the lth generation, each matrix element denotes the probability that the disassembly task numbered r of the lth generation is located in individual c k (k ∈ [1, n x m]) position, the probability matrix is initialized uniformly, i.e. Generate new individuals through probability matrix sampling operation, and construct a candidate set P' of n rows and m columns to represent m disassembly tasks of n workpieces as shown in formula (16): Considering the disassembly task constraints, determine the column vector P' of the set of optional processes P' from the set of alternative processes P that meet the process constraints o , i.e. select the first non-zero element of each row in the set P' as P' o , if all elements in a row in P' are 0, skip this row and continue to select the first non-zero element of the next row as P' o , and determine P' according to the probability matrix P(l) o The selection probability of each task at the current position c k (k∈[1,n×m]) in P k , assign the element at the corresponding position in P' to 0, and update P' o , until all tasks are completed, i.e. all elements in P' are 0, and a new individual is generated According to the maximum completion time, all individuals are sorted from small to large, and the first 20% are elite individuals, and the last 30% are non-elite individuals. According to the population elite individuals and non-elite individuals, the probability matrix is updated, and the definition is As shown in formula (17): In formula (17), s=a represents an elite individual, s=b represents a non-elite individual, and each element of the probability matrix is updated as shown in formula (18), indicates the probability that the disassembly task numbered r in the l+1th generation is located in the individual c k , and a indicates a learning rate, A size indicates the number of all elite individuals, B size indicates the number of all non-elite individuals; (c) Designing a Q-learning-based local search method for the above elite individuals: Q table is established: three key time indicators are selected, i.e. total time of all disassembly tasks, AGV transportation time and machine idle time; discrete operation is performed on the three indicators: all solutions are arranged in descending order according to the indicators, and three different states are obtained by three equal divisions, and a state space with a total of 3*3*3=27 state combinations can be constructed by combining the three levels; the action space is composed of three local search operators, which are exchange, insertion and reversal; exchange refers to exchanging two disassembly tasks in different positions in the task sequence; insertion refers to inserting a task taken from the task sequence into another position; reversal refers to selecting a continuous disassembly task in the task sequence and reversing the order; a Q table with a size of 27*3 is constructed, each element represents a state-action pair, and the initial value of all elements of the Q table is 0; Action selection: the state of the elite solution is determined according to the three time indicators, the action with the maximum Q value in the state is selected from the Q table, and the action is executed to generate a new individual; Update Q-table: judge the state of new individual and calculate the reward, the reward function is composed of the total time C of all disassembly tasks completed max , AGV transportation time T AGV and machine idle time T ms Improvement amount, for each time index X (X ∈ {C max , T AGV , T ms}), the improvement amount ΔX is calculated by formula (19), wherein, X last represents the value of time index X in the last state, X new represents the value of time index X in the current state; The weighted sum of the three time index improvement amounts r' is obtained from formula (20) t+1 where ΔC max , ΔT AGV , ΔT ms is the improvement amount of the three time indexes calculated from formula (19), and the weight ω c of the total time for completing all disassembly tasks is set to 0.6, the weight ω A of the AGV transportation time is set to 0.2, and the weight ω m of the machine idle time is set to 0.2; the reward function r t+1 As shown in formula (21), if any time index is improved, a positive reward is given, otherwise, a negative reward is given; r' t+1 = ω c · ΔC max + ω A · ΔT AGV + ω m · ΔT ms (20) The Q table is updated using the Q learning update formula shown in formula (22); Q'(s t ,a t ) = Q(s t ,a t ) + σ[r t+1 + γmax a Q(s t+1 ,a) - Q(s t ,a t )] (22) where σ (0 < σ < 0.7) represents a learning rate, γ (0.5 < γ < 1) is a discount factor, Q(s t ,a t ) represents a Q value corresponding to a state s t and an action a t before update, Q'(s t ,a t ) represents a Q value corresponding to a new state s t and an action a t , and max a Q(s t+1 ,a) represents a maximum Q value of all possible actions in the new state s t+1 . Store the optimal solution: after updating the Q table, if the new solution is better than the old solution, replace the old solution and store the new solution information; (3) Obtain the AGV scheduling scheme of the disassembly workshop: according to the steps in (2), after initializing the probability matrix, the population operation is generated by repeating the iteration step and the probability matrix is updated, and the state of the elite solution is selected based on the Q table to select different actions, update the Q table and store the optimal solution operation, until the iteration number reaches l max = 100, the optimal solution is obtained, and the optimal AGV scheduling scheme is output through the decoding rule, and the optimization scheduling of the AGV of the disassembly workshop is realized.

Images