A joint scheduling method for job shop machines and AGVs based on deep reinforcement learning

Abstract

The present invention belongs to the field of workshop scheduling, and specifically relates to a method for co-scheduling machines and AGVs in a job shop based on deep reinforcement learning. The method comprises the following steps: Step 1: Expressing the co-scheduling problem of AGVs and machines in a job shop as a directed disjunctive graph G; Step 2: Separating the machine steps and transfer steps in the disjunctive graph G through topological sorting, and converting the problem into a Markov decision process; Step 3: Establishing a Markov decision process model; Step 4: Establishing a deep reinforcement learning network architecture and agent model for solving the problem; Step 5: Initializing and optimizing the parameters of the deep reinforcement learning agent network to train and obtain an optimal scheduling strategy; Step 6: Deploying the trained deep reinforcement learning agent in the shop and scheduling according to the shop's real-time production situation. The present invention can solve the technical problems of low efficiency and poor robustness of existing co-scheduling methods for AGVs and machines in job shops.

Description

Technical Field

[0001] The present invention belongs to the field of workshop scheduling, and specifically relates to a joint scheduling method for workshop machines and AGVs based on deep reinforcement learning. Background Art

[0002] The manufacturing industry's paradigm shift from mass production to mass customization requires more flexible production methods to meet society's demand for a large number of goods. Against this backdrop, flexible manufacturing systems (FMS) have emerged. An FMS consists of a unified information control system, a material storage and transportation system, and a data processing system, enabling manufacturers to dynamically adjust to fluctuations in market demand. Traditional labor-intensive material handling costs account for approximately 30%-70% of total manufacturing costs. To save costs, improve production efficiency, and shorten product interaction time, material handling is often performed by automated guided vehicles (AGVs). Job shops are the most common form of FMS, and their scheduling problem is one of the most difficult. However, due to the inclusion of AGVs and the sequential nature of process processing, workpieces must be transferred between machines multiple times by different AGVs, further increasing the complexity and difficulty of the problem.

[0003] In recent years, many researchers have begun studying the joint scheduling problem of automated guided vehicles (AGVs) and machines in job shops, often employing linear programming, constraint programming, heuristic rules, and metaheuristic algorithms to solve it. However, with the exception of heuristic rules, most of these methods rely on iterative solutions, which are subject to numerous limitations. First, to find a near-optimal solution, the algorithm requires numerous iterations of population updates or iterative searches. This is particularly true for extremely complex production and logistics problems, resulting in long runtimes for optimization. Second, even minor changes to the problem, such as changes in the number of workpieces, number of processes, processing time, or number of AGVs, may necessitate re-running the algorithm. Third, when encountering new scheduling problems, or even new instances of similar problems, the algorithm must be modified to achieve optimal results. This is known as the no-free-lunch theorem. Fourth, while these methods can certainly achieve high performance, they require specific algorithm design for the specific problem scenario.

[0004] Therefore, how to solve the technical problems of low efficiency and poor robustness of the existing joint scheduling methods of AGVs and machines in job workshops, and stably and efficiently achieve the simultaneous optimization of the two sub-problems of process sequencing and AGV assignment, has become an urgent problem to be solved. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the existing technology, the present invention provides a joint scheduling method for machines and AGVs in a job shop based on deep reinforcement learning, which can solve the technical problems of low efficiency and poor robustness of the existing joint scheduling method for AGVs and machines in a job shop.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for joint scheduling of machines and AGVs in a job shop based on deep reinforcement learning, comprising the following steps:

[0008] Step 1: Express the joint scheduling problem of AGVs and machines in the job shop as a directed disjunctive graph G;

[0009] Step 2: Use topological sorting to separate the machine processes and transfer processes in the disjunctive graph G, transforming the job shop AGV and machine joint scheduling problem into a sequential decision problem related to process sorting, and then converting it into a Markov decision process.

[0010] Step 3: Based on the Markov decision process obtained in step 2, a Markov decision process model for the joint scheduling problem of AGVs and machines in the job shop is established;

[0011] Step 4: Based on the Markov decision process model established in step 3, a deep reinforcement learning network architecture and intelligent agent model are established to solve the joint scheduling problem of AGVs and machines in the job shop.

[0012] Step 5: Initialize and optimize the parameters of the deep reinforcement learning agent network established in Step 4, and train the network to obtain the optimal scheduling strategy for the joint scheduling problem of AGVs and machines in the job shop.

[0013] Step 6: Deploy the trained deep reinforcement learning agent for joint scheduling of AGVs and machines in the workshop and perform scheduling based on the real-time production situation of the workshop.

[0014] Preferably, in step 1, the directed disjunctive graph G is G=(V O ∪V T ,C∪D m ∪D r );

[0015] The disjunctive graph G contains V O and V T Two types of vertices, two virtual vertices s and e, and three types of directed edge sets C and D m and D r ; Among them, V O Indicates that the scheduling system includes all machining processes, V TThe scheduling system includes all transportation and delivery processes; s represents the start; e represents the end; C represents the priority constraint between processes, and D m Indicates the processing sequence of the process on the machine, D r Indicates the assignment of the process; arrive The weight of the directed arc is the machining time; arrive The weight of the directed arc is the transportation time; V O A vertex in V T A vertex in ; directed arc D r After assigning the process to the AGV, D r The weight of is the pickup time.

[0016] Preferably, in step 2, the topological sorting process includes: placing the vertex V O Convert the vertex V into a linear sequence T Arrange them into linear sequences of the same number according to the number of AGVs; and add virtual operations to each job so that the number of machine steps and transportation steps are consistent, so that any pair of vertices in the disjunctive graph G has a clear order;

[0017] After converting the job shop AGV and machine joint scheduling problem into a Markov decision process, the state transfer process includes: at step t, the DRL agent observes the state s t and select action a t The next step of the processing is done by action a t The decision is made and the greedy scheduling rule is used to assign the process to the corresponding AGV to transfer the workpiece. Then the workshop environment feeds back the corresponding reward to the DRL agent and transfers it to the next state.

[0018] Glossary: ​​DRL, namely deep reinforcement learning.

[0019] Preferably, in step 3, when establishing the Markov decision process model, the construction content includes state design, action design and reward function design.

[0020] Preferably, in step 3, when performing state design, the state is constructed by extracting the abstract information of the graph G and the preset artificial features;

[0021] Express the abstract information of the disjunctive graph G as a four-dimensional matrix S t ={PT t ,ST t ,ET t ,D t}, St The size is 4×W×n; W represents the total number of workpieces, and n represents the maximum number of processes;

[0022] Among them, PT t It is used to represent the disjunctive arc C in the disjunctive graph; and based on the two kinds of information contained in the disjunctive arc C, PT t Represents a matrix of the following form:

[0023]

[0024] in, represents the delivery time of the jth transportation process of workpiece i, p ij represents the processing time of the jth process of workpiece i;

[0025] The disjunctive arc C contains two types of information: one is the information of the directed arc from the transportation process to the machining process, which represents the delivery time of the workpiece from the current machining position to the next machining position; the other is the information of the directed arc from the machining process to the transportation process, which represents the processing time required for the current machining process;

[0026] ST t Used to indicate the start time of the disjunctive arc in the disjunctive graph G; ET t Used to indicate the end time of the disjunctive arc in the disjunctive graph G; at step t, ST t Represents the start time matrix of all completed transportation and processing operations, ET t Represents the end time matrix of all completed transportation and processing procedures, and the position corresponding to each procedure is related to PT t The matrices are completely consistent; at step t, for a given action a, ST t The state transfer will end by assigning the start time of the current transportation process and processing process, ET t The state transfer will end by assigning completion time to the current transportation process and processing process;

[0027] D t Used to represent the assignment of each process in the disjunctive graph G; D t The parameters consist of the machine index and the AGV index; at step t, for a given action a, D t The state transition ends with the position of the current transport process being assigned to the index of the AGV assigned to the corresponding action; for transport processes and virtual processes that are not assigned AGVs, the parameters of the corresponding positions are set to 0.

[0028] Preferably, in step 3, when performing state design, the preset artificial feature is a one-dimensional vector The length is 3+r+m; r represents the total number of AGVs, and m represents the total number of machines;

[0029] Among them, PT art represents the average remaining processing time of all jobs; Indicates AGV R i The ratio of the current total transport time to its current completion time; pRRT t It represents the remaining average effective transportation time, that is, the AGV transfers the workpiece from the current position to the next processing position; Indicates machine M i The ratio of the current load to its current completion time; pML t It represents the ratio of the total machine load to the machine completion time.

[0030] Preferably, in step 3, when designing the action, the action is expressed as a vector a t =(OR t ,VR t ), used to represent the scheduling behavior at the decision point; OR t represents the process selection rule at the decision point t, VR t represents the AGV allocation rule at the decision point t steps; the AGV allocation rule is a first-come-first-served scheduling rule;

[0031] OR consists of 8 heuristic rules, OR = {FCFS, SOPT, SJPT, SRW, PDJT, PDRW, PMJT, PMRW}, which are described in the following table:

[0032]

[0033] Preferably, in step 3, the reward function design process includes: based on the characteristics of the job shop AGV and machine joint scheduling problem, without considering the heterogeneity of AGVs in the absence of machine flexibility, taking the sum of machine processing time and delivery time as a fixed value K; K is calculated as follows:

[0034]

[0035] Among them, h i represents the total number of processes for workpiece i;

[0036] On this basis, at step t, the reward r(t) of the DRL agent is defined as:

[0037] r(t)=U(t)-U(t-1);

[0038] in,

[0039] Where U(t) represents the average utilization of the machine and AGV at time step t, and U(0) = 0; Ct represents the completion time of time step t, K(t) represents the total workload completed by the machine and AGV in step t; F i Indicates the number of completed processing steps for workpiece i, FT i Indicates the number of completed transportation processes for workpiece i.

[0040] Preferably, in step 4, D3QN is used as a deep reinforcement learning algorithm for the joint scheduling problem of AGVs and machines in the job shop; and priority experience replay is used as the experience pool in the learning process of D3QN;

[0041] The D3QN network construction process includes:

[0042] The abstract information S of the disjunctive graph is extracted through convolutional neural network t For feature extraction, in the convolutional neural network, the first hidden layer contains 12 filters with a kernel size of (3,3), a stride of 1, a padding of 1, and an activation function of ReLU. The output size after the first hidden layer is (12, W, n); the second hidden layer is a maximum pooling layer with a size of (2, 2) and a stride of 2, which can reduce the dimension of the features and extract the main features. The output size after the second hidden layer is (12, int(W / 2), int(n / 2)); when the second hidden layer is expanded, a fully connected layer with a length of 12×int(W / 2)×int(n / 2) is obtained;

[0043] After that, the fully connected layer with a length of 12×int(W / 2)×int(n / 2) is combined with the artificial feature S t ～ The concatenation is performed to obtain a fully connected layer with a length of 12×int(W / 2)×int(n / 2)+3+r+m; then, the concatenated features are input into a fully connected layer with 258 units and connected to the duel network; the duel network divides the network into a value function network and an advantage function network; the first layer of the value function network and the advantage function network are both fully connected layers with 128 nodes; the second layer of the value function network is a value function of 7 nodes; the second layer of the advantage function network is an advantage function of 1 linear node.

[0044] Preferably, in step 5, the process of network parameter optimization includes:

[0045]

[0046] θ t represents the network parameters at step t, Denotes the target network function value at step t, Q is the action value function. In D3QN, the dueling network architecture is used to estimate the state value function V(s) and the advantage function A(s,a). Then, the state value function V(s) and the advantage function A(s,a) are combined to estimate the action value function Q(s,a). The calculation method is as follows:

[0047]

[0048] Where α and β are the weight parameter and bias parameter of the corresponding fully connected layer respectively.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) The present invention provides a problem conversion method for the joint scheduling problem of AGVs and machines in a job shop. By modeling the joint scheduling problem of AGVs and machines in a job shop using a disjunctive graph and converting it into a topological sorting form, the joint scheduling problem of AGVs and machines in a job shop can be clearly converted into a sequential decision problem, which is more conducive to constructing an MDP model.

[0051] (2) The present invention designs a state expression method based on disjunctive graphs and artificial feature values. Compared with the state expression method that only uses artificial feature values, this expression method can better reflect the state of the environment and is more conducive to the interactive learning between the intelligent agent and the environment.

[0052] (3) The present invention aims to construct a reward function based on the characteristic that the processing time and the effective transportation time in the job shop are fixed values. The cumulative reward is inversely proportional to the completion time, which can effectively guide the intelligent agent to learn in the direction of target optimization, resulting in better optimization performance.

[0053] (4) The present invention adopts the D3QN algorithm as the learning algorithm for the joint scheduling problem of AGVs and machines in the job shop, and builds a network structure that can be used to solve two state expressions, so that the environmental state and action can be better fitted, so that appropriate scheduling actions can be given according to the corresponding state during the scheduling process, making the scheduling solution more robust.

[0054] (5) The key advantage of the deep reinforcement learning-based joint scheduling method for job shop AGVs and machines proposed in this invention is that it learns how to determine the optimal scheduling action based on various state characteristics in the job shop AGV and machine joint scheduling environment, which can further improve the adaptability of the algorithm compared with a single scheduling rule.

[0055] (6) When designing actions, OR consists of 8 heuristic rules, which can avoid invalid actions in the scheduling process and overcome the short-sightedness of a single rule.

[0056] In summary, the present invention can solve the technical problems of low efficiency and poor robustness of the existing joint scheduling method of AGVs and machines in job shops, and stably and efficiently realize the simultaneous optimization of the two sub-problems of process sequencing and AGV assignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to make the purpose, technical solutions and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0058] Figure 1 is a flow chart of the present invention;

[0059] Figure 2 This is a schematic diagram of an example of a disjunctive graph in an embodiment;

[0060] Figure 3 A schematic diagram of topological sorting in an embodiment;

[0061] Figure 4 Schematic diagram of the state transition process in the embodiment;

[0062] Figure 5 Schematic diagram of the network structure of D3QN in the embodiment;

[0063] Figure 6 Schematic diagram of the scheduling process of the deep reinforcement learning agent in the workshop in the embodiment. DETAILED DESCRIPTION

[0064] The following is a further detailed description through specific implementation methods:

[0065] Example:

[0066] like Figure 1 As shown, the present invention provides a joint scheduling method for machine and AGV in a job shop based on deep reinforcement learning, comprising the following steps:

[0067] Step 1: Express the joint scheduling problem of AGVs and machines in the job shop as a directed disjunctive graph G;

[0068] In specific implementation, the directed disjunctive graph G is G=(V O ∪V T ,C∪D m ∪D r );

[0069] The disjunctive graph G contains V O and V T Two types of vertices, two virtual vertices s and e, and three types of directed edge sets C and D m and D r ; Among them, V O Indicates that the scheduling system includes all machining processes, V TThe scheduling system includes all transportation and delivery processes; s represents the start; e represents the end; C represents the priority constraint between processes, and D m Indicates the processing sequence of the process on the machine, D r Indicates the assignment of the process; arrive The weight of the directed arc is the machining time; arrive The weight of the directed arc is the transportation time; V O A vertex in V T A vertex in ; directed arc D r After assigning the process to the AGV, D r The weight of is the pickup time.

[0070] For easier understanding, let’s take an example to illustrate. Figure 2 As shown, Figure 2 (a) is an undirected disjunctive graph of a simple example before solving. To solve the joint scheduling problem of AGV and machines in the job shop, it is necessary to set all the undirected arcs D m Convert to directed arc and add directed arc D r Assign AGV to each transport process, where D r The weight of is the pickup time, so the disjunctive graph of a feasible solution to the problem is as follows Figure 2 (b) shown.

[0071] Step 2: Separate the machine process and transfer process in the disjunctive graph G through topological sorting, and transform the joint scheduling problem of AGV and machines in the job shop into a sequential decision problem related to process sorting, and then convert it into a Markov decision process.

[0072] The topological sorting process includes: O Convert the vertex V into a linear sequence T Arrange them into linear sequences of the same number according to the number of AGVs; and add virtual operations to each job so that the number of machine steps and transportation steps are consistent, so that any pair of vertices in the disjunctive graph G has a clear order;

[0073] After converting the job shop AGV and machine joint scheduling problem into a Markov decision process, the state transfer process includes: at step t, the DRL agent observes the state s t and select action a t The next step of the processing is done by action a t The decision is made and the greedy scheduling rule is used to assign the process to the corresponding AGV to transfer the workpiece. Then the workshop environment feeds back the corresponding reward to the DRL agent and transfers it to the next state.

[0074] Among them, DRL stands for deep reinforcement learning.

[0075] Deep reinforcement learning (DRL), a combination of deep learning (DL) and reinforcement learning (RL), demonstrates powerful data processing and environmental interaction capabilities in terms of self-adaptation and self-learning, offering new solutions for complex workshop manufacturing systems. Unlike many classic methods, DRL-based approaches can solve similar problems without retraining for each new instance, making them easy to deploy in real-world applications.

[0076] The joint scheduling problem of AGVs and machines in a job shop modeled with a disjunctive graph is difficult to be directly viewed as a sequential decision problem. In order to intuitively mine the information of the directed disjunctive graph, the machine process and the transfer process are separated by topological sorting. Therefore, the vertex V O is converted into a linear sequence, the vertex V T It is arranged into several linear sequences according to the number of AGVs. In order to match the machine steps and transportation steps, virtual operations are added to each job to make the number of machine steps and transportation steps consistent. Thus, any pair of vertices in G has a clear order. In this way, the final scheduling result of topological sorting is consistent with the scheduling result of the directed disjunctive graph. Figure 2 Taking (b) as an example, for the three jobs in the directed disjunctive graph, three virtual processes O are added. 14 , O 24 and O 34 .so, Figure 2 The feasible scheduling solution in (b) can be transformed into Figure 3 The topological sorting shown in Figure 1 is used. Through this transformation, the joint scheduling problem of AGVs and machines in the job shop becomes a sequential decision problem related to process sorting, which can be converted into a Markov decision process. The order of processes is determined by using a deep reinforcement learning agent, and the allocation of robots is determined by the greedy scheduling rule. The specific process of state transition is shown in Figure 1. Figure 4 As shown, at step t, the DRL agent observes the state s t and select action a t The next step of the processing is done by action a t The decision is made and a greedy scheduling rule is used to assign the process to the corresponding AGV to transfer the workpiece. The workshop environment then feeds back the corresponding reward to the DRL agent and simultaneously transitions to the next state. It is worth noting that state transitions are independent of time and the node of the state transition is the completion of the current process.

[0077] Step 3: Based on the Markov decision process obtained in Step 2, a Markov decision process model for the joint scheduling problem of AGVs and machines in the job shop is established. Specifically, the Markov decision process model is constructed by designing states, actions, and reward functions.

[0078] The state represents the environmental information perceived by the DRL agent and the changes brought about by its own behavior. The state serves as the basis for the DRL agent's decision-making and long-term reward evaluation. It also serves as the basis for evaluating the convergence, convergence speed, and ultimate execution of the DRL algorithm. Therefore, the design of the state requires a deep understanding of the task logic. The joint scheduling problem of AGVs and machines in a job shop involves transforming a disjunctive graph from an undirected disjunctive graph to a directed disjunctive graph with the goal of minimizing the completion time.

[0079] When designing a state, the state is constructed by extracting the abstract information of the graph G and the preset artificial features;

[0080] Express the abstract information of the disjunctive graph G as a four-dimensional matrix S t ={PT t ,ST t ,ET t ,D t}, S t The size is 4×W×n; W represents the total number of workpieces, and n represents the maximum number of processes;

[0081] Among them, PT t It is used to represent the disjunctive arc C in the disjunctive graph; and based on the two kinds of information contained in the disjunctive arc C, PT t Represents a matrix of the following form:

[0082]

[0083] in, represents the delivery time of the jth transportation process of workpiece i, p ij represents the processing time of the jth process of workpiece i;

[0084] The disjunctive arc C contains two types of information: one is the information of the directed arc from the transportation process to the machining process, which represents the delivery time of the workpiece from the current machining position to the next machining position; the other is the information of the directed arc from the machining process to the transportation process, which represents the processing time required for the current machining process;

[0085] ST t Used to indicate the start time of the disjunctive arc in the disjunctive graph G; ET t Used to indicate the end time of the disjunctive arc in the disjunctive graph G; at step t, ST tRepresents the start time matrix of all completed transportation and processing operations, ET t Represents the end time matrix of all completed transportation and processing procedures, and the position corresponding to each procedure is related to PT t The matrices are completely consistent; at step t, for a given action a, ST t The state transfer will end by assigning the start time of the current transportation process and processing process, ET t The state transfer will end by assigning completion time to the current transportation process and processing process;

[0086] D t Used to represent the assignment of each process in the disjunctive graph G; D t The parameters consist of the machine index and the AGV index; at step t, for a given action a, D t The state transition ends with the position of the current transport process being assigned to the index of the AGV assigned to the corresponding action; for transport processes and virtual processes that are not assigned AGVs, the parameters of the corresponding positions are set to 0.

[0087] The preset artificial feature is a one-dimensional vector The length is 3+r+m; r represents the total number of AGVs, and m represents the total number of machines;

[0088] Among them, PT art represents the average remaining processing time of all jobs; Indicates AGV R i The ratio of the current total transport time to its current completion time; pRRT t It represents the remaining average effective transportation time, that is, the AGV transfers the workpiece from the current position to the next processing position; Indicates machine M i The ratio of the current load to its current completion time; pML t It represents the ratio of the total machine load to the machine completion time.

[0089] When designing an action, express the action as a vector a t =(OR t ,VR t ), used to represent the scheduling behavior at the decision point; OR t represents the process selection rule at the decision point t, VR t represents the AGV allocation rule at the decision point t steps; the AGV allocation rule is a first-come-first-served scheduling rule;

[0090] OR consists of 8 heuristic rules, OR = {FCFS, SOPT, SJPT, SRW, PDJT, PDRW, PMJT, PMRW}, which are described in the following table:

[0091]

[0092] OR consists of 8 heuristic rules, which can avoid invalid actions in the scheduling process and overcome the short-sightedness of a single rule.

[0093] The reward function design process includes: based on the characteristics of the joint scheduling problem of AGVs and machines in the job shop, without considering the heterogeneity of AGVs in the absence of machine flexibility, and taking the sum of machine processing time and delivery time as a fixed value K; K is calculated as follows:

[0094]

[0095] Among them, h i represents the total number of processes for workpiece i;

[0096] On this basis, at step t, the reward r(t) of the DRL agent is defined as:

[0097] r(t)=U(t)-U(t-1);

[0098] in,

[0099] Where U(t) represents the average utilization of the machine and AGV at time step t, and U(0) = 0; C t represents the completion time of time step t, K(t) represents the total workload completed by the machine and AGV in step t; F i Indicates the number of completed processing steps for workpiece i, FT i Indicates the number of completed transportation processes for workpiece i.

[0100] Step 4: Based on the Markov decision process model established in step 3, a deep reinforcement learning network architecture and intelligent agent model are established to solve the joint scheduling problem of AGVs and machines in the job shop.

[0101] By transforming the job shop AGV and machine co-scheduling problem into a Markov decision process, D3QN can be used for training and solving. Therefore, in step 4, D3QN is used as the deep reinforcement learning algorithm for the job shop AGV and machine co-scheduling problem, and prioritized experience replay is used as the experience pool during D3QN's learning process.

[0102] D3QN consists of an online network and a target network, both of which adopt the structure of convolutional neural network and duel network. The network structure of D3QN is as follows Figure 5 shown.

[0103] The D3QN network construction process includes:

[0104] The abstract information S of the disjunctive graph is extracted through convolutional neural network t For feature extraction, in the convolutional neural network, the first hidden layer contains 12 filters with a kernel size of (3,3), a stride of 1, a padding of 1, and an activation function of ReLU. The output size after the first hidden layer is (12, W, n); the second hidden layer is a maximum pooling layer with a size of (2, 2) and a stride of 2, which can reduce the dimension of the features and extract the main features. The output size after the second hidden layer is (12, int(W / 2), int(n / 2)); when the second hidden layer is expanded, a fully connected layer with a length of 12×int(W / 2)×int(n / 2) is obtained;

[0105] After that, the fully connected layer with a length of 12×int(W / 2)×int(n / 2) is combined with the artificial feature S t ～ The concatenation is performed to obtain a fully connected layer with a length of 12×int(W / 2)×int(n / 2)+3+r+m; then, the concatenated features are input into a fully connected layer with 258 units and connected to the duel network; the duel network divides the network into a value function network and an advantage function network; the first layer of the value function network and the advantage function network are both fully connected layers with 128 nodes; the second layer of the value function network is a value function of 7 nodes; the second layer of the advantage function network is an advantage function of 1 linear node.

[0106] Step 5: Initialize and optimize the parameters of the deep reinforcement learning agent network established in Step 4, and train the network to obtain the optimal scheduling strategy for the joint scheduling problem of AGVs and machines in the job shop.

[0107] In specific implementation, the process of network parameter optimization includes:

[0108]

[0109] θ t represents the network parameters at step t, Denotes the target network function value at step t, Q is the action value function. In D3QN, the dueling network architecture is used to estimate the state value function V(s) and the advantage function A(s,a). Then, the state value function V(s) and the advantage function A(s,a) are combined to estimate the action value function Q(s,a). The calculation method is as follows:

[0110]

[0111] Where α and β are the weight parameter and bias parameter of the corresponding fully connected layer respectively.

[0112] Step 6: Deploy the trained deep reinforcement learning agent for joint scheduling of AGV and machines in the workshop and perform scheduling based on the real-time production situation of the workshop. Figure 6 As shown in the figure, when a workpiece is released to the workshop, scheduling is triggered immediately. The workshop environment is abstracted into disjunctive graph states and human states. The D3QN agent and greedy AGV assignment rules complete the scheduling of existing processing tasks until the undirected disjunctive graph is transformed into a directed disjunctive graph, and processing is completed. The disjunctive graph transformation is synchronized with the actual workshop conditions, and changes in processing conditions, such as changes in processing time and AGV transport time, are reflected in the disjunctive graph in real time.

[0113] The present invention provides a problem conversion method for the joint scheduling problem of AGVs and machines in a job shop. By modeling the joint scheduling problem of AGVs and machines in a job shop using a disjunctive graph and converting it into a topological sorting form, the joint scheduling problem of AGVs and machines in a job shop can be clearly converted into a sequential decision problem, which is more conducive to constructing an MDP model. In addition, the present invention designs a state expression method based on a disjunctive graph and artificial eigenvalues. Compared with the state expression method that only uses artificial eigenvalues, this expression method can better reflect the state of the environment and is more conducive to the interactive learning between the intelligent agent and the environment. Moreover, when designing actions, OR is composed of 8 heuristic rules, which can avoid invalid actions in the scheduling process and overcome the short-sightedness of a single rule. In addition, the present invention constructs a reward function based on the characteristic that the processing time and effective transportation time in the job shop are certain values. The cumulative reward is inversely proportional to the completion time, which can effectively guide the intelligent agent to learn in the direction of target optimization, resulting in better optimization performance.

[0114] This paper uses the D3QN algorithm as a learning algorithm for the joint scheduling problem of AGVs and machines in job shops. It builds a network structure that can be used to solve two state expressions, allowing for a better fit between environmental states and actions. This allows for appropriate scheduling actions to be given according to the corresponding states during the scheduling process, making the scheduling solution more robust. Furthermore, the key advantage of the deep reinforcement learning-based joint scheduling method for AGVs and machines in job shops proposed in this paper is that it learns how to determine the optimal scheduling action based on various state characteristics in the joint scheduling environment of AGVs and machines in the job shop. This can further improve the adaptability of the algorithm compared to a single scheduling rule.

[0115] In summary, the present invention can solve the technical problems of low efficiency and poor robustness of the existing joint scheduling method of AGVs and machines in job shops, and stably and efficiently realize the simultaneous optimization of the two sub-problems of process sequencing and AGV assignment.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the technical solutions. Those skilled in the art should understand that modifications or equivalent replacements of the technical solutions of the present invention that do not depart from the purpose and scope of the technical solutions of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A joint scheduling method for machine and AGV in a job shop based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Express the joint scheduling problem of AGVs and machines in the job shop as a directed disjunctive graph G; Step 2: Use topological sorting to separate the machine processes and transfer processes in the disjunctive graph G, transforming the job shop AGV and machine joint scheduling problem into a sequential decision problem related to process sorting, and then converting it into a Markov decision process. Step 3: Based on the Markov decision process obtained in step 2, a Markov decision process model for the joint scheduling problem of AGVs and machines in the job shop is established; Step 4: Based on the Markov decision process model established in step 3, a deep reinforcement learning network architecture and intelligent agent model are established to solve the joint scheduling problem of AGVs and machines in the job shop. Step 5: Initialize and optimize the parameters of the deep reinforcement learning agent network established in Step 4, and train the network to obtain the optimal scheduling strategy for the joint scheduling problem of AGVs and machines in the job shop. Step 6: Deploy the trained deep reinforcement learning agent for joint scheduling of AGVs and machines in the workshop and perform scheduling based on the real-time production situation of the workshop; Among them, in step 1, the directed disjunctive graph G is G=(V O ∪V T ,C∪D m ∪D r ); The disjunctive graph G contains V O and V T Two types of vertices, two virtual vertices s and e, and three types of directed edge sets C and D m and D r ; Among them, V O Indicates that the scheduling system includes all machining processes, V T The scheduling system includes all transportation and delivery processes; s represents the start; e represents the end; C represents the priority constraint between processes, and D m Indicates the processing sequence of the process on the machine, D r Indicates the assignment of the process; arrive The weight of the directed arc is the machining time; arrive The weight of the directed arc is the transportation time; V O A vertex in V T A vertex in ; directed arc D r After assigning the process to the AGV, D r The weight of is the pickup time.

2. The method for joint scheduling of machines and AGVs in a job shop based on deep reinforcement learning according to claim 1, characterized in that: In step 2, the topological sorting process includes: placing the vertex V O Convert the vertex V into a linear sequence T Arrange them into linear sequences of the same number according to the number of AGVs; and add virtual operations to each job so that the number of machine steps and transportation steps are consistent, so that any pair of vertices in the disjunctive graph G has a clear order; After converting the job shop AGV and machine joint scheduling problem into a Markov decision process, the state transfer process includes: at step t, the DRL agent observes the state s t and select action a t The next step of the processing is done by action a t The decision is made and the greedy scheduling rule is used to assign the process to the corresponding AGV to transfer the workpiece. Then the workshop environment feeds back the corresponding reward to the DRL agent and transfers to the next state.

3. The method for joint scheduling of machines and AGVs in a job shop based on deep reinforcement learning according to claim 2, characterized in that: In step 3, when establishing the Markov decision process model, the construction content includes state design, action design and reward function design.

4. The method for joint scheduling of machine and AGV in a job shop based on deep reinforcement learning according to claim 3, characterized in that: In step 3, when designing the state, the state is constructed by extracting the abstract information of graph G and the preset artificial features; Express the abstract information of the disjunctive graph G as a four-dimensional matrix S t ={PT t ,ST t ,ET t ,D t }, S t The size is 4×W×n; W represents the total number of workpieces, and n represents the maximum number of processes; Among them, PT t It is used to represent the disjunctive arc C in the disjunctive graph; and based on the two kinds of information contained in the disjunctive arc C, PT t Represents a matrix of the following form: in, represents the delivery time of the jth transportation process of workpiece i, p ij represents the processing time of the jth process of workpiece i; The disjunctive arc C contains two types of information: one is the information of the directed arc from the transportation process to the machining process, which represents the delivery time of the workpiece from the current machining position to the next machining position; the other is the information of the directed arc from the machining process to the transportation process, which represents the processing time required for the current machining process; ST t Used to indicate the start time of the disjunctive arc in the disjunctive graph G; ET t Used to indicate the end time of the disjunctive arc in the disjunctive graph G; at step t, ST t Represents the start time matrix of all completed transportation and processing operations, ET t Represents the end time matrix of all completed transportation and processing procedures, and the position corresponding to each procedure is related to PT t The matrices are completely consistent; at step t, for a given action a, ST t The state transfer will end by assigning the start time of the current transportation process and processing process, ET t The state transfer will end by assigning completion time to the current transportation process and processing process; D t Used to represent the assignment of each process in the disjunctive graph G; D t The parameters consist of the machine index and the AGV index; at step t, for a given action a, D t The state transition ends with the position of the current transport process being assigned to the index of the AGV assigned to the corresponding action; for transport processes and virtual processes that are not assigned AGVs, the parameters of the corresponding positions are set to 0.

5. The method for joint scheduling of machines and AGVs in a job shop based on deep reinforcement learning according to claim 4, characterized in that: In step 3, when performing state design, the preset artificial feature is a one-dimensional vector The length is 3+r+m; r represents the total number of AGVs, and m represents the total number of machines; Among them, PT art represents the average remaining processing time of all jobs; Indicates AGV R i The ratio of the current total transport time to its current completion time; pRRT t It represents the remaining average effective transportation time, that is, the AGV transfers the workpiece from the current position to the next processing position; Indicates machine M i The ratio of the current load to its current completion time; pML t It represents the ratio of the total machine load to the machine completion time.

6. The method for joint scheduling of machines and AGVs in a job shop based on deep reinforcement learning according to claim 5, characterized in that: In step 3, when designing the action, the action is expressed as a vector a t =(OR t ,VR t ), used to represent the scheduling behavior at the decision point; OR t represents the process selection rule at the decision point t, VR t represents the AGV allocation rule at the decision point t steps; the AGV allocation rule is a first-come-first-served scheduling rule; OR consists of 8 heuristic rules, OR = {FCFS, SOPT, SJPT, SRW, PDJT, PDRW, PMJT, PMRW}, which are described in the following table:

7. The method for joint scheduling of machines and AGVs in a job shop based on deep reinforcement learning according to claim 6, characterized in that: In step 3, the reward function design process includes: based on the characteristics of the joint scheduling problem of AGVs and machines in the job shop, without considering the heterogeneity of AGVs in the absence of machine flexibility, and taking the sum of machine processing time and delivery time as a fixed value K; K is calculated as follows: Among them, h i represents the total number of processes for workpiece i; On this basis, at step t, the reward r(t) of the DRL agent is defined as: r(t)=U(t)-U(t-1); in, Where U(t) represents the average utilization of the machine and AGV at time step t, and U(0) = 0; C t represents the completion time of time step t, K(t) represents the total workload completed by the machine and AGV in step t; F i Indicates the number of completed processing steps for workpiece i, FT i Indicates the number of completed transportation processes for workpiece i.

8. The method for joint scheduling of machines and AGVs in a job shop based on deep reinforcement learning according to claim 7, characterized in that: In step 4, D3QN is used as the deep reinforcement learning algorithm for the joint scheduling problem of AGVs and machines in the job shop; and priority experience replay is used as the experience pool in the learning process of D3QN; The D3QN network construction process includes: The abstract information S of the disjunctive graph is extracted through convolutional neural network t For feature extraction, in the convolutional neural network, the first hidden layer contains 12 filters with a kernel size of (3,3), a stride of 1, a padding of 1, and an activation function of ReLU. The output size after the first hidden layer is (12, W, n); the second hidden layer is a maximum pooling layer with a size of (2, 2) and a stride of 2, which can reduce the dimension of the features and extract the main features. The output size after the second hidden layer is (12, int(W / 2), int(n / 2)); when the second hidden layer is expanded, a fully connected layer with a length of 12×int(W / 2)×int(n / 2) is obtained; After that, the fully connected layer with a length of 12×int(W / 2)×int(n / 2) is combined with the artificial features The concatenation is performed to obtain a fully connected layer with a length of 12×int(W / 2)×int(n / 2)+3+r+m; then, the concatenated features are input into a fully connected layer with 258 units and connected to the duel network; the duel network divides the network into a value function network and an advantage function network; the first layer of the value function network and the advantage function network are both fully connected layers with 128 nodes; the second layer of the value function network is a value function of 7 nodes; the second layer of the advantage function network is an advantage function of 1 linear node.

9. The method for joint scheduling of machines and AGVs in a job shop based on deep reinforcement learning according to claim 8, characterized in that: In step 5, the network parameter optimization process includes: θ t represents the network parameters at step t, Denotes the target network function value at step t, and Q is the action value function. In D3QN, the dueling network architecture is used to estimate the state value function V(s) and the advantage function A(s,a). Then, the state value function V(s) and the advantage function A(s,a) are combined to estimate the action value function Q(s,a). The calculation method is as follows: Where α and β are the weight parameter and bias parameter of the corresponding fully connected layer respectively.

Images