A job-shop scheduling method under transportation constraints based on deep reinforcement learning

Abstract

The application discloses a kind of job shop scheduling methods under transportation constraint based on deep reinforcement learning, comprising the following steps: obtaining workshop scheduling environment data and workshop scheduling order feature data, construct training dataset;Based on the state of job shop scheduling under transportation constraint, construct deep reinforcement learning model;The deep reinforcement learning model includes Markov process and scheduler;According to training dataset, utilize multi-agent near-end strategy optimization algorithm to train deep reinforcement learning model;The workshop scheduling task is input into the trained deep reinforcement learning model, and the scheduling result is output.The application can quickly and effectively find the optimal solution of workshop scheduling problem, improve the efficiency and quality of scheduling.The application can be widely applied to industrial scheduling technical field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial scheduling, and in particular to a job shop scheduling method under transportation constraints based on deep reinforcement learning. BACKGROUND

[0002] With the development of intelligent manufacturing and Industry 4.0, industrial enterprises are facing more and more challenges in production scheduling. Among them, transportation constraints, as an important factor affecting production efficiency and cost, put higher requirements on job shop scheduling.

[0003] Transportation constraints, as one of the key factors affecting job shop scheduling, involve material handling, storage and distribution, and directly affect the smoothness of production processes and job efficiency. However, existing research has not been deep enough in considering the impact of transportation constraints on job shop scheduling.

[0004] Traditional scheduling methods based on mixed integer linear programming (MILP) have high model complexity and are difficult to solve due to the introduction of a large number of constraints when dealing with large-scale production and constraint problems. In order to solve these problems, a scheduling method that can efficiently handle high-dimensional variables and complex constraints is needed.

[0005] Deep reinforcement learning (DRL) as a new intelligent optimization technology, with its advantage of not requiring models and prior knowledge, shows great potential in solving complex optimization problems. Although DRL has been successfully applied to job shop scheduling problems, how to effectively combine DRL for job shop scheduling under transportation constraints is still a problem worth further study. SUMMARY

[0006] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a job shop scheduling method under transportation constraints based on deep reinforcement learning, an electronic device and a medium.

[0007] The first technical solution adopted by the present application is:

[0008] A job shop scheduling method under transportation constraints based on deep reinforcement learning, comprising the following steps:

[0009] Obtain job shop scheduling environment data and job shop scheduling order feature data, and construct a training data set;

[0010] Based on the state of job shop scheduling under transportation constraints, a deep reinforcement learning model is constructed; the deep reinforcement learning model includes a Markov process and a scheduler;

[0011] According to the training data set, a deep reinforcement learning model is trained by using a multi-agent proximal policy optimization algorithm;

[0012] The job shop scheduling task is input into the trained deep reinforcement learning model, and a scheduling result is output.

[0013] Further, the job shop scheduling environment data includes the transportation time of workpieces between different machines in the job shop and the number of automatic guided vehicles (hereinafter referred to as AGVs) responsible for transporting the workpieces; and the job shop scheduling order feature data includes the number of workpieces in the order, the machining process of the workpieces in the order, the machines corresponding to the machining process of the workpieces in the order, and the machining time corresponding to the machining process of the workpieces in the order.

[0014] Further, the job shop scheduling state based on the transportation constraint comprises:

[0015] A job shop scheduling state representation method based on disjunctive graphs and heterogeneous graphs under transportation constraints is established, and is used as the environment state in the Markov process;

[0016] Based on the environment state in the Markov process, the estimated completion time of the scheduling task is calculated, and the difference between the estimated completion time of the scheduling task calculated this time and the estimated completion time of the scheduling task calculated last time is used as the immediate reward of the Markov process.

[0017] The actions of the Markov process include process scheduling operations and AGV scheduling operations; the scheduler includes a process scheduler for performing the process scheduling operations, an AGV scheduler for performing the AGV scheduling operations, and an evaluation network for calculating the state value function.

[0018] Further, the disjunctive graph includes a disjunctive graph structure and a disjunctive graph original feature vector; the expression of the disjunctive graph structure is G=(O, C∪D), wherein O represents a process node set, C represents a disjunctive arc set, and D represents a conjunctive arc set; and the disjunctive graph original feature vector includes a process node feature vector μ ij .

[0019] The heterogeneous graph includes a heterogeneous graph structure and a heterogeneous graph original feature vector; the expression of the heterogeneous graph structure is H=(O, A, C), wherein O represents a process node set, A represents an AGV node set, and C represents a disjunctive arc set; and the heterogeneous graph original feature vector includes a process node feature vector μ ij , an AGV node feature vector v k , and an O-A arc feature vector λ ijk .

[0020] Further, the calculation formula of the estimated completion time of the scheduling task is:

[0021] c(s t ) = max i，j (c1(s t O ij ))

[0022] In the formula, s t c(s) represents the scheduling state at step t. t c1(s) represents the estimated completion time. t O ij ) indicates process O ij The estimated completion time; max i，j This represents the maximum estimated completion time for all processes.

[0023] In state s t If the processing of workpiece i has reached step n (n < j), then the completion time of the preceding process of the workpiece can be accurately obtained, and the subsequent completion time of the workpiece is:

[0024]

[0025] In the formula, m ik Indicates process O ik Processing time, v ik Indicates process O ik The transportation time; of which, O in The meaning is the process node located at (i, j), which should be combined with the function c1(s) in the formula. t O ij As you can see, the meaning of this function has been explained earlier, so it will not be elaborated here. in ;

[0026] The formula for calculating the immediate reward of the Markov process is as follows:

[0027] r(s t )=c(s t )-c(s t-1 )

[0028] In the formula, r(s) t () indicates an instant reward.

[0029] Furthermore, the process scheduler includes a disjunctive graph neural network feature extraction module and a process selection module;

[0030] The disjunctive graph neural network feature extraction module is used to take the disjunctive graph structure and the original feature vector of the disjunctive graph as input, and to obtain the disjunctive graph node embedding vector through graph pooling and multilayer perceptron processing.

[0031] The process selection module is configured to extract a feature vector of a candidate process, and input the feature vector fused with the disjunction graph node embedding vector into a process Actor neural network to obtain a selection probability distribution of each candidate process.

[0032] The AGV scheduler comprises a heterogeneous graph neural network feature extraction module and an AGV selection module.

[0033] The heterogeneous graph neural network feature extraction module is configured to take the heterogeneous graph structure and the heterogeneous graph original feature vector as input, and obtain an AGV node embedding vector in the heterogeneous graph through graph attention network and multilayer perception processing calculation.

[0034] The AGV selection module is configured to input the AGV node embedding vector in the heterogeneous graph into an AGV Actor neural network to obtain a selection probability distribution of each candidate AGV.

[0035] The evaluation network is configured to input the disjunction graph node embedding vector and the AGV node embedding vector in the heterogeneous graph into a Critics network to obtain a state value function value v(s t ).

[0036] Further, the disjunction graph node embedding vector is obtained through graph pooling, multilayer perception processing calculation, comprising:

[0037] A disjunction graph adjacency matrix A is obtained according to the disjunction graph structure: c ;

[0038] The average pooling is used to calculate the pooled node feature vector:

[0039]

[0040] In the formula, I is a vector with all elements being 1.

[0041] The disjunction graph original feature vector is processed through multilayer graph pooling and multilayer perception processing calculation to obtain the disjunction graph node embedding vector:

[0042]

[0043] In the formula, MLP is multilayer perception, and θ1 is a parameter of the multilayer perception.

[0044] Further, the AGV node embedding vector in the heterogeneous graph is obtained through graph attention network and multilayer perception processing calculation, comprising:

[0045] Three learnable parameter matrices W1, W2, W3 and a parameter vector a are defined.

[0046] The fused feature is calculated:

[0047] h ijk = [W1v k ||W2μ ij ||W3λ ijk ]

[0048] In the formula, [||] is a matrix merging operation;

[0049] Calculate the attention score of the node in the heterogeneous graph:

[0050] α ijk = Softmax(LeakyReLu(a T h ijk ))

[0051] In the formula, LeakyReLU is an activation function, Softmax is to regard all attention scores as elements of a matrix, and do normalization operation; T is transposition;

[0052] According to the heterogeneous graph, obtain a heterogeneous graph adjacency matrix A h .

[0053] Combine the attention score with the heterogeneous graph adjacency matrix to perform shielding processing, and then fuse the node features to obtain the AGV node embedding vector:

[0054]

[0055] The AGV node embedding vector is obtained after multi-layer graph pooling and multi-layer perceptron processing and calculation in the heterogeneous graph:

[0056]

[0057] In the formula, MLP is a multi-layer perceptron, and θ2 is a parameter of the multi-layer perceptron.

[0058] Further, the expressions of the process Actor neural network and the AGV Actor neural network are:

[0059] Process Actor:

[0060] AGV Actor:

[0061] In the formula, π μ , π v are the action probability distribution of process selection and the action probability distribution of AGV selection respectively, and θ3, θ4 are parameters of the process Actor neural network and the AGV Actor neural network respectively.

[0062] Further, the loss function of the multi-agent proximal policy optimization algorithm includes entropy loss, value loss and policy loss.

[0063] The formula for calculating the entropy loss is:

[0064]

[0065] In the formula, β is the loss weight of the agent; π μ π v These are the probability distributions of actions selected for process steps and the probability distributions of actions selected by the AGV, respectively.

[0066] The value loss is the mean square error between the state function value and the immediate reward, and the calculation formula is as follows:

[0067]

[0068] The formula for calculating the strategy loss is as follows:

[0069]

[0070] In the formula, r t It is the policy sampling ratio, clip means to limit the policy ratio to a range, and ∈ is a coefficient used to control the range;

[0071] The Adam optimizer is used to train a deep reinforcement learning model by minimizing the loss function of the multi-agent proximal policy optimization algorithm.

[0072] The second technical solution adopted in this invention is:

[0073] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a deep reinforcement learning-based job shop scheduling method under transportation constraints as described above.

[0074] The third technical solution adopted in this invention is:

[0075] A computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a deep reinforcement learning-based job shop scheduling method under transportation constraints as described above.

[0076] The fourth technical solution adopted in this invention is:

[0077] A computer program product or computer program comprises computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned job shop scheduling method under transportation constraints based on deep reinforcement learning.

[0078] The beneficial effects of the present application include:

[0079] (1) The present application is used for job shop scheduling problem under transportation constraints, which converts this problem into a Markov process based on disjunctive graph and heterogeneous graph state description. This method allows to represent the complexity of the job shop in a structured way, while taking into account the impact of transportation constraints on scheduling. By constructing a multi-agent deep reinforcement learning network, we can simulate and learn the optimal strategy in this complex environment.

[0080] (2) By designing disjunctive graph and heterogeneous graph, the environment features are extracted for process agents and AGV agents respectively; through the graph attention mechanism, the complex correlation between transportation constraints and job shop scheduling is learned. This mechanism enables the agent to capture key information in the scheduling process and make more accurate decisions accordingly; through the training of multi-agent proximal policy optimization algorithm, the agent can learn how to cooperate to minimize the total completion time, thereby significantly improving the production efficiency of the workshop.

[0081] (3) The algorithm of the present application provides technical support for job shop scheduling problem under transportation constraints, and provides a new idea for the research of job shop scheduling problem. BRIEF DESCRIPTION OF DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0083] Figure 1 is a flow chart of the job shop scheduling method under transportation constraints based on deep reinforcement learning according to the embodiments of the present application;

[0084] Figure 2 is a disjunctive graph structure schematic diagram in the job shop scheduling method under transportation constraints based on deep reinforcement learning according to the embodiments of the present application;

[0085] Figure 3It is a heterogeneous graph structure schematic diagram in the job shop scheduling method under transportation constraints based on deep reinforcement learning according to an embodiment of the application;

[0086] Figure 4 It is an interaction process schematic diagram of an agent and an environment in the job shop scheduling method under transportation constraints based on deep reinforcement learning according to an embodiment of the application;

[0087] Figure 5 It is an AGV and machine Gantt chart of an embodiment;

[0088] Figure 6 It is a training optimization process schematic diagram of an embodiment. DETAILED DESCRIPTION

[0089] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0090] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0091] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of technical features indicated.

[0092] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0093] To solve the existing technical problems, the application provides a job shop scheduling method under transportation constraints based on deep reinforcement learning, which is particularly suitable for solving the problems of model complexity, great difficulty in solving, lack of flexibility, and dependence on prior knowledge in traditional scheduling methods. The method interacts with the simulated job shop scheduling environment through the construction of a multi-agent system, enabling the agent to continuously learn and optimize strategies, ultimately mastering an efficient optimal strategy to cope with job shop scheduling challenges under transportation constraints. This method can effectively and quickly find a better solution to the job shop scheduling problem, improving the efficiency and quality of scheduling.

[0094] Embodiment 1

[0095] As shown in Figure 1 The embodiment provides a job shop scheduling method under transportation constraints based on deep reinforcement learning, which includes the following steps:

[0096] S1, obtain job shop scheduling environment data and job shop scheduling order feature data, and construct a training data set.

[0097] In this embodiment, the job shop scheduling environment data includes the transportation time of workpieces between different machines in the workshop and the number of automatic guided vehicles (hereinafter referred to as AGVs) responsible for transporting workpieces; the job shop scheduling order feature data includes the number of workpieces in the order, the processing procedure of workpieces in the order, the machines corresponding to the processing procedure of workpieces in the order, and the processing time corresponding to the processing procedure of workpieces in the order.

[0098] S2, based on the job shop scheduling state under transportation constraints, construct a deep reinforcement learning model; the deep reinforcement learning model includes a Markov process and a scheduler.

[0099] In this embodiment, a Markov decision process based on graph state is first constructed to describe the job shop scheduling problem under transportation constraints. Then, a scheduler combining disjunctive graph neural network and heterogeneous graph neural network is designed. The scheduler provides an effective solution strategy for state feature extraction and scheduling action decision in the Markov decision process.

[0100] As an optional implementation, step S2 specifically includes the following steps:

[0101] S21, establish a job shop scheduling state representation method under transportation constraints based on disjunctive graph and heterogeneous graph, and use it as the environment state in the Markov process.

[0102] In some embodiments, referring to Figure 2, the disjunction graph includes a disjunction graph structure and a disjunction graph original feature vector; the expression of the disjunction graph structure is G=(O, C∪D), wherein O represents a process node set, C represents a disjunction arc set, and D represents a conjunction arc set; and the disjunction graph original feature vector includes a process node feature vector μ ij .

[0103] The heterogeneous graph includes a heterogeneous graph structure and a heterogeneous graph original feature vector; see Figure 3 , the expression of the heterogeneous graph structure is H=(O, A, C), wherein O represents a process node set, A represents an AGV node set, and C represents a disjunction arc set; and the heterogeneous graph original feature vector includes a process node feature vector μ ij , an AGV node feature vector v k , and an O-A arc feature vector λ ijk .

[0104] S22, based on an environment state in a Markov process, calculate an estimated completion time of a scheduling task, and take a difference between the estimated completion time of the scheduling task calculated this time and the estimated completion time of the scheduling task calculated last time as an instant reward of the Markov process.

[0105] In the description of the embodiment, the instant reward of the Markov process includes:

[0106] S221, calculate an estimated completion time of a scheduling task, and the calculation formula is:

[0107] c(s t )=max i，j (c1(s t , O ij ))

[0108] In the formula, s t represents a scheduling state at the tth step, c(s t ) represents the estimated completion time, and c1(s t , O ij ) represents the estimated completion time of a process O ij . In the state s t , if the machining of a workpiece i has proceeded to the nth (n

[0109]

[0110] That is, the fastest completion time of an unscheduled process, that is, the sum of the machining time and the transportation time is not considered.

[0111] S222, the instant reward of the Markov process, and the calculation formula is:

[0112] r(s t )=c(s t )-c(s t-1 )

[0113] wherein r(s t ) represents an immediate reward.

[0114] The actions of the Markov process include a process scheduling operation and an AGV scheduling operation; the scheduler includes a process scheduler for performing the process scheduling operation, an AGV scheduler for performing the AGV scheduling operation, and an evaluation network for calculating a state value function.

[0115] In some embodiments, referring to Figure 4 , the process scheduler includes an abductive graph neural network feature extraction module and a process selection module.

[0116] Specifically, the abductive graph neural network feature extraction module takes an abductive graph structure and an abductive graph original feature vector as input, and obtains an abductive graph node embedding vector through graph pooling, multilayer perceptron processing calculation.

[0117] Illustratively, the graph pooling, multilayer perceptron processing calculation includes:

[0118] A1, obtaining an abductive graph adjacency matrix A c from the abductive graph structure.

[0119] A2, calculating a pooled node feature vector using average pooling:

[0120]

[0121] wherein I is a vector with all elements being 1, and the vector division operation is a division of corresponding elements of the vectors.

[0122] A3, the multilayer perceptron processing calculation includes a linear layer, a normalization layer, and a ReLU nonlinear layer.

[0123] A4, the abductive graph original feature vector obtains an abductive graph node embedding vector after multilayer graph pooling and multilayer perceptron processing calculation.

[0124]

[0125] wherein MLP is a multilayer perceptron, and θ1 is a parameter of the multilayer perceptron.

[0126] Specifically, the process selection module is configured to extract a feature vector of a candidate process, and input the feature vector and the abductive graph node embedding vector into a process Actor neural network after fusion to obtain a selection probability distribution of each candidate process.

[0127] In some embodiments, referring to Figure 4 , the AGV scheduler comprises a heterogeneous graph neural network feature extraction module and an AGV selection module.

[0128] Specifically, the heterogeneous graph neural network feature extraction module is configured to take the heterogeneous graph structure and the heterogeneous graph original feature vector as input, and obtain an AGV node embedding vector in the heterogeneous graph through graph attention network and multilayer perception processing calculation.

[0129] Illustratively, the graph attention network and multilayer perception processing calculation comprises:

[0130] B1, define three learnable parameter matrices W1, W2, W3 and a parameter vector a;

[0131] B2, calculate the fusion feature:

[0132] h ijk = [W1v k ||W2μ ij ||W3λ ijk ]

[0133] In the formula, [||] is a matrix merging operation;

[0134] B3, calculate the attention score of the node in the heterogeneous graph:

[0135] α ijk = Softmax ijk (LeakyReLU(a T h ijk ))

[0136] In the formula, LeakyReLU is an activation function, Softmax is to regard all attention scores as elements of a matrix and perform normalization operation; T is transposition;

[0137] B4, obtain a heterogeneous graph adjacency matrix A h from the heterogeneous graph;

[0138] B5, combine the heterogeneous graph adjacency matrix to perform shielding processing on the attention score, and then fuse the node features to obtain the AGV node embedding vector:

[0139]

[0140] B6, the AGV node embedding vector is obtained after multilayer graph pooling and multilayer perception processing calculation:

[0141]

[0142] In the formula, MLP is a multilayer perceptron, and θ2 is a parameter of the multilayer perceptron. The multilayer perceptron processing calculation includes a linear layer, a normalization layer, and a ReLU nonlinear layer, and the multilayer perceptron structure is consistent with the multilayer perceptron in step A3, but the parameters are different.

[0143] Specifically, the AGV selection module inputs the AGV node embedding vector in the heterogeneous graph into the AGVActor neural network to obtain a selection probability distribution of each candidate AGV.

[0144] The expressions of the process Actor neural network and the AGVActor neural network are as follows:

[0145] Process Actor:

[0146] AGVActor:

[0147] In the formula, π μ and π v are an action probability distribution of process selection and an action probability distribution of AGV selection, respectively, and θ3 and θ4 are parameters of the process Actor neural network and the AGVActor neural network, respectively.

[0148] In some embodiments, the evaluation network inputs the disjunctive graph node embedding vector and the AGV node embedding vector in the heterogeneous graph into the Critics network to obtain a state value function value v(s t ).

[0149] S3, according to the training data set, a deep reinforcement learning model is trained by using a multi-agent proximal policy optimization algorithm.

[0150] In this embodiment, step S3 specifically includes the following steps:

[0151] S31, the loss function of the multi-agent proximal policy optimization algorithm includes an entropy loss, a value loss, and a policy loss.

[0152] S32, the entropy loss is calculated by π μ and π v .

[0153]

[0154] In the formula, β is an agent loss weight; π μ and π v are an action probability distribution of process selection and an action probability distribution of AGV selection, respectively.

[0155] S33, the value loss is a mean square error of a state function value and an immediate reward, and the calculation formula is as follows:

[0156]

[0157] S34, the calculation formula of policy loss is:

[0158]

[0159] In the formula, r t is the policy sampling ratio, clip represents limiting the policy ratio in a range, and ∈ is a coefficient for controlling the range.

[0160] S35, the depth reinforcement learning model is trained by minimizing the loss function using the Adam optimizer.

[0161] S4, input the workshop scheduling task into the trained depth reinforcement learning model, and output the scheduling result.

[0162] The above technical solution mainly describes the overall framework of the application, and the effect is verified by a specific embodiment. Here, we take a mechanical part processing workshop as an application scenario.

[0163] In this workshop, there are 6 processing equipment, each of which is responsible for different production processes. At the same time, there are 4 AGVs to be responsible for the transportation of workpieces. We take the production order of a certain day as the experimental task data, and the goal is to shorten the total processing time of all orders as much as possible.

[0164] In order to achieve this goal, we use the production order data of the past 6 months in the workshop data platform, which is divided into training set and test set. Among them, 5 months of data are used for model training, and the remaining 1 month of data is used for model verification. During the training process, every 100 times of training, we apply the current model to the test set, and record the maximum completion time predicted by the model. After 5000 times of training, we observe that the maximum completion time gradually tends to be stable, basically reaching the convergence state, as shown in the accompanying Figure 6 At this time, we think that the model has been trained maturely and can be used for further testing and application.

[0165] Use 8 order task data of a recent production cycle in the workshop for experiment.

[0166] The experimental steps are as follows:

[0167] Step 1, load the trained scheduling model

[0168] Step 2, convert the 8 order task data into disjunctive graph and heterogeneous graph as the initial state, and set the time step t=0;

[0169] Step 3, the process selection agent extracts the disjunction graph embedding vector from the disjunction graph;

[0170] Step 4, input the embedding vector into the scheduler of the process agent, and calculate the probability distribution of each possible action through the process Actor network;

[0171] Step 5, a greedy sampling strategy is adopted to select an operation action according to the probability distribution for execution;

[0172] Step 6, the AGV selection agent extracts the heterogeneous graph embedding vector from the heterogeneous graph;

[0173] Step 7, input the embedding vector into the scheduler of the AGV agent, and calculate the probability distribution of each possible transportation action through the AGV Actor network;

[0174] Step 8, a greedy sampling strategy is adopted to select an AGV action according to the probability distribution for execution;

[0175] Step 9, according to the selected process action and AGV action, perform state transition to obtain a new state, and update the time step t=t+1;

[0176] Step 10, repeat steps 3 to 9 until the time step t=48, at this time the whole scheduling process is completed, and the final scheduling scheme is output.

[0177] The scheduling scheme output by the model application of the present application is visualized as a Gantt chart, so that the execution order and time arrangement of each task can be observed intuitively. The horizontal axis of the Gantt chart represents the processing time, and the vertical axis represents different AGVs or machines. Each rectangle in the graph corresponds to a transportation or processing task. Figure 5 is the running result of the above verification case, it can be seen that the algorithm can reasonably arrange the operation order of AGV and machine, and ensure the continuity and efficiency of the production process.

[0178] From the above implementation case, it can be seen that the algorithm of the present application can quickly and efficiently train the model, and the trained model can quickly solve the job shop scheduling problem under transportation constraints. The model not only can quickly solve such job shop scheduling problems, but also can get good optimization effect. The trained model can cope with different scales of similar problems and has good generalization performance.

[0179] In summary, with the technical solutions of the present application, the job shop scheduling problem under transportation constraints is converted into a Markov process based on disjunctive graph and heterogeneous graph state description. This method allows to represent the complexity of the job shop in a structured way, while taking into account the impact of transportation constraints on scheduling. By constructing a multi-agent deep reinforcement learning network, we are able to simulate and learn the optimal strategy in this complex environment. By designing disjunctive graph and heterogeneous graph, the environment features are extracted for process agents and AGV agents respectively; through the graph attention mechanism, the complex correlation between transportation constraints and job shop scheduling is learned. This mechanism enables the agent to capture key information in the scheduling process and make more accurate decisions accordingly; through the training of multi-agent proximal policy optimization algorithm, the agent can learn how to collaborate to minimize the total completion time, thereby significantly improving the production efficiency of the workshop. The algorithm of the present application provides technical support for the job shop scheduling problem under transportation constraints, and provides a new idea for the research of job shop scheduling problem.

[0180] Embodiment 2

[0181] The embodiment of the present application also provides an electronic device, which comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the job shop scheduling method under transportation constraints based on deep reinforcement learning as shown in the method. Figure 1

[0182] It can be understood that the memory can include random access memory (RAM) and read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing each of the above-mentioned method embodiments, etc.; and the data storage area can store data created according to the use of the server, etc.

[0183] ​The processor can include one or more processing cores. The processor connects various parts within the entire server by various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with one or several combinations of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be realized by a single chip.

[0184] Since the electronic device is an electronic device corresponding to the job shop scheduling method under transportation constraints based on deep reinforcement learning of the embodiment of the application, and the principle of solving problems by the electronic device is similar to the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be described again.

[0185] Embodiment 3

[0186] The embodiment of the application further provides a computer readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set are stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to realize the job shop scheduling method under transportation constraints based on deep reinforcement learning as shown in Figure 1

[0187] ​Those skilled in the art can understand that all or part of the steps of various methods of the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0188] Since the storage medium is a storage medium corresponding to the job shop scheduling method under transportation constraints based on deep reinforcement learning of the embodiments of the present application, and the principle of solving problems of the storage medium is similar to the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described again.

[0189] Embodiment 4

[0190] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the method for job shop scheduling under transportation constraints based on deep reinforcement learning according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Wherein the executable computer program codes or "codes" for executing various embodiments can be written in high-level programming languages such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0191] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so on.

[0192] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0193] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A job-shop scheduling method under transportation constraints based on deep reinforcement learning, characterized in that, The method comprises the following steps: acquiring workshop scheduling environment data and workshop scheduling order feature data, and constructing a training data set; constructing a deep reinforcement learning model based on a job shop scheduling state under transportation constraints; the deep reinforcement learning model comprises a Markov process and a scheduler; training the deep reinforcement learning model by using a multi-agent proximal policy optimization algorithm according to the training data set; inputting a workshop scheduling task into the trained deep reinforcement learning model, and outputting a scheduling result; the scheduler comprises a process scheduler for performing process scheduling operations, an AGV scheduler for performing AGV scheduling operations, and an evaluation network for calculating a state value function; the process scheduler comprises an extraction graph neural network feature extraction module and a process selection module; the extraction graph neural network feature extraction module is configured to take the extraction graph structure and the extraction graph original feature vector as inputs, and obtain an extraction graph node embedding vector through graph pooling and multilayer perceptron processing calculation; the process selection module is configured to extract a feature vector of a candidate process, and input the feature vector and the extraction graph node embedding vector into a process Actor neural network after fusion to obtain a selection probability distribution of each candidate process; the AGV scheduler comprises a heterogeneous graph neural network feature extraction module and an AGV selection module; the heterogeneous graph neural network feature extraction module is configured to take the heterogeneous graph structure and the heterogeneous graph original feature vector as inputs, and obtain an AGV node embedding vector in the heterogeneous graph through graph attention network and multilayer perceptron processing calculation; and the AGV selection module is configured to input the AGV node embedding vector in the heterogeneous graph into an AGV Actor neural network to obtain a selection probability distribution of each candidate AGV; The evaluation network is used for inputting the AGV node embedding vector into a Critics network to obtain a state value function value v(s t ). the AGV node embedding vector in the heterogeneous graph is obtained through graph attention network and multilayer perceptron processing calculation, which comprises: defining three learnable parameter matrices W1, W2 and W3 and a parameter vector a; calculating the fusion feature: h ijk = [W1b k ||W2μ ij ||W3λ ijk ] where [||] is a matrix merging operation; calculating the attention score of the node in the heterogeneous graph: a ijk = Softmax(LeakyReLU(a T h ijk )) where LeakyReLU is an activation function, Softmax is a normalization operation on all attention scores as elements of a matrix, and T is a transpose; According to the heterogeneous graph, a heterogeneous graph adjacency matrix A is obtained h ; shielding the attention score in combination with the heterogeneous graph adjacency matrix, and then fusing the node features to obtain the AGV node embedding vector: the AGV node embedding vector is obtained through multilayer graph pooling and multilayer perceptron processing calculation: where MLP is a multilayer perceptron, and θ2 is a parameter of the multilayer perceptron.

2. The method of claim 1, wherein, The workshop scheduling environment data comprises transportation time of workpieces between different machines in the workshop and the number of automatic guided vehicles responsible for transporting the workpieces; and the workshop scheduling order feature data comprises the number of workpieces in an order, the machining processes of the workpieces in the order, the machines corresponding to the machining processes of the workpieces in the order, and the machining time corresponding to the machining processes of the workpieces in the order.

3. The method of claim 1, wherein, The deep reinforcement learning model is constructed based on a job shop scheduling state under transportation constraints, which comprises: A disjunction graph and a heterogeneous graph-based job-shop scheduling state representation method under transportation constraints is established, and is used as an environment state in the Markov process; Based on the environment state in the Markov process, an estimated completion time of a scheduling task is calculated, and a difference between the estimated completion time of the scheduling task calculated this time and the estimated completion time of the scheduling task calculated last time is used as an immediate reward of the Markov process. The action of the Markov process includes a process scheduling operation and an AGV scheduling operation.

4. The method of claim 3, wherein, The disjunction graph includes a disjunction graph structure and a disjunction graph original feature vector; an expression of the disjunction graph structure is G=(O, C∪D), wherein O represents a process node set, C represents a disjunction arc set, and D represents a conjunction arc set; and the disjunction graph original feature vector includes a process node feature vector μ ij ; The isomorphic graph includes an isomorphic graph structure and an isomorphic graph original feature vector; an expression of the isomorphic graph structure is H=(O,A,C), wherein O represents a process node set, A represents an AGV node set, and C represents a disjunctive arc set; and the isomorphic graph original feature vector includes a process node feature vector μ ij , an AGV node feature vector v k , and an O-A arc feature vector λ ijk .

5. The method of claim 3, wherein, The calculation formula of the estimated completion time of the scheduling task is: c(s t ) = max i,j (c1(s t ,O ij )) In the formula, s t c(s) represents the scheduling state at step t. t c1(s) represents the estimated completion time. t O ij ) indicates process O ij The estimated completion time; max i,j This represents the maximum estimated completion time for all processes. At state s t If the machining of workpiece i has proceeded to the nth step, the completion time of the preceding process of the workpiece can be accurately obtained, and the subsequent completion time of the workpiece is: wherein m ik represents the processing time of the process O ik represents the processing time of the process O ik represents the transport time of the process O ik represents the transport time of the process O The calculation formula of the immediate reward of the Markov process is: r(s t ) = c(s t ) - c(s t-1 ) where r(s t ) represents the immediate reward.

6. The method of claim 1, wherein, The disjunction graph node embedding vector is obtained through graph pooling and multi-layer perception processing and calculation, and includes: According to the disjunctive graph structure, a disjunctive graph adjacency matrix A is obtained c ; The node feature vector after pooling is calculated by using average pooling: In the formula, I is a vector with all elements being 1. The disjunction graph node embedding vector is obtained through multi-layer graph pooling and multi-layer perception processing and calculation on the original feature vector of the disjunction graph: In the formula, MLP is a multi-layer perception, and θ1 is a parameter of the multi-layer perception.

7. The method of claim 1, wherein, The expressions of the process Actor neural network and the AGV Actor neural network are: Process Actor: AGVActor: In the formula, π μ , π v are the action probability distribution of the process selection and the action probability distribution of the AGV selection, respectively, and θ3, θ4 are the parameters of the process Actor neural network and the AGV Actor neural network, respectively. 8.The method of claim 1, wherein, The loss function of the multi-agent proximal policy optimization algorithm includes an entropy loss, a value loss and a policy loss. The calculation formula of the entropy loss is: In the formula, β is the loss weight of the agent; π μ , π v are respectively the action probability distribution of the process selection and the action probability distribution of the AGV selection; The value loss is a mean square error of a state function value and an immediate reward, and the calculation formula is: The calculation formula of the policy loss is: In the formula, r t is a strategy sampling ratio, clip represents limiting the strategy ratio in a range, and ∈ is a coefficient for controlling the range. An Adam optimizer is used to train the deep reinforcement learning model by minimizing the loss function of the multi-agent proximal policy optimization algorithm.

Images