Job-shop scheduling method under transportation constraint based on deep reinforcement learning

Abstract

The invention discloses a job shop scheduling method under transportation constraint based on deep reinforcement learning, and the method comprises the following steps: obtaining workshop scheduling environment data and workshop scheduling order feature data, and constructing a training data set; constructing a deep reinforcement learning model based on the job shop scheduling state under the transportation constraint; the deep reinforcement learning model comprises a Markov process and a scheduler; according to the training data set, using a multi-agent near-end strategy optimization algorithm to train a deep reinforcement learning model; and inputting the workshop scheduling task into the trained deep reinforcement learning model, and outputting a scheduling result. According to the method, the optimal solution of the workshop scheduling problem can be quickly and effectively found, and the scheduling efficiency and quality are improved. The method can be widely applied to the technical field of industrial scheduling.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial scheduling, and in particular to a job shop scheduling method under transportation constraints based on deep reinforcement learning. Background Art

[0002] With the development of intelligent manufacturing and Industry 4.0, industrial enterprises are facing more and more challenges in production scheduling. Among them, transportation constraints, as an important factor affecting production efficiency and cost, put forward higher requirements for job shop scheduling.

[0003] Transportation constraints, as one of the key factors affecting job shop scheduling, involve material handling, storage and distribution, which directly affect the smoothness of the production process and the efficiency of the operation. However, existing research is not deep enough in considering the impact of transportation constraints on job shop scheduling.

[0004] Traditional scheduling methods based on mixed integer linear programming (MILP) introduce a large number of constraints when dealing with large-scale production and constraints, resulting in high model complexity, difficulty in solving, and high development and maintenance costs. In order to solve these problems, a scheduling method that can efficiently handle high-dimensional variables and complex constraints is needed.

[0005] As an emerging intelligent optimization technology, deep reinforcement learning (DRL) shows great potential in solving complex optimization problems with its advantages of no need for models and prior knowledge. Although DRL has been successfully applied to job shop scheduling problems, how to effectively combine DRL for job shop scheduling under the consideration of transportation constraints is still a problem worthy of in-depth study. Summary of the invention

[0006] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a job shop scheduling method, electronic device and medium under transportation constraints based on deep reinforcement learning.

[0007] The first technical solution adopted by the present invention is:

[0008] A method for job shop scheduling under transportation constraints based on deep reinforcement learning includes the following steps:

[0009] Obtain workshop scheduling environment data and workshop scheduling order feature data to build a training data set;

[0010] Based on the job shop scheduling state under transportation constraints, a deep reinforcement learning model is constructed; the deep reinforcement learning model includes a Markov process and a scheduler;

[0011] Based on the training data set, a deep reinforcement learning model is trained using a multi-agent proximal policy optimization algorithm;

[0012] Input the workshop scheduling task into the trained deep reinforcement learning model and output the scheduling result.

[0013] Furthermore, the workshop scheduling environment data includes the transportation time of workpieces between different machines in the workshop and the number of automatic guided vehicles (hereinafter referred to as AGVs) responsible for transporting the workpieces; the workshop scheduling order feature data includes the number of workpieces in the order, the processing procedures of the workpieces in the order, the machines corresponding to the processing procedures of the workpieces in the order, and the processing time corresponding to the processing procedures of the workpieces in the order.

[0014] Furthermore, the deep reinforcement learning model is constructed based on the job shop scheduling state under transportation constraints, including:

[0015] Establish a method for representing the state of workshop scheduling under transportation constraints based on disjunctive graphs and heterogeneous graphs, and use it as the environmental state in the Markov process;

[0016] Based on the environmental state in the Markov process, the estimated completion time of the scheduled task is calculated, and the difference between the estimated completion time of the scheduled task calculated this time and the estimated completion time of the scheduled task calculated last time is used as the immediate reward of the Markov process;

[0017] The actions of the Markov process include process scheduling operations and AGV scheduling operations; the scheduler includes a process scheduler for executing the process scheduling operations, an AGV scheduler for executing the AGV scheduling operations, and an evaluation network for calculating a state value function.

[0018] Furthermore, the disjunctive graph includes a disjunctive graph structure and an original feature vector of the disjunctive graph; the expression of the disjunctive graph structure is G=(O, C∪D), where O represents a process node set, C represents a disjunctive arc set, and D represents a conjunctive arc set; the original feature vector of the disjunctive graph includes a process node feature vector μ ij ;

[0019] The heterogeneous graph includes a heterogeneous graph structure and an original feature vector of the heterogeneous graph; the expression of the heterogeneous graph structure is H=(O, A, C), where O represents a process node set, A represents an AGV node set, and C represents a disjunctive arc set; the original feature vector of the heterogeneous graph includes a process node feature vector μ ij , AGV node feature vector v k and the OA arc eigenvector λ ijk .

[0020] Furthermore, the calculation formula for the estimated completion time of the scheduled task is:

[0021] c(s t )=max i，j (c1(s t , O ij ))

[0022] In the formula, s t represents the scheduling state of step t, c(s t ) represents the estimated completion time, c1(s t , O ij ) represents process O ij Estimated completion time; max i，j It indicates the maximum value of the estimated completion time of all processes;

[0023] In status t Under this condition, if the processing of workpiece i has reached the nth (n<j) step, then the completion time of the previous process of the workpiece can be accurately obtained, and the subsequent completion time of the workpiece is:

[0024]

[0025] In the formula, m ik Indicates process O ik Processing time, v ik Indicates process O ik The transportation time of O in The meaning is the process node located at (i, j), which should be combined with the function c1(s t , O ij ) and the meaning of this function has been explained in the previous text, so it will not be explained here. in ;

[0026] The calculation formula of the instantaneous reward of the Markov process is:

[0027] r(s t )=c(s t )-c(s t-1 )

[0028] In the formula, r(s t ) indicates an immediate reward.

[0029] Further, the process scheduler includes a disjunctive graph neural network feature extraction module and a process selection module;

[0030] The disjunctive graph neural network feature extraction module is used to take the disjunctive graph structure and the original feature vector of the disjunctive graph as input, and obtain the disjunctive graph node embedding vector through graph pooling and multi-layer perceptron processing and calculation;

[0031] The process selection module is used to extract the feature vector of the candidate process, and fuse it with the embedding vector of the disjunctive graph node and input it into the process Actor neural network to obtain the selection probability distribution of each candidate process;

[0032] The AGV scheduler includes a heterogeneous graph neural network feature extraction module and an AGV selection module;

[0033] The heterogeneous graph neural network feature extraction module is used to take the heterogeneous graph structure and the original feature vector of the heterogeneous graph as input, and obtain the AGV node embedding vector in the heterogeneous graph through graph attention network and multi-layer perceptron processing and calculation;

[0034] The AGV selection module is used to input the AGV node embedding vector in the heterogeneous graph into the AGVActor neural network to obtain the selection probability distribution of each candidate AGV;

[0035] The evaluation network is used to input the node embedding vector of the disjunctive graph and the AGV node embedding vector of the heterogeneous graph into the critic network to obtain the state value function value v(s t ).

[0036] Furthermore, the extraction of graph node embedding vectors by graph pooling and multi-layer perceptron processing and calculation includes:

[0037] According to the disjunctive graph structure, the disjunctive graph adjacency matrix A is obtained. c ;

[0038] Average pooling is used to calculate the pooled node feature vector:

[0039]

[0040] Where I is a vector whose elements are all 1;

[0041] After the original feature vector of the disjunctive graph is processed and calculated by multi-layer graph pooling and multi-layer perceptron, the node embedding vector of the disjunctive graph is obtained:

[0042]

[0043] Where MLP is a multi-layer perceptron and θ1 is the parameter of the multi-layer perceptron.

[0044] Furthermore, the AGV node embedding vector in the heterogeneous graph is obtained by processing and calculating through the graph attention network and the multi-layer perceptron, including:

[0045] Define three learnable parameter matrices W1, W2, W3 and parameter vector a;

[0046] Calculate fusion features:

[0047] h ijk =[W1v k ||W2μ ij ||W3λ ijk ]

[0048] Where [||] is the matrix merging operation;

[0049] Calculate the attention score of a node in a heterogeneous graph:

[0050] α ijk =Softmax(LeakyReLu(a T h ijk ))

[0051] In the formula, LeakyReLU is the activation function, Softmax treats all attention scores as elements of the matrix and performs normalization operations; T is the transpose;

[0052] According to the heterogeneous graph, the heterogeneous graph adjacency matrix A is obtained h .

[0053] The attention scores are masked in combination with the heterogeneous graph adjacency matrix, and then the node features are fused to obtain the AGV node embedding vector:

[0054]

[0055] The AGV node embedding vector is obtained after multi-layer graph pooling and multi-layer perceptron processing and calculation to obtain the AGV node embedding vector in the heterogeneous graph:

[0056]

[0057] Where MLP is a multi-layer perceptron and θ2 is the parameter of the multi-layer perceptron.

[0058] Furthermore, the expressions of the process Actor neural network and AGVActor neural network are:

[0059] Process Actor:

[0060] AGVActor:

[0061] In the formula, π μ , π v are the action probability distributions of process selection and AGV selection, θ3 and θ4 are the parameters of process Actor neural network and AGV Actor neural network, respectively.

[0062] Furthermore, the loss function of the multi-agent proximal strategy optimization algorithm includes entropy loss, value loss, and strategy loss;

[0063] The calculation formula of the entropy loss is:

[0064]

[0065] Where β is the agent loss weight; π μ , π v They are the probability distribution of actions selected by the process and the probability distribution of actions selected by the AGV respectively;

[0066] The value loss is the mean square error between the state function value and the immediate reward, and the calculation formula is:

[0067]

[0068] The calculation formula of the strategy loss is:

[0069]

[0070] In the formula, r t is the strategy sampling ratio, clip means limiting the strategy ratio to a range, ∈ is the coefficient used to control the range;

[0071] The Adam optimizer is used to train the deep reinforcement learning model by minimizing the loss function of the multi-agent proximal policy optimization algorithm.

[0072] The second technical solution adopted by the present invention is:

[0073] An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a job shop scheduling method under transportation constraints based on deep reinforcement learning as described above.

[0074] The third technical solution adopted by the present invention is:

[0075] A computer-readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a job shop scheduling method under transportation constraints based on deep reinforcement learning as described above.

[0076] The fourth technical solution adopted by the present invention is:

[0077] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method for job shop scheduling under transportation constraints based on deep reinforcement learning.

[0078] The beneficial effects of the present invention include:

[0079] (1) This paper is applied to the job shop scheduling problem under transportation constraints, transforming this problem into a Markov process based on disjunctive graph and heterogeneous graph state description. This approach allows the complexity of the job shop to be represented in a structured way while taking into account the impact of transportation constraints on scheduling. By building a multi-agent deep reinforcement learning network, we are able to simulate and learn the optimal strategy in this complex environment.

[0080] (2) By designing disjunctive graphs and heterogeneous graphs, environmental features are extracted for process agents and AGV agents respectively; the complex relationship between transportation constraints and job shop scheduling is learned through the graph attention mechanism. This mechanism enables the agent to capture key information in the scheduling process and make more accurate decisions based on it; through the training of the multi-agent proximal policy optimization algorithm, the agents can learn how to collaborate to minimize the total completion time, thereby significantly improving the production efficiency of the workshop.

[0081] (3) The algorithm of the present invention provides technical support for the job shop scheduling problem under transportation constraints and provides new ideas for the study of job shop scheduling problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0083] Figure 1 is a flow chart of a job shop scheduling method under transportation constraints based on deep reinforcement learning according to an embodiment of the present invention;

[0084] Figure 2 is a schematic diagram of a disjunctive graph structure in a job shop scheduling method under transportation constraints based on deep reinforcement learning according to an embodiment of the present invention;

[0085] Figure 3is a schematic diagram of a heterogeneous graph structure in a job shop scheduling method under transportation constraints based on deep reinforcement learning according to an embodiment of the present invention;

[0086] Figure 4 is a schematic diagram of the interaction process between an agent and an environment in a job shop scheduling method under transportation constraints based on deep reinforcement learning according to an embodiment of the present invention;

[0087] Figure 5 It is the Gantt chart of AGV and machine of the embodiment;

[0088] Figure 6 Schematic diagram of the training optimization process of the embodiment. DETAILED DESCRIPTION

[0089] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0090] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0091] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0092] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0093] In view of the existing technical problems, the present invention proposes a method for job shop scheduling under transportation constraints based on deep reinforcement learning. This method is particularly suitable for solving the problems of model complexity, difficulty in solving, lack of flexibility, and reliance on prior knowledge in traditional scheduling methods. This method interacts with the simulated shop scheduling environment by constructing a multi-agent system, so that the agent can continuously learn and optimize strategies, and finally master a set of efficient optimal strategies to cope with the challenges of shop scheduling under transportation constraints. This method can effectively and quickly find a better solution to the shop scheduling problem, improving the efficiency and quality of scheduling.

[0094] Example 1

[0095] like Figure 1 As shown, this embodiment provides a job shop scheduling method under transportation constraints based on deep reinforcement learning, and the scheduling method includes:

[0096] S1. Obtain the workshop scheduling environment data and the workshop scheduling order feature data to build a training data set.

[0097] In this embodiment, the workshop scheduling environment data includes the transportation time of workpieces between different machines in the workshop and the number of automatic guided vehicles (hereinafter referred to as AGVs) responsible for transporting the workpieces; the workshop scheduling order feature data includes the number of workpieces in the order, the processing procedures of the workpieces in the order, the machines corresponding to the processing procedures of the workpieces in the order, and the processing time corresponding to the processing procedures of the workpieces in the order.

[0098] S2. Based on the job shop scheduling status under transportation constraints, a deep reinforcement learning model is constructed; the deep reinforcement learning model includes a Markov process and a scheduler.

[0099] Among them, this embodiment first constructs a graph-state-based Markov decision process to describe the job shop scheduling problem under transportation constraints. Then, a scheduler combining disjunctive graph neural network and heterogeneous graph neural network is designed. The scheduler provides an effective solution strategy for state feature extraction and scheduling action decision in the Markov decision process.

[0100] As an optional implementation, step S2 specifically includes the following steps:

[0101] S21. Establish a workshop scheduling state representation method under transportation constraints based on disjunctive graphs and heterogeneous graphs, and use it as the environmental state in the Markov process.

[0102] In some embodiments, see Figure 2, the disjunctive graph includes a disjunctive graph structure and an original feature vector of the disjunctive graph; the expression of the disjunctive graph structure is G=(O, C∪D), where O represents a process node set, C represents a disjunctive arc set, and D represents a conjunctive arc set; the original feature vector of the disjunctive graph includes a process node feature vector μ ij .

[0103] The heterogeneous graph includes a heterogeneous graph structure and an original feature vector of the heterogeneous graph; see Figure 3 , the heterogeneous graph structure is expressed as H = (O, A, C), where O represents the process node set, A represents the AGV node set, and C represents the disjunctive arc set; the heterogeneous graph original feature vector includes the process node feature vector μ ij , AGV node feature vector v k and the OA arc eigenvector λ ijk .

[0104] S22. Based on the environmental state in the Markov process, the estimated completion time of the scheduling task is calculated, and the difference between the estimated completion time of the scheduling task calculated this time and the estimated completion time of the scheduling task calculated last time is used as the immediate reward of the Markov process.

[0105] In the description of this embodiment, the immediate rewards of the Markov process include:

[0106] S221. Calculate the estimated completion time of the scheduled task, using the following formula:

[0107] c(s t )=max i，j (c1(s t , O ij ))

[0108] In the formula, s t represents the scheduling state of step t, c(s t ) represents the estimated completion time, c1(s t , O ij ) represents process O ij The estimated completion time of t Under this condition, if the processing of workpiece i has reached the nth (n<j) step, then the completion time of the previous process of the workpiece can be accurately obtained, and the subsequent completion time of the workpiece is:

[0109]

[0110] That is, the fastest completion time of the unscheduled process, that is, the sum of the processing time and transportation time without considering the scheduling.

[0111] S222. The instant reward of the Markov process is calculated as follows:

[0112] r(s t )=c(s t )-c(s t-1 )

[0113] In the formula, r(s t ) indicates an immediate reward.

[0114] The actions of the Markov process include process scheduling operations and AGV scheduling operations; the scheduler includes a process scheduler for executing the process scheduling operations, an AGV scheduler for executing the AGV scheduling operations, and an evaluation network for calculating a state value function.

[0115] In some embodiments, see Figure 4 The process scheduler includes a disjunctive graph neural network feature extraction module and a process selection module.

[0116] Specifically, the disjunctive graph neural network feature extraction module takes the disjunctive graph structure and the original feature vector of the disjunctive graph as input, and obtains the disjunctive graph node embedding vector through graph pooling and multi-layer perceptron processing and calculation.

[0117] Exemplarily, the graph pooling and multi-layer perceptron processing calculations include:

[0118] A1. Obtain the disjunctive graph adjacency matrix A based on the disjunctive graph structure c .

[0119] A2. Use average pooling to calculate the node feature vector after pooling:

[0120]

[0121] Where I is a vector whose elements are all 1, and the vector division operation is the division of the corresponding elements of the vector.

[0122] A3. The multi-layer perceptron processing calculation includes linear layer, normalization layer, and ReLU nonlinear layer.

[0123] A4. The original feature vector of the disjunctive graph is processed and calculated by multi-layer graph pooling and multi-layer perceptron to obtain the disjunctive graph node embedding vector.

[0124]

[0125] Where MLP is a multi-layer perceptron and θ1 is the parameter of the multi-layer perceptron.

[0126] Specifically, the process selection module is used to extract the feature vector of the candidate process, and after fusing it with the disjunctive graph node embedding vector, input it into the process Actor neural network to obtain the selection probability distribution of each candidate process.

[0127] In some embodiments, see Figure 4 , the AGV scheduler includes a heterogeneous graph neural network feature extraction module and an AGV selection module.

[0128] Specifically, the heterogeneous graph neural network feature extraction module is used to take the heterogeneous graph structure and the original feature vector of the heterogeneous graph as input, and obtain the AGV node embedding vector in the heterogeneous graph through graph attention network and multi-layer perceptron processing and calculation.

[0129] Exemplarily, the graph attention network and multi-layer perceptron processing calculation include:

[0130] B1. Define three learnable parameter matrices W1, W2, W3 and parameter vector a;

[0131] B2. Calculate fusion features:

[0132] h ijk =[W1v k ||W2μ ij ||W3λ ijk ]

[0133] Where [||] is the matrix merging operation;

[0134] B3. Calculate the attention scores of nodes in heterogeneous graphs:

[0135] α ijk =Softmax ijk (LeakyReLU(a T h ijk ))

[0136] In the formula, LeakyReLU is the activation function, Softmax treats all attention scores as elements of the matrix and performs normalization operations; T is the transpose;

[0137] B4. Obtain the heterogeneous graph adjacency matrix A based on the heterogeneous graph h ;

[0138] B5. Combine the heterogeneous graph adjacency matrix to mask the attention score, and then fuse the node features to obtain the AGV node embedding vector:

[0139]

[0140] B6. The AGV node embedding vector is processed and calculated by multi-layer graph pooling and multi-layer perceptron to obtain the AGV node embedding vector in the heterogeneous graph:

[0141]

[0142] In the formula, MLP is a multi-layer perceptron, and θ2 is a parameter of the multi-layer perceptron. The multi-layer perceptron processing calculation includes a linear layer, a normalization layer, and a ReLU nonlinear layer. The structure of the multi-layer perceptron is consistent with the multi-layer perceptron in step A3 above, but the parameters are different.

[0143] Specifically, the AGV selection module inputs the embedding vector of the AGV node in the heterogeneous graph into the AGVActor neural network to obtain the selection probability distribution of each candidate AGV.

[0144] The expressions of the process Actor neural network and AGVActor neural network are:

[0145] Process Actor:

[0146] AGVActor:

[0147] In the formula, π μ , π v are the action probability distributions of process selection and AGV selection, θ3 and θ4 are the parameters of process Actor neural network and AGV Actor neural network, respectively.

[0148] In some embodiments, the evaluation network takes the disjunctive graph node embedding vector and the AGV node embedding vector in the heterogeneous graph as input to the Critics network to obtain the state value function value v(s t ).

[0149] S3. Based on the training data set, the deep reinforcement learning model is trained using the multi-agent proximal strategy optimization algorithm.

[0150] In this embodiment, step S3 specifically includes the following steps:

[0151] S31. The loss functions of the multi-agent proximal strategy optimization algorithm include entropy loss, value loss, and strategy loss.

[0152] S32, entropy loss is π μ , π v The calculation results are:

[0153]

[0154] Where β is the agent loss weight; π μ , π v They are the action probability distribution of process selection and the action probability distribution of AGV selection respectively.

[0155] S33, value loss is the mean square error between the state function value and the immediate reward, and the calculation formula is:

[0156]

[0157] S34. The calculation formula of strategy loss is:

[0158]

[0159] In the formula, r t is the strategy sampling ratio, clip means limiting the strategy ratio to a range, and ∈ is the coefficient used to control the range.

[0160] S35. Use the Adam optimizer to train the deep reinforcement learning model by minimizing the loss function.

[0161] S4. Input the workshop scheduling task into the trained deep reinforcement learning model and output the scheduling result.

[0162] The above technical solution mainly describes the overall framework of the present invention, and its effect is verified by a specific embodiment below. Here, we take a mechanical parts processing workshop as the application scenario.

[0163] In this workshop, there are 6 processing equipment, each of which is responsible for different production processes. At the same time, there are 4 AGVs to transport the workpieces. We use the production orders of a certain day as the experimental task data, and the goal is to shorten the total processing time of all orders as much as possible.

[0164] To achieve this goal, we used the production order data of the past 6 months in the workshop data platform and divided it into a training set and a test set. Among them, 5 months of data were used for model training, and the remaining 1 month of data was used for model verification. During the training process, every 100 training times, we applied the current model to the test set and recorded the maximum completion time predicted by the model. After 5,000 training times, we observed that the maximum completion time gradually stabilized and basically reached a convergence state, as shown in the attached figure. Figure 6 At this point, we believe that the model has been trained and can be used for further testing and application.

[0165] The experiment was conducted using 8 order task data from a recent production cycle of the workshop.

[0166] The experimental steps are as follows:

[0167] Step 1: Load the trained scheduling model

[0168] Step 2: Convert the 8 order task data into a disjunctive graph and a heterogeneous graph, use this as the initial state, and set the time step t = 0;

[0169] Step 3: The process selection agent extracts the disjunctive graph embedding vector from the disjunctive graph;

[0170] Step 4: Input the embedding vector into the scheduler of the process agent, and calculate the probability distribution of each possible action through the process actor network;

[0171] Step 5: Use the greedy sampling strategy to select a process action for execution based on the probability distribution;

[0172] Step 6: AGV selects an agent to extract a heterogeneous graph embedding vector from the heterogeneous graph;

[0173] Step 7: Input the embedding vector into the scheduler of the AGV agent and calculate the probability distribution of each possible transportation action through the AGV Actor network;

[0174] Step 8: Use the greedy sampling strategy to select an AGV action to execute based on the probability distribution;

[0175] Step 9: According to the selected process action and AGV action, execute state transition, obtain a new state, and update the time step t=t+1;

[0176] Step 10: Repeat steps 3 to 9 until time step t=48, at which time the entire scheduling process is completed and the final scheduling plan is output.

[0177] The present invention visualizes the scheduling scheme output after the model is applied in the form of a Gantt chart, so that the execution order and time arrangement of each task can be intuitively observed. The horizontal axis of the Gantt chart represents the processing time, and the vertical axis represents different AGVs or machines. Each rectangle in the figure corresponds to a transportation or processing task. Figure 5 This is the running result of the above verification case. It can be seen that the algorithm can reasonably arrange the operation sequence of AGV and machines to ensure the continuity and efficiency of the production process.

[0178] Through the above implementation cases, it can be seen that the algorithm of the present invention can train the model quickly and efficiently, and the trained model can quickly solve the job shop scheduling problem under transportation constraints. The model can not only quickly solve such shop scheduling problems, but also obtain good optimization effects. The trained model can cope with similar problems of different scales and has good generalization performance.

[0179] In summary, with the help of the technical solution of the present invention, the present invention is used for the job shop scheduling problem under transportation constraints, and this problem is converted into a Markov process based on the state description of the disjunctive graph and the heterogeneous graph. This method allows the complexity of the job shop to be represented in a structured way, while taking into account the impact of transportation constraints on scheduling. By constructing a multi-agent deep reinforcement learning network, we can simulate and learn the optimal strategy in this complex environment. By designing disjunctive graphs and heterogeneous graphs, environmental features are extracted for process agents and AGV agents respectively; the complex relationship between transportation constraints and job shop scheduling is learned through the graph attention mechanism. This mechanism enables the agent to capture key information in the scheduling process and make more accurate decisions based on it; through the training of the multi-agent proximal strategy optimization algorithm, the agent can learn how to collaborate to minimize the total completion time, thereby significantly improving the production efficiency of the workshop. The algorithm of the present invention provides technical support for the job shop scheduling problem under transportation constraints and provides new ideas for the study of workshop scheduling problems.

[0180] Example 2

[0181] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 1 A job shop scheduling method under transportation constraints based on deep reinforcement learning is shown.

[0182] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0183] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0184] Since the electronic device is an electronic device corresponding to a job shop scheduling method under transportation constraints based on deep reinforcement learning in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0185] Example 3

[0186] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 A job shop scheduling method under transportation constraints based on deep reinforcement learning is shown.

[0187] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0188] Since the storage medium is a storage medium corresponding to a job shop scheduling method under transportation constraints based on deep reinforcement learning in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0189] Example 4

[0190] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a method for scheduling a job shop under transportation constraints based on deep reinforcement learning according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0191] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0192] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0193] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A job shop scheduling method under transportation constraints based on deep reinforcement learning, characterized in that: The following steps are involved: Obtain workshop scheduling environment data and workshop scheduling order feature data to build a training data set; Build a deep reinforcement learning model based on the job shop scheduling status under transportation constraints; The deep reinforcement learning model includes a Markov process and a scheduler; Based on the training data set, a deep reinforcement learning model is trained using a multi-agent proximal policy optimization algorithm; Input the workshop scheduling task into the trained deep reinforcement learning model and output the scheduling result.

2. According to claim 1, a method for job shop scheduling under transportation constraints based on deep reinforcement learning is characterized in that: The workshop scheduling environment data includes the transportation time of workpieces between different machines in the workshop and the number of automatic guided vehicles responsible for transporting the workpieces; the workshop scheduling order feature data includes the number of workpieces in the order, the processing procedures of the workpieces in the order, the machines corresponding to the processing procedures of the workpieces in the order, and the processing time corresponding to the processing procedures of the workpieces in the order.

3. The method for job shop scheduling under transportation constraints based on deep reinforcement learning according to claim 1 is characterized in that: The deep reinforcement learning model is constructed based on the job shop scheduling state under transportation constraints, including: Establish a method for representing the state of workshop scheduling under transportation constraints based on disjunctive graphs and heterogeneous graphs, and use it as the environmental state in the Markov process; Based on the environmental state in the Markov process, the estimated completion time of the scheduled task is calculated, and the difference between the estimated completion time of the scheduled task calculated this time and the estimated completion time of the scheduled task calculated last time is used as the immediate reward of the Markov process; The actions of the Markov process include process scheduling operations and AGV scheduling operations; the scheduler includes a process scheduler for executing the process scheduling operations, an AGV scheduler for executing the AGV scheduling operations, and an evaluation network for calculating a state value function.

4. The method for job shop scheduling under transportation constraints based on deep reinforcement learning according to claim 3 is characterized in that: The disjunctive graph includes a disjunctive graph structure and an original feature vector of the disjunctive graph; the expression of the disjunctive graph structure is G=(O, C∪D), where O represents a process node set, C represents a disjunctive arc set, and D represents a conjunctive arc set; the original feature vector of the disjunctive graph includes a process node feature vector μ ij ; The heterogeneous graph includes a heterogeneous graph structure and an original feature vector of the heterogeneous graph; the expression of the heterogeneous graph structure is H=(O, A, C), where O represents a process node set, A represents an AGV node set, and C represents a disjunctive arc set; the original feature vector of the heterogeneous graph includes a process node feature vector μ ij , AGV node feature vector v k and the OA arc eigenvector λ ijk .

5. The method for job shop scheduling under transportation constraints based on deep reinforcement learning according to claim 3 is characterized in that: The calculation formula for the estimated completion time of the scheduled task is: c(s t )=max i，j (c1(s t ,O ij )) In the formula, s t represents the scheduling state of step t, c(s t ) represents the estimated completion time, c1(s t , O ij ) represents process O ij Estimated completion time; max i，j It indicates the maximum value of the estimated completion time of all processes; In status t Under this condition, if the processing of workpiece i has reached step n, then the completion time of the previous process of the workpiece can be accurately obtained, and the subsequent completion time of the workpiece is: In the formula, m ik Indicates process O ik Processing time, v ik Indicates process O ik transportation time; The calculation formula of the instantaneous reward of the Markov process is: r(s t )=c(s t )-c(s t-1 ) In the formula, r(s t ) indicates an immediate reward.

6. The method for job shop scheduling under transportation constraints based on deep reinforcement learning according to claim 4 is characterized in that: The process scheduler includes a disjunctive graph neural network feature extraction module and a process selection module; The disjunctive graph neural network feature extraction module is used to take the disjunctive graph structure and the original feature vector of the disjunctive graph as input, and obtain the disjunctive graph node embedding vector through graph pooling and multi-layer perceptron processing and calculation; The process selection module is used to extract the feature vector of the candidate process, and fuse it with the embedding vector of the disjunctive graph node and input it into the process Actor neural network to obtain the selection probability distribution of each candidate process; The AGV scheduler includes a heterogeneous graph neural network feature extraction module and an AGV selection module; The heterogeneous graph neural network feature extraction module is used to take the heterogeneous graph structure and the original feature vector of the heterogeneous graph as input, and obtain the AGV node embedding vector in the heterogeneous graph through graph attention network and multi-layer perceptron processing and calculation; the AGV selection module is used to input the AGV node embedding vector in the heterogeneous graph into the AGVActor neural network to obtain the selection probability distribution of each candidate AGV; The evaluation network is used to input the node embedding vector of the disjunctive graph and the AGV node embedding vector of the heterogeneous graph into the critic network to obtain the state value function value v(s t ).

7. The method for job shop scheduling under transportation constraints based on deep reinforcement learning according to claim 6 is characterized in that: The above-mentioned process of obtaining the disjunctive graph node embedding vector by graph pooling and multi-layer perceptron processing and calculation includes: According to the disjunctive graph structure, the disjunctive graph adjacency matrix A is obtained. c ; Average pooling is used to calculate the pooled node feature vector: Where I is a vector whose elements are all 1; After the original feature vector of the disjunctive graph is processed and calculated by multi-layer graph pooling and multi-layer perceptron, the node embedding vector of the disjunctive graph is obtained: Where MLP is a multi-layer perceptron and θ1 is the parameter of the multi-layer perceptron.

8. The method for job shop scheduling under transportation constraints based on deep reinforcement learning according to claim 6 is characterized in that: The AGV node embedding vector in the heterogeneous graph is obtained by the graph attention network and multi-layer perceptron processing calculation, including: Define three learnable parameter matrices W1, W2, W3 and parameter vector a; Calculate fusion features: h ijk =[W1v k ||W2μ ij ||W3λ ijk ] Where [||] is the matrix merging operation; Calculate the attention score of a node in a heterogeneous graph: a ijk =Softmax(LeakyReLU(a T h ijk )) In the formula, LeakyReLU is the activation function, Softmax treats all attention scores as elements of the matrix and performs normalization operations; T is the transpose; According to the heterogeneous graph, the heterogeneous graph adjacency matrix A is obtained h ; The attention scores are masked in combination with the heterogeneous graph adjacency matrix, and then the node features are fused to obtain the AGV node embedding vector: The AGV node embedding vector is obtained after multi-layer graph pooling and multi-layer perceptron processing and calculation to obtain the AGV node embedding vector in the heterogeneous graph: Where MLP is a multi-layer perceptron and θ2 is the parameter of the multi-layer perceptron.

9. The method for job shop scheduling under transportation constraints based on deep reinforcement learning according to claim 6 is characterized in that: The expressions of the process Actor neural network and AGVActor neural network are: Process Actor: AGVActor: In the formula, π μ , π v are the action probability distributions of process selection and AGV selection, θ3 and θ4 are the parameters of process Actor neural network and AGV Actor neural network, respectively.

10. The method for job shop scheduling under transportation constraints based on deep reinforcement learning according to claim 1, characterized in that: The loss function of the multi-agent proximal strategy optimization algorithm includes entropy loss, value loss, and strategy loss; The calculation formula of the entropy loss is: Where β is the agent loss weight; π μ , π v They are the probability distribution of actions selected by the process and the probability distribution of actions selected by the AGV respectively; The value loss is the mean square error between the state function value and the immediate reward, and the calculation formula is: The calculation formula of the strategy loss is: In the formula, r t is the strategy sampling ratio, clip means limiting the strategy ratio to a range, ∈ is the coefficient used to control the range; The Adam optimizer is used to train the deep reinforcement learning model by minimizing the loss function of the multi-agent proximal policy optimization algorithm.

Images