A robust scheduling optimization method for discrete manufacturing workshops based on deep reinforcement learning

Abstract

The application discloses a kind of robust scheduling optimization methods of discrete manufacturing workshop based on deep reinforcement learning, comprising: taking historical processing data, using neural network to fit the function of process processing duration about equipment, operator and production initial time;Establish the processing environment model of factory workshop, and the processing environment model includes the number of available equipment, on-duty operator and the inventory number of intermediate product;Obtain the product quantity that needs to be processed today;According to the processing environment model and the product quantity that needs to be processed today, construct the robust scheduling problem of discrete manufacturing workshop, and the objective function of the robust scheduling problem of discrete manufacturing workshop is to minimize the maximum completion time and minimize the difference between completion time and delivery time;Solve the robust scheduling problem of discrete manufacturing workshop.The application can obtain more actual, more robust production scheme.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of workshop scheduling, and particularly relates to a robust scheduling optimization method for a discrete manufacturing workshop based on deep reinforcement learning. BACKGROUND

[0002] Production scheduling, as a key link of production management in manufacturing industry, directly affects the efficiency of enterprise production and processing and the delivery time of products. A good scheduling scheme can improve the core competitiveness of enterprises such as production capacity. The scheduling function of a traditional factory workshop is usually arranged by a scheduler with rich experience after obtaining the processing state of all workshop equipment, order information, equipment operator and other workshop states. Due to insufficient experience, unpredictable events in the production workshop and other factors, the artificially designed scheduling scheme is difficult to respond to the sudden events in the production workshop in time.

[0003] With the rapid development of computer technology and the emergence of a large number of intelligent algorithms, meta-heuristic algorithms, artificial intelligence and other algorithms have been successfully applied in the field of production scheduling. Practice shows that the results obtained by intelligent algorithms have the advantages of faster response speed and better scheme quality compared with artificially designed schemes.

[0004] Intelligent algorithms have fast calculation speed, but there are various uncertain factors in the implementation process of intelligent algorithms in production scheduling. The actual industrial site environment is complex, and there are many uncontrollable events. It is difficult to predict all short-term and long-term uncertainties in advance when establishing the model. At present, the research of scheduling theory mainly focuses on complete information and static deterministic environment, and there are few literatures on project scheduling in risky and uncertain environments. Therefore, it is necessary to further study the scheduling theory under uncertain conditions, improve the robustness and stability of the scheduling scheme, and avoid the objective factors of environmental uncertainty from causing the original scheduling plan to be unable to continue to be executed or to have a large deviation in execution. SUMMARY

[0005] The application aims to provide a robust scheduling optimization method for a discrete manufacturing workshop based on deep reinforcement learning, which can obtain a more practical and robust production scheme.

[0006] To achieve the above-mentioned purpose, the technical solution adopted by the application is as follows:

[0007] A robust scheduling optimization method for a discrete manufacturing workshop based on deep reinforcement learning, the robust scheduling optimization method for a discrete manufacturing workshop based on deep reinforcement learning comprises:

[0008] Step 1, taking historical processing data, using neural network to fit the process processing time as a function of equipment, operator and production initial time;

[0009] Step 2, establishing a processing environment model of the factory workshop, the processing environment model including the number of operable equipment, on-duty operators and the inventory number of intermediate products;

[0010] Step 3, obtaining the product quantity to be processed on the day;

[0011] Step 4, constructing a discrete manufacturing workshop robust scheduling problem according to the processing environment model and the product quantity to be processed on the day, the objective function of the discrete manufacturing workshop robust scheduling problem being to minimize the maximum completion time and minimize the difference between the completion time and the delivery time;

[0012] Step 5, solving the discrete manufacturing workshop robust scheduling problem, including:

[0013] Step 51, selecting features, including four two-dimensional matrices of process processing time, end time of process, equipment utilization and process time left before delivery;

[0014] Step 52, describing the production procedure state as a four-channel image based on real-time processing data according to the four two-dimensional matrices, and approximating the state action value based on the four-channel image using a deep convolutional neural network;

[0015] Step 53, taking heuristic rules as the action space, outputting a scheduling optimization scheme using a Double DQN algorithm based on the state action value output by the deep convolutional neural network.

[0016] The following also provides several optional modes, but not as an additional limitation to the above overall scheme, just a further supplement or preferred, without technical or logical contradiction, each optional mode can be combined alone for the above overall scheme, but also can be combined between multiple optional modes.

[0017] As a preferred, the product quantity to be processed on the day includes: the product quantity in the order on the day, and the product quantity not completed processing in the order before the day.

[0018] As a preferred, the discrete manufacturing workshop robust scheduling problem is constructed according to the processing environment model and the product quantity to be processed on the day, including:

[0019] Let the number of operable equipment of the factory workshop be m, the product quantity to be processed on the day be n, the equipment set be M=(M1, M2,..., M m ), the workpiece set in the order on the day be J=(J1, J2,..., J n ), the i-th workpiece Ji There are a total of P i Processing steps, i = 1, 2, ..., n, O ij For workpiece J i The j-th process, j = 1, 2, ..., P i The inventory of intermediate products is denoted as the workpiece set L = (L1, L2, ..., L...). P-1 ), where P is the maximum number of processing steps for the products in this workshop, and S ijk For workpiece J i The j-th process O ij On the kth device M k The start time of construction, k = 1, 2, ..., m, T ijk For process O ij In machine M k The processing time of the above steps, of which T ijk The numerical values ​​are output from the fitted function obtained in step 1.

[0020] Let C ij For process O ij If the completion time is 0, then the maximum completion time C is 0. max =Max(C ij Therefore, minimizing the maximum completion time in the objective function can be described as the function f1 = minMax(C). ij ), and make Where RT represents the time required for reporting work, and S... i For workpiece J i The start time of the first processing step;

[0021] Order for workpiece J i The delivery time is D oi workpiece J i The predicted delivery time during the processing is D. pi , The difference between the completion time and delivery time in the objective function can then be described as the function f2 = minD.

[0022] Therefore, the objective function is ultimately described as: F = min(α·C) max +β·D);

[0023] The constraints are as follows:

[0024] x i，j，k ∈{0,1}

[0025]

[0026]

[0027] C i，j ≥0

[0028] α+β=1

[0029]

[0030]

[0031] In the formula, x i，j，k is a decision variable, and takes the value 1 when the process O i，j is processed on the equipment M k ; otherwise, it takes the value 0, and α and β are weighting weights, S i，j+1 represents the start time of the j+1th process of the ith workpiece.

[0032] As a preferred, the calculation method of the process processing time in the feature is as follows:

[0033] If the current product is an intermediate product, the processing time of the process completed by the current product is set to 0;

[0034] If the process of the current product has been assigned to the equipment, the processing time of the assigned process is set to 0;

[0035] If the current process is not completed and not assigned, the processing time of the process is obtained according to the fitting function trained in step 1.

[0036] As a preferred, the calculation method of the equipment utilization in the feature is as follows:

[0037] The equipment utilization is the utilization of the equipment used for processing each process, and the initial state is 0, and the utilization is calculated as follows: utilization = equipment working time / (current time - start time of the first process of all workpieces).

[0038] As a preferred, the calculation method of the process time to delivery in the feature is as follows:

[0039] The process time to delivery is the difference between the delivery time of the workpiece and the predicted delivery time during the processing of the workpiece, and in the initial state, all the differences of the same product are the same, and are initialized as the difference between the delivery time and the initial moment of production. During the process of workpiece assignment, if the process has been completed, the process time to delivery remains unchanged, otherwise, the process time to delivery of the unfinished process is updated to the delivery time of the workpiece minus the completion time of the latest process of the workpiece.

[0040] The deep reinforcement learning-based robust scheduling optimization method for discrete manufacturing workshops provided by the present application has the following beneficial effects compared with the prior art:

[0041] 1) Compared with the existing processing time determined scheduling model, the processing time of the process of the application is calculated according to the previous large amount of processing history data about time, personnel and equipment to predict and select the scheduling scheme, and the robustness of the scheduling optimization target and the scheme is balanced, so that the calculated scheme is more in line with the actual situation.

[0042] 2) The application takes the number of intermediate products in the workshop processing process into the production environment when calculating the scheduling scheme, so that the scheme conforms to the processing environment of the factory, and relieves the intermediate product inventory.

[0043] 3) Compared with the single-target scheduling method, the optimization target of the application is to minimize the maximum processing time and minimize the delivery time, which improves the processing efficiency, and judges the priority of the task according to the delivery date of the product, avoiding only considering the processing efficiency and making the order delayed.

[0044] 4) The method combines the advantages of deep convolutional neural network and reinforcement learning in real-time response and flexibility, describes the production process state as a 4-channel image, uses a deep convolutional neural network to approximate the state action value, and uses a Double Q network to relieve the transition estimation of Q value, uses multiple heuristic rules as the action space, and overcomes the short-sightedness of a single rule. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 Flow chart of the discrete manufacturing workshop robust scheduling optimization method based on deep reinforcement learning of the application;

[0046] Figure 2 Flow chart of the application using Double DQN algorithm to output scheduling optimization scheme. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs. The terms used in the specification of the application herein are only for the purpose of describing the specific embodiments and are not intended to limit the application.

[0049] In order to balance the optimization target of the scheduling scheme and the robustness of the scheme, so that the calculated scheme is more in line with the actual situation, an embodiment proposes a robust scheduling optimization method for discrete manufacturing workshops based on deep reinforcement learning. In the face of different workers, different equipment, different time periods and other factors, the same process processing time has uncertainty, and a large number of past processing data in the workshop is used to fit the function according to the workers, equipment, and time period through a neural network. When the deep reinforcement learning training method is used, the longest processing time of the adjacent time period before and after the time point is input as the input time, and the method can obtain a more realistic and more robust production scheme according to the running equipment and operating personnel on the same day, and more reasonably allocate the processing orders on the same day.

[0050] The robust scheduling optimization method for discrete manufacturing workshops based on deep reinforcement learning of the embodiment describes the production process state as a 4-channel image of the processing time of the process, the end time of the process, the utilization rate of the equipment, and the remaining time of the process to delivery, wherein the processing time of the process is predicted and selected according to a large number of past processing historical data about time, personnel, and equipment, to obtain a production scheme and further improve its robustness, as shown in Figure 1 The method specifically includes the following steps:

[0051] Step 1: Take historical processing data, and use a neural network to fit the function of the process processing time with respect to the equipment, the operator, and the initial production time.

[0052] Taking a steel cabinet furniture factory A as an example, there is a large amount of processing data in the factory production line. A large amount of processing data of different equipment in the past production process is used, the equipment serial number, the operator number, the initial production time, and the processing time of the process are taken as input and output, and a neural network is used to fit the function of the process processing time with respect to the equipment, the operator, and the initial production time.

[0053] Step 2: Establish a processing environment model of the factory workshop, wherein the processing environment model includes the number of running equipment, the number of on-duty operators, and the number of intermediate product inventories.

[0054] The processing process of the furniture factory A can be classified into six processes of cutting, carving, edge bonding, edge grinding, packaging, and warehousing. After one process is completed, the workpiece is put into the intermediate warehouse. The warehousing process is the last process, and the completion of the warehousing process represents that the workpiece is put into the finished product warehouse, and the workpiece processing is completed. The number of running equipment, the number of on-duty equipment operators, and the number of intermediate product inventories are obtained before the processing task scheduling is performed.

[0055] Step 3: Obtain the number of products to be processed on the same day.

[0056] Synchronize the orders of the day from the ERP system of factory A, and count the number of products that need to be processed. The number of products processed on the day includes the number of products in the orders of the day and the number of products (i.e., workpieces) that have not been completed in the orders before the day.

[0057] Step 4: Construct a robust scheduling problem for the discrete manufacturing workshop based on the processing environment model and the number of products that need to be processed on the day. The objective function of the robust scheduling problem is to minimize the maximum completion time and the difference between the completion time and the delivery time.

[0058] Establish the optimization objective based on the production environment and processing task characteristics of factory A. The production requirements of factory A are to improve production efficiency while not delaying the delivery period of orders as much as possible. Therefore, the optimization objective of the production plan of factory A is to minimize the maximum completion time f1 and the difference between the completion time and the delivery time f2:

[0059] f1 = min C max

[0060] f2 = min D

[0061] In the formula, C max is the maximum completion time, and D is the difference between the completion time and the delivery time.

[0062] The processing workshop of factory A can be classified as a flexible discrete processing workshop. The robust scheduling problem of the discrete manufacturing workshop can be described as n workpieces being processed on m machines. According to the process route requirements of the workpieces, each workpiece consists of one or more sequential processes, and each process may have multiple machines that can be selected for processing. The processing workshop of factory A has six processes: cutting, engraving, edge bonding, edge grinding, packaging, and warehousing. The edge grinding and edge bonding processes can share two machines.

[0063] During processing, the following conditions need to be met: At the same time, at most one machine can be selected for processing for each process. Once processing starts, it cannot be interrupted until it is completed. At the same time, at most one process can be processed on each machine. There are sequential constraints between the processes of the same workpiece, and there are no sequential constraints between the processes of different workpieces.

[0064] Let the number of machines that can run in the factory workshop be m, the number of products that need to be processed on the day be n, the set of machines be M = (M1, M2,..., M m ), the set of workpieces in the orders of the day be J = (J1, J2,..., J n ), the i-th workpiece J i have P i processes, i = 1, 2,..., n.

[0065] The process O ijIt can be any machine M from a set of selectable machines. k The processing is carried out on the same surface. In reality, workpieces in the same workshop have the same processing steps. Therefore, based on the processing environment of the phenolic resin board workshop in Factory A, the above definition can be simplified to each workpiece having a total of 6 processing steps. ij For workpiece J i The j-th process, j = 1, 2, ..., P i The inventory of intermediate products is denoted as the workpiece set L = (L1, L2, ..., L...). P-1 ), where P is the maximum number of processing steps for the products in this workshop, and S ijk For workpiece J i The j-th process O ij On the kth device M k The start time of construction, k = 1, 2, ..., m, T ijk For process O ij In machine M k The processing time of the above steps, of which T ijk The numerical values ​​are output from the fitting function obtained in step 1. When fitting the output from the fitting function, the operator of the equipment can specify or randomly assign the values ​​as needed.

[0066] The number of processes required in a workshop is the number of workpieces (i.e., the number of products) to be processed in the order minus the number of workpieces for the corresponding intermediate products. If a workshop processes the same product, the processing task can be described as follows:

[0067]

[0068] Where L a This represents the number of steps in an original processing task after subtracting intermediate products. The data in parentheses indicates which step in the original processing task actually requires a certain number of processing steps. If there are no intermediate products in the original processing task, the number of processing steps for each step is the same. If there are intermediate products, the number of steps for which the intermediate products have already been processed needs to be subtracted.

[0069] Let C ij For process O ij If the completion time is 0, then the maximum completion time C is 0. max =Max(C ij Therefore, minimizing the maximum completion time in the objective function can be described as the function f1 = minMax(C). ij ), and make Where RT represents the time required for reporting work, and the reporting time RT can be understood as a slack, while the required time is a fixed value, S. i For workpiece J i The start time of the first processing step is the initial production time of the processing task.

[0070] Order for workpiece J i The delivery time is D oi workpiece J i The predicted delivery time during the processing is D. pi , The difference between the completion time and delivery time in the objective function can then be described as the function f2 = minD.

[0071] Therefore, the objective function is ultimately described as: F = min(α·C) max +β·D).

[0072] The constraints are as follows:

[0073] 1) When process O i，j In device M k Decision variable x during processing i，j，k The value is 1; otherwise, the value is 0.

[0074] x i，j，k ∈{0,1}

[0075] 2) Ensure that a machine can only process one operation at a time:

[0076]

[0077] 3) Ensure that only one process is allowed to be in the processing state for a workpiece at any given time:

[0078]

[0079] 4) Ensure the workpiece processing time is normal:

[0080] C i，j ≥0

[0081] 5) Constraints on weighted weights:

[0082] α+β=1

[0083] 6) This indicates that the difference between the completion time of the j-th operation of the i-th workpiece, the processing time of that operation on the k-th machine, and the initial processing time of that operation is greater than 0, ensuring the smooth processing of that operation and satisfying the rationality of its processing procedure:

[0084]

[0085] 7) represents the start time S of the (j+1)th process of the i-th workpiece. i，j+1 The time taken for the j-th process to start and process on the k-th machine must be greater than the sum of the start time and processing time of the j-th process to ensure the processing sequence of the workpiece:

[0086]

[0087] In the formula, x i，j，k S is the decision variable, α and β are the weighting weights, and S is the weighting weight. i，j+1 This represents the start time of the (j+1)th process for the i-th workpiece.

[0088] Step 5: Solve the robust scheduling problem for the discrete manufacturing workshop.

[0089] Step 51: The first step is to extract model features. The selected features should reflect both the global and local states. The features are selected as four two-dimensional matrices: process duration, process end time, equipment utilization rate, and remaining time before delivery.

[0090] For a two-dimensional matrix of process duration: the number of workpieces and the number of processes required in the matrix are referenced to the number of processes L obtained by subtracting intermediate products from the original processing task. a If the product is an intermediate product, the processing time of the previously completed processes is set to 0; if the process has been assigned to equipment, the processing time of that process is also set to 0. The processing time value of each process comes from the fitting function completed by the neural network training in step 1. In order to enhance the robustness of the solution obtained in subsequent calculations, when calculating the processing time, the longest processing time value among the three time periods before and after the current moment is taken.

[0091] A two-dimensional matrix for the completion time of each process: The process completion time matrix represents the time when each process is completed.

[0092] A two-dimensional matrix for equipment utilization: Equipment utilization is the utilization rate of equipment used in each processing step. The initial state is 0, and the subsequent utilization rate is calculated as follows: Utilization rate = Equipment working time / (Current time - Start time of the first process of all workpieces).

[0093] For the two-dimensional matrix of time remaining until delivery of a process: the time remaining until delivery of a process is the difference between the delivery time of the workpiece and the predicted delivery time during the workpiece processing. In the initial state, all differences for the same product are the same, initialized as the difference between the delivery time and the initial moment of production. During the workpiece allocation process, if the process has been completed, the time remaining until delivery of the process remains unchanged (for example, if the second process of workpiece 1 needs to be allocated, that is, the first process has been completed, then when allocating the second process, the value of the first process does not change, only the current process and subsequent processes are changed). Otherwise, the time remaining until delivery of an unfinished process is updated to the delivery time of the workpiece minus the completion time of the most recent process of the workpiece.

[0094] Table 1 shows the initial states of four two-dimensional matrices, and Table 2 shows the states after two dispatching processes based on the initial states in Table 1.

[0095] Table 1 Initial states of the four two-dimensional matrices

[0096]

[0097] Table 2 shows the status after two dispatching processes.

[0098]

[0099] In the four two-dimensional matrices, each row represents a workpiece, and each column represents a process. The rows of all four matrices represent workpieces with the same sequence number, and the columns represent the same processes. For example, the 3rd row and 4th column of all four matrices represent the fourth process of workpiece 3. Two dispatches were made for the first and second processes of workpiece 1. The end time matrix is ​​updated as the processes are completed. The equipment utilization matrix represents the utilization rate of the equipment processing that process during the current processing phase. It's important to note that normalization is required during state storage and calculation: matrices one and two are divided by the maximum value of the current time step, and the processing time of each process of workpiece four is divided by the initial value.

[0100] Step 52: Based on the real-time processing data, describe the production process status as a four-channel image using four two-dimensional matrices, and approximate the state action value using a deep convolutional neural network based on the four-channel image.

[0101] To extract features, a CNN network is built. A CNN mainly consists of an input layer, a convolutional layer, a ReLU layer, a pooling layer, and a fully connected layer. In practical applications, the convolutional layer and the ReLU layer are collectively referred to as a convolutional layer.

[0102] The four two-dimensional feature matrices can be described as an image with a depth of 4, a length equal to the number of processed product parts, and a width equal to the maximum number of product processes. The convolutional layer is set to have 4 input channels and 6 output channels. After zero padding, the output of the convolutional layer is an image with the same length and width but a depth of 6.

[0103] The purpose of pooling layers is to gradually reduce the size of the data space, decrease the number of parameters in the network, reduce computational resource consumption, and control overfitting. Pooling layers use the MAX operation, operating independently on each depth slice of the input data volume, with a kernel size of 2.

[0104] Step 53: Using heuristic rules as the action space, and based on the state-action values ​​output by the deep convolutional neural network (the deep convolutional neural network is part of q_net, and q_net outputs state-action values), the Double DQN algorithm is used to output a scheduling optimization scheme. The specific process is as follows... Figure 2 As shown:

[0105] Initialize the DQN agent's memory bank, and randomly initialize the current value function Q and the target action value function with weights.* .

[0106] The agent, as the main body of DQN decision-making, makes action decisions in the environment.

[0107] To balance development and exploration, an ∈-greedy action decision-making strategy is adopted. This strategy means that when an agent selects an action, there is a probability of ∈ that it will randomly select an action from the action space set, and a probability of 1-∈ that it will select the action with the highest value among the historical actions.

[0108] The reward function is set as the difference between the utilization rate of the equipment processing the current operation and the utilization rate of the equipment during the last operation, plus the sum of the values ​​of the last operation of all workpieces in the remaining time matrix before delivery.

[0109] The action space, as the set of actions that an intelligent agent can select during the decision-making process, uses scheduling rules as the action space. Specific heuristic rules are shown in Appendix Table 3, which are common scheduling rules in the field of production scheduling. Among them, the OOD (latest) and OOD (farthest) rules are related to the delivery date, selecting the workpieces closest to and furthest from the delivery time of the current process, respectively.

[0110] Table 3 Heuristic Rules

[0111] Serial number Scheduling rule Rule description 1 LWKR Select the workpiece with the shortest remaining processing time 2 MWKR Select the workpiece with the longest remaining processing time 3 SPT Select the workpiece with the shortest process processing time 4 LPT Select the workpiece with the longest process processing time 5 SPT / TWK Select the workpiece with the smallest ratio of process processing time to total processing time 6 LPT / TWK Select the workpiece with the largest ratio of process processing time to total processing time 7 SPT / TWKR Select the workpiece with the smallest ratio of process processing time to remaining processing time 8 LPT*TWKR Select the workpiece with the largest product of process processing time and remaining processing time 9 SPT*TWK Select the workpiece with the smallest product of process processing time and total processing time 10 LPT*TWK Select the workpiece with the largest product of process processing time and total processing time 11 SPT*TWKR Select the workpiece with the smallest product of process processing time and remaining processing time 12 LPT*TWKR Select the workpiece with the largest product of process processing time and remaining processing time 13 SRM Select the workpiece with the shortest remaining processing time excluding the current process 14 LRM Select the workpiece with the longest remaining processing time excluding the current process 15 SSO Select the workpiece with the shortest successor process processing time 16 LSO Select the workpiece with the longest successor process processing time 17 SPT+SSO Select the workpiece with the longest current process processing time and successor process processing time 18 LPT+SSO Select the workpiece with the longest current process processing time and successor process processing time 19 SPT / LSO Select the workpiece with the smallest ratio of current process processing time to successor process processing time 20 LPT / SSO Select the workpiece with the largest ratio of current process processing time to successor process processing time 21 OOD(latest) Select the workpiece with the closest current process to the delivery time 22 OOD(farthest) Select the workpiece with the farthest current process to the delivery time

[0112] After the scheduling rules are selected, the specific actions to be performed are calculated.

[0113] Execute the specific action, calculate the reward and the next state after the action is executed. Store the state, reward, and next action of the agent at the time of the current action execution.

[0114] If the agent's memory counter value is greater than or equal to the batch size, the agent has reached the learning condition. The agent randomly selects memory data from the memory bank to train the Q network, calculates the current target Q value, and updates all parameters of the Q network through backpropagation of the gradient of the neural network using the mean squared error loss function. If the current state is the termination state, the current iteration is complete.

[0115] After a certain number of training steps, Q is made * Synchronize with the parameters of the Q network.

[0116] To mitigate overestimation of the Q-value, this method uses Double DQN to decouple the target Q. * The selection of the value action and the calculation of the target Q value are no longer directly in the target Q. * Instead of selecting the action with the maximum Q-value in the current Q-network, this method selects the action corresponding to the maximum Q-value in the current Q-network and then uses this selected action in the target network Q-network. *The target value is calculated inside.

[0117] The current target Q value is calculated as follows:

[0118]

[0119] In the formula, Q(s) j+1 (a,w) represents the current Q-network at s j+1 The state, and the Q value for executing action 'a';

[0120] Q * (s j+1 argmax a Q(s j+1 ,a,w),w * )

[0121] This represents the Q-value of the target network Q* after it executes the action corresponding to the maximum Q-value in the current Q-network; 'a' represents the action taken in the current state; 's' represents the action taken in the current state. j+1 γ represents the next state after performing action 'a', and γ represents the discount factor, indicating the importance of future rewards.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A robust scheduling optimization method for discrete manufacturing shop based on deep reinforcement learning, characterized in that, The robust scheduling optimization method for a discrete manufacturing workshop based on deep reinforcement learning comprises the following steps: Step 1: Obtain historical processing data and use a neural network to fit a function of process processing time with respect to equipment, operators, and initial production time; Step 2: Establish a processing environment model of the factory workshop, which includes the number of available equipment, on-duty operators, and the inventory of intermediate products; Step 3: Obtain the number of products that need to be processed on the day; Step 4: Construct a robust scheduling problem for a discrete manufacturing workshop based on the processing environment model and the number of products that need to be processed on the day, with the objective function being to minimize the maximum completion time and the difference between the minimum completion time and the delivery time; Step 5: Solve the robust scheduling problem for a discrete manufacturing workshop, comprising the following steps: Step 51: Select features, including four two-dimensional matrices of process processing time, process end time, equipment utilization, and process time remaining until delivery; Step 52: Based on the four two-dimensional matrices, describe the production procedure state as a four-channel image based on real-time processing data, and use a deep convolutional neural network to approximate the state-action value based on the four-channel image; Step 53: Use heuristic rules as the action space, and based on the state-action value output by the deep convolutional neural network, use the Double DQN algorithm to output a scheduling optimization scheme.

2. The deep reinforcement learning based robust scheduling optimization method for discrete manufacturing job-shop as claimed in claim 1, wherein, The number of products that need to be processed on the day includes the number of products in the order on the day and the number of products that have not been completed in the order before the day.

3. The deep reinforcement learning based robust scheduling optimization method for discrete manufacturing job-shop of claim 1, wherein, The robust scheduling problem for a discrete manufacturing workshop is constructed based on the processing environment model and the number of products that need to be processed on the day, comprising the following steps: Let the number of devices that can run in the factory workshop be m, the number of products that need to be processed on that day be n, the device set be M=(M1, M2,..., M m ), the set of workpieces in the order of that day be J=(J1, J2,..., J n ), the i-th workpiece J i have P i machining processes, i=1, 2,..., n, O ij be the j-th process of workpiece J i , j=1, 2,..., P i , the inventory number of intermediate products be the workpiece set L=(L1, L2,..., L P-1 ), where P is the maximum number of processes for product processing in the workshop, S ijk be the start time of the j-th process O i of workpiece J ij on the k-th device M k , k=1, 2,..., m, T ijk be the process processing time of process O ij on machine M k , where T ijk the value is output by the fitting function trained in step 1; Let C ij be the completion time of process O ij , then the maximum completion time C max = Max(C ij ), thus the minimization of the maximum completion time in the objective function can be described as a function f1 = minMax(C ij ), and let where RT is the time consumed by the reporting work, S i is the start time of the first process to be processed of the workpiece J i ; Order for workpiece J i The delivery time is D oi workpiece J i The predicted delivery time during the processing is D. pi , The objective function can then be described as minimizing the difference between the completion time and the delivery time, which can be expressed as the function f2 = minD. The objective function is therefore ultimately described as: F = min (a - C max + β - D); The constraint condition is: x i,j,k ∈{0,1} C i,j ≥0 α+β＝1 where x i,j,k is the decision variable, and is equal to 1 when the process O i,j is processed on the equipment M k ; otherwise, it is equal to 0, and α, β are weighting weights, and S i,j+1 represents the start time of the j+1th process of the ith workpiece.

4. The deep reinforcement learning based robust scheduling optimization method for discrete manufacturing job-shop of claim 1, wherein, The calculation method of the process processing time in the features is as follows: If the current product is an intermediate product, set the processing time of the completed process of the current product to 0; If the process of the current product has been assigned to a device, set the processing time of the assigned process to 0; If the current process is not completed and not assigned, obtain the processing time of the process according to the fitting function obtained in step 1.

5. The deep reinforcement learning based robust scheduling optimization method for discrete manufacturing job-shop of claim 1, wherein, The calculation method of the equipment utilization in the features is as follows: The equipment utilization is the utilization of the equipment used for each process, and the initial state is 0. The utilization is calculated as follows: utilization = equipment working time / (current time - start time of the first process of all workpieces).

6. The deep reinforcement learning based robust scheduling optimization method for discrete manufacturing job-shop of claim 1, wherein, The calculation method of the process time remaining until delivery in the features is as follows: The process time remaining until delivery is the difference between the delivery time of a workpiece and the predicted delivery time during the processing of the workpiece. In the initial state, all differences for the same product are the same, and are initialized as the difference between the delivery time and the initial production time. During the allocation of the workpiece, if the process has been completed, the process time remaining until delivery remains unchanged, otherwise, the process time remaining until delivery for the uncompleted process is updated to the delivery time of the workpiece minus the completion time of the most recent process of the workpiece.

Images