DQN-based intelligent factory job shop scheduling method and system

Abstract

The invention discloses a DQN-based intelligent factory job shop scheduling method and system, and belongs to the technical field of production scheduling, and the method specifically comprises the following steps: obtaining real-time data generated by intelligent equipment, and extracting key shop system features; calculating a current workshop system state based on the key workshop system characteristics; based on the current workshop system state, using the DQN network to select and execute an optimal scheduling rule, and obtaining a scheduling result; and calculating and obtaining a reward function rt based on a scheduling result. And finally, updating the weight of the DQN network according to the workshop system state and the scheduling rule, obtaining the updated DQN network according to the updated weight of the DQN network, and selecting a better scheduling rule in future decisions by using the updated DQN network. According to the method provided by the invention, a large amount of data can be processed in real time, and the optimal scheduling rule is automatically explored and learned through deep reinforcement learning, so that the scheduling efficiency and the capability of coping with a complex environment are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of production scheduling, and in particular to a DQN-based intelligent factory job shop scheduling method and system. Background Art

[0002] As an important part of the modern economy, manufacturing plays a key role globally. In the manufacturing industry, production scheduling is a key link, which has a significant impact on production efficiency, product quality and corporate competitiveness. Production scheduling refers to the reasonable allocation of product manufacturing resources and the determination of the order of work order scheduling and processing under limited time and resources to complete specific scheduling tasks and achieve pre-set optimization goals. With the changes in market demand, consumers' demand for personalized and differentiated products has become increasingly strong, resulting in the production orders of manufacturing companies increasingly developing in the direction of small batches and multiple varieties. This trend of orders requires companies to have higher production planning and scheduling management capabilities. However, traditional workshop scheduling methods are usually based on experience and rules, lack flexibility and intelligence, and are difficult to adapt to complex production environments and dynamic market demands. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for scheduling a smart factory job shop based on DQN, which can improve the flexibility and intelligence of shop scheduling based on the real-time data generated by smart devices and extract key shop system features. The present invention is implemented through the following technical solutions.

[0004] In a first aspect, the present invention provides a DQN-based smart factory job shop scheduling method, comprising:

[0005] Acquire real-time data generated by smart devices and extract key shop floor system features;

[0006] Calculate the current workshop system status based on the key workshop system characteristics ;

[0007] Based on the current workshop system status Using DQN network to select the optimal scheduling rule And execute to get the scheduling result; execute the scheduling rules After that, the workshop system status changes from Transfer to ;

[0008] Based on the scheduling results, calculate and obtain the reward function ;

[0009] based on The tuple updates the weight of the DQN network, obtains an updated DQN network according to the updated weight of the DQN network, and uses the updated DQN network to select a better scheduling rule in future decision-making.

[0010] In the actual production scheduling process of smart factory workshops, in order to adapt to complex production environments and dynamic market demands, according to the method proposed in the present invention, by preprocessing workshop data and calculating the workshop system status, the continuous system status value is used as the input of the deep reinforcement learning algorithm, and the optimal scheduling rules are automatically explored and learned through deep reinforcement learning, thereby improving the scheduling efficiency and the ability to cope with complex environments.

[0011] Optionally, the key workshop system features include: The number of orders being produced in the workshop system at the moment , Total Order Quantity , The number of equipment being processed in the workshop system at the moment , Total number of devices , the minimum processing time of candidate orders in the workshop system , Maximum processing time of candidate orders in the workshop system , the minimum remaining processing time of candidate orders in the workshop system , the maximum remaining processing time of candidate orders in the workshop system , the minimum delay time of candidate orders in the workshop system and the maximum delay time of candidate orders in the workshop system .

[0012] Optionally, the current workshop system status Includes: Orders already scheduled , Equipment Utilization , processing time ratio , delivery time ratio and delay ratio ,in They are calculated by the following formulas:

[0013] ,

[0014] ,

[0015] ,

[0016] ,

[0017] .

[0018] Optionally, the current workshop system status Using DQN network to select the optimal scheduling rule The method comprises the following steps: the DQN network selects one of the four pre-set composite scheduling rules as the optimal scheduling rule according to the current system state; and the selected scheduling rule is applied to the actual production scheduling to determine the workpiece to be processed, the process and the corresponding processing machine.

[0019] Optionally, the four preset composite scheduling rules include:

[0020] DR1, when there is no workpiece delay, the workpiece with the shortest changeover time is assigned to the machine with the earliest start time;

[0021] DR2, when there is no workpiece delay, the workpiece with the highest pre-allocated balanced production score is allocated to the machine with the earliest start time;

[0022] DR3, when there is a workpiece delay, the workpiece with the smallest changeover time is allocated to the machine with the highest pre-allocated balanced production score;

[0023] DR4, when there is a workpiece delay, the workpiece with the highest pre-allocation balanced production score is allocated to the machine with the shortest changeover time.

[0024] Optionally, the optimal scheduling rule is selected The selection criteria are such that the reward function maximum.

[0025] Optionally, the scheduling results include: number of orders delivered on time, equipment The completion time of all equipment, the average completion time of all equipment, the total switching time and the total production time of all equipment, where The serial number of the equipment that is processing the order.

[0026] Optionally, the reward function is calculated and obtained based on the scheduling result. Includes: Calculate the delivery rate score at time t based on the scheduling results , Balanced Production Score and the conversion time score , where the setting Orders in The production process on the equipment, the order number is 1, 2, ..., i, ..., n, the equipment number is 1, 2, ..., j, ... m, each order i has only one process, i and j are the order and equipment numbers respectively; the calculation of the delivery rate score at time t , Balanced Production Score and the conversion time score They are obtained by the following formulas:

[0027] ,

[0028] ,

[0029] ,

[0030] Among them, on-time delivery means: the date with the maximum completion time of all work orders in the order is ≤ the order requirement date; the completion time of equipment j is equal to the end time of the last work order of the equipment minus the start time of the first work order; the total production time of equipment j is equal to the end time of all work orders on the equipment minus the start time;

[0031] Scoring based on delivery rate at time t , Balanced Production Score and the conversion time score The production scheduling effect score at time t is calculated using the following formula :

[0032] ,

[0033] Score of production scheduling effect based on time t The reward function is calculated by the following formula: :

[0034] Reward Function , where It is the production scheduling effect score at time t-1.

[0035] The purpose of selecting the optimal scheduling rule in this invention is to make the reward function Maximum, thus making the score based on delivery rate , Balanced Production Score and the conversion time score Maximum, to achieve the goal of maximizing delivery rate, maximizing balanced production and minimizing changeover time.

[0036] In a second aspect, the present invention provides a DQN-based intelligent factory job shop scheduling system, comprising:

[0037] The acquisition module is used to obtain real-time data generated by smart devices and extract key workshop system features;

[0038] A calculation module, used to calculate the current workshop system status based on the key workshop system characteristics ;

[0039] Execution module, used to execute Using DQN network to select the optimal scheduling rule And execute to get the scheduling result; among them, execute the scheduling rules After that, the workshop status changed from Transfer to ;

[0040] Reward function calculation module, used to calculate and obtain the reward function based on the scheduling results .

[0041] Select module for The tuple updates the weight of the DQN network, obtains an updated DQN network according to the updated weight of the DQN network, and uses the updated DQN network to select a better scheduling rule in future decision-making.

[0042] Beneficial Effects

[0043] The DQN-based smart factory job shop scheduling method proposed in the present invention can process large amounts of data in real time, automatically explore and learn the optimal scheduling rules through deep reinforcement learning, thereby improving scheduling efficiency and the ability to cope with complex environments.

[0044] The present invention designs four new composite scheduling rules, which can comprehensively consider multiple factors, such as equipment balance, switching time, workpiece delay rate, etc., to cope with more complex and conflicting workshop environments. The present invention pre-processes workshop data and calculates workshop system status, and uses continuous workshop system status values ​​as input to the deep reinforcement learning algorithm, which can more effectively integrate and process workshop data, mine valuable information, and provide strong support for scheduling decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The figure is a flow chart of a DQN-based smart factory job shop scheduling method in one embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following is a further description in conjunction with the accompanying drawings and specific embodiments. In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features.

[0047] Example 1

[0048] This embodiment introduces a DQN-based smart factory job shop scheduling method, which specifically includes the following contents:

[0049] Acquire real-time data generated by smart devices and extract key shop floor system features;

[0050] Calculate the current workshop system status based on the key workshop system characteristics ;

[0051] Based on the current workshop system status Using DQN network to select the optimal scheduling rule And execute to get the scheduling result; among them, execute the scheduling rules After that, the workshop system status changes from Transfer to ;

[0052] Based on the scheduling results, calculate and obtain the reward function ;

[0053] based on The tuple updates the weight of the DQN network, obtains an updated DQN network according to the updated weight of the DQN network, and uses the updated DQN network to select a better scheduling rule in future decision-making.

[0054] In the actual production scheduling process of smart factory workshops, in order to adapt to complex production environments and dynamic market demands, according to the method proposed in the present invention, by preprocessing workshop data and calculating the workshop system status, the continuous workshop system status value is used as the input of the deep reinforcement learning algorithm, and the optimal scheduling rules are automatically explored and learned through deep reinforcement learning, thereby improving the scheduling efficiency and the ability to cope with complex environments.

[0055] Example 2

[0056] Based on Example 1, this example introduces a specific implementation process of a DQN-based smart factory job shop scheduling method. Figure 1 As shown, specifically including the following:

[0057] 1. Methodology

[0058] When the smart factory workshop is actually working, various sensors, smart devices and production management systems in the smart factory generate a large amount of data in real time. These data are collected and pre-processed, and key workshop system features are extracted and analyzed, including: The number of orders being produced in the workshop system at the moment , Total Order Quantity , The number of equipment being processed in the workshop system at the moment , Total number of devices , the minimum processing time of candidate orders in the workshop system , Maximum processing time of candidate orders in the workshop system , the minimum remaining processing time of candidate orders in the workshop system , the maximum remaining processing time of candidate orders in the workshop system , the minimum delay time of candidate orders in the workshop system and the maximum delay time of candidate orders in the workshop system .

[0059] Based on the key features of the workshop system extracted above, the current workshop system status is calculated and input into the DQN network. The current workshop system status includes: the ratio of scheduled orders to , Equipment Utilization , processing time ratio , delivery time ratio and delay ratio ,in They are calculated by the following formulas:

[0060] ,

[0061] ,

[0062] ,

[0063] ,

[0064] .

[0065] The DQN network selects and executes the optimal scheduling rule based on the current workshop system state. Specifically, the DQN network selects one of the four pre-set composite scheduling rules as the optimal scheduling rule based on the current workshop system state. The selection criteria for selecting the optimal scheduling rule are as follows: The maximum, and based on the selected optimal scheduling rules, the scheduling results are obtained. The scheduling results include: the number of orders delivered on time, the number of equipment The completion time of all equipment, the average completion time of all equipment, the total switching time and the total production time of all equipment, where The serial number of the equipment that is processing the order.

[0066] The four pre-set composite scheduling rules include:

[0067] DR1, when there is no workpiece delay, the workpiece with the shortest changeover time is assigned to the machine with the earliest start time;

[0068] DR2, when there is no workpiece delay, the workpiece with the highest pre-allocated balanced production score is allocated to the machine with the earliest start time;

[0069] DR3, when there is a workpiece delay, the workpiece with the smallest changeover time is allocated to the machine with the highest pre-allocated balanced production score;

[0070] DR4, when there is a workpiece delay, the workpiece with the highest pre-allocation balanced production score is allocated to the machine with the shortest changeover time.

[0071] The DQN network selects and executes the optimal scheduling rules based on the current workshop system status, including: the number of orders delivered on time, equipment The completion time of all equipment, the average completion time of all equipment, the total switching time and the total production time of all equipment, where The serial number of the equipment that is processing the order.

[0072] Finally, based on the scheduling results, calculate and obtain the reward function Includes: Calculate the delivery rate score at time t based on the scheduling results , Balanced Production Score and the conversion time score , where the setting Orders in The production process on the equipment, the order number is 1, 2, ..., i, ..., n, the equipment number is 1, 2, ..., j, ... m, each order i has only one process, i and j are the order and equipment numbers respectively; the calculation of the delivery rate score at time t , Balanced Production Score and the conversion time score They are obtained by the following formulas:

[0073] ,

[0074] ,

[0075] ,

[0076] Among them, on-time delivery means: the date with the maximum completion time of all work orders in the order is ≤ the order requirement date; the completion time of equipment j is equal to the end time of the last work order of the equipment minus the start time of the first work order; the total production time of equipment j is equal to the end time of all work orders on the equipment minus the start time;

[0077] Scoring based on delivery rate at time t , Balanced Production Score and the conversion time score The production scheduling effect score at time t is calculated using the following formula :

[0078] ,

[0079] Score of production scheduling effect based on time t The reward function is calculated by the following formula: :

[0080] Reward Function , where It is the production scheduling effect score at time t-1.

[0081] The purpose of selecting the optimal scheduling rule in this invention is to make the reward function Maximum, thus making the score based on delivery rate , Balanced Production Score and the conversion time score Maximum, to achieve the goal of maximizing delivery rate, maximizing balanced production and minimizing changeover time.

[0082] 2. Experiment

[0083] The standard data set of flexible job workshop, Bran-dimarte data set (Mk01-Mk010), is used, and compared with the commonly used algorithms for solving flexible job workshops: genetic algorithm GA, PSO particle swarm algorithm and ACO ant colony algorithm.

[0084] Table 1 Algorithm comparison results

[0085] Dataset GA PSO ACO DQN Mk01 49.8 40 73 40 Mk02 37.5 27 44 26 Mk03 215.0 206.4 292 204 Mk04 68.3 63 119 61 Mk05 335 410 196 216 Mk06 148 258 97 111 Mk07 305 537 249 221 Mk08 748 977 651 606 Mk09 788 1249 464 425 Mk010 603 582 444 390

[0086] As can be seen from the table, in problems with different data sizes, compared with the genetic algorithm GA, PSO particle swarm algorithm and ACO ant colony algorithm, the DQN algorithm has a stronger ability to get rid of the local optimal solution, and the algorithm has higher accuracy, and basically can obtain better solutions. Overall, the traditional meta-heuristic algorithm achieves high stability and solution accuracy through repeated iterations and multiple runs, which will inevitably lead to the problem of long algorithm running time. The deep reinforcement learning algorithm is to train the decision model offline, call the model online to quickly optimize multiple examples in parallel, and does not need to update the model parameters. Under the premise of ensuring relatively high stability and solution accuracy, the problem of long algorithm running time is avoided. This verifies the effectiveness of the algorithm proposed in this paper.

[0087] In terms of overall performance, DQN shows relatively good performance on most datasets (Mk01, Mk02, Mk03, Mk04, Mk06, Mk07, Mk08, Mk09, Mk010), especially on Mk02, Mk03, Mk04, Mk06, Mk08 and Mk09 datasets, where its performance is significantly better than the other three algorithms. This shows that DQN may have stronger adaptability and optimization capabilities when dealing with complex problems or large-scale datasets.

[0088] GA, PSO, and ACO have their own advantages and disadvantages on different data sets, but in general, their performance on some data sets is not as good as DQN. This may be due to the limitations of these traditional optimization algorithms in solving high-dimensional, nonlinear, or dynamically changing problems.

[0089] In terms of performance stability, DQN can maintain relatively stable performance on multiple data sets, which shows that it has strong generalization ability and robustness. DQN approximates the Q function through a deep neural network, which can process complex input data and learn effective strategies, thereby maintaining good performance in different environments. In contrast, the performance of GA, PSO, and ACO on different data sets fluctuates greatly, which may be related to their parameter settings, initial populations, or the choice of search strategies. When facing complex problems, these algorithms may require more parameter adjustment to ensure stable performance.

[0090] 3. Data Description

[0091] Preliminary data description: The preliminary round provides a total of 20 products, 10 equipment, 4 process routes, and 2 bottleneck materials. Each process has corresponding machine types and different quantities. Among them, an order in the product demand data will contain multiple work orders, each work order will correspond to a process, and each product will correspond to a process route. In the process route data, each process route will contain several processes, and each process will have several equipment that can be processed. Each equipment has a unit processing time (seconds) for each product. Multiplying by the product demand quantity of the work order can get the processing time of the work order on the equipment. The work calendar data contains the start time and end time that each device can work every day. The switching time refers to the time taken to switch from product A to product B on a certain device. If it is the same product, the switching time is 0. When calculating the processing time of the work order, the switching time is uniformly added to the processing time of the later work order. Bottleneck materials refer to the quantity of this material that can be supplied on the day. The number of work orders that include this material in daily production cannot exceed the total supply of this material on the day. Only some products are restricted by bottleneck materials. If the "bottleneck material number" field is empty, it means that there is no bottleneck material and production can be scheduled normally. Product demand data is as follows:

[0092] Requirement Date Demand Order Number Work Order Number Process No. Product Number Process route type Bottleneck material number 2024-04-11 100 1001 1001-1 1 3001 G1 4001 2024-04-11 200 1001 1001-2 2 3001 G1 4001

[0093] Process route data

[0094] Process route type Process No. Available device numbers Processing time per product unit (seconds) G1 1 T001 10 G1 2 T001 20

[0095] Work calendar data

[0096] Device Number Start time End Time Calendar Type T001 2024-04-08 08:00:00 2024-04-08 12:00:00 day shift T002 2024-04-08 13:00:00 2024-04-08 18:00:00 day shift

[0097] Switching time data

[0098] Device Number Previous product After product Switching time (minutes) T001 3001 3002 1 T001 3001 3003 4

[0099] Bottleneck material data

[0100] Material Number Availability Date Supply Quantity 4001 2024-04-08 2000 4002 2024-04-08 3500

[0101] The scoring rules for the preliminary round are as follows: Consider the production process of n work orders on m equipment, and each work order i has only one process. Among them, the order numbers are 1, 2, ..., i, ..., n, and the equipment numbers are 1, 2, ..., j, ... m. Each order i has only one process, i and j are the order and equipment numbers respectively; the calculation of delivery rate score , Balanced Production Score and the conversion time score They are obtained by the following formulas: ,

[0102] ,

[0103] ,

[0104] Note: (1) On-time delivery: the date with the maximum completion time of all work orders in the order is ≤ the order requirement date; (2) The completion time of equipment j is equal to the end time of the last work order of the equipment minus the start time of the first work order; (3) The total production time of equipment j is equal to the sum of the end time of all work orders on the equipment minus the start time;

[0105] Based on delivery rate scoring , Balanced Production Score and the conversion time score The production scheduling effect score at time t is calculated using the following formula :

[0106] ,

[0107] Preliminary round score To rank.

[0108] Example 3

[0109] This embodiment introduces a DQN-based smart factory job shop scheduling system, including:

[0110] The acquisition module is used to obtain real-time data generated by smart devices and extract key workshop system features;

[0111] A calculation module, used to calculate the current workshop system status based on the key workshop system characteristics ;

[0112] Execution module, used to execute Using DQN network to select the optimal scheduling rule And execute to get the scheduling result; among them, execute the scheduling rules After that, the workshop system status changes from Transfer to ;

[0113] Reward function calculation module, used to calculate and obtain the reward function based on the scheduling results .

[0114] Select module for The tuple updates the weight of the DQN network, obtains an updated DQN network according to the updated weight of the DQN network, and uses the updated DQN network to select a better scheduling rule in future decision-making.

[0115] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A DQN-based smart factory job shop scheduling method, characterized in that: include: Acquire real-time data generated by smart devices and extract key shop floor system features; Calculate the current workshop system status based on the key workshop system characteristics ; Based on the current workshop system status Using DQN network to select the optimal scheduling rule And execute to get the scheduling result; execute the scheduling rules After that, the workshop system status changes from Transfer to ; Based on the scheduling results, calculate and obtain the reward function ; based on The tuple updates the weight of the DQN network, obtains an updated DQN network according to the updated weight of the DQN network, and uses the updated DQN network to select a better scheduling rule in future decision-making.

2. The method for scheduling a job shop in an intelligent factory according to claim 1, characterized in that: The key shop floor system features include: The number of orders being produced in the workshop system at the moment , Total Order Quantity , The number of equipment being processed in the workshop system at the moment , Total number of devices , Minimum processing time of candidate orders in the workshop system , Maximum processing time of candidate orders in the workshop system , the minimum remaining processing time of candidate orders in the workshop system , the maximum remaining processing time of candidate orders in the workshop system , the minimum delay time of candidate orders in the workshop system and the maximum delay time of candidate orders in the workshop system .

3. The method for scheduling a job shop in an intelligent factory according to claim 2, characterized in that: The current workshop system status Includes: Orders already scheduled , Equipment Utilization , processing time ratio , delivery time ratio and delay ratio ,in They are calculated by the following formulas: ， ， ， ， 。 4. The method for scheduling a job shop in an intelligent factory according to claim 3, characterized in that: Based on the current workshop system status Using DQN network to select the optimal scheduling rule The method comprises the following steps: the DQN network selects one of the four pre-set composite scheduling rules as the optimal scheduling rule according to the current workshop system state; and the selected scheduling rule is applied to the actual production scheduling to determine the workpiece to be processed, the process and the corresponding processing machine.

5. The method for scheduling a job shop in an intelligent factory according to claim 4, characterized in that: The four pre-set composite scheduling rules include: DR1, when there is no workpiece delay, the workpiece with the shortest changeover time is assigned to the machine with the earliest start time; DR2, when there is no workpiece delay, the workpiece with the highest pre-allocated balanced production score is allocated to the machine with the earliest start time; DR3, when there is a workpiece delay, the workpiece with the smallest changeover time is allocated to the machine with the highest pre-allocated balanced production score; DR4, when there is a workpiece delay, the workpiece with the highest pre-allocation balanced production score is allocated to the machine with the shortest changeover time.

6. The method for scheduling a job shop in an intelligent factory according to claim 4, characterized in that: The optimal scheduling rule is selected The selection criteria are such that the reward function maximum.

7. The method for scheduling a job shop in an intelligent factory according to claim 4, characterized in that: The scheduling results include: number of orders delivered on time, equipment The completion time of all equipment, the average completion time of all equipment, the total switching time and the total production time of all equipment, where It is the serial number of the equipment that is processing the order.

8. The method for scheduling a job shop in an intelligent factory according to claim 7, characterized in that: Based on the scheduling results, the reward function r is calculated and obtained. t Including: Calculate the delivery rate at time t based on the scheduling results , Balanced Production Score and the conversion time score , where the setting Orders in The production process on the equipment, the order number is 1, 2, ..., i, ..., n, the equipment number is 1, 2, ..., j, ... m, each order i has only one process, i and j are the order and equipment numbers respectively; the calculation of the delivery rate score at time t , Balanced Production Score and the conversion time score They are obtained by the following formulas: ， ， ， Scoring based on delivery rate at time t , Balanced Production Score and the conversion time score The production scheduling effect score at time t is calculated using the following formula : ， Score of production scheduling effect based on time t The reward function is calculated by the following formula: : Reward Function , where It is the production scheduling effect score at time t-1.

9. A DQN-based intelligent factory job shop scheduling system, characterized in that: include: The acquisition module is used to obtain real-time data generated by smart devices and extract key workshop system features; A calculation module, used to calculate the current workshop system status based on the key workshop system characteristics ; Execution module, used to execute Using DQN network to select the optimal scheduling rule And execute to get the scheduling result; among them, execute the scheduling rules After that, the workshop system status changes from Transfer to ; Reward function calculation module, used to calculate and obtain the reward function based on the scheduling results .

10. Select the module for The tuple updates the weight of the DQN network, obtains an updated DQN network according to the updated weight of the DQN network, and uses the updated DQN network to select a better scheduling rule in future decision-making.

Images