Intelligent production scheduling dynamic scheduling method based on deep reinforcement learning

Abstract

The present invention relates to the technical field of intelligent production scheduling, and discloses an intelligent production scheduling dynamic scheduling method based on deep reinforcement learning, comprising the following methods: 1) reading information; 2) data processing; 3) building a deep reinforcement learning framework; 4) considering the start time and end time of each process; 5) splitting the order end time into each process, building a deep reinforcement learning framework, and using an Asynchronous Advantage Actor Critic (A3C) model. In addition to requiring a maximum reward value, it also requires that the entropy of each action output be maximum. In this way, the strategy is randomized, and the probability of each action output is dispersed as much as possible, rather than concentrated on one action. The A3C deep learning framework is used, and the solution speed is fast, which can support the factory's use requirements of performing intelligent production scheduling twice a day.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent production scheduling, and in particular to an intelligent production scheduling dynamic scheduling method based on deep reinforcement learning. Background Art

[0002] In the current related technologies, most of the intelligent production scheduling dynamic scheduling methods are based on optimization methods and approximate / heuristic algorithms. In recent years, many scholars have also begun to use deep reinforcement learning to solve various dynamic scheduling problems, including intelligent production scheduling dynamic scheduling problems. Optimization methods mainly include mixed integer linear programming (MILP), branch and bound method and Lagrange relaxation method; approximate / heuristic methods were originally introduced because of their small computational complexity and easy algorithm implementation, mainly including priority dispatch rule (PDR), neural network (NN) and neighborhood search method (NS). Neighborhood search method also includes taboo search (TS), genetic algorithm (GA) and simulated annealing (SA) and other approximate optimization methods that can be called meta-heuristic. Optimization methods are mainly limited by the scale of calculation. Since there are (n!)×m possible solutions to an n×m intelligent production scheduling dynamic scheduling problem, it is not feasible to use exact solution calculation methods for large-scale problems.

[0003] At present, the research on the problem of intelligent production dynamic scheduling using deep reinforcement learning models (DRL) has made great progress, and deep reinforcement learning is widely used to solve various dynamic scheduling problems. This type of model is more flexible than the traditional priority scheduling rule heuristic. The reinforcement learning environment can model random decisions and flexible problems, such as non-deterministic job re-entry, serial and parallel order between processes, multiple production lines for processes, multiple production lines for equipment, etc. However, most of the above processing methods are still in the stage of theoretical research and cannot model complex constraints for the real needs of the factory. For some random factory downtime, random processing time, order deadline and other problems, it is impossible to provide intelligent production dynamic scheduling methods that meet the real needs of the factory; in addition, the real factory needs usually take into account the advanced planning and scheduling (APS), separate the short-term plan and the medium- and long-term plan, and ensure the accuracy of the short-term plan and the rapid solution of the long-term plan. This is also an area that the current mainstream deep reinforcement learning model has not covered.

[0004] To this end, we proposed an intelligent production dynamic scheduling method based on deep reinforcement learning. Summary of the invention

[0005] The purpose of the present invention is to provide an intelligent production scheduling dynamic scheduling method based on deep reinforcement learning, which solves the problem that it is difficult to achieve real-time, autonomous and unmanned factory intelligent production scheduling in the background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: an intelligent production scheduling dynamic scheduling method based on deep reinforcement learning, comprising the following steps:

[0007] S1: Read the order status, material quantity, worker shift calendar, and production line calendar received by the factory at the current moment;

[0008] S2: Process the raw data read in and distinguish short-term plans from long-term plans based on order delivery dates and required materials.

[0009] S3: Build a deep reinforcement learning framework and input the production line, process, and capacity feature vectors into the training to obtain the target strategy network of the target intelligent agent;

[0010] S4: Consider the start and end time of each process, and the time calendar of each production line and machine;

[0011] S5: Split the order end time into each process.

[0012] Furthermore, in S1, the order data includes the quantity of required products and the product delivery deadline. Each product needs to go through several processes to be completed. There is a certain serial and parallel sequence between the processes, and switching machines or materials on the production line requires a certain equipment transfer time. There is usually a minimum waiting time or maximum waiting time constraint between serial processes.

[0013] Furthermore, in S2, short-term plans need to be processed in detail in minutes, and all processes in this part of the order must be scheduled to the production line; long-term planned orders only need to evaluate the resource conditions such as material quantity, production line, machine, production capacity, etc., give an early warning when there is a resource bottleneck, and provide a rough scheduling result in days when there is no resource bottleneck. Secondly, calculate the number of workers, machines, and production lines at each time node, and combine the worker shift calendar and production line production calendar to generate a resource timeline in minutes.

[0014] Furthermore, in S3, the Asynchronous Advantage Actor Critic (A3C) model is used. In addition to requiring the maximum reward value, it also requires the maximum entropy of each action output. This approach randomizes the strategy and makes the probability of each action output as dispersed as possible, rather than concentrating on one action.

[0015] Furthermore, in S3, the scheduling target of the deep neural network in the current target strategy network is obtained, and the feature vectors of the production line, process, and capacity status are processed and input into the classification function to obtain the selection probability corresponding to each optimization target.

[0016] Furthermore, for the constraints described in S1, the earliest start time and expected end time of the process are introduced, and all constraints are converted into the timeline of the process and production line for unified control and update.

[0017] Furthermore, S4 includes the following steps:

[0018] S41: First, for the part with pre-process requirements, the start time needs to be initialized to a larger value;

[0019] S42: When all the preceding processes are completed, the start time is updated to the maximum end time of all its preceding processes;

[0020] S43: Secondly, for situations with minimum waiting time or maximum waiting time constraints, after the preceding process is completed, the start time needs to be updated to the end time of the preceding process plus the minimum / maximum waiting time.

[0021] Furthermore, in S5, since the deep reinforcement learning model needs to repeatedly update the reward value, each order needs to be completed before the delivery date as much as possible, and considering the importance of the order, it is necessary to split the overall available time into the available time of each process according to certain rules, and design the reward function based on static attributes such as whether the order is urgent.

[0022] Furthermore, offline training includes the following steps:

[0023] S01: Generate a target strategy network for each production line as an intelligent agent;

[0024] S02: Update the reward function network used by the reward value;

[0025] S03: Store the state feature vector of each intermediate state and initialize the parameters of each network.

[0026] In each training cycle, a new training environment is randomly generated, and all agents are pre-trained offline using A3C. The optimal process-line allocation plan is generated based on the target policy network of each agent. The reward function network is generated based on the target decision states of each production line, such as its latest end time, idle time ratio, and whether each process ends later than expected. The target state value network and state feature vector of the target agent are updated through the minimum square error loss function (MSE). This process is continued until the allocation plan for all processes meets the usage requirements.

[0027] Furthermore, in the process of intelligent scheduling, the production line scheduling feature vector at the current moment is first read, and then the current executable process vector is screened according to the previous process and material conditions. At the same time, the production line and process vectors are used as inputs to train in the deep reinforcement learning agent network to obtain the process-production line assignment rules at the current moment. Then, based on whether all processes have been assigned to the production line, if not, the time needs to be updated according to the timeline movement rule, and then the agent reward network is updated according to the reward value. Finally, the production line and process are updated according to the completed tasks, and a new process-production line assignment rule is entered. If so, it continues to determine whether the maximum number of iterations has been reached or the objective function has converged. If so, the deep reinforcement learning intelligent scheduling result is output. If not, the production line scheduling feature vector is re-read until the deep reinforcement learning intelligent scheduling result is output.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The intelligent production scheduling dynamic scheduling method based on deep reinforcement learning proposed in the present invention builds a deep reinforcement learning framework and uses the Asynchronous Advantage Actor Critic (A3C) model. In addition to requiring the maximum reward value, it also requires the maximum entropy of the output of each action selection. In this way, the strategy is randomized, and the probability of each action output is as dispersed as possible instead of being concentrated on one action. The deep learning framework of A3C is used, and the solution speed is fast, which can support the factory's use requirements of intelligent production scheduling twice a day.

[0030] 2. The intelligent production scheduling dynamic scheduling method based on deep reinforcement learning proposed in the present invention takes into account the start time and end time of each process, the time calendar of each production line and machine, introduces the earliest start time and expected end time of the process, and converts all constraints into the time axis of the process and production line for unified control and update. First, for the part with predecessor process requirements, it is necessary to initialize the start time to a larger value. When all predecessor processes are completed, the start time is updated to the maximum end time of all its predecessor processes; secondly, for the case with minimum waiting time or maximum waiting time constraints, after the predecessor process is completed, the start time needs to be updated to the end time of the predecessor process plus the minimum / maximum waiting time, so that the actual needs of the factory can be fully considered. There is a certain serial and parallel order between the processes. Switching machines or materials on the production line requires a certain equipment transfer time. There is usually a minimum waiting time or maximum waiting time constraint between serial processes. Each order has an order deadline and there is a priority.

[0031] 3. The intelligent production scheduling dynamic scheduling method based on deep reinforcement learning proposed in the present invention processes the raw data read in, firstly distinguishes short-term plans from long-term plans according to the order delivery date and the required materials, wherein the short-term plans need to be processed in minutes, and all the processes in this part of the orders are scheduled to the production line; the orders for long-term plans only need to evaluate the resource conditions such as material quantity, production line, machine, production capacity, etc., give an early warning when there is a resource bottleneck, and provide a rough scheduling result in days when there is no resource bottleneck. Secondly, the number of workers, machines, and production lines at each time node is calculated, and the resource timeline in minutes is generated by combining the worker shift calendar and the production line production calendar. Orders can be distinguished according to advanced planning and scheduling (APS), and short-term plans and medium- and long-term plans can be modeled separately, while ensuring the accuracy of short-term plans and the rapid solution of long-term plans, which can greatly reduce the time required to solve the factory intelligent scheduling problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0033] Figure 1 This is an overall flow chart of the intelligent production scheduling dynamic scheduling method based on deep reinforcement learning of the present invention;

[0034] Figure 2 This is a flowchart of intelligent production scheduling of the intelligent production scheduling dynamic scheduling method based on deep reinforcement learning of the present invention;

[0035] Figure 3 This is a flow chart of offline training of the intelligent production scheduling dynamic scheduling method based on deep reinforcement learning of the present invention;

[0036] Figure 4 This is a flow chart of the time control method of the intelligent production scheduling dynamic scheduling method based on deep reinforcement learning in the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0040] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to specific circumstances.

[0041] In addition, the term "plurality" shall mean two or more.

[0042] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] See also Figure 1 , the intelligent production scheduling dynamic scheduling method based on deep reinforcement learning includes the following steps:

[0044] S1: Read the order status, material quantity, worker shift calendar, and production line calendar received by the factory at the current moment;

[0045] S2: Process the raw data read in and distinguish short-term plans from long-term plans based on order delivery dates and required materials.

[0046] S3: Build a deep reinforcement learning framework and input the production line, process, and capacity feature vectors into the training to obtain the target strategy network of the target intelligent agent;

[0047] S4: Consider the start and end time of each process, and the time calendar of each production line and machine;

[0048] S5: Split the order end time into each process.

[0049] In S1, the order data includes the quantity of required products and the product delivery deadline. Each product needs to go through several processes to be completed. There is a certain serial and parallel sequence between the processes. Switching machines or materials on the production line requires a certain equipment transfer time. There is usually a minimum waiting time or maximum waiting time constraint between serial processes.

[0050] In S2, short-term plans need to be processed in detail in minutes, and all processes in this part of the order must be scheduled to the production line; long-term planned orders only need to evaluate resource conditions such as material quantity, production line, machine, and production capacity, give an early warning when there is a resource bottleneck, and provide a rough production scheduling result in days when there is no resource bottleneck. Secondly, calculate the number of workers, machines, and production lines at each time node, and combine the worker shift calendar and production line production calendar to generate a resource timeline in minutes.

[0051] In S3, the Asynchronous Advantage Actor Critic (A3C) model is used. In addition to requiring the maximum reward value, the entropy of each action output is also required to be maximized. This approach allows the strategy to be randomized, and the probability of each action output is as dispersed as possible, rather than concentrated on one action. The scheduling target of the deep neural network in the current target strategy network is obtained, and the feature vectors of the production line, process, and capacity status are processed and input into the classification function to obtain the selection probability corresponding to each optimization target.

[0052] See also Figure 1 and Figure 4 For the constraints described in S1, the earliest start time and expected end time of the process are introduced, and all constraints are converted into the time axis of the process and production line for unified control and update.

[0053] The following steps are included in S4:

[0054] S41: First, for the part with pre-process requirements, the start time needs to be initialized to a larger value;

[0055] S42: When all the preceding processes are completed, the start time is updated to the maximum end time of all its preceding processes;

[0056] S43: Secondly, for situations with minimum waiting time or maximum waiting time constraints, after the preceding process is completed, the start time needs to be updated to the end time of the preceding process plus the minimum / maximum waiting time.

[0057] In S5, since the deep reinforcement learning model needs to repeatedly update the reward value, each order needs to be completed before the delivery date as much as possible, and considering the importance of the order, it is necessary to split the overall available time into the available time of each process according to certain rules, and design the reward function based on static attributes such as whether the order is urgent.

[0058] See also Figure 1 and Figure 3 , offline training includes the following steps:

[0059] S01: Generate a target strategy network for each production line as an intelligent agent;

[0060] S02: Update the reward function network used by the reward value;

[0061] S03: Store the state feature vector of each intermediate state and initialize the parameters of each network.

[0062] In each training cycle, a new training environment is randomly generated, and all agents are pre-trained offline using A3C. The optimal process-line allocation plan is generated based on the target policy network of each agent. The reward function network is generated based on the target decision states of each production line, such as its latest end time, idle time ratio, and whether each process ends later than expected. The target state value network and state feature vector of the target agent are updated through the minimum square error loss function (MSE). This process is continued until the allocation plan for all processes meets the usage requirements.

[0063] See also Figure 2 In the process of intelligent scheduling, the current production line scheduling feature vector is first read, and then the current executable process vector is screened according to the previous process and material conditions. At the same time, the production line and process vectors are used as inputs to train in the deep reinforcement learning agent network to obtain the process-production line assignment rule at the current moment. Then, according to whether all processes have been assigned to the production line, if not, the time needs to be updated according to the timeline movement rule, and then the agent reward network is updated according to the reward value. Finally, the production line and process are updated according to the completed tasks, and the new process-production line assignment rule is entered. If yes, it continues to determine whether the iteration has reached the maximum number of times or the objective function has converged. If so, the deep reinforcement learning intelligent scheduling result is output. If not, the production line scheduling feature vector is re-read until the deep reinforcement learning intelligent scheduling result is output.

[0064] In summary: the intelligent production scheduling dynamic scheduling method based on deep reinforcement learning provided by the present invention includes: S1: reading the order situation, material quantity, worker shift calendar, and production line production calendar received by the factory at the current moment; S2: processing the read raw data, and distinguishing short-term plans from long-term plans according to the order delivery date and the required material situation; S3: building a deep reinforcement learning framework, and inputting the production line, process, and capacity feature vectors into the training to obtain the target strategy network of the target intelligent agent; S4: considering the start time and end time of each process, and the time calendar of each production line and machine; S5: splitting the order end time into each process, building a deep reinforcement learning framework, and using the Asynchronous Advantage Actor The Critic (A3C) model, in addition to requiring the maximum reward value, also requires the maximum entropy of each action output. This approach allows the strategy to be randomized, and the probability of each output action is as dispersed as possible, rather than concentrated on one action. Using the A3C deep learning framework, the solution speed is fast, which can support the factory's use requirements of twice-daily intelligent production scheduling. Secondly, considering the start time and end time of each process, the time calendar of each production line and machine, the earliest start time and expected end time of the process are introduced, and all constraints are converted into the time axis of the process and production line for unified control and update. First, for the part with the requirements of the predecessor process, the start time needs to be initialized to a larger value. When all the predecessor processes are completed, the start time is updated to the maximum end time of all its predecessor processes; secondly, for the case with minimum waiting time or maximum waiting time constraints, after the predecessor process is completed, the start time needs to be updated to the end time of the predecessor process plus the minimum / maximum waiting time, so that the real needs of the factory can be fully considered, and there is a certain series and parallel relationship between the processes. The order of the production line is determined by the order of the short-term plan and the long-term plan. The short-term plan needs to be processed in minutes, and all the processes in this part of the order are scheduled to the production line. The long-term plan only needs to evaluate the resource conditions such as the quantity of materials, production lines, machines, and production capacity. When there is a resource bottleneck, an early warning is given. When there is no resource bottleneck, a rough scheduling result in days is provided. Secondly, the number of workers, machines, and production lines at each time node is calculated, and the resource timeline in minutes is generated by combining the worker shift calendar and the production line production calendar. Orders can be distinguished according to the advanced planning and scheduling (APS), and short-term plans and medium- and long-term plans can be modeled separately. At the same time, the accuracy of short-term plans and the rapid solution of long-term plans can be guaranteed, which can greatly reduce the time required to solve the factory intelligent scheduling problem.

[0065] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0066] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. Intelligent production scheduling dynamic scheduling method based on deep reinforcement learning, characterized by: The following steps are involved: S1: Read the order status, material quantity, worker shift calendar, and production line calendar received by the factory at the current moment; S2: Process the raw data read in and distinguish short-term plans from long-term plans based on order delivery dates and required materials. S3: Build a deep reinforcement learning framework and input the production line, process, and capacity feature vectors into the training to obtain the target strategy network of the target intelligent agent; S4: Consider the start and end time of each process, and the time calendar of each production line and machine; S5: Split the order end time into each process; In S1, the order data includes the quantity of required products and the product delivery deadline. Each product needs to go through several processes to be completed. There is a serial and parallel sequence between the processes. In addition, switching machines or materials on the production line requires equipment transfer time. There is a minimum waiting time or maximum waiting time constraint between the serial processes. In S2, short-term plans need to be processed in minutes, and all processes in this part of the order should be scheduled to the production line; long-term planned orders only need to evaluate the quantity of materials, production lines, machines, and production capacity resources, give warnings when there are resource bottlenecks, and provide rough production scheduling results in days when there are no resource bottlenecks. Secondly, calculate the number of workers, machines, and production lines at each time node, and combine the worker shift calendar and production line production calendar to generate a resource timeline in minutes; In S3, the Asynchronous Advantage Actor Critic (A3C) model is used. In addition to requiring the maximum reward value, the entropy of each action output is also required to be maximized. This approach randomizes the strategy and disperses the probability of each action output instead of concentrating on one action. The scheduling target of the deep neural network in the current target strategy network is obtained, and the feature vectors of the production line, process, and capacity status are processed and input into the classification function to obtain the selection probability corresponding to each optimization target; For the constraints described in S1, the earliest start time and expected end time of the process are introduced, and all constraints are converted into the time axis of the process and production line for unified control and update; The following steps are included in S4: S41: First, for the part with pre-process requirements, the start time needs to be initialized to a larger value; S42: When all the preceding processes are completed, the start time is updated to the maximum end time of all its preceding processes; S43: Secondly, for the case where there is a minimum waiting time or maximum waiting time constraint, after the preceding process is completed, the start time needs to be updated to the end time of the preceding process plus the minimum / maximum waiting time; In S5, since the deep reinforcement learning model needs to repeatedly update the reward value, each order needs to be completed before the delivery date, and considering the importance of the order, it is necessary to split the overall available time into the available time of each process according to the rules, and design the reward function according to the static attribute of whether the order is urgent; The specific steps include: S01: Generate a target strategy network for each production line as an intelligent agent; S02: Update the reward function network used by the reward value; S03: Store the state feature vector of each intermediate state and initialize the parameters of each network; In each training cycle, a new training environment is randomly generated, and all agents are pre-trained offline using A3C. The optimal process-production line allocation plan is generated based on the target policy network of each agent. The reward function network is generated based on the latest end time of each production line, the proportion of idle time, and whether the end time of each process is later than the expected target decision state. The target state value network and state feature vector of the target agent are updated through the minimum square error loss function (MSE). This process is repeated until the allocation plan for all processes finally meets the usage requirements.

2. The intelligent production scheduling dynamic scheduling method based on deep reinforcement learning according to claim 1, characterized in that: In the process of intelligent scheduling, the current production line scheduling feature vector is first read, and then the current executable process vector is screened according to the previous process and material conditions. At the same time, the production line and process vectors are used as input to train in the deep reinforcement learning agent network to obtain the process-production line assignment rule at the current moment. Then, based on whether all processes have been assigned to the production line, if not, the time needs to be updated according to the timeline movement rule, and then the agent reward network is updated according to the reward value. Finally, the production line and process are updated according to the completed tasks, and a new process-production line assignment rule is entered. If so, it continues to determine whether the maximum number of iterations has been reached or the objective function has converged. If so, the deep reinforcement learning intelligent scheduling result is output. If not, the production line scheduling feature vector is re-read until the deep reinforcement learning intelligent scheduling result is output.

Images