Multi-target distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning

The distributed workshop scheduling problem is decomposed through the layered selective deep reinforcement learning method, and the multi-layer agent and deep reinforcement learning algorithm are designed, which solves the complexity of the traditional scheduling method that cannot handle the multi-workshop scheduling problem, and achieves the optimal solution of multi-objectives and quick decision making.

CN118917567APending Publication Date: 2024-11-08NANJING TECH UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410290398.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-03-14
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional single-workshop scheduling methods cannot effectively deal with complex scheduling problems in distributed workshops, and cannot achieve overall optimal resource allocation and operation schedule, especially when facing multiple workshops, resources and constraints.

Method used

Using a method based on layered selection deep reinforcement learning, the distributed flexible work workshop scheduling problem is broken down into three-layer structures, the design process selection agent, the factory selection agent and the machine selection agent, and the depth is used. Reinforcement learning algorithms are trained and scheduling decisions are optimized through experience playback pools.

Benefits of technology

The stratified reinforcement learning method can reduce problem complexity, improve system response speed and flexibility, achieve multi-objective optimal solutions, and adapt to distributed workshop scheduling problems of different scales and complexities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917567A_ABST
    Figure CN118917567A_ABST
Patent Text Reader

Abstract

The invention provides a multi-target distributed flexible job shop scheduling method based on hierarchical selection type deep reinforcement learning, and the method is characterized in that the method mainly comprises the steps: simplifying a distributed flexible job shop scheduling problem; creating a process selection intelligent agent, a factory selection intelligent agent and a machine selection intelligent agent which are respectively responsible for determining a to-be-optimized process, selecting a factory for a workpiece and selecting a processing machine for the process, and respectively arranging the three types of intelligent agents in an upper structure, a middle structure and a lower structure; designing a scheduling rule of each agent in combination with the task of the agent; training a process selection agent, a factory selection agent and a machine selection agent by using a deep reinforcement learning algorithm; and training the intelligent agent to complete the multi-objective optimization problem of distributed flexible job shop scheduling.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of workshop scheduling, and utilizes hierarchical reinforcement learning theory to deal with multi-objective distributed flexible job workshop scheduling problems. Specifically, the present invention relates to a multi-objective distributed flexible job workshop scheduling method based on hierarchical selective deep reinforcement learning. Background Art

[0002] The continuous expansion of enterprise scale has made the distributed workshop production mode a norm, and at the same time has greatly increased the complexity of workshop scheduling problems. The traditional single-workshop scheduling method has certain limitations in dealing with distributed workshop scheduling problems. The single-workshop scheduling method is designed to optimize the production scheduling and resource utilization within a single workshop. It can usually only consider the tasks and resources within the workshop, and cannot obtain the real status and needs of other workshops in a timely manner, which makes it impossible to achieve the overall optimal resource allocation and job scheduling; the traditional single-workshop scheduling method cannot effectively handle the various constraints that exist, such as order priority, production capacity differences, material dependence, etc. In order to overcome these problems, it is necessary to design and apply new methods and algorithms suitable for distributed workshop scheduling. Summary of the invention

[0003] The purpose of this invention is to propose a multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning to solve the distributed flexible job shop scheduling problem and meet actual production needs.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning includes the following steps

[0006] S1. The distributed flexible job shop scheduling problem is simplified by adopting the upper, middle and lower three-layer structure, and the process selection agent, factory selection agent and machine selection agent are designed respectively and correspond to the three-layer structure;

[0007] S2. Determine the deep reinforcement learning algorithm used in each of the upper, middle and lower layers to train the process selection agent, the factory selection agent and the machine selection agent;

[0008] S3, the workpiece selection agent selects the workpiece according to the current state of the workpiece Selecting a dispatch rule in the action space Output the process of the workpiece to be optimized as the optimization target of the factory selection agent and the machine selection agent, and obtain the workpiece state at the next moment And based on the change of the previous and next states, the reward r is obtained according to the reward function tu ,Will Stored in the experience replay pool for training the upper-level deep reinforcement learning algorithm;

[0009] S4: The factory selection agent determines whether the workpiece corresponding to the process to be optimized at the current moment has been assigned to a factory. If not, the factory selection agent determines whether the workpiece corresponding to the process to be optimized at the current moment has been assigned to a factory. Select Scheduling Rules Select a factory for the workpiece and get the status of each factory at the next moment and reward r t m ,collect After all process scheduling is completed, reschedule and t m After correction, Stored in the experience replay pool for training mid-level deep reinforcement learning algorithms; if already assigned, skip the factory selection agent and hand it over directly to the machine selection agent for processing;

[0010] S5. Set the same number of machine selection agents as the number of factories to solve the scheduling problem of each factory respectively. The designated machine selection agent is based on the factory status corresponding to the current time. Select Scheduling Rules Assign processing machines to workpiece processes and obtain the factory status at the next moment and reward r t d ,Will Stored in the experience replay pool for training the underlying deep reinforcement learning algorithm.

[0011] The state space of the process selection agent includes: the average completion rate of workpieces, the standard deviation of the completion rate, the estimated delay rate and the actual delay rate; the action space includes: selecting workpiece processes with low completion levels, selecting workpiece processes with high urgency, and selecting workpiece processes with long remaining processing time.

[0012] The state space of the factory selection agent includes: the average utilization rate of all factory machines and the standard deviation of utilization rate; the action space includes: selecting the factory with the shortest workpiece processing time, selecting the factory with the lowest average machine utilization rate, and selecting the currently idle factory.

[0013] The state space of the machine selection agent includes: the average utilization rate of factory machines and the standard deviation of utilization rate; the action space includes: selecting the machine with the shortest processing time, selecting the machine with the lowest utilization rate, and selecting the idle machine in the factory.

[0014] The deep reinforcement learning algorithm of the upper, middle and lower layers is the same, using the DDQN algorithm, using a neural network Q wThe output of the action with the greatest value is selected But when using the value of the action, another neural network Calculate the value of the action to avoid overestimation of the final value of the action. The optimization goal is:

[0015]

[0016] Among them, r t Represents the agent from state s t Transfer to t+1 The reward is, a represents the action performed by the agent, Representing a neural network In status t+1 and actions The Q value obtained under , γ represents the discount rate, and done represents whether the current round is completed;

[0017] To make the algorithm stable, the target network It will not be updated at every step, but the training network Q w The parameters of the target network are updated at each step in the training, and they are synchronized with the training network every C steps;

[0018] The loss function adopts the mean square error loss function, as shown below;

[0019] L(w)=E[(y t -Q w (s t ,a)) 2 ] (2)

[0020] Among them, y t represents the optimization objective, Q w (s t ,a) represents the neural network Q w In status t+1 And the Q value obtained under action a.

[0021] The search strategy in deep reinforcement learning with three layers of structure: upper, middle and lower is:

[0022] In order to balance the relationship between development (maximizing the expected return at the current moment) and exploration (maximizing the total return from a long-term perspective), the ε greedy strategy is selected. In the initial stage or in the early stage of training, a higher ε value is set to more comprehensively understand the environment and gain more experience. As the number of action selections increases, the agent gradually gains enough experience and understanding of the environment. At this time, the value of ε is gradually reduced, and the known optimal action is used to obtain the maximum long-term return. ε is defined as follows:

[0023] ε=max{0.1,ε-0.0001×n} (3)

[0024] Where n is the number of times the agent currently performs action selection.

[0025] Experience replay is used in deep reinforcement learning with a three-layer structure of upper, middle and lower layers. By setting up an experience replay pool, the <current state, action, reward, next state> sampled from the environment is stored in the pool. When training the deep learning algorithm, some data is randomly sampled from it for training to improve sample utilization efficiency.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The hierarchical reinforcement learning method adopted in the present invention can decompose complex problems and decompose the entire problem into multiple subtasks, making the problem easier to understand and solve. Distributed workshop scheduling problems are usually large in scale and complexity, involving multiple workshops, resources and constraints. Hierarchical reinforcement learning can decompose the problem layer by layer, and each subtask can be independently modeled and optimized, reducing the complexity of the problem.

[0028] The present invention has good scalability and can adapt to distributed workshop scheduling problems of different scales and complexities; it allows rapid decision-making and adjustment at various levels, thereby improving the response speed and flexibility of the system; and it can optimize various objectives at different decision-making levels, thereby achieving the optimal solution of multiple objectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the operation structure of the method for multi-objective dynamic flexible job shop scheduling on energy consumption based on deep reinforcement learning of the present invention;

[0030] Figure 2 This is the algorithm framework diagram. DETAILED DESCRIPTION

[0031] In order to clarify the technical problems, technical solutions, implementation processes and performance demonstrations, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are for explanation only. The present invention is not intended to limit the present invention. Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0032] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0033] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0034] Example 1

[0035] like Figure 1-Figure 2 As shown, a multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning is characterized by comprising the following steps:

[0036] S1. The distributed flexible job shop scheduling problem is simplified by adopting the upper, middle and lower three-layer structure, and the process selection agent, factory selection agent and machine selection agent are designed respectively and correspond to the three-layer structure;

[0037] S2. Determine the deep reinforcement learning algorithm used in each of the upper, middle and lower layers to train the process selection agent, the factory selection agent and the machine selection agent;

[0038] S3, the workpiece selection agent selects the workpiece according to the current state s t u Selecting a dispatch rule in the action space Output the process of the workpiece to be optimized as the optimization target of the factory selection agent and the machine selection agent, and obtain the workpiece state at the next moment And based on the change of the previous and next states, the reward r is obtained according to the reward function t u ,Will Stored in the experience replay pool for training the upper-level deep reinforcement learning algorithm;

[0039] S4: The factory selection agent determines whether the workpiece corresponding to the process to be optimized at the current moment has been assigned to a factory. If not, the factory selection agent determines whether the workpiece corresponding to the process to be optimized at the current moment has been assigned to a factory. Select Scheduling Rules Select a factory for the workpiece and get the status of each factory at the next moment and reward r t m ,collect After all process scheduling is completed, reschedule and t m After correction, Stored in the experience replay pool for training mid-level deep reinforcement learning algorithms; if already assigned, skip the agent and hand it over directly to the machine selection agent for processing;

[0040] S5. Set the same number of machine selection agents as the number of factories to solve the scheduling problem of each factory respectively. The designated machine selection agent is based on the factory status corresponding to the current time. Select Scheduling Rules Assign processing machines to workpiece processes and obtain the factory status at the next moment and reward r t d ,Will Stored in the experience replay pool for training the underlying deep reinforcement learning algorithm.

[0041] In order to describe the scheduling problem, the following symbols are defined, as shown in Table 1:

[0042] Table 1 Mathematical symbols and definitions for shop scheduling

[0043]

[0044]

[0045] Taking minimizing the completion time C, maximizing the machine utilization rate U, and minimizing the delay time as the optimization goal T, a multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning is constructed. The optimization model is:

[0046] minC=min(max{C i}) (4)

[0047]

[0048] In formula (5), F q Machine utilization in Utilization rate When O i,j exist If processing is performed, X=1; otherwise, X=0.

[0049] The state space of the process selection agent includes: the average completion rate of the workpiece, the standard deviation of the completion rate, the estimated delay rate and the actual delay rate, where the average completion rate of the workpiece Completion rate standard deviation Among them J i Completion rate Estimated delay rate T est The calculation process is shown in Table 2:

[0050] Table 2 Estimated delay rate calculation process

[0051]

[0052]

[0053] Actual delay rate T act The calculation process is shown in Table 3:

[0054] Table 3 Actual delay rate calculation process

[0055]

[0056] C ave ,C std ,T est ,T act Constructing the state space

[0057] The action space of the process selection agent includes: selecting the process of the workpiece with low completion degree, selecting the process of the workpiece with high urgency, and selecting the process of the workpiece with long remaining processing time. The urgency is the known initial condition, workpiece J i Remaining process time The calculation process is shown in the following table: If J i Allocated Factory F q , Among them, i,j In F q Average processing time otherwise J i In F q Processing time

[0058] The reward calculation of the process selection agent is shown in Table 4:

[0059] Table 4 Reward calculation process of process selection agent

[0060]

[0061] The state space of the factory selection agent includes: the average utilization rate of all factory machines and the standard deviation of utilization rate, where the average utilization rate of all factory machines is Utilization standard deviation U ave ,U std Constructing the state space

[0062] The action space of the factory selection agent includes: selecting the factory with the shortest workpiece processing time, selecting the factory with the lowest average machine utilization, and selecting the factory that is currently idle. i In F q The processing time is The average machine utilization rate is U ave .

[0063] The reward calculation of the factory selection agent is shown in Table 5:

[0064] Table 5 Reward calculation process of factory selection agent

[0065]

[0066] The state space of the machine selection agent includes: the average utilization rate of the factory machines and the standard deviation of the utilization rate, where the average utilization rate of the factory machines is U q The standard deviation of utilization is

[0067] The action space of the machine selection agent includes: selecting the machine with the shortest processing time, selecting the machine with the lowest utilization rate, and selecting the idle machine in the factory, where the machine utilization rate is

[0068] The reward calculation for the machine-selected agent is shown in Table 6:

[0069] Table 6 Reward calculation process for machine selection agents

[0070]

[0071]

[0072] The deep reinforcement learning algorithms of the upper, middle and lower layers are the same. Using the DDQN algorithm, the deep learning algorithms all contain an input layer, four hidden layers and an output layer. The activation function used is the Relu function, the loss function uses the mean square error loss function, and the optimizer uses Adam. The input layer of the upper structure contains 4 neuron nodes, each hidden layer contains 12 neuron nodes, and the output layer contains 3 neuron nodes. The middle and lower layers have the same structure, the input layer contains 2 neuron nodes, each hidden layer contains 10 neuron nodes, and the output layer contains 3 neuron nodes. The input layer of the structure contains 2 neuron nodes, each hidden layer contains 10 neuron nodes, and the output layer contains 3 neuron nodes.

[0073] In terms of experience replay, the workpiece selection agent and the machine selection agent use the same replay method, storing a set of experiences consisting of their current state, selected action, next state, and reward in the experience replay pool for training deep reinforcement learning algorithms.

[0074] Based on the experience replay of the factory selection agent, a rescheduling correction algorithm is designed to optimize the factory selection: determine the specific workpiece J spec For its processing factory F q The impact on machine utilization, set the following algorithm:

[0075] Table 7 Rescheduling correction algorithm flow

[0076]

[0077]

[0078] After rescheduling, the calculation The sum of the processing time of each process WT m

[0079]

[0080] After using the algorithm to modify the rewards in the factory selection agent's experience, the experience is stored in the experience replay pool for training the deep reinforcement learning algorithm.

[0081] The hierarchical reinforcement learning method adopted in the present invention can decompose complex problems and decompose the entire problem into multiple subtasks, making the problem easier to understand and solve. Distributed workshop scheduling problems are usually large in scale and complexity, involving multiple workshops, resources and constraints. Hierarchical reinforcement learning can decompose the problem layer by layer, and each subtask can be independently modeled and optimized, reducing the complexity of the problem.

[0082] The present invention has good scalability and can adapt to distributed workshop scheduling problems of different scales and complexities; it allows rapid decision-making and adjustment at various levels, thereby improving the response speed and flexibility of the system; and it can optimize various objectives at different decision-making levels, thereby achieving the optimal solution of multiple objectives.

[0083] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning, characterized in that: The following steps are included S1. The distributed flexible job shop scheduling problem is simplified by adopting the upper, middle and lower three-layer structure, and the process selection agent, factory selection agent and machine selection agent are designed respectively and correspond to the three-layer structure; S2. Determine the deep reinforcement learning algorithm used in each of the upper, middle and lower layers to train the process selection agent, the factory selection agent and the machine selection agent; S3, the workpiece selection agent selects the workpiece according to the current state of the workpiece Selecting a dispatch rule in the action space Output the process of the workpiece to be optimized as the optimization target of the factory selection agent and the machine selection agent, and obtain the workpiece state at the next moment And based on the change of the previous and next states, the reward r is obtained according to the reward function t u ,Will Stored in the experience replay pool for training the upper-level deep reinforcement learning algorithm; S4: The factory selection agent determines whether the workpiece corresponding to the process to be optimized at the current moment has been assigned to a factory. If not, the factory selection agent determines whether the workpiece corresponding to the process to be optimized at the current moment has been assigned to a factory. Select Scheduling Rules Select a factory for the workpiece and get the status of each factory at the next moment and reward r t m ,collect After all process scheduling is completed, reschedule and t m After correction, Stored in the experience replay pool for training mid-level deep reinforcement learning algorithms; if already assigned, skip the factory selection agent and hand it over directly to the machine selection agent for processing; S5. Set the same number of machine selection agents as the number of factories to solve the scheduling problem of each factory respectively. The designated machine selection agent is based on the factory status corresponding to the current time. Select Scheduling Rules Assign processing machines to workpiece processes and obtain the factory status at the next moment and reward r t d ,Will Stored in the experience replay pool for training the underlying deep reinforcement learning algorithm.

2. The multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning according to claim 1 is characterized in that: The state space of the process selection agent includes: the average completion rate of workpieces, the standard deviation of the completion rate, the estimated delay rate and the actual delay rate; the action space includes: selecting workpiece processes with low completion levels, selecting workpiece processes with high urgency, and selecting workpiece processes with long remaining processing time.

3. The multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning according to claim 1 is characterized in that: The state space of the factory selection agent includes: the average utilization rate of all factory machines and the standard deviation of utilization rate; the action space includes: selecting the factory with the shortest workpiece processing time, selecting the factory with the lowest average machine utilization rate, and selecting the currently idle factory.

4. The multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning according to claim 1 is characterized in that: The state space of the machine selection agent includes: the average utilization rate of factory machines and the standard deviation of utilization rate; the action space includes: selecting the machine with the shortest processing time, selecting the machine with the lowest utilization rate, and selecting the idle machine in the factory.

5. The multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning according to claim 1 is characterized in that: The deep reinforcement learning algorithm of the upper, middle and lower layers is the same, using the DDQN algorithm, using a neural network Q w The output of the action with the greatest value is selected But when using the value of the action, another neural network Calculate the value of the action to avoid overestimation of the final value of the action. The optimization goal is: Among them, r t Represents the agent from state s t Transfer to t+1 The reward is, a represents the action performed by the agent, Representing a neural network In status t+1 and actions The Q value obtained under , γ represents the discount rate, and done represents whether the current round is completed; To make the algorithm stable, the target network It will not be updated at every step, but the training network Q w The parameters of the target network are updated at each step in the training, and they are synchronized with the training network every C steps; The loss function adopts the mean square error loss function, as shown below; L(w)=E[(y t -Q w (s t ,a)) 2 ] (2) Among them, y t represents the optimization objective, Q w (s t ,a) represents the neural network Q w In status t+1 And the Q value obtained under action a.

6. The multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning according to claim 1 is characterized in that: The search strategy in deep reinforcement learning with three layers of structure: upper, middle and lower is: In order to balance the relationship between development (maximizing the expected return at the current moment) and exploration (maximizing the total return from a long-term perspective), the ε greedy strategy is selected. In the initial stage or in the early stage of training, a higher ε value is set to more comprehensively understand the environment and gain more experience. As the number of action selections increases, the agent gradually gains enough experience and understanding of the environment. At this time, the value of ε is gradually reduced, and the known optimal action is used to obtain the maximum long-term return. ε is defined as follows: ε=max{0.1,ε-0.0001×n} (3) Where n is the number of times the agent currently performs action selection.

7. The multi-objective distributed flexible job shop scheduling method based on hierarchical selective deep reinforcement learning according to claim 1 is characterized in that: Experience replay is used in deep reinforcement learning with a three-layer structure of upper, middle and lower layers. By setting up an experience replay pool, the <current state, action, reward, next state> sampled from the environment is stored in the pool. When training the deep learning algorithm, some data is randomly sampled from it for training to improve sample utilization efficiency.

Citation Information

Cited By

  • Industrial green micro-grid multi-target flexible job shop dynamic intelligent scheduling method

    CN118966715A

  • Kit production workshop scheduling method and system based on hierarchical reinforcement learning

    CN119247907A

  • Dynamic flexible workshop multi-target online rescheduling method based on hierarchical MADDQN drive

    CN119575905A