A Distributed Flexible Job Shop Scheduling Method and System Based on Dual Deep Reinforcement Learning and Multi-layer Agents

Abstract

The present application relates to the field of intelligent manufacturing technology, and in particular to a distributed flexible job shop scheduling method and system based on dual-deep reinforcement learning and multi-layer intelligent agents. By introducing a multi-level intelligent agent scheduling framework, the top-level, middle-level and bottom-level intelligent agents work together and make hierarchical decisions to decompose global complex problems into multiple local problems; in addition, different scheduling tasks of the distributed flexible job shop are decentralizedly executed by intelligent agents at each level, without the need for a central coordinator to manage each decision. When the environment changes locally, each level can adapt quickly and make the best decision based on its specific context, while still serving the overall goal of the entire system. It aims to solve the problem of how to improve the scheduling accuracy of distributed flexible job shops.

Description

Technical Field

[0001] This application relates to the field of intelligent manufacturing technology, and in particular to a distributed flexible job shop scheduling method and system based on dual deep reinforcement learning and multi-layer intelligent agents. Background Technology

[0002] In recent years, with the rise of multi-factory production models, distributed flexible manufacturing has gradually become an important research direction in the manufacturing industry. Flexible job shop scheduling, as a core technology in flexible manufacturing systems, can help companies maintain production efficiency and profitability in a dynamic market environment through flexible scheduling and resource optimization. Distributed manufacturing can not only effectively allocate resources across different factories, but also reduce manufacturing costs and management risks by optimizing the rational allocation of production tasks, thus helping companies achieve higher production efficiency. Therefore, production scheduling in a distributed manufacturing environment is currently one of the hot research topics in the field of intelligent manufacturing.

[0003] Among the relevant technical solutions, traditional distributed flexible job shop scheduling schemes mainly focus on mathematical programming and heuristic algorithms, which have shortcomings such as limited computing power and limitation to small-scale problems. Currently, some researchers have combined deep reinforcement learning (DRL) with job shop scheduling, using the powerful computing power of DRL to optimize complex job shop scheduling problems.

[0004] However, most existing research on DRL and job shop scheduling focuses on single-factory or single-agent environments. In distributed environments, however, the collaborative scheduling, resource allocation, and task optimization among multiple factories or equipment involve more complex interactions and decision-making processes, which limits the application of single-agent DRL methods in such scenarios. A single agent cannot fully grasp the global information in a distributed environment, making it difficult to simultaneously optimize resource scheduling across multiple factories. Furthermore, when handling cross-factory task collaboration, a single agent lacks effective coordination mechanisms, easily leading to resource waste or unbalanced scheduling. Its performance in distributed scheduling problems is also limited, failing to adequately address the multi-objective and large-scale scheduling needs of distributed environments.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents, aiming to solve the problem of how to improve the scheduling accuracy of distributed flexible job shops.

[0007] To achieve the above objectives, this application provides a distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layered agents, applied to a distributed flexible job shop scheduling system. The system includes: a top-level agent for determining the target workpiece with the highest processing priority based on workpiece information; an intermediate-level agent for determining the target processing plant corresponding to the target workpiece based on plant information; and a bottom-level agent for determining the target processing machine corresponding to the target workpiece based on processing machine information in the target processing plant. The top-level agent, the intermediate-level agent, and the bottom-level agent all employ a dual deep reinforcement learning algorithm for parameter optimization during centralized training. The method includes the following steps:

[0008] S1, the top-level agent obtains the current top-level state characteristics at time t from the scheduling environment, determines the workpiece decision information based on the current state characteristics, and transmits the workpiece decision information to the intermediate layer agent and the bottom layer agent;

[0009] S2, the intermediate layer agent obtains the current intermediate layer state characteristics from the scheduling environment, determines the factory decision information based on the workpiece decision information and the current intermediate layer state characteristics, and transmits the factory decision information to the bottom layer agent;

[0010] S3, the underlying intelligent agent obtains the current underlying state characteristics from the scheduling environment, and determines the equipment decision information based on the workpiece decision information, the factory decision information, and the current underlying state characteristics;

[0011] S4. Based on the workpiece decision information, the factory decision information, and the equipment decision information, determine joint decision information, execute a scheduling strategy based on the joint decision information, and perform a state transition on the scheduling environment so that the top-level agent can obtain the top-level state characteristics and rewards at time t+1 and feed them back to the top-level agent.

[0012] S5, repeat steps S1-S4 until each workpiece is processed.

[0013] Optionally, the top-level state features in the top-level agent include:

[0014] Completed processing features CP i (s t If all workpieces have been processed by time t, then CP i (s t ) = 1, otherwise CP i (s t )=0;

[0015] Workpiece processing average time characteristics (TPT)i (s t ), representing the average processing time of workpiece i at time t across all available machines, is mathematically expressed as:

[0016]

[0017] Pre-process features OCT i (s t ), where, if at time t, the completion time of the preceding process of the workpiece i's machinable process is OCT. i (s t If ) = 1, then OCT is used if there is no preceding process. i (s t )=0;

[0018] Processing time characteristics PD i (s t The processing time feature is characterized by the processing time of workpiece i at time t.

[0019] Optionally, the intermediate layer state features in the intermediate layer agent include:

[0020] Factory load characteristics FPL f (s t ), which represents the load of factory f at time t;

[0021] Pre-scheduled features SPD f (s t ), which represents the predicted maximum completion time obtained by the factory f performing a complete pre-schedule of the workpiece i selected by the top-level decision action at time t, wherein the machine with the shortest processing time for each process is selected for pre-schedule of the operation without changing the existing scheduling scheme of the factory.

[0022] Optionally, the underlying state features of the underlying intelligent agent include:

[0023] Machine Readiness Time Characteristics (MCT) k (s t ), which represents the ready time of machine m at time t;

[0024] Machine Load Characteristics MPL k (s t ), representing the load of machine m at time t, where:

[0025]

[0026] In the formula, O i,j For the j-th process of workpiece i, p k,i,j For O i,jThe processing time on machine k, where OPMk is the set of operations that can be processed on machine k;

[0027] Machine utilization characteristics UR m (s t ), representing the utilization rate of machine m at time t, where:

[0028] UR m (s t ) = MCT k (s t MPL k (s t )

[0029] Machine processing time characteristics PT j (s t ), which represents the processing time of workpiece i selected by the top-level decision at time t on machine m.

[0030] Optionally, the step of using a dual deep reinforcement learning algorithm for parameter optimization during the centralized training process for the top-level agent, the intermediate-level agent, and the bottom-level agent includes:

[0031] In the initial stage of training, at time t+1, pre-decision processing is performed on the top-level agent, the intermediate-level agent, and the bottom-level agent to obtain the experience tuples corresponding to each layer agent. The experience tuples include the top-level experience tuple, the intermediate-level experience tuple, and the bottom-level experience tuple.

[0032] After the pre-decision processing is completed, the top-level experience tuple, the intermediate-level experience tuple, and the bottom-level experience tuple are stored in their respective experience buffers by the buffer controller. The experience buffers include the top-level experience buffer, the intermediate-level experience buffer, and the bottom-level experience buffer.

[0033] When the experience buffer is greater than its corresponding preset lower limit, D experience tuples in the experience buffer are randomly sampled for network parameter updates.

[0034] Optionally, the pre-decision processing steps include:

[0035] The top-level state features at time t+1 are obtained from the scheduling environment. The current top-level network parameters of the top-level agent are preloaded and processed to obtain the top-level state features, thereby obtaining the action corresponding to the top-level agent at time t+1. The action corresponding to the top-level agent includes a top-level action space, which consists of n workpieces.

[0036] By using the actions corresponding to the top-level agent, the intermediate-level state features of the intermediate-level agent are obtained from the scheduling environment. The current intermediate-level network parameters of the intermediate-level agent are preloaded and the intermediate-level state features are processed to obtain the actions corresponding to the intermediate-level agent at time t+1. The actions corresponding to the intermediate-level agent include the intermediate-level action space, which is composed of F factories.

[0037] By using the action space corresponding to the intermediate layer agent, the underlying state features of the bottom layer agent are obtained from the scheduling environment. The current underlying network parameters of the bottom layer agent are preloaded to process the underlying state features, thereby obtaining the action corresponding to the bottom layer agent at time t+1. The action corresponding to the intermediate layer agent includes the underlying action space, which is a set of M processing machines in the action space corresponding to the top layer agent that execute the workpiece decision information.

[0038] Optionally, the step of randomly sampling D empirical tuples in the empirical buffer to update network parameters when the empirical buffer is greater than its corresponding preset lower limit includes:

[0039] When the top-level experience buffer is greater than the preset lower limit of the top-level buffer, D top-level experience tuples in the top-level experience buffer are randomly sampled to update the top-level network parameters;

[0040] When the intermediate layer experience buffer is greater than the preset lower limit of the intermediate layer buffer, D intermediate layer experience tuples in the intermediate layer experience buffer are randomly sampled to update the intermediate layer network parameters;

[0041] When the underlying experience buffer is greater than the preset lower limit of the underlying buffer, D underlying experience tuples in the underlying experience buffer are randomly sampled to update the underlying network parameters.

[0042] Optionally, the network parameter update step includes:

[0043] Obtain target network parameters, wherein the target network parameters include at least one of target top-level network parameters, target intermediate-level network parameters, and target bottom-level network parameters;

[0044] Calculate the maximum Q value of the next state of the corresponding agent based on the target network parameters;

[0045] Determine whether the current state characteristics of the agent are terminal states;

[0046] If so, the reward r at time t t As the TD error, the reward is characterized by minimizing the completion time as the common objective;

[0047] Otherwise, calculate the TD error using the following formula:

[0048]

[0049] In the formula, y represents the TD error, and r t Let be the reward at time t, and γ be the reward discount factor. For the maximum Q value, s t+1 Let a be the current state feature of the agent at time t+1. t+1 Let θ represent the action of the agent at time t+1, where θ is the online network parameter and θ′ is the target network parameter.

[0050] After calculating the TD error, according to loss(yQ(s) t ,a t ;θ)) 2 Update the online network parameters using gradient descent.

[0051] After the number of updates meets the preset target network update frequency, the currently obtained online network parameters are used as the target network parameters to complete the network parameter update.

[0052] Furthermore, to achieve the above objectives, this application also provides a distributed flexible job shop scheduling system, which includes: a memory, a processor, and a distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents, stored in the memory and executable on the processor. When the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents is executed by the processor, it implements the steps of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents as described in any of the above claims.

[0053] This application has at least the following beneficial effects:

[0054] 1. A multi-level intelligent agent scheduling framework is introduced. Through the collaborative work of top-level, middle-level and bottom-level intelligent agents, and through hierarchical decision-making, the global complex problem is decomposed into multiple local problems, which improves the system's decision-making efficiency and scheduling accuracy, and achieves a dynamic balance between global scheduling and local optimization.

[0055] 2. Different scheduling tasks in the distributed flexible workshop are executed in a decentralized manner by intelligent agents at various levels. There is no need for a central coordinator to manage every decision. When the environment changes locally, each level can adapt quickly and make the optimal decision according to its specific context, while still serving the overall goal of the whole system.

[0056] 3. Introduce a dual deep reinforcement learning algorithm to train each agent in a centralized manner. This allows the system to access the state, action, and reward information of all agents during the training phase. As a result, the system can use global environmental information to optimize the policy of each agent, enabling better cooperation among the agents and achieving the optimal solution for the overall goal. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the hardware operating environment of the distributed flexible job shop scheduling system involved in the embodiments of this application;

[0058] Figure 2 This is a schematic diagram of the architecture of the distributed flexible job shop scheduling system according to the first embodiment of this application;

[0059] Figure 3 This is a flowchart illustrating the first embodiment of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents in this application.

[0060] Figure 4 This is a schematic diagram of a 2×3×3 distributed flexible job shop scheduling example involved in the first embodiment of this application;

[0061] Figure 5 This is a detailed schematic diagram of the distributed flexible job shop scheduling system architecture involved in the second embodiment of this application;

[0062] Figure 6 This is a schematic diagram of the training curve on a 2×6×10 distributed flexible job shop scheduling system example according to the second embodiment of this application;

[0063] Figure 7 This is a Gantt chart of the processing time of the factory involved in the second embodiment of this application;

[0064] Figure 8 This is a Gantt chart of the processing time of another factory involved in the second embodiment of this application.

[0065] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0066] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0067] As one implementation scheme, Figure 1 This is a schematic diagram of the hardware operating environment of the distributed flexible job shop scheduling system involved in the embodiments of this application.

[0068] like Figure 1 As shown, the distributed flexible job shop scheduling system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0069] Those skilled in the art will understand that Figure 1 The distributed flexible job shop scheduling system architecture shown in the figure does not constitute a limitation on the distributed flexible job shop scheduling system. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0070] like Figure 1 As shown, the memory 1005, serving as a storage medium, may include an operating system, a network communication module, a user interface module, and a distributed flexible job shop scheduler based on dual deep reinforcement learning and multi-layered intelligent agents. The operating system is a program that manages and controls the hardware and software resources of the distributed flexible job shop scheduler system, and it also manages the operation of the distributed flexible job shop scheduler based on dual deep reinforcement learning and multi-layered intelligent agents, as well as other software or programs.

[0071] exist Figure 1 In the distributed flexible job shop scheduling system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the back-end server; and the processor 1001 can be used to call the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents stored in the memory 1005.

[0072] In this embodiment, the distributed flexible job shop scheduling system includes: a memory 1005, a processor 1001, and a distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents, stored in the memory and executable on the processor, wherein:

[0073] When processor 1001 calls the distributed flexible job shop scheduler based on dual deep reinforcement learning and multi-layer intelligent agents stored in memory 1005, it performs the following operations:

[0074] S1, the top-level agent obtains the current top-level state characteristics at time t from the scheduling environment, determines the workpiece decision information based on the current state characteristics, and transmits the workpiece decision information to the intermediate layer agent and the bottom layer agent;

[0075] S2, the intermediate layer agent obtains the current intermediate layer state characteristics from the scheduling environment, determines the factory decision information based on the workpiece decision information and the current intermediate layer state characteristics, and transmits the factory decision information to the bottom layer agent;

[0076] S3, the underlying intelligent agent obtains the current underlying state characteristics from the scheduling environment, and determines the equipment decision information based on the workpiece decision information, the factory decision information, and the current underlying state characteristics;

[0077] S4. Based on the workpiece decision information, the factory decision information, and the equipment decision information, determine joint decision information, execute a scheduling strategy based on the joint decision information, and perform a state transition on the scheduling environment so that the top-level agent can obtain the top-level state characteristics and rewards at time t+1 and feed them back to the top-level agent.

[0078] S5, repeat steps S1-S4 until each workpiece is processed.

[0079] When processor 1001 calls the distributed flexible job shop scheduler based on dual deep reinforcement learning and multi-layer intelligent agents stored in memory 1005, it performs the following operations:

[0080] In the initial stage of training, at time t+1, pre-decision processing is performed on the top-level agent, the intermediate-level agent, and the bottom-level agent to obtain the experience tuples corresponding to each layer agent. The experience tuples include the top-level experience tuple, the intermediate-level experience tuple, and the bottom-level experience tuple.

[0081] After the pre-decision processing is completed, the top-level experience tuple, the intermediate-level experience tuple, and the bottom-level experience tuple are stored in their respective experience buffers by the buffer controller. The experience buffers include the top-level experience buffer, the intermediate-level experience buffer, and the bottom-level experience buffer.

[0082] When the experience buffer is greater than its corresponding preset lower limit, D experience tuples in the experience buffer are randomly sampled for network parameter updates.

[0083] When processor 1001 calls the distributed flexible job shop scheduler based on dual deep reinforcement learning and multi-layer intelligent agents stored in memory 1005, it performs the following operations:

[0084] The top-level state features at time t+1 are obtained from the scheduling environment. The current top-level network parameters of the top-level agent are preloaded and processed to obtain the top-level state features, thereby obtaining the action corresponding to the top-level agent at time t+1. The action corresponding to the top-level agent includes a top-level action space, which consists of n workpieces.

[0085] By using the actions corresponding to the top-level agent, the intermediate-level state features of the intermediate-level agent are obtained from the scheduling environment. The current intermediate-level network parameters of the intermediate-level agent are preloaded and the intermediate-level state features are processed to obtain the actions corresponding to the intermediate-level agent at time t+1. The actions corresponding to the intermediate-level agent include the intermediate-level action space, which is composed of F factories.

[0086] By using the action space corresponding to the intermediate layer agent, the underlying state features of the bottom layer agent are obtained from the scheduling environment. The current underlying network parameters of the bottom layer agent are preloaded to process the underlying state features, thereby obtaining the action corresponding to the bottom layer agent at time t+1. The action corresponding to the intermediate layer agent includes the underlying action space, which is a set of M processing machines in the action space corresponding to the top layer agent that execute the workpiece decision information.

[0087] When processor 1001 calls the distributed flexible job shop scheduler based on dual deep reinforcement learning and multi-layer intelligent agents stored in memory 1005, it performs the following operations:

[0088] When the top-level experience buffer is greater than the preset lower limit of the top-level buffer, D top-level experience tuples in the top-level experience buffer are randomly sampled to update the top-level network parameters;

[0089] When the intermediate layer experience buffer is greater than the preset lower limit of the intermediate layer buffer, D intermediate layer experience tuples in the intermediate layer experience buffer are randomly sampled to update the intermediate layer network parameters;

[0090] When the underlying experience buffer is greater than the preset lower limit of the underlying buffer, D underlying experience tuples in the underlying experience buffer are randomly sampled to update the underlying network parameters.

[0091] When processor 1001 calls the distributed flexible job shop scheduler based on dual deep reinforcement learning and multi-layer intelligent agents stored in memory 1005, it performs the following operations:

[0092] Obtain target network parameters, wherein the target network parameters include at least one of target top-level network parameters, target intermediate-level network parameters, and target bottom-level network parameters;

[0093] Calculate the maximum Q value of the next state of the corresponding agent based on the target network parameters;

[0094] Determine whether the current state characteristics of the agent are terminal states;

[0095] If so, the reward r at time t t As the TD error, the reward is characterized by minimizing the completion time as the common objective;

[0096] Otherwise, calculate the TD error using the following formula:

[0097]

[0098] In the formula, y represents the TD error, and r t Let be the reward at time t, and γ be the reward discount factor. For the maximum Q value, s t+1 Let a be the current state feature of the agent at time t+1. t+1 Let θ represent the action of the agent at time t+1, where θ is the online network parameter and θ′ is the target network parameter.

[0099] After calculating the TD error, according to loss(yQ(s) t ,a t ;θ)) 2 Update the online network parameters using gradient descent.

[0100] After the number of updates meets the preset target network update frequency, the currently obtained online network parameters are used as the target network parameters to complete the network parameter update.

[0101] Based on the hardware architecture of the distributed flexible job shop scheduling system based on the above-mentioned intelligent manufacturing technology, this application proposes an embodiment of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents.

[0102] First Embodiment

[0103] In the first embodiment, referring to Figure 2The diagram shows the architecture of a distributed flexible job shop scheduling system. The distributed flexible job shop scheduling system includes: a top-level intelligent agent for determining the target workpiece with the highest processing priority based on workpiece information; an intermediate-level intelligent agent for determining the target processing factory corresponding to the target workpiece based on factory information; and a bottom-level intelligent agent for determining the target processing machine corresponding to the target workpiece based on the processing machine information in the target processing factory.

[0104] Specifically, the top-level agent, located at the very top of the scheduling architecture, is responsible for global production scheduling planning and job selection. Its main function is to analyze the overall production needs and status of the current system and select jobs that need to be prioritized for processing. Based on parameters such as delivery deadlines, processing requirements, and the availability of production resources, it evaluates the priority of all jobs to be processed, determining which jobs require priority handling in the current production cycle. After evaluation, the top-level agent selects the highest-priority job that best suits the current production capacity and prepares to distribute it to the intermediate-level agents for more specific factory allocation.

[0105] For the intermediate-layer agent, the responsibility is to assign the most suitable factory for processing the workpieces selected by the top-layer agent. Factories in a distributed manufacturing system may have different processing capabilities, equipment configurations, and resource usage. The intermediate-layer agent's task is to maximize the utilization of these distributed resources. The intermediate-layer agent monitors the production status of each factory in real time, including data such as equipment availability, current factory workload, production capacity, and processing history. This allows it to grasp the real-time status of factory resources. After workpiece selection, the intermediate-layer agent assigns the workpiece to the most suitable factory based on the status of each factory and the workpiece's processing requirements.

[0106] For the bottom-level intelligent agent, the specific machine scheduling is responsible for ensuring that workpieces receive optimal processing within the designated factory. After the workpiece is allocated to the factory by the intermediate-level intelligent agent, the bottom-level intelligent agent selects the most suitable processing equipment for the workpiece based on the specific machine resources within the factory, and performs real-time scheduling and monitoring of the processing process. The bottom-level intelligent agent is not only responsible for machine allocation, but also for real-time monitoring of the processing process to ensure that the processing progress remains consistent with the plan.

[0107] Furthermore, the distributed flexible job shop scheduling system described in this embodiment also includes a Double Deep Q Network (DDQN) architecture, in which the top-level agent, the middle-level agent, and the bottom-level agent all use the Double Deep Q Network algorithm for parameter optimization during centralized training.

[0108] Furthermore, in this embodiment, the distributed flexible workshop scheduling system (hereinafter referred to as the system) includes a scheduling environment as one of its core components. This environment contains the state space, action space, state transition mechanism, reward function, and termination condition of each level of intelligent agent. The dynamic changes and feedback mechanisms within the environment provide decision-making basis for intelligent agents at each level, ensuring the smooth execution of the entire scheduling process.

[0109] The state space stores the state features of agents at each level. Each state feature is assigned a timestamp to record the update time of the state feature. Agents at different levels have different state features. Agents can schedule the environment to obtain features that match their target state features at the current time (i.e., time t) based on the target state features they need to find.

[0110] Further and optionally, the top-level state features in the top-level agent include:

[0111] Completed processing features CP i (s t If all workpieces have been processed by time t, then CP i (s t ) = 1, otherwise CP i (s t )=0;

[0112] Workpiece processing average time characteristics (TPT) i (s t ), representing the average processing time of workpiece i at time t across all available machines, is mathematically expressed as:

[0113]

[0114] Pre-process features OCT i (s t ), where, if at time t, the completion time of the preceding process of the workpiece i's machinable process is OCT. i (s t If ) = 1, then OCT is used if there is no preceding process. i (s t )=0;

[0115] Processing time characteristics PD i (s t The processing time feature is characterized by the processing time of workpiece i at time t.

[0116] Further and optionally, the intermediate layer state features in the intermediate layer agent include:

[0117] Factory load characteristics FPL f (st ), which represents the load of factory f at time t;

[0118] Pre-scheduled features SPD f (s t ), which represents the predicted maximum completion time obtained by the factory f performing a complete pre-schedule of the workpiece i selected by the top-level decision action at time t, wherein the machine with the shortest processing time for each process is selected for pre-schedule of the operation without changing the existing scheduling scheme of the factory.

[0119] Further and optionally, the underlying state features in the underlying agent include:

[0120] Machine Readiness Time Characteristics (MCT) k (s t ), which represents the ready time of machine m at time t;

[0121] Machine Load Characteristics MPL k (s t ), representing the load of machine m at time t, where:

[0122]

[0123] In the formula, O i,j For the j-th process of workpiece i, p k,i,j For O i,j The processing time on machine k, where OPMk is the set of operations that can be processed on machine k;

[0124] Machine utilization characteristics UR m (s t ), representing the utilization rate of machine m at time t, where:

[0125] UR m (s t ) = MCT k (s t MPL k (s t )

[0126] Machine processing time characteristics PT j (s t ), which represents the processing time of workpiece i selected by the top-level decision at time t on machine m.

[0127] In a complete scheduling process, there are several different states, denoted by s. tThis represents the current state. The state representation primarily reflects the state characteristics of candidate actions at each level within the state. These states are independent but interconnected with other states. The top-level state characteristics can be directly obtained from the scheduling environment, while the middle and bottom-level state characteristics need to be obtained from the scheduling environment in conjunction with the decisions of the upper levels. The top-level state consists of characteristics associated with all currently processable operations across all jobs.

[0128] Each level has its own independent motion space. and At time 0, the first operation of all workpieces is machinable, and the top-level motion space... Composed of n workpieces, as the scheduling environment changes, the workpiece processes are gradually completed, leading to a reduction in the motion space. When the motion space reaches zero, it indicates that all workpieces have been processed, marking the end of scheduling. (Intermediate layer motion space) It consists of F factories, and each workpiece can be assigned to any factory for processing, with a low-level motion space. It is to perform an action A set of compatible machines.

[0129] The state transition mechanism updates the state features in the state space based on certain judgment conditions. For example, the top-level state features of the top-level agent include the completed processing feature CP. i (s t If the condition is met: all workpieces are processed at time t, then the state transition mechanism will change CP. i (s t Update from 0 to 1, otherwise CP i (s t ) = 0.

[0130] The reward function, also known as the return, is implemented in the system described in this embodiment, where each layer of agents shares the common goal of minimizing the completion time. Therefore, the reward r at time t is... t =C max (s t )-C max (s t+1 For unprocessed processes, the average available processing time is used for virtual scheduling and filling until all processes are filled and the predicted completion time is obtained. The difference between the two scheduling times is the reward at time t.

[0131] Optionally, the system in this embodiment satisfies the following constraints when executing scheduling tasks:

[0132] minC max (1)

[0133]

[0134] Equation (1) is the objective function, which represents minimizing the completion time;

[0135] Equation (2) indicates that all processes of a workpiece can only be processed in the same factory;

[0136] Equation (3) represents the flexible constraint for workpiece machining;

[0137] Equation (4) indicates that there is a priority relationship between different processes of the same workpiece;

[0138] Equation (5) indicates that any process of any workpiece can be processed on the machine at time 0;

[0139] Equation (6) indicates that the processing time of a process cannot be earlier than the start time;

[0140] Equations (7) and (8) indicate that the same machine in the factory can only process one operation at a time;

[0141] Equation (9) means that at any given time, the same process can only be processed by one machine;

[0142] Equation (10) represents the maximum completion time of the factory;

[0143] Equation (11) represents the maximum completion time of the entire scheduling task.

[0144] The meanings of the parameters in the formula are shown in Table 1 below:

[0145]

[0146]

[0147] Furthermore, the system in this embodiment also includes a key architecture: a distributed execution architecture composed of multi-level intelligent agents. Each level can independently make decisions during execution based on its own local observations and information obtained from higher levels. (Refer to...) Figure 3 The distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents includes the following steps:

[0148] S1, the top-level agent obtains the current top-level state characteristics at time t from the scheduling environment, determines the workpiece decision information based on the current state characteristics, and transmits the workpiece decision information to the intermediate layer agent and the bottom layer agent;

[0149] S2, the intermediate layer agent obtains the current intermediate layer state characteristics from the scheduling environment, determines the factory decision information based on the workpiece decision information and the current intermediate layer state characteristics, and transmits the factory decision information to the bottom layer agent;

[0150] S3, the underlying intelligent agent obtains the current underlying state characteristics from the scheduling environment, and determines the equipment decision information based on the workpiece decision information, the factory decision information, and the current underlying state characteristics;

[0151] S4. Based on the workpiece decision information, the factory decision information, and the equipment decision information, determine joint decision information, execute a scheduling strategy based on the joint decision information, and perform a state transition on the scheduling environment so that the top-level agent can obtain the top-level state characteristics and rewards at time t+1 and feed them back to the top-level agent.

[0152] S5, repeat steps S1-S4 until each workpiece is processed.

[0153] In this embodiment, during a complete scheduling process, the top-level agent first obtains the current top-level state characteristics at time t from the scheduling environment, and then determines the workpiece decision information based on the current state characteristics. This workpiece decision information is then passed to the lower-level agents, namely the intermediate-level agents and the bottom-level agents. Next, the intermediate-level agents obtain the current intermediate-level state characteristics from the scheduling environment, and determine the factory decision information based on the workpiece decision information and the current intermediate-level state characteristics. This factory decision information is then passed to the bottom-level agents. Finally, the bottom-level agents obtain the current bottom-level state characteristics from the scheduling environment, and determine the equipment decision information based on the workpiece decision information, the factory decision information, and the current bottom-level state characteristics.

[0154] In this embodiment, the top layer only needs to consider its own current state characteristics. The current decision can be made immediately. Next, the decision information The decision information is passed to the middle layer and the bottom layer, and the middle layer receives the decision information. Then, the state characteristics of the intermediate layer are obtained by combining the scheduling environment. Make decisions independently and share decision information The information is passed to the lower layer, which, after receiving all the decision information, also combines it with the scheduling environment to obtain the lower-level state characteristics. And make independent decisions.

[0155] Furthermore, after all three levels have made a decision, the information from the three decisions is combined to obtain joint decision information. Then, a state transition is performed on the environment to obtain the state at the next moment. and r t Repeat the above process until all workpieces are processed.

[0156] For example, refer to Figure 4The diagram illustrates a 2×3×3 distributed flexible job shop scheduling example. At time t (left) and t+1 (right), the red-circled workpiece nodes represent the processes available for processing at time t. The solid colored lines connecting to these nodes represent the selected processing plants and machines within those plants, while the dashed lines represent available machines. At time t, the instance environment executes the joint action pair (2,1,3), assigning workpiece 2 to plant 1 and processing the currently available processing steps of plant 2 on machine 3 in plant 1. The state then transitions to time t+1.

[0157] In the technical solution provided in this embodiment, by introducing a multi-level intelligent agent scheduling framework, the top-level, middle-level, and bottom-level intelligent agents work together and make decisions in a hierarchical manner, decomposing the global complex problem into multiple local problems, thereby improving the system's decision-making efficiency and scheduling accuracy, and achieving a dynamic balance between global scheduling and local optimization. The intelligent agents at each level perform different scheduling tasks in the distributed flexible work workshop in a decentralized manner, without the need for a central coordinator to manage every decision. When the environment changes locally, each level can adapt quickly and make the optimal decision according to its specific context, while still serving the overall goal of the entire system.

[0158] Second Embodiment

[0159] Based on the first embodiment, referring to Figure 5 The diagram shows a detailed schematic of the distributed flexible job shop scheduling system architecture. To illustrate how to optimize the parameters of agents at each level of the system based on a dual deep reinforcement learning algorithm, this embodiment adds the following to the architecture of the first embodiment: Figure 5 The multiple levels of experience buffers and corresponding networks shown on the right are used to update the system's network parameters, helping each agent establish a unified optimization goal and preventing each agent from focusing too much on its own goal and neglecting overall cooperation.

[0160] The following explains how to conduct intensive training:

[0161] Step S10: In the initial stage of training, at time t+1, pre-decision processing is performed on the top-level agent, the intermediate-level agent, and the bottom-level agent to obtain the experience tuples corresponding to each layer agent. The experience tuples include the top-level experience tuple, the intermediate-level experience tuple, and the bottom-level experience tuple.

[0162] In this step, during the initial stage of training, the states at time t+1 required by the intermediate and bottom layers cannot be obtained directly from the environment because the bottom layer needs the decision of the upper layer to obtain the current state from the environment. At time t+1, the upper layer has not made any decision, so a pre-decision needs to be made for each layer at time t+1.

[0163] Optionally, the pre-decision processing steps include:

[0164] Step S11: Obtain the top-level state features at time t+1 from the scheduling environment, preload the current top-level network parameters of the top-level agent to process the top-level state features, and obtain the action corresponding to the top-level agent at time t+1. The action corresponding to the top-level agent includes a top-level action space, which consists of n workpieces.

[0165] Step S12: Based on the action corresponding to the top-level agent, obtain the intermediate layer state features of the intermediate layer agent from the scheduling environment, preload the current intermediate layer network parameters of the intermediate layer agent to process the intermediate layer state features, and obtain the action corresponding to the intermediate layer agent at time t+1. The action corresponding to the intermediate layer agent includes the intermediate layer action space, which is composed of F factories.

[0166] Step S13: Obtain the underlying state features of the bottom layer agent from the scheduling environment through the action space corresponding to the intermediate layer agent, preload the current underlying network parameters of the bottom layer agent to process the underlying state features, and obtain the action corresponding to the bottom layer agent at time t+1. The action corresponding to the intermediate layer agent includes the underlying action space, which is a set of M processing machines in the action space corresponding to the top layer agent that execute the workpiece decision information.

[0167] Each level has its own independent motion space. and At time 0, the first operation of all workpieces is machinable, and the top-level motion space... Composed of n workpieces, as the scheduling environment changes, the workpiece processes are gradually completed, leading to a reduction in the motion space. When the motion space reaches zero, it indicates that all workpieces have been processed, marking the end of scheduling. (Intermediate layer motion space) It consists of F factories, and each workpiece can be assigned to any factory for processing, with a low-level motion space. It is to perform an action A set of compatible machines.

[0168] In the pre-decision processing step, the top-level state features at time t+1 are obtained from the environment. Then, preload the current online network parameters at the top level. Processing to obtain the top-level actions at time t+1 in, Then through Obtain the intermediate layer from the environment Repeated operations are performed on the intermediate layer to obtain the underlying layer. This completes the pre-decision processing.

[0169] Step S20: After the pre-decision processing is completed, the top-level experience tuple, the intermediate-level experience tuple and the bottom-level experience tuple are stored into their respective experience buffers by the buffer controller. The experience buffers include the top-level experience buffer, the intermediate-level experience buffer and the bottom-level experience buffer.

[0170] Step S30: When the experience buffer is greater than its corresponding buffer preset lower limit, randomly sample D experience tuples in the experience buffer to update the network parameters.

[0171] Furthermore, after the pre-decision processing is completed, the buffer controller sets the corresponding experience tuples for each layer. The data is stored in the corresponding experience buffer. The experience buffer is used by the agent to uniformly draw samples from its corresponding experience buffer for centralized training when it needs to update parameters.

[0172] Furthermore, after the experience tuples are stored in the experience buffer, when it is determined that the experience buffer is greater than its corresponding preset lower limit, it is determined that the network parameters need to be updated, and D experience tuples in the experience buffer are randomly sampled to update the network parameters.

[0173] Further and optionally, step S30 specifically includes:

[0174] Step S31: When the top-level experience buffer is greater than the preset lower limit of the top-level buffer, randomly sample D top-level experience tuples in the top-level experience buffer to update the top-level network parameters;

[0175] Step S32: When the intermediate layer experience buffer is greater than the preset lower limit of the intermediate layer buffer, randomly sample D intermediate layer experience tuples in the intermediate layer experience buffer to update the intermediate layer network parameters;

[0176] Step S33: When the underlying experience buffer is greater than the preset lower limit of the underlying buffer, randomly sample D underlying experience tuples in the underlying experience buffer to update the underlying network parameters.

[0177] For example, when the top-level experience buffer R J Size > Lower bound of top-level experience buffer B wJM If t, then randomly sample empirical tuples of batch size D in the top-level empirical buffer. Update the parameters of the top-level network. Repeat the same process for the remaining layers.

[0178] As an optional implementation, the pseudocode execution logic based on the above centralized training logic is given below:

[0179]

[0180]

[0181]

[0182] Further and optionally, in this embodiment, the network parameter update step includes:

[0183] Step S100: Obtain target network parameters, wherein the target network parameters include at least one of target top-level network parameters, target intermediate-level network parameters, and target bottom-level network parameters;

[0184] Step S200: Calculate the maximum Q value of the next state of the corresponding agent based on the target network parameters;

[0185] Step S300: Determine whether the current state feature of the intelligent agent is a terminal state;

[0186] Step S400, if so, take the reward r at time t. t As the TD error, the reward is characterized by minimizing the completion time as the common objective;

[0187] Step S500, otherwise, calculate the TD error according to the following formula:

[0188]

[0189] In the formula, y represents the TD error, and r t Let be the reward at time t, and γ be the reward discount factor. For the maximum Q value, s t+1 Let a be the current state feature of the agent at time t+1. t+1 Let θ represent the action of the agent at time t+1, where θ is the online network parameter and θ′ is the target network parameter.

[0190] Step S600: After calculating the TD error, according to loss(yQ(s) t ,a t ;θ)) 2 Update the online network parameters using gradient descent.

[0191] Step S700: After the number of updates meets the preset target network update frequency, the currently obtained online network parameters are used as the target network parameters to complete the network parameter update.

[0192] As an optional implementation, the pseudocode execution logic based on the above network parameter updates is given below:

[0193]

[0194] For example, Figure 6 This diagram illustrates the training curves on a 2×6×10 distributed flexible job shop scheduling system instance. The X-axis represents the number of training iterations, and the Y-axis represents the maximum completion time. The yellow curve represents the average completion time variation of the 100 instances in the validation set that did not participate in the training, while the blue curve represents the maximum completion time of the instances participating in the actual training in each iteration of the training set. It can be seen that the convergence of the validation set perfectly matches that of the training set, achieving good convergence even on data that has not appeared before. Figure 7 and Figure 8 These are Gantt charts showing the processing times for the two factories in this example.

[0195] In the technical solution provided in this embodiment, a dual deep reinforcement learning algorithm is introduced to train each agent in a centralized manner. This allows the system to access the state, action, and reward information of all agents during the training phase, thereby enabling the system to utilize global environmental information to optimize the strategy of each agent. This allows the agents to better cooperate and achieve the optimal solution for the overall goal.

[0196] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the distributed flexible job shop scheduling system to implement the process steps of the embodiments of the above methods.

[0197] Therefore, this application also provides a computer-readable storage medium storing a distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents. When the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents is executed by a processor, it implements the various steps of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents as described in the above embodiments.

[0198] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0199] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0200] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0204] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0205] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0206] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents, characterized in that, An application is made in a distributed flexible job shop scheduling system. The system includes: a top-level agent for determining the target workpiece with the highest processing priority based on workpiece information; an intermediate-level agent for determining the target processing plant corresponding to the target workpiece based on plant information; and a bottom-level agent for determining the target processing machine corresponding to the target workpiece based on processing machine information in the target processing plant. The top-level, intermediate-level, and bottom-level agents all employ a dual-deep reinforcement learning algorithm for parameter optimization during centralized training. The method includes the following steps: S1, the top-level agent obtains the current top-level state characteristics at time t from the scheduling environment, determines the workpiece decision information based on the current top-level state characteristics, and transmits the workpiece decision information to the intermediate layer agent and the bottom layer agent; S2, the intermediate layer agent obtains the current intermediate layer state characteristics from the scheduling environment, determines the factory decision information based on the workpiece decision information and the current intermediate layer state characteristics, and transmits the factory decision information to the bottom layer agent; S3, the underlying intelligent agent obtains the current underlying state characteristics from the scheduling environment, and determines the equipment decision information based on the workpiece decision information, the factory decision information, and the current underlying state characteristics; S4. Based on the workpiece decision information, the factory decision information, and the equipment decision information, determine joint decision information, execute a scheduling strategy based on the joint decision information, and perform a state transition on the scheduling environment so that the top-level agent can obtain the top-level state characteristics and rewards at time t+1 and feed them back to the top-level agent. S5, repeat steps S1-S4 until each workpiece is processed; The steps for parameter optimization using a dual deep reinforcement learning algorithm during the centralized training process for the top-level agent, the intermediate-level agent, and the bottom-level agent include: In the initial stage of training, at time t+1, pre-decision processing is performed on the top-level agent, the intermediate-level agent, and the bottom-level agent to obtain the experience tuples corresponding to each layer agent. The experience tuples include the top-level experience tuple, the intermediate-level experience tuple, and the bottom-level experience tuple. After the pre-decision processing is completed, the top-level experience tuple, the intermediate-level experience tuple, and the bottom-level experience tuple are stored in their respective experience buffers by the buffer controller. The experience buffers include the top-level experience buffer, the intermediate-level experience buffer, and the bottom-level experience buffer. When the experience buffer is greater than its corresponding preset lower limit, D experience tuples in the experience buffer are randomly sampled for network parameter updates.

2. The method as described in claim 1, characterized in that, The top-level state features of the top-level agent include: Finished features If in Whether all workpieces have been processed at any given time. ,otherwise ; Average processing time characteristics of workpieces Characterized as workpiece exist The average processing time of a given time-bound process across all available machines is expressed mathematically as follows: ； Pre-process characteristics Among them, if in Time workpiece The completion time of the preceding process of the process that can be processed is If there is no preceding process, then ; Processing time characteristics The processing time characteristic is characterized by the workpiece exist The processing time of the moment.

3. The method as described in claim 1, characterized in that, The intermediate layer state features of the intermediate layer agent include: Factory load characteristics , characterized as in Time Factory The load; Pre-scheduling features , characterized as in Time Factory The workpiece for the top-level decision-making action selection The predicted maximum completion time is obtained by performing a complete pre-scheduling, in which the machine with the shortest processing time for each operation is selected for pre-scheduling without changing the existing scheduling scheme of the factory.

4. The method as described in claim 1, characterized in that, The underlying state features of the underlying intelligent agent include: Machine readiness time characteristics , characterized as in Time Machine Readiness time; Machine load characteristics , characterized as in Time Machine The load, where: ； In the formula, For workpiece The Each process, for In the machine The processing time for the upper processing. Let k be the set of operations that can be processed on machine k. Machine utilization characteristics , characterized as in Time Machine The utilization rate, of which: ； Machine processing time characteristics , characterized as in The workpiece selected at the top level of time. In the machine Processing time.

5. The method as described in claim 1, characterized in that, The steps of the pre-decision processing include: The top-level state features at time t+1 are obtained from the scheduling environment. The current top-level network parameters of the top-level agent are preloaded and processed to obtain the top-level state features, thereby obtaining the action corresponding to the top-level agent at time t+1. The action corresponding to the top-level agent includes a top-level action space, which consists of n workpieces. By using the actions corresponding to the top-level agent, the intermediate-level state features of the intermediate-level agent are obtained from the scheduling environment. The current intermediate-level network parameters of the intermediate-level agent are preloaded and the intermediate-level state features are processed to obtain the actions corresponding to the intermediate-level agent at time t+1. The actions corresponding to the intermediate-level agent include the intermediate-level action space, which is composed of F factories. By using the action space corresponding to the intermediate layer agent, the underlying state features of the bottom layer agent are obtained from the scheduling environment. The current underlying network parameters of the bottom layer agent are preloaded to process the underlying state features, thereby obtaining the action corresponding to the bottom layer agent at time t+1. The action corresponding to the intermediate layer agent includes the underlying action space, which is a set of M processing machines in the action space corresponding to the top layer agent that execute the workpiece decision information.

6. The method as described in claim 1, characterized in that, The step of randomly sampling D empirical tuples in the empirical buffer to update network parameters when the empirical buffer is greater than its corresponding preset lower limit includes: When the top-level experience buffer is greater than the preset lower limit of the top-level buffer, D top-level experience tuples in the top-level experience buffer are randomly sampled to update the top-level network parameters; When the intermediate layer experience buffer is greater than the preset lower limit of the intermediate layer buffer, D intermediate layer experience tuples in the intermediate layer experience buffer are randomly sampled to update the intermediate layer network parameters; When the underlying experience buffer is greater than the preset lower limit of the underlying buffer, D underlying experience tuples in the underlying experience buffer are randomly sampled to update the underlying network parameters.

7. The method as described in claim 1, characterized in that, The steps for updating the network parameters include: Obtain target network parameters, wherein the target network parameters include at least one of target top-level network parameters, target intermediate-level network parameters, and target bottom-level network parameters; Calculate the maximum Q value of the next state of the corresponding agent based on the target network parameters; Determine whether the current state characteristics of the agent are terminal states; If so, the reward r at time t t As the TD error, the reward is characterized by minimizing the completion time as the common objective; Otherwise, calculate the TD error using the following formula: ； In the formula, y represents the TD error, and r t The reward at time t, As a reward discount factor, For the maximum Q value, Let be the current state features of the agent at time t+1. This represents the action of the agent at time t+1. For online network parameters: For target network parameters; After calculating the TD error, according to Update the online network parameters using gradient descent. After the number of updates meets the preset target network update frequency, the currently obtained online network parameters are used as the target network parameters to complete the network parameter update.

8. A distributed flexible job shop scheduling system, characterized in that, The distributed flexible job shop scheduling system includes: a memory, a processor, and a distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents, which is stored in the memory and can run on the processor. When the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents is executed by the processor, it implements the steps of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents as described in any one of claims 1 to 7.

Images