Distributed flexible job shop scheduling method and system based on double deep reinforcement learning and multi-layer intelligent agent

Abstract

The invention relates to the technical field of intelligent manufacturing, in particular to a distributed flexible job shop scheduling method and system based on double deep reinforcement learning and a multi-layer agent. A global complex problem is decomposed into a plurality of local problems by introducing a multi-level agent scheduling framework and through cooperative work and hierarchical decision making of a top layer agent, a middle layer agent and a bottom layer agent; besides, different scheduling tasks of the distributed flexible job shop are executed dispersedly through the intelligent agent of each level, each decision does not need to be managed by a central coordinator, and when the environment is locally changed, each level can quickly adapt and make an optimal decision according to the specific context of the environment, so that the scheduling efficiency of the distributed flexible job shop is improved. And meanwhile, the overall goal of the whole system is still served. The objective of the invention is to solve the problem of how to improve the scheduling precision of the distributed flexible job shop.

Description

Technical Field

[0001] The present application relates to the field of intelligent manufacturing technology, and in particular to a distributed flexible job shop scheduling method and system based on dual deep reinforcement learning and multi-layer intelligent agents. Background Art

[0002] In recent years, due to the rise of multi-factory production models, distributed flexible manufacturing has gradually become an important research direction in the manufacturing industry. Flexible job shop scheduling, as the core technology in flexible manufacturing systems, can help companies maintain production efficiency and profitability in a dynamic market environment through flexible scheduling and resource optimization. Distributed manufacturing can not only effectively allocate resources from different factories, but also reduce manufacturing costs and management risks by optimizing the reasonable allocation of production tasks, helping companies achieve higher production efficiency. Therefore, the production scheduling problem in a distributed manufacturing environment is one of the current research hotspots in the field of intelligent manufacturing.

[0003] Among the relevant technical solutions, traditional distributed flexible job shop scheduling solutions mainly focus on mathematical programming and heuristic algorithms, which have defects such as limited computing power and limitation to small-scale problems. At present, some researchers have combined deep reinforcement learning (DRL) with job shop scheduling, and used the powerful computing power of DRL to optimize complex job shop scheduling problems.

[0004] However, most of the existing research on DRL and job shop scheduling is in a single factory or single agent environment. In a distributed environment, the collaborative scheduling, resource allocation and task optimization between multiple factories or equipment involve more complex interactions and decision-making processes. The application of single-agent DRL methods in this scenario is subject to many limitations. A single agent cannot fully grasp the global information in a distributed environment and it is difficult to optimize the resource scheduling of multiple factories at the same time. In addition, when dealing with cross-factory task collaboration, a single agent lacks an effective coordination mechanism, which easily leads to resource waste or unbalanced scheduling. In addition, its performance on distributed scheduling problems is limited and it cannot fully cope with the multi-objective and large-scale scheduling requirements in a distributed environment.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents, aiming to solve the problem of how to improve the scheduling accuracy of distributed flexible job shops.

[0007] To achieve the above-mentioned purpose, the present application provides a distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer agents, which is applied to a distributed flexible job shop scheduling system. The system includes: a top-level agent for determining a target workpiece to be processed with the highest processing priority according to workpiece information, an intermediate-level agent for determining a target processing factory corresponding to the target workpiece to be processed according to factory information, and a bottom-level agent for determining a target processing machine corresponding to the target workpiece to be processed according to processing machine information in the target processing factory. The top-level agent, the intermediate-level agent, and the bottom-level agent all use a dual deep reinforcement learning algorithm for parameter optimization during centralized training. The method includes the following steps:

[0008] S1, the top-level agent obtains the current top-level state characteristics at time t from the scheduling environment, determines the workpiece decision information according to the current state characteristics, and transmits the workpiece decision information to the middle-level agent and the bottom-level agent;

[0009] S2, the middle-layer agent obtains the current middle-layer state characteristics from the scheduling environment, determines the factory decision information according to the workpiece decision information and the current middle-layer state characteristics, and transmits the factory decision information to the bottom-layer agent;

[0010] S3, the underlying intelligent agent obtains the current underlying state characteristics from the scheduling environment, and determines the equipment decision information according to the workpiece decision information, the factory decision information and the current underlying state characteristics;

[0011] S4, determining joint decision information according to the workpiece decision information, the factory decision information and the equipment decision information, executing a scheduling strategy based on the joint decision information, and performing a state transfer on the scheduling environment, so that the top-level agent obtains the top-level state characteristics and rewards at time t+1, and feeds back to the top-level agent;

[0012] S5, looping through steps S1-S4 until each workpiece to be processed is completed.

[0013] Optionally, the top-level state features in the top-level agent include:

[0014] Processing completed feature CP i (s t ), if all workpieces are processed at time t, then CP i (s t )=1, otherwise CP i (s t )=0;

[0015] Average time consumption feature of workpiece processing TPTi (s t ), which is represented by the average processing time of the workpiece i on all the machinable machines at time t. Its mathematical expression is:

[0016]

[0017] Pre-process characteristics OCT i (s t ), where if the completion time of the predecessor process of the processable process of workpiece i at time t is OCT i (s t )=1, if there is no pre-process, then OCT i (s t )=0;

[0018] Processing time feature PD i (s t ), the processing time feature is characterized by the processing time of workpiece i at time t.

[0019] Optionally, the intermediate layer state characteristics in the intermediate layer agent include:

[0020] Factory Load Characteristics FPL f (s t ), represented by the load of plant f at time t;

[0021] Pre-dispatch characteristic SPD f (s t ), represented by the predicted maximum completion time obtained by factory f performing complete pre-scheduling on job i selected by the top-level decision action at time t, where the machine with the shortest processing time for each process is selected for pre-scheduling without changing the factory's existing scheduling plan.

[0022] Optionally, the underlying state characteristics in the underlying agent include:

[0023] Machine Ready Time Characteristics MCT k (s t ), represented as the ready time of machine m at time t;

[0024] Machine Load Signature MPL k (s t ), represented by the load of machine m at time t, where:

[0025]

[0026] In the formula, O i,j is the jth process of workpiece i, p k,i,j O i,jThe processing time for processing on machine k, OPMk is the set of operations that can be processed on machine k;

[0027] Machine utilization characteristic UR m (s t ), represented by the utilization of machine m at time t, where:

[0028] UR m (s t )=MCT k (s t )MPL k (s t )

[0029] Machine processing time characteristics PT j (s t ), represented as the processing time of job i on machine m selected by the top-level decision at time t.

[0030] Optionally, the step of using a dual deep reinforcement learning algorithm to perform parameter optimization on the top-layer agent, the middle-layer agent, and the bottom-layer agent during centralized training includes:

[0031] In the initial stage of training, pre-decision processing is performed on the top-level agent, the middle-level agent and the bottom-level agent at time t+1 to obtain experience tuples corresponding to each layer agent, respectively, wherein the experience tuples include the top-level experience tuple, the middle-level experience tuple and the bottom-level experience tuple;

[0032] After the pre-decision processing is completed, the top-level experience tuple, the middle-level experience tuple and the bottom-level experience tuple are respectively stored in their corresponding experience buffers through a buffer controller, and the experience buffer includes a top-level experience buffer, a middle-level experience buffer and a bottom-level experience buffer;

[0033] When the experience buffer is larger than the corresponding preset lower limit of the buffer, D experience tuples in the experience buffer are randomly sampled to update the network parameters.

[0034] Optionally, the pre-decision processing step includes:

[0035] Acquire the top-level state characteristics at time t+1 from the scheduling environment, preload the current top-level network parameters of the top-level agent to process the top-level state characteristics, and obtain the action corresponding to the top-level agent at time t+1, wherein the action corresponding to the top-level agent includes a top-level action space, and the top-level action space is composed of n workpieces;

[0036] Obtaining the intermediate layer state characteristics of the intermediate layer agent from the scheduling environment through the action corresponding to the top-level agent, preloading the current intermediate layer network parameters of the intermediate layer agent to process the intermediate layer state characteristics, and obtaining the action corresponding to the intermediate layer agent at time t+1, wherein the action corresponding to the intermediate layer agent includes an intermediate layer action space, and the intermediate layer action space is composed of F factories;

[0037] Through the action space corresponding to the middle-layer intelligent agent, the underlying state characteristics of the bottom-level intelligent agent are obtained from the scheduling environment, and the current underlying network parameters of the bottom-level intelligent agent are preloaded to process the underlying state characteristics to obtain the action corresponding to the bottom-level intelligent agent at time t+1, wherein the action corresponding to the middle-layer intelligent agent includes the underlying action space, and the underlying action space is a set of M processing machines that execute the workpiece decision information in the action space corresponding to the top-level intelligent agent.

[0038] Optionally, when the experience buffer is larger than a corresponding preset lower limit of the buffer, the step of randomly sampling D experience tuples in the experience buffer to update the network parameters includes:

[0039] When the top-level experience buffer is larger than the preset lower limit of the top-level buffer, randomly sampling D top-level experience tuples in the top-level experience buffer to update the top-level network parameters;

[0040] When the intermediate layer experience buffer is larger than the preset lower limit of the intermediate layer buffer, randomly sampling D intermediate layer experience tuples in the intermediate layer experience buffer to update the intermediate layer network parameters;

[0041] When the underlying experience buffer is larger than a preset lower limit of the underlying buffer, D underlying experience tuples in the underlying experience buffer are randomly sampled to update underlying network parameters.

[0042] Optionally, the network parameter updating step includes:

[0043] Acquire a target network parameter, wherein the target network parameter includes at least one of a target top-layer network parameter, a target middle-layer network parameter, and a target bottom-layer network parameter;

[0044] Calculate the maximum Q value of the next state of the corresponding intelligent agent according to the target network parameters;

[0045] Determining whether the current state characteristic of the agent is a terminal state;

[0046] If so, the reward r at time t t As TD error, the reward is characterized by the common goal of minimizing the completion time;

[0047] Otherwise, the TD error is calculated as follows:

[0048]

[0049] Where y is the TD error, r t is the reward at time t, γ is the reward discount factor, is the maximum Q value, s t+1 is the current state feature of the agent at time t+1, a t+1 is the action corresponding to the agent at time t+1, θ is the online network parameter: θ′ is the target network parameter;

[0050] After calculating the TD error, according to loss(yQ(s t ,a t ;θ)) 2 Perform gradient descent to update online network parameters;

[0051] After the number of updates meets the preset target network update frequency, the currently obtained online network parameters are used as the target network parameters to complete the network parameter update.

[0052] In addition, to achieve the above-mentioned objectives, the present application also provides a distributed flexible job shop scheduling system, which includes: a memory, a processor, and a distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents stored on the memory and run on the processor. When the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents is executed by the processor, the steps of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer agents as described in any of the above items are implemented.

[0053] This application has at least the following beneficial effects:

[0054] 1. Introducing a multi-level intelligent agent scheduling framework, through the top-level, middle-level and bottom-level intelligent agents working together, through hierarchical decision-making, global complex problems are decomposed into multiple local problems, which improves the decision-making efficiency and scheduling accuracy of the system and achieves a dynamic balance between global scheduling and local optimization;

[0055] 2. Through the decentralized execution of different scheduling tasks of the distributed flexible job shop by agents at all levels, there is no need for a central coordinator to manage every decision. When the environment changes locally, each level can adapt quickly and make the best decision based on its specific context while still serving the overall goal of the entire system;

[0056] 3. Introduce a dual deep reinforcement learning algorithm to centrally train each intelligent agent, allowing the system to access the state, action and reward information of all intelligent agents during the training phase, so that the system can use global environmental information to optimize the strategy of each intelligent agent, so that each intelligent agent can better collaborate and achieve the optimal solution for the overall goal. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the architecture of the hardware operating environment of the distributed flexible job shop scheduling system involved in the embodiment of the present application;

[0058] Figure 2 This is a schematic diagram of the architecture of a distributed flexible job shop scheduling system involved in the first embodiment of the present application;

[0059] Figure 3 This is a flow chart of the first embodiment of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer agents of the present application;

[0060] Figure 4 This is a schematic diagram of a 2×3×3 distributed flexible job shop scheduling example involved in the first embodiment of the present application;

[0061] Figure 5 This is a detailed schematic diagram of the distributed flexible job shop scheduling system architecture involved in the second embodiment of the present application;

[0062] Figure 6 A schematic diagram of a training curve on a 2×6×10 distributed flexible job shop scheduling system example according to the second embodiment of the present application;

[0063] Figure 7 A Gantt chart of the processing time of a factory involved in the second embodiment of the present application;

[0064] Figure 8 This is a Gantt chart of processing time of another factory involved in the second embodiment of the present application.

[0065] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0066] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0067] As an implementation plan, Figure 1 This is a schematic diagram of the architecture of the hardware operating environment of the distributed flexible job shop scheduling system involved in the embodiment of the present application.

[0068] like Figure 1 As shown, the distributed flexible job shop scheduling system may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0069] Those skilled in the art will understand that Figure 1 The distributed flexible job shop scheduling system architecture shown in the figure does not constitute a limitation on the distributed flexible job shop scheduling system, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0070] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents. Among them, the operating system is a program that manages and controls the hardware and software resources of the distributed flexible job shop scheduling system, and the operation of the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents and other software or programs.

[0071] exist Figure 1 In the distributed flexible job shop scheduling system shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used for the background server and communicates data with the background server; the processor 1001 can be used to call the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents stored in the memory 1005.

[0072] In this embodiment, the distributed flexible job shop scheduling system includes: a memory 1005, a processor 1001, and a distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents stored in the memory and executable on the processor, wherein:

[0073] When the processor 1001 calls the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents stored in the memory 1005, the following operations are performed:

[0074] S1, the top-level agent obtains the current top-level state characteristics at time t from the scheduling environment, determines the workpiece decision information according to the current state characteristics, and transmits the workpiece decision information to the middle-level agent and the bottom-level agent;

[0075] S2, the middle-layer agent obtains the current middle-layer state characteristics from the scheduling environment, determines the factory decision information according to the workpiece decision information and the current middle-layer state characteristics, and transmits the factory decision information to the bottom-layer agent;

[0076] S3, the underlying intelligent agent obtains the current underlying state characteristics from the scheduling environment, and determines the equipment decision information according to the workpiece decision information, the factory decision information and the current underlying state characteristics;

[0077] S4, determining joint decision information according to the workpiece decision information, the factory decision information and the equipment decision information, executing a scheduling strategy based on the joint decision information, and performing a state transfer on the scheduling environment, so that the top-level agent obtains the top-level state characteristics and rewards at time t+1, and feeds back to the top-level agent;

[0078] S5, looping through steps S1-S4 until each workpiece to be processed is completed.

[0079] When the processor 1001 calls the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents stored in the memory 1005, the following operations are performed:

[0080] In the initial stage of training, pre-decision processing is performed on the top-level agent, the middle-level agent and the bottom-level agent at time t+1 to obtain experience tuples corresponding to each layer agent, respectively, wherein the experience tuples include the top-level experience tuple, the middle-level experience tuple and the bottom-level experience tuple;

[0081] After the pre-decision processing is completed, the top-level experience tuple, the middle-level experience tuple and the bottom-level experience tuple are respectively stored in their corresponding experience buffers through a buffer controller, and the experience buffer includes a top-level experience buffer, a middle-level experience buffer and a bottom-level experience buffer;

[0082] When the experience buffer is larger than the corresponding preset lower limit of the buffer, D experience tuples in the experience buffer are randomly sampled to update the network parameters.

[0083] When the processor 1001 calls the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents stored in the memory 1005, the following operations are performed:

[0084] Acquire the top-level state characteristics at time t+1 from the scheduling environment, preload the current top-level network parameters of the top-level agent to process the top-level state characteristics, and obtain the action corresponding to the top-level agent at time t+1, wherein the action corresponding to the top-level agent includes a top-level action space, and the top-level action space is composed of n workpieces;

[0085] Obtaining the intermediate layer state characteristics of the intermediate layer agent from the scheduling environment through the action corresponding to the top-level agent, preloading the current intermediate layer network parameters of the intermediate layer agent to process the intermediate layer state characteristics, and obtaining the action corresponding to the intermediate layer agent at time t+1, wherein the action corresponding to the intermediate layer agent includes an intermediate layer action space, and the intermediate layer action space is composed of F factories;

[0086] Through the action space corresponding to the middle-layer intelligent agent, the underlying state characteristics of the bottom-level intelligent agent are obtained from the scheduling environment, and the current underlying network parameters of the bottom-level intelligent agent are preloaded to process the underlying state characteristics to obtain the action corresponding to the bottom-level intelligent agent at time t+1, wherein the action corresponding to the middle-layer intelligent agent includes the underlying action space, and the underlying action space is a set of M processing machines that execute the workpiece decision information in the action space corresponding to the top-level intelligent agent.

[0087] When the processor 1001 calls the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents stored in the memory 1005, the following operations are performed:

[0088] When the top-level experience buffer is larger than the preset lower limit of the top-level buffer, randomly sampling D top-level experience tuples in the top-level experience buffer to update the top-level network parameters;

[0089] When the intermediate layer experience buffer is larger than the preset lower limit of the intermediate layer buffer, randomly sampling D intermediate layer experience tuples in the intermediate layer experience buffer to update the intermediate layer network parameters;

[0090] When the underlying experience buffer is larger than a preset lower limit of the underlying buffer, D underlying experience tuples in the underlying experience buffer are randomly sampled to update underlying network parameters.

[0091] When the processor 1001 calls the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents stored in the memory 1005, the following operations are performed:

[0092] Acquire a target network parameter, wherein the target network parameter includes at least one of a target top-layer network parameter, a target middle-layer network parameter, and a target bottom-layer network parameter;

[0093] Calculate the maximum Q value of the next state of the corresponding intelligent agent according to the target network parameters;

[0094] Determining whether the current state characteristic of the agent is a terminal state;

[0095] If so, the reward r at time t t As TD error, the reward is characterized by the common goal of minimizing the completion time;

[0096] Otherwise, the TD error is calculated as follows:

[0097]

[0098] Where y is the TD error, r t is the reward at time t, γ is the reward discount factor, is the maximum Q value, s t+1 is the current state feature of the agent at time t+1, a t+1 is the action corresponding to the agent at time t+1, θ is the online network parameter: θ′ is the target network parameter;

[0099] After calculating the TD error, according to loss(yQ(s t ,a t ;θ)) 2 Perform gradient descent to update online network parameters;

[0100] After the number of updates meets the preset target network update frequency, the currently obtained online network parameters are used as the target network parameters to complete the network parameter update.

[0101] Based on the hardware architecture of the distributed flexible job shop scheduling system of the above-mentioned intelligent manufacturing technology, an embodiment of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents of the present application is proposed.

[0102] First embodiment

[0103] In the first embodiment, referring to Figure 2The schematic diagram of the architecture of a distributed flexible job shop scheduling system is shown, and the distributed flexible job shop scheduling system includes: a top-level intelligent agent for determining a target workpiece to be processed with the highest processing priority based on workpiece information, an intermediate-level intelligent agent for determining a target processing factory corresponding to the target workpiece to be processed based on factory information, and a bottom-level intelligent agent for determining a target processing machine corresponding to the target workpiece to be processed based on processing machine information in the target processing factory.

[0104] Specifically, the top-level agent is located at the top level of the scheduling architecture and is responsible for the planning and workpiece selection of global production scheduling. Its main function is to select the workpieces that need to be processed first by analyzing the overall production demand and status of the current system. According to the delivery deadline, processing requirements, availability of production resources and other parameters of each workpiece, the priority of all workpieces to be processed is evaluated to determine which workpieces need to be processed first in the current production cycle. After the evaluation is completed, the top-level agent selects the workpiece with the highest priority and suitable for the current production capacity, and prepares to send it to the middle-level agent for more specific factory allocation.

[0105] For the middle-layer agent, it is responsible for assigning the most suitable factory for processing the workpiece selected by the top-level agent. Factories in a distributed manufacturing system may have different processing capabilities, equipment configurations, and resource usage. The task of the middle-layer agent is to maximize the use of these distributed resources. The middle-layer agent monitors the production status of each factory in real time, including data such as equipment availability, the factory's current task load, production capacity, and processing history. This enables it to grasp the real-time status of factory resources. After the workpiece is selected, the middle-layer agent assigns the workpiece to the most suitable factory based on the status of each factory and the processing requirements of the workpiece.

[0106] For the bottom-level agent, the bottom-level agent is responsible for specific machine scheduling to ensure that the workpiece is optimally processed in the designated factory. After the workpiece is assigned to the factory by the middle-level agent, the bottom-level agent selects the most suitable processing equipment for the workpiece based on the specific machine resources within the factory, and schedules and monitors the processing process in real time. The bottom-level agent is not only responsible for machine allocation, but also monitors the processing process in real time to ensure that the processing progress is consistent with the plan.

[0107] In addition, the distributed flexible job shop scheduling system described in this embodiment also includes a double deep reinforcement learning architecture (Double Deep Q Network, DDQN), in which the top-level intelligent agent, the middle-level intelligent agent and the bottom-level intelligent agent all use a double deep reinforcement learning algorithm for parameter optimization during the centralized training process.

[0108] Furthermore, in this embodiment, the distributed flexible job shop scheduling system (hereinafter referred to as the system) includes a scheduling environment as one of the core components, which includes the state space, action space, state transition mechanism, reward function and termination conditions of each level of intelligent agents. The dynamic changes and feedback mechanisms in the environment provide decision-making basis for intelligent agents at all levels, ensuring the smooth execution of the entire scheduling process.

[0109] The state space stores the state features of agents at each level. Each state feature is assigned a timestamp to record the update time of the state feature. Agents at different levels have different state features. Agents can dispatch the environment to obtain features that match their target state features at the current time (i.e., time t) based on the target state features they need to find.

[0110] Further and optionally, the top-level state features in the top-level agent include:

[0111] Processing completed feature CP i (s t ), if all workpieces are processed at time t, then CP i (s t )=1, otherwise CP i (s t )=0;

[0112] Average time consumption feature of workpiece processing TPT i (s t ), which is represented by the average processing time of the workpiece i on all the machinable machines at time t. Its mathematical expression is:

[0113]

[0114] Pre-process characteristics OCT i (s t ), where if the completion time of the predecessor process of the processable process of workpiece i at time t is OCT i (s t )=1, if there is no pre-process, then OCT i (s t )=0;

[0115] Processing time feature PD i (s t ), the processing time feature is characterized by the processing time of workpiece i at time t.

[0116] Further and optionally, the intermediate layer state characteristics in the intermediate layer agent include:

[0117] Factory Load Characteristics FPL f (st ), represented by the load of plant f at time t;

[0118] Pre-dispatch characteristic SPD f (s t ), represented by the predicted maximum completion time obtained by factory f performing complete pre-scheduling on job i selected by the top-level decision action at time t, where the machine with the shortest processing time for each process is selected for pre-scheduling without changing the factory's existing scheduling plan.

[0119] Further and optionally, the underlying state characteristics in the underlying agent include:

[0120] Machine Ready Time Characteristics MCT k (s t ), represented as the ready time of machine m at time t;

[0121] Machine Load Signature MPL k (s t ), represented by the load of machine m at time t, where:

[0122]

[0123] In the formula, O i,j is the jth process of workpiece i, p k,i,j O i,j The processing time for processing on machine k, OPMk is the set of operations that can be processed on machine k;

[0124] Machine utilization characteristic UR m (s t ), represented by the utilization of machine m at time t, where:

[0125] UR m (s t )=MCT k (s t )MPL k (s t )

[0126] Machine processing time characteristics PT j (s t ), represented as the processing time of job i on machine m selected by the top-level decision at time t.

[0127] In a complete scheduling process, there are a number of different states, represented by s tRepresents the current state. The state representation mainly reflects the state characteristics of candidate actions at each level in the state. These states are independent, but interrelated with other states. The state characteristics of the top level can be obtained directly from the scheduling environment, while the state characteristics of the middle and bottom levels need to be obtained from the scheduling environment in combination with the upper-level decisions. The top-level state consists of the characteristics associated with all currently processable operations in all jobs.

[0128] For the action space, each level has an independent action space and At time 0, the first process of all workpieces is processable, and the top-level action space It consists of n workpieces. As the scheduling environment changes, the workpiece processes are gradually completed, resulting in a reduction in the action space. When the action space is zero, it means that all workpieces have been processed, marking the end of scheduling. It consists of F factories. Each workpiece can be assigned to any factory for processing. The underlying action space Is to perform an action A collection of compatible machines.

[0129] The state transition mechanism updates the state features in the state space according to certain judgment conditions. For example, the top-level state features of the top-level agent include the processing completion feature CP i (s t ), if the condition is met: all workpieces are processed at time t, then the state transition mechanism will CP i (s t ) is updated from 0 to 1, otherwise CP i (s t )=0.

[0130] The reward function is also called return. In the system involved in this embodiment, the agents at each layer have the common goal of minimizing the completion time. So the reward r at time t is t =C max (s t )-C max (s t+1 ), for the unprocessed processes, the average value of their available processing time is taken for virtual scheduling filling until all processes are filled and the predicted completion time is obtained. The difference between the two scheduling times is the return at time t.

[0131] Optionally, the system in this embodiment satisfies the following constraints when executing the scheduling task:

[0132] minC max (1)

[0133]

[0134] Among them, formula (1) is the objective function, which means minimizing the completion time;

[0135] Formula (2) indicates that all processes of a workpiece can only be processed in the same factory;

[0136] Formula (3) represents the flexibility constraint of workpiece processing;

[0137] Formula (4) indicates that there is a priority relationship between different processes of the same workpiece;

[0138] Formula (5) indicates that any process of any workpiece can be processed on the machine at time 0;

[0139] Formula (6) indicates that the processing end time of a process cannot be earlier than the start time;

[0140] Equations (7) and (8) indicate that the same machine in a factory can only process one process at the same time;

[0141] Formula (9) indicates that the same process can only be processed by one machine at the same time;

[0142] Formula (10) represents the maximum completion time of the factory;

[0143] Formula (11) represents the maximum completion time of the entire scheduling task

[0144] The meanings of the parameters in the formula are shown in Table 1 below:

[0145]

[0146]

[0147] Furthermore, the system in this embodiment also includes a key architecture: a distributed execution architecture consisting of multiple levels of intelligent agents. Each level can make independent decisions based on its own local observations and information obtained from the upper level during the execution process. Figure 3 The distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer agents includes the following steps:

[0148] S1, the top-level agent obtains the current top-level state characteristics at time t from the scheduling environment, determines the workpiece decision information according to the current state characteristics, and transmits the workpiece decision information to the middle-level agent and the bottom-level agent;

[0149] S2, the middle-layer agent obtains the current middle-layer state characteristics from the scheduling environment, determines the factory decision information according to the workpiece decision information and the current middle-layer state characteristics, and transmits the factory decision information to the bottom-layer agent;

[0150] S3, the underlying intelligent agent obtains the current underlying state characteristics from the scheduling environment, and determines the equipment decision information according to the workpiece decision information, the factory decision information and the current underlying state characteristics;

[0151] S4, determining joint decision information according to the workpiece decision information, the factory decision information and the equipment decision information, executing a scheduling strategy based on the joint decision information, and performing a state transfer on the scheduling environment, so that the top-level agent obtains the top-level state characteristics and rewards at time t+1, and feeds back to the top-level agent;

[0152] S5, looping through steps S1-S4 until each workpiece to be processed is completed.

[0153] In this embodiment, in a complete scheduling process, the top-level agent first obtains the current top-level state characteristics at time t as the current time from the scheduling environment, and the top-level agent determines the workpiece decision information based on the current state characteristics, and then transmits the workpiece decision information to the lower-level agents, namely the middle-level agent and the bottom-level agent; then, the middle-level agent obtains the current middle-level state characteristics from the scheduling environment, determines the factory decision information based on the workpiece decision information and the current middle-level state characteristics, and transmits the factory decision information to the bottom-level agent; finally, the bottom-level agent obtains the current bottom-level state characteristics from the scheduling environment, and determines the equipment decision information based on the workpiece decision information, the factory decision information and the current bottom-level state characteristics.

[0154] In this embodiment, the top layer only needs to be based on its current state characteristics Make the current decision Then the decision information Passed to the middle layer and the bottom layer, the middle layer receives the decision information Finally, combine the scheduling environment to obtain the status characteristics of the middle layer Make decisions independently and share decision information Passed to the bottom layer, after receiving all the decision information, the bottom layer also combines the scheduling environment to obtain the bottom layer status characteristics and make independent decisions.

[0155] Furthermore, when all three levels have made a decision, the three decision information are combined to obtain the joint decision information. And perform a state transfer on the environment to get the state at the next moment and r t , repeat the above process until all workpieces are processed.

[0156] For example, refer to Figure 4The diagram shows a 2×3×3 distributed flexible job shop scheduling example. At time t (left) and t+1 (right), the red circle workpiece node represents the process that can be processed at the current time t. The solid line with color connected to it represents the processing factory and the machine in the factory that has been selected for the process. The dotted line represents the optional machine. At time t, the instance environment executes the joint action pair (2,1,3), assigns workpiece 2 to factory 1, and processes the process that can be processed in factory 2 on machine 3 in factory 1. Then the state is transferred to time t+1.

[0157] In the technical solution provided in this embodiment, a multi-level intelligent agent scheduling framework is introduced, and the top-level, middle-level and bottom-level intelligent agents work together. Through hierarchical decision-making, global complex problems are decomposed into multiple local problems, thereby improving the decision-making efficiency and scheduling accuracy of the system, and achieving a dynamic balance between global scheduling and local optimization. Different scheduling tasks of distributed flexible job shops are decentralizedly executed by intelligent agents at various levels, without the need for a central coordinator to manage each decision. When the environment changes locally, each level can adapt quickly and make the best decision based on its specific context while still serving the overall goal of the entire system.

[0158] Second embodiment

[0159] Based on the first embodiment, refer to Figure 5 The schematic diagram of the distributed flexible job shop scheduling system architecture is shown. Regarding how to optimize the parameters of the agents at each level in the system based on the dual deep reinforcement learning algorithm, this embodiment adds the following to the architecture of the first embodiment: Figure 5 The multiple levels of experience buffers and corresponding networks shown on the right are used to update the system's network parameters, helping each intelligent agent to establish a unified optimization goal and preventing each intelligent agent from focusing too much on its own goals and ignoring overall collaboration.

[0160] Here is how to focus on training:

[0161] Step S10, in the initial stage of training, pre-decision processing is performed on the top-level agent, the middle-level agent and the bottom-level agent at time t+1, and experience tuples corresponding to each layer agent are obtained respectively, wherein the experience tuples include the top-level experience tuple, the middle-level experience tuple and the bottom-level experience tuple;

[0162] In this step, at the initial stage of training, the state at time t+1 required by the middle and bottom layers cannot be directly obtained from the environment, because the bottom layer needs the decision of the upper layer to obtain the current state from the environment. At time t+1, the upper layer has not made any decision, so it is necessary to make a preliminary decision for each layer at time t+1.

[0163] Optionally, the pre-decision processing step includes:

[0164] Step S11, obtaining the top-level state characteristics at time t+1 from the scheduling environment, preloading the current top-level network parameters of the top-level agent to process the top-level state characteristics, and obtaining the action corresponding to the top-level agent at time t+1, wherein the action corresponding to the top-level agent includes a top-level action space, and the top-level action space is composed of n workpieces;

[0165] Step S12, obtaining the intermediate layer state characteristics of the intermediate layer agent from the scheduling environment through the action corresponding to the top-level agent, preloading the current intermediate layer network parameters of the intermediate layer agent to process the intermediate layer state characteristics, and obtaining the action corresponding to the intermediate layer agent at time t+1, wherein the action corresponding to the intermediate layer agent includes an intermediate layer action space, and the intermediate layer action space is composed of F factories;

[0166] Step S13, through the action space corresponding to the middle-layer intelligent agent, obtain the underlying state characteristics of the bottom-level intelligent agent from the scheduling environment, preload the current underlying network parameters of the bottom-level intelligent agent to process the underlying state characteristics, and obtain the action corresponding to the bottom-level intelligent agent at time t+1, wherein the action corresponding to the middle-layer intelligent agent includes the underlying action space, and the underlying action space is a set of M processing machines that execute the workpiece decision information in the action space corresponding to the top-level intelligent agent.

[0167] For the action space, each level has an independent action space and At time 0, the first process of all workpieces is processable, and the top-level action space It consists of n workpieces. As the scheduling environment changes, the workpiece processes are gradually completed, resulting in a reduction in the action space. When the action space is zero, it means that all workpieces have been processed, marking the end of scheduling. It consists of F factories. Each workpiece can be assigned to any factory for processing. The underlying action space Is to perform an action A collection of compatible machines.

[0168] In the pre-decision processing step, the top-level state features at time t+1 are obtained from the environment After that, preload the top-level current online network parameter pair Process the top-level action at time t+1 in, Then through Get the middle layer from the environment Repeat the operation on the middle layer to obtain the bottom layer Thus, the pre-decision processing is completed.

[0169] Step S20, after the pre-decision processing is completed, the top-level experience tuple, the middle-level experience tuple and the bottom-level experience tuple are respectively stored in their corresponding experience buffers through a buffer controller, and the experience buffer includes a top-level experience buffer, a middle-level experience buffer and a bottom-level experience buffer;

[0170] Step S30: When the experience buffer is larger than the corresponding preset lower limit of the buffer, D experience tuples in the experience buffer are randomly sampled to update the network parameters.

[0171] Furthermore, after the pre-decision processing is completed, the buffer controller sends the corresponding experience tuples of each layer to The experience buffer is used to store the corresponding experience buffer for the agent to evenly extract samples from its corresponding experience buffer for centralized training when the agent needs to update parameters.

[0172] Furthermore, after the experience tuple is stored in the experience buffer, when it is determined that the experience buffer is larger than its corresponding preset lower limit, it is determined that the network parameters need to be updated, and D experience tuples in the experience buffer are randomly sampled to update the network parameters.

[0173] Further and optionally, step S30 specifically includes:

[0174] Step S31, when the top-level experience buffer is larger than the preset lower limit of the top-level buffer, randomly sampling D top-level experience tuples in the top-level experience buffer to update the top-level network parameters;

[0175] Step S32, when the intermediate layer experience buffer is greater than a preset lower limit of the intermediate layer buffer, randomly sampling D intermediate layer experience tuples in the intermediate layer experience buffer to update the intermediate layer network parameters;

[0176] Step S33: When the underlying experience buffer is larger than a preset lower limit of the underlying buffer, D underlying experience tuples in the underlying experience buffer are randomly sampled to update underlying network parameters.

[0177] For example, when the top experience buffer R J Size > top experience buffer lower limit B wJM t, then randomly sample experience tuples of batch size D in the top-level experience buffer Update the top layer network parameters. The same goes for the remaining layers.

[0178] As an optional implementation, the pseudo code execution logic set based on the above centralized training logic is given below:

[0179]

[0180]

[0181]

[0182] Further and optionally, in this embodiment, the step of updating the network parameters includes:

[0183] Step S100, obtaining target network parameters, wherein the target network parameters include at least one of target top-layer network parameters, target middle-layer network parameters and target bottom-layer network parameters;

[0184] Step S200, calculating the maximum Q value of the next state of the corresponding agent according to the target network parameter;

[0185] Step S300, determining whether the current state characteristic of the agent is a terminal state;

[0186] Step S400: If yes, set the reward r at time t t As TD error, the reward is characterized by the common goal of minimizing the completion time;

[0187] Step S500, otherwise, calculate the TD error according to the following formula:

[0188]

[0189] Where y is the TD error, r t is the reward at time t, γ is the reward discount factor, is the maximum Q value, s t+1 is the current state feature of the agent at time t+1, a t+1 is the action corresponding to the agent at time t+1, θ is the online network parameter: θ′ is the target network parameter;

[0190] Step S600, after calculating the TD error, according to loss(yQ(s t ,a t ;θ)) 2 Perform gradient descent to update online network parameters;

[0191] Step S700, after the update times meet the preset target network update frequency, the currently obtained online network parameters are used as the target network parameters to complete the network parameter update.

[0192] As an optional implementation, the pseudo code execution logic set based on the update of the above network parameters is given below:

[0193]

[0194] For example, Figure 6 The following is a diagram of the training curve on a 2×6×10 distributed flexible job shop scheduling system instance. The X-axis represents the number of training iterations, the Y-axis represents the maximum completion time, the yellow curve represents the average completion time change curve of the 100 validation set instances that did not participate in the training, and the blue represents the maximum completion time of the instances that participated in the actual training in each iteration of the training set. It can be seen that the convergence of the validation set is completely consistent with the training set, and good convergence can be achieved even on data that has not appeared before. Figure 7 and Figure 8 They are the Gantt charts of the processing time of the two factories under this scale example.

[0195] In the technical solution provided in this embodiment, a dual deep reinforcement learning algorithm is introduced to conduct centralized training of each intelligent agent, allowing the system to access the status, action and reward information of all intelligent agents during the training phase, so that the system can use the global environmental information to optimize the strategy of each intelligent agent, so that the various intelligent agents can better collaborate and achieve the optimal solution for the overall goal.

[0196] In addition, it can be understood by a person skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the distributed flexible job shop scheduling system to implement the process steps of the embodiment of the above method.

[0197] Therefore, the present application also provides a computer-readable storage medium, which stores a distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents. When the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer intelligent agents is executed by a processor, it implements the various steps of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer intelligent agents described in the above embodiment.

[0198] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which may store program codes.

[0199] It should be noted that since the storage medium provided in the embodiment of the present application is the storage medium used to implement the method of the embodiment of the present application, based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the storage medium, so it is not repeated here. All storage media used in the method of the embodiment of the present application belong to the scope of protection of this application.

[0200] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0201] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0204] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present application may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0205] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0206] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer agents, characterized in that: Applied to a distributed flexible job shop scheduling system, the system comprises: a top-level intelligent agent for determining a target workpiece to be processed with the highest processing priority according to workpiece information, a middle-level intelligent agent for determining a target processing factory corresponding to the target workpiece to be processed according to factory information, and a bottom-level intelligent agent for determining a target processing machine corresponding to the target workpiece to be processed according to processing machine information in the target processing factory. The top-level intelligent agent, the middle-level intelligent agent and the bottom-level intelligent agent all use a dual deep reinforcement learning algorithm for parameter optimization during centralized training. The method comprises the following steps: S1, the top-level agent obtains the current top-level state characteristics at time t from the scheduling environment, determines the workpiece decision information according to the current state characteristics, and transmits the workpiece decision information to the middle-level agent and the bottom-level agent; S2, the middle-layer agent obtains the current middle-layer state characteristics from the scheduling environment, determines the factory decision information according to the workpiece decision information and the current middle-layer state characteristics, and transmits the factory decision information to the bottom-layer agent; S3, the underlying intelligent agent obtains the current underlying state characteristics from the scheduling environment, and determines the equipment decision information according to the workpiece decision information, the factory decision information and the current underlying state characteristics; S4, determining joint decision information according to the workpiece decision information, the factory decision information and the equipment decision information, executing a scheduling strategy based on the joint decision information, and performing a state transfer on the scheduling environment, so that the top-level agent obtains the top-level state characteristics and rewards at time t+1, and feeds back to the top-level agent; S5, looping through steps S1-S4 until each workpiece to be processed is completed.

2. The method according to claim 1, characterized in that The top-level state features in the top-level agent include: Processing completed feature CP i (s t ), if all workpieces are processed at time t, then CP i (s t )=1, otherwise CP i (s t )=0; Average time consumption feature of workpiece processing TPT i (s t ), which is represented by the average processing time of the workpiece i on all the machinable machines at time t. Its mathematical expression is: Pre-process characteristics OCT i (s t ), where if the completion time of the predecessor process of the processable process of workpiece i at time t is OCT i (s t )=1, if there is no preceding process, then OCT i (s t )=0; Processing time feature PD i (s t ), the processing time feature is characterized by the processing time of workpiece i at time t.

3. The method according to claim 1, characterized in that The intermediate layer state characteristics in the intermediate layer agent include: Factory Load Characteristics FPL f (s t ), represented by the load of plant f at time t; Pre-dispatch characteristic SPD f (s t ), represented by the predicted maximum completion time obtained by factory f performing complete pre-scheduling on job i selected by the top-level decision action at time t, where the machine with the shortest processing time for each process is selected for pre-scheduling without changing the factory's existing scheduling plan.

4. The method according to claim 1, characterized in that The underlying state characteristics in the underlying agent include: Machine Ready Time Characteristics MCT k (s t ), represented as the ready time of machine m at time t; Machine Load Signature MPL k (s t ), represented by the load of machine m at time t, where: In the formula, O i,j is the jth process of workpiece i, p k,i,j O i,j The processing time for processing on machine k, OPMk is the set of operations that can be processed on machine k; Machine utilization characteristic UR m (s t ), represented by the utilization of machine m at time t, where: UR m (s t )=MCT k (s t )MPL k (s t ) Machine processing time characteristics PT j (s t ), represented as the processing time of job i on machine m selected by the top-level decision at time t.

5. The method according to claim 1, characterized in that The steps of optimizing parameters of the top-level agent, the middle-level agent, and the bottom-level agent using a dual-depth reinforcement learning algorithm during centralized training include: In the initial stage of training, pre-decision processing is performed on the top-level agent, the middle-level agent and the bottom-level agent at time t+1 to obtain experience tuples corresponding to each layer agent, respectively, wherein the experience tuples include the top-level experience tuple, the middle-level experience tuple and the bottom-level experience tuple; After the pre-decision processing is completed, the top-level experience tuple, the middle-level experience tuple and the bottom-level experience tuple are respectively stored in their corresponding experience buffers through a buffer controller, and the experience buffer includes a top-level experience buffer, a middle-level experience buffer and a bottom-level experience buffer; When the experience buffer is larger than the corresponding preset lower limit of the buffer, D experience tuples in the experience buffer are randomly sampled to update the network parameters.

6. The method according to claim 5, characterized in that The steps of the pre-decision processing include: Acquire the top-level state characteristics at time t+1 from the scheduling environment, preload the current top-level network parameters of the top-level agent to process the top-level state characteristics, and obtain the action corresponding to the top-level agent at time t+1, wherein the action corresponding to the top-level agent includes a top-level action space, and the top-level action space is composed of n workpieces; Obtaining the intermediate layer state characteristics of the intermediate layer agent from the scheduling environment through the action corresponding to the top-level agent, preloading the current intermediate layer network parameters of the intermediate layer agent to process the intermediate layer state characteristics, and obtaining the action corresponding to the intermediate layer agent at time t+1, wherein the action corresponding to the intermediate layer agent includes an intermediate layer action space, and the intermediate layer action space is composed of F factories; Through the action space corresponding to the middle-layer intelligent agent, the underlying state characteristics of the bottom-level intelligent agent are obtained from the scheduling environment, and the current underlying network parameters of the bottom-level intelligent agent are preloaded to process the underlying state characteristics to obtain the action corresponding to the bottom-level intelligent agent at time t+1, wherein the action corresponding to the middle-layer intelligent agent includes the underlying action space, and the underlying action space is a set of M processing machines that execute the workpiece decision information in the action space corresponding to the top-level intelligent agent.

7. The method according to claim 5, characterized in that When the experience buffer is larger than the corresponding preset lower limit of the buffer, the step of randomly sampling D experience tuples in the experience buffer to update the network parameters includes: When the top-level experience buffer is larger than the preset lower limit of the top-level buffer, randomly sampling D top-level experience tuples in the top-level experience buffer to update the top-level network parameters; When the intermediate layer experience buffer is larger than the preset lower limit of the intermediate layer buffer, randomly sampling D intermediate layer experience tuples in the intermediate layer experience buffer to update the intermediate layer network parameters; When the underlying experience buffer is larger than a preset lower limit of the underlying buffer, D underlying experience tuples in the underlying experience buffer are randomly sampled to update underlying network parameters.

8. The method according to claim 5, characterized in that The network parameter updating step comprises: Acquire a target network parameter, wherein the target network parameter includes at least one of a target top-layer network parameter, a target middle-layer network parameter, and a target bottom-layer network parameter; Calculate the maximum Q value of the next state of the corresponding intelligent agent according to the target network parameters; Determining whether the current state characteristic of the agent is a terminal state; If so, the reward r at time t t As TD error, the reward is characterized by the common goal of minimizing the completion time; Otherwise, the TD error is calculated as follows: Where y is the TD error, r t is the reward at time t, γ is the reward discount factor, is the maximum Q value, s t+1 is the current state feature of the agent at time t+1, a t+1 is the action corresponding to the agent at time t+1, θ is the online network parameter: θ′ is the target network parameter; After calculating the TD error, according to loss(yQ(s t ,a t ;θ)) 2 Perform gradient descent to update online network parameters; After the number of updates meets the preset target network update frequency, the currently obtained online network parameters are used as the target network parameters to complete the network parameter update.

9. A distributed flexible job shop scheduling system, characterized in that: The distributed flexible job shop scheduling system includes: a memory, a processor, and a distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents stored in the memory and executable on the processor. When the distributed flexible job shop scheduling program based on dual deep reinforcement learning and multi-layer agents is executed by the processor, the steps of the distributed flexible job shop scheduling method based on dual deep reinforcement learning and multi-layer agents as described in any one of claims 1 to 8 are implemented.

Images