A multi-agent approach to solving dynamic flexible job shop scheduling based on D3QN network

Abstract

The present invention discloses a method for solving dynamic flexible job shop scheduling based on a multi-agent D3QN network, which belongs to the field of deep reinforcement learning and production scheduling technology of intelligent agents. The present invention can monitor the dynamic changes of processing workpieces and machines in real time, and promptly and efficiently give real-time scheduling decisions for different emergencies, such as emergency insertion of workpieces, machine damage, etc., effectively reducing the time cost and resource cost of production. The method includes: in the actual production process of the dynamic flexible job shop, real-time acquisition of processing information in the dynamic flexible job shop, the processing information includes processing workpiece information and processing machine information; when it is determined based on the processing information that a rescheduling event occurs, a rescheduling strategy is generated based on the mathematical model of the dynamic flexible job shop scheduling problem for the real-time status of the workpiece and the machine at the rescheduling point where the dynamic event occurs, and the rescheduling is executed until the production process is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of deep reinforcement learning of intelligent agents and production scheduling, and in particular to a method for solving dynamic flexible job shop scheduling based on a D3QN network using multiple intelligent agents. Background Art

[0002] Production scheduling is an essential and core component of production management for manufacturing enterprises, directly impacting production costs and productivity. It is a crucial component of modern supply chains and manufacturing systems, as appropriate production scheduling solutions can improve machine utilization, ensure timely delivery, and reduce inventory costs. The flexible job-shop scheduling problem (FJSP) is one of the most widely used problems in this field. FJSP is an extension of the job-shop scheduling problem. The number of processing machines for each process is uncertain. Each process step for each workpiece can be performed on multiple selectable machines, and the processing time required for each machine varies. FJSP has been extensively studied for decades. Many existing approaches to address FJSP are based on a static job-shop production environment and provide a deterministic scheduling solution. However, this approach differs from reality. In real-world manufacturing, typical internal disruptions, such as machine breakdown and operator absence, require swift handling to ensure efficient production. Furthermore, in a fiercely competitive market, companies strive to enhance their competitiveness by offering customers a high degree of flexibility, which in turn leads to a rapid increase in order additions and cancellations. Consequently, existing FJSP scheduling solutions may no longer be suitable for dynamic production systems and may even result in poor scheduling performance. Therefore, it is necessary to develop dynamic scheduling to handle these interference events so that the production system can operate continuously and efficiently.

[0003] In recent years, research on dynamic FJSP (DFJSP) has garnered significant attention. Numerous approaches have been proposed to address this problem, with scheduling rules and metaheuristics being the most widely used. Scheduling rules immediately respond to dynamic events, achieving optimal time efficiency. However, scheduling rules cannot guarantee local optimality for decisions made at the time of dynamic events. Furthermore, due to the variability of the dynamic environment, selecting the optimal rule at a specific time point is challenging and highly dependent on expert knowledge. The basic idea of ​​metaheuristics is to decompose the dynamic scheduling problem into corresponding static scheduling subproblems. These static problems are then solved using metaheuristic algorithms, such as genetic algorithms (GAs), ant colony algorithms (ACOs), particle swarm optimization (PSOs), and artificial bee colony algorithms (ABCs). These solutions (for new workpiece insertion) often offer higher quality, but are often accompanied by high time consumption and infeasible real-time scheduling. The primary motivation of this paper is to determine how to select the appropriate scheduling rule from a multitude of available scheduling rules at each decision point, while ensuring real-time efficiency in the ever-changing dynamic environment of the shop floor.

[0004] In order to select the most appropriate scheduling rule at each scheduling decision time point, the dynamic scheduling process is regarded as a Markov decision process (MDP), and a dynamic selection strategy is formulated through the deep reinforcement learning (DRL) algorithm.

[0005] Traditional deep reinforcement learning considers only a single agent and controls it through a DRL algorithm. However, as job shop scheduling environments become increasingly complex, traditional deep reinforcement learning algorithms suffer from poor scalability. When multiple agents explore the environment simultaneously, training becomes chaotic and inefficient as the dimensionality of the action space increases. This challenge is exacerbated in dynamic job shop scheduling problems, as the results (rewards) of agent actions are often delayed and sparse, making it difficult to allocate credit for rewards and formulate cooperative strategies for all agents. Summary of the Invention

[0006] To overcome the above technical deficiencies, the present invention provides a multi-agent method for solving dynamic flexible job shop scheduling based on a D3QN network. The main scheme is as follows:

[0007] During the actual production process of the dynamic flexible job shop, the processing information in the dynamic flexible job shop is obtained in real time, and the processing information includes the processing workpiece information and the processing machine information;

[0008] When a rescheduling event is determined to occur based on processing information, a rescheduling strategy is generated based on the mathematical model of the dynamic flexible job shop scheduling problem according to the real-time status of the workpiece and machine at the rescheduling point where the dynamic event occurs, and the rescheduling is executed until the production process is completed;

[0009] The mathematical model of the dynamic flexible job shop scheduling problem is constructed based on the initial parameter information of the dynamic flexible job shop, including scheduling objectives and constraints. The scheduling objectives are: minimizing the total delay time of all workpieces and minimizing the variance of all machine utilization. The constraints are: n workpieces have a total of O processes and m machines, each workpiece has a fixed process sequence, and each process has a fixed set of processing machines. These include the general constraints and dynamic constraints of FJSP.

[0010] Rescheduling events are dynamic / uncertain events including urgent workpiece insertion and unexpected machine failure. The rescheduling point is the time point when the workpiece is urgently inserted or the machine fails.

[0011] Optionally, in some possible implementations, generating a rescheduling policy and performing specific rescheduling operations include:

[0012] 1) Update of workpiece and machine status at rescheduling point:

[0013] Based on the obtained processing information, initially, the rescheduling point workpiece and machine status update module updates the action status information, and obtains the optional workpiece set based on whether the predecessor process of the current process of the workpiece has been processed; the optional machine set is obtained based on the processing machine set and current machine status set of the current process of the workpiece; when a rescheduling event occurs, the rescheduling point workpiece and machine status update module updates the status information again until all workpieces are processed.

[0014] 2) The artifact agent generates artifact scheduling information:

[0015] After the status information of the rescheduling point is updated, the workpiece agent allocates the next processing workpiece according to the optional workpiece set and generates workpiece scheduling information, which includes the workpiece serial number and process serial number.

[0016] 3) The machine agent generates machine scheduling information:

[0017] Based on the workpiece scheduling information and the rescheduling optional machine set, the machine agent arranges the processing machine of the workpiece and generates machine scheduling information, which includes the machine serial number.

[0018] Optionally, in some possible implementations, the general constraints of FJSP include: ① Every machine is available at time zero, ② All arrived workpieces can be processed at time zero, ③ There is no priority between the processes of different workpieces, ④ Each machine can only process at most one operation at a time, ⑤ Each operation should be processed in a non-preemptive manner without interruption, and ⑥ All operations belonging to the same job should be processed one after another in a fixed order, i.e., priority constraints.

[0019] Optionally, in some possible implementations, the dynamic constraints include: ①) transportation time and setup time are negligible, ② the buffer between machines is unlimited, ③ if an operation is interrupted due to a machine failure, the remaining processing time is equal to the total processing time minus the completed processing time, ④ each job contains different types of operations, and each job has a delivery deadline that must be met, otherwise it will be considered timed out.

[0020] Alternatively, in some possible implementations, minimizing the total delay time of all workpieces is expressed as: .

[0021] Alternatively, in some possible implementations, minimizing the variance of all machine utilizations is expressed as: .

[0022] In addition, the present invention also provides a multi-agent scheduling system, including: a rescheduling point workpiece and machine status update module, a workpiece agent module, a machine agent module, and a joint reward mapping agent module, which is used to execute the above-mentioned D3QN network-based multi-agent solution method for dynamic flexible job shop scheduling.

[0023] The present invention adopts the above technical solution to achieve the following technical effects:

[0024] By introducing a multi-agent scheduling system based on D3QN, the dynamic flexible job shop scheduling problem is solved. In view of the variability and randomness existing in the scheduling problem, a multi-agent and distributed agent architecture is adopted to distribute and parallelize the problem-solving steps, and achieve the consistency from step optimization to global optimization. In the actual factory processing process, the invention of this article can monitor the dynamic changes of the processed workpieces and machines in real time, and make real-time scheduling decisions in a timely and efficient manner for different emergencies, such as emergency insertion of workpieces, machine damage, etc., effectively reducing the time cost and resource cost of production. Different from the traditional single-agent deep reinforcement learning method, the multi-agent scheduling system of the present invention can meet both the high efficiency of production and processing and the intelligence of production resource allocation when dealing with production scheduling problems with huge data volumes and a large number of emergencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0026] Figure 1Schematic diagram of the basic structure of the interaction between the reinforcement learning agent and the environment in the present invention;

[0027] Figure 2 A flowchart of a method for solving dynamic flexible job shop scheduling based on a multi-agent D3QN network in the present invention;

[0028] Figure 3 A flowchart of the rescheduling process in the method for solving dynamic flexible job shop scheduling based on a multi-agent D3QN network in the present invention;

[0029] Figure 4 A flowchart of an optional machine set update method in the present invention;

[0030] Figure 5 This is a system diagram of the D3QN network-based multi-agent solution to the dynamic flexible job shop scheduling network framework in the present invention;

[0031] Figure 6 The scheduling Gantt chart for the example of the present invention where the number of workpieces is initialized to 14, the number of new workpieces is inserted to 8, and the number of machines is 8;

[0032] Figure 7 Schematic diagram of the change in total delay time during the training phase of the D3QN network-based multi-agent solution for dynamic flexible job shop scheduling in the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0034] It should be noted that the descriptions of "first", "second", etc. in the embodiments of the present invention are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0035] In the description of the present invention, it should be understood that the numerical labels before the steps do not identify the order in which the steps are executed, but are only used to facilitate the description of the present invention and to distinguish each step, and therefore should not be understood as a limitation of the present invention.

[0036] Reinforcement learning (RL) is one of the most powerful sequential decision-making tools in Markov decision processes (MDPs). It is suitable for solving dynamic optimization problems in workshop production and can achieve high average performance. The RL algorithm drives the learning of intelligent agents through rewards. The intelligent agents interact with the environment, perform operations on the environment based on the perceived state, and then obtain rewards from the environment. Figure 1 , Figure 1 middle Indicates the action at the moment. represents the reward from time t to time t+1, represents the state from time t to time t+1. Through continuous interaction, the agent is continuously trained by the RL algorithm to learn the optimal action policy, thereby solving real-time control tasks. In previous research, the RL algorithm Q-learning selects the most appropriate allocation rule for different states by constructing a Q-table. This Q-table displays the estimated maximum value of the Q function for all discrete state-action pairs. However, due to the complex environment of real production workshops with a large state space and a large action space, it is impossible for a Q-table to list all states, limiting the application of RL in dynamic scheduling. However, with the rise of deep learning (DL), DL research results in fields such as image recognition and natural language processing have demonstrated its powerful fitting and representation capabilities for processing high-dimensional data. The characteristics of DL provide new approaches for solving high-dimensional problems. Deep learning has accelerated the progress of reinforcement learning, and the use of deep learning algorithms in RL has defined "deep reinforcement learning" (DRL). Currently, deep Q-networks (DQNs) are one of the most commonly used DRL methods in DRL research. DQNs are expected to efficiently learn nonlinear relationships between states and actions, achieving optimal policies in complex environments. The DQN algorithm can be used to train intelligent agents to select the most appropriate action at each decision point. DRL combines the powerful representation capabilities of deep neural networks with the decision-making power of reinforcement learning, making it highly effective for solving decision-making problems with high-dimensional state and action spaces.

[0037] The purpose of this invention is to address the current difficulties in allocating reward credits to multiple agents in dynamic and flexible job shops, which results in disrupted training between agents, poor scheduling efficiency, and wasted production resources. In this patent, the job shop is represented as a multi-agent system consisting of a workpiece agent, a machine agent, and a joint reward agent. The workpiece agent selects the appropriate workpiece to be processed, the machine agent selects the appropriate processing machine, and the joint reward agent generates a mapping of joint rewards. This patent utilizes a monotonic reward function decomposition algorithm and a neural network to approximate the joint reward of all agent actions. This method can represent the corresponding relationship between the joint reward and the reward of each agent, ultimately forming a scheduling production strategy.

[0038] In the technical solution of the embodiment of the present invention, first, a mathematical model for solving the dynamic flexible job shop scheduling problem using a multi-agent network based on the D3QN network is constructed. The construction scheme and model are as follows:

[0039] 1) Obtain the initial parameter information of the dynamic flexible job shop.

[0040] The parameter information includes: the number of workpieces, the number of machines, the number of newly inserted workpieces, the status of the machine, the number of processes, the processing relationship between the workpieces and the machine, the process sequence of the workpiece processing, and the processing time of the workpiece on the machine.

[0041] It should be noted that the parameter information here is the initial parameter information, which represents the overall workpiece and machine information obtained before scheduling begins; the workpiece and machine information in the scheduling process below refers to the information obtained at that moment during each rescheduling.

[0042] 2) Based on the obtained parameter information of the dynamic flexible job shop, a mathematical model of the dynamic flexible job shop scheduling problem is constructed, including constraints and scheduling objectives.

[0043] Due to the presence of urgent workpiece insertions and unexpected machine failures, the goal of this study is to minimize the total delay time of all jobs and the variance of the system's machine utilization in the event of such dynamic events. Therefore, the scheduling objectives of the mathematical model in this invention are: minimizing the total delay time of all jobs and minimizing the variance of all machine utilization; minimizing the total delay time of all jobs is defined as the time when the actual processing completion time of all jobs is greater than the sum of the extensions to the construction period after scheduling is completed; and the variance of machine utilization is defined as the average of the squared differences between the utilization of each machine and the overall average utilization after scheduling is completed.

[0044] Furthermore, the constraints are: there are a total of O processes for n workpieces and m machines. The process processing of each workpiece has a fixed order, and each process has a fixed set of processing machines. n is the total number of workpieces, and m is the total number of machines.

[0045] Specifically, the constraints include general constraints and dynamic constraints of FJSP.

[0046] Among them, the general constraints of FJSP are as follows:

[0047] ①Every machine is available at time zero.

[0048] ②All workpieces can be processed at time zero.

[0049] ③ There is no priority between the processes of different workpieces.

[0050] ④Each machine can only process one operation at a time.

[0051] ⑤ Each operation should be processed in a non-preemptive manner without interruption.

[0052] ⑥ All operations belonging to the same job should be processed one after another in a fixed order (precedence constraint).

[0053] Correspondingly, the dynamic constraints are as follows:

[0054] ①Transportation time and setup time can be ignored.

[0055] ②The buffer between machines is not limited.

[0056] ③ If the operation is interrupted due to machine failure, the remaining processing time is equal to the total processing time minus the completed processing time.

[0057] ④Each job contains different types of operations, and each job has a delivery deadline that must be met, otherwise it will be considered timed out.

[0058] A mathematical model based on the D3QN network and multi-agent solution to the dynamic flexible job shop scheduling problem is constructed. The dynamic flexible job shop scheduling problem is briefly described as follows:

[0059] have assignments exist Machine Each operation Include operations, Represents the set representation of O processes. Indicates homework No. Operations. Processing Any processing machine All belong to the set of processable machines .machine Actual processing The time is , Represented as an operation Completion time, workpiece The arrival time and duration are and .

[0060] The data model constructed based on the operations described in 1) and 2) above is as follows:

[0061] (1)

[0062] Where, Indicates the total number of jobs, Indicates the index of the job. ∈ ; express The completion time of the workpiece after each operation; Represents workpiece duration of the project; Indicates the part of the completion time that exceeds the construction period;

[0063] (2)

[0064] Where, represents the variance of machine utilization in the entire system; m represents the total number of machines; represents the total processing time of the kth machine, k∈m; represents the processing time of all machines, l∈m, k∈l;

[0065] (3)

[0066] Where, Indicates that it belongs to the artifact The number of operations, An index indicating the number of operations. ∈ ; Indicates an operation completion time;

[0067] (4)

[0068] Where, =0 means initial operation The completion time is 0;

[0069] (5)

[0070] Where, Indicates an operation The set of processable machines; Represents a binary variable, that is, an operation Whether to allocate to the kth machine;

[0071] (6)

[0072] Where, Indicates an operation The start time of Indicates an operation In the machine The processing time on Indicates the first machines;

[0073] (7)

[0074] In the formula, when =1, Indicates the completion time of the first operation; Indicates the machine under the first operation Processing time on Represents workpiece Arrival time; represents the binary variable under the first operation;

[0075] (8)

[0076] Where, Indicates the completion time of the predecessor operation on the same workpiece;

[0077] (9)

[0078] Where, represents the completion time of other workpiece operations, i.e., the machine conflict constraint; Indicates the processing time of other workpiece operations; binary variables representing other artifact operations; Indicates an operation and Indicates the order in which other workpieces are operated on the same machine;

[0079] (10)

[0080] Where, Represents workpiece Non-processing time before completion time; Indicates the time at the rescheduling point after the dynamic time occurs; Represents workpiece duration of the project;

[0081] (11)

[0082] Where, Indicates an operation Average processing time on the machines it can process;

[0083] (12)

[0084] Where, Represents workpiece The remaining operation preprocessing time; Indicates the sequence number of the operation to be performed at the rescheduling point after a dynamic event occurs;

[0085] (13)

[0086] Where, express The idle delay time of the workpiece;

[0087] (14)

[0088] Where, express Delay time of workpiece; Indicates the final completion time;

[0089] (15)

[0090] Where, Indicates the total delay rate caused by scheduling;

[0091] (16);

[0092] Formula (1) represents the total delay of all jobs. Formula (2) represents the variance of machine utilization in the entire system. Formula (3) represents the sum of the completion times of all jobs. Formula (4) indicates that the completion time of each operation must be non-negative. Formula (5) indicates that each operation can only be assigned to one machine. Formula (6) indicates that the completion time of an operation minus the start time equals the processing time. Formula (7) ensures that a job can only be processed after its arrival time. Formula (8) ensures the priority constraint of operations on the same workpiece. Formula (9) ensures the execution order constraint of operations on each workpiece on each machine. Formula (10) represents the workpiece The remaining time from now to the expiration time. Formula (11) represents the operation The average processing time on a machine in the optional machine set. Formula (12) expresses the average processing time of a workpiece when rescheduling occurs. The remaining processing time of the operation. Formula (13) represents the workpiece The idle time of the workpiece. Formula (14) represents The delay time of the scheduling system is shown in Formula (15). The delay rate of the scheduling system is shown in Formula (16). The total processing time on .

[0093] The decision variables corresponding to the above mathematical model are:

[0094] ,and, .

[0095] Secondly, based on the mathematical model of the dynamic flexible job shop scheduling problem, the rescheduling events that occur in the actual production process of the dynamic flexible job shop are scheduled. Figure 2 , the specific plan is as follows:

[0096] 201. During the actual production process of the dynamic flexible operation workshop, the processing information in the dynamic flexible operation workshop is obtained in real time.

[0097] The processing information includes processing workpiece information and processing machine information; processing workpiece information includes: the number of processed workpieces, the number of newly inserted workpieces, the number of workpiece processes, the process sorting of workpieces, the processing time of each process on different machines, the average processing time of the process, the arrival time of the workpiece, and the completion time of the workpiece; processing machine information includes: machine status information, and the processing ownership relationship between the workpiece and the machine.

[0098] 202. When a rescheduling event is determined to occur based on processing information, a rescheduling strategy is generated based on the mathematical model of the dynamic flexible job shop scheduling problem according to the real-time status of the workpiece and the machine at the rescheduling point where the dynamic event occurs, and rescheduling is executed until the production process is completed.

[0099] This paper addresses the question of how to efficiently generate real-time rescheduling strategies to handle dynamic events. Dynamic / uncertain events in actual production processes typically include urgent workpiece insertions and unexpected machine failures. Therefore, the rescheduling events in this invention are dynamic / uncertain events such as urgent workpiece insertions and unexpected machine failures. Accordingly, the rescheduling point is the point in time when an urgent workpiece is inserted or a machine failure occurs.

[0100] Therefore, this example focuses on scheduling at rescheduling points. The rescheduling process for the dynamic flexible job shop scheduling problem is as follows: At the start of scheduling, all workpieces are processable on all machines. Once all machines have been scheduled for processing, the production process proceeds. When a workpiece is urgently inserted or a machine fails, a rescheduling point occurs. At this point, the set of remaining workpieces and the set of available machines are updated, new scheduling information is generated, and the production process continues. Similarly, each rescheduling event and rescheduling point generates new scheduling information until the production process is complete.

[0101] The present invention solves the dynamic flexible job shop scheduling problem by introducing a multi-agent scheduling system based on D3QN. In view of the variability and randomness existing in the scheduling problem, a multi-agent and distributed agent architecture is adopted to distribute and parallelize the problem-solving steps, thereby achieving the consistency of step optimization to global optimization. In the actual factory processing process, the invention of this article can monitor the dynamic changes of the processed workpieces and machines in real time, and make real-time scheduling decisions in a timely and efficient manner for different emergencies, such as emergency insertion of workpieces, machine damage, etc., effectively reducing the time cost and resource cost of production. Unlike the traditional single-agent deep reinforcement learning method, the multi-agent scheduling system of the present invention can meet both the high efficiency of production and processing and the intelligence of production resource allocation when dealing with production scheduling problems with huge data volumes and a large number of emergencies.

[0102] For a detailed description of the rescheduling process in 202, see Figure 3 , its scheduling process specifically includes:

[0103] S1. Obtaining workpiece information and processing machine information.

[0104] S2, rescheduling point workpiece and machine status update.

[0105] Based on the obtained processing information, at the beginning, the rescheduling point workpiece and machine status update module updates the action status information, and obtains the optional workpiece set according to whether the predecessor process of the current process of the workpiece is completed; obtains the optional machine set according to the processing machine set of the current process of the workpiece and the current machine status set, and refers to Figure 4 When a rescheduling event occurs during the production process, the workpiece and machine status update module at the rescheduling point will update the status information again until all workpieces are processed.

[0106] S3. The artifact agent generates artifact scheduling information.

[0107] After the status information of the rescheduling point is updated, the workpiece agent allocates the next processing workpiece according to the optional workpiece set and generates workpiece scheduling information. The workpiece agent configures the trained workpiece D3QN evaluation network. The workpiece scheduling information includes the workpiece serial number and process serial number.

[0108] S4. The machine agent generates machine scheduling information.

[0109] Based on the workpiece scheduling information and the set of rescheduling optional machines, the machine agent arranges the processing machines of the workpiece and generates machine scheduling information. The machine agent configures the trained machine D3QN evaluation network to evaluate whether the workpiece has been processed. If not, it returns to S2 to re-update the status. If processing is completed, it outputs the optimal scheduling plan.

[0110] Reinforcement learning can be modeled as a Markov decision model represented as a quintuple . represents the state space, represents a finite action space, Expressed as state transition distribution, . Represents the discount factor. represents the reward function, In reinforcement learning, the agent follows a specific strategy Interact with the surrounding environment. For each decision point , the agent observes the current state , according to the strategy Select an action , then by Sample transition to next state , and get immediate rewards In the embodiment of the present invention, three agents are included. Taking the workpiece agent as an example, whenever a rescheduling event occurs, the environment state space is updated to At this time, the agent extracts state features based on the new state. The state features extracted by each agent are different, but these state features are interrelated. It is precisely because the state features are interrelated that the subsequent action decisions between the agents are relevant.

[0111] The state characteristics of the workpiece agent are shown in Table 1:

[0112] Table 1: Status characteristics of artifact agents

[0113]

[0114] The state characteristics of the machine agent are shown in Table 2:

[0115] Table 2: State characteristics of machine agents

[0116]

[0117] The state characteristics of the joint reward agent are shown in Table 3:

[0118] Table 3: State characteristics of the joint reward agent

[0119]

[0120] After the state characteristics are updated, the workpiece and machine state update modules are rescheduled to update the action state information. The workpiece agent and the machine agent select different actions based on their respective state characteristics and action state information. and , the actions are executed together, and the environment promptly feeds back the joint reward The joint reward agent is based on The state characteristics obtained at all times, perform actions , get different rewards for the workpiece agent and the machine agent , the environmental state characteristics are updated to , the rewards obtained The goal of deep reinforcement learning is to learn decision-making problems by maximizing the discounted cumulative reward. For any policy , state-action value function ( function) is defined as:

[0121] Q π ( s , a ) = E π [ ∑ t = 0 T γ t r ( s t , a t ) | s 0 = s , a a = a ] ;

[0122] in Representation Strategy Under expectations, Indicates the time range, Indicates the initial state, Indicates the initial action;

[0123] The ultimate goal of the three intelligent agents in this invention is to learn the decision-making problem of dynamic job shop scheduling through the maximum discounted cumulative reward, so that the final scheduling strategy meets the scheduling goal.

[0124] like Figure 5 The following diagram shows a multi-agent network framework system for solving dynamic and flexible job shop scheduling based on the D3QN network. At each rescheduling point, the workpiece agent and the machine agent perform a joint action based on the workpiece and machine states in the current environment. The environment provides a total reward, and the reward agent maps the total reward to the workpiece agent and the machine agent based on the current environment and the total reward. The two agents then evaluate the effectiveness of their actions based on the resulting mapped rewards. The pseudo code for the training phase of the D3QN network multi-agent network framework system for solving dynamic and flexible job shop scheduling is as follows:

[0125] enter:

[0126] Workshop parameters: number of workpieces , the number of newly inserted workpieces , operand , number of machines ; Iterations: ; Experience replay buffer and buffer lower limit: ; ;Batch size: ;

[0127] Output:

[0128] Artifact Agent Evaluation Network Parameters , the machine agent evaluates network parameters , Jointly reward the agent to evaluate the network parameters ;

[0129] Initial artifact agent evaluation network parameters And assign the evaluation network parameters to the target network parameters ;

[0130] Initialize the machine agent evaluation network parameters And assign the evaluation network parameters to the target network parameters ;

[0131] Combined reward agent evaluation network parameters And assign the evaluation network parameters to the target network parameters ;

[0132] Initialize the workpiece agent, machine agent, and reward joint agent experience replay buffer respectively , the buffer size is set to ;

[0133] for episode = 1: do:

[0134] Generate instances based on workshop parameter information ;

[0135] Reset the scheduling environment and generate the initial state , and ;

[0136] for do:

[0137] The rescheduling point workpiece and machine status update module updates the action status information;

[0138] The legal action space of the workpiece machine is formed;

[0139] if The action space is not empty then;

[0140] according to Computing workpiece agent actions ;

[0141] according to Computing robot agent actions ;

[0142] Execute an action , scheduling environment feedback joint reward R;

[0143] Forming the initial state of the reward joint agent ;

[0144] according to Calculate joint rewards for agent actions ;

[0145] Execute an action , scheduling environment feedback reward ,get ;

[0146] The scheduling environment is transferred to ;

[0147] Store experience tuples ( , , , ) to the artifact agent experience replay buffer ;

[0148] Store experience tuples ( , , , ) to the machine agent experience replay buffer ;

[0149] Store experience tuples ( , , , ) to the joint reward agent experience replay buffer ;

[0150] if artifact agent experience replay buffer Size > lower limit of top experience buffer then:

[0151] The priority replay size in the workpiece agent experience replay buffer is The experience tuple ( , , , ) Update network parameters (see Network Update 1);

[0152] In the machine agent experience replay buffer, the priority replay size is The experience tuple of

[0153] In the machine agent experience replay buffer, the priority replay size is The experience tuple ( , , , ) Update network parameters (see Network Update 1);

[0154] Endif.

[0155] if joint reward agent experience replay buffer Size > lower limit of underlying experience buffer then:

[0156] In the joint reward agent experience replay buffer, the priority replay size is The experience tuple ( , , , ) Update network parameters (see Network Update 2);

[0157] Endif.

[0158] Endif.

[0159] Endfor.

[0160] End for.

[0161] The pseudo code for network update is as follows:

[0162] 1: Input: Experience tuple: ( , , , ); Evaluate network parameters: ;Target network parameters: ; Target network update frequency: ; Reward discount factor: ;

[0163] 2: Calculate the maximum Q value of the next state based on the target network parameters ;

[0164] 3:if Scheduling ends then;

[0165] 4: TD error ;

[0166] 5:Else;

[0167] 6: TD error ;

[0168] 7:Endif;

[0169] 8: Gradient Descent Updated assessment network;

[0170] Every Update target network .

[0171] The pseudo code for network update 2 is as follows:

[0172] 1: Input: Experience tuple: ( , , , ); Evaluate network parameters: ;Target network parameters: ; Target network update frequency: ; Reward discount factor: ;

[0173] 2: Calculate the maximum Q value of the next state based on the target network parameters V ( s ; θ , β ) + [ A ( s , a ; θ , α ) − 1 | A | ∑ a ' A ( s , a ' ; θ , α ) ] ,in represents the action space, represents the candidate actions in the action space, ∈ ; is the value function, which is related to the state characteristics, is the advantage function, which is related to both state characteristics and actions;

[0174] 3:if Scheduling ends then;

[0175] 4: TD error ;

[0176] 5:Else;

[0177] 6: TD error ;

[0178] 7:Endif;

[0179] 8: Gradient Descent Update the evaluation network every Update target network .

[0180] Training is complete. See the Gantt chart for an instance with 14 initial workpieces, 8 newly inserted workpieces, and 8 machines. Figure 6.

[0181] During the system training phase, the total delay time changes. Figure 7 In the line graph shown, it can be clearly seen that after the number of training times reaches 150, the total delay time can reach the lowest.

[0182] In addition, an embodiment of the present invention provides a multi-agent scheduling system comprising four modules: a rescheduling point artifact and machine status update module, an artifact proxy module, a machine proxy module, and a joint reward mapping proxy module. The rescheduling point artifact and machine status update module is used to update artifact and machine status when rescheduling occurs in the production process; the artifact proxy module is used to generate artifact scheduling information; the machine proxy module is used to generate machine scheduling information; and the joint reward mapping proxy module is used to assign joint reward mappings to corresponding agents.

[0183] The above-mentioned multi-agent scheduling system is used to perform all operations in the method for solving dynamic flexible job shop scheduling based on a multi-agent D3QN network as described in the above-mentioned method embodiment.

[0184] Obviously, those skilled in the art should understand that the various modules or steps of the above-mentioned embodiments of the present invention can be implemented using general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices. Alternatively, they can be implemented using program codes executable by the computer device, so that they can be stored in a storage device and executed by the computer device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0185] It should be noted that the above are only preferred embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present invention.

Claims

1. A multi-agent method for solving dynamic flexible job shop scheduling based on a D3QN network, characterized in that: include: During the actual production process of the dynamic flexible job shop, the processing information in the dynamic flexible job shop is obtained in real time, and the processing information includes the processing workpiece information and the processing machine information; When a rescheduling event is determined to occur based on processing information, a rescheduling strategy is generated based on the mathematical model of the dynamic flexible job shop scheduling problem according to the real-time status of the workpiece and machine at the rescheduling point where the dynamic event occurs, and the rescheduling is executed until the production process is completed; At each rescheduling point, the workpiece agent and the machine agent perform a joint action based on the workpiece and machine states in the current environment. The joint action is performed as follows: the environment gives a total reward, and the joint reward agent uses the current environment and total reward as its state. It maps the total reward to the workpiece agent and the machine agent respectively. The two agents then evaluate the effectiveness of the actions they have taken based on the resulting mapped rewards. Generate a rescheduling strategy. The specific operations for executing rescheduling include: 1) Update of workpiece and machine status at rescheduling point: Based on the obtained processing information, the rescheduling point workpiece and machine status update module initially updates the action status information and obtains the optional workpiece set based on whether the predecessor process of the workpiece's current process has been processed. The optional machine set is obtained based on the processing machine set of the workpiece's current process and the current machine status set. When a rescheduling event occurs, the rescheduling point workpiece and machine status update module updates the status information again until all workpieces are processed. 2) The artifact agent generates artifact scheduling information: After the status information of the rescheduling point is updated, the workpiece agent allocates the next processing workpiece according to the optional workpiece set and generates workpiece scheduling information. The workpiece agent configures the trained workpiece D3QN evaluation network. The workpiece scheduling information includes the workpiece number and process number. 3) The machine agent generates machine scheduling information: Based on the workpiece scheduling information and the set of rescheduling optional machines, the machine agent arranges the processing machines of the workpiece and generates machine scheduling information. The machine agent configures the trained machine D3QN evaluation network to evaluate whether the workpiece is processed. If the processing is completed, it outputs the optimal scheduling solution; Whenever a rescheduling event occurs, the environment state space is updated, and the agent extracts state features based on the new state; The state characteristics of the workpiece agent include: the average processing time of the next unfinished operation that can be selected for the workpiece at time t; the shortest processing time of the next unfinished operation that can be selected for the workpiece at time t; the standard deviation of the completion rate of each workpiece process at time t; the tightness of the remaining deadline for each workpiece at time t; The state characteristics of the machine agent include: a list of machine processing times for the selected workpieces at time t; the utilization rate of each machine at time t; the standard deviation of the completion rate of the machine at time t; The state characteristics of the joint reward agent include: the change in workpiece delay at time t; the change in the standard deviation of the workpiece completion rate at time t; the change in the standard deviation of the machine utilization rate at time t; the change in the standard deviation of the workpiece delay at time t; joint reward; the joint reward agent uses the DDQN network framework; After the momentary state features are updated, the workpiece and machine state update modules at the rescheduling point update the action state information. The workpiece agent and machine agent combine their respective state features and action state information to select different actions. After the actions are jointly executed, the environment promptly provides joint rewards. The mathematical model of the dynamic flexible job shop scheduling problem is constructed based on the initial parameter information of the dynamic flexible job shop, including the scheduling objectives and constraints. The scheduling objectives are to minimize the total delay time of all workpieces and minimize the variance of all machine utilization. The constraints are: n There are a total of O process, m The processing sequence of each workpiece is fixed, and each process has a fixed set of processing machines, including the general and dynamic constraints of FJSP. Rescheduling events are dynamic / uncertain events including urgent workpiece insertion and unexpected machine failure. The rescheduling point is the time point when the workpiece is urgently inserted or the machine fails.

2. The method for solving dynamic flexible job shop scheduling based on a D3QN network according to claim 1 is characterized in that: The general constraints of FJSP include: ① Every machine is available at time zero, ② All arrived workpieces can be processed at time zero, ③ There is no priority between the processes of different workpieces, ④ Each machine can only process at most one operation at a time, ⑤ Each operation should be processed in a non-preemptive manner without interruption, and ⑥ All operations belonging to the same job should be processed one after another in a fixed order, that is, priority constraints.

3. The method for solving dynamic flexible job shop scheduling based on a D3QN network according to claim 1, characterized in that: The dynamic constraints include: ① transportation time and setup time are negligible, ② the buffer between machines is unlimited, ③ if an operation is interrupted due to machine failure, the remaining processing time is equal to the total processing time minus the completed processing time, ④ each job contains different types of operations, and each job has a delivery deadline that must be met, otherwise it will be considered timed out.

4. The method for solving dynamic flexible job shop scheduling based on a D3QN network according to claim 1, characterized in that: Minimizing the total delay time of all workpieces is expressed as: ； Where, Indicates the total number of jobs, Indicates the index of the job, ∈ ; express The completion time of the workpiece after each operation; Represents workpiece duration of the project; Indicates the portion of completion time that exceeds the construction period.

5. The method for solving dynamic flexible job shop scheduling based on a D3QN network according to claim 1, characterized in that: Minimizing the variance of all machine utilization is expressed as: ； Where, represents the variance of machine utilization in the entire system; m Indicates the total number of machines; Indicates the k The total processing time of each machine, k ∈ m ; represents the processing time of all machines, l ∈ m , k ∈ l .

6. A multi-agent scheduling system, characterized in that: include: The rescheduling point workpiece and machine status update module, the workpiece agent module, the machine agent module, and the joint reward mapping agent module are used to execute the method for solving dynamic flexible job shop scheduling based on a multi-agent D3QN network as described in any one of claims 1 to 5 above.

Images