Multi-target dynamic distributed flexible job shop scheduling method based on deep Q network

Abstract

The invention discloses a multi-target dynamic distributed flexible job shop scheduling method based on a deep Q network, and belongs to the technical field of dynamic shop scheduling. The scheduling method comprises the following steps: S1, establishing a mathematical model of the multi-target dynamic distributed flexible job shop according to a scheduling requirement of the multi-target dynamic distributed flexible job shop; s2, respectively constructing state spaces of the distribution agent and the scheduling agent, and designing corresponding scheduling rules and reward functions for the two agents; s3, solving the problem of the multi-target dynamic distributed flexible job shop by using a multi-agent collaborative algorithm; s4, respectively constructing a distribution agent structure and a scheduling agent structure corresponding to the distribution agent and the scheduling agent; s5, formulating a tardiness-based speed regulation energy-saving strategy; according to the speed regulation energy-saving strategy, the corresponding machining time is adjusted by adjusting the machining speed of each workpiece on the corresponding machine, and therefore the overall energy consumption is adjusted.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of dynamic job shop scheduling, and in particular to a multi-objective dynamic distributed flexible job shop scheduling method based on a deep Q network. Background Art

[0002] Most of the related technologies of multi-objective dynamic distributed flexible job shop scheduling are based on two methods, one is a simple scheduling rule, and the other is a meta-heuristic algorithm. Common simple scheduling rules include First in First out (FIFO), Earliest due date (EDD), Most remaining processing time (MRPT), etc. When rescheduling is required, it selects a workpiece to be processed on a machine at the time of rescheduling, which can quickly respond to uncertain events and has high timeliness, but the response is short-sighted in the long run and cannot get a better scheduling solution.

[0003] Meta-heuristic algorithms mainly include genetic algorithms, particle swarm optimization algorithms, ant colony optimization algorithms, etc. They decompose the dynamic scheduling problem into a multi-stage static scheduling problem for solution, that is, at each rescheduling moment, all remaining unfinished workpieces are considered uniformly, and each search branch is first evaluated in the state space to obtain the best branch, and then continue searching from this branch to reach the target and form a new scheduling plan. In this way, a better rescheduling plan can be generated. However, it takes a lot of time to find a better scheduling plan in the huge feasible solution space, and it cannot adapt well to the situation where dynamic events occur frequently.

[0004] Most of the traditional dynamic flexible job shop scheduling methods are aimed at a single factory, that is, only the dynamic scheduling in a single factory needs to be considered. In the distributed flexible job shop scheduling problem, the processing status of each factory should be measured, so that the workpieces can be reasonably sent to the corresponding factory for processing at the rescheduling point. Compared with the traditional dynamic flexible job shop scheduling problem, the distributed flexible job shop must consider not only sending each workpiece to the appropriate factory reasonably, but also the processing status of the workpiece in the factory. It is more complicated than the flexible job shop scheduling problem and has been proven to be an NP-hard problem. Therefore, in a processing environment where various uncertain factors (such as emergency workpiece insertion and machine failure) coexist and occur frequently, the traditional distributed flexible job shop dynamic scheduling algorithm, especially the complex heuristic algorithm, cannot generate a good scheduling plan in time because it consumes a lot of computing time. Although the simple heuristic rules can quickly generate scheduling plans, they are short-sighted and cannot play a long-term optimization role in the scheduling plan. Summary of the invention

[0005] In order to overcome the defects existing in the above-mentioned existing scheduling technical solutions, the embodiment of the present application is based on a hierarchical collaborative multi-agent method of deep reinforcement learning to solve the multi-objective dynamic scheduling problem of distributed flexible job shops with energy saving. This method can quickly and adaptively generate a good scheduling plan at each rescheduling moment.

[0006] Specifically, the embodiment of the present application provides a multi-objective dynamic distributed flexible job shop scheduling method based on a deep Q network, which overcomes the short-sightedness of the scheduling scheme generated by a single priority scheduling rule; overcomes the huge time required to obtain the scheduling scheme using heuristic algorithms and mathematical programming. The method can quickly and adaptively generate a good scheduling scheme at each rescheduling moment. The technical solution is as follows:

[0007] A multi-objective dynamic distributed flexible job shop scheduling method based on deep Q network, comprising:

[0008] S1. According to the scheduling requirements of the multi-objective dynamic distributed flexible job shop, a mathematical model of the multi-objective dynamic distributed flexible job shop is established;

[0009] The mathematical model of the multi-objective dynamic distributed flexible job shop consists of three objective functions and constraints. The three objective functions are the total delay time function of the workpiece, the total energy consumption function and the maximum completion time function of the workpiece.

[0010] S2. Construct the state space of the distribution agent and the scheduling agent respectively, and design corresponding scheduling rules and reward functions for the two agents;

[0011] The state spaces of the distribution agent and the scheduling agent are inconsistent. The state space of the distribution agent comprehensively reflects the allocation status of the rescheduling point or decision point and the production status of each factory, and the state space of the scheduling agent reflects the rescheduling production status; the corresponding scheduling rules designed for the two agents include: 8 workpiece selection rules set for the distribution agent, and, 5 workpiece selection rules and 3 machine selection rules set for the scheduling agent; the corresponding reward functions designed for the two agents are the distribution agent incentive function and the scheduling agent reward function;

[0012] S3, using multi-agent collaborative algorithm to solve the problem of multi-objective dynamic distributed flexible job shop;

[0013] The multi-agent collaborative algorithm includes the scheduling agent algorithm and the distributed agent algorithm, both of which are composed of DDQN. The interaction between the DDQN multi-agent and the multi-objective dynamic distributed flexible job shop scheduling environment is a discrete-time Markov decision process model;

[0014] S4, construct the distribution agent structure and the scheduling agent structure corresponding to the distribution agent and the scheduling agent respectively;

[0015] Both the distribution agent structure and the scheduling agent structure are composed of DDQN, which are connected by the environment of undistributed artifacts;

[0016] S5. Formulate a speed regulation and energy-saving strategy based on delay; the speed regulation and energy-saving strategy is to adjust the corresponding processing time by adjusting the processing speed of each workpiece on the corresponding machine, thereby adjusting the overall energy consumption.

[0017] Specifically, in some possible implementations, the constraints are set as follows: 1) Each workpiece can only be assigned to one factory; 2) All operations on each workpiece should be performed in the same factory; 3) Each operation on each workpiece can only be assigned to one available machine; 4) The sum of the number of machines used for all corresponding operations of each workpiece should be greater than the total number of machines; 5) Each workpiece can only be processed after the workpiece arrives; 6) The process of each workpiece must follow a priority order from front to back; 7) When the workpieces are processed on the same machine, the processing time will not overlap; 8) All operations on each workpiece have a non-negative start time; 9) The transportation time between workshops and equipment is not considered.

[0018] Specifically, in some possible implementations, the function expression for minimizing the total delay time function of the workpiece is: In the formula, the total delay time function TWT of all factories is given by workpiece J i Deadline D i and the corresponding completion time Calculate, n is the total number of workpieces, n i For workpiece J i The total number of operations, For workpiece J i The processing urgency, the larger the value, the more urgent it is;

[0019] The function expression for minimizing the maximum completion time of a job is: in Workpiece J i The completion time of the last operation;

[0020] The total energy consumption of the factory TEC is composed of the processing loss of the equipment EC p and idle loss EC I Calculated, its function expression is: TEC = EC p +EC I ;

[0021] Among them, the processing loss of equipment EC p for:

[0022] Idle loss EC I for:

[0023] Where, PP f,k,v is the machine M in factory f k Energy consumption during processing with speed v, EI f,k is the machine M in factory f k Specific no-load energy consumption when idle, p i,j,f,k,v For workpiece J i In factory f, machine M k The actual processing time when processing at speed v.

[0024] Specifically, in some possible implementations, the eight artifact selection rules set for the distribution agent are as follows:

[0025] Rule 1: Randomly select a workpiece from the queued workpieces and assign it to a factory randomly selected from the available factories; Rule 2: Select the workpiece with the shortest arrival time from the queued workpieces and assign it to the factory with the least number of stored workpieces; Rule 3: Select the workpiece with the shortest arrival time from the queued workpieces and assign it to the factory with the lowest average machine utilization; Rule 4: Select the workpiece with the shortest processing time from the queued workpieces and assign it to the factory with the least number of stored workpieces; Rule 5: Select the workpiece with the shortest processing time from the queued workpieces and assign it to the factory with the lowest average machine utilization; Rule 6: Select the workpiece with the longest delay from the queued workpieces and assign it to the factory with the smallest total delay in the current factory; Rule 7: Select the workpiece with the longest delay from the queued workpieces and assign it to the factory with the smallest machine utilization in the current factory; Rule 8: Select the workpiece with the longest delay from the queued workpieces and assign it to the factory with the least number of currently stored workpieces;

[0026] The five artifact selection rules set for the scheduling agent are as follows:

[0027] Rule 1: Select the workpiece with the smallest current slack time from the unfinished workpieces; Rule 2: Select the workpiece with the smallest estimated slack time after the remaining operations are executed from the unfinished workpieces; Rule 2: Select the workpiece with the longest queue time from the unfinished workpieces. If there are multiple workpieces with the longest queue time, select the workpiece with the smallest slack time corresponding to multiple workpieces; Rule 3: Select the workpiece that arrives first from the unfinished workpieces. If there are multiple workpieces that arrive at the same time and they are all the first to arrive at time t, randomly select a workpiece from them; Rule 4: Select the workpiece with the shortest delivery time from the unfinished workpieces; Rule 5: The workpiece with the shortest remaining processing time;

[0028] The three machine selection rules set for the scheduling agent are as follows:

[0029] Rule 1: Select the machine with the lowest utilization; Rule 2: Select the machine with the earliest availability; Rule 3: Select the machine with the lowest processing load.

[0030] Specifically, in some possible implementations, the scheduling agent reward function is:

[0031] Where TWT(t) is the total delay at time t, It is operation O i,j+1 The start time of It is operation O i,j The completion time of , α, β are hyper parameters;

[0032] The distribution agent reward function is as follows: R a (t)=-(max(R f (t)-min(R f (t)))), where R f (t) is the reward value of the scheduling agent in factory f.

[0033] Specifically, in some possible implementations, the scheduling agent algorithm is: Figure 1 As shown, the distributed agent algorithm is Figure 2 shown.

[0034] Specifically, in some possible implementations, the DDQNs of the distribution agent and the scheduling agent each consist of 1 input layer, 5 hidden layers, and 1 output layer;

[0035] The distributed agent has 20 input nodes, corresponding to 20 state features; each hidden layer has 200 nodes; the output layer has 8 nodes, corresponding to 8 optional actions;

[0036] The scheduling agent has 15 input nodes, corresponding to 15 input nodes; each hidden layer has 200 nodes; the output layer has 12 nodes, corresponding to 12 optional actions.

[0037] The above technical solution can achieve the following technical effects in the embodiments of the present application:

[0038] 1. The present invention proposes a mathematical model of a multi-objective dynamic distributed flexible job shop.

[0039] In view of the existing workshop scheduling problem, the current methods for generating workshop scheduling solutions are mostly based on single scheduling rules (FIFO, EDD, MRPT) and meta-heuristic algorithms. The present invention makes up for the deficiency of few available methods in the scheduling field according to the characteristics of the workshop, and makes a meaningful expansion in the scheduling field of multi-objective dynamic distributed flexible job shops.

[0040] 2. The present invention constructs two state spaces in a multi-objective dynamic distributed flexible job shop, designs corresponding action spaces for the two state spaces, and proposes corresponding reward functions. By designing exclusive states, actions, and rewards for the allocation agent and the scheduling agent, the two agents can learn the corresponding strategies to deal with the dynamic situation of random arrival of workpieces and generate good scheduling solutions during the training process.

[0041] 3. The present invention uses the DDQN algorithm to select the scheduling rules formulated according to the current state of the multi-objective flexible job shop. Specifically, the agent inputs the corresponding characteristic states into the network, and the Q value of each action is obtained after network calculation. The action with a larger Q value proves that the action has the best impact on the environment in this state, so the action with the largest Q value, that is, the scheduling rule corresponding to the largest Q value, is selected. In addition, since the reward of the scheduling agent has multi-objective orientation, during the training process of the scheduling agent, the scheduling agent will give different rewards for delays and completion times under different environments. The scheduling agent continuously trains with the maximum total reward as the goal to update the parameters of the scheduling agent, so as to achieve a balance between delays and completion times.

[0042] 4. The present invention uses a speed control strategy to solve the problem of minimum energy consumption while meeting the minimum delay. The speed control strategy adjusts the corresponding processing time by adjusting the processing speed of each workpiece on the corresponding machine, thereby adjusting the overall energy consumption, thereby achieving the purpose of saving energy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0044] Figure 1 A flowchart of a multi-objective dynamic distributed flexible job shop scheduling method based on a deep Q network in this application;

[0045] Figure 2 The workflow of the scheduling agent DDQN of the scheduling agent algorithm in this application;

[0046] Figure 3 The workflow of the distributed agent DDQN of the distributed agent algorithm in this application;

[0047] Figure 4 It is the main flow chart of the distribution agent and scheduling agent algorithm in this application;

[0048] Figure 5This is a structural diagram of the distribution agent and the scheduling agent in this application;

[0049] Figure 6 A schematic diagram of a speed control algorithm for the energy-saving speed control strategy based on delay in this application;

[0050] Figure 7 This is a schematic diagram of the energy-saving speed regulation steps based on delay in this application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0052] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0053] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0054] Under the pressure of a series of environmental impacts and energy costs, green manufacturing has attracted more and more attention from various industries, especially in the automotive manufacturing industry. This industry is not only an important part of the global economy, but also one of the main sources of greenhouse gas emissions. In response to growing environmental regulations and consumer demand for sustainable products, automakers are actively seeking solutions to reduce carbon footprints and improve energy efficiency. In the past few decades, manufacturing has consumed about one-third of the world's energy, and this phenomenon is particularly prominent in the automotive industry. Traditional flexible manufacturing systems usually focus on improving production efficiency, such as reducing costs through automation and process optimization, while ignoring their impact on the environment. This short-sightedness may increase corporate profits in the short term, but in the long run, it may lead to companies facing severe environmental responsibilities and social pressure.

[0055] With the acceleration of global trade, automakers have gradually increased their reliance on the global supply chain. However, centralized flexible manufacturing systems often lag behind in responding to market changes and cannot quickly adapt to the needs of different regions. Collaborative production between different factories can provide obvious advantages in response speed and production costs, so that products are usually processed in multiple countries and regions. Take Toyota Motor Corporation as an example. The company has multiple manufacturing bases in North America, Europe and Asia. In order to optimize resource utilization, the company implemented a distributed flexible manufacturing system to rationally utilize local human and material resources by assigning production tasks of different models to factories in different regions. However, this approach also brings new challenges: how to keep the system energy-efficient and efficient in a dynamic and unpredictable production environment. For example, one of the company's factories faces a machine failure, which causes the production line to stagnate. If the production plan is not adjusted in time, it may lead to order delays and customer loss. In addition, the random arrival of workpieces and the cancellation of orders will also have a negative impact on production efficiency. Researchers have found that the ideal distributed flexible job shop design often cannot cope with these practical problems. Therefore, studying the energy-saving distributed flexible job shop problem in a real-time production environment has very important practical significance and potential value. With the help of data analysis technology, factories can optimize production scheduling to minimize energy consumption while ensuring output. This kind of research can not only help companies improve production efficiency, but also enhance their sustainable development capabilities in global competition. As consumers' demand for environmentally friendly cars increases, having an efficient and environmentally friendly production system will enable companies to stand out in the market, meet increasingly stringent environmental regulations and consumer expectations, and achieve dual economic and environmental benefits.

[0056] Therefore, a multi-objective dynamic distributed flexible job shop scheduling method based on a deep Q network is provided in an embodiment of the present application, and the scheme involves a total of two agents, namely, an allocation agent and a scheduling agent, both of which are composed of a DDQN framework. The main function of the allocation agent is to allocate unallocated workpieces to suitable factories, and the main function of the scheduling agent is to schedule the workpieces assigned to the factory by the allocation agent and generate feasible and good scheduling schemes. Each agent continuously interacts with the corresponding environment and selects a better scheduling rule at each rescheduling point; a strategy of online generation of scheduling schemes and offline training is adopted, and scheduling schemes under various environments are fully explored during training, so that it can adaptively and quickly generate a better solution when applied, and after the scheduling scheme is generated, the scheduling scheme adopts a speed regulation energy-saving strategy for energy-saving operation.

[0057] In order to use multi-agents to solve the multi-objective dynamic distributed flexible job shop problem and achieve the expected goal, the technical solution in the embodiment of the present application is as follows:

[0058] Embodiment 1

[0059] like Figure 1 As shown, the multi-objective dynamic distributed flexible job shop scheduling method includes:

[0060] S1. According to the scheduling requirements of the multi-objective dynamic distributed flexible job shop, a mathematical model of the multi-objective dynamic distributed flexible job shop is established.

[0061] The mathematical model of the multi-objective dynamic flexible job shop consists of multi-objective functions and a series of constraints: there are three multi-objective functions, namely, the total delay time function of the workpiece, the total energy consumption function and the maximum completion time function of the workpiece. The total delay time function of the workpiece includes the deadline of each workpiece, the time required to complete the corresponding process of the workpiece, the arrival time of the workpiece, and the decision variable calculation; the total energy consumption function is calculated by the factory's machine processing energy consumption and machine idle energy consumption; the maximum completion time of the workpiece is calculated based on the completion time of the last operation of all the largest workpieces.

[0062] The constraints are set as follows: 1) Each workpiece can only be assigned to one factory; 2) All operations of each workpiece should be performed in the same factory; 3) Each operation of each workpiece can only be assigned to one available machine; 4) The sum of the number of machines used by all corresponding operations of each workpiece should be greater than the total number of machines; 5) Each workpiece can only be processed after the workpiece arrives; 6) The process of each workpiece must follow the priority order from front to back; 7) When the workpieces are processed on the same machine, the processing time will not overlap; 8) All operations of each workpiece have a non-negative start time; 9) The transportation time between workshops and equipment is not considered.

[0063] For a detailed description of the multi-objective functions and constraints in S1, please refer to the second embodiment below.

[0064] S2. Construct the state space of the distribution agent and the scheduling agent respectively, and design corresponding scheduling rules and reward functions for the two agents.

[0065] The state space of the distribution agent and the scheduling agent are inconsistent. The state space of the distribution agent comprehensively reflects the allocation state of the rescheduling point or decision point and the production state of each factory. The state space of the scheduling agent reflects the rescheduling production state.

[0066] Based on the above states, such as allocation state and production state, 8 workpiece selection rules are set for the distribution agent, and 5 workpiece selection rules and 3 machine selection rules are set for the scheduling agent, which together constitute 16 composite scheduling rules.

[0067] Different reward functions are set for the distribution agent and the scheduling agent, namely the distribution agent incentive function and the scheduling agent reward function.

[0068] For a detailed description of the state space, corresponding scheduling rules and reward function described in S2, please refer to the third embodiment below.

[0069] S3. Use multi-agent collaborative algorithm to solve the problem of multi-objective dynamic distributed flexible job shop.

[0070] Specifically, the multi-agent collaborative algorithm includes the scheduling agent algorithm and the distributed agent algorithm, both of which are composed of DDQN. The interaction between the DDQN multi-agent and the multi-objective dynamic distributed flexible job shop scheduling environment is a discrete-time Markov decision process model. DDQN (Double Deep Q-Network) is an algorithm based on deep reinforcement learning. It aims to solve the problem that the traditional Q-learning algorithm faces state dimension explosion in multiple states and that the traditional DQN (Deep Q-Network) may overestimate the value of actions in some cases.

[0071] Scheduling agent algorithm such as Figure 2 As shown, the distributed agent algorithm is Figure 3 As shown, the detailed description of the algorithm can be found in Example 4 and will not be repeated here.

[0072] For the specific solutions for the scheduling agent algorithm, distribution agent algorithm and multi-objective dynamic distributed flexible job shop described in S3, please refer to the fourth embodiment of the following text.

[0073] S4. Construct the distribution agent structure and scheduling agent structure corresponding to the distribution agent and scheduling agent respectively.

[0074] The distributed agent structure and the scheduling agent structure are both composed of DDQN, which are connected by the environment of unassigned artifacts. The framework structure of the two is as follows Figure 5 As shown. Figure 5 In the example, the allocation agent selects the most suitable workpiece through the corresponding network according to the information of the current unallocated workpiece and the processing status of each factory, and allocates the selected workpiece to the corresponding factory. After receiving the workpiece, the factory pre-processes all the workpieces in the factory, integrates the status of each factory after the pre-processing is completed, and then transmits the status to the allocation agent. Repeat the above process until all workpieces are allocated and all factories are processed. In addition, the DDQN of the distribution agent and the scheduling agent consists of 1 input layer, 5 hidden layers and 1 output layer, and the number of nodes in them is different. The distribution agent has 20 input nodes, corresponding to 20 state features; each hidden layer has 200 nodes; the output layer has 8 nodes, corresponding to 8 optional actions. The scheduling agent has 15 input nodes, corresponding to 15 input nodes; each hidden layer has 200 nodes; the output layer has 12 nodes, corresponding to 12 optional actions.

[0075] S5. Develop a speed regulation and energy-saving strategy based on delay.

[0076] Specifically, the speed regulation energy-saving strategy is to adjust the corresponding processing time by adjusting the processing speed of each workpiece on the corresponding machine, thereby adjusting the overall energy consumption, so as to achieve the purpose of saving energy. In the dynamic event of the random arrival of workpieces, due to the inconsistency of the arrival time of the workpieces, the processing time of the workpieces must be after the arrival time, which greatly increases the interval of the workpiece processing time. At the same time, since the machine operates at a certain energy consumption when the workpiece has not arrived, it will cause great loss of energy use. Therefore, adjusting the speed of the machine without changing the total delay can greatly reduce the loss of energy.

[0077] Figure 6 The algorithm of speed regulation based on delay is described, and a series of speed regulation operations are performed according to the scheduling plan generated by the scheduling agent to obtain the speed-regulated scheduling plan; Figure 7 The steps of delay-based speed regulation are described, and the specific contents are as follows.

[0078] First, find the critical path of the scheduling scheme and derive the non-critical operation set N o , assuming that all these non-critical operations meet the deceleration requirements, the arrival time of J2 is A2, the arrival time of the remaining workpieces is 0, the delivery time of J1 is D1, and the delivery time of the remaining workpieces is after the completion time. Next, calculate the adjustable time margin V of all non-critical path operations i,j , and its calculation formula is as follows:

[0079] Where S i,j,k It is operation O i,j On machine M k The start time of C i,j-1,k' It is operation O i,j-1 On machine M k' The completion time on C i,j,k Workpiece J i The last operation is on machine M k The completion time on i',j',k It corresponds to C i,j,k The start time of the next workpiece on the machine. If there is no next workpiece, S i',j',k =C max Then, calculate the available time margin of all non-critical path operations. The available time margin is the maximum time that the machine speed can reach. Finally, adjust the speed of the machine corresponding to the non-critical path operation. Note: When the adjustable time margin V i,jWhen it is less than 0, it means that the completion time of the workpiece is greater than the delivery date of the workpiece. At this time, the workpiece has been delayed, so the speed of the operation should not be adjusted.

[0080] Embodiment 2

[0081] For the multiple objective functions and constraints in S1:

[0082] In the embodiment of the present application, a multi-objective function and constraint conditions corresponding to the planning model are established, the main purpose of which is to reduce the total delay time of workpieces in all factories, minimize the completion time of all factories, and reduce the total energy consumption of all factories. This step includes establishing a multi-objective function and establishing a series of constraint conditions. Table 1 contains the parameters of the mathematical model, Table 2 contains the parameters of the characteristic variables of the scheduling agent, and Table 3 contains the characteristic variable representation of the distribution agent.

[0083] Objective function 1: Total workpiece delay time function - reduce the total workpiece delay time of all factories. Its function expression is:

[0084] In formula (1), the total delay time function TWT of all factories is given by workpiece J i Deadline D i and the corresponding completion time Calculate, n is the total number of workpieces, n i For workpiece J i The total number of operations, For workpiece J i The processing urgency, the larger the value, the more urgent it is.

[0085] Objective function 2: Total energy consumption function - reduce the total energy consumption of all factories

[0086] The total energy consumption of the factory TEC is composed of the processing loss of the equipment EC p and idle loss EC I Calculated, its function expression is: TEC = EC p +EC I (2);

[0087] Equipment processing loss EC p for:

[0088]

[0089] Among them, PP f,k,v is machine M in factory f k Specific energy consumption when running at speed v, p i,j,f,k,v In factory f, workpiece J i At speed v on device M k The actual processing time on

[0090] Idle loss EC I for:

[0091] Among them, X i,j,f,k,v Denote a device M with speed v in factory f k Is process O in progress? i,j If yes, then it is 1, otherwise it is 0. EI f,k Indicates that equipment M in factory f k Unit idle energy consumption.

[0092] Objective function 3: Maximum completion time function of the workpiece - minimize the completion time of all factories. Its expression is: in Is workpiece J i The completion time of the last operation.

[0093] The constraints in S1 above are specifically distributed as follows:

[0094] 1) Each workpiece can only be assigned to one factory, that is: Among them, Z i,f Indicates workpiece J i Is it assigned to factory f? If J i Assigned to Factory J i , then Z i,f is 1, otherwise it is 0;

[0095] 2) All operations for each workpiece should be performed in the same factory, that is:

[0096] 3) Each operation of each workpiece can only be assigned to one available machine, that is:

[0097] 4) The sum of the number of machines used by all operations corresponding to each workpiece should be greater than the total number of machines, that is:

[0098] 5) Each workpiece can only be processed after the workpiece arrives, that is:

[0099] 6) The process of each workpiece must follow the priority order from front to back, that is:

[0100] 7) When workpieces are processed on the same machine, the processing time will not overlap, that is:

[0101]

[0102] 8) All operations on each workpiece have non-negative start times, i.e.:

[0103] The mathematical model of the multi-objective dynamically distributed flexible job shop proposed in the embodiments of the present application addresses the existing workshop scheduling problems and makes up for the shortage of available methods in the scheduling field according to the characteristics of the workshop, and makes a meaningful expansion to the scheduling field of the multi-objective dynamically distributed flexible job shop.

[0104] Embodiment 3

[0105] For the state space, corresponding rules and reward function described in S2:

[0106] 1. State space of distributed agents and scheduling agents

[0107] The state space of the distributed agent includes the basic information of the unassigned workpieces and the processing information of each factory, including 20 pieces of state information in total, including three types of state information: information about the workpieces that have been assigned in the factory, information about the machines where the workpieces are located in the factory, and information about the workpieces that have arrived but have not been assigned. There are 8 types of information about the workpieces that have been assigned in the factory, namely: the maximum and minimum total delay time max (Tard sum ),min(Tard sum ), the maximum and minimum average delay time of different factories max(Tard ave ),min(Tard ave ), the maximum and minimum total processing time of different factories max(PT sum ),min(PT sum ), the maximum and minimum number of workpieces in different factories max(m), min(m); there are three types of machine information in the factory where the assigned workpieces are located, namely: the maximum and minimum average machine utilization rate of different factories max(U ave ),min(U ave ), the maximum and minimum machine completion time max(T cur ),min(T cur ), the average processing load of the largest and smallest machines in different factories (PT ave ),min(PT ave ); There are three types of workpiece information that has arrived but not yet assigned, namely: the largest and smallest delivery time of all workpieces max(DT), min(DT), the largest and smallest average processing time of all workpieces max(PT a ' ve ),min(PT a ' ve), the maximum and minimum arrival time of all workpieces max(AT), min(AT).

[0108] The state space of the scheduling agent includes the production status of the rescheduling, including a total of 15 pieces of status information, namely: the expected delay rate of the workpiece Tard e , the actual delay rate of the workpiece Tard a , average workpiece completion rate CRJ, standard deviation of average workpiece completion rate CRJ std , the total estimated processing time TEPT of unfinished workpieces, the average processing time TEPT of workpieces to be processed avg , the total waiting time TTWT of unfinished workpieces, the average total waiting time TTWT of unfinished workpieces ave , the completion rate of all operations of the workpiece CRO, the total relaxation time TRT of the workpiece to be processed, the average relaxation time TRT of the workpiece to be processed avg , the average utilization of all machines U ave , the standard deviation of the average utilization of all machines U std , the sum of machine idle time SITM, the average value of machine idle time SITM ave .

[0109] 1) Expected delay rate of workpiece:

[0110] The expected delay rate of the workpiece: Then operation O i,j is delayed. OP i (t) is workpiece J i The number of operations at time t, Is workpiece J i Estimated machining time for remaining operations, CT k (t) At time t, machine M k The completion time of workpiece J i The estimated delayed operation at the rescheduling point t can be expressed as:

[0111] The estimated delay rate of the workpiece Tard e is defined as the estimated number of delayed operations divided by the number of all unscheduled operations as follows:

[0112] Among them, UC job (t) = {i|OP i (t)<n i}.

[0113] 2) The actual delay rate of the workpiece:

[0114] definition The set of delayed jobs at the rescheduling point t. The actual delay rate Tard a It is defined as the number of jobs actually delayed divided by the number of all jobs that were not completed as planned, as follows:

[0115]

[0116] 3) The average completion rate of workpieces CRJ is defined as follows:

[0117]

[0118] 4) Standard deviation of the average completion rate of workpieces CRJ std , defined as follows:

[0119]

[0120] 5) The total estimated processing time TEPT of unfinished workpieces is defined as follows:

[0121]

[0122] 6) Estimated average processing time TEPT for unfinished workpieces ave , defined as follows:

[0123]

[0124] 7) The total waiting time TTWT of unfinished workpieces is defined as follows:

[0125]

[0126] 8) The average total waiting time of unfinished workpieces TTWT ave , defined as follows:

[0127]

[0128] 9) The completion rate of all operations of the workpiece, CRO, is defined as follows:

[0129]

[0130] 10) The total relaxation time TRT of unfinished workpieces is defined as follows:

[0131] Among them, now is the current time.

[0132] 11) The average relaxation time TRT of the workpiece to be processed avg , defined as follows:

[0133]

[0134] 12) Average utilization of all machines U ave , defined as follows:

[0135]

[0136] 13) Variance U of the average utilization of all machines std , defined as follows:

[0137]

[0138] 14) The sum of machine idle time SITM is defined as follows:

[0139]

[0140] 15) SITM, the average idle time of the machine ave , defined as follows:

[0141]

[0142] Second. Dispatching rules corresponding to distribution agents and scheduling agents

[0143] Eight artifact selection rules are set for the distribution agent, as follows:

[0144] Rule 1: Randomly select an artifact from the queue and assign it to a factory randomly selected from the available factories.

[0145] Rule 2: Select the workpiece with the shortest arrival time from the queued workpieces and assign it to the factory with the least number of stored workpieces.

[0146] Rule 3: Select the workpiece with the shortest arrival time from the queued workpieces and assign it to the factory with the lowest average machine utilization.

[0147] Rule 4: Select the workpiece with the shortest processing time from the queued workpieces and assign it to the factory with the least number of stored workpieces.

[0148] Rule 5: Select the job with the shortest processing time from the queued jobs and assign it to the factory with the lowest average machine utilization.

[0149] Rule 6: Select the workpiece with the largest delay from the queued workpieces and assign it to the factory with the smallest total delay among the current factories.

[0150] Rule 7: Select the job with the longest delay from the queued jobs and assign it to the factory with the lowest machine utilization rate.

[0151] Rule 8: Select the workpiece with the largest delay from the queued workpieces and assign it to the factory with the least number of currently stored workpieces.

[0152] Five workpiece selection rules and three machine selection rules are set for the scheduling agent, as follows:

[0153] Artifact selection rules:

[0154] Rule 1: Select the workpiece with the smallest current slack time from the unfinished workpieces.

[0155] Rule 2: Select the unfinished workpiece with the smallest estimated slack time after the remaining operations are executed from the unfinished workpieces.

[0156] Rule 2: Select the workpiece with the longest queue time from the unfinished workpieces. If there are multiple workpieces with the longest queue time, select the workpiece with the smallest slack time corresponding to the multiple workpieces.

[0157] Rule 3: Select the first workpiece to arrive from the unfinished workpieces. If there are multiple workpieces that arrive at the same time and they are all the first to arrive at time t, randomly select a workpiece from them.

[0158] Rule 4: Select the workpiece with the shortest delivery time from the unfinished workpieces.

[0159] Rule 5: The workpiece with the shortest remaining processing time.

[0160] Machine selection rules:

[0161] Rule 1: Choose the machine with the lowest utilization.

[0162] Rule 2: Choose the machine with the earliest available time.

[0163] Rule 3: Select the machine with the lowest processing load.

[0164] 3. Distribution agent reward function and scheduling agent reward function

[0165] Artifact Delay Collection: Tard job (t) = {i|OP i (t)<n i &&D i <T cur}(30),

[0166] Estimated relaxation time:

[0167] Different reward functions are designed for the distribution agent and the scheduling agent in this application. The scheduling agent contains the total delay time variable of the workpiece and the waiting time of the workpiece to be processed in the buffer pool. When the total delay and completion time of the factory are small, a larger reward is obtained, otherwise a small reward is given to punish the action; the distribution agent is related to the maximum and minimum values ​​of the corresponding rewards of each factory. When the delay and completion time of each factory are relatively balanced, a larger reward is given to affirm it, otherwise it is based on a small reward and punishment.

[0168] The scheduling agent reward function is as follows:

[0169] Where TWT(t) is the total delay at time t, It is operation O i,j+1 The start time of It is operation O i,j The completion time of , α, β are hyper parameters.

[0170] This is a local reward function, that is, at each time step t step Afterwards, the scheduling agent is provided with immediate feedback on the quality of its actions and their impact on the environment. Although this reward is simple, it shows multi-objective behavior. First, the reward function positively affects the total delay of all jobs at the end by incentivizing the total delay of each operation. By supporting the selection of operations that incur little or no delay, the agent prioritizes processing jobs that will not incur delays or processing jobs that may incur large delays, which helps to continuously reduce the total delay. Second, the ratio of the average idle time of the machine to the total queue time also has a direct impact on the idle time of the machine. During the idle waiting time, jobs remain in the buffer without being processed. By minimizing the queue time of the job, the reward function encourages the agent to prioritize and efficiently assign jobs to idle machines, thereby reducing the idle time of the machine before processing the job, thereby increasing the utilization of the machine, and ultimately making the maximum completion time smaller and smaller. At the same time, when the machine utilization rate increases, the scheduling agent processes the job more efficiently, which indirectly but effectively reduces the delay of the job. Finally, the reward function addresses machine downtime. By penalizing the accumulated idle time of the job, it encourages the efficient allocation of work to the machine, thereby avoiding long downtime. This helps minimize machine downtime due to lack of available work, thereby improving overall resource utilization.

[0171] The distribution agent should satisfy the intention of minimizing the total tardiness and the maximum completion time in a single factory. Therefore, the design of the distribution agent reward should be related to the research objectives of different factories.

[0172] The distribution agent reward function is as follows: R a (t)=-(max(Rf (t)-min(R f (t))))(33) Among them, R f (t) is the reward value of the scheduling agent in factory f.

[0173] Embodiment 4

[0174] For the solution of the scheduling agent algorithm, distribution agent algorithm and multi-objective dynamic distributed flexible job shop described in S3:

[0175] First, about the scheduling agent algorithm

[0176] Scheduling agent algorithm such as Figure 2 As shown in the flow chart, Figure 4 As shown in the figure. In this algorithm, the scheduling agent applies the DDDQ training algorithm, which aims to eliminate the problem of overestimating Q values ​​in DQN and provide a more stable algorithm for the training of the scheduling agent. In addition, the algorithm also introduces the experience replay pool mechanism, which can eliminate the correlation between data and provide more data for agent training, which can speed up the training process.

[0177] Specifically, we first initialize the initial parameters of the playback experience pool N, the target network Q, and the online network Qμ. Then we iterate, and the iterative parameter L is a hyperparameter. At the beginning of each iteration, we need to p1 Perform initialization operation, then select the action with the highest probability And execute. Note: At the beginning of the iteration, the greedy strategy value ε is used to select the corresponding action. With the continuous iteration, the greedy strategy value ε keeps decreasing until it is the minimum. This can better explore the execution of different actions in different environments and avoid the algorithm falling into the local optimum too quickly. After executing the corresponding action, the corresponding next state s is obtained p(t+1) and the reward r for this action pt , put the initial state, action, next state and reward into the replay experience pool N. When the data in the replay experience pool N exceeds a certain limit, start to extract small sample data from the replay experience pool, and then calculate the y of the data pj Value and loss function (y pj -Q(φ pj ,a pj θ p )) 2 , then perform a gradient descent operation on the loss function and continuously update the parameters θ of the online network Q p , update the target network parameters every C steps Repeat the above steps until the algorithm ends. Figure 4 In the example, termination condition 1 indicates whether the operation has been run L times, and termination condition 2 indicates whether all operations have been processed.

[0178] Second, about the distributed agent algorithm

[0179] The algorithm of distributed agents is as follows Figure 3 As shown in the main flow chart Figure 4 As shown in the figure. In this algorithm, the distribution agent is also the training algorithm of the applied DDQN. In this algorithm, except for the different way in which the distribution agent obtains the state, the rest of the parts are consistent with the scheduling agent algorithm. After the distribution agent assigns the workpiece, the scheduling agent obtains the mean of all the final states after multiple pre-processing, and uses these means as the state of the distribution agent. In addition, during the training process of the distribution agent, the scheduling agent will also be trained at the same time.

[0180] Specifically, the algorithm steps of the distribution agent are similar to those of the scheduling agent. The difference between the distribution agent lies in its timeliness and state. In the multi-agent training process, since the distribution agent has a certain lag, that is, at the beginning of the training, the performance of the distribution agent has not been trained many times, so the pros and cons of the distribution agent in allocating workpieces are unstable and uncertain. In order to solve this problem, the invention adopts a two-stage algorithm for multi-agent training. In the first stage of the algorithm, the scheduling agent is first trained separately, and the second stage of training begins after the scheduling agent achieves good performance. In the second stage of the algorithm, the distribution agent is responsible for operating the unallocated workpieces until all workpieces are allocated. Whenever the scheduling agent allocates a workpiece, it will put the workpiece into a temporary pool F corresponding to each factory, and then the scheduling agent will pre-process the workpiece in the temporary pool F. Then the actions, current eigenvalues, mean of eigenvalues ​​and rewards of the distribution agent are placed in the experience replay pool. In addition, their states are different. The scheduling agent continues to work until all operations of the corresponding factory have been completed. At the decision point t, the scheduling agent observes the state s pt To take an action pt The workpiece operation and its corresponding processable machine determine the corresponding rules through actions. After the job is scheduled, the reward r is calculated pt , and updates the state s for the next decision p(t+1) . Experience(s) pt ,a pt ,r pt ,s p(t+1) ) is stored in the agent's replay memory. At each decision point, the allocation agent should check whether new work has arrived. Before allocating a workpiece, record the allocation agent state s corresponding to each factory at , and take action a according to the current state at, where the action can be used to achieve job allocation through the composite allocation rule corresponding to the action. Since there is random selection in the scheduling agent's rules, the result of each processing fluctuates. Therefore, the temporary pool F is processed N times, and the corresponding allocation agent feature value is averaged N times. After the workpiece is allocated, the reward r is calculated at , and update the status to s according to the processed factory information a(t+1) Finally, the experience (s at ,a at ,r at ,s a(t+1) ) is stored in the replay memory of the allocation agent. Note that when there are jobs that are not allocated, the allocation agent will continue to work until the artifacts are fully allocated.

[0181] Table 1: Relevant variables of mathematical model

[0182]

[0183]

[0184] Table 2: Parameters of the characteristic variables of the scheduling agent

[0185]

[0186]

[0187] Table 3: Characteristic variables of distribution agents

[0188]

[0189] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0190] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-objective dynamic distributed flexible job shop scheduling method based on deep Q network, characterized in that: include: S1. According to the scheduling requirements of the multi-objective dynamic distributed flexible job shop, a mathematical model of the multi-objective dynamic distributed flexible job shop is established; The mathematical model of the multi-objective dynamic distributed flexible job shop consists of three objective functions and constraints. The three objective functions are the total delay time function of the workpiece, the total energy consumption function and the maximum completion time function of the workpiece. S2. Construct the state space of the distribution agent and the scheduling agent respectively, and design corresponding scheduling rules and reward functions for the two agents; The state spaces of the distribution agent and the scheduling agent are inconsistent. The state space of the distribution agent comprehensively reflects the allocation status of the rescheduling point or decision point and the production status of each factory, while the state space of the scheduling agent reflects the rescheduling production status. The corresponding scheduling rules designed for the two agents include: 8 workpiece selection rules set for the distribution agent, and 5 workpiece selection rules and 3 machine selection rules set for the scheduling agent; the corresponding reward functions designed for the two agents are the distribution agent incentive function and the scheduling agent reward function; S3, using multi-agent collaborative algorithm to solve the problem of multi-objective dynamic distributed flexible job shop; The multi-agent collaborative algorithm includes the scheduling agent algorithm and the distributed agent algorithm, both of which are composed of DDQN. The interaction between the DDQN multi-agent and the multi-objective dynamic distributed flexible job shop scheduling environment is a discrete-time Markov decision process model; S4, construct the distribution agent structure and the scheduling agent structure corresponding to the distribution agent and the scheduling agent respectively; Both the distribution agent structure and the scheduling agent structure are composed of DDQN, which are connected by the environment of undistributed artifacts; S5. Formulate a speed regulation and energy saving strategy based on delay; The speed regulation energy-saving strategy is to adjust the corresponding processing time by adjusting the processing speed of each workpiece on the corresponding machine, thereby adjusting the overall energy consumption.

2. The multi-objective dynamic distributed flexible job shop scheduling method according to claim 1 is characterized in that: The constraints are set as follows: 1) Each workpiece can only be assigned to one factory; 2) All operations of each workpiece should be performed in the same factory; 3) Each operation of each workpiece can only be assigned to one available machine; 4) The sum of the number of machines used by all operations corresponding to each workpiece should be greater than the total number of machines; 5) Each workpiece can be processed only after the workpiece arrives; 6) The process of each workpiece must follow the priority order from front to back; 7) When the workpieces are processed on the same machine, the processing time will not overlap; 8) All operations of each workpiece have a non-negative start time; 9) The transportation time between workshops and equipment is not considered.

3. The multi-objective dynamic distributed flexible job shop scheduling method according to claim 1, characterized in that: The function expression for minimizing the total delay time function of the workpiece is: In the formula, the total delay time function TWT of all factories is given by workpiece J i Deadline D i and the corresponding completion time Calculate, n is the total number of workpieces, n i For workpiece J i The total number of operations, For workpiece J i The processing urgency, the larger the value, the more urgent it is; The function expression for minimizing the maximum completion time of a job is: in Is workpiece J i The completion time of the last operation; The total energy consumption of the factory TEC is composed of the processing loss of the equipment EC p and idle loss EC I Calculated, its function expression is: TEC = EC p +EC I ; Among them, the processing loss of equipment EC p for: Idle loss EC I for: Where, PP f,k,v is the machine M in factory f k Energy consumption during processing with speed v, EI f,k is the machine M in factory f k Specific no-load energy consumption when idle, p i,j,f,k,v For workpiece J i In factory f, machine M k The actual processing time when processing at speed v.

4. The multi-objective dynamic distributed flexible job shop scheduling method according to claim 1, characterized in that: The eight artifact selection rules set for the distribution agent are as follows: Rule 1: Randomly select a workpiece from the queue and assign it to a factory randomly selected from the available factories; Rule 2: Select the workpiece with the shortest arrival time from the queue and assign it to the factory with the least number of stored workpieces; Rule 3: Select the workpiece with the shortest arrival time from the queued workpieces and assign it to the factory with the lowest average machine utilization rate; Rule 4: Select the workpiece with the shortest processing time from the queued workpieces and assign it to the factory with the least number of stored workpieces; Rule 5: Select the workpiece with the shortest processing time from the queued workpieces and assign it to the factory with the lowest average machine utilization rate; Rule 6: Select the workpiece with the largest delay from the queued workpieces and assign it to the factory with the smallest total delay among the current factories; Rule 7: Select the longest-delayed workpiece from the queue and assign it to the factory with the lowest machine utilization rate. Rule 8: Select the most delayed workpiece from the queue and assign it to the factory with the least number of workpieces currently stored; The five artifact selection rules set for the scheduling agent are as follows: Rule 1: Select the workpiece with the smallest current slack time from the unfinished workpieces; Rule 2: Select the workpiece with the smallest estimated slack time after the remaining operations are executed from the unfinished workpieces; Rule 2: Select the workpiece with the longest queue time from the unfinished workpieces. If there are multiple workpieces with the longest queue time, select the workpiece with the smallest slack time corresponding to the multiple workpieces. Rule 3: Select the first workpiece to arrive from the unfinished workpieces. If there are multiple workpieces that arrive at the same time and they are all the first to arrive at time t, randomly select one of them; Rule 4: Select the workpiece with the shortest delivery time from the unfinished workpieces; Rule 5: Select the workpiece with the shortest remaining processing time; The three machine selection rules set for the scheduling agent are as follows: Rule 1: Select the machine with the lowest utilization; Rule 2: Select the machine with the earliest availability; Rule 3: Select the machine with the lowest processing load.

5. The multi-objective dynamic distributed flexible job shop scheduling method according to claim 1, characterized in that: The scheduling agent reward function is: Where TWT(t) is the total delay at time t, It is operation O i,j+1 The start time of It is operation O i,j The completion time of , α, β are hyper parameters; The distribution agent reward function is as follows: R a (t)=-(max(R f (t)-min(R f (t)))), where R f (t) is the reward value of the scheduling agent in factory f.

6. The multi-objective dynamic distributed flexible job shop scheduling method according to claim 1, characterized in that: The scheduling agent algorithm is the training algorithm of DDDQ, and the experience replay pool mechanism is introduced into the algorithm; The distributed agent algorithm is also the training algorithm of DDQN. In this algorithm, except for the different way in which the distributed agent obtains the state, the rest of the parts are consistent with the scheduling agent algorithm.

7. The multi-objective dynamic distributed flexible job shop scheduling method according to claim 1, characterized in that: The DDQNs of both the distribution agent and the scheduling agent consist of 1 input layer, 5 hidden layers, and 1 output layer; The distributed agent has 20 input nodes, corresponding to 20 state features; each hidden layer has 200 nodes; the output layer has 8 nodes, corresponding to 8 optional actions; The scheduling agent has 15 input nodes, corresponding to 15 input nodes; each hidden layer has 200 nodes; the output layer has 12 nodes, corresponding to 12 optional actions.

Images