A multi-agent d3qn-based real-time scheduling method for hybrid flow manufacturing workshops

Abstract

The application provides a kind of based on multi-agent D3QN's mixed flow manufacturing workshop real-time scheduling method, comprising: the MPN model of workshop is constructed and input to be executed order is initialized;Random scheduling simulation is carried out, and Markov decision chain data is constructed and loaded into sample data pool;Initialize multiple different scheduling target agents, construct D3QN model for each agent;Randomly extract data from sample data pool and train each agent;Finally, the real-time state of workshop is input into MPN model;After calling the D3QN model of each agent trained, the real-time MPN model is scheduled and calculated, and several workshop scheduling strategies oriented to different scheduling targets are obtained, and production is scheduled according to the workshop scheduling strategy;The present application can enable production managers to consider and decide in multiple dimensions, so as to better optimize production scheduling and resource utilization, improve overall production efficiency and economic benefit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep reinforcement learning and job shop intelligent scheduling, and more particularly to a mixed flow manufacturing job shop real-time scheduling method based on multi-agent D3QN (Double Dueling Deep Q-Network). BACKGROUND

[0002] Job shop scheduling problem as a combination problem with a wide range of engineering background, the research on its theory and solving method has important significance to guide the production of manufacturing enterprises. The solving algorithms of this kind of problem in the past mainly include heuristic scheduling algorithm and meta-heuristic scheduling algorithm, but they have problems such as time-consuming solving process, weak algorithm scene applicability, etc.

[0003] With the proposal of deep reinforcement learning theory and its successful application in game decision-making, combinatorial optimization problems and other fields, it provides a new idea and method for solving job shop scheduling problem. Deep reinforcement learning combined with scheduling rules can make up for the defects of traditional scheduling methods in the application of historical data, and obtain a scheduling scheme that meets the actual needs of production. Reinforcement learning is the process in which an agent obtains the maximum reward through continuous interaction with the environment. In the learning process, the agent will output an action according to the current state of the environment, which will affect the environment and change the state of the environment. At the same time, the agent will be rewarded for the action of decision-making, and the agent will optimize its subsequent action selection according to the reward obtained. In the process of continuous trial and error, the agent's policy optimization is completed.

[0004] Although many scholars have achieved good results in job shop scheduling problem using deep reinforcement learning, it is found that the previous solutions to job shop scheduling problem, whether using approximate calculation method or deep reinforcement learning method, mostly focus on a single scheduling target, such as minimizing the maximum completion time. However, due to the different requirements of different departments of the enterprise for production, the goal of the production department is to minimize the cost of products and complete the task in the shortest time, the sales department hopes that the production department can deliver on time, and the top management of the enterprise hopes to reduce resource waste, so the job shop scheduling problem often appears the problem of multiple objectives needing to be optimized at the same time.

[0005] The existing patent file discloses a workpiece flow-oriented intelligent workshop real-time scheduling method based on D3QN reinforcement learning, first collects the job data of the workshop and processes it into Markov decision chain data, trains an online Q network and a target Q network of D3QN, and uses the trained online Q network as a knowledge network to match the real-time state of the workshop, optimizes production scheduling, and repeatedly executes until the production is completed; However, this scheme is to overcome the problems of traditional algorithms in large-scale manufacturing workshops, such as easy to fall into local optimal solution and long solving time, and still belongs to the solution of a single scheduling target, and the optimization effect for multiple scheduling targets is poor. SUMMARY

[0006] In order to overcome the defects of the prior art that it is difficult to handle the workshop scheduling problem of multiple targets needing to be optimized at the same time, a mixed flow manufacturing workshop real-time scheduling method based on multi-agent D3QN is provided, which can obtain mixed flow manufacturing workshop scheduling strategies oriented by different targets in one scheduling calculation process, and improve overall production efficiency and economic benefits.

[0007] To solve the above technical problems, the technical scheme of the present application is as follows:

[0008] A mixed flow manufacturing workshop real-time scheduling method based on multi-agent D3QN, comprising the following steps:

[0009] S1: Constructing a MPN (Manufacturing Petri Net) model based on Petri net according to the operation relationship of workpieces and processing equipment in the mixed flow manufacturing workshop;

[0010] S2: Obtaining the job data of the to-be-executed order of the mixed flow manufacturing workshop and inputting the MPN model, obtaining the initialized MPN model, and initializing a production scene for the mixed flow manufacturing workshop after initialization, so as to introduce multiple agents subsequently; The job data includes: the number and type of workpieces of the to-be-executed order, and the state information of the processing equipment in the processing process of each workpiece;

[0011] S3: Random scheduling simulation based on the initialized MPN model, obtaining simulation data; constructing Markov decision chain data according to the simulation data and the job data and loading it into a preset sample data pool;

[0012] S4: Initializing multiple agents, constructing a D3QN model for each agent; the scheduling target of each agent is different;

[0013] The D3QN model includes two EfficientNet neural networks with the same structure, which are used as the online evaluation network Q and the target value network Q of the D3QN model - ;

[0014] S5: Randomly extract a plurality of Markov decision chain data from the sample data pool and train the D3QN model of each agent to obtain the trained D3QN model of each agent;

[0015] S6: Obtain the real-time state of the mixed flow manufacturing workshop and input the MPN model to obtain the real-time MPN model; after calling the trained D3QN model of each agent to perform scheduling calculation on the real-time MPN model, a plurality of workshop scheduling strategies oriented to different scheduling targets are obtained, and production is scheduled according to the workshop scheduling strategies.

[0016] Preferably, the step S1 comprises:

[0017] Each parallel running machining device in each process of the mixed flow manufacturing workshop is abstracted into a library, representing that the workpiece can be transferred to the machining device when being processed in a process; different libraries are connected through an intermediate transition path, representing that the workpiece is transferred between different processes, i.e. a path from one machining device to the next machining device after the machining device is finished processing;

[0018] The expression of the whole MPN model is as follows:

[0019] MPN = <Token, Place.Transition>

[0020] Token = {Token1, Token2, …, Token i}

[0021] Place = {Place1, Place2, …, Place j}

[0022] Transition = {Transition1, Transition2, …, Transition h}

[0023] Wherein, Token is a set of expressions of the workpiece in the MPN model, Token i is the expression of the i-th workpiece; Place is a set of expressions of the machining device in the MPN model, Place j is the expression of the j-th machining device; Transition is a set of expressions of the transition path in the MPN model, Transition h is the h-th transition path.

[0024] Preferably, the expression of the i-th workpiece in the MPN model is as follows:

[0025]

[0026] Token i represents the i-th workpiece, and n is the number of workpieces; ID1 is the number of the i-th workpiece, satisfying ID1∈N, N is a natural number set; C is the workpiece type, satisfying C∈{Color1, Color2, …, Color c}, Color1~Color c are the 1st~c-th workpiece types, respectively; is the processing time of the i-th workpiece on the j-th processing equipment, and m is the number of processing equipment; P is the number of the processing equipment where the i-th workpiece is currently located, satisfying P∈N; T is the remaining processing time of the i-th workpiece in the current equipment, satisfying T∈R, R is a real number set;

[0027] The expression of the j-th processing equipment in the MPN model is as follows:

[0028]

[0029] Token j represents the j-th processing equipment; ID2 is the number of the j-th processing equipment, satisfying ID2∈N; Capacity is the number of workpieces that the processing equipment can accommodate, is the workpiece in the j-th processing equipment;

[0030] The expression of the h-th transition path in the MPN model is as follows:

[0031] Transition h =<ID3, inPlace, outPlace>

[0032] ID3 is the number of the transition, satisfying ID3∈N, inPlace is the number of the current processing equipment, satisfying inPlace∈N; outPlace is the number of the next processing equipment, satisfying outPlace∈N.

[0033] Preferably, in the step S2, the number of workpieces, the workpiece number and the workpiece type of each workpiece, and the processing equipment where each workpiece is located during processing, the remaining processing time of the workpiece, are extracted from the job data, and all the extracted information is loaded into the MPN model for initialization, to obtain the initialized MPN model.

[0034] Preferably, in the step S3, the whole process of starting to ending of the machining of all workpieces of the to-be-executed order is simulated using a random scheduling strategy, a batch of simulation data is generated, and the job data is combined to construct a workpiece scheduling time matrix ST (Schedule Time Matrix), a workpiece position transfer marking matrix MM (Marking Matrix), and a workpiece device delay matrix MT (Machining Time Matrix);

[0035] According to the three matrices obtained, a three-channel workshop state s i is constructed. i i The selected action a k , the action a k , the corresponding reward r, and the next workshop state s i+1 jointly constitute a Markov decision chain data <s i , a k , r, s i+1 > and are loaded into a preset sample data pool.

[0036] Preferably, the expression of the workshop state s i is as follows:

[0037] s i = <ST n×m , MM n×m , MT n×m >

[0038] Wherein, n x m is the size of the matrix, n is the number of workpieces in the MPN model, and m is the number of machining devices in the MPN model.

[0039] The set of actions Job(s i ) that can be triggered by the MPN model under the current workshop state s i is obtained, wherein the action refers to the process of transferring a specific workpiece from a certain process device to a next process device, and the action a k is expressed as follows:

[0040] a k = <ID1, inPlace, outPlace, Time>

[0041] Wherein, Time is the time when the action is triggered, and Time ∈ R is satisfied.

[0042] The set Job(s i ) is composed of a plurality of actions that can be triggered by the workpiece under the workshop state s i , and the expression of the action set Job(s i ) is as follows:

[0043] ​Job(s i )={a1,a2,...,a k}

[0044] The action set Job(s i ) is used to represent the direction of the workpiece transferable under the workshop state s i , and when the actual production transfer is performed, one action is selected from a plurality of actions to execute, and after the execution is completed, the workshop state s i is updated to s i+1 , and the action set Job(s i ) is also updated to Job(s i+1 );

[0045] According to the degree of fit between the result t of the random scheduling simulation and the preset scheduling target tar, the action a k executed under the workshop state s i enters a new state s i+1 , and a corresponding reward r is given;

[0046] The mixed flow manufacturing workshop starts from the order placement state s0 and ends at the finished state s n , and forms a piecewise Markov decision chain data <s i , a k , r, s i+1 > in the order of n times of scheduling.

[0047] Preferably, in the step S4, initializing a plurality of agents includes: setting different agents and different reward functions respectively with the shortest completion time, the shortest average completion time and the highest equipment utilization as the scheduling target tar;

[0048] The reward function is:

[0049]

[0050] Wherein, R + and R- respectively represent fixed positive and negative reward values, and tar is the scheduling target;

[0051] Each of the agents is respectively provided with a D3QN model;

[0052] The D3QN model includes two EfficientNet neural networks with the same structure, which are respectively used as an online evaluation network Q and a target value network Q- of the D3QN model.

[0053] Preferably, in the step S5, the estimated value is obtained by inputting the batch data read from the sample data pool into the online evaluation network Q, and the target value is obtained by inputting the batch data into the target value network Q-, and the mean square error of the estimated value and the target value is used as the loss function of the D3QN model for supervised training.

[0054] The loss function of the D3QN model is specifically:

[0055]

[0056]

[0057] Wherein, L e (θ e ) is the loss function value of the D3QN model; subscript e is the training step, represents the mean square error calculation, (s i , a k , r, s i+1 ) ~ U(DB) represents the training sample data extracted from the sample data pool, Y e represents the target value; Q(s i , a k ; θ e ) represents the estimated value obtained by inputting the current workshop state s i and action a k to the online evaluation network Q; γ is a discount factor, and the larger the discount factor, the more the network training focuses on the change of the future state; argmaxaQ(s i+1 , a; θ e ) represents the most valuable action a for the next state s i+1 evaluated by the online evaluation network Q;

[0058] Using the stochastic gradient descent strategy, the weight parameters of the online evaluation network Q are reversely updated and trained; every training step, the parameters of the online evaluation network Q last updated are copied to update the parameters of the target value network Q-; repeatedly training until the loss function value of the D3QN model is less than the preset threshold, and the trained D3QN model is obtained.

[0059] Preferably, in the step S6, the real-time state of the mixed flow manufacturing workshop is obtained and input into the MPN model, the real-time MPN model and the to-be-scheduled task under the real-time state are obtained;

[0060] The real-time state is input into the D3QN model trained by each intelligent agent, and each intelligent agent selects the optimal value scheduling action from the to-be-scheduled task under the real-time state according to the preset scheduling target, and updates the next real-time state and the corresponding to-be-scheduled task; the optimal value scheduling action is repeatedly selected and executed until all workpiece production is completed, and a plurality of workshop scheduling strategies oriented to different scheduling targets are obtained;

[0061] For the same to-be-executed order, the corresponding workshop scheduling strategy is called according to different scheduling targets to schedule production.

[0062] Preferably, the step S6 further comprises:

[0063] Each workshop scheduling strategy is scored under different scheduling objectives using a weighted comprehensive scoring method to obtain the priority of each workshop scheduling strategy under different scheduling objectives;

[0064] According to the priority of each workshop scheduling strategy under different scheduling objectives, a workshop scheduling strategy with priority meeting preset conditions under different scheduling objectives is selected as an optimal scheduling strategy, and production is comprehensively scheduled according to the optimal scheduling strategy.

[0065] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0066] The present application provides a kind of based on multi-agent D3QN's mixed flow manufacturing workshop real-time scheduling method, first according to the running relationship of workpiece and processing equipment in mixed flow manufacturing workshop constructs MPN model based on Petri net;Job data of mixed flow manufacturing workshop to be executed order are acquired and input MPN model, and the MPN model after initialization is acquired;Subsequently, random scheduling simulation is carried out based on the MPN model after initialization, and simulation data are acquired;Markov decision chain data are constructed according to simulation data and job data and loaded into preset sample data pool;Initialize multiple agents, and construct D3QN model for each agent;The scheduling target of each agent is different;Afterwards, a plurality of Markov decision chain data are randomly extracted from sample data pool and the D3QN model of each agent is trained, and the D3QN model of each agent after training is acquired;Finally, the real-time state of mixed flow manufacturing workshop is acquired and input MPN model, and real-time MPN model is acquired;After calling the D3QN model of each agent after training, the real-time MPN model is scheduled and calculated, and a plurality of workshop scheduling strategies oriented to different scheduling objectives are obtained, and scheduling production is carried out according to the workshop scheduling strategy;

[0067] The present application uses EfficientNet neural network as the basic network of D3QN model, utilizes the high-efficiency and lightweight characteristics of EfficientNet neural network, significantly reduces the amount of calculation and parameter while maintaining high precision, thereby improving the performance and training speed of D3QN algorithm in processing large-scale scheduling problems;At the same time, by introducing the mechanism of multi-agent, the scheduling algorithm can obtain mixed flow manufacturing workshop scheduling strategies oriented to different objectives in one scheduling calculation process, so that production managers can consider and decide in multiple dimensions, thereby better optimizing production scheduling and resource utilization, improving overall production efficiency and economic benefits. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1A multi-agent D3QN-based real-time scheduling method for a mixed-model manufacturing workshop provided in Embodiment 1.

[0069] Figure 2 A three-channel workshop state transition schematic diagram provided in Embodiment 2.

[0070] Figure 3 A multi-agent D3QN algorithm framework schematic diagram provided in Embodiment 2. DETAILED DESCRIPTION

[0071] The accompanying drawings are only intended to illustrate, and cannot be understood as a limitation to the patent;

[0072] In order to better illustrate the embodiments, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0073] It can be understood by those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0074] The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0075] Embodiment 1

[0076] As shown in Figure 1 , the present embodiment provides a multi-agent D3QN-based real-time scheduling method for a mixed-model manufacturing workshop, comprising the following steps:

[0077] S1: constructing an MPN model based on Petri net according to the operation relationship between workpieces and processing equipment in the mixed-model manufacturing workshop;

[0078] S2: obtaining the job data of the to-be-executed order of the mixed-model manufacturing workshop and inputting the MPN model to obtain the initialized MPN model; the job data includes the number and type of workpieces of the to-be-executed order, and the state information of the processing equipment in the processing process of each workpiece;

[0079] S3: performing random scheduling simulation based on the initialized MPN model to obtain simulation data; constructing a Markov decision chain data according to the simulation data and the job data and loading it into a preset sample data pool;

[0080] S4: initializing a plurality of agents and constructing a D3QN model for each agent; the scheduling target of each agent is different;

[0081] The D3QN model includes two EfficientNet neural networks with the same structure, which are respectively used as the online evaluation network Q and the target value network Q - of the D3QN model.

[0082] S5: Randomly extract several pieces of Markov decision chain data from the sample data pool and train the D3QN model of each agent to obtain the trained D3QN model of each agent;

[0083] S6: Obtain the real-time state of the mixed-model manufacturing workshop and input the MPN model to obtain the real-time MPN model; after calling the trained D3QN model of each agent to perform scheduling calculation on the real-time MPN model, several workshop scheduling strategies oriented to different scheduling targets are obtained, and production is scheduled according to the workshop scheduling strategies.

[0084] In the specific implementation process, first, an MPN model based on Petri net is constructed according to the running relationship of workpieces and processing equipment in the mixed-model manufacturing workshop; job data of orders to be executed in the mixed-model manufacturing workshop are obtained and input into the MPN model to obtain an initialized MPN model; then, random scheduling simulation is performed based on the initialized MPN model to obtain simulation data; Markov decision chain data are constructed according to the simulation data and the job data and loaded into a preset sample data pool; a plurality of agents are initialized, and a D3QN model is constructed for each agent; the scheduling targets of each agent are different; then, several pieces of Markov decision chain data are randomly extracted from the sample data pool, and the D3QN model of each agent is trained to obtain the trained D3QN model of each agent; finally, the real-time state of the mixed-model manufacturing workshop is obtained and input into the MPN model to obtain a real-time MPN model; after calling the trained D3QN model of each agent to perform scheduling calculation on the real-time MPN model, several workshop scheduling strategies oriented to different scheduling targets are obtained, and production is scheduled according to the workshop scheduling strategies;

[0085] The method adopts an EfficientNet neural network as the basic network of the D3QN model, utilizes the high-efficiency and lightweight characteristics of the EfficientNet neural network, significantly reduces the amount of calculation and the amount of parameters while maintaining high precision, thereby improving the performance and training speed of the D3QN algorithm in processing large-scale scheduling problems; at the same time, the method introduces the mechanism of multiple agents, can obtain mixed-model manufacturing workshop scheduling strategies oriented to different targets in one scheduling calculation process of the scheduling algorithm, and enables production managers to consider and decide in multiple dimensions, thereby better optimizing production scheduling and resource utilization and improving overall production efficiency and economic benefits.

[0086] Embodiment 2

[0087] The embodiment provides a mixed-model manufacturing workshop real-time scheduling method based on multiple-agent D3QN, which comprises the following steps:

[0088] S1: An MPN model based on Petri net is constructed according to the running relationship of workpieces and processing equipment in the mixed-model manufacturing workshop;

[0089] S2: Obtain job data of orders to be executed in the mixed flow manufacturing workshop and input the MPN model, and obtain an initialized MPN model; the job data includes: the number and types of workpieces of the orders to be executed, and state information of processing equipment of each workpiece in the processing process;

[0090] S3: Perform random scheduling simulation based on the initialized MPN model to obtain simulation data; and construct Markov decision chain data according to the simulation data and the job data and load the data into a preset sample data pool;

[0091] S4: Initialize a plurality of agents, and construct a D3QN model for each agent; the scheduling targets of each agent are different;

[0092] The D3QN model includes two EfficientNet neural networks with consistent structures, which are used as an online evaluation network Q and a target value network Q- of the D3QN model, respectively;

[0093] S5: Randomly extract a plurality of Markov decision chain data from the sample data pool and train the D3QN model of each agent to obtain a trained D3QN model of each agent;

[0094] S6: Obtain a real-time state of the mixed flow manufacturing workshop and input the MPN model to obtain a real-time MPN model; after calling the trained D3QN model of each agent to perform scheduling calculation on the real-time MPN model, a plurality of workshop scheduling strategies oriented to different scheduling targets are obtained, and production is scheduled according to the workshop scheduling strategies;

[0095] The step S1 includes:

[0096] Each of a plurality of processing equipment running in parallel in each process of the mixed flow manufacturing workshop is abstracted as a library, indicating that the workpiece can be transferred to the processing equipment when being processed in a process; different libraries are connected through an intermediate transition path, indicating that the workpiece is transferred between different processes, that is, a path from one processing equipment to the next processing equipment after the processing equipment is processed;

[0097] The expression of the entire MPN model is as follows:

[0098] MPN = <Token, Place.Transition>

[0099] Token = {Token1, Token2, …, Token i}

[0100] Place = {Place1, Place2, …, Place j}

[0101] Transition = {Transition1, Transition2, …, Transition h}

[0102] Token = {Token1, Token2, …, Token i}, Place = {Place1, Place2, …, Place j}, Transition = {Transition1, Transition2, …, Transition h}

[0103] The expression of the ithworkpiece in the MPN model is as follows:

[0104]

[0105] Token i represents the ithworkpiece, and n is the number of workpieces; ID1 is the number of the ithworkpiece, and satisfies ID1 ∈ N, N is a natural number set; C is the workpiece type, and satisfies C ∈ {Color1, Color2, …, Color c}, Color1 ~ Color c are the 1st ~ cthworkpiece types, respectively; is the processing time of the ithworkpiece on the jthprocessing equipment, and m is the number of processing equipment; P is the number of the processing equipment where the ithworkpiece is currently located, and satisfies P ∈ N; T is the remaining processing time of the ithworkpiece in the current equipment, and satisfies T ∈ R, R is a real number set;

[0106] The expression of the jthprocessing equipment in the MPN model is as follows:

[0107]

[0108] Place j represents the jthprocessing equipment; ID2 is the number of the jthprocessing equipment, and satisfies ID2 ∈ N; Capacity is the number of workpieces that the processing equipment can accommodate, is the number of workpieces in the jthprocessing equipment;

[0109] The expression of the hthtransition path in the MPN model is as follows:

[0110] Transition h = <ID3, inPlace, outPlace>

[0111] Wherein, ID3 is the number of transition, ID3∈N is met, inPlace is the number of the current processing equipment, inPlace∈N is met; outPlace is the number of the next processing equipment, outPlace∈N is met;

[0112] In the step S2, the workpiece quantity, the workpiece number and the workpiece type of each workpiece, and the processing equipment where each workpiece is located in the processing process, the remaining processing time of the workpiece are extracted from the job data, and all the extracted information is loaded into the MPN model for initialization to obtain the initialized MPN model;

[0113] In the step S3, the random scheduling strategy is used to simulate the whole process from the start to the end of the processing of all workpieces of the to-be-executed order, batch simulation data is generated, and the job data is combined to construct a workpiece scheduling time matrix ST, a workpiece position transfer identification matrix MM and a workpiece device delay matrix MT;

[0114] The three-channel workshop state s i is constructed according to the three matrices. i i The selected action a k , the action a k , the corresponding reward r and the next workshop state s i+1 jointly constitute a Markov decision chain data <s i , a k , r, s i+1 > and are loaded into a preset sample data pool.

[0115] The expression of the workshop state s i is as follows:

[0116] s i =<ST n×m , MM n×m , MT n×m >

[0117] Wherein, n×m is the size of the matrix, n is the number of workpieces in the MPN model, and m is the number of processing equipment in the MPN model.

[0118] The action set Job(s i ) that can be triggered by the MPN model under the current workshop state s i is obtained, wherein the action refers to the process that a specific workpiece is transferred from a certain process equipment to a next process equipment, and the action a k expression is as follows:

[0119] a k =<ID1, inPlace, outPlace, Time> ​

[0120] wherein Time is the time when the action is triggered, satisfying Time∈R;

[0121] from the workshop state s i The set of actions Job(s i ) triggered by the workpiece under the workshop state s i is expressed as follows:

[0122] Job(s i )={a1,a2,...,a k}

[0123] The action set Job(s i ) is used to represent the direction of the workpiece transfer under the workshop state s i , and when the actual production transfer is performed, one action is selected from the plurality of actions, and after the execution is completed, the workshop state s i is updated to s i+1 , and the action set Job(s i ) is also updated to Job(s i+1 );

[0124] According to the degree of fit between the result t of the random simulation scheduling and the preset scheduling target tar, the action a i executed under the workshop state s k is given a corresponding reward r when entering a new state s i+1 ;

[0125] The mixed flow manufacturing workshop starts from the order placement state s0 to the finished state s n , and forms a piecewise Markov decision chain data <s i , a k , r, s i+1 > in the order of n times of scheduling;

[0126] In the step S4, initializing the plurality of agents includes: setting different agents and different reward functions respectively with the shortest completion time, the shortest average completion time and the highest equipment utilization as the scheduling target tar;

[0127] The reward function is:

[0128]

[0129] wherein R + and R- respectively represent fixed positive and negative reward values, and tar is the scheduling target;

[0130] Each of the agents is respectively provided with a D3QN model;

[0131] The D3QN model comprises two structurally consistent EfficientNet neural networks, which are respectively an online evaluation network Q and a target value network Q- of the D3QN model;

[0132] In the step S5, data is read in batches from the sample data pool and input into the online evaluation network Q to obtain an estimated value, and input into the target value network Q- to obtain a target value, and the mean square error of the estimated value and the target value is taken as a loss function of the D3QN model for supervised training;

[0133] The loss function of the D3QN model is specifically:

[0134]

[0135]

[0136] Wherein, L e (θ e ) is the loss function value of the D3QN model; the subscript e is a training step length, represents the mean square error calculation, (s i , a k , r, s i+1 ) ~ U(DB) represents the training sample data extracted from the sample data pool, Y e represents the target value; Q(s i , a k ; θ e ) represents the estimated value obtained by inputting the current workshop state s i and the action a k into the online evaluation network Q; γ is a discount factor, and the greater the discount factor, the more the network training focuses on the change of the future state; argmaxaQ(s i+1 , a; θ e ) represents the most valuable action a of the next state s i+1 evaluated by the online evaluation network Q;

[0137] Using the stochastic gradient descent strategy, the weight parameters of the online evaluation network Q are reversely updated and trained; every training step length, the parameters of the online evaluation network Q last updated are copied to update the parameters of the target value network Q-; the training is repeated until the loss function value of the D3QN model is less than a preset threshold, and the trained D3QN model is obtained;

[0138] In the step S6, the real-time state of the mixed flow manufacturing workshop is obtained and input into the MPN model, and the real-time MPN model and the to-be-scheduled task under the real-time state are obtained;

[0139] The real-time state is input into the D3QN model trained by each agent, each agent selects an optimal value scheduling action from the to-be-scheduled tasks in the real-time state according to a preset scheduling target, and updates the next real-time state and the corresponding to-be-scheduled task; the optimal value scheduling action is repeatedly selected and executed until all workpieces are produced, and a plurality of workshop scheduling strategies oriented to different scheduling targets are obtained;

[0140] For the same to-be-executed order, a corresponding workshop scheduling strategy is called according to different scheduling targets to schedule production;

[0141] In addition, the step S6 further comprises:

[0142] Each workshop scheduling strategy under different scheduling targets is scored by using a weighted comprehensive scoring method, and the priority of each workshop scheduling strategy under different scheduling targets is obtained;

[0143] According to the priority of each workshop scheduling strategy under different scheduling targets, a workshop scheduling strategy whose priority under different scheduling targets meets a preset condition is selected as an optimal scheduling strategy, and comprehensive scheduling production is performed according to the optimal scheduling strategy.

[0144] In the specific implementation process, first, an MPN model based on Petri net is constructed according to the running relationship between workpieces and machining devices in the mixed flow manufacturing workshop, and the process is as follows:

[0145] Each machining device running in parallel in each process of the mixed flow manufacturing workshop is abstracted into a place, indicating that the workpiece can be transferred to the machining device when being processed in a process; different places are connected through an intermediate transition path, indicating that the workpiece is transferred between different processes, that is, a path from one machining device to the next machining device after being processed by the machining device;

[0146] The expression of the whole MPN model is as follows:

[0147] MPN = <Token, Place.Transition>

[0148] Token = {Token1, Token2, …, Token i}

[0149] Place = {Place1, Place2, …, Place j}

[0150] Transition = {Transition1, Transition2, …, Transition h}

[0151] Where Token is the expression set of the workpiece in the MPN model, Token i is the expression of the i-th workpiece; Place is the expression set of the processing equipment in the MPN model, Place j is the expression of the j-th processing equipment; Transition is the expression set of the transition path in the MPN model, Transition h is the h-th transition path;

[0152] The expression of the i-th workpiece in the MPN model is as follows:

[0153]

[0154] Where Token i represents the i-th workpiece, and n is the number of workpieces; ID1 is the number of the i-th workpiece, satisfying ID1∈N, N is a set of natural numbers; C is the workpiece type, satisfying C∈{Color1, Color2, …, Color c}, Color1~Color c are the 1st~c-th workpiece types, respectively; is the processing time of the i-th workpiece on the j-th processing equipment, and m is the number of processing equipment; P is the number of the processing equipment where the i-th workpiece is currently located, satisfying P∈N; T is the remaining processing time of the i-th workpiece in the current equipment, satisfying T∈R, R is a set of real numbers;

[0155] The expression of the j-th processing equipment in the MPN model is as follows:

[0156]

[0157] Where Place j represents the j-th processing equipment; ID2 is the number of the j-th processing equipment, satisfying ID2∈N; Capacity is the number of workpieces that the processing equipment can accommodate, is the number of workpieces in the j-th processing equipment;

[0158] The expression of the h-th transition path in the MPN model is as follows:

[0159] Transition h =<ID3, inPlace, outPlace>

[0160] Where ID3 is the number of the transition, satisfying ID3∈N, inPlace is the number of the current processing equipment, satisfying inPlace∈N; outPlace is the number of the next processing equipment, satisfying outPlace∈N;

[0161] Subsequently, the job data of the orders to be executed in the hybrid manufacturing workshop is acquired and input into the MPN model, and an initialized MPN model is obtained;

[0162] The number, the number and the type of each workpiece in the job data are extracted, and the machining equipment in which each workpiece is located during processing, the remaining processing time of the workpiece, and all the extracted information are loaded into the MPN model for initialization, and an initialized MPN model is obtained;

[0163] After that, random scheduling simulation is carried out based on the initialized MPN model to obtain simulation data; Markov decision chain data is constructed according to the simulation data and the job data and loaded into a preset sample data pool;

[0164] The whole process from the start to the end of the processing of all workpieces of the order to be executed is simulated using a random scheduling strategy, a batch of simulation data is generated, and a workpiece scheduling time matrix ST, a workpiece position transfer identification matrix MM and a workpiece device delay matrix MT are constructed according to the job data;

[0165] The three-channel workshop state s i is constructed according to the three matrices; i , s i The selected action a k , the action a k The corresponding reward r, and the next workshop state s i+1 Together construct Markov decision chain data <s i , a k , r, s i+1 > and load it into a preset sample data pool;

[0166] The expression of the workshop state s i is as follows:

[0167] s i =<ST n×m , MM n×m , MT n×m >

[0168] Wherein, n x m is the size of the matrix, n is the number of workpieces in the MPN model, and m is the number of machining equipment in the MPN model;

[0169] The action set Job(s i ) that can be triggered by the MPN model under the current workshop state s i is obtained, wherein the action refers to the process of transferring a specific workpiece from a certain process equipment to a next process equipment, and the action a k The expression is as follows:

[0170] a k=<ID1, inPlace, outPlace, Time>

[0171] Where Time is the time when the action is triggered, satisfying Time∈R;

[0172] The workshop state s i The set Job(s) consists of multiple actions that can be triggered by the next workpiece. i ), action set Job(s i The expression is as follows:

[0173] Job(s i )={a1,a2,...,a k}

[0174] The action set Job(s) i ) is used to represent the workshop state s i The workpiece can be transferred in a selectable direction. During actual production transfer, one action will be selected from many actions to be executed. After execution, the workshop status s will be displayed. i Updated to s i+1 Action set Job(s) i The corresponding update is Job(s) i+1 );

[0175] Based on the degree of fit between the result t of the random scheduling simulation and the preset scheduling target tar, the workshop state s is assigned. i Next, execute action a k Entering a new state i+1 The corresponding reward r;

[0176] The mixed-flow manufacturing workshop starts from the order placement state s0 and ends at the completion state s0. n End, forming a Markov decision chain data according to the nth scheduling sequence. i a k ,r,s i+1 >;

[0177] like Figure 2 As shown, the workpiece scheduling time matrix ST represents the time points when different workpieces leave the current processing equipment. Figure 2 Chinese i In the workpiece scheduling time matrix ST under the current state, the matrix element in the second row and fourth column is 79, which means that the time point when workpiece number T2 is completed in processing equipment number P4 is 79 units of time after the start of scheduling; -1 in the matrix has no practical meaning and is only used to fill the matrix;

[0178] The workpiece position transfer identifier matrix MM represents the current processing equipment where the workpiece is located and the processing equipment to which the next process can be transferred; Figure 2 Chinese​i The elements in the second row of the workpiece position transfer identifier matrix under the current state are as follows: element 0 indicates that workpiece T2 is currently in processing equipment P4; element 1 indicates that workpiece T2 can be transferred to processing equipment P5 or P6 in the next process; element -1 indicates that the workpiece is not currently in any equipment or is not in any processing equipment in the next process.

[0179] The workpiece delay matrix MT is a matrix that does not change with state, representing the processing time of the workpiece on different machines; where... Figure 2 Chinese i If the first and last columns of the workpiece position transfer identifier matrix are all 0, it can be assumed that the workpiece is not processed at the initial equipment and the last equipment it arrives at, but is only stored.

[0180] like Figure 2 As shown, state s i After action a k Then enter state s i+1 , where a k The expression <T4,P1,P2,0> indicates that workpiece T4 was transferred from processing equipment P1 to processing equipment P4 at time point 0. Figure 2 medium state s i+1 The change of the first element in the fourth row of the workpiece scheduling time matrix from 0 to -1 and the change of the second element from -1 to 23 indicates that the T4 component leaves the P1 processing equipment and is completed at the P2 processing time point 23 units after the start of scheduling. Figure 2 medium state s i+1 The change of the first element in the fourth row of the workpiece position transfer identifier matrix from 0 to -1, the change of the second element from 1 to 0, the change of the third and fourth elements from 1 to -1, and the change of the fifth and sixth elements from -1 to 1 indicates that the T4 component leaves the P1 processing equipment and is transferred to the P2 processing setting. The original P3 and P4 processing equipment changes from a transferable state to a non-transferable state, and the P5 and P6 processing equipment changes from a non-transferable state to a transferable state.

[0181] Next, multiple agents are initialized, and a D3QN model is built for each agent. Each agent has a different scheduling objective. The scheduling objective includes, but is not limited to, the shortest completion time objective, the shortest average completion time objective, and the highest equipment utilization objective.

[0182] Different agents and different reward functions are set up with scheduling objectives tar based on the shortest completion time, the shortest average completion time, and the highest equipment utilization rate, respectively;

[0183] The reward function is:

[0184]

[0185] wherein R + and R- respectively represent fixed positive and negative reward values, and tar is a scheduling target;

[0186] Each of the agents is respectively provided with a D3QN model;

[0187] The D3QN model comprises two structurally identical EfficientNet neural networks, which are respectively used as an online evaluation network Q and a target value network Q- of the D3QN model;

[0188] A plurality of Markov decision chain data are randomly extracted from a sample data pool, and the D3QN model of each agent is trained, and a trained D3QN model of each agent is obtained, and the training specifically comprises:

[0189] Data are batch-read from the sample data pool and input into the online evaluation network Q to obtain an estimated value, and input into the target value network Q- to obtain a target value, and a mean square error of the estimated value and the target value is used as a loss function of the D3QN model for supervised training;

[0190] The loss function of the D3QN model is specifically:

[0191]

[0192]

[0193] wherein L e (θ e ) is a loss function value of the D3QN model; subscript e is a training step length, represents a mean square error calculation, (s i , a k , r, s i+1 ) ~ U(DB) represents training sample data extracted from the sample data pool, Y e represents a target value; Q(s i , a k ; θ e ) represents an estimated value obtained by inputting a current workshop state s i and an action a k into the online evaluation network Q; γ is a discount factor, and the greater the discount factor, the more the network training focuses on the change of the future state; argmax a Q(s i+1 , a; θ e ) represents an action a with the most valuable state s i+1 evaluated by the online evaluation network Q;

[0194] The weight parameters of the online evaluation network Q are reversely updated and trained using a random gradient descent strategy; every certain step of training, the parameters of the online evaluation network Q last updated are copied to update the parameters of the target value network Q-; the training is repeated until the loss function value of the D3QN model is less than a preset threshold, and the trained D3QN model is obtained;

[0195] As shown in Figure 3 The real-time state of the mixed flow manufacturing workshop is finally acquired and input into the MPN model to acquire the real-time MPN model and the to-be-scheduled task under the real-time state;

[0196] The real-time state is input into the D3QN model trained by each agent, and each agent selects an optimal value scheduling action from the to-be-scheduled task under the real-time state according to a preset scheduling target, and updates the next real-time state and the corresponding to-be-scheduled task; the optimal value scheduling action is repeatedly selected and executed until the production of all workpieces is completed, and a plurality of workshop scheduling strategies oriented to different scheduling targets are obtained;

[0197] For the same to-be-executed order, a corresponding workshop scheduling strategy is directly called for scheduling production according to different scheduling targets;

[0198] In addition, a weighted comprehensive scoring method can also be used to score each workshop scheduling strategy under different scheduling targets to obtain the priority of each workshop scheduling strategy under different scheduling targets;

[0199] According to the priority of each workshop scheduling strategy under different scheduling targets, a workshop scheduling strategy whose priority under different scheduling targets all meets a preset condition is selected as an optimal scheduling strategy, and comprehensive scheduling production is performed according to the optimal scheduling strategy, so that a scheduling strategy considering various targets is obtained after balancing the priorities of different strategies;

[0200] The method adopts the EfficientNet neural network as the basic network of the D3QN model, utilizes the high-efficiency and lightweight characteristics of the EfficientNet neural network, significantly reduces the calculation amount and parameter amount while maintaining high accuracy, thereby improving the performance and training speed of the D3QN algorithm in processing large-scale scheduling problems; meanwhile, the method introduces the mechanism of multiple agents, so that the scheduling algorithm can obtain mixed flow manufacturing workshop scheduling strategies oriented to different targets in one scheduling calculation process, so that production managers can consider and decide in multiple dimensions, thereby better optimizing production scheduling and resource utilization, and improving overall production efficiency and economic benefits.

[0201] The same or similar reference signs correspond to the same or similar components;

[0202] The terms describing the positional relationship in the drawings are used only for illustrative purposes and should not be construed as limiting the present patent;

[0203] Obviously, the above-mentioned embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the implementation modes of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, it is not necessary and also impossible to exhaust all the implementation modes. Any modification, equivalent replacement and improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A multi-agent D3QN-based real-time scheduling method for hybrid flow manufacturing workshops, characterized in that, The method comprises the following steps: S1: constructing a MPN model based on Petri net according to the running relationship between workpieces and processing equipment in the mixed flow manufacturing workshop, comprising: Abstracting each set of processing equipment running in parallel in the mixed flow manufacturing workshop into a library, indicating that the workpiece can be transferred to the processing equipment when being processed in a certain process; different libraries are connected through an intermediate transition path, indicating that the workpiece is transferred between different processes, i.e. a path from one processing equipment to the next processing equipment after being processed; The expression of the entire MPN model is as follows: wherein, is a set of expressions for workpieces in the MPN model, is an expression for the i-th workpiece; is a set of expressions for processing equipment in the MPN model, is an expression for the j-th processing equipment; is a set of expressions for transition paths in the MPN model, is the h-th transition path; The expression of the i-th workpiece in the MPN model is as follows: wherein, represents the ith workpiece, is the number of workpieces; is the number of the ith workpiece, satisfying , is a natural number set; is the workpiece type, satisfying , ~ are the 1st to cth workpiece types, respectively; is the processing time of the ith workpiece on the jth processing equipment, is the number of processing equipment; is the processing equipment number where the ith workpiece is currently located, satisfying ; is the remaining processing time of the ith workpiece in the current equipment, satisfying , is a real number set; The expression of the j-th processing equipment in the MPN model is as follows: wherein, represents the jth processing equipment; is the number of the jth processing equipment, satisfying ; is the number of workpieces that the processing equipment can accommodate, is the workpiece present in the jth processing equipment; The expression of the h-th transition path in the MPN model is as follows: wherein, is the number of the transition path, satisfying , is the number of the current processing device, satisfying ; is the number of the next processing device, satisfying ; S2: obtaining the job data of the to-be-executed order of the mixed flow manufacturing workshop and inputting the MPN model to obtain an initialized MPN model; the job data comprises: the number and type of workpieces of the to-be-executed order, and the state information of the processing equipment of each workpiece in the processing process; S3: performing random scheduling simulation based on the initialized MPN model to obtain simulation data; constructing Markov decision chain data according to the simulation data and the job data and loading the data into a preset sample data pool; S4: initializing a plurality of agents and constructing a D3QN model for each agent; the scheduling target of each agent is different; The D3QN model includes two structurally consistent EfficientNet neural networks respectively as an online evaluation network and a target value network of the D3QN model and a target value network ; S5: randomly extracting a plurality of Markov decision chain data from the sample data pool and training the D3QN model of each agent to obtain the trained D3QN model of each agent; S6: obtaining the real-time state of the mixed flow manufacturing workshop and inputting the MPN model to obtain a real-time MPN model; after calling the trained D3QN model of each agent to perform scheduling calculation on the real-time MPN model, a plurality of workshop scheduling strategies oriented to different scheduling targets are obtained, and production is scheduled according to the workshop scheduling strategies. 2.The multi-agent D3QN based real-time scheduling method for hybrid flow shop according to claim 1, wherein, In the step S2, the number, number and type of each workpiece are extracted from the job data, the processing equipment of each workpiece in the processing process, the remaining processing time of the workpiece, and all the extracted information is loaded into the MPN model for initialization to obtain an initialized MPN model. 3.The multi-agent D3QN based real-time scheduling method for hybrid flow shop according to claim 2, wherein, In the step S3, the whole process of starting and ending of the to-be-executed order is simulated using a random scheduling strategy to generate a batch of simulation data, and the job data is combined to construct a workpiece scheduling time matrix ST, a workpiece position transfer identification matrix MM and a workpiece in device delay matrix MT; The three-channel plant state is constructed according to the three matrices obtained The plant state 、 The selected action The action The corresponding reward , and the next plant state Together constitute the Markov decision chain data And load the preset sample data pool.

4. The multi-agent D3QN-based real-time scheduling method for hybrid flow manufacturing shop according to claim 3, characterized in that, The plant status The expression is as follows: wherein, is the size of the matrix, is the number of workpieces in the MPN model, is the number of machining devices in the MPN model; Acquiring the MPN model in the current workshop state The set of actions that can be triggered next Wherein the action refers to the process of transferring a specific workpiece from a process equipment to a next process equipment, the action The expression is as follows: wherein, is the time at which the action is triggered, satisfies ; from the cell state a set of actions that can be triggered by the workpiece , the set of actions The expression is as follows: the set of actions for indicating the state of the workshop the direction of the selectable transfer of the lower workpiece, the actual production transfer will select an action from the set of actions, and the state of the workshop after the execution of the action is updated to , the set of actions is also updated to ; According to the results of the random scheduling simulation The degree of fit between the preset scheduling target The state of the workshop is given The action is executed Enter a new state The corresponding reward ; The mixed flow manufacturing shop goes from an order placement state start to finish state end, by sequentially dispatching production orders to form a piecewise Markov decision chain data .

5. The multi-agent D3QN-based real-time scheduling method for hybrid flow manufacturing shop according to claim 4, wherein, In the step S4, initializing the plurality of agents comprises: setting different agents and different reward functions respectively with the shortest completion time, the shortest average completion time and the highest equipment utilization rate as the scheduling target ​ The reward function is: wherein, and represent fixed positive and negative reward values, respectively, is a scheduling objective; Each agent is provided with a D3QN model; The D3QN model includes two structurally consistent EfficientNet neural networks respectively as an online evaluation network and a target value network of the D3QN model . .

6. The multi-agent D3QN-based real-time scheduling method for hybrid flow manufacturing shop according to claim 5, wherein, In the step S5, data is read in batches from the sample data pool and input into the online evaluation network An estimated value is obtained, and a target value network is input A target value is obtained, and a mean square error between the estimated value and the target value is used as a loss function of the D3QN model for supervised training The loss function of the D3QN model is specifically: wherein, is the loss function value of the D3QN model; subscript is the training step length, represents the mean square error calculation, represents the training sample data extracted from the sample data pool, represents the target value; represents inputting the current workshop state and action to the online evaluation network to obtain the estimated value; is a discount factor, the larger the discount factor, the more the network training focuses on the change of the future state; represents that the online evaluation network evaluates the most valuable action for the next state ; The network is evaluated online using a stochastic gradient descent strategy. The weight parameters are updated in reverse during training; after each training step, the online evaluation network is copied. The parameters of the last update for the target value network The parameters are updated; the training is repeated until the loss function value of the D3QN model is less than the preset threshold, and then the trained D3QN model is obtained.

7. The multi-agent D3QN-based real-time scheduling method for hybrid flow manufacturing shop according to claim 6, wherein, In the step S6, the real-time state of the mixed flow manufacturing workshop is obtained and input into the MPN model to obtain a real-time MPN model and a to-be-scheduled task under the real-time state. The real-time state is input into the D3QN model trained by each agent, each agent selects an optimal value scheduling action from the to-be-scheduled tasks in the real-time state according to a preset scheduling target, and updates the next real-time state and the corresponding to-be-scheduled task; the optimal value scheduling action is repeatedly selected and executed until the production of all workpieces is completed, and a plurality of workshop scheduling strategies oriented to different scheduling targets are obtained; For the same to-be-executed order, a corresponding workshop scheduling strategy is called according to different scheduling targets to schedule production.

8. The multi-agent D3QN-based real-time scheduling method for hybrid flow manufacturing workshops according to any one of claims 1-7, characterized in that, The step S6 further comprises: Each workshop scheduling strategy is scored under different scheduling targets by using a weighted comprehensive scoring method, and the priority of each workshop scheduling strategy under different scheduling targets is obtained; According to the priority of each workshop scheduling strategy under different scheduling targets, a workshop scheduling strategy whose priority under different scheduling targets all satisfies a preset condition is selected as an optimal scheduling strategy, and comprehensive scheduling production is performed according to the optimal scheduling strategy.

Images