Dynamic intelligent scheduling method for multi-objective flexible workshop in industrial green microgrid

Abstract

The present invention provides a method for dynamic intelligent scheduling of a multi-objective flexible workshop in an industrial green microgrid, which is implemented by constructing a theoretical model for flexible workshop scheduling that includes a public power grid, a new energy power supply system, and a factory. By using deep reinforcement learning, especially an improved DQN neural network, the agent learning model can dynamically adjust the scheduling strategy according to real-time data, optimize the workpiece processing time, reduce production costs, and minimize energy consumption. The method guides the agent to perform multi-objective optimization through an instant compound reward and a global compound reward mechanism to achieve automation and intelligence in production scheduling. Compared with the prior art, the present invention can effectively reduce production delays, reduce energy consumption and costs, and improve the flexibility and efficiency of scheduling, which is of great significance to promoting industrial green production and improving the intelligence level of production scheduling.

Description

Technical Field

[0001] The present invention relates to the technical field of flexible job shop scheduling, and in particular to a dynamic intelligent scheduling method for a multi-objective flexible job shop in an industrial green microgrid. Background Art

[0002] With the advancement of renewable energy technology, more and more manufacturing companies are beginning to adopt green energy such as solar energy, wind energy and hydropower in order to achieve the goal of energy conservation and emission reduction. In addition, the development of microgrid technology provides technical support for the widespread application of these green energy sources in the manufacturing industry, enabling factories to build small power systems including distributed power sources, energy storage equipment and power loads, thereby more effectively utilizing the surrounding renewable energy and peak and valley power of the public power grid.

[0003] In the field of industrial production, especially in the manufacturing industry, the workshop scheduling problem has always been the key to improving production efficiency and reducing costs. With the increasing global awareness of sustainable development and environmental protection, the manufacturing industry is facing huge challenges of transformation and upgrading. In this context, how to achieve intelligent and green workshop scheduling has become a hot topic of research.

[0004] At present, the research on green and low-carbon manufacturing workshop scheduling mainly relies on some swarm intelligence optimization technologies for static planning, such as genetic algorithms, ant colony algorithms, etc. These studies are usually based on the assumption of static environment, such as the cost of power supply and output power are relatively stable within a certain time range, and energy saving is achieved by adjusting the processing strategy or the working state of the machine. However, these methods do not fully consider the comprehensive management and control of production equipment and unstable energy supply in the microgrid environment, nor have they formed a dynamic strategy to optimize production and energy costs, which makes it difficult for traditional optimization measures to effectively reduce the power cost of the factory and adapt to the dynamic changes of power supply.

[0005] In the prior art, the invention patent application with publication number CN114186749A proposed a flexible workshop scheduling method based on reinforcement learning and genetic algorithm. This method optimizes the scheduling strategy of workshop operations by combining the global search capability of genetic algorithm and the adaptive characteristics of reinforcement learning. However, this method mainly focuses on the optimization of search algorithm and does not fully consider energy consumption and environmental impact.

[0006] The invention patent application with publication number CN115271293A proposes a multi-agent flexible job shop autonomous scheduling method, which improves the speed of solving scheduling problems by building multiple scheduling agents for parallel computing. Although this method improves scheduling efficiency, it still has certain limitations when dealing with multi-objective problems related to energy supply and cost optimization.

[0007] The invention patent application with publication number CN116644902A discloses a method for multi-objective dynamic flexible job shop scheduling based on energy consumption based on deep reinforcement learning. This method takes energy consumption optimization into account, but in practical applications, it may be necessary to further enhance the adaptability to the instability of renewable energy.

[0008] On the whole, although the existing green and low-carbon manufacturing workshop scheduling technology has promoted energy conservation, emission reduction and cost control to a certain extent, further innovation and improvement are still needed to achieve more efficient, flexible and adaptable dynamic intelligent scheduling, especially how to link production operations and energy supply in microgrid environments in real time, analyze their relationships, set global optimization goals for intelligent entities, and find the best action plan to achieve the long-term goals of manufacturing enterprises in energy conservation, emission reduction, cost reduction and efficiency improvement. Summary of the invention

[0009] The purpose of the present invention is to provide a dynamic intelligent scheduling method for a multi-objective flexible workshop in an industrial green microgrid. By adopting deep reinforcement learning technology, it can link production operations and energy supply in a microgrid environment in real time, analyze the relationship between them, set global optimization goals for intelligent agents, and find the best action plan to achieve the long-term goals of manufacturing enterprises in energy conservation, emission reduction, cost reduction and efficiency improvement.

[0010] Based on the first main aspect of the present invention, a method for dynamic intelligent scheduling of a multi-objective flexible workshop in an industrial green microgrid is provided, comprising:

[0011] S1, construct a theoretical model of flexible workshop operation scheduling for green and low-carbon factories in a computer system. The theoretical model of flexible workshop operation scheduling for green and low-carbon factories consists of three major units: a public power grid, a new energy power supply system, and a factory. A workshop model is set up in the factory, and the workshop model includes an intelligent agent learning model. The workshop model provides environmental information for the intelligent agent, and the intelligent agent learning model is responsible for making scheduling decisions for the workshop based on this information;

[0012] S2, constructing an objective function for deep reinforcement learning of the agent learning model, and integrating different scheduling rules into the agent learning model; the optimization objectives of the scheduling rules include: optimizing the processing time of the workpiece to reduce the delay, optimizing the time-of-use electricity price of the public grid to reduce the production cost and minimizing the maximum completion time based on the insufficient energy supply of the microgrid;

[0013] The objective function is constructed as follows: ;

[0014] ;

[0015] in,

[0016] : Indicates minimizing the total delay time; : represents the last process of each workpiece i The completion time of job i is the actual completion time of job i. : is the expected delivery time of workpiece i ;

[0017] like , then the actual completion time is greater than the expected delivery time, and the processing of the workpiece is delayed;

[0018] : represents minimizing energy cost, : represents energy cost;

[0019] : It means to minimize the maximum value of the completion time of all workpieces, that is, the maximum completion time. This is also one of the commonly used criteria in workshop scheduling. : Indicates the maximum completion time of all workpieces at the end of an iteration;

[0020] S3, setting state characteristic variables of the workshop model, setting state characteristic variable definitions abstracted from the workshop environment, and adding different scheduling rules to the agent learning model;

[0021] S4, constructing an improved deep reinforcement learning DQN neural network to train the agent learning model to perform dynamic intelligent scheduling on the flexible job shop, constructing an immediate compound reward and a global compound reward mechanism in the deep reinforcement learning DQN neural network, and designing a compound scheduling rule strategy so that the agent can effectively respond to various constraints and guide the agent to perform multi-objective optimization during the scheduling process;

[0022] S5, evaluating the training effect of the intelligent agent learning model; integrating the evaluated intelligent agent learning model into the control system or embedded device in the factory to achieve autonomous scheduling.

[0023] As a further preferred solution, the learning process of the agent in the agent learning model includes the following cyclic iterative process: at each time step, the agent obtains the current state ; Based on the state, select actions through greedy strategy ; Execute action , the environment will feedback the new state and rewards ; Store the experience in the experience replay buffer; Randomly sample N steps of experience from the experience replay buffer to update the online network; Soft update the target network; Calculate the loss function for back propagation and update the weights of the online network; Repeat the process until the training converges.

[0024] As a further preferred solution, in step S3, setting the state characteristic variables of the workshop model includes:

[0025] : represents the total number of machines, all m here have this meaning;

[0026] : represents the total number of workpieces, all n here have this meaning;

[0027] : represents the process number of workpiece i;

[0028] : represents the total number of all artifacts;

[0029] : represents the first A process;

[0030] :Represents these different artifacts Different processes On different machines Processing time on

[0031] :For machine The completion time of the last process at the rescheduling point t;

[0032] : It is the process index currently completed by the workpiece Ji;

[0033] : Represents a machine The utilization rate at time t is specifically expressed as:

[0034] ;

[0035] In the present invention, the processing time of each workpiece on different machines is different for different processes, so Represents different artifacts Different processes On different machines Processing time on

[0036] Represents at the moment ,machine The total time actually used to process the workpiece accounts for the total available time of the machine (i.e., the completion time of the last operation);

[0037] in, , is a binary decision variable used to describe the allocation of workpiece processes on machines, then It means that the jth process of workpiece i is assigned to machine k. Indicates the machine number assigned to the j-th process of workpiece i;

[0038] : represents the completion rate of job Ji at time t: .

[0039] As a further preferred solution, in step S3, the setting of the state characteristic variable definition abstracted from the workshop environment includes:

[0040] Average machine utilization: ;

[0041] Standard deviation of machine utilization: ;

[0042] Average process completion rate: ;

[0043] Average completion rate of workpieces: ;

[0044] Standard deviation of workpiece completion rate: ;

[0045] Estimated delay rate ; The calculation steps are:

[0046] Step 1: Initialize the number of delayed steps To 0, initialize the number of remaining processes is 0;

[0047] Step 2: Calculate the current average completion time : The completion time of all machines at the current moment Divide by the number of machines ;

[0048] Step 3: Traverse all artifacts;

[0049] Step 4: Check whether the workpiece has been processed: the number of processes that have been completed for the workpiece at the current moment If it is less than the total number of processes of the workpiece, it is considered as unfinished. The number of unfinished processes is recorded and accumulated to middle;

[0050] Step 5: Initialize the remaining time 0, used to record the estimated processing time of the remaining steps of the workpiece;

[0051] Step 6: Check the remaining processes of all workpieces and calculate the average processing time of the remaining processes , and its specific calculation method is: for each remaining process of the workpiece, calculate the average processing time on all available machines;

[0052] Step 7: Cumulative processing time of the remaining steps of the workpiece ;

[0053] Step 8: Determine whether a delay occurs: If + If it is greater than the deadline of the workpiece, it is considered that the workpiece may be delayed;

[0054] Step 9: If it is determined that the workpiece is expected to be delayed, the number of remaining processes for the workpiece is accumulated to middle;

[0055] Step 10: After all workpieces have been processed as above, update the statistics of remaining processes and delayed processes;

[0056] Step 11: Calculate the estimated latency rate ;

[0057] Actual delay rate ; The calculation steps are:

[0058] Step 1: Initialize the number of delayed steps 0, remaining number of processes is 0;

[0059] Step 2: Traverse all artifacts;

[0060] Step 3: Check whether each workpiece is completed, calculate the number of remaining processes, and accumulate the number of remaining processes;

[0061] Step 4: Check whether the current completion time of the workpiece is later than the estimated delivery time;

[0062] Step 5: If it is later than the expected delivery time, the workpiece is judged to be delayed, and the remaining number of processes of the workpiece is regarded as the number of delayed processes and added to the number of delayed processes;

[0063] Step 6: If it is not later than the expected delivery time, it is determined that the workpiece is not delayed, and the accumulation of delayed process number is skipped;

[0064] Step 7: After all workpieces have been calculated for the accumulation of the remaining number of processes and the accumulation of the delayed number of processes;

[0065] Step 8: Calculate the actual delay rate: .

[0066] As a further preferred solution, in step S4, constructing an improved deep reinforcement learning DQN neural network includes:

[0067] S41, using the ε-greedy strategy for action selection to balance exploration and exploitation;

[0068] S42, double DQN technology is applied to reduce the overestimation of Q value;

[0069] S43, introduces the dueling DQN network structure to evaluate state value and action advantage;

[0070] S44, implement the N step learning method, accumulate N step rewards to update the Q value;

[0071] S45, construct a loss function, and use the mean square error loss function to define the error between the evaluation network and the target network;

[0072] S46, designs immediate reward functions and global reward functions to guide the agent to perform multi-objective optimization, including reducing costs and delays.

[0073] In addition, scheduling rule strategies can be designed to enable the agent to effectively cope with various constraints and optimization objectives during the scheduling process.

[0074] Furthermore, in step S42, the double DQN technique is applied to reduce the overestimation of the Q value, and the target Q value calculation formula is:

[0075] ;

[0076] in, : represents the number of time steps, where ;

[0077] : represents the return of DQN in the st time step;

[0078] : represents the return value of the st+1th time step;

[0079] : is the discount factor;

[0080] : represents the action taken by the agent;

[0081] : represents the parameters in the online network in DDQN;

[0082] : represents the parameters in the target network in DDQN;

[0083] :Indicates that based on Parameters of the online network obtained The action that maximizes the Q value function in the state .

[0084] Furthermore, in step S43, the dueling DQN network structure is introduced to evaluate the state value and action advantage, including:

[0085] Add dueling net to DDQN, modify the neural network part of double DQN, introduce state value; calculate the benefit and action advantage in a certain state; evaluate the possible advantage of taking action a compared to other actions in state s; to:

[0086] ;

[0087] in, .

[0088] in, : Represents the network parameters in the dueling network, corresponding to the parameters of the action advantage function (AdvantageFunction, A): controls how the neural network estimates the advantage of an action over other actions;

[0089] : Represents the network parameters in the dueling network, corresponding to the parameters of the state value function (Value Function, V);

[0090] These two parameters are mainly used to separate action advantages from state values, thereby improving the estimation effect of Q-learning;

[0091] :refers to the given state When evaluating the value of the state itself (the potential benefit of the state), it does not depend on the action ;

[0092] :Evaluation is in state Next, execute the action Advantages over other possible actions;

[0093] and : represents any action in the action set;

[0094] : Indicates that the state The sum of the action advantages of all possible actions in the case;

[0095] : represents the number of time steps, where .

[0096] Furthermore, based on the above, we add the N-step learning method so that the Q value is updated not in a single step, but in an accumulated N-step reward, that is, ; So the Q value at this time becomes:

[0097] .

[0098] in: Here is the cumulative discounted reward, which means the cumulative discounted reward between time steps st to st+N-1. Each reward Multiply by the discount factor to the corresponding power of to reduce the weight of future rewards.

[0099] is each time step The discount factor ensures that the reward weight is smaller the farther away from the current moment.

[0100] The rest remains the same as above.

[0101] in: Here is the cumulative discounted reward, which means the cumulative discounted reward between time steps st to st+N-1. Each reward Multiply by the discount factor to the corresponding power of to reduce the weight of future rewards.

[0102] is each time step The discount factor ensures that the reward weight is smaller the farther away from the current moment.

[0103] The rest remains the same as above.

[0104] Furthermore, the loss function constructed in step S45 is: , updated through back-propagation and the optimizer.

[0105] Among them, batchsize: is the batch size. In DQN, we usually update the model in small batches instead of the entire dataset;

[0106] : is the online network Q value;

[0107] : is the target network Q value;

[0108] The loss function here is expressed as the average error, which is minimized , to optimize the error.

[0109] Based on the second main aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed, the above-mentioned industrial green microgrid multi-objective flexible job shop dynamic intelligent scheduling method is implemented.

[0110] According to a third main aspect of the present invention, there is provided an electronic device, comprising:

[0111] at least one processor;

[0112] a memory communicatively coupled to the at least one processor;

[0113] The memory stores a computer program, which, when executed by the at least one processor, enables the at least one processor to implement the aforementioned industrial green microgrid multi-objective flexible job shop dynamic intelligent scheduling method.

[0114] Compared with the prior art, the present invention has the following beneficial effects:

[0115] The present invention achieves a major innovation in traditional scheduling methods by constructing a multi-objective flexible workshop dynamic intelligent scheduling method for industrial green microgrid systems. The method applies deep reinforcement learning technology to workshop scheduling problems and achieves real-time optimization of production scheduling through iterative training of intelligent agent learning models. Compared with traditional static planning methods, the present invention can dynamically respond to changes in microgrid energy supply, real-time production task requirements, and effectively adapt to time-of-use electricity price policies, significantly improving scheduling flexibility and response speed.

[0116] Based on the principle of autonomous learning of intelligent agents in deep reinforcement learning algorithms, this invention explores and analyzes the internal mechanism of energy supply and demand, and develops a composite reward function to guide intelligent agents to select the optimal strategy. Compared with other traditional scheduling strategies, this invention can effectively reduce the consumption of electricity costs of manufacturing enterprises, reasonably reduce the maximum completion time, reduce the delay time of workpieces, etc., and combine with the real-time dynamic production status of enterprises to achieve the purpose of reducing costs and green and low-carbon production.

[0117] By constructing a composite reward function and introducing multi-level scheduling rules, the present invention achieves an effective balance between multiple objectives, including reducing production costs, reducing maximum completion time, and reducing workpiece delay time. This approach not only optimizes production efficiency and energy consumption, but also improves the greenness and sustainability of the production process. In addition, by integrating the agent learning model into the control system or embedded device, the present invention supports highly automated and intelligent production scheduling, reduces human intervention, and improves the system's autonomous decision-making ability.

[0118] The implementation effect of the present invention shows that the method can significantly reduce energy consumption and improve energy efficiency while maintaining production efficiency. This optimization not only helps manufacturing companies achieve the goals of cost control and energy conservation and emission reduction, but also meets the current global pursuit of green manufacturing and sustainable development. Since the present invention has a high degree of generalization and adaptability, its application is not limited to specific workshops or industries, but can also be extended to various production environments and industrial fields, and has broad application prospects and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 A composition structure of a theoretical model of flexible job shop scheduling for green and low-carbon factories in one embodiment of the present invention is shown;

[0120] Figure 2 A schematic diagram of a multi-objective flexible workshop dynamic intelligent scheduling strategy for an industrial green microgrid system according to an embodiment of the present invention is shown;

[0121] Figure 3 A flowchart of a multi-objective flexible workshop dynamic intelligent scheduling strategy framework for an industrial green microgrid system in one embodiment of the present invention is shown;

[0122] Figure 4 A flow chart of a dispatching rule algorithm for reducing electricity costs based on a time-of-use electricity price optimization target in one embodiment of the present invention is shown;

[0123] Figure 5 A flowchart of a scheduling rule algorithm for optimizing the maximum completion time based on an optimization target in one embodiment of the present invention is shown;

[0124] Figure 6 A flow chart of a dispatching rule algorithm based on reducing delay time and saving costs as optimization goals in one embodiment of the present invention is shown. DETAILED DESCRIPTION

[0125] The preferred embodiments of the present invention will be described in detail below so that the purpose, features and advantages of the present invention can be more clearly understood. It should be understood that the following embodiments are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0126] In the following description, certain specific details are set forth for the purpose of illustrating the various disclosed embodiments to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0127] References throughout the specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0128] The main purpose of this invention is to analyze the impact of factors such as job allocation, processing time of production machines, and completion time of workpieces on energy consumption by combining classic cases of workshop scheduling in green and low-carbon factories. Integrate the real case of actual microgrid energy supply, according to the relationship between workshop operations and microgrid energy supply, and link with the external power grid time-of-use electricity price policy, comprehensively consider the energy cost problem in the scheduling system, study the energy supply characteristics of the microgrid and the reasons for factory power consumption, consider the impact of the instability of renewable energy (wind energy, solar energy, etc.) on production tasks, improve the strategy rules of workshop operation scheduling at the energy consumption end, and form a scheduling strategy based on multi-objectives of green microgrids to make its optimization target effect more obvious, and optimize and improve the deep reinforcement learning method, so that the action decision-making ability and learning ability of the intelligent body are improved.

[0129] In addition, the present invention designs two new composite reward functions, which are closely related to the scheduling optimization objectives and effectively guide the intelligent agent to identify actions, which effectively improves the real-time performance of the solution and realizes dynamic intelligent scheduling of workshops for green and low-carbon factories.

[0130] The present invention provides a multi-objective flexible workshop dynamic intelligent scheduling strategy for an industrial green microgrid system, which is implemented through the following technical solutions:

[0131] S1. A theoretical model for flexible workshop operation scheduling for green and low-carbon factories is constructed in a computer system. The theoretical model for flexible workshop operation scheduling for green and low-carbon factories is composed of three major units: a public power grid, a new energy power supply system, and a factory. In the theoretical model for flexible workshop operation scheduling for green and low-carbon factories, the workshop model provides environmental information for the intelligent agent, and the intelligent agent learning model is responsible for making scheduling decisions for the workshop based on this information.

[0132] like Figure 1 As shown, the flexible workshop operation scheduling theoretical model for green and low-carbon factories constructed by the present invention consists of three major units: public power grid, new energy power supply system and factory. In the flexible workshop operation scheduling theoretical model for green and low-carbon factories, the workshop model provides environmental information for the intelligent agent, and the intelligent agent learning model is responsible for making scheduling decisions for the workshop based on this information. The details are as follows:

[0133] The theoretical model of flexible workshop scheduling for green and low-carbon factories includes the new energy of microgrid as the energy supply unit, the public power grid as the backup energy support, and the load of the production machine in the flexible workshop is the core load source of the whole system. The system contains multiple machines of different types, and the production load is volatile and flexible. According to the arrival time, delivery date and production process of the workpiece, the scheduling strategy will be dynamically adjusted to optimize the energy efficiency. The model mainly covers the three major elements of green energy supply source, public power grid and production equipment load of flexible workshop, and establishes a coordinated scheduling mechanism on this basis, aiming to reduce costs, reduce the maximum completion time and reduce the delay rate.

[0134] Shop floor model: In deep reinforcement learning to solve shop floor scheduling problems, especially DQN to solve the flexible job shop scheduling problem (FJSP), we often need to map the flexible job shop problem into a shop floor model that can be accepted by the agent in DQN. In layman's terms, it is to apply the concept of the flexible job shop scheduling problem for low-carbon factories to the agent learning model. In this shop floor model, the variable information required for the flexible shop floor operation for green and low-carbon factories is integrated through the following elements:

[0135] (1) Status: In this study, we use some information that can reflect the status of the workshop as the status in DQN. These statuses can effectively reflect the situation of the workshop and can be accepted and processed by the neural network in DQN. In the present invention, its status can be expressed as the average utilization rate of the machine, the standard deviation of the machine utilization rate, the average completion rate of the process, the average completion rate of the workpiece, the standard deviation of the completion rate of the workpiece, the expected delay rate and the actual delay rate. Setting the state characteristic variables of the workshop model is to set the status in this step.

[0136] (2) Action: Here, as described above, different scheduling rules are used as actions.

[0137] (3) Reward: In deep reinforcement learning, reward refers to the ability to guide the agent to choose actions (scheduling rules). It evaluates the pros and cons of the selected scheduling rules based on the different states (information about the workshop state) at different time points and the optimization goal, thereby guiding the selection of the next action (scheduling rule).

[0138] (4) Environment: includes dynamic changes in the workshop, such as the arrival of workpieces, actual power required by the workshop, power supplied by new energy sources, etc. In the present invention, information such as power changes and public grid charges is included.

[0139] In the present invention, a flexible workshop operation scheduling theoretical model for green and low-carbon factories is constructed according to the following method:

[0140] The flexible job shop scheduling problem is described as Workpieces and Machine , each workpiece have Process , and each process has a set of processing machines. The decision point of scheduling is the time when the next process of each workpiece will be scheduled. The purpose of scheduling is to select the most suitable machine for each process, determine the best processing sequence and start time of each workpiece process on each machine, so that the optimization index of the entire system can be optimized.

[0141] The following constraints are met during the processing: the same machine can only process one part at the same time; the processing of the workpiece is not allowed to be interrupted; there is no sequence constraint between the processes of different workpieces; all machines are available at time 0; the processing time of workpieces on different machines is different; energy limitation constraint: in the scenario of green and low-carbon factories, the energy supply is volatile, and it is reasonable to reduce the scheduling of too many flexible processes with high energy consumption to avoid production interruptions caused by energy shortages and other consequences that affect production; dynamic adjustment constraint on the priority of expedited workpieces: in the case of insufficient energy supply, according to special circumstances, urgent workpieces are called first to meet the scheduling optimization goals of minimizing the maximum completion time, minimizing production costs, and minimizing the average delay time of workpieces.

[0142] In the present invention, the agent learning model is an algorithm model for flexible job shop scheduling in the flexible job shop scheduling theoretical model for green and low-carbon factories. The agent learning model provides an effective decision framework for the flexible job shop scheduling problem by combining state space, action space and reward function.

[0143] The specific learning process of the agent is an iterative process, which mainly includes the following steps:

[0144] (1) At each time step, the agent obtains the current state ;

[0145] (2) Based on the state, select actions through greedy strategy ;

[0146] (3) Execute action , the environment will feedback the new state and rewards ;

[0147] (4) Storing the experience in the experience playback buffer;

[0148] (5) Randomly sample N steps of experience from the experience revisit buffer to update the online network;

[0149] (6) Soft update target network;

[0150] (7) Calculate the loss function for back propagation and update the weights of the online network;

[0151] (8) Repeat this process until the training converges.

[0152] S2, constructing an objective function for deep reinforcement learning of the agent learning model, and adding different scheduling rules to the agent learning model; the optimization objectives of the scheduling rules include: optimizing the processing time of the workpiece to reduce the delay, optimizing the production cost based on the insufficient energy supply of the microgrid, and minimizing the production cost;

[0153] The objective function is constructed as follows: ;

[0154] ;

[0155] in, : Indicates minimizing the total delay time. : represents the last process of each workpiece i The completion time of job i is the actual completion time of job i. is the expected delivery time of workpiece i, ;

[0156] like , then the actual completion time is greater than the expected delivery time, and the processing of the workpiece is delayed;

[0157] : represents minimizing energy cost, : represents energy cost;

[0158] : It means to minimize the maximum value of the completion time of all workpieces, that is, the maximum completion time. This is also one of the commonly used criteria in workshop scheduling. : Indicates the maximum completion time of all workpieces at the end of an iteration.

[0159] The present invention is realized based on the linkage between the time-of-use electricity price policy of the external power grid and the energy supply of the microgrid. The electricity cost is generated from the external power grid. The time-of-use electricity price can be expressed as the following piecewise function. The specific value can be changed according to the actual application scenario:

[0160] ;

[0161] in, Represents the time-of-use electricity price charging function at time t;

[0162] Represents the time period [ ] electricity price;

[0163] Represents the time period electricity price;

[0164] Represents the time period electricity price.

[0165] The present invention also takes into account the unstable energy supply problem output by the microgrid. The power generation of renewable energy in the microgrid changes over time. The dynamic output power can be expressed as the following piecewise function. The specific value can be changed according to the actual application scenario:

[0166] ;

[0167] in, : represents the time period The total power function of wind power plus solar power plus other renewable energy sources.

[0168] : represents the time period The total power function of wind power plus solar power plus other renewable energy sources.

[0169] : represents the time period The total power function of wind power plus solar power plus other renewable energy sources.

[0170] in represents the wind power generation power, represents the solar power generation power, and Indicates the power generated by other types of energy.

[0171] The independent variable t represents time, and the domain is derived from the power generation of renewable energy (wind energy, solar energy, etc.) in different time periods. Corresponding to different power output combinations: : represents wind power generation power, : represents the solar power generation power, and : Represents the sum of power generated by other types of energy.

[0172] S3, setting state characteristic variables of the workshop model, setting state characteristic variable definitions abstracted from the workshop environment, and adding different scheduling rules to the agent learning model;

[0173] Set the status characteristic variables of the workshop model:

[0174] : represents the total number of machines, all m here have this meaning;

[0175] : represents the total number of workpieces, all n here have this meaning;

[0176] : represents the process number of workpiece i;

[0177] : represents the total number of processes for all workpieces;

[0178] :For machine The completion time of the last process at the rescheduling point t;

[0179] : It is the process index currently completed by the workpiece Ji;

[0180] : Represents a machine The utilization rate at time t can be specifically expressed as:

[0181] ;

[0182] In the present invention, the processing time of each workpiece on different machines is different for different processes, so :Represents these different artifacts Different processes On different machines Processing time on. : is a binary decision variable used to describe the allocation of workpiece processes on machines. : represents the time t, the machine The ratio of the total time actually used to process the workpiece to the total available time of the machine (that is, the completion time of the last process) is: It means that the j-th process of workpiece i is assigned to machine k.

[0183] in, , is a binary decision variable used to describe the allocation of workpiece processes on machines, then It means that the jth process of workpiece i is assigned to machine k. Indicates the machine number assigned to the j-th process of workpiece i;

[0184] : represents the completion rate of job Ji at time t: .

[0185] Set the state characteristic variable definition abstracted from the workshop environment:

[0186] (1) Average machine utilization: ;

[0187] (2) Standard deviation of machine utilization: ;

[0188] (3) Average completion rate of each process: ;

[0189] (4) Average completion rate of workpieces: ;

[0190] (5) Standard deviation of workpiece completion rate: ;

[0191] (6) Estimated delay rate :The calculation steps are as follows;

[0192] Step 1: Initialize the number of delayed steps To 0, initialize the number of remaining processes is 0;

[0193] Step 2: Calculate the current average completion time : The completion time of all machines at the current moment Divide by the number of machines ;

[0194] Step 3: Traverse all artifacts;

[0195] Step 4: Check whether the workpiece has been processed: the number of processes that have been completed for the workpiece at the current moment If it is less than the total number of processes of the workpiece, it is considered as unfinished. The number of unfinished processes is recorded and accumulated to middle;

[0196] Step 5: Initialize the remaining time 0, used to record the estimated processing time of the remaining steps of the workpiece;

[0197] Step 6: Check the remaining processes of all workpieces and calculate the average processing time of the remaining processes , and its specific calculation method is: for each remaining process of the workpiece, calculate the average processing time on all available machines;

[0198] Step 7: Cumulative processing time of the remaining steps of the workpiece ; where “+=” is the programming language, i.e. T left = T left +t i,j ;

[0199] Step 8: Determine whether a delay occurs: If + If it is greater than the deadline of the workpiece, it is considered that the workpiece may be delayed;

[0200] Step 9: If it is determined that the workpiece is expected to be delayed, the number of remaining processes for the workpiece is accumulated to middle;

[0201] Step 10: After all workpieces have been processed as above, update the statistics of remaining processes and delayed processes.

[0202] Step 11: Calculate the estimated latency rate ;

[0203] (7) Actual delay rate :The calculation steps are as follows;

[0204] Step 1: Initialize the number of delayed steps 0, remaining number of processes is 0;

[0205] Step 2: Traverse all artifacts;

[0206] Step 3: Check whether each workpiece is completed, calculate the number of remaining processes, and accumulate the number of remaining processes;

[0207] Step 4: Check whether the current completion time of the workpiece is later than the estimated delivery time;

[0208] Step 5: If it is later than the expected delivery time, the workpiece is judged to be delayed, and the remaining number of processes of the workpiece is regarded as the number of delayed processes and added to the number of delayed processes;

[0209] Step 6: If it is not later than the expected delivery time, it is determined that the workpiece is not delayed, and the accumulation of delayed process number is skipped;

[0210] Step 7: After all workpieces have been calculated for the accumulation of the remaining number of processes and the accumulation of the delayed number of processes;

[0211] Step 8: Calculate the actual delay rate: .

[0212] Setting scheduling rules: Based on different optimization objectives, the composite scheduling rules are also different, including reducing energy costs, reducing average delay time, and reducing maximum completion time: considering some key workpieces that cannot be flexibly scheduled, changes in the rated power output of the microgrid, and making feasible scheduling, and performing the lowest cost scheduling based on real-time changes in electricity prices.

[0213] S4, constructing an improved deep reinforcement learning DQN neural network to train the agent learning model to perform dynamic intelligent scheduling on the flexible job shop, constructing an immediate compound reward and a global compound reward mechanism in the deep reinforcement learning DQN neural network and designing different scheduling rules to guide the agent to perform multi-objective optimization;

[0214] Build an improved DQN neural network for deep reinforcement learning, including:

[0215] (1) Action selection: During the training phase, the probability of randomly selecting an action at each scheduling point is , otherwise the action with a larger Q value is selected, and this probability will decrease as the number of training steps increases;

[0216] (2) Double DQN is combined with the original DQN to reduce the overestimation of Q value. The target Q value calculation formula is:

[0217] ;

[0218] in, : represents the number of time steps, where ;

[0219] : represents the return of DQN in the st time step;

[0220] : represents the return value of the st+1th time step;

[0221] : is the discount factor;

[0222] : represents the action taken by the agent;

[0223] : represents the parameters in the online network in DDQN;

[0224] : represents the parameters in the target network in DDQN;

[0225] :Indicates that based on Parameters of the online network obtained The action that maximizes the Q value function in the state .

[0226] (3) Adding dueling net to DDQN, modifying the neural network part of double DQN, introducing state value: calculating the benefit in a certain state, action advantage: evaluating the possible advantage of taking action a in state s compared to other actions, reducing complexity and improving stability. to:

[0227] ;

[0228] in, .

[0229] in, : Represents the network parameters in the dueling network, corresponding to the parameters of the action advantage function (AdvantageFunction, A): controls how the neural network estimates the advantage of an action over other actions;

[0230] : Represents the network parameters in the dueling network, corresponding to the parameters of the state value function (Value Function, V);

[0231] These two parameters are mainly used to separate action advantages from state values, thereby improving the estimation effect of Q-learning;

[0232] :refers to the given state When evaluating the value of the state itself (the potential benefit of the state), it does not depend on the action ;

[0233] :Evaluation is in state Next, execute the action Advantages over other possible actions;

[0234] and : represents any action in the action set;

[0235] : Indicates that the state The sum of the action advantages of all possible actions in the case;

[0236] : represents the number of time steps, where .

[0237] (4) In the double DQN method with dueling DQN, the N-step learning method is added so that the Q value is updated not in a single step, but in an accumulated N-step reward, that is, ; So the Q value at this time becomes:

[0238] ;

[0239] Explanation: is the reward obtained at time step st; is the discount factor; is the step length of N step; is the online network parameter; are the target network parameters.

[0240] Here is the cumulative discounted reward, which means the cumulative discounted reward between time steps st to st+N-1. Each reward Multiply by the discount factor to the corresponding power of to reduce the weight of future rewards.

[0241] is each time step The discount factor ensures that the reward weight is smaller the farther away from the current moment.

[0242] (5) Construct loss function: , updated through back-propagation and the optimizer.

[0243] Variable Description:

[0244] Batchsize: is the batch size. In DQN we usually update the model in small batches instead of the entire dataset.

[0245] : is the online network Q value;

[0246] : is the target network Q value;

[0247] The loss function here is expressed as the average error, which is minimized , to optimize the error.

[0248] (6) Constructing an immediate reward function: Immediate reward: ;

[0249] in: Indicates the impact of changes in the proportion of delayed processes on rewards; Indicates the impact of changes in the actual proportion of delayed processes on rewards; represents the effect of changes in average machine utilization on latency; Indicates the actual delayed process ratio, Indicates the expected delay ratio of the current time step, Indicates the actual delayed process ratio for the next time step, represents the expected delayed process ratio at the next time step, represents the average machine utilization at the current time step, represents the average machine utilization rate in the next time step, which is accumulated according to certain weights; then the reward is adjusted according to different values ​​of electricity cost: if the electricity cost is low, then Increase by 1; if the electricity cost is high, then Decrease by 1; if the electricity cost remains unchanged, Unchanged. Guide the agent to act.

[0250] Construct a global reward: Give the agent a reward taking into account energy costs and artifact delays. Calculate the global reward:

[0251] 1) The global reward is calculated based on the total electricity cost and the delay time. If the current total electricity cost is less than the minimum electricity price, the reward r0 is increased by one; if the current total electricity cost is greater than the maximum electricity price, the reward r0 is reduced by one.

[0252] 2) Calculate r1 based on the delay time of the workpiece. If the current delay time is less than the minimum delay time, the reward r1 is increased by one; if the current total electricity cost is greater than the maximum delay time, the reward r1 is reduced by one. Finally, calculate the total global reward reward = r1+r0. This can guide the agent to choose the optimal strategy.

[0253] The following describes the specific important components and the differences between the present invention and the traditional DQN:

[0254] (1) Q network: A deep neural network that estimates a given state and actions The corresponding Q value, this paper improves on the traditional DQN network and incorporates the core ideas of dueling network, DDQN and n-step learning:

[0255] ;

[0256] is the reward obtained at time synchronization st; is the discount factor; is the step length of N step; is the online network parameter; are the target network parameters.

[0257] in: Here is the cumulative discounted reward, which means the cumulative discounted reward between time steps st to st+N-1. Each reward Multiply by the discount factor to the corresponding power of to reduce the weight of future rewards.

[0258] is each time step The discount factor ensures that the reward weight is smaller the farther away from the current moment.

[0259] The rest remains the same as above.

[0260] (2) Target network: Calculate the target Q value. Here, we use the DDQN idea and use soft update of the target network to reduce the overestimation of Q value during training.

[0261] (3) Loss function: used to evaluate the gap between the current network and the target network;

[0262] (4) Greedy strategy: According to the greedy strategy, the action with a higher Q value or the randomly selected action is selected. The actions here correspond to different scheduling rules. As the number of training rounds increases, the probability of selecting an action with a higher Q value increases;

[0263] (5) Discount factor : Regulate the impact of future rewards on current decisions;

[0264] (6) Q-learning algorithm: optimizes the decision-making of the intelligent agent by iteratively approximating the optimal Q value;

[0265] (7) Experience replay: The traditional reinforcement learning algorithm DQN stores the experience of the single-step interaction between the intelligent agent and the environment in an experience buffer and performs random sampling to update the Q value. In this invention, the idea of ​​n-step learning is combined to train the intelligent agent instead of single-step rewards by considering rewards of multiple time steps. This is more effective for intelligent agent learning in complex environments.

[0266] Add different scheduling rules to the agent model: it represents the actions described in its model It is replaced by different scheduling rules, so as to calculate the Q value between different scheduling rules and the loss function, etc. Thus, the mapping between the actual flexible job shop and the algorithm is obtained.

[0267] S5, evaluating the training effect of the intelligent agent learning model; integrating the evaluated intelligent agent learning model into the control system or embedded device in the factory to achieve autonomous scheduling.

[0268] Training strategy model and evaluation: The total number of training rounds L is set to 10,000. The intelligent agent is trained and tested using the simulated workshop environment, microgrid energy output characteristics, and public grid electricity price distribution characteristics. The performance of the improved version of DQN and the optimization effects of different scheduling strategies are evaluated, and the output parameter changes are observed.

[0269] The present invention can implement each process in the flowchart or block diagram through computer program instructions to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0270] The application of the present invention generally involves integrating the learning model and scheduling decision process of the intelligent agent into a specific control system or embedded device to achieve autonomous scheduling. For example, the following embodiments:

[0271] The first step is to choose a suitable hardware platform to achieve DQN autonomous scheduling. Usually choose:

[0272] Edge computing devices: such as edge servers or dedicated AI chips, are suitable for processing more complex deep learning tasks.

[0273] Embedded systems: such as Arduino, Raspberry Pi, etc., suitable for simple scheduling tasks.

[0274] Industrial controllers: such as PLC (Programmable Logic Controller), suitable for operation in industrial environments.

[0275] Step 2: Real-time data collection and processing: The machine needs to monitor its vehicle in real time.

[0276] The agent can collect data from the machine state (such as workload, machine availability, workpiece status, etc.) and pass this information as input to the DQN model. Data collection can be achieved through sensors and data acquisition systems, and the data is pre-processed and input to the agent.

[0277] Step 3: Agent decision and scheduling: The DQN model receives the workshop status input during operation and outputs the corresponding action (scheduling rule decision) through the policy network. This process usually involves:

[0278] Status acquisition: Regularly update the status information of the machine.

[0279] Action selection: Select the scheduling action based on the current state (using a greedy strategy).

[0280] Execute action: Execute the selected composite scheduling rule into the machine control system.

[0281] Step 4: Continuous learning and adaptation: To enable the machine to continuously adapt to the changing production environment, online learning can be integrated into the system. By continuously collecting new experience data and updating the parameters of the agent learning model, it can make better decisions in the new state.

[0282] Step 5: Integration and testing: After the agent learning model is integrated into the specific scheduling system, sufficient testing and verification are required to ensure the stability and reliability of the system. This includes:

[0283] Simulation testing: Test scheduling performance in a simulation environment.

[0284] Field testing: Conduct trial runs in actual production environments to adjust and optimize the model.

[0285] In the fields of industrial manufacturing, logistics, and supply chain management, some companies and research teams have successfully implemented autonomous scheduling system cases based on DQN. For example: Smart factory: By embedding reinforcement learning algorithms in machines on the production line, autonomous scheduling and resource optimization are achieved.

[0286] Warehouse management: Use DQN to schedule cargo handling and sorting in automated warehouses.

[0287] The following embodiment provides a complete algorithm process of a multi-objective flexible workshop dynamic intelligent scheduling method for an industrial green microgrid system of the present invention, including the following steps:

[0288] S1: As 2 and Figure 3 As shown, enter the main loop and set the iteration termination condition, that is, when the iteration number t is greater than the maximum iteration number, jump out of the loop.

[0289] S2: According to the situation of the flexible job shop, define the number of workpieces N, the number of machines M, the processing time of each process of each workpiece on different machines, the delivery date of the workpiece, the arrival date of the workpiece, initialize the environment in deep reinforcement learning, and abstract it into feature states (average machine utilization, standard deviation of machine utilization, average process completion rate, average workpiece completion rate, standard deviation of workpiece completion rate, expected delay rate). The scheduling decision point is defined as the moment when each process needs to be scheduled.

[0290] S3: For each process of the workpiece, (1- ) with a probability of selecting an action with a larger Q value. Each action corresponds to a scheduling rule with different optimization objectives. It will decrease as the number of iterations increases.

[0291] S4: According to the improved composite scheduling rule regarding the optimization objective, an index of workpieces and machines is generated.

[0292] S5: Make a decision on the start time based on the end time of the previous process of the workpiece, the completion time of the machine and the arrival time of the workpiece, such as Figure 5 shown.

[0293] S6: After the decision is made, it is considered that a process is scheduled and the relevant characteristic data of the workshop environment is updated: average machine utilization, standard deviation of machine utilization, average process completion rate, average workpiece process completion rate, standard deviation of workpiece process completion rate, expected delay rate, actual delay rate, average electricity price and other information.

[0294] S7: Calculate the instant reward based on the updated information:

[0295] ;

[0296] in: Indicates the impact of changes in the proportion of delayed processes on rewards; Indicates the impact of changes in the actual proportion of delayed processes on rewards; Represents the effect of changes in average machine utilization on latency.

[0297] S8: Next, adjust the reward according to different values ​​of electricity cost: if the electricity cost is small, then Increase by 1; if the electricity cost is high, then Decrease by 1; if the electricity cost remains unchanged, constant.

[0298] S9: Put the experience information: status, action, reward and other information into a temporary storage room.

[0299] S10: The electricity price and other related parameters of each process are updated.

[0300] S11: If all processes of all workpieces have not been scheduled to be completed, return to S2 and then proceed to the selection of workpieces and machines for the next process.

[0301] S12: After multiple production task allocations, all workpiece processes are scheduled and completed, which is considered as the completion of a round of iteration. The start time, end time, and machine list of each workpiece are recorded, and the electricity cost of each workpiece is calculated.

[0302] S13: Determine whether each workpiece has exceeded its deadline. If so, calculate the average delay time of the workpieces in each iteration. The number of empirical samples k is increased by 1.

[0303] S14: Calculate the global reward: (1) Calculate the global reward reward based on the total electricity cost and delay time. If the current total electricity cost is less than the minimum electricity price, the reward r0 is increased by one; if the current total electricity cost is greater than the maximum electricity price, the reward r0 is reduced by one. (2) Calculate r1 based on the delay time of the workpiece. If the current delay time is less than the minimum delay time, the reward r1 is increased by one; if the current total electricity cost is greater than the maximum delay time, the reward r1 is reduced by one. Finally, calculate the total global reward reward = r1+r0.

[0304] S15: Determine whether the iteration is finished. If not, return to step S1.

[0305] S16: After collecting enough experience samples, put the experience into the experience playback area and perform N steplearning method: after collecting N consecutive experiences, calculate the reward of the accumulated N experiences and adjust the discount factor to .

[0306] S17: Using the improved network model:

[0307] , to output the Q value estimate, where is the discounted return from time step st to st+N-1, is the discount factor. If The last time step , then the Q value will depend on the actual reward obtained.

[0308] S18: Calculate the loss function and soft-update the target network.

[0309] S19: Perform back propagation to update the weights of the online network.

[0310] S20: End of iteration, output average delay time, average maximum completion time, and electricity cost of each round. Graphical analysis.

[0311] S21: Using the improved deep reinforcement learning algorithm of the present invention, the green flexible job shop scheduling problem is solved, and after obtaining the minimum maximum completion time, a Gantt chart is generated. In the Gantt chart, the horizontal axis is the processing time, the vertical axis is each machine, each bar represents the information of the workpiece and the process and the electricity cost, and the length of the bar represents the processing time of each process of each workpiece.

[0312] The model of the present invention is a composite dispatching rule formulated with the optimization objectives of reducing the average delay time, reducing the cost, and reducing the maximum completion time, and performs job scheduling in flexible job shops based on industrial green microgrids and public power grids. Tables 1, 2, and 3 are data comparisons based on different DQN variant algorithms before and after combining the composite dispatching rules and the improved reward function. The optimization objectives of the tables are respectively the average delay time, the maximum completion time, and the electricity cost. After data comparison, it can be found that the improved DQN version of the present invention combined with the improved composite dispatching rules can have the advantages of reducing the average delay time, reducing the cost, and reducing the maximum completion time, and the effect is more obvious than other DQN variants.

[0313] Table 1. Comparison of average delay time training optimization results

[0314]

[0315] Table 2. Comparison of maximum completion time training optimization results

[0316]

[0317] Table 3. Comparison of electricity cost training optimization results

[0318]

[0319] The matters not described in detail in the present invention are all known technologies to those skilled in the art.

[0320] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A dynamic intelligent scheduling method for multi-objective flexible workshops in industrial green microgrids, characterized in that: include: S1, constructing a theoretical model of flexible workshop operation scheduling for green and low-carbon factories in a computer system, wherein the theoretical model of flexible workshop operation scheduling for green and low-carbon factories is composed of three major units: a public power grid, a new energy power supply system, and a factory. A workshop model is set in the factory, and the workshop model includes an intelligent agent learning model; S2, constructing an objective function for deep reinforcement learning of the agent learning model, and integrating different scheduling rules into the agent learning model; The objective function is constructed as follows: ; ； in, : Indicates minimizing the total delay time; : represents the last process of each workpiece i The completion time of job i is the actual completion time of job i. : is the expected delivery time of workpiece i, where: ; like , then the actual completion time is greater than the expected delivery time, and the processing of the workpiece is delayed; : represents minimizing energy cost, : represents energy cost; : represents the maximum value of the completion time of all workpieces that is minimized, that is, the maximum completion time; : Indicates the maximum completion time of all workpieces at the end of an iteration; S3, setting state characteristic variables of the workshop model, setting state characteristic variable definitions abstracted from the workshop environment, and adding different scheduling rules to the agent learning model; S4, constructing an improved deep reinforcement learning DQN neural network to train the agent learning model to perform dynamic intelligent scheduling on the flexible job shop, constructing an immediate compound reward and a global compound reward mechanism in the deep reinforcement learning DQN neural network to guide the agent to perform multi-objective optimization; S5, evaluating the training effect of the intelligent agent learning model; integrating the evaluated intelligent agent learning model into the control system or embedded device in the factory to achieve autonomous scheduling; In step S4, the step of constructing an improved deep reinforcement learning DQN neural network includes: S41, using the ε-greedy strategy for action selection to balance exploration and exploitation; S42, double DQN technology is applied to reduce the overestimation of Q value; S43, introduces the dueling DQN network structure to evaluate state value and action advantage; S44, implement the N step learning method, accumulate N step rewards to update the Q value; S45, construct a loss function, and use the mean square error loss function to define the error between the evaluation network and the target network; S46, designing immediate reward functions and global reward functions to guide the agent to perform multi-objective optimization, including reducing costs and delays; In step S42, the double DQN technique is applied to reduce the overestimation of the Q value, and the target Q value calculation formula is: ； in, : represents the number of time steps, where ; : represents the return of DQN in the st time step; : represents the return value of the st+1th time step; : is the discount factor; : represents the action taken by the agent; : represents the parameters in the online network in DDQN; : represents the parameters in the target network in DDQN; :Indicates that based on Parameters of the online network obtained The action that maximizes the Q value function in the state ; In step S43, the dueling DQN network structure is introduced to evaluate the state value and action advantage, including: Add dueling net to DDQN, modify the neural network part of double DQN, introduce state value; calculate the benefit and action advantage in a certain state; evaluate the possible advantage of taking action a compared to other actions in state s; to: ； in, ; in, : Represents the network parameters in the dueling network, corresponding to the parameters of the action advantage function: controls how the neural network estimates the advantage of an action over other actions; : Represents the parameters of the corresponding state value function in the dueling network; these two parameters are used to separate action advantage from state value, thereby improving the estimation effect of Q-learning; :refers to the given state When the state itself is evaluated, it does not depend on the action. ; :Evaluation is in state Next, execute the action Advantages over other possible actions; and : represents any action in the action set; : Indicates that the state The sum of the action advantages of all possible actions in the case; : represents the number of time steps, where ; The loss function constructed in step S45 is: , updated through back-propagation and the optimizer; Among them, batchsize: is the batch size; : is the online network Q value; : is the target network Q value; the loss function is expressed as the average error, which is minimized , to optimize the error.

2. The industrial green microgrid multi-objective flexible workshop dynamic intelligent scheduling method according to claim 1 is characterized in that: The learning process of the agent in the agent learning model includes the following cyclic iterative process: at each time step, the agent obtains the current state ; Based on the state, select actions through greedy strategy ; Execute action , the environment will feedback the new state and rewards ; Store the experience in the experience replay buffer; Randomly sample N steps of experience from the experience replay buffer to update the online network; Soft update the target network; Calculate the loss function for back propagation and update the weights of the online network; Repeat the process until the training converges.

3. The industrial green microgrid multi-objective flexible workshop dynamic intelligent scheduling method according to claim 1 is characterized in that: In step S3, the setting of the state characteristic variables of the workshop model includes: : represents the total number of machines, all m here have this meaning; : represents the total number of workpieces, all n here have this meaning; : represents the process number of workpiece i; : represents the total number of processes for all workpieces; : represents the first A process; :Represents these different artifacts Different processes On different machines Processing time on :For machine The completion time of the last process at the rescheduling point t; : It is the process index currently completed by the workpiece Ji; : Represents a machine The utilization rate at time t is specifically expressed as: ； in, : Represents different workpieces Different processes On different machines Processing time on : represents the time at which the machine The total time actually used to process the workpiece accounts for the total available time of the machine, that is, the completion time of the last operation; in, , is a binary decision variable used to describe the allocation of workpiece processes on machines, then It means that the jth process of workpiece i is assigned to machine k. Indicates the machine number assigned to the j-th process of workpiece i; :Indicates job J i Completion rate at time t: .

4. The industrial green microgrid multi-objective flexible workshop dynamic intelligent scheduling method according to claim 3 is characterized in that: In step S3, the setting of the state characteristic variable definition abstracted from the workshop environment includes: Average machine utilization: ; Standard deviation of machine utilization: ; Average process completion rate: ; Average completion rate of workpieces: ; Standard deviation of workpiece completion rate: ; Estimated delay rate ; Actual delay rate .

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the industrial green microgrid multi-objective flexible workshop dynamic intelligent scheduling method as described in any one of claims 1 to 4 is implemented.

6. An electronic device, characterized in that: include: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, enables the at least one processor to implement the industrial green microgrid multi-objective flexible job shop dynamic intelligent scheduling method as described in any one of claims 1-4.

Images