Multi-objective dynamic flexible job shop scheduling optimization method and device

Abstract

This invention relates to the field of flexible work shop scheduling technology, and discloses a multi-objective dynamic flexible work shop scheduling optimization method and apparatus. The method includes: constructing a flexible work shop scheduling model integrating multiple agent scheduling units; constructing a deep reinforcement learning objective function for the multiple agent scheduling units, and transforming scheduling rules into constraints and weight coefficients; acquiring the microgrid energy status, dynamic work queues, and real-time equipment status, defining flexible work shop state characteristic variables based on constraints, and embedding scheduling rules to construct a unified state space; based on an improved near-end strategy optimization algorithm, jointly training the multiple agent scheduling units using the state space, and setting a multi-dimensional composite reward mechanism to guide the multiple agent scheduling units to achieve optimized scheduling in a dynamic environment. This invention can achieve autonomous, intelligent, low-carbon, and efficient scheduling of flexible work shops in environments with intermittent renewable energy supply and frequent dynamic events within the work shop.

Description

Technical Field

[0001] This invention relates to the field of flexible job shop scheduling technology, and in particular to a multi-objective dynamic flexible job shop scheduling optimization method. Background Technology

[0002] In flexible manufacturing processes, shop floor scheduling remains a crucial aspect of optimizing production efficiency and reducing operating costs. Traditional scheduling methods, largely based on static job sets and deterministic power supply, struggle to adapt to the dynamic changes in microgrid environments, such as unstable energy output and frequent job disturbances. Therefore, developing an intelligent scheduling method that comprehensively incorporates the intermittency of renewable energy in microgrids and the dynamic load of shop floors will be a core direction for breakthroughs in green manufacturing scheduling.

[0003] Existing research often employs swarm intelligence optimization algorithms (such as genetic algorithms and particle swarm optimization) to address the multi-objective flexible job shop scheduling problem. While this has yielded some results, it generally suffers from limitations such as poor environmental adaptability and weak real-time response capabilities. Some methods have introduced reinforcement learning to enhance the adaptive capabilities of the scheduling model, but most remain limited to traditional production objectives and have not yet achieved systematic modeling of multiple energy inputs, energy storage variations, and carbon emission constraints in the scheduling decision-making process. In particular, they lack a dynamic optimization mechanism that can coordinate resource allocation and job sequencing within a multi-agent framework.

[0004] Therefore, current green and intelligent scheduling technologies still face several challenges: First, it is difficult to simultaneously coordinate multiple production objectives (such as completion time and delays) and low-carbon objectives (such as energy consumption and carbon emissions); second, the modeling of the volatility of new energy sources in microgrid environments is insufficient; and third, the scheduling system lacks real-time response and optimization capabilities for disturbance scenarios. Therefore, there is an urgent need to propose a multi-objective dynamic flexible job shop scheduling optimization method to achieve dynamic coupling between job shop scheduling and energy optimization control, thereby improving the operational efficiency and environmental adaptability of green manufacturing systems. Summary of the Invention

[0005] The purpose of this invention is to overcome the deficiencies in the prior art and provide a multi-objective dynamic flexible workshop scheduling optimization method and device, which can achieve autonomous, intelligent, low-carbon and efficient scheduling of flexible workshops in environments with intermittent new energy supply and frequent dynamic events within the workshop, thereby achieving the goals of energy conservation, emission reduction, cost reduction and efficiency improvement.

[0006] In a first aspect, embodiments of the present invention provide a multi-objective dynamic flexible job shop scheduling optimization method, including:

[0007] A flexible job shop scheduling model is constructed, which includes a multi-agent scheduling unit, comprising a resource allocation agent and a job sequencing agent.

[0008] The deep reinforcement learning objective function of the multi-agent scheduling unit is constructed based on the production scheduling rules, and the scheduling rules are transformed into its constraints and weight coefficients.

[0009] The real-time status of microgrid energy, dynamic work queues, and production equipment is obtained, and the state characteristic variables of the flexible workshop are defined in combination with the constraints.

[0010] The scheduling rules are embedded into the state feature variables to construct a unified state space containing variable relationships;

[0011] Based on the improved near-end policy optimization algorithm, the resource allocation agent and the job sorting agent are jointly trained using the state space, and a multi-dimensional composite reward mechanism associated with the weight coefficients is set to guide the multiple agents to perform optimization scheduling in a dynamic environment.

[0012] In a preferred embodiment, the deep reinforcement learning objective function of the multi-agent scheduling unit is constructed using the following formula:

[0013]

[0014]

[0015] in, : Indicates minimizing the job completion time; : indicates homework Completion time; : Represents the set of all jobs; : Indicates minimizing the average delay of a task; : Indicates the number of assignments; Indicate homework The deadline; : Indicates minimizing carbon emissions during the production process; : Indicates the number of machines; : Indicates time machine power, This represents the power that the microgrid can provide at time t; : This indicates minimizing the total energy consumption of the production process.

[0016] In a preferred embodiment, the flexible workshop state characteristic variables include at least a combination of the following variables:

[0017] Total number of assignments Total number of machines ,Operation operands ,Operation The One operation ,operate In the machine On the processing time ,operate In the machine On the processing power ,Operation Arrival time and completion time Waiting at the machine Up-processing job operation set ,machine Expected idle time Photovoltaic system power generation Power generation of wind power systems Residual energy of energy storage systems Machine utilization rate Expected electricity purchase rate , work slack ,machine Expected delay rate of the above work and machines Actual delay rate of the above work .

[0018] As a preferred embodiment, the expected delay rate Or actual delay rate The calculation methods include:

[0019] Define the criteria for determining whether a task is overdue;

[0020] For the job waiting queue of the current machine at the corresponding time, select any job in the queue and move it to the first position of the queue. Construct a new job sorting queue based on the adjusted job order.

[0021] For each job in the newly constructed job sorting queue, calculate the time taken for all jobs in the queue that are currently waiting to be processed, and determine the total time taken as the waiting time for that job.

[0022] By combining the waiting time of each task, the current time of calculation, and the deadline of the task, the expected delay rate or the actual delay rate can be calculated.

[0023] As a preferred implementation, the step of jointly training the resource allocation agent and the job ranking agent using the state space based on the improved near-end policy optimization algorithm includes:

[0024] The differentiated state variables in the state space are respectively input into the resource allocation agent and the job sorting agent, and the collaborative scheduling action is obtained through reasoning and sampling by their respective policy networks;

[0025] The scheduling action is sent to the flexible workshop environment, and the status data and instant rewards for the next moment are collected. The instant rewards include resource allocation instant rewards and job sorting instant rewards.

[0026] Based on the improved near-end policy optimization algorithm, combined with the state data and immediate reward, the policy update amplitude is limited by the shearing mechanism, the policy diversity is adjusted by adding an entropy term to the policy loss, and the error is calculated by multi-step temporal difference, and the action advantage estimate is accumulated to correct the policy update direction.

[0027] By combining the trajectory data collected in parallel, the policy network and value network parameters of the resource allocation agent and the job sorting agent are updated synchronously.

[0028] Repeat the above process until the policy network converges, completing the joint training.

[0029] As a preferred embodiment, the policy network objective optimization function of the improved near-end policy optimization algorithm for:

[0030] ;

[0031] ;

[0032] in, This represents the ratio of the probabilities of the new and old strategies; This is the estimated value of the dominance function; This is a hyperparameter representing the shear range; The parameters of the current and old policy networks, For TD error; These are GAE parameters used to adjust bias and variance; This represents a discount factor used to decay future rewards in order to balance immediate and long-term gains.

[0033] In a preferred embodiment, the total loss function used to update the policy network parameters and value network parameters in the improved near-end policy optimization algorithm is:

[0034] ;

[0035] in, For the error of the state value function, The entropy of the strategy is used to encourage exploration. These are the weighting coefficients.

[0036] As a preferred implementation, the instant reward function of the resource allocation agent Instant reward function for task ranking agents They are respectively:

[0037] ;

[0038] ;

[0039] in, As a guideline for electricity prices, For energy efficiency optimization items, For resource balance, For relaxation guide terms, For processing time guidance, This is the term for suppressing the delay rate disturbance. This refers to the expected delays and interference.

[0040] In a preferred embodiment, the method further includes:

[0041] Run the trained resource allocation agent and job sorting agent in a simulation environment and collect performance index data during the process.

[0042] The optimization effect of the flexible job shop scheduling model is evaluated based on the performance index data.

[0043] Secondly, embodiments of the present invention also provide a multi-objective dynamic flexible job shop scheduling optimization device, comprising:

[0044] The workshop scheduling model construction module is used to construct a flexible workshop scheduling model that includes a multi-agent scheduling unit, wherein the multi-agent scheduling unit includes a resource allocation agent and a job sequencing agent;

[0045] The deep reinforcement learning objective function construction module is used to construct the deep reinforcement learning objective function of the multi-agent scheduling unit based on the production scheduling rules, and to transform the scheduling rules into its constraints and weight coefficients;

[0046] The state feature variable definition module is used to obtain the real-time status of microgrid energy status, dynamic work queue and production equipment, and define the state feature variables of flexible work workshop in combination with the constraints.

[0047] A unified state space construction module is used to embed the scheduling rules into the state feature variables to construct a unified state space containing variable relationships.

[0048] The multi-agent scheduling unit joint training module is used to jointly train the resource allocation agent and the job sorting agent based on the improved near-end policy optimization algorithm using the state space, and to set up a multi-dimensional composite reward mechanism associated with the weight coefficients to guide the multi-agent to perform optimization scheduling in a dynamic environment.

[0049] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0050] One or more processors;

[0051] Storage device for storing one or more programs;

[0052] When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-objective dynamic flexible job shop scheduling optimization method described in any embodiment of the present invention.

[0053] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-objective dynamic flexible job shop scheduling optimization method described in any embodiment of the present invention.

[0054] To achieve the above objectives, the beneficial effects of the present invention are:

[0055] (1) This invention innovatively introduces the near-end strategy optimization (PPO) algorithm into the job scheduling and machine allocation process. By dividing the resource allocation agent and the job scheduling agent and giving them differentiated state information and optimization objectives, it can achieve coordinated control of reducing carbon emissions and energy consumption under dynamic job arrival and energy supply fluctuation conditions, thereby meeting the green transformation needs of manufacturing systems under the "dual carbon" background.

[0056] (2) Compared with traditional scheduling methods that rely on heuristic scheduling rules or static modeling, this invention fully leverages the advantages of combining policy iteration and value assessment in reinforcement learning to continuously optimize decision-making strategies and gradually learn low-energy, low-emission, and timely scheduling behaviors. By constructing a refined reward function system, the ranking agent is guided to minimize delays and completion times, thereby reducing energy consumption and carbon emission intensity and significantly improving the environmental adaptability and global optimization capability of the overall scheduling strategy.

[0057] (3) This invention achieves an effective trade-off between key performance indicators such as maximum completion time, operation delay, carbon emissions and total energy consumption through multi-objective optimization. It is applicable to various flexible work workshop scheduling scenarios, and is particularly suitable for the supply of new energy to the workshop in the microgrid industrial environment, as well as flexible manufacturing scenarios where the load fluctuates drastically inside the factory workshop. It can significantly reduce manual intervention, enhance the manufacturing system's ability to respond to real-time dynamic conditions, and help enterprises achieve the goal of cost reduction, efficiency improvement and green manufacturing synergistic development. Attached Figure Description

[0058] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0059] Figure 1 This is a flowchart of a multi-objective dynamic flexible job shop scheduling optimization method provided in Embodiment 1 of the present invention;

[0060] Figure 2 This is a structural diagram of a multi-agent reinforcement learning collaborative framework for multi-objective dynamic flexible job shop scheduling optimization provided in Embodiment 1 of the present invention;

[0061] Figures 3(a)-3(i) The second embodiment of the present invention provides a Pareto solution distribution diagram for different methods under the three objectives of completion time, delay rate, and energy consumption;

[0062] Figure 4 This is a schematic diagram of the structure of a multi-objective dynamic flexible workshop scheduling optimization device provided in Embodiment 4 of the present invention;

[0063] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention;

[0064] Wherein, 10: electronic device, 11: processor, 12: read-only memory (ROM), 13: random access memory (RAM), 14: bus, 15: I / O interface, 16: input unit, 17: output unit, 18: storage unit, 19: communication unit. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0066] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations (or steps) can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0067] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations. It should be noted that existing industry solutions, such as software, components, or models, may be mentioned in the embodiments of this application. These should be considered exemplary and their purpose is solely to illustrate the feasibility of implementing the technical solution of this application.

[0068] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0069] Example 1

[0070] Figure 1 This is a flowchart of a multi-objective dynamic flexible job shop scheduling optimization method provided in Embodiment 1 of the present invention. This embodiment is applicable to various flexible job shop scheduling scenarios. The method of this embodiment can be executed by a multi-objective dynamic flexible job shop scheduling optimization device, which can be implemented in hardware and / or software. This device can be configured in the server of the multi-objective dynamic flexible job shop scheduling optimization device. The method specifically includes the following steps:

[0071] S110. Construct a flexible job shop scheduling model that includes a multi-agent scheduling unit, wherein the multi-agent scheduling unit includes a resource allocation agent and a job sequencing agent.

[0072] Among them, the resource allocation agent is used to make decisions on machine selection, and the job ranking agent is used to make decisions on job ranking.

[0073] Optional, combined Figure 2 It can be seen that the flexible job shop scheduling model is a theoretical model of flexible job scheduling for green and low-carbon factories built in a computer system. This model should also include at least the following core components:

[0074] Microgrid energy supply unit: integrates photovoltaic system (solar energy), wind power system (wind energy) and energy storage system to provide renewable energy power generation in real time, form a complementary power supply with the main grid, and realize dynamic energy distribution through power flow;

[0075] Dynamic job insertion unit: Integrated into the flexible workshop module, its core functions include:

[0076] Dynamic task awareness: Real-time monitoring of workpiece arrival signals and identification of priority indicators for urgent tasks;

[0077] Disturbance response mechanism: When a new job arrives or an urgent task is inserted, the multi-agent scheduling unit is triggered to perform an immediate rescheduling process, suspend the current non-critical job sorting calculation, prioritize updating the job queue data and synchronizing it to the state feature variables;

[0078] Queue optimization and adjustment: Based on characteristics such as job slack and deadline, the queue of jobs to be processed is reordered to ensure that urgent tasks are given priority in resource allocation, while minimizing the disturbance to the original scheduling scheme.

[0079] Flexible workshop execution unit: receives multi-agent scheduling instructions, controls machine processing flow, and provides feedback on real-time processing status and energy consumption data;

[0080] Resource allocation agent: It consists of an actor head, a shared network, and a commentator head. The actor head is used to output the machine selection probability distribution to realize the matching decision between the job and the machine. The shared network is used to extract common features from the state feature variables to provide unified feature input for the actor and the commentator. The commentator head is used to evaluate the value of the current resource allocation action, calculate the advantage function estimate, and assist in policy update.

[0081] Job ranking agent: The structure is consistent with the resource allocation agent, focusing on job queue priority decision-making. Its actor head outputs the job ranking probability, and the commentator head evaluates the value of the ranking action.

[0082] Experience buffer: Used to store state-action-reward-next state trajectory data during the scheduling process. The data is managed using a first-in-first-out mechanism. When the amount of data reaches a threshold (e.g., 10,000 records), data is extracted in batches for the synchronous update of the multi-agent policy network and value network, thereby improving training stability.

[0083] Main grid: As a supplementary power source for the microgrid energy supply unit, when the power generation capacity of renewable energy (wind power, solar power) is insufficient, power is transmitted to the flexible operation workshop through power flow. The direction of power flow is dynamically determined by the energy state characteristic variables of the microgrid to ensure the continuity of power supply.

[0084] Power flow refers to the transmission path and power distribution relationship of electrical energy between the main power grid, microgrid system and flexible work workshop. Its magnitude and direction are calculated by real-time monitoring of the power data of each node.

[0085] This model deeply integrates the energy fluctuation characteristics of microgrids with flexible workshop scheduling, which can effectively cope with the fluctuation of new energy supply and the dynamic disturbance of workshops, and realize green and low-carbon manufacturing.

[0086] S120. Construct the deep reinforcement learning objective function of the multi-agent scheduling unit based on the production scheduling rules, and transform the scheduling rules into its constraints and weight coefficients.

[0087] Optionally, the flexible job shop scheduling problem can be formalized as follows: Given several jobs and machines, each job consists of multiple processes, and each process can be processed from a specified set of machines. The scheduling decision point corresponds to the timing of the next process for a job. The objective is to select the appropriate machine and determine the processing sequence and start time, thereby achieving multi-objective optimization.

[0088] The actual production scheduling rules must meet the following requirements: a single machine can only process one workpiece at any given time; the processing process cannot be interrupted; there is no mandatory sequence between processes; all machines are available at the initial time; the processing time for the same process varies on different machines; energy constraints reflect the fluctuations in energy supply in green and low-carbon factories, and high-energy-consuming operations are reasonably controlled to avoid production interruptions due to insufficient energy; the priority of urgent workpieces is dynamically adjusted to ensure the minimization of maximum completion time, production cost, and average delay.

[0089] Optionally, the deep reinforcement learning objective function of the aforementioned multi-agent scheduling unit is set based on the operating characteristics of the flexible work workshop in the microgrid fluctuating renewable energy supply environment, while also taking into account the actual production scheduling rules and multi-dimensional scheduling performance indicators. Its optimized scheduling indicators include, but are not limited to:

[0090] Optimize workpiece processing time to reduce overall delays while ensuring delivery deadlines;

[0091] In scenarios where renewable energy supply is insufficient, intelligent adjustment of work load allocation strategies can improve power utilization efficiency, thereby effectively reducing total system energy consumption and carbon emissions.

[0092] Furthermore, with the goal of minimizing the overall scheduling cost, the policy network function of the multi-agent scheduling unit is driven to continuously iterate and learn.

[0093] Specifically, the deep reinforcement learning objective function described above is constructed using the following formula:

[0094]

[0095] ;

[0096] in, : Indicates minimizing the job completion time; : indicates homework Completion time; : Represents the set of all jobs; : Indicates minimizing the average delay of a task; : Indicates the number of assignments; Indicate homework The deadline; : Indicates minimizing carbon emissions during the production process; : Indicates the number of machines; : Indicates time machine power, This represents the power that the microgrid can provide at time t; : This indicates minimizing the total energy consumption of the production process.

[0097] Furthermore, this invention also considers the significant time-dependent fluctuations in the output power of renewable energy sources (such as wind and solar power) in green microgrids. To accurately simulate the power supply capacity at different times, the power supply capacity of the microgrid in each scheduling cycle is set as a piecewise function to reflect the dynamic change trend of renewable energy output power over time. The specific form and parameter values ​​of this piecewise function can be flexibly adjusted according to the actual operating scenario to improve the model's adaptability and generalization ability.

[0098] Wherein, the new energy power supply capacity function is represented in the first... The total combined power generation from wind, solar and other clean energy sources during the specified time period. , denoted as:

[0099] ;

[0100] in, This indicates the power generation capacity of wind energy during that time period. This indicates the power generation capacity of solar energy. This indicates the power generation capacity of the energy storage system.

[0101] Over time, the power supply capacity of various energy sources will exhibit periodic fluctuations or random changes. To more effectively address the dynamic nature of energy supply, the scheduling system established in this invention can receive and analyze the current moment in real time. The total available power is used to adjust the scheduling strategy and achieve flexible scheduling based on energy status awareness.

[0102] S130. Obtain the real-time status of microgrid energy, dynamic work queue and production equipment, and define the flexible workshop status characteristic variables in combination with the constraints.

[0103] This step aims to acquire the real-time status of microgrid energy, dynamic work queues, and production equipment through a multi-source heterogeneous data acquisition system, and define the state characteristic variables of the flexible workshop in conjunction with scheduling constraints, so as to provide accurate state input for the multi-agent scheduling unit.

[0104] Among them, the energy status of the microgrid can be obtained through photovoltaic / wind power inverter monitoring modules, energy storage management systems, and smart meters, including but not limited to photovoltaic power, wind power, remaining energy of energy storage, and total power supply of the microgrid.

[0105] Dynamic job queue information is obtained through the interfaces of the MES system and the job scheduling system, including but not limited to the total number of jobs, the number of job operations, the processing time, the deadline, the set of operations to be processed, and the job slack.

[0106] The machine collects real-time status data of production equipment through built-in sensors and edge computing devices, including but not limited to machine utilization, expected idle time, processing power, and expected electricity purchase rate.

[0107] Optionally, based on the collected microgrid energy status, dynamic work queues, and real-time status of production equipment, and in conjunction with scheduling constraints, the following key state characteristic variables are defined, which include at least a combination of the following variables:

[0108] Dynamic job queue characteristics: total number of jobs Total number of machines ,Operation operands ,Operation The One operation ,operate In the machine On the processing time ,operate In the machine On the processing power ,Operation Arrival time and completion time ,Operation Delivery period Waiting at the machine Up-processing job operation set ,Operation Completed operands ,machine Expected idle time ;

[0109] Microgrid energy state characteristics: machines Power consumption Photovoltaic system power generation Power generation of wind power systems Total available power supply of the microgrid Residual energy of energy storage systems ;

[0110] Real-time status of production equipment: Expected energy consumption Machine utilization rate Expected electricity purchase rate With work slack .

[0111] Specifically, the aforementioned expected energy consumption Machine utilization rate Expected electricity purchase rate With work slack It can also be calculated using the following formula:

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] in, Indicates if operation Assigned to machine If the process is applied, the value is 1; otherwise, it is 0. Indicates machine At any moment The machine power.

[0117] Optionally, for the differentiated roles of resource allocation agent and job sorting agent in the multi-agent scheduling framework, targeted state feature input dimensions are designed for resource allocation agent and job sorting agent respectively.

[0118] The state characteristic variables of the resource allocation agent include at least the following:

[0119] Machine utilization set ;

[0120] Set of electricity purchase rates for operations on different machines ;

[0121] Set of processing times for a task on different machines ;

[0122] Processing power set of the operation on different machines ;

[0123] in, It is the set of all machines.

[0124] The state feature variables of the job ranking agent include at least:

[0125] machine Set of processing times for the above operations: ;

[0126] machine Set of relaxation values ​​for the above processing: ;

[0127] machine Set of expected delay rates for assignments: ;

[0128] machine Set of actual overdue rates for assignments: .

[0129] The aforementioned state characteristic variables, as inputs to the multi-agent scheduling unit, accurately reflect the dynamics and complexity of the scheduling environment.

[0130] Optionally, the calculation process for the above-mentioned expected delay rate / actual delay rate includes the following steps:

[0131] Define the criteria for determining whether a task is overdue;

[0132] For the job waiting queue of the current machine at the corresponding time, select any job in the queue and move it to the first position of the queue. Construct a new job sorting queue based on the adjusted job order.

[0133] For each job in the newly constructed job sorting queue, calculate the time taken for all jobs in the queue that are currently waiting to be processed, and determine the total time taken as the waiting time for that job.

[0134] By combining the waiting time of each task, the current time of calculation, and the deadline of the task, the expected delay rate or the actual delay rate can be calculated.

[0135] Specifically, the aforementioned expected delay rate and actual delay rate The calculation process includes the following steps (1) to (4):

[0136] Step 1: First, introduce the delay judgment function: ;

[0137] Step 2: In each decision, the machine Upload to the job queue A certain assignment Move to the top and build a new queue. ;

[0138] Step 3: For each job in the new queue The waiting time is:

[0139] ;

[0140] Step 4: Expected Delay Rate:

[0141] ;

[0142] in Indicates operation The expected processing time.

[0143] Actual delay rate:

[0144] .

[0145] S140. Embed the scheduling rules into the state feature variables to construct a unified state space containing variable relationships.

[0146] Optionally, the scope of core state characteristic variables in the unified state space should cover the above three types of core information: microgrid energy status, dynamic work queue information, and real-time status of production equipment.

[0147] Next, the scheduling rules are transformed into logical relationships between state characteristic variables: for example, the "low-energy machine selection" rule is associated with "job processing power characteristics" and "microgrid power supply characteristics", the "job priority scheduling" rule is associated with "job slack characteristics" and "expected delay rate characteristics", and the "carbon emission reduction" rule is associated with "machine power consumption characteristics" and "microgrid power supply gap characteristics".

[0148] Finally, by integrating the aforementioned state characteristic variables and the relationships between them, a unified state space covering energy, operations, and equipment coordination is formed, ensuring that the logical relationships between variables can support the strategy learning and scheduling decisions of multi-agent scheduling units.

[0149] S150. Based on the improved near-end policy optimization algorithm, the resource allocation agent and the job sorting agent are jointly trained using the state space, and a multi-dimensional composite reward mechanism associated with the weight coefficients is set to guide the multiple agents to perform optimization scheduling in a dynamic environment.

[0150] The improved proximal policy optimization algorithm (PPO) is based on a dual-network separated Actor-Critic structure, including a policy network (Actor) and a value network (Critic). The two networks form a feedback loop through a dominance function, ensuring the stability and reliability of the policy optimization path.

[0151] Policy Network (Actor): The resource allocation agent and the job sequencing agent each have their own independent policy network, corresponding to... Figure 2 In the routing agent network and the ranking agent network, the policy network input of the resource allocation agent is a differentiated state variable such as machine utilization rate and expected electricity purchase rate. The output machine selection probability distribution is used. The policy network of the job-ordering agent takes differentiated state variables such as job slackness and delay rate as input, and similarly outputs the job queue priority probability distribution using the same formula. In the formula... This is the fully connected layer mapping function for the actor's head.

[0152] Value Network (Critic): Two types of intelligent agents share the backbone layer of the value network, corresponding to... Figure 2 The middle route is the "shared network" part of the intelligent agent network and the ranking intelligent agent network, and its core is through Assess the current status The value assessment. In the formula... The unified state feature vector extracted for the shared network. This is a mapping function for the commenter's header, and its output is the long-term value assessment of the current state.

[0153] After the policy network outputs the scheduled action, the value network simultaneously evaluates the value of the next state after the action is executed. The two form a dynamic optimization closed loop through the advantage function, specifically:

[0154] State value assessment: executed through a value network Calculate the current state Valuation Simultaneously calculate the next state. Valuation ;

[0155] Time-Difference (TD) Objective and Error Calculation: By Calculate the temporal difference (TD) objective, where This is a discount factor (range [0,1]), used to balance immediate and long-term returns; then through... (as attached) Figure 2 (As shown in the TD error logic) Calculate the single-step TD error;

[0156] Dominance function estimation: Based on multi-step TD error, through Calculate the cumulative advantage of discount (GAE) to balance bias and variance, making action-reward evaluation more accurate;

[0157] Strategy loss calculation: using Limit the policy update range to avoid performance fluctuations. (In the formula...) The probability ratio between the old and new strategies. This is the shearing hyperparameter (default value is 0.2). This is the estimated value of the dominance function;

[0158] Value network loss calculation: by (or ) Calculate the value network error to optimize the accuracy of value network state assessment.

[0159] Loss function construction: Taking into account policy loss, value network loss, and entropy regularization, the final loss function is as follows:

[0160] ;

[0161] in, For value network loss, The entropy of the strategy is used to encourage exploration. , These are the weighting coefficients.

[0162] Call Attachment Figure 2 The parallel trajectory data stored in the intermediate experience buffer is used to perform gradient backpropagation based on the total loss function, and the policy network parameters of the resource allocation agent and the job ranking agent are updated synchronously, as well as the backbone layer parameters of the shared value network, to ensure the decision-making coordination and consistency of the two types of agents.

[0163] Repeat the above steps until the policy network converges, then stop training and save the optimal model parameters.

[0164] Optionally, based on the improved proximal policy optimization algorithm described above, the resource allocation agent and the job ranking agent are jointly trained using the state space, including the following steps:

[0165] The differentiated state variables in the state space are respectively input into the resource allocation agent and the job scheduling agent (e.g., the resource allocation agent inputs machine utilization rate and electricity purchase rate, and the job scheduling agent inputs job slack and delay rate), and the collaborative scheduling action is obtained through inference and sampling by their respective policy networks.

[0166] The scheduling action is sent to the flexible workshop environment, and the status data and instant rewards for the next moment are collected. The instant rewards include resource allocation instant rewards and job sorting instant rewards.

[0167] Based on the improved near-end policy optimization algorithm, combined with the state data and immediate reward, the policy update amplitude is limited by the shearing mechanism, the policy diversity is adjusted by adding an entropy term to the policy loss, and the error is calculated by multi-step temporal difference, and the action advantage estimate is accumulated to correct the policy update direction.

[0168] By combining the trajectory data collected in parallel, the policy network and value network parameters of the resource allocation agent and the job sorting agent are updated synchronously.

[0169] Repeat the above process until the policy network converges (for example, set the maximum number of training rounds to 10,000 to avoid overtraining) to complete the joint training.

[0170] Optionally, after training, the performance of the multi-agent PPO learning model is further evaluated, and the trained agent policy model is integrated into the control platform or edge device of the manufacturing system to realize the autonomous scheduling function of the workshop. For example, in a constructed simulation environment, factors such as dynamic arrival of workshop operations, microgrid power supply fluctuations, and time-varying electricity prices are simulated to train two agent models based on the PPO algorithm. By recording the changing trends of key indicators (including maximum completion time, average delay, total energy consumption, and carbon emissions), the optimization performance of the scheduling model under the green manufacturing goal is evaluated.

[0171] The training process can be implemented through program instructions in a general-purpose computer, edge computing device, or embedded processor. This program is used to drive the agent to complete operations such as state input, action output, and policy update during the scheduling process, so as to achieve continuous intelligent optimization scheduling.

[0172] The embodiments of the present invention achieve the following beneficial effects:

[0173] (1) This invention innovatively introduces the near-end strategy optimization (PPO) algorithm into the job scheduling and machine allocation process. By dividing the resource allocation agent and the job scheduling agent and giving them differentiated state information and optimization objectives, it can achieve coordinated control of reducing carbon emissions and energy consumption under dynamic job arrival and energy supply fluctuation conditions, thereby meeting the green transformation needs of manufacturing systems under the "dual carbon" background.

[0174] (2) Compared with traditional scheduling methods that rely on heuristic scheduling rules or static modeling, this invention fully leverages the advantages of combining policy iteration and value assessment in reinforcement learning to continuously optimize decision-making strategies and gradually learn low-energy, low-emission, and timely scheduling behaviors. By constructing a refined reward function system, the ranking agent is guided to minimize delays and completion times, thereby reducing energy consumption and carbon emission intensity and significantly improving the environmental adaptability and global optimization capability of the overall scheduling strategy.

[0175] (3) This invention achieves an effective trade-off between key performance indicators such as maximum completion time, operation delay, carbon emissions and total energy consumption through multi-objective optimization. It is applicable to various flexible work workshop scheduling scenarios, and is particularly suitable for the supply of new energy to the workshop in the microgrid industrial environment, as well as flexible manufacturing scenarios where the load fluctuates drastically inside the factory workshop. It can significantly reduce manual intervention, enhance the manufacturing system's ability to respond to real-time dynamic conditions, and help enterprises achieve the goal of cost reduction, efficiency improvement and green manufacturing synergistic development.

[0176] Example 2

[0177] Embodiment 2 of this invention focuses on improving algorithm performance in flexible job shop scheduling scenarios. Addressing the needs of multi-agent collaborative scheduling, it provides a core optimization method for the near-end strategy optimization (PPO) algorithm, adaptable to microgrid energy fluctuations and dynamic operational disturbances. Specifically, it includes the following steps:

[0178] Step (21) Action selection mechanism: at each scheduling time The agent, based on the current state Input Policy Network Select scheduling actions in a probabilistic manner .

[0179] Step (22) Policy update method: The ClippedPPO loss function is used to constrain the magnitude of each policy update, preventing drastic policy changes from causing performance instability. The optimization objective is as follows:

[0180] ;

[0181] in, For the current strategy Compared to the old strategy The ratio, This is the shear range hyperparameter. This is the estimated value of the advantage function.

[0182] Step (23) Advantage function estimation: To improve the stability of policy updates, the generalized advantage estimation (GAE) method is introduced, combined with immediate rewards. and state value Calculate the cumulative discount advantage over multiple steps :

[0183] ;

[0184] in, This represents a discount factor used to decay future rewards in order to balance immediate and long-term gains.

[0185] Step (24) Loss Function Construction: Taking into account the policy loss, value function loss, and entropy regularization term, the final loss function is as follows:

[0186] ;

[0187] in, For the error of the state value function, The entropy of the strategy is used to encourage exploration. , These are the weighting coefficients.

[0188] It is important to emphasize that, in order to guide different scheduling agents to achieve multi-objective optimization in the production scheduling process, real-time reward functions were designed for resource allocation agents and job sorting agents respectively, to incentivize them to make reasonable trade-offs between objectives such as energy consumption control, scheduling efficiency and resource balance.

[0189] Specifically, the immediate reward function of the resource allocation agent. The composition is as follows:

[0190] ;

[0191] in, The electricity price incentive term, used to incentivize agents to prioritize operations performed during periods of low electricity prices, is calculated as follows:

[0192] ;

[0193] in, This indicates the machine currently selected by the resource allocation agent. In time The electricity purchase ratio.

[0194] This is an energy efficiency optimization term, used to encourage resource allocation agents to select machines with relatively low processing energy consumption. Its definition is as follows:

[0195] ;

[0196] in, , This indicates the energy consumption of the workpiece on the selected machine. This represents the average energy consumption across all candidate machines.

[0197] This is a resource balancing term used to encourage balanced overall resource utilization. If the current scheduling action leads to a decrease in average utilization, a negative reward is given.

[0198] ;

[0199] in, and The average machine utilization rates before and after scheduling are as follows:

[0200] ;

[0201] Optionally, the immediate reward function of the task ranking agent described above consists of the following four parts:

[0202] ;

[0203] in, The relaxation guide is used to encourage prioritizing tasks with the lowest relaxation level. If the task... Normalized relaxation A reward is given if the difference between the minimum relaxation and the set threshold is less than 0.1.

[0204] ;

[0205] This is a processing time guideline used to indicate a preference for selecting jobs with shorter processing times within the relaxation range.

[0206] ;

[0207] in, This is the normalized processing time.

[0208] Expected delay interference items :

[0209] ;

[0210] Actual delay interference items :

[0211] ;

[0212] By constructing the aforementioned instant reward mechanism, the multi-agent reinforcement learning system in this invention can achieve multi-objective integrated scheduling of energy consumption control, scheduling balance, resource optimization and production line delay control, guiding the agent strategy to evolve towards a green and efficient direction.

[0213] Finally, the training effect of the multi-agent PPO learning model is evaluated, and the trained agent policy model is integrated into the control platform or edge device of the manufacturing system to realize the workshop autonomous scheduling function.

[0214] Compared to traditional rule-based scheduling systems, the PPO multi-agent learning framework constructed in this invention does not require explicit design of scheduling rules. Instead, it learns the dynamic optimal strategy through autonomous interaction and gradient optimization, exhibiting strong adaptability and online response capabilities.

[0215] Furthermore, the PPO-based intelligent agent strategy model proposed in this invention supports deployment in actual production systems, enabling the automation and intelligentization of green scheduling tasks. The specific implementation process includes the following steps:

[0216] Step 1: Hardware platform selection. Choose a suitable deployment platform based on the complexity of the task, including but not limited to the following:

[0217] Edge servers: support GPU acceleration and are suitable for large-scale model and multi-objective optimization scheduling;

[0218] Embedded terminal devices, such as Raspberry Pi and Jetson Nano, are suitable for lightweight scheduling tasks;

[0219] Industrial Controllers (PLCs): Suitable for actual factory control scenarios, they combine industrial communication protocols to issue action commands.

[0220] The second step is state awareness and data acquisition. Through the MES system, sensor network and energy consumption monitoring system, the state information required for scheduling is collected in real time, including work queues, processing time, renewable energy supply capacity, electricity price distribution, machine load, etc., and converted into input features for PPO model inference.

[0221] Step 3: Agent Action Decision-Making and Scheduling Execution

[0222] Status input: The system collects the current workshop status at each scheduling time step;

[0223] Action output: The PPO policy network generates scheduling actions, including job routing decisions and job sorting priorities;

[0224] Action execution: Map scheduling actions to equipment control commands, and control the equipment to perform specific operations through the MES or PLC system.

[0225] Step 4: Strategy fine-tuning and online learning mechanism. To enhance the system's adaptability, an online learning module can be introduced to continuously collect scheduling experience and periodically update the agent model without affecting scheduling execution, thereby improving the model's generalization ability to new scenarios and tasks.

[0226] Step 5: System integration testing and verification. After the model deployment is completed, multiple rounds of testing are conducted to verify the system's stability and optimization capabilities.

[0227] Simulation testing: The convergence and scheduling effect of the model were tested under different workshop sizes and different energy consumption constraints;

[0228] On-site trial operation: Run the scheduling model in an actual factory to test its responsiveness to real-time conditions and the effectiveness of its decisions.

[0229] The embodiments of the present invention achieve the following beneficial effects:

[0230] (1) Compared with traditional rule-based scheduling systems, the PPO multi-agent learning framework constructed in this invention does not require explicit design of scheduling rules, but learns the dynamic optimal strategy through autonomous interaction and gradient optimization, and has strong adaptability and online response capability.

[0231] (2) By using the clipping mechanism of the ClippedPPO loss function, the performance oscillation caused by the excessive single update amplitude of the strategy in traditional reinforcement learning is avoided, ensuring that the multi-agent can still iterate smoothly in the scenario of microgrid energy fluctuation and dynamic operation disturbance.

[0232] (3) Introducing the Generalized Advantage Estimation (GAE) method, combined with multi-step time-series difference error calculation to calculate cumulative discount advantage, balances the high variance of traditional single-step TD estimation and the high bias of Monte Carlo estimation, making the resource allocation and job ranking agent more accurate in evaluating action-reward, and improving the policy convergence speed by 20%-30%.

[0233] (4) Add an entropy regularization term to the total loss function and guide the strategy to maintain action diversity through weight coefficients to avoid the agent from falling into local optimal scheduling schemes too early. Especially when there are many machines in the workshop and the types of work are complex, the probability of exploring low energy consumption and low delay scheduling strategies is significantly improved.

[0234] Example 3

[0235] Embodiment 3 of the present invention illustrates the complete algorithm flow and experimental comparison results of the multi-objective dynamic flexible job shop scheduling optimization method proposed in this invention, specifically including the following steps:

[0236] S1: Enter the main loop and set the iteration termination condition, that is, when the number of training rounds exceeds the maximum number of iteration rounds, exit the main loop.

[0237] S2: Initialize the environmental parameters of the flexible workshop, including the workpiece set, process set, machine set, and processing time of each process on different machines. With power Etc.; Construct a multi-agent scheduling unit system environment, and define the state characteristic variables and their corresponding state spaces for the resource allocation agent and the job sorting agent, respectively.

[0238] S3: At each scheduling decision point, update the state information of the resource allocation agent and the job sequencing agent.

[0239] S4: The resource allocation agent uses PPO to make decisions about the current state and outputs the index of the selected target machine. This enables the assignment of machine tools for each process.

[0240] S5: The job scheduling agent selects a job from the task queue for scheduling based on the PPO. It records the job's start time, estimated completion time, and machine information used.

[0241] S6: After scheduling is completed, update the system environment status, including the utilization rate of each machine, job completion status, changes in the task queue, etc.

[0242] S7: Calculate the immediate reward functions for the resource allocation agent and the job ranking agent, respectively.

[0243] S8: Store the current scheduling status, action, reward, and other samples in the temporary storage area for subsequent training.

[0244] S9: Update the system time and determine whether all workpiece processes have been scheduled to be completed. If not, return to S2 for the next round of decision-making.

[0245] S10: If all workpiece process scheduling is completed, it indicates that one round of simulation is over. Record the start time, end time, processing machine and scheduling path of each workpiece.

[0246] S11: Calculate the maximum completion time, average delay, total carbon emissions, and total energy consumption in each round, and update the data.

[0247] S12: Determine if the training round has ended. If not, reinitialize the environment and return to S1 to start the next round.

[0248] S13: Use the improved PPO algorithm to train the policy networks of the resource allocation agent and the job ranking agent.

[0249] S14: Calculate the loss functions of the resource allocation agent and the job ranking agent respectively, perform gradient backpropagation, and update the policy network parameters.

[0250] S15: Use a soft update mechanism to update the target network parameters and improve policy stability.

[0251] S16: After the iterative training is completed, output the final model performance metrics.

[0252] S17: Draw a Gantt chart based on the final scheduling results. The horizontal axis represents time, the vertical axis represents machine number, each bar represents the job processing process, and the job number and operation power consumption are marked.

[0253] In summary, the model of this invention aims to reduce job delays and energy consumption. It optimizes the scheduling strategy through a multi-agent PPO reinforcement learning method, thereby achieving green and efficient production scheduling.

[0254] Table 1 below shows the simulation results of the proposed method, specifically the combined effect (RA_SA) of the resource allocation agent (RA) and job scheduling agent (SA) on four indicators. Compared with traditional rules, the proposed method shows superior performance in both delay rate control and energy consumption optimization, demonstrating good practical value. Meanwhile, Figures 3(a)-3(i) show the Pareto solution distributions of different methods under the three objectives of completion time, delay rate, carbon emissions, and energy consumption, which intuitively reflects the advantages of the proposed method in multi-objective optimization. Its solution set is superior to the comparative methods in both distribution and superiority.

[0255] Table 1

[0256]

[0257] According to embodiments of the present invention, the following effects are achieved:

[0258] (1) This invention innovatively introduces the near-end strategy optimization (PPO) algorithm into the job scheduling and machine allocation process. By dividing the resource allocation agent and the job scheduling agent and giving them differentiated state information and optimization objectives, it can achieve coordinated control of reducing carbon emissions and energy consumption under dynamic job arrival and energy supply fluctuation conditions, thereby meeting the green transformation needs of manufacturing systems under the "dual carbon" background.

[0259] (2) Compared with traditional scheduling methods that rely on heuristic scheduling rules or static modeling, this invention fully leverages the advantages of combining policy iteration and value assessment in reinforcement learning to continuously optimize decision-making strategies and gradually learn low-energy, low-emission, and timely scheduling behaviors. By constructing a refined reward function system, the ranking agent is guided to minimize delays and completion times, thereby reducing energy consumption and carbon emission intensity and significantly improving the environmental adaptability and global optimization capability of the overall scheduling strategy.

[0260] (3) This invention achieves an effective trade-off between key performance indicators such as maximum completion time, operation delay, carbon emissions and total energy consumption through multi-objective optimization. It is applicable to various flexible work workshop scheduling scenarios, and is particularly suitable for the supply of new energy to the workshop in the microgrid industrial environment, as well as flexible manufacturing scenarios where the load fluctuates drastically inside the factory workshop. It can significantly reduce manual intervention, enhance the manufacturing system's ability to respond to real-time dynamic conditions, and help enterprises achieve the goal of cost reduction, efficiency improvement and green manufacturing synergistic development.

[0261] Example 4

[0262] Figure 4This is a schematic diagram of a multi-objective dynamic flexible workshop scheduling optimization device provided in Embodiment 4 of the present invention. The technical solution of this embodiment is applicable to various flexible workshop scheduling scenarios. The device can be implemented by software and / or hardware and is generally integrated into any electronic device with network communication capabilities, including but not limited to: servers, computers, personal digital assistants, etc. Figure 4 As shown, the multi-objective dynamic flexible job shop scheduling optimization device provided in this embodiment may include: a job shop scheduling model construction module 410, a deep reinforcement learning objective function construction module 420, a state feature variable definition module 430, a unified state space construction module 440, and a multi-agent scheduling unit joint training module 450; wherein,

[0263] The workshop scheduling model construction module 410 is used to construct a flexible work workshop scheduling model that includes a multi-agent scheduling unit, wherein the multi-agent scheduling unit includes a resource allocation agent and a job sequencing agent.

[0264] The deep reinforcement learning objective function construction module 420 is used to construct the deep reinforcement learning objective function of the multi-agent scheduling unit based on the production scheduling rules, and to convert the scheduling rules into its constraints and weight coefficients.

[0265] The state feature variable definition module 430 is used to acquire the real-time status of microgrid energy status, dynamic work queue and production equipment, and define the state feature variables of flexible work workshop in combination with the constraints.

[0266] The unified state space construction module 440 is used to embed the scheduling rules into the state feature variables and construct a unified state space containing variable relationships.

[0267] The multi-agent scheduling unit joint training module 450 is used to jointly train the resource allocation agent and the job sorting agent based on the improved near-end policy optimization algorithm using the state space, and to set a multi-dimensional composite reward mechanism associated with the weight coefficients to guide the multi-agent to perform optimization scheduling in a dynamic environment.

[0268] Based on the above embodiments, optionally, the multi-agent scheduling unit joint training module is specifically used for:

[0269] The differentiated state variables in the state space are respectively input into the resource allocation agent and the job sorting agent, and the collaborative scheduling action is obtained through reasoning and sampling by their respective policy networks;

[0270] The scheduling action is sent to the flexible workshop environment, and the status data and instant rewards for the next moment are collected. The instant rewards include resource allocation instant rewards and job sorting instant rewards.

[0271] Based on the improved near-end policy optimization algorithm, combined with the state data and immediate reward, the policy update amplitude is limited by the shearing mechanism, the policy diversity is adjusted by adding an entropy term to the policy loss, and the error is calculated by multi-step temporal difference, and the action advantage estimate is accumulated to correct the policy update direction.

[0272] By combining the trajectory data collected in parallel, the policy network and value network parameters of the resource allocation agent and the job sorting agent are updated synchronously.

[0273] Repeat the above process until the policy network converges, completing the joint training.

[0274] Optionally, based on the above embodiments, the device further includes an operation evaluation module, specifically used for:

[0275] Run the trained resource allocation agent and job sorting agent in a simulation environment and collect performance index data during the process.

[0276] The optimization effect of the flexible job shop scheduling model is evaluated based on the performance index data.

[0277] The multi-objective dynamic flexible workshop scheduling optimization device provided in the embodiments of the present invention can execute the multi-objective dynamic flexible workshop scheduling optimization method provided in any of the embodiments of the present invention, and has the corresponding functions and beneficial effects of executing the method. For details, please refer to the relevant operations of the multi-objective dynamic flexible workshop scheduling optimization method in the foregoing embodiments.

[0278] Example 5

[0279] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0280] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0281] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0282] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as multi-objective dynamic flexible job shop scheduling optimization methods.

[0283] In some embodiments, the multi-objective dynamic flexible job shop scheduling optimization method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multi-objective dynamic flexible job shop scheduling optimization method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the multi-objective dynamic flexible job shop scheduling optimization method by any other suitable means (e.g., by means of firmware).

[0284] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0285] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0286] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0287] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0288] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0289] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0290] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0291] The above embodiments are merely illustrative examples and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multi-objective dynamic flexible job shop scheduling optimization method, characterized in that, include: A flexible job shop scheduling model is constructed, which includes a multi-agent scheduling unit, comprising a resource allocation agent and a job sequencing agent. The deep reinforcement learning objective function of the multi-agent scheduling unit is constructed based on the production scheduling rules, and the scheduling rules are transformed into its constraints and weight coefficients. The real-time status of microgrid energy, dynamic work queues, and production equipment is obtained, and the state characteristic variables of the flexible workshop are defined in combination with the constraints. The scheduling rules are embedded into the state feature variables to construct a unified state space containing variable relationships; Based on the improved near-end policy optimization algorithm, the resource allocation agent and the job sorting agent are jointly trained using the state space, and a multi-dimensional composite reward mechanism associated with the weight coefficients is set to guide the multiple agents to perform optimization scheduling in a dynamic environment.

2. The method according to claim 1, characterized in that, The deep reinforcement learning objective function of the multi-agent scheduling unit is constructed using the following formula: ； ； in, : Indicates minimizing the job completion time; : indicates homework Completion time; : Represents the set of all jobs; : Indicates minimizing the average delay of a task; : Indicates the number of assignments; Indicate homework The deadline; : Indicates minimizing carbon emissions during the production process; : Indicates the number of machines; : Indicates time machine power, This represents the power that the microgrid can provide at time t; : This indicates minimizing the total energy consumption of the production process.

3. The method according to claim 2, characterized in that, The state characteristic variables of the flexible workshop include at least a combination of the following variables: Total number of assignments Total number of machines ,Operation operands ,Operation The One operation ,operate In the machine On the processing time ,operate In the machine On the processing power ,Operation Arrival time and completion time Waiting at the machine Up-processing job operation set ,machine Expected idle time Photovoltaic system power generation Power generation of wind power systems Residual energy of energy storage systems Machine utilization rate Expected electricity purchase rate , work slack ,machine Expected delay rate of the above work and machines Actual delay rate of the above work .

4. The method according to claim 3, characterized in that, The expected delay rate Or actual delay rate The calculation methods include: Define the criteria for determining whether a task is overdue; For the job waiting queue of the current machine at the corresponding time, select any job in the queue and move it to the first position of the queue. Construct a new job sorting queue based on the adjusted job order. For each job in the newly constructed job sorting queue, calculate the time taken for all jobs in the queue that are currently waiting to be processed, and determine the total time taken as the waiting time for that job. By combining the waiting time of each task, the current time of calculation, and the deadline of the task, the expected delay rate or the actual delay rate can be calculated.

5. The method according to claim 1, characterized in that, The method based on the improved proximal policy optimization algorithm, which utilizes the state space to jointly train the resource allocation agent and the job ranking agent, includes: The differentiated state variables in the state space are respectively input into the resource allocation agent and the job sorting agent, and the collaborative scheduling action is obtained through reasoning and sampling by their respective policy networks; The scheduling action is sent to the flexible workshop environment, and the status data and instant rewards for the next moment are collected. The instant rewards include resource allocation instant rewards and job sorting instant rewards. Based on the improved near-end policy optimization algorithm, combined with the state data and immediate reward, the policy update amplitude is limited by the shearing mechanism, the policy diversity is adjusted by adding an entropy term to the policy loss, and the error is calculated by multi-step temporal difference, and the action advantage estimate is accumulated to correct the policy update direction. By combining the trajectory data collected in parallel, the policy network and value network parameters of the resource allocation agent and the job sorting agent are updated synchronously. Repeat the above process until the policy network converges, completing the joint training.

6. The method according to claim 5, characterized in that, The improved proximal policy optimization algorithm's policy network objective function for: ； ； in, This represents the ratio of the probabilities of the new and old strategies; This is the estimated value of the dominance function; This is a hyperparameter representing the shear range; The parameters of the current and old policy networks are... For TD error; These are GAE parameters used to adjust bias and variance. This represents a discount factor used to decay future rewards in order to balance immediate and long-term gains.

7. The method according to claim 5, characterized in that, The total loss function used to update the policy network parameters and value network parameters in the improved proximal policy optimization algorithm is: ； in, For the error of the state value function, The entropy of the strategy is used to encourage exploration. , These are the weighting coefficients.

8. The method according to claim 7, characterized in that, The instant reward function of the resource allocation agent Instant reward function for task ranking agents They are respectively: ； ； in, As a guideline for electricity prices, For energy efficiency optimization items, For resource balance, For relaxation guide terms, For processing time guidance, This is the term for suppressing the delay rate disturbance. This refers to the expected delays and interference.

9. The method according to claim 1, characterized in that, The method further includes: Run the trained resource allocation agent and job sorting agent in a simulation environment and collect performance index data during the process. The optimization effect of the flexible job shop scheduling model is evaluated based on the performance index data.

10. A multi-objective dynamic flexible job shop scheduling optimization device, characterized in that, include: The workshop scheduling model construction module is used to construct a flexible workshop scheduling model that includes a multi-agent scheduling unit, wherein the multi-agent scheduling unit includes a resource allocation agent and a job sequencing agent; The deep reinforcement learning objective function construction module is used to construct the deep reinforcement learning objective function of the multi-agent scheduling unit based on the production scheduling rules, and to transform the scheduling rules into its constraints and weight coefficients; The state feature variable definition module is used to obtain the real-time status of microgrid energy status, dynamic work queue and production equipment, and define the state feature variables of flexible work workshop in combination with the constraints. A unified state space construction module is used to embed the scheduling rules into the state feature variables to construct a unified state space containing variable relationships. The multi-agent scheduling unit joint training module is used to jointly train the resource allocation agent and the job sorting agent based on the improved near-end policy optimization algorithm using the state space, and to set up a multi-dimensional composite reward mechanism associated with the weight coefficients to guide the multi-agent to perform optimization scheduling in a dynamic environment.

Images