Flexible job-shop energy-saving scheduling method oriented to excitation-driven demand response

Abstract

The invention relates to the technical field of industrial scheduling, and discloses an excitation-driven demand response-oriented flexible job shop energy-saving scheduling method, which comprises the following steps of: representing an energy-saving scheduling state of a flexible job shop based on an improved heterogeneous disjunction graph, and constructing a graph reinforcement learning model; training the graph reinforcement learning model by adopting a near-end strategy optimization algorithm; and outputting a corresponding scheduling scheme according to the production task and the day-ahead peak clipping invitation based on the trained graph reinforcement learning model. The method effectively coordinates and optimizes the three targets of the completion period, the total energy consumption and the incentive return, provides technical support for power grid operation personnel to design incentive policies, and helps to encourage more industrial loads to participate in power system operation.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial scheduling, and in particular to a flexible job shop energy-saving scheduling method oriented to incentive-driven demand response. Background Art

[0002] Electricity is one of the most commonly used energy sources in the manufacturing industry. With the increase in the penetration rate of renewable energy and the advent of the "producer-seller" era, the power grid is facing increasing challenges in balancing power supply and demand. In order to alleviate the pressure of power supply and demand imbalance from the load side, the optimization focus of energy-aware scheduling has gradually shifted from consumption level to consumption mode, and demand response (DR) is the catalyst for this process.

[0003] DR is usually divided into two categories: price-based plans (PBPs) and incentive-based plans (IBPs). For energy storage systems (EES), the introduction of PBPs requires considering electricity prices at different times when making scheduling decisions, and indirectly achieving the purpose of peak load reduction by reducing electricity costs. Correspondingly, the scheduling scheme under IBPs needs to be adjusted autonomously according to the peak load reduction request of the power grid (usually during peak power consumption or when there are problems with power grid reliability), so as to obtain corresponding economic returns from the load reduction level. Therefore, PBPs-EES can be regarded as a user-centric model. When or to what extent to take advantage of lower electricity prices depends entirely on the manufacturer itself, and the impact of this model on grid operations is difficult to accurately predict. In contrast, IBPs-EES is a grid-centric model.

[0004] Under the guidance of the DR signal formulated by the power grid, demand-side resources are used as an additional means of power balancing, and the reliability and resilience of the power grid are improved by using source-load interaction. Statistics from the power market also show that IBPs (93%) are more effective than PBPs (7%) in reducing peak loads.

[0005] DRPs-EES are important for sustainable industrial production operations, but they still have the following limitations. On the one hand, existing flexible job shop energy-saving scheduling mainly considers the coordination of production and electricity under PBPs, especially the time-of-use electricity price (TOU) scheme. The characterization of the power system operation characteristics in these studies is rough (relatively fixed time windows), and when multiple users adopt TOU, it will have negative effects, such as the occurrence of "rebound peaks". IBPs can more finely depict the real-time operation status of the power grid (dynamic time windows), providing different opportunities for balancing supply and demand, so they need to be focused on when formulating scheduling schemes. On the other hand, current studies combining IBPs with production scheduling either determine the optimal load reduction scheme under given incentives based on operating costs, or manually divide the load reduction levels and study the changing trends of production and energy targets under different peak load constraints. For flexible job shops, subtle adjustments to machine allocation schemes may lead to huge changes in completion time, total energy consumption (TEC), and total effective power consumption during demand response requests. This changing relationship is difficult to solve and characterize using manual division of constraints and linear relationships. Therefore, it is worth further exploring the potential of industrial loads as DR resources based on multi-objective Pareto analysis.

[0006] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0007] In view of the problems in the related art, the present invention proposes a flexible job shop energy-saving scheduling method for incentive-driven demand response to overcome the above-mentioned technical problems existing in the existing related art.

[0008] To this end, the specific technical solution adopted by the present invention is as follows:

[0009] A flexible job shop energy-saving scheduling method for incentive-driven demand response includes the following steps:

[0010] S1. Based on the improved heterogeneous disjunctive graph, the energy-saving scheduling state of the flexible job shop is represented, and a graph reinforcement learning model is constructed;

[0011] S2, using the proximal strategy optimization algorithm to train the graph reinforcement learning model;

[0012] S3: Based on the trained graph reinforcement learning model, output the corresponding scheduling plan according to the production tasks and the day-ahead peak shaving invitation;

[0013] The day-ahead peak shaving invitation includes a response time, a reduction requirement and an incentive rate; the reduction requirement represents the difference between the historical power load measured before the subscription incentive-driven demand response and the reduction threshold of this invitation.

[0014] Furthermore, the method of representing the energy-saving scheduling state of the flexible job shop based on the improved heterogeneous disjunctive graph and constructing a graph reinforcement learning model includes the following steps:

[0015] S11. Construct a Markov decision process module that uses an improved heterogeneous disjunctive graph to describe the scheduling state, and design the reward function of the Markov decision process module;

[0016] S12. Establishing a graph neural network scheduler for extracting embedded information and selecting scheduling actions, wherein the graph neural network scheduler includes a representation learning network and a policy learning network;

[0017] S13. Based on the representation learning network, the improved heterogeneous disjunctive graph embedding is realized, and the scheduling action is selected according to the strategy learning network and the embedding result;

[0018] S14. Based on the Markov decision process module and the graph neural network scheduler, a graph reinforcement learning model is constructed.

[0019] Furthermore, the graph structure of the improved heterogeneous disjunctive graph is expressed as:

[0020]

[0021] In the formula, Represents a set of process nodes; Represents a set of machine nodes; represents the set of connected arcs; express Arc Set;

[0022] The original features of the improved heterogeneous disjunctive graph include process node original features, machine node original features, disjunctive arc original features and global features.

[0023] Furthermore, the reward function of the Markov decision process module is designed to include:

[0024] Define reward functions with multiple levels and objectives;

[0025] Based on the multi-level factors, a double-layer reward of trajectory and action is defined, where the trajectory reward is the actual value of the optimization target and the action reward is the difference between the expected optimization targets in adjacent time steps;

[0026] Based on multi-objective factors, the energy-saving scheduling problem of flexible job shop for incentive-driven response is decomposed into N scalar optimization sub-problems, and different weight vectors λ are used to optimize the energy-saving scheduling problem. b , b∈[1, 2, …, N] for identification;

[0027] Set a weighted hybrid multi-objective aggregate reward function to determine the reward for each sub-problem.

[0028] Furthermore, the weighted hybrid multi-objective aggregate reward function is set to determine the reward for each sub-problem including:

[0029] Define the action reward r using Chebyshev method act , based on the weighted sum method to model the trajectory reward r tra , for the b-th sub-problem, the expression of the reward function at time step t is:

[0030]

[0031] Where ω1 and ω2 represent the weights of trajectory reward and action reward respectively; and They represent the trajectory reward and action reward of the b-th subproblem at time step t respectively; TE t (λ b ) represents the Chebyshev value of the b-th subproblem at time step t; objs represents the number of optimization objectives; represents the weight of the g-th objective of the b-th subproblem; f g (t) represents the predicted value of the g-th target at time step; represents the reference value of the g-th target; A t represents the action space set at time step t; Represents a set of process nodes.

[0032] Furthermore, the embedding of the improved heterogeneous disjunctive graph based on the representation learning network includes:

[0033] Based on meta-path-edge attention, machine nodes are embedded; based on heterogeneous attention, process nodes are embedded;

[0034] Use a multi-layer perceptron to extract the graph-level feature ψ, and add the calculation result to the node-level embedding result to get the complete graph embedding;

[0035] Among them, the expression of complete graph embedding is:

[0036]

[0037] In the formula, represents the complete graph embedding at time step t; O ij Indicates a process node; Represents the process node O after Z iterations ij The final embedding of ;|| represents the cascade operator; Represents a set of machine nodes; M k Represents a machine node; Represents the machine node M after Z iterations kThe final embedding of ; MLP represents multi-layer perceptron; θ represents the weights and bias parameters of the connections between neurons in MLP.

[0038] Furthermore, the embedding of machine nodes based on meta-path-edge attention includes:

[0039] In terms of meta-path neighbors, a path consisting of a disjunctive arc in the machine processing capability subgraph and two different machine nodes connected by it and the same process node is defined as a meta-path; at time step t, the path connected to the machine node M k The set of all machine nodes that have a meta-path connection relationship is called its meta-path neighbors. Use graph attention to embed the neighbor machine nodes based on the meta-path, and get the embedded machine node M k The eigenvector of The expression is:

[0040]

[0041] Where, ν k Represents the machine node M before embedding k The feature vector of MetapathGAT(v k ) represents the meta-path attention embedding method; Θ1 represents the learnable weight matrix used to transform machine node features to the specified embedding dimension; c represents the meta-path neighbor An element in a set; α ck is the attention coefficient of different neighbor nodes; vc represents the feature vector of c element; LeakyReLU represents the activation function; softmax represents the normalization of all meta-path neighbors; a0 represents the learnable weight matrix; T represents the transpose operator; Θ2 represents the learnable weight matrix for transforming process node features; ‖ represents the cascade operator; μ ij Represents process node O ij The feature vector of , where i represents the workpiece, j represents the jth process of the i-th workpiece; φ(ck) represents the simultaneous c and M k A set of process nodes that have a meta-path connection relationship;

[0042] In terms of edge feature processing, the disjunctive arc connecting the process and the machine node carries the processing time and power information of the process machine pair; the edge graph attention network EdgeGAT is used to integrate the edge features into the message passing and neighborhood aggregation process, where the embedding process of the graph topology neighbors with edge features is expressed as:

[0043]

[0044] α ijk=softmax(LeakyReLU(a T [Θ1·v k ||Θ2·μ ij ||Θ3·δ ijk ]))

[0045] In the formula, represents the embedding result of machine nodes in terms of edge features; α ijk Represents the attention coefficient of different neighbor process nodes; Indicates the current time step and machine node M k The set of adjacent process nodes; Θ3 represents the learnable weight matrix used to transform the edge feature dimension; δ ijk represents the edge eigenvector; a T represents the transpose of the learnable weight matrix representing edge attention;

[0046] The final embedding results of meta-paths and edges are expressed as:

[0047]

[0048] Where MLP represents a multi-layer perceptron; ELU represents the function used for hidden layer activation.

[0049] Furthermore, the embedding process node based on heterogeneous attention includes:

[0050] For process node O ij For example, the process constraint subgraph and the processing queue subgraph define five types of process neighbor nodes: the process node itself O ij , process front node O i，j-1 , process subsequent node O i，j+1 , queue front node O i′，j′ and queue post-order node O i″，j″ , the corresponding encoding results are: the encoding result of the process node itself f1 = MLP θ (μ ij ), the encoding result of the previous node of the process f2 = MLP θ (μ i，j-1 ), the encoding result of the subsequent node of the process f3 = MLP θ (μ i′，j′+1 ), the encoding result f4 of the preceding node of the queue = MLP θ (μ i′，j′ ), the encoding result f5 of the queue's subsequent node = MLP θ (μ i″，j″ );

[0051] For the neighbor machine nodes defined by the processing capacity subgraph, the encoding formula is:

[0052]

[0053] The attention mechanism is used to aggregate the encoding results of each neighbor to obtain the final process node embedding, which is expressed as:

[0054]

[0055] In the formula, k represents the machine number of the process node; Indicates the current time step and process node O ij The set of adjacent machine nodes; α 1，x Indicates the attention coefficients of different types of machine neighbors; represents the learnable attention parameter matrix; f x Representing a collection Any element in , and f x ∈{f1∪f2∪f3∪f4∪f5∪f6}.

[0056] Furthermore, the selecting a scheduling action according to the strategy learning network and the embedding result includes:

[0057] Define a preference network for encoding the associated weight vector, where the preference network includes 1 linear layer and a ReLU activation layer, and the encoding process is represented by Λ b =ReLU(Linear(λ b )), where Λ b represents the final encoding result, Linear represents the linear layer, and λ b represents the weight of the b-th sub-problem;

[0058] The feature embedding and preference encoding are weighted and aggregated through matrix multiplication, and the results are input into the Actor and Critic networks for strategy learning. The input of the Actor network is:

[0059]

[0060] Input of Critic network:

[0061]

[0062] Based on the actions generated by the Actor network, the Critic network evaluates the action value and finally selects the scheduling action.

[0063] Furthermore, the training of the graph reinforcement learning model using the proximal strategy optimization algorithm includes the following steps:

[0064] Set the total number of training iterations, prepare a preset number of scheduling problem instances for each round of training, and process them in parallel;

[0065] Record the agent's state, actions taken, and rewards at each time step; use the proximal policy optimization algorithm to adjust the scheduler's parameters based on the collected data;

[0066] After a preset number of trainings, the model performance is evaluated using independent validation examples, and based on the validation results, new training examples are reloaded to refresh the model and avoid overfitting.

[0067] The beneficial effects of the present invention are:

[0068] 1) The present invention is used for the production scheduling problem of a flexible workshop operation workshop under incentive-driven demand response, selecting completion period, total energy consumption, and total effective power consumption during demand response request as three optimization objectives. The problem is converted into a Markov decision process in which the scheduling state is represented by an improved heterogeneous disjunctive graph and contains trajectory-action double-layer rewards. A graph neural network scheduler including a representation learning network and a policy learning network is established, and a proximal policy optimization algorithm is used to train the graph reinforcement learning model. The scheduling efficiency and the ability to respond to incentive-driven demand can be significantly improved, providing an efficient, flexible and highly generalized optimization method for the field of production scheduling.

[0069] 2) The present invention innovatively designs an improved heterogeneous disjunctive graph that integrates the inherent attributes of nodes, deeply revealing the intrinsic interaction mechanism of production and energy consumption behavior in the context of incentive-driven demand response; further, the present invention designs a graph neural network scheduler, which effectively captures and models the key features of the scheduling environment under incentive-driven demand response through machine node embedding based on meta-path-edge attention and process node embedding based on heterogeneous attention; the present invention not only optimizes the manufacturing process from the dual dimensions of production and energy efficiency, but also provides a practical technical solution for flexible job shops in realizing industrial-grade demand response, thereby promoting the development of intelligent manufacturing and energy management. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 is a flow chart of a flexible job shop energy-saving scheduling method for incentive-driven demand response according to an embodiment of the present invention;

[0072] Figure 2It is a schematic diagram of an improved heterogeneous disjunctive graph structure in a flexible job shop energy-saving scheduling method for incentive-driven demand response according to an embodiment of the present invention;

[0073] Figure 3 is a schematic diagram of a machine node embedding process in a flexible job shop energy-saving scheduling method for incentive-driven demand response according to an embodiment of the present invention;

[0074] Figure 4 is a schematic diagram of a process node embedding process in a flexible job shop energy-saving scheduling method for incentive-driven demand response according to an embodiment of the present invention;

[0075] Figure 5 is a schematic diagram of an approximate Pareto front obtained in scenario 1 of a flexible job shop energy-saving scheduling method for incentive-driven demand response according to an embodiment of the present invention;

[0076] Figure 6 It is a Gantt chart and a schematic diagram of estimated power consumption curve of a scenario 1 benchmark solution in a flexible job shop energy-saving scheduling method for incentive-driven demand response according to an embodiment of the present invention;

[0077] Figure 7 is a schematic diagram of an approximate Pareto front obtained in scenario 2 of a flexible job shop energy-saving scheduling method for incentive-driven demand response according to an embodiment of the present invention;

[0078] Figure 8 It is a Gantt chart and a schematic diagram of the estimated power consumption curve of a scenario 2 benchmark solution in a flexible job shop energy-saving scheduling method for incentive-driven demand response according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0080] According to an embodiment of the present invention, a flexible job shop energy-saving scheduling method for incentive-driven demand response is provided.

[0081] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1-Figure 8 As shown, the flexible job shop energy-saving scheduling method for incentive-driven demand response according to an embodiment of the present invention includes the following steps:

[0082] S1. Based on the improved heterogeneous disjunctive graph, the energy-saving scheduling state of the flexible job shop is represented, and a graph reinforcement learning model is constructed, including a Markov decision process and a graph neural network scheduler;

[0083] Among them, in this embodiment, a Markov decision process module is first constructed using an improved heterogeneous disjunctive graph to describe the scheduling state, and then a graph neural network scheduler is established to extract embedded information and select scheduling actions. Specifically, the energy-saving scheduling state of the flexible job shop is represented based on the improved heterogeneous disjunctive graph, and the graph reinforcement learning model is constructed, which includes the following steps:

[0084] S11. Construct a Markov decision process module that uses an improved heterogeneous disjunctive graph to describe the scheduling state (such as Figure 2 As shown, the improved heterogeneous disjunctive graph includes its graph structure and original features), and the reward function of the Markov decision process module is designed;

[0085] Specifically, a Markov decision process module is constructed to describe the scheduling state using an improved heterogeneous disjunctive graph, including:

[0086] Establish a Markov decision process module and use an improved heterogeneous disjunctive graph to describe the scheduling state; integrate process selection and machine allocation decisions into scheduling actions in the Markov decision process module; design the reward function of the Markov decision process module;

[0087] Specifically, the graph structure of the improved heterogeneous disjunctive graph is expressed as:

[0088]

[0089] In the formula, represents a set of process nodes, It includes all processing steps and virtual steps that represent the start and end of the processing steps. The processing time of the virtual steps is zero.

[0090] represents a set of machine nodes, It includes all processing machines in a shop environment;

[0091] represents the connection arc set, which contains the directed arcs of n paths formed by the workpiece from start to completion;

[0092] express The arc set, which contains undirected arcs connecting process nodes and machine nodes;

[0093] The original features of the improved heterogeneous disjunctive graph include the original features μ of the process nodes ij , the original feature of the machine node ν k , extract the original feature of the arc δijk and the global feature ψ.

[0094] The original features of the process node include: (1) Scheduling ID: ij The scheduled value is 1, otherwise it is 0; (2) The number of neighboring machine nodes (3) Number of remaining workpiece processes: J i The number of unscheduled processes; (4) Start time: Partial scheduling Estimated start time of processing (depending on the scheduling flag); (5) Completion time: Estimated completion time; (6) Remaining workload of the workpiece: J i The estimated total processing time of the unscheduled process; (7) Processing energy consumption: estimated processing energy consumption; (8) Number of neighbor nodes:

[0095] The original characteristics of the machine node include: (1) Release time: M k The time to complete all the processes assigned before t; (2) Standby time: the cumulative total standby time before t; (3) Standby energy consumption: the cumulative total standby energy consumption before t;

[0096] The original features of the disjunctive arc include: (1) Processing time: The processing time corresponding to the disjunctive arc t ijk ; (2) Processing power: The processing power corresponding to the extraction arc p ijk ; (3) Number of processes: The total number of processes in which all processing periods and performance periods intersect;

[0097] Global features include: (1) Number of successful incentives: The number of time windows that meet the load reduction conditions; (2) Average power: The average power of all time windows that meet the load reduction condition.

[0098] Specifically, the reward function of the Markov decision process module includes:

[0099] Develop a reward function that takes into account both multi-level and multi-objective factors;

[0100] Based on the multi-level factors, a double-layer reward of trajectory and action is defined, where the trajectory reward is the actual value of the optimization target, which is used to guide the global optimization direction, and the action reward is the difference between the expected optimization targets of adjacent time steps, which further evaluates the contribution of each action;

[0101] Based on multi-objective factors, the energy-saving scheduling problem of flexible job shop for incentive-driven response is decomposed into N scalar optimization sub-problems, and different weight vectors λ are used to optimize the energy-saving scheduling problem. b , b∈[1, 2, …, N] for identification;

[0102] Set a weighted hybrid multi-objective aggregate reward function to determine the reward for each sub-problem.

[0103] Specifically, the weighted hybrid multi-objective aggregate reward function is set to determine the reward for each sub-problem including:

[0104] The action reward r is defined using the Tchebycheff method act , based on the weighted sum method to model the trajectory reward r tra , for the b-th sub-problem, the expression of the reward function at time step t is:

[0105]

[0106] Where ω1 and ω2 represent the weights of trajectory reward and action reward respectively, and satisfy ω1+ω2=1ω1, ω2∈[0,1]; and They represent the trajectory reward and action reward of the b-th subproblem at time step t respectively; TE t (λ b ) represents the Chebyshev value of the b-th subproblem at time step t; objs represents the number of optimization targets (in this embodiment, objs = 3; represents the weight of the g-th objective of the b-th subproblem; f g (t) represents the predicted value of the g-th target at time step t; represents the reference value of the g-th target, that is, the minimum value of the target found during the training process; A t represents the action space set at time step t; Represents a set of process nodes.

[0107] S12. Establishing a graph neural network scheduler for extracting embedded information and selecting scheduling actions, wherein the graph neural network scheduler includes a representation learning network and a policy learning network;

[0108] Among them, the graph neural network scheduler in this embodiment includes a representation learning network and a policy learning network. The representation learning network is used to realize the embedding of improved heterogeneous disjunctive graphs. Graph embedding refers to converting graph structure data into low-dimensional vectors, thereby capturing the structural characteristics of nodes in the graph, which is convenient for direct application in machine learning; the policy learning network is used to select scheduling actions based on the graph embedding results.

[0109] S13. Based on the representation learning network, the improved heterogeneous disjunctive graph embedding is realized, and the scheduling action is selected according to the strategy learning network and the embedding result;

[0110] Specifically, the embedding of the improved heterogeneous disjunctive graph based on the representation learning network includes:

[0111] A dual heterogeneous attention mechanism is designed to embed process nodes and machine nodes respectively;

[0112] Embedding machine nodes based on meta-path-edge attention, including using graph attention to embed neighbor machine nodes based on meta-paths, and using attention mechanisms to process edge features;

[0113] Based on heterogeneous attention, process nodes are embedded, including considering process constraints, processing capabilities, and neighbor nodes in the processing queue subgraph, and two MLPs with the same structure are used to encode machine and process features respectively;

[0114] The meta-path-edge attention-based embedding of machine nodes includes:

[0115] In terms of metapath neighbors, the path consisting of the disjunctive arc in the machine processing capability subgraph and its connecting two different machine nodes and the same process node is defined as a metapath; at time step t, the path consisting of the disjunctive arc and the connecting two different machine nodes and the same process node is defined as a metapath; k The set of all machine nodes that have a meta-path connection relationship is called its meta-path neighbors (including itself), which is expressed as In this embodiment, graph attention (GAT) is used to embed neighbor machine nodes based on meta-paths to obtain the embedded machine node M k The eigenvector of The expression is:

[0116]

[0117] In the formula, v k Represents the machine node M before embedding k The feature vector of MetapathGAT(v k ) represents the meta-path attention embedding method; Θ1 represents the learnable weight matrix used to transform machine node features to the specified embedding dimension; c represents the meta-path neighbor An element in a set; α ck is the attention coefficient of different neighbor nodes; v c represents the feature vector of c element; LeakyReLU represents the activation function; softmax represents the normalization of all meta-path neighbors; a0 represents the learnable weight matrix; T represents the transpose operator; Θ2 represents the learnable weight matrix used to transform the process node features; || represents the cascade operator; μ ij Represents process node O ij where i represents the workpiece, j represents the jth process of the i-th workpiece; μ(ck) represents the process that is simultaneously connected to the machine node M c and M k A set of process nodes that have a meta-path connection relationship;

[0118] In terms of edge feature processing, the disjunctive arc connecting the process and the machine node carries the processing time and power information of the process machine pair; in this embodiment, the edge graph attention network EdgeGAT is used to integrate the edge features into the message passing and neighborhood aggregation process, where the embedding process of the graph topology neighbors with edge features is expressed as:

[0119]

[0120] αijk=softmax(LeakyReLU(a T [Θ1·v k ||Θ2·μ ij ||Θ3·δ ijk ]))

[0121] In the formula, represents the embedding result of machine nodes in terms of edge features; α ijk Represents the attention coefficient of different neighbor process nodes; Indicates the current time step and machine node M k The set of adjacent process nodes; Θ3 represents the learnable weight matrix used to transform the edge feature dimension; δ ijk represents the edge eigenvector; a T The transpose of the learnable weight matrix representing the edge attention;

[0122] The final embedding results of meta-paths and edges are expressed as:

[0123]

[0124] Where MLP stands for multi-layer perceptron, which has a 2d-dimensional input layer, a d-dimensional output layer, and two d h dimensional hidden layer; ELU represents the function used for hidden layer activation.

[0125] Use a multi-layer perceptron (input layer dimension is 3, 2d-dimensional output layer, two d h dimensional hidden layer and ReLU activation function) to extract the global feature ψ, and add the calculation result to the node-level embedding result to get the complete graph embedding;

[0126] Among them, the expression of complete graph embedding is:

[0127]

[0128] In the formula, represents the complete graph embedding at time step t; O ij Indicates a process node; Represents the process node O after Z iterations ij The final embedding of ;|| represents the cascade operator; Represents a set of machine nodes; M k Represents a machine node; Represents the machine node M after Z iterations k The final embedding of ; θ represents the weights and bias parameters of the connections between neurons in the MLP.

[0129] The embedding process nodes based on heterogeneous attention include:

[0130] For process node O ij For example, the process constraint subgraph and the processing queue subgraph define five types of process neighbor nodes: the process node itself O ij , process front node O i，j-1 , process subsequent node O i，j+1 , queue front node O i′，j′ and queue post-order node O i″，j″ , the corresponding encoding results are: the encoding result of the process node itself f1 = MLP θ (μ ij ), the encoding result of the previous node of the process f2 = MLP θ (μ i，j-1 ), the encoding result of the subsequent node of the process f3 = MLP θ (μ i，j+1 ), the encoding result f4 of the preceding node of the queue = MLP θ (μ i′，j′ ), the encoding result f5 of the queue's subsequent node = MLP θ (μ i″，j″ );

[0131] For the neighbor machine nodes defined by the processing capacity subgraph, the encoding formula is:

[0132]

[0133] The attention mechanism is used to aggregate the encoding results of the above neighbors to obtain the final process node embedding, which is expressed as:

[0134]

[0135] In the formula, k represents the machine number of the process node; Indicates the current time step and process node O ij The set of adjacent machine nodes; α 1，x Indicates the attention coefficients of different types of machine neighbors; represents the learnable attention parameter matrix; f x Representing a collection Any element in , and fx∈{f1∪f2∪f3∪f4∪f5∪f6}.

[0136] Specifically, the selecting of the scheduling action according to the strategy learning network and the embedding result includes:

[0137] In this embodiment, the energy-saving scheduling problem of a flexible job shop for incentive-driven response is transformed into multiple sub-problems, and different strategy learning networks (with the same structure but different weight parameters) are used to solve them;

[0138] In order to enable the policy learning network to generate appropriate strategies according to the input preferences, the present embodiment first designs a preference network to encode the associated weight vector. The network includes a linear layer and a ReLU activation layer. The encoding process is expressed as Λ b =ReLU(Linear(λ b )), where: the linear layer input dimension is 3, the output dimension is 2d, where Λ b represents the final encoding result, Linear represents the linear layer, and λ b represents the weight of the b-th sub-problem;

[0139] The feature embedding and preference encoding are weighted and aggregated through matrix multiplication, and the results are input into the Actor and Critic networks for strategy learning. The input of the Actor network is:

[0140]

[0141] Input of Critic network:

[0142]

[0143] Based on the actions generated by the Actor network, the Critic network evaluates the action value and finally selects the scheduling action.

[0144] S14. Based on the Markov decision process module and the graph neural network scheduler, a graph reinforcement learning model is constructed.

[0145] S2, using the proximal strategy optimization algorithm to train the graph reinforcement learning model;

[0146] The method of training the graph reinforcement learning model using the proximal strategy optimization algorithm includes the following steps:

[0147] Set training parameters. Determine the total number of training cycles and the number of scheduling problem instances that need to be solved in each cycle. The specific steps are: set the total number of iterations required during the training process; prepare a certain number of scheduling problem instances for each round of training and process them in parallel;

[0148] Data collection and parameter update. Collect state, action, and reward data at each time step, and use this data to update the parameters of the scheduler. The specific steps are: record the state of the agent at each time step, the action taken, and the reward obtained; use the proximal policy optimization algorithm (PPO) to adjust the parameters of the scheduler based on the collected data to improve the strategy;

[0149] The specific scheduler parameter adjustment process includes the following steps:

[0150] Determine the scheduling task: optimize the completion time, total energy consumption (TEC) and total effective power consumption during the demand response request at the same time; define the state: the scheduling state described based on the improved heterogeneous disjunctive graph; actions and rewards: a Markov decision process with trajectory-action double-layer rewards.

[0151] Determine the scheduler model: including representation learning network and policy learning network.

[0152] Set the proximal policy optimization algorithm parameters: Learning rate: Determine the learning rate of the optimization algorithm, which is used to update the parameters of the policy network and the value network; Discount factor: Set the discount factor, which is used to calculate the discounted sum of future rewards; Batch size: Determine the sample batch size used for each training update; Optimizer: Select the Adam optimizer, use the Adam optimizer and set the clipping parameter ε = 0.2, policy loss coefficient c1 = 1, value function coefficient c2 = 0.5, entropy coefficient c3 = 0.01, subset size M = 256, and number of updates K = 4; Objective function: Define the objective function of the proximal policy optimization algorithm, and limit the update amplitude of the policy to ensure stability.

[0153] Model training process: Initialize the model and environment: initialize the parameters of the policy network and the value network, and initialize the environment of the scheduler; collect experience data: let the scheduler run in the environment, select actions according to the current state, and execute actions; record samples of the state, action, reward, and next state, and store them in the experience buffer; calculate the advantage function: use the collected experience data to calculate the advantage function of each state-action pair; update the policy and value network: use the proximal policy optimization algorithm to update the parameters of the policy network and the value network; according to the objective function, maximize the expected cumulative reward of the strategy while limiting the update range of the strategy; repeat the above steps: keep repeating the steps of collecting experience data, calculating the advantage function, and updating the policy and value network until the model converges or reaches the preset number of training times.

[0154] Periodic validation and model updating: Prevent the model from overfitting during training and ensure that the model has good generalization ability. The specific steps are as follows: After every ten rounds of training, use a set of independent validation examples to evaluate model performance; based on the validation results, reload a new set of training examples to refresh the model and avoid overfitting.

[0155] S3: Based on the trained graph reinforcement learning model, output the corresponding scheduling plan according to the production tasks and the day-ahead peak shaving invitation;

[0156] The Day-Ahead Peak Shaving Request (DA-PCR) is an invitation or notification sent by the grid operator or market manager to users to inform them of the upcoming peak period and request them to take measures to reduce electricity consumption to help balance the grid load. The Day-Ahead Peak Shaving Request contains three key parameters: (1) Response time: the time period during which peak shaving is required; (2) Shaving requirement (FSL): the upper limit of the power load within the predefined response time, where HEL represents the historical power load measured before subscribing to the Incentive Driven Demand Response (IBDR), and THR represents the reduction threshold for this invitation; (3) Incentive rate: if the manufacturing workshop accepts the DA-PSR and achieves the agreed reduction target within the performance period, it will receive a corresponding monetary reward. In this embodiment, CNY / kW is used as the unit of the incentive rate.

[0157] The above technical solution describes the overall framework of the present invention through modeling and training processes. The following application verification is carried out using a mechanical parts manufacturing workshop as an example.

[0158] The workshop includes 5 CNC equipment to process 4 major categories of products. The present invention uses 10 order data in a production cycle of the workshop in August 2023 for experiments, including the following steps:

[0159] (1) The 10 orders in the production cycle are visualized as an improved heterogeneous disjunctive graph used in the present invention to describe the production scheduling status;

[0160] (2) Setting the reward function in the Markov decision process module of the present invention;

[0161] (3) Using the representation learning network in the neural network scheduler of the present invention, extracting the state features in the improved heterogeneous disjunctive graph, the initial number of iterations t = 0;

[0162] (4) Using the policy learning network in the neural network scheduler of the present invention, actions are generated according to the Actor network, the Critic network evaluates the value of the action, and finally the scheduling action is selected;

[0163] (5) Iteration number t = t + 1;

[0164] (6) Determine whether the scheduling plan output is complete. If t<500, return to (3) to continue execution. If t=500, the decision is completed and the scheduling plan is output.

[0165] The present invention visualizes the scheduling scheme output after the model is applied in the form of a Gantt chart, and draws the corresponding load curve according to the Gantt chart.

[0166] In the Gantt chart, the vertical axis (Y-axis) is mapped to a specific production equipment or workstation in the workshop, while the horizontal axis (X-axis) represents the unit of measurement of time: hours (h). Each rectangular bar represents a specific processing task, and its starting and ending points on the horizontal axis accurately define the planned start and end time of the task. Correspondingly, the position of the rectangular bar on the vertical axis corresponds to the specific equipment that performs the task.

[0167] In the load curve diagram, power consumption is associated with a time series to show a graphical representation of the change in power demand over time. In the load curve diagram of the present invention, the vertical axis (Y-axis) quantifies the unit of measurement of power: kilowatts (kW), and the horizontal axis (X-axis) represents the unit of measurement of time: hours (h). The color blocks in the figure represent energy consumption, representing the energy consumption profile of a single device, and its projected area is equivalent to the cumulative energy consumption of the device during the production process. The height change of the color block reflects the power consumption level of the device at each time point. By superimposing the energy consumption blocks of all devices along the time axis, the total load curve can be obtained.

[0168] The present invention is divided into two scenarios when applying verification: (1) not considering incentive-driven demand response; (2) not considering incentive-driven demand response;

[0169] Scenario 1: No incentive-driven demand response:

[0170] The original problem is converted into a dual-objective optimization problem of the completion period and total energy consumption considering standby energy consumption. The present invention uses the ∈ constraint method to convert the total energy consumption into constraints, and uses the commercial solver Gurobi10.0.3 to optimize the completion period (the solution time is set to 1200s). By repeatedly tightening the constraints, we finally get Figure 5 The approximate Pareto front, Gantt chart and estimated power consumption curve are shown in Figure 6 shown.

[0171] Scenario 2: Consider incentive-driven demand response:

[0172] Consider that the DA-PSR event starts at the third hour and lasts for 2 hours. Other key parameters are as follows: (1) HEL is 20kW, (2) THR is 12kW; (3) incentive rate is 3CNY / kW. The three-objective Pareto optimal frontier obtained by the multi-objective graph neural network scheduler is as follows: Figure 7 Compared with the scheduling scheme in scenario 1, the new scheme sacrifices 1.75h of completion time, but gains 1.16kWh of energy consumption and 387.9CNY of incentives. The new Gantt chart and estimated power consumption curve are shown in Figure 8 shown.

[0173] From the above implementation cases, it can be seen that the present invention has successfully developed a flexible job shop energy-saving scheduling method for incentive-driven demand response through graph reinforcement learning. The model involved not only has high training efficiency and fast solution speed, but also can provide high-quality optimization results. This method shows excellent generalization ability on similar problems of different scales, and provides a general and efficient intelligent optimization solution for production scheduling problems in the field of intelligent manufacturing.

[0174] In summary, with the help of the above-mentioned technical scheme of the present invention, the present invention is used for the production scheduling problem of a flexible workshop operation workshop under incentive-driven demand response, selecting completion period, total energy consumption, and total effective power consumption during demand response request as three optimization objectives, and transforming the problem into a Markov decision process in which the scheduling state is represented by an improved heterogeneous disjunctive graph and a trajectory-action double-layer reward is included. A graph neural network scheduler including a representation learning network and a policy learning network is established, and the proximal policy optimization algorithm is used to train the graph reinforcement learning model; the scheduling efficiency and the ability to respond to incentive-driven demand can be significantly improved, and an efficient, flexible and highly generalized optimization method is provided for the field of production scheduling.

[0175] In addition, the present invention innovatively designs an improved heterogeneous disjunctive graph that integrates the inherent attributes of nodes, deeply revealing the inherent interaction mechanism of production and energy consumption behavior in the context of incentive-driven demand response; further, the present invention designs a graph neural network scheduler, which effectively captures and models the key features of the scheduling environment under incentive-driven demand response through machine node embedding based on meta-path-edge attention and process node embedding based on heterogeneous attention; the present invention not only optimizes the manufacturing process from the dual dimensions of production and energy efficiency, but also provides a practical technical solution for flexible job shops in realizing industrial-level demand response, promoting the development of intelligent manufacturing and energy management.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A flexible job shop energy-saving scheduling method for incentive-driven demand response, characterized in that: The following steps are involved: S1. Based on the improved heterogeneous disjunctive graph, the energy-saving scheduling state of the flexible job shop is represented, and a graph reinforcement learning model is constructed; S2, using the proximal strategy optimization algorithm to train the graph reinforcement learning model; S3: Based on the trained graph reinforcement learning model, output the corresponding scheduling plan according to the production tasks and the day-ahead peak shaving invitation; The day-ahead peak shaving invitation includes a response time, a reduction requirement and an incentive rate; the reduction requirement represents the difference between the historical power load measured before the subscription incentive-driven demand response and the reduction threshold of this invitation.

2. The method for energy-saving scheduling of a flexible job shop for incentive-driven demand response according to claim 1, characterized in that: The method of representing the energy-saving scheduling state of the flexible job shop based on the improved heterogeneous disjunctive graph and constructing a graph reinforcement learning model includes the following steps: S11. Construct a Markov decision process module that uses an improved heterogeneous disjunctive graph to describe the scheduling state, and design the reward function of the Markov decision process module; S12. Establishing a graph neural network scheduler for extracting embedded information and selecting scheduling actions, wherein the graph neural network scheduler includes a representation learning network and a policy learning network; S13. Based on the representation learning network, the improved heterogeneous disjunctive graph embedding is realized, and the scheduling action is selected according to the strategy learning network and the embedding result; S14. Based on the Markov decision process module and the graph neural network scheduler, a graph reinforcement learning model is constructed.

3. The method for energy-saving scheduling of a flexible job shop for incentive-driven demand response according to claim 2 is characterized in that: The expression of the graph structure of the improved heterogeneous disjunctive graph is: In the formula, Represents a set of process nodes; Represents a set of machine nodes; represents the set of connected arcs; Represents the process node-machine node arc set; The original features of the improved heterogeneous disjunctive graph include process node original features, machine node original features, disjunctive arc original features and global features.

4. The method for energy-saving scheduling of a flexible job shop for incentive-driven demand response according to claim 2 is characterized in that: The reward function of the Markov decision process module is designed to include: Define reward functions with multiple levels and objectives; Based on the multi-level factors, a double-layer reward of trajectory and action is defined, where the trajectory reward is the actual value of the optimization target and the action reward is the difference between the expected optimization targets in adjacent time steps; Based on multi-objective factors, the energy-saving scheduling problem of flexible job shop for incentive-driven response is decomposed into N scalar optimization sub-problems, and different weight vectors λ are used to optimize the energy-saving scheduling problem. b , b∈[1, 2, …, N] for identification; Set a weighted hybrid multi-objective aggregate reward function to determine the reward for each sub-problem.

5. The method for energy-saving scheduling of a flexible job shop for incentive-driven demand response according to claim 4 is characterized in that: The weighted hybrid multi-objective aggregate reward function is set to determine the reward for each sub-problem, including: Define the action reward r using Chebyshev method act , based on the weighted sum method to model the trajectory reward r tra , for the b-th sub-problem, the expression of the reward function at time step t is: Where ω1 and ω2 represent the weights of trajectory reward and action reward respectively; and They represent the trajectory reward and action reward of the b-th subproblem at time step t respectively; TE t (λ b ) represents the Chebyshev value of the b-th subproblem at time step t; objs represents the number of optimization objectives; represents the weight of the g-th objective of the b-th subproblem; f g (t) represents the predicted value of the g-th target at time step t; represents the reference value of the g-th target; A t represents the action space set at time step t; Represents a set of process nodes.

6. The method for energy-saving scheduling of a flexible job shop for incentive-driven demand response according to claim 2, characterized in that: The improved embedding of heterogeneous disjunctive graph based on representation learning network includes: Based on meta-path-edge attention, machine nodes are embedded; based on heterogeneous attention, process nodes are embedded; Use a multi-layer perceptron to extract the global feature ψ, and add the calculation result to the node-level embedding result to get the complete graph embedding; Among them, the expression of complete graph embedding is: In the formula, represents the complete graph embedding at time step t; O ij Indicates a process node; Represents the process node O after Z iterations ij The final embedding of ;|| represents the cascade operator; Represents a set of machine nodes; M k Represents a machine node; Represents the machine node M after Z iterations k The final embedding of ; MLP represents multi-layer perceptron; θ represents the weights and bias parameters of the connections between neurons in MLP.

7. The method for energy-saving scheduling of a flexible job shop for incentive-driven demand response according to claim 6 is characterized in that: The meta-path-edge attention-based embedding of machine nodes includes: In terms of meta-path neighbors, a path consisting of a disjunctive arc in the machine processing capability subgraph and two different machine nodes connected by it and the same process node is defined as a meta-path; at time step t, the path connected to the machine node M k The set of all machine nodes that have a meta-path connection relationship is called its meta-path neighbors. Use graph attention to embed the neighbor machine nodes based on the meta-path, and get the embedded machine node M k The eigenvector of The expression is: In the formula, v k Represents the machine node M before embedding k The feature vector of MetapathGAT(v k ) represents the meta-path attention embedding method; Θ1 represents the learnable weight matrix used to transform machine node features to the specified embedding dimension; c represents the meta-path neighbor An element in a set; α ck is the attention coefficient of different neighbor machine nodes; v c represents the feature vector of c elements; LeakyReLU represents the activation function; softmax represents the normalization of all meta-path neighbors; a0 represents the learnable weight matrix of meta-path attention; T represents the transpose operator; Θ2 represents the learnable weight matrix for transforming process node features; || represents the cascade operator; μ ij Represents process node O ij ; Φ(ck) represents the feature vector of the machine node M c and M k A set of process nodes that have a meta-path connection relationship; In terms of edge feature processing, the disjunctive arc connecting the process and the machine node carries the processing time and power information of the process machine pair; the edge graph attention network EdgeGAT is used to integrate the edge features into the message passing and neighborhood aggregation process, where the embedding process of the graph topology neighbors with edge features is expressed as: a ijk =softmax(LeakyReLU(a T [Θ1·v k ||Θ2·m ij ||Θ3·d ijk ])) In the formula, represents the embedding result of machine nodes in terms of edge features; α ijk Represents the attention coefficient of different neighbor process nodes; Indicates the current time step and machine node M k The set of adjacent process nodes; Θ3 represents the learnable weight matrix used to transform the edge feature dimension; δ ijk represents the edge eigenvector; a T The transpose of the learnable weight matrix representing the edge attention; The final embedding results of meta-paths and edges are expressed as: Where MLP represents a multi-layer perceptron; ELU represents the function used for hidden layer activation.

8. The method for energy-saving scheduling of a flexible job shop for incentive-driven demand response according to claim 6, characterized in that: The embedding process nodes based on heterogeneous attention include: For process node O ij For example, the process constraint subgraph and the processing queue subgraph define five types of process neighbor nodes: the process node itself O ij , process front node O i，j-1 , process subsequent node O i，j+1 , queue front node O i′,j′ and queue post-order node O i″，j″ , the corresponding encoding results are: the encoding result of the process node itself f1 = MLP θ (μ ij ), the encoding result of the previous node of the process f2 = MLP θ (μ i，j-1 ), the encoding result of the subsequent node of the process f3 = MLP θ (μ i，j+1 ), the encoding result f4 of the preceding node of the queue = MLP θ (μ i′，j′ ), the encoding result f5 of the queue's subsequent node = MLP θ (μ i″，j″ ); For the neighbor machine nodes defined by the processing capacity subgraph, the encoding formula is: The attention mechanism is used to aggregate the encoding results of each neighbor to obtain the final process node embedding, which is expressed as: In the formula, k represents the machine number of the process node; Indicates the current time step and process node O ij The set of adjacent machine nodes; α 1，x Indicates the attention coefficients of different types of machine neighbors; represents the learnable attention parameter matrix; f x Representing a collection Any element in , and f x ∈{f1∪f2∪f3∪f4∪f5∪f6}.

9. The method for energy-saving scheduling of a flexible job shop for incentive-driven demand response according to claim 6, characterized in that: The selecting of the scheduling action according to the strategy learning network and the embedding result includes: Define a preference network for encoding the associated weight vector, where the preference network includes 1 linear layer and a ReLU activation layer, and the encoding process is represented by Λ b =ReLU(Linear(λ b )), where Λ b represents the final encoding result, Linear represents the linear layer, and λ b represents the weight of the b-th sub-problem; The feature embedding and preference encoding are weighted and aggregated through matrix multiplication, and the results are input into the Actor and Critic networks for strategy learning. The input of the Actor network is: Input of Critic network: Based on the actions generated by the Actor network, the Critic network evaluates the action value and finally selects the scheduling action.

10. The method for energy-saving scheduling of a flexible job shop for incentive-driven demand response according to claim 1, characterized in that: The method of training the graph reinforcement learning model using the proximal strategy optimization algorithm includes the following steps: Set the total number of training iterations, prepare a preset number of scheduling problem instances for each round of training, and process them in parallel; Record the agent's state, actions taken, and rewards at each time step; use the proximal policy optimization algorithm to adjust the scheduler's parameters based on the collected data; After a preset number of trainings, the model performance is evaluated using independent validation examples, and based on the validation results, new training examples are reloaded to refresh the model and avoid overfitting.

Images