A hierarchical graph attention reinforcement learning method for demand response under a multi-industry chain

Abstract

The patent proposes a hierarchical graph attention reinforcement learning method for demand response under multi-industrial chain. First, the long short-term memory network is used to make decisions based on the dynamic characteristics of electricity prices. Then, based on the decision, the hierarchical spatial graph attention mechanism is used to evaluate the impact of dynamic factors on the multi-industrial chain structure. Finally, this impact is used to update the decision. The hierarchical spatial graph attention mechanism uses spatial graph attention to aggregate information from multiple nodes in the network into a single key node. Then, the decision-making agent evaluates the current decision based on the graph attention mechanism and the information from each key node. The spatial graph attention technology can capture the physical spatial relationship between nodes and remove the influence of redundant node information on the evaluation strategy. Compared with traditional demand response algorithms based on reinforcement learning, the patent can model the impact of dynamic factors on the internal and inter-network of the multi-industrial chain network, thereby improving the quality of energy management strategies.

Description

Technical Field

[0001] This invention relates to a hierarchical graph attention reinforcement learning method for demand response in a multi-industry chain network. Specifically, the algorithm comprehensively considers the changing trend of electricity prices and information such as the working status and power consumption of each network node in the multi-industry chain network, and adaptively allocates production tasks to nodes in each network, so that production tasks are concentrated more during periods of low electricity prices, thereby achieving the goal of reducing the energy costs of the industry chain network. Background Technology

[0002] With globalization and advancements in artificial intelligence, demand response is playing an increasingly important role in supply chain networks. Demand response refers to the ability of companies within an industrial supply chain to dynamically adjust their energy management plans based on real-time electricity prices, which has become a crucial means of reducing production costs. Simultaneously, current supply chain structures are becoming increasingly complex, exhibiting a new characteristic of multi-layered industrial supply chain networks where each networked supply chain consists of a cascaded topology and related entities (enterprises), with multiple networked supply chains coupling together to complete the entire production task. Compared to traditional single-layered networks, this multi-layered supply chain structure presents unique challenges for effective demand response implementation.

[0003] In a multi-layered networked industrial chain system, the constant changes in external real-time dynamic electricity prices lead to changes in the electricity costs for production agents (enterprises) to perform tasks. This impacts energy management strategies within the network layer and propagates throughout the entire multi-layered industrial chain network through cascading within and between network layers, negatively affecting the overall energy management strategy. To address this negative impact, multi-agent reinforcement learning methods have been introduced as an effective approach. First, compared to traditional model-based optimization methods, it avoids the need for complex mathematical modeling using extensive prior knowledge. It can learn the complex dynamic topological relationships between multiple agents in the multi-layered industrial chain end-to-end through deep neural networks. Second, it possesses strong adaptability, updating its strategies through continuous interaction with the environment. This allows it to adapt to dynamically changing needs and environments, providing more flexible and robust responsiveness.

[0004] However, some task allocation methods based on multi-agent reinforcement learning typically only focus on the agent's situation, failing to fully consider the impact of dynamic factors on the internal and inter-network aspects of the multi-industry chain network. This makes the algorithm unable to adapt to dynamic changes in electricity prices, resulting in energy management strategies failing to achieve global optimum. Simultaneously, the task execution of production agents is affected by both cascading relationships within network layers and inter-layer coupling relationships, significantly increasing the solution space. To address these challenges, this algorithm first uses an LSTM (Long Short-Term Memory) temporal network to learn the dynamic characteristics of electricity price changes and make decisions. Then, based on these decisions, a hierarchical spatial graph attention mechanism is used to evaluate the impact of dynamic factors on the multi-industry chain structure. Finally, this impact is used to update the decision. The hierarchical spatial graph attention mechanism refers to using spatial graph attention to aggregate information from multiple network nodes into a single network node. Then, the decision agent evaluates the current decision based on the graph attention mechanism, combining the information from each network node. Spatial graph attention technology can capture the logical connections and physical spatial relationships between nodes and remove the influence of redundant node information on the evaluation strategy. Summary of the Invention

[0005] Technical Problem: The purpose of this invention is to propose a hierarchical graph attention reinforcement learning method for demand response in a multi-industry chain, aiming to reduce energy costs in the industry chain network. This algorithm comprehensively considers electricity price trends and production task completion, allocating production tasks to each agent in each network from the perspective of the overall network benefit. This achieves energy cost reduction without affecting production tasks, thus meeting demand response requirements.

[0006] Technical Solution: This paper proposes a multi-agent reinforcement learning method based on a hierarchical temporal spatial graph attention mechanism within an actor-critic framework. Each agent possesses two types of networks: Actor and Critic. The Actor network makes action choices based on the agent's local observations. The Critic network, in order to evaluate the value of the Actor network's decisions within the entire system, needs to comprehensively consider the observations and actions of other agents. This algorithm, considering the characteristics of multi-chain network operation, designs and implements mechanisms for capturing the influence of temporal dynamic data in both the Actor and Critic networks, as well as a two-layer cooperation mechanism between and within the chain networks. The main technical solutions of this algorithm are as follows:

[0007] During the training phase, to address the impact of dynamic electricity prices on agent policies, the Actor network employs LSTM in the actor part to capture this impact feature and provide action selection. The Critic network then integrates various information to evaluate the Actor network's actions and guide its learning. During the execution phase, only the agent's observations need to be input into the Actor network to output the agent's production tasks for that phase in real time.

[0008] The Critic network of the agents provides the value of the Actor network's decisions within the overall system. However, most task allocation methods based on multi-agent reinforcement learning typically only focus on the agents' situations, failing to adequately consider the impact of dynamic factors on the internal and inter-network aspects of multi-industry chain networks. This results in the Critic network being unable to make reasonable value judgments, affecting inter-network collaboration. To address these issues, this patent proposes a network-based information aggregation mechanism: a hierarchical spatial graph attention mechanism, used to aggregate information within the network. This mechanism can express the complex state of a single industry chain network using concise feature vectors and achieve two-layer collaboration between agents within the network and other networks.

[0009] The following section introduces this hierarchical spatial graph attention mechanism. Since the devices in each industry chain network are linearly arranged, the products produced by each network have been processed by all agents within that network. That is, the working state of the last agent in the network implicitly contains the action choices of all preceding agents. From the perspective of any industry chain network, observing the other networks, the most important and direct focus for better coordinating the production tasks of different networks is the state of the last agent in each network. Therefore, this paper selects the last agent in the industry chain network as the network's "key node." Based on this node, a self-attention mechanism is applied to weighted summation of information from all nodes, forming a "network code" to aggregate network information. After the aggregation of network information is completed, agents in other networks can use the "network code" to achieve inter-network collaboration.

[0010] The following describes the collaboration mechanism within the Critic network. Since there may be numerous agents within the industry chain network, and each agent possesses corresponding local observations and spatial location information, the main challenge in network collaboration is how to comprehensively utilize the observations and spatial location information of each agent to provide an evaluation of the Actor network's actions from a global perspective. This invention utilizes a spatial graph attention mechanism to address this problem. This patent uses the distance between nodes to generate corresponding spatial codes to describe the spatial relationships between nodes. After obtaining the spatial codes between nodes, the agents need to combine the information of each node within the network with the spatial structure information. Each node adaptively calculates the weights of the remaining nodes in the network using a self-attention mechanism. These weights are then summed with the spatial codes to generate spatial graph attention weights. Finally, the weighted summation of the node feature vectors yields the aggregated information within the network. This patent argues that this information not only includes the characteristics of each agent and the production network but also implicitly contains collaboration information and spatial relationships between nodes, thus efficiently expressing the state of the entire production system and ensuring that the Critic network evaluates the value of actions from the perspective of the overall system benefit.

[0011] Beneficial effects:

[0012] (1) Balancing task execution among various industry chain networks: By adopting a hierarchical spatial graph attention mechanism, task strategies can be rationally selected based on the cascading relationships within the industry chain network and the coupling relationships between networks. This can effectively solve the problem that previous multi-agent reinforcement learning methods did not consider the network topology, resulting in the allocation results getting trapped in local optima.

[0013] (2) Enhanced adaptability to external dynamic electricity prices. Due to the adaptive ability of reinforcement learning and the predictive ability of LSTM network for time series data, the agent's strategy can adapt to dynamic electricity prices in real time, enabling the entire industry chain network to complete all tasks at a lower cost.

[0014] (3) Improved stability of the industrial chain network. In the industrial chain network, fluctuations in external electricity prices pose a challenge to the stability of the industrial chain network and may lead to overproduction at some production nodes. The method proposed in this patent aims to provide a better allocation scheme in real time based on the volatility of electricity prices, so as to minimize the possibility of overproduction by enterprises and thus ensure the stability of the industrial chain system. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the Actor network in this invention, which uses LSTM to learn the fluctuation pattern of electricity prices and assists the agent in adjusting the production plan in advance according to the trend of electricity prices.

[0016] Figure 2This is a schematic diagram illustrating the application of a self-attention mechanism to aggregate information flow among nodes within a key node of an industry chain network with m nodes.

[0017] Figure 3 This diagram illustrates how information from nodes within various supply chain networks is aggregated to form multiple "network nodes," which are then shared across the networks. The circular nodes represent agents within the network, different quadrilaterals indicate which supply chain networks the nodes belong to, and the arrows indicate the direction of goods migration.

[0018] Figure 4 This is a schematic diagram of the attention head of intelligent agents within the industry chain network. This attention head will aggregate spatial location information and feature vectors from nodes outside and inside the network.

[0019] Figure 5 This is a schematic diagram of the main principle of the method of the present invention. Detailed Implementation

[0020] 1. Initial stage

[0021] In the initial stage, a multi-chain supply chain network was perceived. This network was divided into multiple supply chain networks based on different production task types, represented as follows: The agents within each network are interconnected, and the front... Each network is coupled to the last network. (Previous) Each network is responsible for producing individual component products, while the last network is responsible for processing these component products into the final product and selling it—that is, the product migrates between different intelligent agents to form the final product. However, migrating the product within the same network and between different networks incurs different costs.

[0022] At the start of the task, any number of... Intelligent agents in the network It can be in a running state or an idle state. In the running state, the intelligent agent... Production consumes electricity and raw materials. Electricity costs are affected by electricity prices and are therefore somewhat uncertain. Raw material costs depend on consumption volume and raw material prices. Furthermore, production is also affected by storage capacity. Due to limitations, production cannot be unrestrained. Each agent, when producing, needs to consider not only the dynamics of electricity prices but also whether its cascaded agents require it.

[0023] In this scenario, due to the cascading relationships within the network, if an agent chooses not to produce due to fluctuations in external electricity prices, it can easily lead to a disruption of the entire supply chain, thereby affecting product production efficiency. Simultaneously, if each production supply chain network only considers maximizing its own output while ignoring the coupling relationships between networks, it will result in low resource utilization of the entire system, thus increasing production costs. Therefore, agents need to maximize the total revenue of the supply chain within a production cycle, taking into account the production and risk interference of other agents both inside and outside the network. The model description is given below:

[0024] A group of production intelligent agents It can be divided according to the type of production task. There are several supply chains, and the intelligent agents within each supply chain are arranged sequentially according to the order in which the goods are processed. The first and second industrial chains are parallel to each other and are related to the third. These supply chains are coupled together to form a complete multi-industry chain network; a multi-industry chain can be described as a network that includes... A collection of linked production intelligent agents Each link represents a supply chain network existing in multiple supply chains, determined by the link type. The set of intelligent agents in a connected industrial chain network is represented as: Such a multi-industry chain structure with multiple industry chain networks can be represented by a super Laplace matrix. express; any element Representing the adjacency matrix If there exists from the intelligent agent To intelligent agents The movement of goods, It is a constant, and its value is the cost of moving the goods; otherwise... The following sections will present the operational characteristics of each agent and the interaction characteristics between agents, and finally summarize the optimization objectives for demand response under multiple industry chains.

[0025] Operating characteristics of a single intelligent agent

[0026] Intelligent agents in multiple supply chains need to consider the uncertainty of energy costs in different time periods due to dynamic changes in electricity prices in the environment; taking the first... Intelligent agents in the network For example, let's model an intelligent agent:

[0027] Each agent in time There are two selectable working states—running or idle; in the running state, the agent... It not only consumes electrical energy but also uses raw materials from outside the system; compared to its operational state, the agent in its idle state only remains in standby mode at low power levels and does not perform production tasks; therefore, the agent... The cost of work per unit time consists of electricity cost and raw material cost; [This is followed by a description of an intelligent agent.] A formal description of work costs;

[0028] a) Electricity costs

[0029] make For intelligent agents Decision variables, if If it is in running state, then ;if If it is in an idle state, then ; in time Within this system, each agent operates in only one state; thus, we obtain... In time Internal power consumption :

[0030]

[0031] In the formula, For intelligent agents Power consumption per unit time during operation This represents the power consumption per unit time in standby mode; it also provides the power consumption of the agent over time. After considering the internal power consumption, the energy consumption cost of the equipment is further derived. :

[0032]

[0033] In the formula, yes Electricity price over time; this data will change over time. Dynamic changes;

[0034] b) Raw material costs

[0035] intelligent agent The raw material consumption within a certain period depends primarily on the production volume within that period; disregarding the decrease in production efficiency due to factors such as insufficient raw material supply or production agent malfunctions, the production volume of the agent per unit time is considered a fixed value independent of external conditions; In time Actual production volume within It can be described as follows:

[0036]

[0037] in, yes Production volume per unit time; the equipment combines various raw materials into new products during operation, and the consumption of each raw material varies; For intelligent agents The set of raw materials consumed during runtime, for any one type of raw material ( ), in time Internal consumption The description is as follows:

[0038]

[0039] In the formula, yes Raw materials required to produce a unit quantity of product The quantity; after obtaining the consumption of each raw material, the quantity can be deduced. In time Internal raw material costs :

[0040]

[0041] In the formula as raw materials The unit cost.

[0042] Interaction characteristics between intelligent agents

[0043] During the production process in the industrial chain, goods will be transferred to different intelligent agents in sequence. After being processed layer by layer, they will eventually form a complete output product. In multi-industry chain scenarios, intelligent agents may transfer goods through different types of links. At the same time, since the transfer of goods is limited by the storage capacity of intelligent agents, it is necessary to describe the cascading relationship between intelligent agents. Not only should the transfer cost of goods under different conditions be given, but also a formal description of the storage capacity of intelligent agents should be given.

[0044] a) Costs of moving goods

[0045] Since the agents before and after the goods migration may belong to different industry chain networks, therefore, intelligent agents are used. and Represent any two intelligent agents; move goods from the intelligent agents Migration to intelligent agents The required cost It is represented in two parts:

[0046]

[0047] when and When located within a supply chain network formed by similar links, the cost of goods migration... equal and The cost of collaboration between them through this network layer link ;when and When goods are located in different network layers, they need to be migrated through links across network layers. The migration cost of goods on such links is greater than the migration cost within the same network.

[0048] b) Storage capacity

[0049] In time Storage capacity Equal to time ( Storage capacity, plus In time Production within, minus exist Internal consumption; in summary, It can be represented as follows:

[0050]

[0051]

[0052] In the formula, for In time Internal The amount of goods consumed.

[0053] The goal of task allocation in a multi-chain industry network is to minimize the energy costs required to perform tasks while maintaining normal production operations, thereby maximizing the total revenue of the industry chain within a production cycle.

[0054] The completion status of the task is evaluated using the revenue obtained by the supply chain network from producing products within a production cycle; the total revenue obtained by the supply chain network is equal to the value of the output product minus the raw material cost of the product and the cost of goods migration during the production process.

[0055]

[0056]

[0057] In the formula It is the total revenue of the industrial chain network; It is time A collection of internally output products; It is the value set of the output products; Indicates time Memory from to Goods relocation, otherwise ;

[0058] The risks involved in performing a task can be measured by the sum of the electricity costs of all agents; the total electricity cost of the supply chain network over a production cycle is:

[0059]

[0060] In summary, the optimization objective can be defined as maximizing the cumulative revenue of the network while considering the costs required to mitigate risk; that is, maximizing the difference between the network's revenue and its risk costs.

[0061]

[0062] 2. Model Building and Problem Formulation

[0063] In a multi-chain supply chain network, each production agent is affected by external factors such as electricity prices, leading to increased production costs. To reduce operational costs without impacting production tasks, agents need to consider both task completion status and electricity price fluctuations, scheduling production tasks during periods of low electricity prices whenever possible. Therefore, during algorithm training, each agent needs to receive information such as electricity price data, production buffer storage capacity, and time from the external environment to rationally allocate actions at each stage based on production tasks and electricity prices. The Markov decision model for this problem is presented below:

[0064] state: The system affects the intelligent agent. The main parameters for decision-making include time. Energy costs within Storage capacity and maximum capacity Specifically, when A higher value indicates that the electricity price is at its peak during that period, and the agent should be more inclined to standby mode to reduce power consumption; when Slightly less than or equal to When this happens, it means the buffer zone is about to fill up, and the agent should stop production to reduce unnecessary waste. In summary, the agent... In time Local observation As shown in the following formula.

[0065]

[0066] External state includes the current time. and time-varying electricity prices received from SG The complete system state is shown in the equation.

[0067]

[0068] In the formula It is the set of observations of all intelligent agents.

[0069] Furthermore, due to the agent's observation o ij It will directly affect its action a ij To ensure that the agent considers dynamic changes in electricity prices when making decisions, this paper adds electricity prices for four time periods from t to t+3 to the observations. The final electricity price used in the environment... As shown in the formula.

[0070]

[0071] action: The action space defines all possible actions that an agent can take. Any intelligent agent has only two working states per unit of time: running or standby. Let... For intelligent agents In time The decision variables can be used to obtain the intelligent agent. Action space:

[0072]

[0073]

[0074] In the formula, 0 represents standby and 1 represents running.

[0075] Reward function: The goal of this algorithm is to maximize the total revenue of a multi-chain supply chain network within a production cycle. The objective of demand response optimization scheduling for the supply chain network is to reduce network energy consumption while ensuring production tasks are completed. To achieve this goal, the agent needs to avoid peak electricity price periods and shift more production processes to off-peak periods. Therefore, it can be obtained that the supply chain network at time [time value missing]... Rewards For time The revenue minus the cost of raw materials and energy consumption is:

[0076]

[0077]

[0078] In the formula, Is the system in time The total value of domestically produced goods and It represents the raw material consumption cost and energy consumption cost of any intelligent agent within each sub-network within time t.

[0079] 3. Reinforcement Learning-Based Demand Response Algorithm for Multiple Supply Chains

[0080] 3.1 Actor Network

[0081] Deep neural networks possess powerful nonlinear modeling capabilities, making them suitable for handling complex observations by intelligent agents. The Actor network designed in this patent mainly consists of two parts: a Long Short Term Memory (LSTM) model and a Multilayer Perceptron (MLP). Figure 1 The structure of the Actor network is shown.

[0082] LSTM, as a type of recurrent neural network (RNN), performs well in modeling sequential data. Compared to traditional RNNs, LSTM uses a gating mechanism to control the forgetting, updating, and deletion of states, thus handling sequential data better. Electricity prices, as time-series data, exhibit fluctuations over time. Therefore, this paper aims to utilize LSTM to learn the fluctuation patterns of electricity prices, assisting an agent in adjusting its production plan in advance based on electricity price trends. Specifically, the agent's observations from... arrive The electricity price information is input into an LSTM, and then the LSTM output is concatenated with the observations to form a new feature vector. This paper argues that this feature vector not only encompasses the agent's current state information but also includes predictions of future electricity price changes. Next, this feature vector is input into a standard fully connected MLP. The MLP consists of an input layer, a single hidden layer, and an output layer. The output layer ultimately produces a probability distribution containing the probabilities of all possible actions the agent might take in the current state. Finally, random sampling is performed according to this probability distribution to determine the specific output action.

[0083] 3.2 Critic Network

[0084] (1) Inter-network collaboration in Critic

[0085] Some task allocation methods based on multi-agent reinforcement learning typically only focus on the agent's situation, failing to fully consider the impact of dynamic factors on the internal and inter-network aspects of multi-industry chain networks, making the algorithms unable to adapt to dynamic changes in electricity prices. Therefore, this patent designs a hierarchical spatial graph attention mechanism, which can express the complex state of a single industry chain network using concise feature vectors and achieve two-layer cooperation between agents within the network and other networks.

[0086] Because the equipment in the supply chain network is linearly arranged, the output product of each network has been processed by all the equipment within that network. Therefore, the working state of the last device in a sub-network implicitly determines the action choices of all its predecessor devices. From the perspective of any i-th network, observing the other networks, the most important and direct focus for better coordinating the production tasks of different networks is the state of the last device in each network. In summary, this patent selects the last device in a sub-network as the "key node" of the network, and uses a self-attention mechanism to aggregate information from all nodes based on this node. A network of nodes The process of aggregating internal information is as follows: Figure 2 As shown.

[0087] The basic idea of ​​self-attention mechanisms is to process the feature vector at each position in the input sequence through a parameter matrix. , , The mapping yields the query vector for that location. Key vector Sum value vector And use these vectors to calculate the attention scores between each position. The query vector and key vector in the network are shown in the following equations. Since the attention mechanism is only applied to the "key nodes" in each network, the following equation only calculates the query vector of the last node. .

[0088]

[0089]

[0090] In the formula Represents the number of nodes in the network; It is the internet The feature vector of any node in the vector is formed by the node's state-action pair. , This is formed through MLP encoding. Furthermore, this paper directly analyzes the feature vectors of the nodes. Weighted summation eliminates the need to calculate the value vector.

[0091] After obtaining the query vector and key vector, the attention score g of the key node to any node can be calculated. mj The calculation process is shown in the following formula.

[0092]

[0093] In the formula This represents the dimension of the key vector.

[0094] Next, normalization is performed using the Softmax function to adjust the attention scores. Converted into corresponding attention weights ,

[0095] Finally, for the feature vector of each node Weighted summation yields the network. eigenvectors :

[0096]

[0097] After information aggregation is complete, it is only necessary to pass the feature vectors of each network between networks. Then, the other agents in the network can use this information to achieve inter-network cooperation. Because eigenvectors With node feature vectors Since the dimensions are equal, therefore for In this regard, the remaining subnetworks can all be viewed as "network nodes" with feature vectors. For example... Figure 2 As shown, "network node" and New relationships are formed between nodes within the network. This paper will handle network nodes and ordinary nodes uniformly in the collaboration within the network.

[0098] (2) Collaboration within the network in Critic

[0099] The Critic network evaluates the value of different actions based on the current state and a value function. Node The value function can be expressed as .in, Parameters representing the Critic network; It is a feature vector, which is often obtained by processing the observations of an agent by a specific MLP; Indicates the MLP layer. express The aggregation results of various types of information in the network. The calculation process is as follows Figure 4 As shown.

[0100] Overall, The feature vectors of each node It is aggregated. In calculation... At that time, for intelligent agents A self-attention mechanism is applied to obtain its attention scores for all nodes. Then, spatial encoding is added, followed by a softmax operation to obtain the attention weights for all nodes. Finally, the feature vectors of each node are weighted and summed to obtain the final attention score. After the above operations, It not only includes the characteristics of each intelligent agent and the production network, but also implies the collaborative information and spatial relationships between nodes, thus efficiently expressing the state of the entire production system and ensuring that the Critic network evaluates the value of actions from the perspective of the overall interests of the system.

[0101] 3.3 Model Training

[0102] This algorithm uses an Actor-Critic architecture, treating each enterprise in the industry chain as an agent and deploying a pre-trained Actor policy network for it. The Actor network provides the action that the agent should take at the current moment, i.e., the production task to be executed. Simultaneously, a Critic value network is used for training to calculate the value estimate of the action.

[0103] Actor Network

[0104] Each agent's Actor network is updated independently using gradient ascent. The gradient here is calculated from the system's objective function, i.e., the expected cumulative reward E. After obtaining the gradient of the expected cumulative reward, the agent only needs to update its policy along the gradient ascent direction to approximate the maximum cumulative reward. The gradient is calculated as shown in the equation:

[0105]

[0106] Critic Network

[0107] The training process of this algorithm is similar to that of MAAC. Because agents within the same network share parameters, the network... The Critic of each agent within the system is updated by minimizing the joint loss function as shown in the equation:

[0108]

[0109]

[0110] In the formula, The policy function representing the agent; and These are the parameters of the target Critic network and the target Actor network, respectively. This loss function essentially calculates the mean squared error between the training network (i.e., the Critic network) and the target network (i.e., the target Critic network). Minimizing this loss function updates the parameters of the training network.

[0111] This method implements a two-layer collaboration mechanism between various industry chain networks and between nodes within each industry chain network. This method solves the problems of communication difficulties between networks in multi-industry chain network scenarios and the difficulty in evaluating the value of actions based on the spatial location and state information of each agent within the network.

Claims

1. A hierarchical graph attention reinforcement learning method for demand response in a multi-industry chain context, characterized by: First, the reinforcement learning actor neural network uses a long short-term memory network to make decisions based on the dynamic changes in electricity prices. Then, a critic network constructed using a hierarchical spatial graph attention mechanism evaluates the impact of dynamic factors on the multi-chain structure. Finally, this impact is used to update the actor and critic networks. Specifically, the hierarchical spatial graph attention mechanism uses spatial graph attention to aggregate information from multiple nodes in the network into a single key node. Then, the critic neural network evaluates the current decision based on the graph attention mechanism, combining the information from each key node. The spatial graph attention mechanism can capture the physical spatial relationships between nodes and remove the influence of redundant node information on the evaluation strategy. A group of production intelligent agents Classified according to the type of production task There are several supply chains, and the intelligent agents within each supply chain are arranged sequentially according to the order in which the goods are processed. The first and second industrial chains are parallel to each other and are related to the third. These supply chains are coupled together to form a complete multi-industry chain network; a multi-industry chain can be described as a network that includes... A collection of linked production intelligent agents Each link represents a supply chain network existing in multiple supply chains, determined by the link type. The set of intelligent agents in a connected industrial chain network is represented as: Such a multi-industry chain structure with multiple industry chain networks is represented by a super Laplace matrix. express; any element Representing the adjacency matrix If there exists from the intelligent agent To intelligent agents The movement of goods, It is a constant, and its value is the cost of moving the goods; otherwise... ; The Actor network consists of two parts: a Long Short-Term Memory (LSTM) model and a Multilayer Perceptron (MLP). Electricity prices, as time-series data, exhibit fluctuations over time. Therefore, the LSTM is used to learn the fluctuation patterns of electricity prices, assisting the agent in adjusting production plans in advance based on electricity price trends. Specifically, the observed data includes energy costs. Storage capacity and maximum capacity The process also includes electricity prices for four time periods from t to t+3. The electricity price information from t to t+3 observed by the agent is input into an LSTM, and then the output of the LSTM is concatenated with the observations to form a new feature vector. Next, this feature vector is input into a standard fully connected MLP. The MLP consists of an input layer, a single hidden layer, and an output layer. The output layer ultimately produces a probability distribution that contains the probabilities of all possible actions that the agent may take in the current state. Finally, random sampling is performed according to this probability distribution to determine the specific output action. The designed critic network selects the last node in the industry chain network as the "critical node" for collaboration between networks. Based on this node, a self-attention mechanism is applied to aggregate information from all nodes to facilitate collaboration. In the formula, For intelligent agents The weights; For intelligent agents The feature vectors are obtained by processing the observations of the agent by an MLP; after information aggregation is completed, it is only necessary to transmit the feature vectors of each network between networks. This means enabling collaboration between networks.

2. The hierarchical graph attention reinforcement learning method for demand response under multiple industrial chains as described in claim 1, characterized in that: Intelligent agents in multiple supply chains need to consider the uncertainty of energy costs in different time periods due to dynamic changes in electricity prices in the environment; taking the first... Intelligent agents of the network For example, modeling an intelligent agent; Each agent in time There are two selectable working states—running or idle; in the running state, the agent... It not only consumes electrical energy, but also uses raw materials from outside the system; Compared to its operational state, an agent in its idle state will only remain in standby mode at low power levels and will not perform any production tasks; therefore, the agent... The cost of work per unit time consists of electricity cost and raw material cost; [This is followed by a description of an intelligent agent.] A formal description of work costs; a) Electricity costs make For intelligent agents Decision variables, if If it is in running state, then ;if If it is in an idle state, then ; in time Within this system, each agent operates in only one state; thus, we obtain... In time Internal power consumption : In the formula, For intelligent agents Power consumption per unit time during operation This represents the power consumption per unit time in standby mode. Get the agent in time After considering the internal power consumption, the energy consumption cost of the equipment is further derived. : In the formula, yes Electricity price over time; this data will change over time. Dynamic changes; b) Raw material costs intelligent agent The amount of raw materials consumed within a certain period depends primarily on the production volume during that period; without considering factors such as... Production efficiency declines due to factors such as insufficient raw material supply and production agent malfunctions will reduce the production capacity of the agent per unit time. The quantity is considered a fixed value independent of external conditions; the intelligent agent In time Actual production volume within The description is as follows: in, yes Production volume per unit time; the equipment combines various raw materials into new products during operation, and the consumption of each raw material varies; For intelligent agents The set of raw materials consumed during runtime, for any one type of raw material ( ), in time Internal consumption The description is as follows: In the formula, yes Raw materials required to produce a unit quantity of product The quantity; after obtaining the consumption of each raw material, the quantity can be deduced. In time Internal raw material costs : In the formula as raw materials The unit cost.

3. The hierarchical graph attention reinforcement learning method for demand response under multiple industrial chains as described in claim 1, characterized in that: During the production process in the industrial chain, goods will be transferred to different intelligent agents in sequence. After being processed layer by layer, they will eventually form a complete output product. In multi-industry chain scenarios, intelligent agents may transfer goods through different types of links. At the same time, since the transfer of goods is limited by the storage capacity of intelligent agents, it is necessary to describe the cascading relationship between intelligent agents. Not only should the transfer cost of goods under different conditions be given, but also a formal description of the storage capacity of intelligent agents should be given. a) Costs of moving goods Since the agents before and after the goods migration may belong to different industry chain networks, therefore, intelligent agents are used. and Represent any two intelligent agents; move goods from the intelligent agents Migration to intelligent agents The required cost It is represented in two parts: when and When located within a supply chain network formed by similar links, the cost of goods migration... equal and The cost of collaboration between them through this industry chain network ;when and When goods are located in different supply chain networks, they need to be moved through links across networks. The cost of moving goods through such links is greater than the cost of moving goods within the same network. b) Storage capacity In time Storage capacity Equal to time ( Storage capacity, plus In time Production within, minus exist Consumption within; In conclusion, It is expressed as follows: In the formula, for In time Internal The amount of goods consumed.

4. The hierarchical graph attention reinforcement learning method for demand response under multiple industrial chains as described in claim 1, characterized in that: The goal of task allocation in a multi-chain industry network is to minimize the energy costs required to perform tasks while maintaining normal production operations, thereby maximizing the total revenue of the industry chain within a production cycle. The completion of the task is evaluated using the revenue generated by the supply chain network within a production cycle; the total revenue generated by the supply chain network equals the value of the output product minus the raw material cost and costs incurred during the production process. Costs of moving goods; In the formula It is the total revenue of the industrial chain network; It is time A collection of internally output products; It is the value set of the output products; Indicates time Memory from to Goods relocation, otherwise ; The risks involved in performing the task are measured by the sum of the electricity costs of all intelligent agents; the total electricity cost of the supply chain network over a production cycle is: In summary, the optimization objective is defined as maximizing the cumulative revenue of the network while considering the costs required to mitigate risk; that is, maximizing the difference between the network's revenue and its risk costs.

5. The hierarchical graph attention reinforcement learning method for demand response under multiple industrial chains according to claim 1, characterized in that: The designed critic network consists of cooperative agents within the network. The value function is expressed as ;in, Parameters representing the Critic network; Indicates the MLP layer. express Aggregation results of various types of information in the network; The feature vectors of each node Aggregated; in calculation At that time, for the node By applying a self-attention mechanism, its attention score to all nodes is obtained; then, spatial encoding is added based on this; finally, the feature vectors of each node are weighted and summed to obtain the final value. ; It not only includes the characteristics of each intelligent agent and the production network, but also implies the collaborative information and spatial relationships between nodes, thus efficiently expressing the state of the entire production system and ensuring that the Critic network evaluates the value of actions from the perspective of the overall interests of the system.

Images