Event-triggered energy system optimization method based on time-event dual-scale framework

Abstract

The invention discloses an event-triggered energy system optimization method based on a time-event dual-scale framework, and belongs to the technical field of information, and the method comprises the steps: firstly constructing an optimization problem of the operation of an industrial energy system; secondly, reinforcement learning elements are constructed to build a simulation environment; thirdly, state representation with explicit time distribution characteristics is obtained based on a neural Kalman model; a near-end strategy optimization method is adopted to construct a reinforcement learning framework, and the reinforcement learning framework, a simulation environment and a neural Kalman model jointly construct a time-driven optimization model based on hidden state representation and train the time-driven optimization model; and finally, constructing an event-driven optimization model, constructing an event-driven optimization model based on state transition matrix link through combination of time-varying state matrix link and a neural Kalman model, training the event-driven optimization model, and finally realizing event-triggered optimization scheduling through time-event dual-scale cooperative calculation. According to the method, an energy scheduling strategy with higher operation economy can be obtained, and the actual industrial scheduling requirement is met.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and relates to an optimization method for an event-triggered energy system based on a time-event dual-scale framework, and specifically relates to a time-event dual-scale energy optimization method combining neural Kalman state representation and reinforcement learning. Background Art

[0002] In industrial practice, energy production and consumption are closely related to the production process, resulting in the lack of explicit regularities or distribution characteristics of energy data in the time dimension. Representing the state of this type of data is beneficial to the optimization effect of learning-based methods and improves the efficiency and effectiveness of the learning process. On the other hand, considering that the formulation of the steel energy system scheduling strategy is usually event-driven (for example, the by-product gas storage is about to exceed the safety limit), that is, event-driven optimization is used to make decisions and scheduling operations are executed when an event occurs. Therefore, in order to ensure the long-term safe and economic operation of the system and meet the actual scheduling requirements of the industry, it is of great practical significance to study event-driven models.

[0003] To mine the features and hidden states of industrial data, existing research mainly uses end-to-end deep neural networks to achieve hierarchical feature representation of complex data, including: using Temporal Convolutional Networks (TCNs) to extract hidden information and long-term temporal relationships (Wang Y., Chen J., Chen X. (2020). Short-term load forecasting for industrial customers based on TCN-Light GBM[J]. IEEE Transactions on Power Systems, 36(3):1984-1997), and using deep contrast learning methods to represent uncertain states in energy optimization environments (Wang T., Zhao J, Wang W. (2023). A condition knowledge representation and feedback learning framework for dynamic optimization of integrated energy systems[J]. IEEE Transactions on Cybernetics, DOI: 10.1109 / TCYB.2023.3234077.). However, the above methods of using neural network layers with "black box" characteristics for state representation have the disadvantages of being unexplainable and having poor learning effect stability when facing industrial scenarios, and they fail to extract the data distribution of feature states in the time dimension, reducing the stability and reliability of reinforcement learning (RL) for optimization decisions.To account for the temporal dependence between adjacent latent states, some scholars have studied this problem in recent years, including: proposing a weighted linear dynamic system for feature representation and soft sensing in nonlinear dynamic industrial processes (Yuan X., Wang Y., Yang C. (2017). Weighted linear dynamic system for feature representation and soft sensor application in nonlinear dynamic industrial processes[J]. IEEE Transactions on Industrial Electronics, 65(2): 1508-1517); using neural network-assisted Kalman filtering for state estimation of dynamic systems, especially for partially known dynamics (Revach G., Shlezinger N., Ni X.(2022). KalmanNet: Neural network aided Kalman filtering for partially known dynamics[J]. IEEE Transactions on Signal Processing, 70: 1532-1547) or even scenarios with invisible conditions (Chen C., Lu C., Wang B. (2021). DynaNet: Neural Kalman dynamical model for motion estimation and prediction[J]. IEEE Transactions on Neural Networks and Learning Systems, 32(12): 5479-5491.). However, the above methods mainly focus on state estimation and lack coordination with decision feedback in the RL framework, making it difficult to achieve the learning mechanism of "state estimation - decision - parameter update".In addition, existing event-driven models (such as (Li Y., Zhang H., Liang X. (2018). Event-triggered-based distributed cooperative energy management formulti-energy systems[J]. IEEE Transactions on Industrial Informatics, 15(4):2008-2022.), (Zhang N., Sun Q., Yang L. (2021). Event-triggered distributedhybrid control scheme for the integrated energy system[J]. IEEE Transactionson Industrial Informatics, 18(2): 835-846.)) only focus on communication efficiency and ignore the potential dynamic connections between states before and after an event occurs, resulting in an incomplete dynamic association modeling of the state-action-reward sequence during the RL learning process, which is not conducive to the convergence of RL strategies and the perception of long-term rewards. Summary of the Invention

[0004] The purpose of the present invention is to provide an event-triggered energy system optimization method based on a time-event dual-scale framework. This method uses a neural Kalman filter to extract time correlations, improving the interpretability of state representations and thus enhancing the stability and reliability of RL learning. The combination of the time-varying state matrix link and the neural Kalman filter in the method of the present invention establishes the potential dynamic connection between states before and after an event occurs, enabling event-driven RL to perceive the long-term rewards of optimal scheduling and thus obtaining a more operationally economical energy scheduling strategy. The present invention meets the scheduling requirements of industrial practice and can provide decision-making guidance for industrial energy optimization scheduling.

[0005] The technical solution of the present invention is as follows:

[0006] An event-triggered energy system optimization method based on a time-event dual-scale framework, including: First, constructing an optimization problem for the operation of the industrial energy system; Second, constructing reinforcement learning elements including state, action, and reward to build a simulation environment; Third, based on the neural Kalman model, obtaining a state representation with explicit time distribution characteristics; Further, using the proximal policy optimization method to construct a reinforcement learning framework, and jointly constructing and training a time-driven optimization model based on hidden state representation with the simulation environment and the neural Kalman model; Finally, using another reinforcement learning framework and the simulation environment to construct an event-driven optimization model, and combining it with the neural Kalman model through a time-varying state matrix link to construct and train an event-driven optimization model based on state transition matrix link, and finally realizing event-triggered optimal scheduling through time-event dual-scale collaborative calculation. Specifically, it includes the following steps:

[0007] Step 1, constructing an optimization problem for the operation of the industrial energy system:

[0008] According to the optimization objective of the industrial energy system and the constraint conditions that need to be considered in actual operation, construct an optimization problem for the operation of the industrial energy system. Among them, the optimization objective is to minimize the economic operation cost within a specific time range , including energy dissipation cost , energy outsourcing cost , carbon emission cost and equipment start-stop consumption ; The constraint conditions include energy balance constraint, mass balance constraint, equipment safe operation constraint and equipment output limit; The optimization variable is the flow rate of the energy medium input to each adjustable device in the industrial energy system.

[0009] Step 2, constructing reinforcement learning elements:

[0010] According to the optimization problem of the industrial energy system, construct reinforcement learning elements to build a simulation environment for interacting with the reinforcement learning framework. The reinforcement learning elements include: state , action and immediate reward ; Among them, state contains the energy data in the industrial energy system, and the collected historical energy data is used as the training set; action is the adjustment amount of the flow rate of the energy medium input to each adjustable device in the industrial energy system; The immediate reward is composed of the objective and constraint conditions in the optimization problem in Step 1.

[0011] Step 3, constructing a hidden state representation based on the neural Kalman model:

[0012] Since the state used for construction The energy data does not have explicit patterns or distribution characteristics in the time dimension. Therefore, the state does not have the Markov property, which affects the optimization effect. A hidden state representation based on the neural Kalman model is constructed. Starting from the time series formed by the state , the neural Kalman filter is used to calculate the hidden state with explicit time distribution characteristics, that is, the Markov state transition property , forming a hidden state sequence . Among them, represents the interaction period of reinforcement learning, represents the interaction moment.

[0013] The neural Kalman model is . In the formula, is described by the iterative process of the Kalman filter, and the iterative process is shown in Equation (1): is the hidden state error covariance matrix, is the hidden state transition matrix, is the state observation matrix, is the covariance matrix of the observation noise, is the hidden state noise covariance matrix.

[0014] (1)

[0015] In the formula, represents the Kalman gain, represents the identity matrix, represents the matrix transpose.

[0016] Different from the conventional Kalman model, the parameters and in the neural Kalman model are determined by neural network learning. Since each state in the hidden state sequence follows a linear-like relationship as shown in Equation (3), the transition matrix contains historical hidden state information. The long short-term memory network (LSTM) in the recurrent neural network can capture the time dependence in time series data and can learn the complex mapping from historical states to the current state. Therefore, LSTM is used to calculate and , as shown in Equation (2). Its inputs are and . is the internal hidden layer variable of LSTM at moment. And as well as the neural network weights in LSTM will be used as the model parameters of the neural Kalman model and updated in the subsequent learning process.

[0017] (2)

[0018] (3)

[0019] Step 4, construct a time-driven optimization model based on hidden state representation:

[0020] Adopt the Proximal Policy Optimization method (PPO) in reinforcement learning to construct a reinforcement learning framework. The constructed reinforcement learning framework includes a policy network and a value network. The policy network includes a new policy network and an old policy network, and the structures of the two policy networks are the same and the parameter settings are the same. Combine the reinforcement learning framework with the simulation environment built in Step 2 and the neural Kalman model constructed in Step 3 to form a time-driven optimization model based on hidden state representation and train it. Specifically:

[0021] First, starting from the state , calculate the hidden state with Markov state transition characteristics through the neural Kalman model. The hidden state generates an action through the old policy network.

[0022] The action generated by the old policy network is a multi-dimensional action and satisfies the Beta distribution, that is , where represents the probability distribution of the old policy network selecting the action under the input of . The distribution parameters and are approximated by the neural layers and that make up the policy network respectively:

[0023] (4)

[0024] (5)

[0025] where is the activation function , is the activation function ; and are the parameter variables of the old policy network respectively, represents the dimension of the action , , represents the number of probability distributions output by the policy network, is the maximum action range. Then each variable in Sampled according to the probability distribution generated by the old policy network. The specific sampling formula is as follows:

[0026] (6)

[0027] where is the Beta distribution perturbation, which is used to control the policy exploration ability.

[0028] After that, the simulation environment constructed from historical energy data in step 2, according to the action , changes the current state to the next moment's state and obtains the immediate reward . Completing one interaction, is fed back to the neural Kalman model to generate the hidden variable at the next moment.

[0029] After performing interactions to complete one interaction cycle, groups of interaction data are obtained. Based on the interaction data, a new policy network and an evaluation network are trained, and the policy parameters and the evaluation network parameters are updated using the gradient descent method. Among them, the policy parameters include the parameters of the new policy network, as well as the neural Kalman model parameters and in step 3, that is . The parameter update process is as follows:

[0030] Define the reward function of the new policy network as:

[0031] (7)

[0032] (8)

[0033] where) represents the expected value of the reward of the new policy network calculated by satisfying the probability distribution generated by the old policy network, is the importance sampling ratio of the new and old policy networks at time is the clipping mechanism in PPO, which means ensuring that is within the range of , is the hyperparameter for controlling the clipping degree, represents the probability that the new policy network selects the action in the generated distribution , Indicates the probability that the old policy network selects an action from the generated distribution . The advantage function is approximated by the temporal difference method, i.e.:

[0034] (9)

[0035] where represents the expected cumulative reward that can be obtained when the state takes an action under the distribution . It can be approximated by the temporal difference method as . represents the output of the evaluation network when the input is . represents the temporal difference, represents the Monte Carlo estimate, represents the discount factor.

[0036] The evaluation network is approximated by a multi-layer perceptron (MLP) network , i.e., . The output of the evaluation network is used to calculate the advantage function through Equation (9). When training the evaluation network, the reward function of the evaluation network is defined as:

[0037] (10)

[0038] (11)

[0039] (12)

[0040] where is the value target, represents the th time step from the current time step to the interaction period , is the TD error at the time step .

[0041] After completing a parameter update based on the reward functions shown in Equations (7) and (10), the updated new policy network parameters are passed one-to-one to the old policy network. After the old policy network is updated, it is used for data collection in the next interaction period, generating new interaction data and performing the next parameter update. The immediate reward is continuously accumulated during the interaction. When the accumulated immediate reward value converges, the training is completed and the optimal parameters are determined. Based on the trained time-driven model, the optimal action can be automatically selected according to the current state of the industrial energy system collected in real time as the time-driven scheduling strategy.

[0042] Step 5, construct an event-driven optimization model based on state transition matrix linkage:

[0043] Adopt another reinforcement learning framework and simulation environment to construct an event-driven optimization model. Among them, the simulation environment is constructed according to Step 2, and the reinforcement learning framework still adopts the proximal policy optimization method in reinforcement learning to construct, that is, the output settings of its policy network and evaluation network have the same structure as the reinforcement learning framework of the time-driven optimization model based on hidden state representation in Step 4; through the time-varying state transition matrix linkage, combine the neural Kalman state representation with the event-driven model to construct an event-driven optimization model based on state transition matrix linkage, which is used to obtain the event-driven state, and obtain the optimized event-driven policy through the time-event dual-scale policy collaborative calculation. Specifically:

[0044] First, the time-driven optimization model based on hidden state representation trained in Step 4 calculates the time-driven scheduling policy at a fixed time period by rolling but does not execute it.

[0045] When a scheduling event is triggered, according to the event-driven state at the time of the previous scheduling event and integrating the time-driven states between the previous scheduling event and the current scheduling event, calculate the event-driven state after the current scheduling event occurs . The calculation process is as follows:

[0046] Assume that the time interval between the previous scheduling event and the current scheduling event is time periods , then according to Equation (3), we can get:

[0047] (13)

[0048] On the premise of not destroying the time-varying relationship of state variables, when an event is triggered, multiply both sides of the equations in Equation (13) to get to integrate state information of adjacent time periods. Because , let to obtain the event-driven state transition relationship, which is the state transition matrix linkage:

[0049] (14)

[0050] It can be seen from the above formula that the event-driven state still retains the time transfer characteristics of the time-driven part that is partially linear, that is, the time distribution characteristics are extended to the event-driven model.

[0051] Let Generate actions using the old policy network in the input event-driven optimization model , and feed the obtained into the simulation environment of the event-driven optimization model to obtain the state and the immediate reward ; Feed back to the time-driven optimization model based on hidden state representation to update its corresponding state . The time-driven optimization model based on hidden state representation continues to perform rolling calculations at a fixed time period until the next scheduling event occurs, realizing the collaborative calculation of the time-event dual-scale policy. For the th scheduling event, based on the collaborative calculation of the time-event dual-scale policy, obtain groups of interaction data. Use this groups of interaction data to train the event-driven optimization model and update its new policy network parameters ( ), old policy network parameters ( ), and evaluation network parameters . The parameter update process is the same as in step 4.

[0052] To reduce the scheduling frequency while ensuring the learning effect as much as possible, the reward of the event-driven optimization model is defined as , where the time-driven reward value is the sum of to within the range of adjacent events, and the event-driven reward value is the sum of the immediate rewards of the event-driven optimization model for adjacent events to . And there is: when the event does not occur, is calculated by the simulation environment in the event-driven model according to , and when the event occurs, is calculated by the simulation environment in the event-driven model according to the action taken . Continuously accumulate during the interaction process. When the accumulated[[ID=�6]] converges, the training is completed, and the optimal parameters of the event-driven model are determined.

[0053] The trained event-driven model based on the state transition matrix link and the trained time-driven optimization model based on hidden state representation jointly realize the time-event dual-scale collaborative calculation, and can calculate the optimal action when the corresponding event occurs according to the current state of the industrial energy system collected in real time, as the event-driven scheduling strategy.

[0054] ​Advantages of the present invention: The present invention uses a neural Kalman filter for state representation to establish an explicit state transition relationship, making the state representation process interpretable and enhancing the stability and reliability of reinforcement learning (RL) during the optimization decision-making process. At the same time, the state representation collaborates with the decision feedback of the RL framework to achieve a learning mechanism of "state estimation - decision - parameter update". In addition, the time-event dual-scale collaborative calculation method establishes a potential dynamic connection between the states before and after the event occurs, improving the convergence of the RL policy and the ability to perceive long-term rewards. Description of the Drawings

[0055] Figure 1 This is the application flow chart of the present invention.

[0056] Figure 2 This is the schematic diagram of the principle of the present invention.

[0057] Figure 3 This is the optimization schematic diagram of the time-driven optimization model based on hidden state representation.

[0058] Figure 4 This is the schematic diagram of the reward value convergence comparison and the deviation of the reward value between the embodiments and PPO-clip, PPO-TCN, A3C, and TRPO;

[0059] Figure 5 This is the comparison diagram between time-driven and event-driven during the time-event dual-scale policy training process. Among them, (a) is the reward value convergence comparison diagram; (b) is the enlarged comparison diagram of the two convergence curves in the later stage of training in (a); (c) is the convergence curve of the reward value difference.

[0060] Figure 6 This is the optimization effect diagram of the coke oven gas holder and blast furnace gas holder capacity scheduling or operation. Among them, (a) is the time-driven scheduling effect diagram of the coke oven gas holder; (b) is the event-driven scheduling effect diagram of the coke oven gas holder; (c) is the operation optimization effect diagram of the blast furnace gas holder capacity during the transition period from peak electricity price to valley electricity price; (d) is the operation optimization effect diagram of the blast furnace gas holder capacity during the transition period from valley electricity price to peak electricity price.

[0061] Figure 7 This is the change of the self-generated electricity after optimization with the purchase and sale price. Among them, (a) is the change of the self-generated electricity during the transition period from peak electricity price to valley electricity price; (b) is the change of the self-generated electricity during the transition period from valley electricity price to peak electricity price.

[0062] Figure 8 This is the box plot of the operating cost values in each scenario; among them, (a) is the cost box plot at peak electricity price; (b) is the cost box plot at valley electricity price; (c) is the cost box plot during the transition period of electricity price; (d) is the overall cost box plot. Detailed Embodiments

[0063] The generation and consumption of industrial energy are affected by factors such as the production process and manual operations, resulting in the reinforcement learning states constructed based on energy data not having obvious distribution characteristics. To meet the supply-demand balance of various energy media and ensure the stable operation of the entire production process, on-site operators need to understand the changes in energy generation and consumption and their storage in real time, and make scheduling actions according to events. If a potential state representation with time distribution characteristics and interpretability can be constructed, a scheduling plan can be well formulated to achieve the purpose of energy conservation and emission reduction. Therefore, obtaining a state representation with explicit time distribution characteristics and implementing event-driven dynamic scheduling can improve the learning efficiency and model performance of RL and meet the actual on-site scheduling requirements. To better understand the technical route and implementation plan of the present invention, this embodiment takes the steel energy system as the background and actual industrial data as the basis to illustrate the specific implementation steps of the present invention. The flow chart of the present invention is as Figure 1 , and the schematic diagram of the principle is as Figure 2 , including the following steps:

[0064] Step 1, constructing an optimization problem for the operation of the industrial energy system:

[0065] Taking the steel energy system as an example, an optimization problem for the operation of the steel energy system is constructed according to the optimization objectives of the steel energy system and the constraint conditions that need to be considered in actual operation. Among them, the optimization variable is the flow rate of the energy medium input to each adjustable device in the steel energy system.

[0066] Step 1.1, constructing the objective of the optimization problem of the steel energy system: minimizing the economic operation cost within a specific time range , including the energy dissipation cost , the energy outsourcing cost , the carbon emission cost and the equipment start-stop consumption . The objective function is expressed as follows:

[0067] (15)

[0068] (16)

[0069] (17)

[0070] (18)

[0071] (19)

[0072] Among them, is the objective function, is the time step of the optimization problem, The unit emission costs of gas and steam respectively, respectively are The gas and steam emission volumes at time The purchase cost of thermal coal, 、 The purchase and sales costs of electricity respectively, The consumption of purchased coal, respectively represent the electricity load demand and the power generation of the self - contained power plant; is the adjustable equipment The start - stop consumption cost, is the total number of adjustable equipment, represents the adjustable equipment The operating status, is off, is on; is the average unit price of carbon emissions, is the carbon emission coefficient.

[0073] Step 1.2, construct the constraint conditions for the optimization problem of the iron and steel energy system:

[0074] The constraint conditions to be considered in the operation optimization process of the iron and steel energy system include energy balance constraints, mass balance constraints, equipment safe operation constraints, and equipment output limitations.

[0075] 1) Energy balance constraints. Various energy media in the iron and steel production process can be converted through energy conversion equipment to meet the required energy balance. The energy balance constraints of energy conversion equipment mainly include the balance of inflowing and outflowing energy media:

[0076] (20)

[0077] Among them, the constants and are efficiency parameters obtained through data - driven regression methods; and respectively represent the flow rates of the inflowing and outflowing energy media of the energy conversion equipment at time point

[0078] 2) Mass balance constraints:

[0079] (21)

[0080] Among them, respectively represent The gas generation and consumption volumes at time , is the gas load required for production, respectively are the​ The gas consumption of the th power plant and the th combined heat and power unit, represents the total number of power plants, represents the capacity of the gas holder, represents the total number of gas holders.

[0081] Similarly, the steam mass balance constraint is:

[0082] (22)

[0083] where represents the steam generation amount of the th power plant or combined heat and power unit at steam pressure level at time , is the total number of power plants and combined heat and power units with steam generation pressure level , is the steam discharge amount, is the steam load at pressure level .

[0084] The power balance constraint is:

[0085] (23)

[0086] In the formula, represents the purchased electricity, represents the sold electricity.

[0087] 3) Equipment safe operation constraints and equipment output limits. The iron and steel energy system includes various energy conversion equipment and energy storage equipment. The safe operation constraints and equipment output limits of these equipment are as follows:

[0088] (24)

[0089] (25)

[0090] (26)

[0091] where represents the output of the energy conversion equipment at time , are the lower and upper limits of the output of the th energy conversion or energy storage equipment respectively, are the lower and upper limits of the input of the th energy conversion or energy storage equipment respectively. Energy conversion equipment The slope of output increase and decrease, For energy storage equipment exist The operating level at the moment, Energy storage devices Lower and upper limits of operating levels.

[0092] Step 2: Construct reinforcement learning elements:

[0093] Based on the steel energy system optimization problem, reinforcement learning elements are constructed to build a simulation environment for interaction with the reinforcement learning framework. Reinforcement learning elements include: state ,action and instant rewards .

[0094] 1) Status :Including the production process and , since electricity prices change in different periods of time, at the current moment Time period Also considered key status factors:

[0095] (27)

[0096] The historical energy data collected at the minute level is used as the training set for model training in subsequent steps.

[0097] 2) Action :action is the decision variable of the optimization problem in step 1. The dynamic scheduling of the steel energy system is achieved by controlling the and The power plant and the combined heat and power unit generate electricity and steam that meet the load demand and input and output constraints. As a supplementary input to power plants and combined heat and power units, and After the determination, the difference between the flow rate of the energy medium output by the energy conversion equipment and its load demand is obtained. Defined as:

[0098] (28)

[0099] 3) Instant Rewards : Including system operating costs and penalties for exceeding limits :

[0100] (29)

[0101] Among them, and respectively represent the weight coefficients in the reward function, and is of a higher order of magnitude than ; The calculation formula is as shown in Equation (15), is the constraint violation amount of the optimization problem.

[0102] Step 3, construct a hidden state representation based on the neural Kalman model (the principle is as Figure 3 shown):

[0103] Since the energy data used to construct the state does not have explicit laws or distribution characteristics in the time dimension, the state does not have Markov property, thus affecting the optimization effect. Construct a hidden state representation based on the neural Kalman model, and calculate the hidden state with Markov state transition characteristics through the neural Kalman filter for the state , and form a hidden state sequence. Specifically:

[0104] The neural Kalman model is , where is described by the iterative process of the Kalman filter, and the iterative process is as shown in Equation (1).

[0105] Different from the conventional Kalman model, the parameters and in the neural Kalman model are determined by neural network learning. Since each state in the hidden state sequence follows a class linear relationship as in Equation (3), the transition matrix contains historical hidden state information. The long short-term memory network in the recurrent neural network can capture the time dependence in time series data and can learn the complex mapping from historical states to the current state. Therefore, LSTM is used to calculate and , as shown in Equation (2), and its input is and is the internal hidden layer variable of LSTM at time. And as well as the neural network weights in LSTM will be used as the model parameters of the neural Kalman model and will be continuously updated in the subsequent learning process.

[0106] Step 4, construct a time-driven optimization model based on the hidden state representation:

[0107] The proximal policy optimization method in reinforcement learning is used to construct a reinforcement learning framework. The constructed reinforcement learning framework includes a policy network and a value network. The policy network includes a new policy network and an old policy network. The structures of the two policy networks are the same and the parameter settings are the same. The reinforcement learning framework, the simulation environment built in step 2, and the neural Kalman model constructed in step 3 are jointly used to form a time-driven optimization model based on hidden state representation and are trained. Specifically:

[0108] First, starting from the state , the hidden state with Markov state transition characteristics is calculated through the neural Kalman model . The hidden state generates an action through the old policy network .

[0109] The action generated by the old policy network is a multi-dimensional action and follows a Beta distribution, that is , where the distribution parameters and are approximated by the neural layers and constituting the policy network, as shown in equations (4) and (5). Then each variable in is sampled according to the probability distribution generated by the old policy network, as shown in equation (6)

[0110] After that, according to the action , the simulation environment constructed from historical energy data transforms the current state into the next state and obtains an immediate reward . One interaction is completed, and is fed back to the neural Kalman model to generate the hidden variable for the next moment.

[0111] Perform times of interactions to complete one interaction cycle, obtaining groups of interaction data. Based on the interaction data, the new policy network and the value network are trained, and the policy parameters and the value network parameters are updated using the gradient descent method. The parameter update process is as follows:

[0112] Define the reward function of the new policy network as shown in equations (7) to (9).

[0113] The value network uses a multi-layer perceptron network for approximation, that is , the output of the evaluation network is used to calculate the advantage function through Equation (9). When training the evaluation network, the reward function of the evaluation network is defined as shown in Equations (10) to (12).

[0114] After one parameter update is completed based on the reward functions shown in Equations (7) and (10), the updated new policy network parameters are passed to the old policy network one by one. After the old policy network is updated, it is used for data collection in the next interaction cycle, generating new interaction data and performing the next parameter update. The immediate reward values are continuously accumulated during the interaction process. , when the accumulated immediate reward values converge, the training is completed and the optimal parameters are determined. Based on the trained time-driven model, the optimal action can be automatically selected according to the current state of the steel energy system collected in real time as the time-driven scheduling strategy.

[0115] Step 5, construct an event-driven optimization model based on the link of the state transition matrix:

[0116] Another reinforcement learning framework and simulation environment are used to construct the event-driven optimization model. Among them, constructing the event-driven optimization model is basically the same as constructing the time-driven optimization model based on hidden state representation. The difference is that the event-driven optimization model does not include the neural Kalman model, that is, the reinforcement learning framework in the event-driven optimization model is also constructed using the proximal policy optimization method in reinforcement learning. In addition, the simulation environment is constructed according to Step 2; through the time-varying state transition matrix link, the neural Kalman state representation is combined with the event-driven model to construct an event-driven optimization model based on the link of the state transition matrix, which is used to obtain the event-driven state and obtain the optimized event-driven policy through the collaborative calculation of the time-event dual-scale strategy. Specifically:

[0117] First, the time-driven optimization model based on hidden state representation trained in Step 4 calculates the time-driven scheduling strategy at a fixed time period but does not execute it.

[0118] When a scheduling event is triggered, according to the event-driven state when the previous scheduling event occurred , combining the time-driven state between the previous scheduling event and the current scheduling event, calculate the event-driven state after the current scheduling event occurs according to Equations (13) and (14). .

[0119] The input is given to the old policy network in the event-driven optimization model to generate an action , and the obtained is given to the simulation environment of the event-driven optimization model to obtain the state and the immediate reward ; the Feed it back into the time-driven optimization model based on hidden state representation to update its corresponding state , and the time-driven optimization model based on hidden state representation continues with a fixed time period to perform rolling calculations until the next scheduling event occurs, achieving collaborative calculations of the time-event dual-scale strategy. For the occurring scheduling events, through the collaborative calculations of the time-event dual-scale strategy, a set of interaction data is obtained. Using this set of interaction data to train the event-driven optimization model, and update its new policy network parameters, old policy network parameters, and evaluation network parameters. The parameter update process is the same as in step 4.

[0120] To ensure the learning effect as much as possible while reducing the scheduling frequency, the reward of the event-driven optimization model is defined as , where is the sum of to within the range of adjacent events, and is the sum of the immediate rewards of the event-driven optimization model within the range of adjacent events to . And there is: when the event does not occur, is calculated by the simulation environment in the event-driven model according to , and when the event occurs, is calculated by the simulation environment in the event-driven model according to the action taken. During the interaction process, [[ID=3,8]] is continuously accumulated. When the accumulated converges, the training is completed, and the optimal parameters of the event-driven model are determined. The trained event-driven model based on the state transition matrix link and the trained time-driven optimization model based on hidden state representation jointly achieve time-event dual-scale collaborative calculations, and can calculate the optimal action when the corresponding event occurs according to the current state of the industrial energy system collected in real time, serving as the event-driven scheduling strategy.

[0121] To verify the effectiveness of this embodiment, the actual data of an energy system of a domestic steel plant from April to May 2022 (collected by the data acquisition and monitoring control system) is selected for experiments, and the data sampling interval is 1 min. Among them, the data of the first 24 days is used as the training set for the model training process, and the remaining data is used as the test set. Since an intraday scheduling strategy at the minute level needs to be formulated during the real-time scheduling process, the time length of the interaction period is set to 2 h,

[0122] and is set to 5 min.

[0123] For the model training effect, using the same training set data, RL based on PPO (PPO-clip), RL based on asynchronous advantage Actor-Critic (A3C), RL based on trust region policy optimization (TRPO), and RL based on PPO combined with temporal convolutional network (PPO-TCN) are selected for comparison with this embodiment. Multiple experiments are conducted for each method, and the learning curves and standard deviations obtained are as Figure 4 shown. It can be seen from the figure that the three PPO-based methods (PPO-clip, PPO-TCN, this embodiment) show better learning stability. Compared with the A3C and TRPO models, this embodiment and PPO-TCN obtain larger reward values in the later stage of training, which proves the effectiveness of the hidden state representation model in the learning process of industrial data.

[0124] Figure 5 In (a), the convergence curves of the time-driven reward value and the event-driven reward value in step 5 of this embodiment are given respectively. Due to the co-training process of time and events, the fluctuation trends of the two convergence curves are similar. In addition, from Figure 5 in (b) and (c), it can be seen that as the number of training times increases, the reward difference between the two convergence curves gradually decreases. At the end of the training process, the event-driven reward even exceeds the time-driven reward. The above results show that the event-driven scheduling method proposed in this embodiment can achieve the overall expected training effect.

[0125] For the scheduling effects of different energy media, Figure 6 in (a) and (b), the operation results of the coke oven gas (COG) tank capacity under the time-driven and event-driven strategies are described respectively. Since the COG generation source has small fluctuations and the daily tank position operation is relatively stable, while the time-driven strategy needs to perform scheduling actions at fixed intervals, resulting in frequent fluctuations in the COG tank position. On the contrary, the event-driven strategy can reduce the number of scheduling times, and its optimized tank position operation results are more in line with the actual on-site process requirements. Figure 6 In (c) and (d), the operation optimization effects of the blast furnace gas (BFG) tank capacity under time-of-use electricity price are shown. From Figure 6 in (c), it can be seen that when the electricity price changes from off-peak to peak, the optimized BFG tank capacity shows a downward trend to use the stored gas to increase the self-generated electricity of the enterprise. And when the electricity price changes from high to low (as shown in Figure 6 in (d)), the tank capacity will increase compared with the original tank capacity value, realizing more gas storage. The change of the optimized self-generated electricity with the purchase and sale price is as Figure 7As shown, the results show that during the low electricity price period, the steel energy system purchases electricity from the external power grid to reduce costs; during the peak electricity price period, the steel energy system uses steam turbines and CHP to increase power generation and reduce the purchased electricity during the peak period. Therefore, as Figure 7 shown in (a) below, when the electricity price transitions from peak to valley, the self-generated electricity decreases; as Figure 7 shown in (b) below, when the electricity price transitions from valley to peak, the self-generated electricity increases. The above results demonstrate that the optimization of energy utilization in this embodiment takes into account the change of electricity price, which helps to reduce the electricity cost.

[0126] To verify the effectiveness of this embodiment in reducing the overall operating cost, traditional manual experience scheduling, some common optimization methods, and the state representation RL method are selected as comparison methods to compare with the method of this embodiment. The comparison methods include: on-site manual experience scheduling (Method a), mixed-integer linear programming method (Method b), model predictive control optimization method (Method c), original PPO-based method (Method d), PPO combined with time convolutional network method (Method e), time-driven optimization method based on neural Kalman state representation (Method f), asynchronous advantage Actor-Critic-based method (Method g), and trust region policy optimization-based method (Method h).

[0127] Based on the same test set data, the optimized operating costs in three scenarios of peak, valley, and transition of electricity price are calculated using this embodiment and Methods a to h respectively, and the mean, minimum, maximum, and standard deviation of the optimized operating costs in each scenario are statistically analyzed. Figure 8 The box plots of the peak (as Figure 8 shown in (a) below), valley (as Figure 8 shown in (b) below), transition period (as Figure 8 shown in (c) below) of electricity price and the overall operating cost value (as Figure 8 shown in (d) below) are given, and Table 1 shows the corresponding numerical statistical results. As Figure 8Combined with Table 1, it can be observed that in different electricity price scenarios, this embodiment is superior to other methods in terms of average cost. Table 2 further summarizes the daily total operating costs of different methods under the test set. From the results in Table 2, it can be seen that: Method c is inferior to Method d and this embodiment in terms of total cost; Method b is prone to falling into local optima and its performance in optimizing the operating cost of the energy system is inferior to this embodiment; Compared with RL baseline methods such as Method d, g, and h, this embodiment establishes the temporal correlation between states to fully exert the optimization effect of RL and obtains lower operating costs; This embodiment outperforms Method e in the tests under three scenarios, which indicates that compared with traditional hidden state representation methods, using neural Kalman for state representation is more suitable for industrial data modeling; Although compared with Method f, the improvement of this embodiment in terms of cost is limited, it can greatly reduce the costs brought by frequent scheduling. In addition, the cost variance of this embodiment is small, indicating that it can better adapt to different working conditions such as energy generation, consumption, and conversion. The above results prove the effectiveness of the time-event dual-scale collaborative calculation method in this embodiment in establishing the potential dynamic connection between states before and after the event occurs, improving the convergence of the RL strategy and the long-term return perception ability, thus bringing higher economic benefits.

[0128] Table 1: Comparison Results of Operating Costs under Different Scenarios

[0129]

[0130] Table 2: Comparison Results of Daily Operating Costs

[0131]

Claims

1. An event-triggered energy system optimization method based on a time-event dual-scale framework, characterized in that Including: Firstly, construct the optimization problem of the operation of the industrial energy system; Secondly, construct the elements of reinforcement learning including state, action and reward to build a simulation environment; Thirdly, based on the neural Kalman model, obtain the state representation with explicit time distribution characteristics; furthermore, use the proximal policy optimization method to construct a reinforcement learning framework, and jointly construct and train a time-driven optimization model based on the hidden state representation with the simulation environment and the neural Kalman model; finally, use another reinforcement learning framework and the simulation environment to construct an event-driven optimization model, and combine it with the neural Kalman model through a time-varying state matrix link to construct and train an event-driven optimization model based on the state transition matrix link, and finally realize event-triggered optimal scheduling through time-event dual-scale collaborative calculation.

2. The event-triggered energy system optimization method based on the time-event dual-scale framework according to claim 1, wherein Specifically, it includes the following steps: Step 1, construct the optimization problem of the operation of the industrial energy system: According to the optimization goal of the industrial energy system and the constraint conditions that need to be considered in actual operation, construct the optimization problem of the operation of the industrial energy system, and the optimization variable is the flow rate of the energy medium input to each adjustable device in the industrial energy system; Step 2, construct the elements of reinforcement learning: Construct reinforcement learning elements according to the industrial energy system optimization problem to build a simulation environment. The reinforcement learning elements include: state , action and immediate reward ; among them, the state contains the energy data in the industrial energy system; the action is the adjustment amount of the energy medium flow rate input to each adjustable device in the industrial energy system; the immediate reward is composed of the objectives and constraints in the optimization problem in step 1; Step 3, construct the hidden state representation based on the neural Kalman model: Construct a hidden state representation based on the Neural Kalman model, starting from the state formed time series, and calculate the hidden state with explicit time distribution characteristics, i.e., Markov state transition characteristics, through the Neural Kalman filter , to form a hidden state sequence; The neural Kalman model is , where is described by the iterative process of the Kalman filter, and the iterative process is shown in Equation (1), is the hidden state error covariance matrix, is the hidden state transition matrix, is the state observation matrix, is the covariance matrix of the observation noise, is the hidden state noise covariance matrix; （1） In the formula, represents the Kalman gain, represents the identity matrix, represents the matrix transpose; Parameters in the neural Kalman model and are determined by neural network learning; since each state in the hidden state sequence follows a class-linear relationship as in Equation (3), therefore, a long short-term memory network is used to calculate and , while and the neural network weights in the LSTM are used as the model parameters of the neural Kalman model and are updated in the subsequent learning process; （3） Step 4, construct a time-driven optimization model based on the hidden state representation: Use the proximal policy optimization method to construct a reinforcement learning framework. The constructed reinforcement learning framework includes a policy network and a value network. The policy network includes a new policy network and an old policy network; jointly construct and train a time-driven optimization model based on the hidden state representation with the simulation environment built in Step 2 and the neural Kalman model constructed in Step 3. Specifically: First, starting from the state calculate the hidden state with Markov state transition characteristics through the neural Kalman model , the hidden state generates an action through the old policy network ; After that, the simulation environment, according to the action , transforms the current state into the state at the next moment and obtains the immediate reward , completing one interaction. Then, it feeds back to the neural Kalman model to generate the hidden variable at the next moment ; After interactions complete an interaction cycle, a set of interaction data is obtained, the reward functions of the new policy network and the evaluation network are defined, the new policy network and the evaluation network are trained based on the interaction data, and the policy parameters and the evaluation network parameters are updated using the gradient descent method, where the policy parameters include the parameters of the new policy network , as well as the parameters of the neural Kalman model in step 3 and ; After a parameter update is completed, the updated new policy network parameters are passed to the old policy network one by one; after the old policy network is updated, it is used for data collection in the next interaction cycle, generating new interaction data and performing the next parameter update; the immediate rewards are continuously accumulated during the interaction process , when the accumulated immediate reward value converges, the training is completed and the optimal parameters are determined. Based on the trained time-driven model, the optimal action can be automatically selected according to the current state of the industrial energy system collected in real time as the time-driven scheduling strategy; Step 5, construct an event-driven optimization model based on the state transition matrix link: Use another reinforcement learning framework and the simulation environment to construct an event-driven optimization model. Among them, the simulation environment is constructed according to Step 2, and the reinforcement learning framework has the same structure as the reinforcement learning framework of the time-driven optimization model based on the hidden state representation in Step 4; through the time-varying state transition matrix link, combine the neural Kalman state representation with the event-driven model to construct an event-driven optimization model based on the state transition matrix link for obtaining the event-driven state, and obtain the optimized event-driven policy through time-event dual-scale policy collaborative calculation. Specifically: First, the time-driven optimization model based on the hidden state representation after training in step 4 calculates the time-driven scheduling policy in a rolling manner at a fixed time period but does not execute it; When a scheduling event is triggered, according to the event-driven state at the time when the previous scheduling event occurred , combining the time-driven state between the previous scheduling event and the current scheduling event, calculate the event-driven state after the current scheduling event occurs ; The calculation process is as follows: Assume that the previous scheduling event is separated from the current scheduling event by time periods , then according to the formula, we can obtain: （13） Without disrupting the time-varying relationship of the state variables, when the event is triggered, multiply both the left and right sides of the system of equations in Equation (13) to obtain , so as to synthesize state information for adjacent time periods; because , let , and obtain the event-driven state transition relationship, which is the state transition matrix link: （14） Input the old policy network in the event-driven optimization model to generate actions and input the obtained into the simulation environment of the event-driven optimization model to obtain the state and the immediate reward ; input the feedback into the time-driven optimization model based on the hidden state representation to update its corresponding state . The time-driven optimization model based on the hidden state representation continues to perform rolling calculations at a fixed time period until the next scheduling event occurs, realizing the collaborative calculation of the time-event dual-scale policy; for the occurred scheduling event, obtain groups of interaction data, and use this groups of interaction data to train the event-driven optimization model, and update its new policy network parameters, old policy network parameters, and evaluation network parameters. The parameter update process is the same as that in step 4; Define the reward of the event-driven optimization model as , where the time-driven reward value is the sum of to within the range of , and the event-driven reward value is the sum of the immediate rewards to of the event-driven optimization model within the range; continuously accumulate during the interaction process . When the accumulated converges, complete the training and determine the optimal parameters of the event-driven model; The trained event-driven model based on the state transition matrix link and the trained time-driven optimization model based on the hidden state representation jointly realize time-event dual-scale collaborative calculation, and can calculate the optimal action when the corresponding event occurs according to the current state collected in real time by the industrial energy system as the event-driven scheduling strategy.

3. The event-triggered energy system optimization method based on a time-event dual-scale framework according to claim 2, wherein In the above-mentioned Step 1, the optimization goal is to minimize the economic operation cost within a specific time range, and the constraint conditions include energy balance constraint, mass balance constraint, equipment safe operation constraint and equipment output limit.

4. The event-triggered energy system optimization method based on a time-event dual-scale framework according to claim 2, wherein In the said step 3, and The calculation formula is as shown in Equation (2), and its inputs are and , is the internal hidden layer variable of the long short-term memory network at time instant; （2）。 5. The event-triggered energy system optimization method based on a time-event dual-scale framework according to claim 2, wherein In the aforementioned step 4, the process of the old policy network generating an action is as follows: Actions generated by the old policy network are multi-dimensional actions and follow a Beta distribution, i.e., , where represents the probability distribution of the old policy network selecting action when the input is . The distribution parameters and are approximated by the neural layers and that make up the policy network, respectively. Then each variable in is sampled according to the probability distribution generated by the old policy network. The specific sampling formula is as follows: （6） Among them, represents the dimension of the action , , represents the number of probability distributions output by the policy network is the Beta distribution perturbation.

6. The event-triggered energy system optimization method based on the time-event dual-scale framework according to claim 5, characterized in that, Distribution parameter and The calculation formula is as follows: （4） （5） Among them, is the activation function , is the activation function ; and are respectively the parameter variables of the old policy network, is the maximum action range.

7. An event-triggered energy system optimization method based on a time-event dual-scale framework according to claim 2, characterized in that In the above-mentioned Step 4, the parameter update process is as follows: Define the reward function of the new policy network as: （7） （8） Among them, represents the probability distribution generated by the old policy network of calculating the expected value of the reward of the new policy network, is the importance sampling ratio of the new and old policy networks at time is the clipping mechanism in the proximal policy optimization method, indicating to ensure within the range, is a hyperparameter for controlling the clipping degree, represents the probability that the new policy network selects an action in the generated distribution of represents the probability that the old policy network selects an action in the generated distribution of is the advantage function, approximated by the temporal difference method; The evaluation network adopts a multi-layer perceptron network Approximately, that is , define the reward function of the evaluation network as: （10） （11） （12） Among them, is the value target value, represents from the current time step starting, to the interaction cycle the th time step, is the error at the time step error, represents the discount factor.

8. An event-triggered energy system optimization method based on a time-event dual-scale framework according to claim 7, characterized in that Advantage function The calculation formula is as follows: （9） Among them, represents the expected cumulative reward that can be obtained by taking action under the state while following the distribution and can be approximated by the temporal difference method as , where represents the temporal difference, and represents the Monte Carlo estimate.

9. The event-triggered energy system optimization method based on the time-event dual-scale framework according to claim 2, characterized in that In step 5 described above, when the event does not occur, is calculated by the simulation environment in the event-driven model according to When the event occurs, is calculated by the simulation environment in the event-driven model according to the action taken and obtained.

Images