Regional cross-seasonal heat storage system regulation and control strategy based on reinforcement learning method

Abstract

The invention provides a regional cross-seasonal heat storage system regulation and control strategy based on a reinforcement learning method, and relates to the technical field of intelligent control, and the method comprises the steps: collecting the operation data of a regional cross-seasonal heat storage system; according to the operation data, a system operation data set is established, and the system operation data set comprises operation state data and system performance indexes; according to the system operation data set, training to obtain a neural network agent model; and adopting a reinforcement learning method to be combined with the neural network agent model to obtain a cross-seasonal heat storage system regulation and control strategy. The technical problems that in the prior art, in a cross-seasonal energy storage system, the operation parameter regulation and control mode is single, and systematic evaluation and optimization are lacked are solved. The technical effects that accurate prediction and real-time optimization control are carried out on the cross-seasonal heat storage system, and the comprehensive energy efficiency of the cross-seasonal heat storage system is improved are achieved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and in particular to a regional cross-seasonal heat storage system control strategy based on a reinforcement learning method. Background Art

[0002] Existing cross-seasonal energy storage systems usually have certain limitations in the regulation of operating parameters, which is manifested in a relatively single regulation method. Most existing systems rely only on independent control modules to adjust certain specific parameters, but the coupling and mutual influence between these parameters are often not fully considered. Due to the lack of a comprehensive assessment of the overall operating status of the system, these systems may appear too one-sided in their control strategies, resulting in low control efficiency and difficulty in achieving the global optimal regulation effect.

[0003] In addition, some systems rely on human intervention to adjust parameters. Although this method can solve control problems in certain specific situations, it may reduce the effectiveness of overall control due to the limitations of human judgment and the lag of real-time response. In practical applications, this single and incomplete control method may cause the system to fail to fully utilize the potential of the energy storage system when operating in different seasons, reducing the overall performance and energy efficiency of the system. Therefore, the shortcomings of the existing control strategy not only affect the stability and reliability of the cross-seasonal energy storage system, but may also increase energy consumption and operating costs in long-term operation.

[0004] The existing technology has technical problems in the cross-season energy storage system, such as a single operating parameter control method and a lack of systematic evaluation and optimization. Summary of the invention

[0005] The present application provides a regional inter-seasonal heat storage system control strategy based on a reinforcement learning method, which is used to solve the technical problems of the existing technology in the inter-seasonal energy storage system, such as the single operating parameter control method and the lack of systematic evaluation and optimization.

[0006] In view of the above problems, the present application provides a regional inter-seasonal heat storage system control strategy based on a reinforcement learning method, the method comprising: collecting operating data of the regional inter-seasonal heat storage system; establishing a system operation data set based on the operating data, the system operation data set comprising operating status data and system performance indicators; training a neural network proxy model based on the system operation data set; and using a reinforcement learning method in combination with the neural network proxy model to obtain a cross-seasonal heat storage system control strategy.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] 1. By introducing reinforcement learning algorithms, the system's multiple parameters are optimized and regulated in real time, enabling the system to maintain the best operating state under different seasons and load conditions, effectively improving the system's overall energy efficiency and achieving the goal of energy conservation and emission reduction.

[0009] 2. Traditional control methods are difficult to handle the complex coupling relationship between multiple parameters in the system. The present invention constructs the state space and action space, so that the intelligent agent can find the optimal control strategy under complex multi-parameter coupling conditions, effectively avoiding the limitations brought by single parameter regulation and achieving the global optimal control effect.

[0010] 3. Compared with traditional control methods, the adaptability and response speed of the regional cross-seasonal heat storage system have been significantly improved, reducing the volatility and uncertainty in operation.

[0011] 4. The control method of the present invention has good scalability, and the state space, action space and reward function in the system can be adjusted according to actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A flow chart of a regional cross-seasonal heat storage system control strategy based on a reinforcement learning method provided in this application;

[0014] Figure 2 A schematic diagram of the process of obtaining a neural network agent model in a regional inter-seasonal heat storage system control strategy based on a reinforcement learning method provided in this application. DETAILED DESCRIPTION

[0015] This application provides a regional cross-seasonal heat storage system control strategy based on reinforcement learning method, which is used to solve the technical problems of single operation parameter control method and lack of systematic evaluation and optimization in the existing cross-seasonal energy storage system. It achieves the technical effect of accurate prediction and real-time optimization control of the cross-seasonal heat storage system and improves the comprehensive energy efficiency of the cross-seasonal heat storage system.

[0016] Below, the technical solutions in the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments described herein. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all of them.

[0017] like Figure 1 As shown, the present application provides a regional inter-seasonal heat storage system control strategy based on a reinforcement learning method, the method is applied to a regional inter-seasonal heat storage system, the system includes a wind-solar green power module, a carbon dioxide heat pump module, an inter-seasonal buried pipe heat storage module and a waste heat recovery module, the method includes:

[0018] Collect operating data of the cross-seasonal heat storage system in the region.

[0019] A system operation data set is established according to the operation data, and the system operation data set includes operation status data and system performance indicators.

[0020] Furthermore, based on the operating data, a system operating data set is established, including: obtaining the operating data of the wind-solar green power module, the carbon dioxide heat pump module, the cross-seasonal buried pipe heat storage module and the waste heat recovery module respectively according to the system monitoring platform; wherein the operating data of the wind-solar green power module includes the green power supply mode and the power output power; the operating data of the carbon dioxide heat pump module includes the carbon dioxide working fluid flow rate, the compressor speed, and the compression / air cooling temperature; the operating data of the cross-seasonal buried pipe heat storage module includes the buried pipe water inlet temperature and the buried pipe water inlet rate; the operating data of the waste heat recovery module includes the waste heat recovery working fluid flow rate, the waste heat recovery working fluid temperature, and the waste heat supply mode; the system performance indicators are calculated based on the operating data; the operating data and the system performance indicators are integrated to obtain the system operation data set.

[0021] Specifically, the regional inter-seasonal heat storage system includes four major modules, namely, the wind and solar green electricity module, the carbon dioxide heat pump module, the inter-seasonal buried pipe heat storage module and the waste heat recovery module. The wind and solar green electricity module mainly converts clean energy such as wind and solar into electricity. The electricity produced first meets the demand for electricity for daily life. The surplus electricity is used to power the components in the system, and the further surplus electricity can be connected to the power grid for power supply. In the wind and solar green electricity module, the main parameters that need to be regulated include but are not limited to the power supply mode and power output power. The carbon dioxide heat pump system module is mainly based on the carbon dioxide heat pump. In spring, summer and autumn, the heat in the building area is stored underground, and in winter, the working state is switched to transfer the underground heat to the building heating. The main parameters that need to be regulated are the carbon dioxide working fluid flow, compressor speed, compression / air cooling temperature, etc. The inter-seasonal buried pipe heat storage module is mainly used for heat storage, and the main parameters that need to be regulated are the water inlet temperature and water inlet rate of the buried pipe. The waste heat recovery module mainly recovers waste heat in the building through the heat exchanger arranged inside the building, and heats the building or incorporates it into the buried pipe system to store heat for demand. The main parameters that need to be controlled are the recovery working fluid flow rate, temperature and waste heat supply mode.

[0022] Based on this, according to the existing system monitoring platforms, such as industrial automation monitoring platform, heat storage system monitoring platform, etc., the operation data of the regional cross-seasonal heat storage system is extracted, that is, the operation parameter data of the wind and solar green power module, carbon dioxide heat pump module, cross-seasonal buried pipe heat storage module and waste heat recovery module of the regional cross-seasonal heat storage system are extracted respectively. The operation data of the wind and solar green power module includes the green power supply mode and power output power, the operation data of the carbon dioxide heat pump module includes the carbon dioxide working fluid flow rate, compressor speed and compression / air cooling temperature, the operation data of the cross-seasonal buried pipe heat storage module includes the buried pipe water inlet temperature and buried pipe water inlet rate, and the operation data of the waste heat recovery module includes the waste heat recovery working fluid flow rate, waste heat recovery working fluid temperature and waste heat supply mode. Then, according to the operation data extracted from each module, the system performance index is calculated and obtained. The system performance index is used to reflect the energy efficiency of the regional cross-seasonal heat storage system, which is obtained by comparing the heat energy output of the system with the electrical energy consumed in a certain period of time. Finally, the operation data of each module of the system and the system performance index are integrated to form a system operation data set. By integrating the operating parameter data of different modules in the regional cross-seasonal heat storage system, comprehensive and accurate data information is provided for subsequent analysis, thereby improving the accuracy and reliability of system control strategy decisions.

[0023] The system runs a data set and trains a neural network proxy model.

[0024] Specifically, the obtained system operation data set is preprocessed, including removing noise, filling missing values, etc., and then the preprocessed data is standardized to ensure that the numerical ranges of different features are consistent. Select a neural network model architecture suitable for predicting inter-seasonal heat storage systems, such as a multi-layer perception model. The standardized system operation data set is used as training data to train the neural network model to obtain a neural network proxy model that can predict the system state and performance. The neural network proxy model obtained through training can predict the comprehensive performance of the current system based on the different state data of the inter-seasonal heat storage system, and provide support for the intelligent regulation of the inter-seasonal heat storage system.

[0025] In one embodiment, Figure 2 As shown, according to the system operation data set, a neural network proxy model is trained, including: reducing the dimension of the operation status data by combining the system performance indicators through the principal component analysis method; training the neural network model by using the reduced-dimensional operation status data and the system performance indicators as training data; evaluating the performance of the neural network model through the mean square error; and obtaining the neural network proxy model when the performance of the neural network model meets the preset conditions.

[0026] Specifically, after preprocessing and standardizing the system operation data set, the operating status data in the system operation data set is reduced in dimension by the principal component analysis method (PCA) to obtain the operating parameters that have a greater impact on the size of the system performance indicators in the regional cross-seasonal heat storage system, and eliminate the parameters with less impact. Then, the operating status data and the system performance indicators after dimensionality reduction are divided into a training set and a test set for training and verifying the accuracy of the model. The neural network model is trained with the training data. The input of the neural network model is the operating status data after dimensionality reduction, and the output is the predicted value of the system performance indicator. The neural network model is obtained by adjusting the structure of the neural network (number of layers, number of neurons, type of activation function) and hyperparameters (learning rate, batchsize, number of training rounds, etc.). During the training process, in order to evaluate the predictive ability of the neural network, the performance of the neural network model is evaluated by the mean square error (MSE), and the MSE is defined as follows: Among them, y i is the actual value, is the predicted value of the neural network, and n is the number of samples. Through continuous iteration, the neural network model will gradually reduce the MSE value, so that the prediction accuracy of the model will gradually improve. After each iteration cycle of training, the trained neural network model is tested for accuracy using the test set. The closer the fit between the data predicted by the neural network and the test set data is to 1, the better the prediction accuracy is. When the performance of the neural network model meets the preset accuracy requirements on both the training set and the test set, that is, the error on the test set reaches a certain predefined threshold, and the error no longer decreases significantly in multiple iterations, it is considered that the prediction performance of the model has reached the optimal level, indicating that the model can accurately reflect the relationship between the operating status and performance of the inter-seasonal heat storage system. When the performance of the neural network model meets the preset conditions, the training process ends, and a neural network proxy model that can accurately predict system performance is obtained, providing support for the intelligent regulation of the inter-seasonal heat storage system.

[0027] The reinforcement learning method is combined with the neural network agent model to obtain the cross-seasonal heat storage system control strategy.

[0028] Specifically, reinforcement learning is an algorithm that learns how to make decisions through interaction with the environment, and is suitable for the optimal control of dynamic complex systems. In the inter-seasonal heat storage system, reinforcement learning can autonomously learn and improve the control strategy through continuous interaction with the system, and ultimately optimize the system energy efficiency. First, through the system operation data set, a neural network agent model was trained, and the performance of the inter-seasonal heat storage system was preliminarily simulated through the neural network agent model, and a basis for environmental interaction was provided for reinforcement learning. The output of the neural network agent model is used to predict the state changes and performance indicators of the system as a basis for state evaluation during the reinforcement learning process. Then, the reinforcement learning method is combined with the neural network agent model to obtain a control strategy that can dynamically optimize the operation of the inter-seasonal heat storage system, ensuring that the system can maintain efficient and stable performance under changing operating conditions, greatly improving energy efficiency and reducing operating costs, and improving the adaptability and response speed of the regional inter-seasonal heat storage system.

[0029] Furthermore, a reinforcement learning method is used in combination with the neural network agent model to obtain a cross-seasonal heat storage system control strategy, including: obtaining a state space of the states of multiple modules in a regional cross-seasonal heat storage system; obtaining an action space for the control of multiple modules in the regional cross-seasonal heat storage system; defining a reward function for the cross-seasonal heat storage system based on the state space and the action space; obtaining a target Q value based on the reward function, and evaluating the accuracy of the Q function based on the loss function; using the gradient ascent method to update the strategy network parameters; through multiple iterative training, when the training reaches the preset conditions, the control strategy of the cross-seasonal heat storage system is obtained.

[0030] Specifically, when combining the reinforcement learning method, the state space, action space and reward function in reinforcement learning are first defined. The state space refers to the set of all possible states of the current state of the environment, including the current operating parameters of each module of the cross-seasonal heat storage system. The action space refers to the set of all possible actions that the agent can take, including the actions regulated in each state of each module. The reward function defines the changes in system performance indicators after taking a certain action in different states, which is used to evaluate the effect of the agent taking a certain action in a specific state. The reward function provides feedback for policy optimization. During the training process of reinforcement learning, the agent interacts with the system (simulated by the neural network agent model) for multiple iterations. At each iteration, the agent observes the current system state, selects an action based on the current strategy, and then predicts the state change of the system after executing the action through the neural network agent model, and calculates the immediate return according to the reward function. The reinforcement learning agent evaluates the effect of each action by repeatedly trying different actions and gradually improves its decision-making strategy. In order to optimize the strategy, the target Q value is obtained according to the reward function to evaluate the future cumulative return after selecting a certain action in each state. In order to ensure the accuracy of the agent's estimate of the return for each state-action pair, the accuracy of the Q function is evaluated based on the loss function. Further, the policy network is optimized and the policy network parameters are updated by the gradient ascent method. The goal is to maximize the Q value so that the policy network can output the optimal action. Through multiple iterations of training, the agent continuously interacts with the cross-seasonal heat storage system and continuously optimizes its strategy. In each iteration, the agent selects an action according to the state, observes the state changes after the action is executed, and updates the strategy and Q value function according to the reward function until the training reaches the preset conditions. The preset conditions include strategy convergence, the reward function tends to be stable or the maximum number of iterations. When the training reaches the preset conditions, the optimized cross-seasonal heat storage system control strategy is obtained. By constructing the state space and action space, the agent can find the optimal control strategy under complex multi-parameter coupling conditions, achieve the global optimal control effect, and dynamically adjust the various parameters of the system in actual operation, so that the system can maintain the best operating state under different seasons and load conditions, and improve the overall energy efficiency of the system.

[0031] Further, the state space of the states of multiple modules in the regional cross-seasonal heat storage system is obtained, including: constructing the state space according to the operation data, the state space is a first multidimensional vector, and the state space matrix s is expressed as:

[0032] Among them, λ elec Characterize the green electricity supply mode, P elec Characterizes the electrical output power, Characterizes the carbon dioxide working fluid flow rate, v comp Characterizes the compressor speed, Tcon / cool Characterizes the compression / air cooling temperature, T in Characterizes the water inlet temperature of the buried pipe, Characterizes the water inflow rate of the buried pipe, v heat Characterize the flow rate of waste heat recovery medium, T hea Characterizes the temperature of the waste heat recovery medium, λ heat Characterize the waste heat supply mode.

[0033] Specifically, in the inter-seasonal heat storage system, the state space should include all operating parameters that affect the system performance. Specifically, the state space can be represented as a multidimensional vector, including but not limited to the following parameters: the power supply mode λ of the green power system elec , power output P elec , Working fluid flow in carbon dioxide heat pump system Compressor speed v comp , Evaporation / air cooling temperature T con / cool , water inlet temperature T of inter-seasonal underground pipe heat storage system in , underground pipe water inflow rate Working fluid flow rate v of waste heat recovery system heat , Waste heat recovery medium temperature T hea , Waste heat supply mode λ heat Etc. Therefore, the state space matrix is ​​expressed as:

[0034] These parameters form a high-dimensional state space, and each state represents the specific state of the system at a certain moment. The definition of the state space directly affects the effect of reinforcement learning. The more complete the state space is, the richer the system information that the reinforcement learning algorithm can obtain, thereby making better decisions and optimizing system performance.

[0035] Furthermore, the action space of the regulation of multiple modules in the regional inter-seasonal heat storage system is obtained, including: defining the action space of the inter-seasonal heat storage system according to the regulation demand of the inter-seasonal heat storage system, the action space is a second multidimensional vector, and the expression of the action space matrix a is: Among them, Δλ elec Characterize and adjust the green power supply mode, ΔP elec Characterizes and adjusts the electrical output power, Characterize the change of CO2 working fluid flow rate, Δv comp Characterizes the adjustment of compressor speed, ΔT con / cool Characterizes the set compression / air cooling temperature, ΔT in Characterize and regulate the water inlet temperature of the buried pipe. Characterizes and regulates the water inflow rate of the buried pipe, Δv heat Characterize the modified waste heat recovery system working fluid flow rate, ΔT heat Characterize and adjust the temperature of the waste heat recovery medium, Δλheat Characterize and regulate waste heat supply mode.

[0036] Specifically, the action space represents all possible actions that the agent can take in different states. In the inter-seasonal heat storage system, the action space usually includes the following operations: adjusting the power output, changing the CO2 working fluid flow, adjusting the compressor speed, setting the compression / air cooling temperature, adjusting the water inlet temperature and rate of the buried pipe, and modifying the working fluid flow rate, temperature and supply mode of the waste heat recovery system. The action space can also be expressed in vector form, and each element represents a specific control action: The expression of the action space matrix a is:

[0037]

[0038] Among them, Δλ elec Characterize and adjust the green power supply mode, ΔP elec Characterizes and adjusts the electrical output power, Characterize the change of CO2 working fluid flow rate, Δv comp Characterizes the adjustment of compressor speed, ΔT con / cool Characterizes the set compression / air cooling temperature, ΔT in Characterize and regulate the water inlet temperature of the buried pipe. Characterizes and regulates the water inflow rate of the buried pipe, Δv heat Characterize the modified waste heat recovery system working fluid flow rate, ΔT heat Characterize and adjust the temperature of the waste heat recovery medium, Δλ heat Characterize and regulate waste heat supply mode.

[0039] Further, according to the state space and the action space, a reward function of the cross-seasonal heat storage system is defined, including: according to the system performance index, the state space and the action space, a reward function r(s,a) is defined, and the reward function expression is: r(s,a)=COP(s')-COP(s) / C energy ; where s' represents the new state after taking action a, s represents the state before taking action a, COP(s') represents the system performance index of the new state s', COP(s) represents the system performance index of the system in state s, C energy Characterizes energy consumption and is used to normalize the reward value. The larger the reward value r(s,a), the better the system performance will be by adjusting action a.

[0040] Specifically, in the inter-seasonal heat storage system, the reward function is directly related to the system performance index (COP). In order to enable the agent to select actions that can improve the system performance, the reward function is defined as the increment of the system performance index: r(s,a)=COP(s')-COP(s) / C energy; where s' represents the new state after taking action a, s represents the state before taking action a, COP(s') represents the system performance index of the new state s', COP(s) represents the system performance index of the system in state s, C energy Characterizes energy consumption and is used to normalize the reward value. The larger the reward value r(s,a), the better the system performance will be by adjusting action a. The reward function is the difference between the COP values ​​corresponding to the two states. Reinforcement learning needs to decide the action that maximizes the reward function r(s,a) before each action is taken, and the overall efficiency of the system is improved by maximizing the reward function.

[0041] Further, a target Q value is obtained according to the reward function, and the Q function is accurately evaluated based on the loss function, including: calculating the target Q value according to the reward function, wherein the target Q value is calculated as follows: t =r t +γQ′(s t+1 ,μ′(s t+1 )); where r t Represents the current state t Next, perform action a t The immediate reward obtained after the event; γ represents the discount factor, with a value range of 0≤γ<1, which is used to balance the importance of immediate rewards and future rewards; Q′(s t+1 ,μ′(s t+1 )) represents the Critic network in the next state s t+1 Complete the next action a t+1 The obtained Q value; the accuracy of the Q value is evaluated through the loss function, where the loss function expression is: Among them, θ Q is the parameter of the Critic network, representing the weight and bias of the Q function, N represents the number of samples in the mini-batch, Q(s I ,a i |θ Q ) represents the current state of the Critic network s I and action a i Next, using the current parameter θ Q Estimated Q value; update the weight of the Critic network by minimizing the difference between the Critic network predicted Q value and the target Q value.

[0042] Specifically, after obtaining whether a certain action helps improve system performance based on the reward function, the future cumulative return is estimated based on the Bellman equation of reinforcement learning, and the target Q value is calculated as follows: t =r t +γQ′(s t+1 ,μ′(s t+1 )); where rt Represents the current state t Next, perform action a t The immediate reward obtained after the event; γ represents the discount factor, with a value range of 0≤γ<1, which is used to balance the importance of immediate rewards and future rewards; Q′(s t+1 ,μ′(s t+1 )) represents the Critic network in the next state s t+1 Complete the next action a t+1 The obtained Q value is used to evaluate the effect of the current action and predict the return of the future execution strategy. That is, the Q function is a function used to evaluate the return of the current state and action. The accuracy of the Q value is crucial for the intelligent agent to make the best decision. In order to ensure the accuracy of the Q value estimation, the loss function is used to measure the actual target Q value y t And the Q value prediction Q(s) of the Critic network I ,a i |θ Q ). The loss function expression is: Among them, θ Q is the parameter of the Critic network, representing the weight and bias of the Q function, N represents the number of samples in the mini-batch, Q(s I ,a i |θ Q ) represents the current state of the Critic network s I and action a i Next, using the current parameter θ Q The estimated Q value can measure the difference between the predicted Q value and the actual target Q value through the loss function. The smaller the loss, the more accurate the Q value estimation is and the closer the agent's strategy is to the optimal. By minimizing the difference between the Q value predicted by the Critic network and the target Q value, the weight of the Critic network is updated, the strategy is gradually optimized, and the action that can maximize the long-term return is selected to improve the accuracy of the control strategy.

[0043] Furthermore, the gradient ascent method is used to update the policy network parameters, including: optimizing the policy network parameters by the gradient ascent method, and the gradient is calculated as follows:

[0044] in, Characterize the parameters θ of the Actor network μ Find the gradient of the objective function J, Represents the gradient of the Q-value function of the Critic network with respect to action a, a = μ(s i |θ μ ) represents the Actor network according to the current state s iThe output action is input into the Critic network to calculate the corresponding Q value. The policy function μ(s) that characterizes the Actor network i ) for the parameter θ μ gradient.

[0045] Specifically, the task of the policy network is to select the best action to take in each state, and optimize the policy based on the Q-value evaluation of the Critic network. The goal of the Actor network is to maximize the Q-value of the Critic network, that is, to find the action that maximizes the Q-value. The goal of updating the policy network is to maximize the Q-value of the Critic network through the gradient ascent method. The gradient is calculated as follows: in, Characterize the parameters θ of the Actor network μ Find the gradient of the objective function J, Represents the gradient of the Q value function of the Critic network relative to action a, indicating the impact of the selected action on the Q value, a = μ(s i |θ μ ) represents the Actor network according to the current state s i The output action is input into the Critic network to calculate the corresponding Q value. The policy function μ(s) that characterizes the Actor network i ) for the parameter θ μ The gradient of indicates how to adjust the parameters of the policy network so that the selected action can maximize the Q value. The weights of the policy network are updated by the gradient ascent method, and the action selection of the agent in each state is gradually optimized, so that it can select the action that can bring the maximum long-term reward, further improve the performance of the system, and optimize the control strategy.

[0046] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0047] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A regional cross-seasonal heat storage system control strategy based on reinforcement learning method, characterized in that: The method is applied to a regional inter-seasonal heat storage system, the system comprising a wind-solar green electricity module, a carbon dioxide heat pump module, an inter-seasonal buried pipe heat storage module and a waste heat recovery module, and the method comprises: Collecting the operation data of the inter-seasonal heat storage system in the region; Establishing a system operation data set according to the operation data, wherein the system operation data set includes operation status data and system performance indicators; Running the data set according to the system, training and obtaining a neural network proxy model; The reinforcement learning method is combined with the neural network agent model to obtain the cross-seasonal heat storage system control strategy.

2. A regional cross-seasonal heat storage system control strategy based on reinforcement learning method as claimed in claim 1, characterized in that: According to the operation data, a system operation data set is established, including: According to the system monitoring platform, the operation data of the wind-solar green power module, the carbon dioxide heat pump module, the inter-seasonal buried pipe heat storage module and the waste heat recovery module are respectively obtained; The operation data of the wind-solar green electricity module includes green electricity supply mode and power output power; The operation data of the carbon dioxide heat pump module includes carbon dioxide working medium flow rate, compressor speed, and compression / air cooling temperature; The operation data of the inter-seasonal buried pipe heat storage module includes the buried pipe water inlet temperature and the buried pipe water inlet rate; The operation data of the waste heat recovery module includes the waste heat recovery working fluid flow rate, the waste heat recovery working fluid temperature, and the waste heat supply mode; Calculate and obtain the system performance index according to the operating data; The operation data and the system performance index are integrated to obtain the system operation data set.

3. The regional cross-seasonal heat storage system control strategy based on reinforcement learning method according to claim 1 is characterized in that: According to the system running data set, a neural network agent model is trained and obtained, including: By combining the system performance index with the principal component analysis method, the operating status data is reduced in dimension; Using the reduced-dimensional operating status data and the system performance index as training data to train a neural network model; The performance of the neural network model was evaluated by the mean square error; When the performance of the neural network model meets the preset conditions, the neural network proxy model is obtained.

4. The regional cross-seasonal heat storage system control strategy based on reinforcement learning method according to claim 1 is characterized in that: The reinforcement learning method is combined with the neural network agent model to obtain the cross-seasonal heat storage system control strategy, including: Obtain the state space of multiple module states in the regional cross-seasonal heat storage system; Obtain the action space of multiple modules in the regional cross-seasonal heat storage system; According to the state space and the action space, a reward function of the cross-seasonal heat storage system is defined; Obtaining a target Q value according to the reward function, and evaluating the accuracy of the Q function based on the loss function; Use the gradient ascent method to update the policy network parameters; Through multiple iterative training, when the training reaches the preset conditions, the control strategy of the cross-seasonal heat storage system is obtained.

5. A regional cross-seasonal heat storage system control strategy based on reinforcement learning method as claimed in claim 4, characterized in that: The state space of multiple module states in the regional cross-seasonal heat storage system is obtained, including: The state space is constructed according to the operation data. The state space is a first multidimensional vector. The state space matrix s is expressed as: Among them, λ elec Characterize the green electricity supply mode, P elec Characterizes the electrical output power, Characterizes the carbon dioxide working fluid flow rate, v comp Characterizes the compressor speed, T con / cool Characterizes the compression / air cooling temperature, T in Characterizes the water inlet temperature of the buried pipe, Characterizes the water inflow rate of the buried pipe, v heat Characterize the flow rate of waste heat recovery medium, T hea Characterizes the temperature of the waste heat recovery medium, λ heat Characterize the waste heat supply mode.

6. A regional cross-seasonal heat storage system control strategy based on reinforcement learning method as claimed in claim 4, characterized in that: The action space of multiple modules in the regional cross-seasonal heat storage system is obtained, including: According to the regulation demand of the inter-seasonal heat storage system, the action space of the inter-seasonal heat storage system is defined. The action space is the second multidimensional vector, and the expression of the action space matrix a is: Among them, Δλ elec Characterize and adjust the green power supply mode, ΔP elec Characterizes and adjusts the electrical output power, Characterize the change of CO2 working fluid flow rate, Δv comp Characterizes the adjustment of compressor speed, ΔT con / cool Characterizes the set compression / air cooling temperature, ΔT in Characterize and regulate the water inlet temperature of the buried pipe. Characterizes and regulates the water inflow rate of the buried pipe, Δv heat Characterize the modified waste heat recovery system working fluid flow rate, ΔT heat Characterize and adjust the temperature of the waste heat recovery medium, Δλ heat Characterize and regulate waste heat supply mode.

7. A regional cross-seasonal heat storage system control strategy based on reinforcement learning method as claimed in claim 6, characterized in that: According to the state space and the action space, a reward function of the cross-seasonal heat storage system is defined, including: According to the system performance index, the state space and the action space, a reward function r(s,a) is defined, and the reward function expression is: r(s,a)=COP(s')-COP(s) / C energy ; Among them, s' represents the new state after taking action a, s represents the state before taking action a, COP(s') represents the system performance index of the new state s', COP(s) represents the system performance index of the system in state s, C energy Characterizes energy consumption and is used to normalize the reward value. The larger the reward value r(s,a), the better the system performance will be by adjusting action a.

8. A regional cross-seasonal heat storage system control strategy based on reinforcement learning method as claimed in claim 7, characterized in that: Obtaining a target Q value according to the reward function, and evaluating the Q function accurately based on the loss function, including: According to the reward function, the target Q value is calculated, where The target Q value is calculated as follows: y t =r t +γQ′(s t+1 ,μ′(s t+1 )); Among them, r t Represents the current state t Next, perform action a t The immediate reward obtained after the event; γ represents the discount factor, with a value range of 0≤γ<1, which is used to balance the importance of immediate rewards and future rewards; Q′(s t+1 ,μ′(s t+1 )) represents the Critic network in the next state s t+1 Complete the next action a t+1 The obtained Q value; The accuracy of the Q value is evaluated through the loss function, where the loss function expression is: Among them, θ Q is the parameter of the Critic network, representing the weight and bias of the Q function, N represents the number of samples in the mini-batch, Q(s I ,a i |θ Q ) represents the current state of the Critic network s I and action a i Next, using the current parameter θ Q Estimated Q value; The weight of the Critic network is updated by minimizing the difference between the Critic network predicted Q value and the target Q value.

9. A regional cross-seasonal heat storage system control strategy based on reinforcement learning method as claimed in claim 8, characterized in that: Use the gradient ascent method to update the policy network parameters, including: The policy network parameters are optimized by the gradient ascent method. The gradient is calculated as follows: in, Characterize the parameters θ of the Actor network μ Find the gradient of the objective function J, Represents the gradient of the Q-value function of the Critic network with respect to action a, a = μ(s i |θ μ ) represents the Actor network according to the current state s i The output action is input into the Critic network to calculate the corresponding Q value. The policy function μ(s) that characterizes the Actor network i ) for the parameter θ μ gradient.

Images