Heat storage system regulation and control method and system based on expert demonstration and deep reinforcement learning

Abstract

The invention relates to a heat storage system regulation and control method and system based on expert demonstration and deep reinforcement learning. The regulation and control method comprises the steps that a multi-dimensional state space used for describing the operation state of a heat storage system is constructed; training a control agent by adopting a deep reinforcement learning algorithm fused with expert demonstration data based on the multi-dimensional state space so as to learn to obtain an optimized control strategy; wherein the training process is oriented by a reward function which maximally and comprehensively reflects the multi-target performance of the system; the trained control agent is deployed in the heat storage system, a control instruction is fed back and output according to the real-time state of the heat storage system, and active regulation and control over the heat storage system are achieved. According to the method, the problem of power response delay caused by inherent thermal inertia of the heat storage system is effectively solved, the dependence of a traditional control strategy on an accurate mathematical model is eliminated, and therefore multi-target self-adaptive optimization operation aiming at a power grid dispatching instruction is achieved.

Description

Technical Field

[0001] This invention relates to the field of energy technology and automatic control, specifically to a method and system for regulating a thermal storage system based on expert demonstration and deep reinforcement learning. Background Technology

[0002] Building a new power system is the core path to achieving the "dual-carbon" strategic goal. With the high proportion of renewable energy connected to the grid, its randomness and intermittency pose a severe challenge to the power system's flexible adjustment capabilities. Against this backdrop, large-capacity thermal storage systems, with their large-scale, long-term storage capacity and superior life-cycle economics, are gradually becoming key infrastructure supporting stable grid operation and the absorption of new energy sources.

[0003] However, while providing large-scale energy storage, thermal energy storage systems suffer from significant dynamic delays in power response due to the inherent large thermal inertia of their heat storage / release processes. This physical characteristic makes it difficult for them to quickly and accurately track high-frequency, high-precision grid dispatch commands such as Automatic Generation Control (AGC), severely restricting their effective deployment in scenarios requiring high dynamic performance, such as frequency regulation and load tracking.

[0004] To overcome the dynamic response bottleneck caused by physical characteristics, adopting advanced control strategies to improve system performance has become a key approach. However, existing control methods still have many limitations in practical engineering applications: First, strategies relying on precise mathematical models (such as model predictive control) are prone to performance degradation due to model mismatch when facing the strong nonlinearity, large inertia, and complex thermo-electric coupling characteristics of thermal storage systems. Second, system operation requires real-time trade-offs between multiple objectives such as rapid response, economy, and equipment lifespan loss, and traditional PID or rule-based strategies are difficult to achieve dynamic optimization decisions under such complex constraints. Third, existing control architectures generally lack adaptive regulation capabilities, making it difficult to cope with uncertainties such as changes in system parameters and external environmental disturbances, and failing to guarantee the continuous effectiveness of control strategies.

[0005] Therefore, developing an intelligent control method that does not rely on a precise system model and has adaptive and multi-objective dynamic optimization capabilities has become an urgent technical requirement for fully tapping the flexible adjustment potential of large-capacity thermal storage systems and supporting the safe and stable operation of high-proportion new energy power systems. Summary of the Invention

[0006] The technical problem to be solved by this invention is to overcome the defects of the prior art and provide a control method for thermal storage systems based on expert demonstration and deep reinforcement learning. It can effectively solve the power response hysteresis problem caused by the inherent thermal inertia of thermal storage systems and eliminate the dependence of traditional control strategies on precise mathematical models, thereby realizing multi-objective adaptive optimization operation in response to power grid dispatching commands.

[0007] To solve the above-mentioned technical problems, the technical solution of the present invention is: a method for regulating a thermal storage system based on expert demonstration and deep reinforcement learning, comprising:

[0008] Construct a multidimensional state space to describe the operating state of the thermal storage system;

[0009] Based on the aforementioned multidimensional state space, a deep reinforcement learning algorithm that integrates expert demonstration data is used to train the control agent to learn an optimized control strategy; wherein, the training process is guided by maximizing the reward function that comprehensively reflects the multi-objective performance of the system;

[0010] The trained control agent is deployed in the thermal storage system and outputs control commands based on the real-time status feedback of the thermal storage system to achieve active regulation of the thermal storage system.

[0011] Furthermore, the multidimensional state space includes the state variables of the thermal storage system as well as external signals and demands; wherein,

[0012] The state variables of the thermal storage system include the current values ​​of the core physical parameters, the historical values ​​of at least some of the core physical parameters, and the rate of change of at least some of the historical values ​​of the core physical parameters.

[0013] The external signals and demands include grid power commands, real-time electricity prices, and forecasts of future short-term heat loads.

[0014] Furthermore, the core physical parameters include the temperature of the heat storage medium, the system pressure, the flow rate of the heat storage medium, and the heating power.

[0015] Furthermore, the deep reinforcement learning algorithm that integrates expert demonstration data is a demonstration-based deep deterministic policy gradient (DDPGfD) algorithm; the training process for the control agent includes:

[0016] Pre-training phase: Supervised learning of the Actor network of the control agent is performed using expert demonstration data from traditional controllers or historical operating data to initialize the network parameters;

[0017] Offline reinforcement learning phase: The pre-trained control agent is placed in a simulation environment and learns autonomously by interacting with the environment and based on the reward function to optimize its control strategy.

[0018] Furthermore, the reward function is a multi-objective composite function, and its output value is obtained by weighting the error term, economic operating cost term, equipment safety status term, and energy efficiency optimization term that reflect the tracking performance.

[0019] Furthermore, the error term of the tracking performance is expressed as: The calculation formula is:

[0020]

[0021] The economic operating cost item is expressed as: The calculation formula is:

[0022]

[0023] The device safety status item is represented as follows: The calculation formula is:

[0024]

[0025] The energy efficiency optimization term is represented as follows: The calculation formula is:

[0026]

[0027] In the formula, Indicates grid power command. Indicates the actual output power of the system. Indicates real-time electricity price. Indicates heating power. Indicates the scheduling time step. Indicates the temperature of the heat storage medium. This represents the safe temperature, which is a constant. This indicates the pump speed command in the thermal storage system. This indicates the energy consumption coefficient of the auxiliary equipment.

[0028] Furthermore, the action space of the control agent is a continuous space, and its output includes control signals for pumps, valves and heaters in the thermal storage system.

[0029] This invention also relates to a thermal storage system control system based on expert demonstration and deep reinforcement learning, comprising:

[0030] The data sensing unit is connected to the key measuring points of the thermal storage system and is used to collect physical parameters in real time that constitute a multi-dimensional state space for describing the operating status of the thermal storage system.

[0031] The intelligent decision-making unit is communicatively connected to the data sensing unit, and its core is a control agent deployed on a computing device. This unit is configured to receive state data from the data sensing unit, run the trained control agent, and generate corresponding control commands.

[0032] The control execution unit is communicatively connected to the intelligent decision-making unit, and is used to receive the control commands and drive the thermal storage system to perform corresponding adjustment actions.

[0033] Furthermore, the thermal storage system control system based on expert demonstration and deep reinforcement learning also includes a simulation training platform. The simulation training platform is used to run the digital model of the thermal storage system before deployment and to provide the control agent with a simulation environment for the pre-training stage and the offline training stage.

[0034] Furthermore, the simulation training platform and the thermal storage system together constitute a digital twin system.

[0035] By adopting the above technical solution, the present invention has the following beneficial effects:

[0036] 1. Model independence and strong robustness: The data-driven deep reinforcement learning mechanism adopted does not rely on the precise mathematical model of the system, and can effectively cope with the inherent strong nonlinearity and multivariate coupling characteristics of the thermal storage system, thus possessing stronger robustness and adaptability.

[0037] 2. Multi-objective self-optimization performance: Through a multi-objective composite reward function, the control agent can automatically weigh multiple objectives rapidly under complex operating conditions, achieving system-level multi-index self-optimization operation. Attached Figure Description

[0038] Figure 1 This is an overall framework diagram of the thermal storage system control system based on expert demonstration and deep reinforcement learning provided for embodiments of the present invention;

[0039] Figure 2 A two-stage training flowchart based on DDPGfD is provided for embodiments of the present invention;

[0040] Figure 3 This is a schematic diagram of the neural network structure of the control agent provided in an embodiment of the present invention;

[0041] Figure 4 A flowchart illustrating the online deployment and proactive control process provided in this embodiment of the invention;

[0042] In the diagram: 1. Cold salt tank; 2. Hot salt tank; 3. Molten salt pump; 4. Electric heater; 5. Steam generator; 6. Steam turbine; 7. Generator;

[0043] 110. Data perception unit; 120. Intelligent decision-making unit; 121. Data preprocessing module; 122. Model loading and inference module; 130. Control execution unit; 140. Simulation training platform. Detailed Implementation

[0044] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0045] This embodiment uses a molten salt thermal storage system equipped with an electric heater 4 as an example. This thermal storage system generates steam to drive a steam turbine generator set, mainly used for secondary frequency regulation and peak shaving of the power grid. Figure 1 As shown, the thermal storage system includes a cold salt tank 1, a hot salt tank 2, a molten salt pump 3, an electric heater 4, a steam generator 5, a steam turbine 6, a generator 7, and related valves and pipes.

[0046] The outlet of the cold salt tank 1 is connected to the inlet of the hot salt tank 2 via a pipeline. The primary side of the steam generator 5 is connected between the outlet of the hot salt tank 2 and the inlet of the cold salt tank 1 via a pipeline, and the secondary side is connected to a steam turbine to provide steam to the steam turbine 6. The steam turbine 6 is connected to a generator 7 to provide power for power generation. The electric heater 4 is used to heat the hot salt tank 2. The molten salt pump 3 is installed on the pipeline connecting the outlet of the hot salt tank 2 and the inlet of the primary side of the steam generator 5, and at least a portion of the pipeline is equipped with valves.

[0047] The thermal storage system control method based on expert demonstration and deep reinforcement learning includes the following steps:

[0048] Step S1: Construct a multidimensional state space

[0049] Multidimensional state space S t These variables, including the state variables of the thermal storage system and external signals and demands, collectively describe the system dynamics and control objectives:

[0050] Current value of core physical parameter: Molten salt temperature T at the outlet of hot salt tank 2 hot (t) (Measuring range: 290℃~565℃), molten salt temperature T at the outlet of cold salt tank 1 cold (t) (Measurement range: 290℃~565℃), molten salt circulation mass flow rateṁ salt (t) (Measuring range: 0~120kg / s), main steam pressure p main (t) (Measurement range: 5~16MPa), Current power P of electric heater 4 heater (t) (rated power 50MW).

[0051] Historical values ​​of core physical parameters: To overcome the large thermal inertia of the system, the values ​​of some core physical parameters in the previous sampling period (e.g., 10s) are introduced, i.e., T. hot (t-1), ṁ salt (t-1), p main (t-1) is used to capture system dynamics.

[0052] Rate of change of core physical parameters: Calculate the average rate of change of some key parameters within the sampling period, i.e. (T hot (t)- T hot (t-1)) / Δt and (p main(t)- p main (t-1)) / Δt is used to allow the controlling agent to perceive the dynamic trend of the system.

[0053] External signals and demands: Grid power command P ref (t) (unit: MW, range -50 ~ +50MW, negative value represents absorbed power), real-time electricity price C elec (t) (unit: yuan / kWh) and the predicted short-term heat load Q load,fst (t) (unit: MW).

[0054] This embodiment is based on a targeted design to address the control bottleneck of the thermal storage system (response hysteresis caused by large inertia): temperature (T) hot ) and pressure (p) main Flow rate (ṁ) is the core physical quantity with the greatest inertia in the system and the most significant impact on the rapid power response. Its historical values ​​and rate of change can directly quantify the system's thermal inertia, and it is a key input for intelligent agents to achieve "predictive" proactive control and overcome delays. salt The historical values ​​of the variables are mainly used to optimize economic efficiency (reflecting pump consumption and avoiding unnecessary frequent adjustments). Therefore, the above-mentioned state variables are selected for the multidimensional state space in this embodiment.

[0055] The above parameter settings are based on a 50MW power generation thermal storage system. Calculations show that, with a molten salt flow rate cap of 120 kg / s and a temperature drop of approximately 275°C (from 290°C to 565°C), the system's thermal power output can reach approximately 150 MW, matching the power generation capacity. The main steam pressure range covers the typical operating range of subcritical to supercritical turbine units.

[0056] Before being input into the control agent, all state variables are normalized using the following formula to map them to the interval [-1, 1], thereby accelerating the convergence of neural network training:

[0057]

[0058] In the formula, Represents the state variables before normalization. This represents the minimum value of the state variable. This represents the maximum value of the state variable. This represents the normalized state variable.

[0059] S2: Training the control agent based on the DDPGfD algorithm

[0060] The core of this embodiment lies in adopting, as follows Figure 2 The two-stage training framework shown is implemented on a simulation training platform. This platform establishes... Figure 1The accurate dynamic model of the system shown has mathematical models for each component (such as cold salt tank 1, hot salt tank 2, molten salt pump 3, electric heater 4, steam generator 5, steam turbine 6, and generator 7) that are consistent with the actual parameter range described in S1.

[0061] S2.1 Pre-training phase (imitation learning)

[0062] Expert data acquisition: In the simulation platform, an optimized traditional PID controller is run to track a series of power commands P covering typical operating conditions (such as step and ramp changes). ref During this process, 100,000 sets of "state-action" data pairs (s, a) were collected as expert demonstration data and stored in a dedicated expert experience playback buffer. Here, action a is the normalized [u] output of the PID controller. heater , u pump , u valve ] instruction, u heater , u pump , u valve These represent the power command for the electric heater, the speed command for the molten salt pump, and the opening command for the turbine intake regulating valve, respectively.

[0063] Actor Network Initialization: Construct the Actor network, its structure as follows: Figure 3 As shown: Input layer (13 neurons, corresponding to state vector S) t The network consists of three layers: a 128-neuron hidden layer (each with 128 neurons, using the ReLU activation function), an output layer (3 neurons, using the Tanh activation function, corresponding to three normalized control actions), and an output layer (3 neurons, corresponding to three normalized control actions). Using collected expert data and mean squared error (MSE) as the loss function, the Adam optimizer is used for supervised learning (training epochs: 500) to obtain a safe and effective initial control policy.

[0064] S2.2 Offline Reinforcement Learning Phase (Self-Evolution):

[0065] Environmental Interaction: The pre-trained control agent is placed in the simulation environment. At each time step t (sampling interval Δt = 10s), the control agent, based on the current state S... t And Ornstein-Uhlenbeck process noise is added to encourage exploration, output action a t .

[0066] Reward feedback: After the environment performs an action, it returns the immediate reward r(t) and the next state S. t+1 The multi-objective composite reward function designed in this embodiment The specific definitions are as follows:

[0067]

[0068] The definitions of each reward component are as follows:

[0069] , representing the error term of tracking performance, used to penalize tracking deviations in power generation (core objective);

[0070] This represents the economic operating cost item, used to reduce operating costs (economic efficiency) during periods of high electricity prices.

[0071] This indicates the equipment safety status item, used to severely penalize high-temperature operating conditions to ensure safety (security). T safe =550℃;

[0072] , represents the energy efficiency optimization term, which is used to encourage the reduction of auxiliary power consumption and the improvement of system energy efficiency, where k=0.01.

[0073] In the formula, Indicates grid power command. Indicates the actual output power of the system. Indicates real-time electricity price. Indicates heating power. Indicates the sampling interval. This indicates the temperature of the heat storage medium (the temperature of the molten salt at the outlet of hot salt tank 2). This represents the safe temperature, which is a constant. This indicates the pump speed command in the thermal storage system (molten salt pump speed command). This indicates the energy consumption coefficient of the auxiliary equipment.

[0074] Example of weighting coefficients: ω1=2.0, ω2=0.5, ω3=5.0, ω4=0.1, which can be adjusted according to the priority of the target.

[0075] This meticulously designed multi-objective composite reward function enables the control agent to automatically balance multiple objectives such as rapid tracking, economical operation, equipment safety, and system energy efficiency under complex operating conditions, achieving system-level multi-indicator self-optimization.

[0076] Network Update: The controlling agent randomly samples a small batch (batch size=128) of data from the comprehensive experience replay buffer (a 1:1 mixture of expert data and the controlling agent's own interaction data), using the Adam optimizer (Actor learning rate α). a =10 -4 Critic learning rate α c =10-3 Update the Critic and Actor networks. The loss function for the Critic network is the mean squared Bellman error. The target network uses a soft update method (smoothing constant τ = 0.005). This process is repeated until the average round reward of the control agent changes by less than 5% over 100 consecutive rounds, which is considered convergence.

[0077] The DDPGfD algorithm and two-stage training framework used in this step provide the agent with a high-performance initial strategy and safety paradigm using expert demonstration data, which greatly accelerates the training convergence speed and fundamentally avoids the safety risks in the early stage of training.

[0078] S3: Online Deployment and Active Control

[0079] The converged Agent model (mainly Actor network parameters) is solidified into an .onnx format file and deployed to the intelligent decision-making unit 120 of the actual thermal storage system. The closed-loop workflow of the system during online operation is as follows: Figure 4 As shown, the details are as follows:

[0080] S3.1 State Awareness and Signal Upload: The data sensing unit 110 synchronously acquires all the original physical signals (including T) required for the 13-dimensional state vector defined in S1 at a frequency of 10Hz. hot (t), p main (t-1), Q load,fst (t) etc.).

[0081] S3.2 Data Preprocessing: Raw data is transmitted to the intelligent decision-making unit 120 via an industrial network. Its internal data preprocessing module 121 first performs filtering, and then strictly follows the normalization formula and parameter upper and lower limits defined in S1 to convert the data into a standardized state vector S. t .

[0082] S3.3 Intelligent Decision Making: Standardized State Vector S t The data is fed into the model loading and inference module 122. This module loads the agent model trained by S2 and performs forward propagation calculations, generating the normalized optimal control command a within 50ms. t = [u heater (t), u pump (t), u valve (t)].

[0083] S3.4 Instruction Verification and Secure Execution: Control Instruction a tThe command is sent to the control execution unit 130. Its core command verification and safety limiting module immediately verifies and limits the command according to the preset safety parameter table (whose limits are consistent with the system physical limits in S1), ensuring that the final command output to the electric heater power controller, molten salt pump frequency converter and steam turbine inlet regulating valve is absolutely safe.

[0084] S3.5 Closed-Loop Control: The actuator drives the field equipment to change the system state, completing one control cycle. The data sensing unit 110 then begins collecting new state data, initiating the next cycle. Through this closed loop, the system achieves control over the grid power command P. ref It provides fast, accurate, safe, and economical automatic tracking and control.

[0085] Among them, such as Figure 1 As shown, the data sensing unit 110, intelligent decision-making unit 120, control execution unit 130, and simulation training platform 140 together constitute the control system of the thermal storage system. This system, together with the physical entity of the thermal storage system, forms a digital twin system. The following is a detailed introduction to each unit of the control system.

[0086] 1. Data Sensing Unit 110

[0087] The data sensing unit 110 serves as the foundation for signal acquisition in the system, communicating with key measurement points of the thermal storage system. Its hardware configuration perfectly matches the multi-dimensional state space defined by S1. This unit includes:

[0088] Temperature sensing component: Employs a type K thermocouple to accurately measure the molten salt temperature T at the outlet of hot salt tank 2. hot (t) (range: 290℃~565℃) and the molten salt temperature T at the outlet of cold salt tank 1 cold (t) (range: 290℃~565℃);

[0089] Pressure sensing component: Employs a high-precision pressure transmitter to measure the main steam pressure p main (t) (range: 5~16MPa);

[0090] Flow sensing component: Utilizing a Coriolis mass flow meter to measure the mass flow rate of molten salt circulation. salt (t) (range: 0~120 kg / s);

[0091] Signal receiving component: Employs an industrial Ethernet communication module to receive power commands P issued by the power grid dispatching system. ref (t) (range: -50~+50 MW) and the real-time electricity price C published by the electricity market. elec (t) signal.

[0092] This unit uploads the raw physical parameters collected to the intelligent decision-making unit 120 in real time via the PROFIBUS-DP industrial network at a frequency of 10 Hz, together forming complete state observation information for the control agent to make decisions.

[0093] 2. Intelligent Decision-Making Unit 120

[0094] The intelligent decision-making unit 120, as the control core of the system, communicates with the data sensing unit 110. Its hardware is an industrial-grade computer (such as an Advantech product, configured with an Intel i7 processor). This unit is configured to perform the following core tasks:

[0095] Data preprocessing module 121: This module receives raw data from the data sensing unit 110, first executes a moving average filtering algorithm to eliminate measurement noise, and then strictly follows the normalization formula defined in S1 and the physical upper and lower limits of each parameter to convert the data into a standardized state vector S. t .

[0096] Model Loading and Inference Module 122: This module stores and retrieves the neural network model parameters of the trained control agent. Specifically, it loads the .onnx format model file, which has been trained and solidified by S2, into memory and receives the preprocessed standardized state vector S. t And through forward propagation calculation of neural network, the normalized optimal control command a is generated within 50ms. t = [u heater (t), u pump (t), u valve [(t)], this instruction is the direct output of a collaborative control strategy that combines rapid response, economic optimization and equipment safety considerations.

[0097] 3. Control Execution Unit 130

[0098] The control execution unit 130 is communicatively connected to the intelligent decision-making unit 120, and is used to receive control commands and drive the actuators in the thermal storage system. This unit includes:

[0099] Execution hardware: Includes a frequency converter (receiver u) that controls the speed of the molten salt pump 3. pump Command), power-regulating electric heater solid-state power regulator SSR (receiver u) heater Instructions), and the turbine inlet electric regulating valve and its servo controller (receiving u) for controlling steam flow. valve instruction);

[0100] Command Verification and Safety Limiting Module: This module is a key hardware defense line ensuring the physical security of the system, embedded in the programmable logic controller (PLC). It is configured to receive control commands a from the intelligent decision-making unit in real time. t And immediately according to its internally preset safety parameter table (whose limits are completely consistent with the system physical limits set in S1, for example: u pump Limit to [20%, 100%] to prevent molten salt from solidifying, T hot The monitoring value (not exceeding 565℃) undergoes secondary verification and dynamic limiting to ensure that the control commands ultimately output to each actuator are absolutely within the safe operating range of the thermal storage system.

[0101] 4. Simulation Training Platform 140—Digital Twin

[0102] The simulation training platform 140 runs on a separate high-performance server (such as one configured with an NVIDIA RTX A6000 GPU) to run a high-fidelity digital model of the thermal storage system (e.g., built on Simulink / Simscape with simulation accuracy >95%) before the control strategy is deployed, and to provide the control agent with the simulation environment for the pre-training phase and offline reinforcement learning phase described in detail in S2.

[0103] The simulation training platform 140 and the physical thermal storage system together constitute a digital twin system. The simulation training platform, as the digital twin, undertakes the entire training process and strategy evaluation of the control agent; the physical thermal storage system, as the physical entity, runs the final control strategy after training. This architecture achieves virtualization and secure isolation of control strategy development, verification, and deployment, forming the foundation for the system's continuous self-optimization.

[0104] The "perception-decision-execution" decoupled architecture proposed in this embodiment, along with the digital twin system formed by integrating a simulation training platform, enables the research, verification, and deployment of intelligent control strategies to be carried out independently of the physical system, greatly reducing the complexity and risk of system intelligent upgrades.

[0105] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for regulating a thermal storage system based on expert demonstration and deep reinforcement learning, characterized in that, include: Construct a multidimensional state space to describe the operating state of the thermal storage system; Based on the aforementioned multidimensional state space, a deep reinforcement learning algorithm that integrates expert demonstration data is used to train the control agent to learn an optimized control strategy; wherein, the training process is guided by maximizing the reward function that comprehensively reflects the multi-objective performance of the system; The trained control agent is deployed in the thermal storage system and outputs control commands based on the real-time status feedback of the thermal storage system to achieve active regulation of the thermal storage system.

2. The thermal storage system control method based on expert demonstration and deep reinforcement learning according to claim 1, characterized in that, The multidimensional state space includes the state variables of the thermal storage system, as well as external signals and demands; wherein... The state variables of the thermal storage system include the current values ​​of the core physical parameters, the historical values ​​of at least some of the core physical parameters, and the rate of change of at least some of the historical values ​​of the core physical parameters. The external signals and demands include grid power commands, real-time electricity prices, and forecasts of future short-term heat loads.

3. The thermal storage system control method based on expert demonstration and deep reinforcement learning according to claim 2, characterized in that, The core physical parameters include the temperature of the thermal storage medium, the system pressure, the flow rate of the thermal storage medium, and the heating power.

4. The thermal storage system control method based on expert demonstration and deep reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning algorithm that integrates expert demonstration data is the demonstration-based deep deterministic policy gradient (DDPGfD) algorithm. The training process for the control agent includes: Pre-training phase: Supervised learning of the Actor network of the control agent is performed using expert demonstration data from traditional controllers or historical operating data to initialize the network parameters; Offline reinforcement learning phase: The pre-trained control agent is placed in a simulation environment and learns autonomously by interacting with the environment and based on the reward function to optimize its control strategy.

5. The thermal storage system control method based on expert demonstration and deep reinforcement learning according to claim 1, characterized in that, The reward function is a multi-objective composite function, and its output value is obtained by weighting the error term, economic operating cost term, equipment safety status term, and energy efficiency optimization term that reflect the tracking performance.

6. The thermal storage system control method based on expert demonstration and deep reinforcement learning according to claim 5, characterized in that, The error term of the tracking performance is expressed as: The calculation formula is: The economic operating cost item is expressed as: The calculation formula is: The device safety status item is represented as follows: The calculation formula is: The energy efficiency optimization term is represented as follows: The calculation formula is: In the formula, Indicates grid power command. Indicates the actual output power of the system. Indicates real-time electricity price. Indicates heating power. Indicates the scheduling time step. Indicates the temperature of the heat storage medium. This represents the safe temperature, which is a constant. This indicates the pump speed command in the thermal storage system. This indicates the energy consumption coefficient of the auxiliary equipment.

7. The thermal storage system control method based on expert demonstration and deep reinforcement learning according to claim 1, characterized in that, The action space of the control agent is a continuous space, and its output includes control signals for pumps, valves and heaters in the thermal storage system.

8. A thermal storage system control system based on expert demonstration and deep reinforcement learning, characterized in that, include: The data sensing unit (110) is connected in communication with the key measuring points of the thermal storage system and is used to collect physical parameters that constitute a multi-dimensional state space for describing the operating status of the thermal storage system in real time. The intelligent decision unit (120) is communicatively connected to the data sensing unit (110), and its core is a control agent deployed on a computing device; the intelligent decision unit (120) is configured to: receive state data from the data sensing unit (110), run the trained control agent, and generate corresponding control commands; The control execution unit (130) is communicatively connected to the intelligent decision unit (120) and is used to receive the control command and drive the thermal storage system to perform corresponding adjustment actions.

9. The thermal storage system control system based on expert demonstration and deep reinforcement learning according to claim 8, characterized in that, It also includes a simulation training platform, which is used to run a digital model of the thermal storage system before deployment and to provide a simulation environment for the control agent during the pre-training and offline training phases.

10. The thermal storage system control system based on expert demonstration and deep reinforcement learning according to claim 9, characterized in that, The simulation training platform and the thermal storage system together constitute a digital twin system.

Images