A method for predicting regional energy load evolution based on deep reinforcement learning

Abstract

This invention discloses a regional energy load evolution prediction method based on deep reinforcement learning. First, it utilizes wavelet transform to decouple multi-source monitoring data in the frequency domain and combines this with causal dilation convolution translation based on pipeline flow velocity to achieve causal alignment of electrical and thermal data. Second, it constructs an equipment graph topology, mapping phase change states and transient thermal impedance as edge weights to generate a situational awareness map. Next, it uses physical evolution equations to perform nonlinear constraint mapping and extracts state vectors of the embedded mechanism based on high-frequency transient electrical energy features. Then, it outputs asynchronous periodic electrical and thermal energy scheduling actions through a two-layer asynchronous architecture and associates them with immediate and delayed reward signals. Finally, it updates the model using a time-difference credit allocation mechanism based on heat transfer loss rate. This invention effectively overcomes the problems of prediction accuracy decay and strategy failure caused by strong nonlinear state abrupt changes and long-period dynamic hysteresis.

Description

Technical Field

[0001] This invention relates to the field of energy management technology, specifically to a method for predicting regional energy load evolution based on deep reinforcement learning. Background Technology

[0002] With the development of agricultural microgrids, multi-energy coupled greenhouse systems integrating photovoltaic thermal energy, heat pumps, and phase change thermal storage have become key carriers for energy conservation and carbon reduction. Accurate prediction of regional energy load evolution is a prerequisite for achieving multi-energy complementarity and efficient dispatch. Currently, the industry mainly relies on deep time series models combined with deep reinforcement learning algorithms, using deep neural networks for load time series prediction, and combining this with reinforcement learning agents to generate energy dispatch strategies, aiming to achieve global supply and demand balance in system operation.

[0003] However, existing technologies are mostly designed for generalized scheduling of conventional commercial buildings or macro-grids, revealing significant limitations when facing the strongly coupled multi-physics scenario of smart agricultural microgrids. First, the phase change materials introduced in agricultural microgrids exhibit a large amount of heat absorption and release within the solid-liquid phase transition region, while maintaining a relatively stable temperature. For purely data-driven reinforcement learning models, this strong nonlinear thermodynamic characteristic represents a step singularity in the data space. Due to the lack of constraints from underlying physical evolution logic, existing models often experience gradient divergence when dealing with such strongly nonlinear state transitions, leading to difficulty in model convergence and a high risk of getting trapped in local optima. Second, existing solutions typically place high-frequency electrical control commands and long-period thermal inertial responses within the same time architecture for prediction and control, failing to achieve effective decoupling. On the data input side, there is a serious time-scale mismatch between transient power fluctuations and smooth, sluggish thermophysical responses. Directly fusing heterogeneous data can easily lead to the prediction model learning incorrect causal relationships. On the decision feedback side, the evolution of agricultural microenvironment climate and the physiological metabolism of crops is an extremely slow process. This results in a long-period dynamic lag effect between the actual feedback of environmental state and the reward signal of reinforcement learning after the system executes scheduling actions, leading to an extremely low signal-to-noise ratio. Consequently, the prediction model cannot accurately capture the spatiotemporal evolution patterns.

[0004] In summary, overcoming the problems of accuracy decay of data-driven load forecasting models and failure of long-term scheduling strategies caused by strong nonlinear state changes and long-period dynamic hysteresis effects within the system is a technical problem that urgently needs to be solved in this field.

[0005] To address this, a regional energy load evolution prediction method based on deep reinforcement learning is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a regional energy load evolution prediction method based on deep reinforcement learning. This invention utilizes wavelet transform to decouple multi-source monitoring data in the frequency domain and combines it with pipeline flow velocity to perform causal dilation convolution translation, achieving causal alignment of electrical and thermal data. Secondly, it constructs an equipment graph topology, mapping phase change states and transient thermal impedance as edge weights to generate a situational awareness map. Next, it uses physical evolution equations to perform nonlinear constraint mapping and combines high-frequency electrical transient features to extract the state vector of the embedded mechanism. Subsequently, it outputs asynchronous periodic electrical and thermal energy scheduling actions through a two-layer asynchronous architecture and associates them with immediate and delayed reward signals. Finally, it updates the model using a time-difference credit allocation mechanism based on heat transfer loss rate. This invention effectively overcomes the problems of prediction accuracy decay and strategy failure caused by strong nonlinear state abrupt changes and long-period dynamic hysteresis.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for predicting regional energy load evolution based on deep reinforcement learning includes: Multi-source monitoring data of the regional energy system is acquired and decomposed into electrical energy detail sequence and thermal energy approximation sequence using wavelet transform; dynamic time delay offset coefficients are extracted based on pipeline flow velocity and spatial topological distance, and causal dilation convolution is applied to time step shift the thermal energy approximation sequence to obtain the fused feature matrix; A graph topology is constructed based on the fusion feature matrix, and the physical phase change state and transient thermal impedance parameters of the phase change thermal storage device are mapped to the dynamic weights of the corresponding connecting edges in the graph topology to generate a situational awareness map. The node feature matrix of the situational awareness map is extracted, and the node features corresponding to the phase change thermal storage device are nonlinearly constrained and mapped using the physical evolution equation. Combined with the transient fluctuation characteristics of the power energy detail sequence, a state vector embedding the physical mechanism is generated. The state vector is input into the reinforcement learning model, and the power dispatching action and thermal dispatching action are output through a two-layer asynchronous architecture. The power dispatching action and thermal dispatching action are respectively associated with an instant reward signal based on the real-time grid-connected power exchange deviation and a delayed reward signal based on the integral of the ambient temperature deviation. By utilizing the time-difference credit allocation mechanism, the reinforcement learning model is updated based on the immediate reward signal and the delayed reward signal, and the regional load evolution prediction results and scheduling strategy are output.

[0008] Preferably, the process of decomposing the data into an electrical energy detail sequence and a thermal energy approximation sequence using wavelet transform, and applying causal dilatation convolution to time-step shift the thermal energy approximation sequence, includes: extracting detail coefficients representing load abrupt changes from multi-source monitoring data using wavelet transform and reconstructing them into an electrical energy detail sequence; extracting approximation coefficients representing the slow evolution of thermophysical quantities and reconstructing them into a thermal energy approximation sequence; calculating a dynamic time-delay offset coefficient by combining pipeline flow velocity and topological distance, and calculating a spatial thermal decay factor representing energy loss based on pipeline heat transfer parameters and the temperature difference between the internal and external environments; setting the expansion rate of the causal dilatation convolution kernel according to the dynamic time-delay offset coefficient and time-shifting the thermal energy approximation sequence; multiplying the calculated spatial thermal decay factor by the shifted sequence and concatenating it with the electrical energy detail sequence to obtain a fusion feature matrix.

[0009] Preferably, constructing a graph topology based on a fused feature matrix includes: extracting device identifiers from the fused feature matrix and retrieving corresponding physical connection relationships from an energy flow database to determine the set of nodes and the initial set of connection edges in the device graph topology; encapsulating the power components, temperature components after translation and alignment, and flow components belonging to the same time section in the fused feature matrix into initial feature vectors corresponding to each node according to a preset feature index order; and generating an adjacency matrix representing the initial physical topology relationships of the regional energy system based on the initial feature vectors and the initial set of connection edges of each node to complete the construction of the graph topology.

[0010] Preferably, the process of mapping the physical phase change state and transient thermal impedance parameters of the phase change thermal storage device to the dynamic weights of the corresponding connecting edges in the graph topology includes: acquiring the material temperature monitoring values ​​of the nodes of the phase change thermal storage device in real time, and determining whether the material temperature monitoring values ​​are within the preset solid-liquid phase change temperature range; if the determination result is no, setting the dynamic weight between the node and its connected edge to the reciprocal of the static thermal resistance; if the determination result is yes, mapping the liquid volume fraction calculated based on the accumulated heat absorption to the phase change smoothing coefficient, and smoothing the partial derivative of the enthalpy value at the current temperature to the heat capacity reference coefficient; multiplying the phase change smoothing coefficient and the heat capacity reference coefficient to obtain the latent heat characteristic gain coefficient; and using the product of the latent heat characteristic gain coefficient and the reciprocal of the transient thermal impedance value of the material as the dynamic weight to update the adjacency matrix.

[0011] Preferably, the process of generating a state vector embedded with the physical mechanism is as follows: a set of partial differential equations containing unsteady-state heat conduction terms, phase change latent heat source terms, and surface convection heat transfer terms is pre-constructed as the physical evolution equation; the initial feature vector of the phase change thermal storage device is input into the physical evolution equation to obtain the theoretical temperature evolution value and the theoretical remaining heat storage capacity of the phase change thermal storage device within a single prediction step in the future; the theoretical temperature evolution value and the theoretical remaining heat storage capacity are input into a regularized mapping network to obtain the mapping output; the mapping output is used as a feature correction operator to act on the hidden layer output of the neural network to obtain the corrected background thermal state component; the amplitude standard deviation and the sum of the first-order absolute differences of the electrical energy detail sequence are calculated as transient fluctuation features; the transient fluctuation features and the background thermal state component are channel-cascaded to output the state vector embedded with the physical mechanism.

[0012] Preferably, the dual-layer asynchronous architecture outputs power scheduling actions and thermal scheduling actions, including: setting the execution frequency of the power action generation network in the deep reinforcement learning model as a first execution frequency, setting the execution frequency of the thermal action generation network as a second execution frequency, and the value of the first execution frequency being a positive integer multiple of the value of the second execution frequency; during the triggering period of the first execution frequency, the power action generation network outputs a power scheduling action containing the active power command for charging and discharging of the energy storage converter according to the state vector; during the triggering period of the second execution frequency, the thermal action generation network outputs a thermal scheduling action containing the frequency percentage of the phase change thermal storage tank circulating pump according to the state vector; at multiple first execution frequency nodes within the same second execution frequency period, the output value of the thermal scheduling action remains constant.

[0013] Preferably, the quantitative calculation method for associating power dispatch actions and thermal dispatch actions with an instant reward signal based on the real-time grid-connected power exchange deviation and a delayed reward signal based on the integral of the ambient temperature deviation is as follows: The instant reward signal is calculated as follows: When executing a power dispatch action, the Euclidean distance between the real-time grid-connected power exchange value and the preset baseline power value is calculated, and the negative mapping value of the Euclidean distance is used as the instant reward signal returned to the power action generation network; The delayed reward signal is calculated as follows: After the following thermal dispatch action, a delayed observation timer is started synchronously. When the delayed observation timer reaches the preset thermophysical response constant time, the absolute value integral of the difference between the actual ambient temperature value and the target temperature value is calculated, and the negative scalar value of the absolute value integral of the difference is used as the delayed reward signal.

[0014] Preferably, the reinforcement learning model is updated using a time-difference credit allocation mechanism based on immediate and delayed reward signals. This includes: constructing two experience replay data pools; the first experience replay data pool stores quadruple trajectory data containing power dispatch actions at a first execution frequency step size; and the second experience replay data pool stores quadruple trajectory data containing thermal dispatch actions at a second execution frequency step size. For the first experience replay data pool, the transient time difference error is calculated using the first Bellman equation, and the parameters of the power action generation network are updated. For the second experience replay data pool, the delayed reward signal is distributed backward along the time sequence to intermediate state nodes using the second Bellman equation containing a time decay factor, the hysteresis time difference error is calculated, and the parameters of the thermal action generation network are updated.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention achieves frequency domain decoupling and spatiotemporal causal alignment of multi-source heterogeneous data by introducing wavelet transform and causal dilation convolution mechanisms. This processing method fully considers the differences in physical time scale between transient changes in electrical energy and long-period evolution of thermal energy, and reasonably aligns the thermophysical response with time lag characteristics and the original electrical signal on the same time section through a dynamic time delay offset coefficient. This not only improves the physical and logical self-consistency of multi-source data fusion, but also enables subsequent models to more accurately capture the temporal evolution law of cross-medium energy flow, thereby effectively improving the accuracy of load evolution prediction.

[0016] 2. This invention deeply embeds physical mechanisms into the extraction of state features and the construction of the topological graph. By mapping the physical phase change state and transient thermal impedance parameters of the phase change thermal storage device to graph topological weights, and using partial differential equations of physical evolution to impose nonlinear constraints on node features, this scheme provides reasonable physical boundary guidance for purely data-driven neural networks. This deep coupling of physical information and data features enables reinforcement learning models to exhibit better policy convergence and robustness when facing complex nonlinear thermodynamic state evolution, ensuring that the generated predictions and scheduling actions are more consistent with actual physical laws.

[0017] 3. This invention constructs a two-layer asynchronous architecture and a time-differential credit allocation mechanism, realizing collaborative optimization of complex multi-energy systems. By setting different execution frequencies for the electrical energy action generation network and the thermal energy action generation network, and respectively associating instantaneous power smoothing rewards and delayed thermal comfort rewards, this architecture achieves effective decoupling of high-frequency electrical control and low-frequency thermal control in the action tensor space. Combined with gradient update calculations of long- and short-period dual Bellman equations, this scheme can more scientifically allocate delayed feedback to the corresponding action nodes, thereby balancing the stable regulation of transient electrical loads in the microgrid with the economical operation of the long-period thermal environment, and improving the overall comprehensive scheduling efficiency of the system. Attached Figure Description

[0018] Figure 1 A flowchart of a regional energy load evolution prediction method based on deep reinforcement learning provided in an embodiment of the present invention; Figure 2 This is a logic diagram for multi-source data physical causal alignment and graph representation provided in an embodiment of the present invention; Figure 3 The physical mechanism embedding and asynchronous multi-scale scheduling decision logic diagram provided for embodiments of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figures 1 to 3 This invention provides a method for predicting the evolution of regional energy load based on deep reinforcement learning, the technical solution of which is as follows: A method for predicting regional energy load evolution based on deep reinforcement learning includes: Multi-source monitoring data of the regional energy system is acquired and decomposed into electrical energy detail sequence and thermal energy approximation sequence using wavelet transform; dynamic time delay offset coefficients are extracted based on pipeline flow velocity and spatial topological distance, and causal dilation convolution is applied to time step shift the thermal energy approximation sequence to obtain the fused feature matrix; A graph topology is constructed based on the fusion feature matrix, and the physical phase change state and transient thermal impedance parameters of the phase change thermal storage device are mapped to the dynamic weights of the corresponding connecting edges in the graph topology to generate a situational awareness map. The node feature matrix of the situational awareness map is extracted, and the node features corresponding to the phase change thermal storage device are nonlinearly constrained and mapped using the physical evolution equation. Combined with the transient fluctuation characteristics of the power energy detail sequence, a state vector embedding the physical mechanism is generated. The state vector is input into the reinforcement learning model, and the power dispatching action and thermal dispatching action are output through a two-layer asynchronous architecture. The power dispatching action and thermal dispatching action are respectively associated with an instant reward signal based on the real-time grid-connected power exchange deviation and a delayed reward signal based on the integral of the ambient temperature deviation. By utilizing the time-difference credit allocation mechanism, the reinforcement learning model is updated based on the immediate reward signal and the delayed reward signal, and the regional load evolution prediction results and scheduling strategy are output.

[0021] First embodiment:

[0022] This embodiment applies to a smart agriculture solar-storage direct-drive flexible-drive and phase-change thermal storage microgrid system. In this context, agricultural microgrids need to simultaneously coordinate transient power fluctuations and the thermodynamic evolution of greenhouse phase change with long-period hysteresis. Traditional single-timescale pure data reinforcement learning prediction and scheduling models are prone to prediction failure and strategy divergence due to data misalignment and physical step characteristics.

[0023] As one embodiment of the present invention, refer to Figure 1 A flowchart of a regional energy load evolution prediction method based on deep reinforcement learning, referring to... Figure 2 Multi-source data physical causal alignment and graph representation logic diagram, refer to Figure 3 Physical mechanism embedding and asynchronous multi-scale scheduling decision logic diagram.

[0024] The regional energy load evolution prediction method of this invention is based on a deep reinforcement learning architecture. In this architecture, the prediction of the future state of the energy system and the optimization of control strategies for equipment are synchronous processes with joint convergence. Therefore, the prediction results of this invention not only include numerical estimates of future physical parameters, but also naturally encompass the scheduling strategies generated to cope with the evolution trend. That is, the scheduling strategy is the materialized response representation of the prediction model of this invention in the action space.

[0025] Furthermore, the process involves using wavelet transform to decompose the data into electrical energy detail sequences and thermal energy approximation sequences, and applying causal dilatation convolution to perform time-step translation of the thermal energy approximation sequence. This includes: using wavelet transform to extract detail coefficients representing load abrupt changes from multi-source monitoring data and reconstructing them into electrical energy detail sequences; extracting approximation coefficients representing the slow evolution of thermophysical quantities and reconstructing them into thermal energy approximation sequences; calculating dynamic time-delay offset coefficients by combining pipeline flow velocity and topological distance, and calculating spatial thermal decay factors representing energy loss based on pipeline heat transfer parameters and internal / external environmental temperature differences; setting the expansion rate parameter of the causal dilatation convolution kernel according to the dynamic time-delay offset coefficient, performing forward one-sided convolution on the thermal energy approximation sequence using the convolution kernel, and performing forward translation on the virtual time axis; using the spatial thermal decay factor calculated based on real-time heat transfer parameters as a dynamic physical scalar, performing element-wise multiplication with the translated sequence, and then concatenating it with the electrical energy detail sequence at the current time section to obtain a fused feature matrix.

[0026] Specifically, firstly, preset continuous wavelet basis functions and 4-level decomposition parameters are obtained. Wavelet transform (preferably discrete wavelet transform) is then performed on the collected electrical power time-series data and thermodynamic temperature time-series data of the microgrid system over the past 24 hours. The preset wavelet basis functions are selected from the Daubechies wavelet family (such as db4, db8) or the Symlets wavelet family (such as sym4). The selection of the basis functions is based on the tight support of the wavelet basis and the similarity of the characteristic signal waveform. The db4 wavelet has good singularity detection capability when processing transient jumps in the power sequence. The number of decomposition levels is usually set to 3 to 6. In this embodiment, 4 levels are selected to balance computational efficiency and the accuracy of frequency domain decoupling.

[0027] In this process, detailed coefficients with periods shorter than 10 seconds are extracted. These coefficients reflect the transient jump characteristics of the microgrid converter load and are reconstructed into a detailed electrical energy sequence. Simultaneously, approximate coefficients with periods longer than 10 seconds are extracted. These coefficients reflect the smooth evolution characteristics of ambient air temperature and heat storage medium temperature and are reconstructed into an approximate thermal energy sequence. In the pipeline network time delay calculation stage, the current operating frequency percentage of the pipeline circulation pump is read as 60%. Combined with the pump's rated parameters and the known pipe cross-sectional area of ​​0.05 square meters, the physical average flow velocity of the heat medium is calculated to be 1.5 meters per second. Subsequently, based on the microgrid spatial vector topology map, the topological distance of the pipeline centerline between the phase change heat source node and the target greenhouse load node is calculated to be 150 meters. Dividing 150 meters by the physical flow velocity of 1.5 meters per second yields a dynamic time delay reference time constant of 100 seconds. To correct for the pipe wall absorption effect during long-distance heat transmission, the comprehensive convective heat transfer coefficient, pipe surface area, and real-time internal and external ambient temperature difference of this pipeline section are simultaneously acquired. Based on Newton's law of cooling and the energy conservation equation, a spatial heat decay factor characterizing energy loss is derived and calculated. The specific formula is as follows: The heat loss rate along the pipe is obtained by multiplying the combined convective heat transfer coefficient, the pipe surface area, and the absolute value of the temperature difference between the inside and outside environments; this heat loss rate is then divided by the initial heat flow rate at the pipe inlet to obtain the heat loss ratio; finally, the spatial heat decay factor within the (0,1) interval is obtained by subtracting this heat loss ratio from the numerical value 1. The dynamic time delay reference time constant is converted to an integer value of 100 for the time step, and a one-dimensional causal dilation convolution kernel in the time dimension is constructed accordingly, with its dilation rate parameter strictly set to 100. By using this causal dilation convolution kernel to perform forward one-sided convolution on the approximate thermal energy sequence, the delayed thermal response is pushed forward on the virtual time axis to achieve large-scale spatiotemporal causal alignment. Simultaneously, the learnable weights within the convolution kernel are used to adaptively smooth and extract temporal features from the local time delay uncertainty caused by dynamic fluctuations in pipeline flow velocity. Subsequently, the spatial thermal decay factor is used as a dynamic physical scalar and multiplied element-wise with the translated approximate thermal energy sequence to achieve thermodynamic dissipation contraction of waveform amplitude at the algorithmic level. After completing the above dual decoupling physical correction of time and amplitude, the dynamic physical scalar and the electrical energy detail sequence of the current time section are matrix-concatenated along the channel dimension to obtain the spatiotemporal causal alignment fusion feature matrix.

[0028] This invention scientifically decomposes multi-source data in the frequency domain and precisely quantifies the time delay constant using physical flow velocity and spatial distance, then relies on dilated convolution to achieve time step compensation and translation. This processing method eliminates the feature response misalignment caused by long-distance heat transmission, enabling the prediction model to extract features based on the correct physical causal time axis, effectively improving the logical consistency of multimodal data fusion and the accuracy of time series prediction.

[0029] Furthermore, the graph topology is constructed based on the fused feature matrix, including: extracting device identifiers from the fused feature matrix and retrieving corresponding physical connection relationships from the energy flow database to determine the set of nodes and the initial set of connection edges in the device graph topology; encapsulating the power components, temperature components after translation and alignment, and flow components belonging to the same time section in the fused feature matrix into initial feature vectors corresponding to each node according to a preset feature index order; and generating an adjacency matrix representing the initial physical topology relationship of the regional energy system based on the initial feature vectors and the initial set of connection edges of each node, thus completing the construction of the graph topology.

[0030] Specifically, the backend includes a pre-configured energy flow database that records the physical connections between devices in the microgrid. The device identifiers carried in the fusion feature matrix are extracted using a feature parsing module; for example, identifier 101 represents a photovoltaic inverter, identifier 201 represents a phase change thermal storage tank, and identifier 301 represents a greenhouse heat load. Based on these identifiers, a relationship search is performed in the database to establish the effective node set in the device graph topology and to create initial connection edge sets such as those from the photovoltaic inverter to the battery and from the phase change thermal storage tank to the greenhouse load. Subsequently, the multi-data elements belonging to the same time segment in the fusion feature matrix are sliced ​​and extracted. The node active power component, the energy storage device state of charge component, the translated and aligned medium temperature component, the pipeline mass flow component, the ambient irradiance component, and the corresponding time step identifier are encapsulated and assembled into an initial feature vector for each node according to a pre-defined feature index order. The pre-defined feature index order is as follows: node active power component, energy storage device state of charge component, translated and aligned medium temperature component, pipeline mass flow component, ambient irradiance component, and time step identifier. The fixed feature index order ensures that the physical meaning of each feature channel remains consistent in the spatiotemporal dimension when the graph convolutional neural network processes the multiplication of the adjacency matrix and the feature matrix. Based on the encapsulated initial feature vectors of the nodes and the established initial set of connecting edges, a two-dimensional adjacency matrix representing the initial physical topological relationships of the regional microgrid system is generated in memory, thus completing the initialization and construction of the infrastructure graph topology.

[0031] This invention reconstructs fragmented multi-source monitoring time-series data into a topological data structure by extracting device identifiers and associating them with a physical database. This method provides a standardized mathematical expression of complex regional microgrid energy flow relationships through node features and adjacency matrices, offering a standardized data input foundation for subsequent graph neural network hierarchical feature extraction and enhancing the algorithm's versatility.

[0032] Furthermore, the process of mapping the physical phase change state and transient thermal impedance parameters of the phase change thermal storage device to the dynamic weights of the corresponding connecting edges in the graph topology includes: acquiring the material temperature monitoring values ​​of the phase change thermal storage device nodes in real time, and determining whether the material temperature monitoring values ​​are within the preset solid-liquid phase change temperature range; if the determination result is no, then setting the dynamic weight between the node and its connected edge to the reciprocal of the static thermal resistance; if the determination result is yes, then mapping the liquid volume fraction calculated based on the accumulated heat absorption to the phase change smoothing coefficient, and smoothing the partial derivative of the enthalpy value at the current temperature to the heat capacity reference coefficient; multiplying the phase change smoothing coefficient and the heat capacity reference coefficient to obtain the latent heat characteristic gain coefficient; and using the product of the latent heat characteristic gain coefficient and the reciprocal of the material transient thermal impedance value as the dynamic weight to update the adjacency matrix.

[0033] Specifically, the material temperature monitoring values ​​of the phase change thermal storage device nodes are acquired in real time via IoT sensors at 5-second sampling intervals. The system's preset solid-liquid phase change temperature range is 45°C to 55°C. When the real-time monitored temperature is determined to be 40°C, this value is outside the phase change range and is determined to be pure solid. The solid thermal conductivity of the material is retrieved, the reciprocal of the static thermal resistance is calculated, and this is directly set as the dynamic weight between the node and the external heating network connection. When the monitored temperature rises to 50 degrees Celsius, it is determined that the value is within the preset phase change temperature range, and dynamic weight calculation is triggered: The system first calculates the current liquid phase volume fraction based on the cumulative heat absorbed by the phase change heat storage device since entering the phase change zone, combined with the total latent heat of the material. This is then used as an input feature and substituted into a preset continuous nonlinear activation function (such as the Sigmoid or Softplus function) to obtain a smoothing coefficient reflecting the progress of the phase change process. At the same time, in order to obtain the true transient heat capacity at this stage, the discrete enthalpy-temperature characteristic curve of the phase change material stored in the database is extracted. The numerical center difference algorithm is used to calculate the first-order partial derivative of the enthalpy value with respect to temperature at the current monitoring temperature point as the dynamic equivalent heat capacity peak value. The Gaussian smoothing function is then called to filter and limit the peak value (the one-dimensional filtering window size is set to 5 to 9 discrete temperature sampling points) to obtain the limited heat capacity reference coefficient. Finally, the smoothing coefficient reflecting the phase change process is multiplied and fused with the heat capacity reference coefficient, and the output comprehensive scalar value is used as the final latent heat characteristic gain coefficient. Subsequently, the transient thermal impedance value of the node at this temperature is retrieved, and the latent heat characteristic gain coefficient is multiplied by the reciprocal of the transient thermal impedance. The calculated value is used as the new dynamic weight of the corresponding connecting edge in the graph topology, and the corresponding element in the adjacency matrix is ​​updated using this weight value.

[0034] This invention cleverly transforms the nonlinear endothermic and exothermic physical processes of phase change materials into a dynamic adjustment logic of the edge weights in a graph topology network. This approach allows the implicit physical state transitions to be represented at the algorithmic level as changes in the connectivity of the graph structure's information flow. This enables data-driven models to intuitively perceive abrupt changes in the system's thermodynamic state, effectively reducing the model's prediction bias in the phase transition region.

[0035] After constructing the situational awareness map and dynamically updating the adjacency matrix, the node feature matrix and adjacency matrix of the graph topology are input as the main neural network for spatial feature extraction. Specifically, in this embodiment, a GAT (Graph Attention Network) is used as the main neural network. The GAT contains two graph attention convolutional layers. The first layer contains eight attention heads, each with an output feature dimension of 8, and the multi-head outputs are concatenated to have a feature dimension of 64. The second layer contains one attention head with an output dimension of 64. Each graph convolutional layer uses the ELU non-linear activation function and performs layer normalization on the node feature dimension. After information aggregation by the graph network, the 64-dimensional output feature vectors corresponding to the nodes of the target phase change thermal storage device are extracted and used as the hidden layer features of the main neural network, which are then fed into the subsequent physical mechanism embedding module for correction and fusion.

[0036] Furthermore, the process of generating the state vector embedded with the physical mechanism involves pre-constructing a set of partial differential equations containing unsteady-state heat conduction terms, phase change latent heat source terms, and surface convection heat transfer terms, which serve as the physical evolution equations. Taking one-dimensional heat conduction as an example, the core governing equation of this set of equations is: ; in, For the density of phase change materials, For temperature, For time, For spatial coordinates, Thermal conductivity; It is the dynamic equivalent heat capacity that includes the latent heat source term of phase change (i.e., the heat capacity reference coefficient mentioned above, which is based on the partial derivative of the enthalpy-temperature curve and smoothed). This refers to a source item within the system.

[0037] Meanwhile, the surface convection heat transfer term is used as the third type of boundary condition (Robin boundary condition) for the above partial differential equation, and its expression is: ; in, To take into account the convective heat transfer coefficient, The surface temperature of the equipment. The ambient temperature.

[0038] Specifically, standard partial differential equations, encompassing unsteady-state heat conduction, phase change latent heat sources, and surface convection heat transfer terms, are pre-loaded into the computation module and used as the physical evolution equations. The initial feature vector of the phase change heat storage device in the situational awareness map is used as the input boundary condition. A finite difference method with adaptive time steps is employed for iterative calculations to determine the theoretical temperature evolution and theoretical remaining heat storage capacity of the phase change device within a 15-minute prediction step. Subsequently, a regularized mapping network with a fully connected layer structure is constructed. This network consists of an input layer, two hidden layers with 64 and 128 neurons respectively (using the ReLU activation function), and an output layer. The theoretical temperature evolution and theoretical remaining heat storage capacity are concatenated along the feature dimension to form a one-dimensional physical prior vector, which is then input into the network. The feature dimension of the network's output layer is strictly aligned with the hidden layer dimension of the main reinforcement learning neural network. After obtaining the mapped output through forward propagation, the Hadamard product (element-wise multiplication) is used as a feature correction operator, and it is multiplied element-wise with the hidden layer features of the main neural network to obtain the physically corrected background thermal state component. Simultaneously, a 15-minute sliding window is opened to calculate the amplitude standard deviation and the sum of the first-order absolute differences of the detailed electrical energy sequence, forming a transient fluctuation feature. Finally, the background thermal state component and this transient fluctuation feature are cascaded along the channel dimension to output a state vector embedding the physical mechanism.

[0039] This invention provides a clear physical boundary for the neural network by solving partial differential equations using finite difference methods, avoiding gradient divergence in pure data optimization. Simultaneously, it incorporates high-frequency transient fluctuation characteristics for splicing and fusion. This approach ensures that the output state vector not only embodies a safety boundary awareness from a thermophysical perspective but also possesses the ability to perceive short-term shock pressures on the power grid, enhancing the comprehensiveness of the multi-energy coupled state representation of the microgrid.

[0040] Furthermore, the two-layer asynchronous architecture outputs power scheduling actions and thermal scheduling actions, including: setting the execution frequency of the power action generation network in the deep reinforcement learning model as the first execution frequency, setting the execution frequency of the thermal action generation network as the second execution frequency, and the value of the first execution frequency being a positive integer multiple of the value of the second execution frequency; during the triggering period of the first execution frequency, the power action generation network outputs a power scheduling action containing the active power command for charging and discharging of the energy storage converter according to the state vector; during the triggering period of the second execution frequency, the thermal action generation network outputs a thermal scheduling action containing the percentage of the frequency of the phase change thermal storage tank circulating pump according to the state vector; at multiple first execution frequency nodes within the same second execution frequency period, the output value of the thermal scheduling action remains constant.

[0041] The reinforcement learning model employs a deep deterministic policy gradient algorithm architecture, comprising independent electrical and thermal Actor-Critic network groups. Both the electrical and thermal action generation networks are Actor networks, internally containing an input normalization layer, a multilayer perceptron hidden layer, and an action output layer with a hyperbolic tangent activation function. Before actual deployment, a pre-built regional microgrid digital twin simulation platform serves as the interactive environment for reinforcement learning. Within this simulation environment, historical multi-source monitoring data is injected to simulate environmental evolution, allowing the reinforcement learning model to complete pre-training through trial and error interaction. After convergence, it is then deployed to the actual physical system for online fine-tuning and inference.

[0042] Specifically, independent power action generation networks and thermal action generation networks are configured for microgrid energy dispatch. Based on the response capability of the grid converter, the first fixed execution frequency of the power action generation network is set to once per second; based on the response inertia of the water valves and phase change materials, the second fixed execution frequency of the thermal action generation network is set to once every 60 seconds. Under this setting, the first execution frequency is strictly a positive integer multiple of the second execution frequency, i.e., 60 times. During system operation, at each 1-second trigger node, the power action generation network receives the latest state vector, independently performs forward propagation, outputs a power dispatch action containing the active power command for charging and discharging the energy storage converter, and immediately sends it to the underlying equipment for execution. At each 60-second trigger node, the thermal action generation network calculates based on the state vector at that moment and outputs a thermal dispatch action containing the frequency command of the phase change thermal storage tank circulating pump. During the 59 1-second durations before entering the next 60-second cycle, the thermal action generation network is not invoked, and the frequency command of the underlying circulating pump is forcibly maintained at a constant output of the phase change thermal storage tank circulating pump frequency command. During this period, the power generation network continues to update the active power command every second, enabling parallel issuance of actions.

[0043] This invention sets dual frequencies for the strategy output based on the actual physical response rate of the equipment, breaking the traditional limitation of a globally unified clock. This multi-scale decoupling in the action tensor space ensures smooth adjustment of high-frequency power on the grid side while avoiding mechanical wear and system oscillations caused by excessively rapid switching of control commands on the heating network side, thus improving the reliability of practical engineering applications.

[0044] Furthermore, the quantitative calculation methods for the power dispatch action and the thermal dispatch action, which are respectively associated with the instant reward signal based on the real-time grid-connected power exchange deviation and the delayed reward signal based on the ambient temperature deviation integral, are as follows: The instant reward signal is calculated as follows: When the power dispatch action is executed, the Euclidean distance between the real-time grid-connected power exchange value and the preset baseline power value is calculated. Using a linear penalty method, the Euclidean distance value is multiplied by a preset negative penalty coefficient to obtain the corresponding negative scalar value, and this negative scalar value is used as the instant reward signal returned to the power action generation network. The delayed reward signal is calculated as follows: After the thermal dispatch action, a delayed observation timer is started synchronously. When the delayed observation timer reaches the preset thermophysical response constant time, the absolute value integral of the difference between the actual ambient temperature value and the target temperature value is calculated, and the negative scalar value of the absolute value integral of the difference is used as the delayed reward signal.

[0045] Specifically, for the immediate reward signal, at the current moment after each power dispatch action, the real-time exchange power of the regional microgrid interconnection line is obtained through the underlying electricity meter, for example, 520 kW; the pre-set baseline power value of the intraday dispatch plan, 500 kW, is retrieved; and the absolute value of the Euclidean distance between 520 kW and 500 kW is calculated to be 20 kW. In this embodiment, the preset negative penalty coefficient is set to -1, which physically means that each 1 kW power deviation is proportionally converted into a reward penalty value of -1. In other implementation scenarios, this coefficient can be adaptively calibrated within the range of [-0.5, -5] according to the microgrid's tolerance for power fluctuations, converting 20 kW into a negative mapping value of -20, and returning it as an immediate reward signal to the power action generation network of reinforcement learning in real time. For the delayed reward signal, after each thermal energy dispatch action such as water pumps or valves is issued, the background program synchronously starts the delayed observation timer. When the timer accumulates to the preset 300-second thermophysical response constant time, a temperature acquisition command is triggered, and the actual ambient temperature value of the regional greenhouse terminal is obtained as 22 degrees Celsius. The user-preset comfort target temperature value of 24 degrees Celsius is read. The difference between 22 degrees Celsius and 24 degrees Celsius is integrally calculated over continuous time to obtain the integral error. This integral error is converted into a negative scalar and used as a delayed reward signal to feed back to the thermal action generation network for subsequent training and evaluation. To ensure that the two reward signals participate in gradient updates on the same order of magnitude, both the immediate reward signal and the delayed reward signal are strictly normalized and aligned within the [-1, 0] interval by dividing by their respective historical maximum absolute value boundaries before being input into the playback pool.

[0046] This invention designs a targeted reward function to address the differences in objectives between electrical and thermal scheduling. It employs a penalty mechanism based on instantaneous power deviation for electrical energy actions and a penalty mechanism for ambient temperature integral error considering time lag for thermal energy actions. This decoupled quantitative index can accurately guide the optimization direction of each agent, helping reinforcement learning models to learn joint scheduling strategies that ensure stable system operation more quickly.

[0047] Furthermore, utilizing a time-differential credit allocation mechanism, the reinforcement learning model is updated based on immediate and delayed reward signals. This includes: constructing two experience replay data pools; the first experience replay data pool stores quadruple trajectory data containing power dispatching actions at a first execution frequency step size, and the second experience replay data pool stores quadruple trajectory data containing thermal energy dispatching actions at a second execution frequency step size; for the first experience replay data pool, the transient time difference error is calculated using the first Bellman equation, and the parameters of the power action generation network are updated; for the second experience replay data pool, the delayed reward signal is distributed backward along the time sequence to intermediate state nodes using the second Bellman equation including a time decay factor, the hysteresis time difference error is calculated, and the parameters of the thermal energy action generation network are updated.

[0048] Specifically, two data storage areas are allocated in memory to construct two independent experience replay data pools. The first experience replay data pool is configured to continuously store the quadruple trajectory data of state, action, reward, and next state generated by the interaction of the energy action generation network at a first execution frequency step of 1 second, with a maximum queue length of 1000. For thermal networks with long-period lag, a macroscopic action design based on a semi-Markov decision process is adopted: the second experience replay data pool strictly follows the second execution frequency step of 60 seconds for aligned sampling, that is, it extracts the macroscopic state vector only at the first time node of each 60-second trigger cycle, spans the intermediate time step of the action lock, and combines it with the corresponding thermal macroscopic action, accumulated delay reward, and macroscopic state vector of the next cycle to store the quadruple trajectory data related to thermal actions, with a maximum queue length of 500. During the parameter update phase, for the data batch extracted from the first experience replay data pool, the standard first Bellman equation is directly applied. The predicted value of the generated action is compared with the weighted value of the actual immediate reward, and the transient time difference error is calculated. Based on this error, the gradient descent algorithm is used to correct the weight parameters of the electrical energy action generation network in real time. For the data batch extracted from the second experience replay data pool, the delayed reward signal is extracted, and the second Bellman equation, including a 0.9 time decay factor, is used. This equation distributes the delayed reward signal uniformly in reverse along the time-series chain, calculating it over the time distance span to the 60 intermediate state nodes experienced in generating the reward. Based on this, the hysteresis time difference error reflecting long-term evaluation is calculated. Finally, this hysteresis error is used to perform targeted gradient updates on the parameters of the thermal energy action generation network.

[0049] This invention addresses the issues of delayed long-cycle reward signal feedback and sparse reward distribution in asynchronous multi-scale actions by establishing a dual empirical replay pool and introducing a reverse allocation calculation with a time decay factor. This credit allocation mechanism enables the model to clearly distinguish the correlation between long-term thermal rewards and immediate electrical rewards, ensuring stable convergence performance of the reinforcement learning model in a multi-frequency action space.

[0050] The steps for calculating the hysteresis time difference error using the second Bellman equation with a time decay factor include: calculating the dynamic heat transfer temperature difference and medium flow rate between the phase change thermal storage device and the external environment in real time, and calculating the real-time heat transfer loss rate of the regional energy system; obtaining the nonlinear mapping reciprocal of the real-time heat transfer loss rate and substituting it as the dynamic eligibility trace decay factor into the second time difference update equation with eligibility trace mechanism; when the real-time heat transfer loss rate increases, indicating that the system has high irreversible heat dissipation, the dynamic eligibility trace decay factor decreases, so that the step size of the delayed reward signal for credit backtracking to the historical action trajectory is adaptively truncated, thereby blocking the invalid long-term credit allocation in the irreversible dissipation process.

[0051] Specifically, during the parameter update phase, when performing credit allocation for low-frequency thermal energy dispatch trajectory data, the absolute surface temperature of the phase change thermal storage device, the absolute temperature of the greenhouse environment, and the current mass flow rate of the pipeline medium are collected in real time. Preset values ​​for the specific heat capacity at constant pressure of the circulating fluid and the system reference ambient absolute temperature (e.g., set to 293.15K) are retrieved. The real-time heat transfer loss rate is calculated based on the entropy increase physical relationship of the heat transfer process. The specific formula is as follows: First, the entropy production rate of the heat conduction process is calculated, which is equal to the product of the mass flow rate, the specific heat capacity at constant pressure, and the absolute value of the difference between the surface absolute temperature and the ambient absolute temperature, divided by the quotient obtained by multiplying the surface absolute temperature and the ambient absolute temperature. This formula is based on the assumption of high fluid velocity and a large heat transfer coefficient, assuming that the fluid outlet temperature is approximately equal to the device surface wall temperature, thus achieving a reasonable simplification of engineering calculations. Subsequently, the entropy production rate is multiplied by the reference ambient absolute temperature to obtain the scalar value of the real-time heat transfer loss rate of the regional energy system.

[0052] Subsequently, a nonlinear inverse mapping method based on a negative exponential function is used to inversely map the real-time heat transfer loss rate scalar value to a preset interval of [0.1, 0.9]. The specific calculation method is as follows: using the natural constant as the base, the negative of the product of a preset shape adjustment parameter (calibrated to 0.3 in this embodiment through grid search, representing the sensitivity of unit loss rate change to attenuation factor adjustment) and the real-time heat transfer loss rate is used as the exponent for exponentiation. The resulting exponent is then multiplied by 0.8 and added to 0.1. The final calculation result is used as a new dynamic qualified trace attenuation factor and substituted into the second time-difference update equation with a qualified trace mechanism, while maintaining the long-term vision discount factor in this equation as a constant. The physical and algorithmic basis for this interval and parameter replacement setting is to ensure that the attenuation factor is strictly between zero and one to satisfy the mathematical boundary of algorithm convergence; simultaneously, the heat transfer loss rate represents the degree of physical irreversible dissipation and the increase in system disorder (entropy production) during energy flow. At the underlying level of reinforcement learning algorithms, the high degree of physically irreversible dissipation means that the thermodynamic state transition matrix of the system has extremely strong nonlinear coupling and random perturbation at this stage. If long-period credit backtracking (i.e., maintaining a large eligibility trace parameter) is continued in such a high dynamic uncertainty interval, the accumulated transition probability error of multiple states will cause the evaluation variance of the time difference objective to surge exponentially, thereby causing the model gradient update to diverge.

[0053] Therefore, this invention creatively establishes a mapping between physical entropy production and algorithm hyperparameters: when high heat transfer loss occurs, the dynamic qualification trace decay factor adaptively decreases to truncate the historical backtracking step size of delayed rewards. This is essentially an adaptive variance reduction mechanism based on underlying physical mechanisms. It enables the model to proactively abandon high-variance long-term causal tracing when facing physical transients with high irreversible dissipation (high noise, low predictability), focusing the optimization objective on short-term action penalties with lower variance and higher signal-to-noise ratio; while when the system is running smoothly (low loss), it restores long-cycle credit allocation. This mechanism effectively solves the problem of algorithm convergence collapse caused by physical irreversible perturbations, forcing the model to stably learn energy-efficient scheduling logic during dynamic evolution.

[0054] This invention integrates the heat transfer loss rate in engineering thermodynamics with the qualification trace credit allocation mechanism in deep reinforcement learning. The design cleverly utilizes the principle that thermodynamic entropy increase represents physical irreversibility, adaptively truncating the historical backtracking span of the reward signal when the system experiences high energy dissipation.

[0055] Second embodiment:

[0056] This embodiment aims to elaborate on the complete technical solution process of the present invention from a global perspective of systems engineering and data flow. This embodiment focuses on demonstrating how the various functional modules collaboratively process multi-source heterogeneous data, achieve deep coupling between physical mechanisms and intelligent algorithms, and ultimately output and iterate the optimal energy scheduling strategy.

[0057] As one embodiment of the present invention, refer to Figure 1 A flowchart of a regional energy load evolution prediction method based on deep reinforcement learning, referring to... Figure 2 Multi-source data physical causal alignment and graph representation logic diagram, refer to Figure 3 Physical mechanism embedding and asynchronous multi-scale scheduling decision logic diagram.

[0058] During system initialization and data preprocessing, the frequency domain decoupling and physical causal alignment process for multi-source heterogeneous data is first executed. The intelligent data acquisition engine in the regional energy system backend continuously receives multi-source monitoring time-series data from the power grid monitoring and control terminal, pipeline flow meters, and temperature and humidity sensors. Addressing the inherent physical gap in electrothermal response rates, the frequency domain analysis module is invoked, utilizing a wavelet transform algorithm with predefined basis functions and decomposition levels to reduce the dimensionality and separate the original mixed time-series signals. Detailed coefficient sequences characterizing load abrupt changes and high-frequency oscillations are extracted and reconstructed into electrical energy detail sequences; simultaneously, approximate coefficient sequences characterizing the slow evolution trend of thermodynamics are extracted and reconstructed into thermal energy approximate sequences. To eliminate dynamic hysteresis caused by spatial transmission, the operating frequency of the pipeline circulation pump and the preset pipeline cross-sectional area are acquired in real time to determine the physical flow velocity of the medium. Combined with the pipeline topological distance from the heat source to each load node in the spatial topology map, the dynamic time delay reference time constant is calculated. This time constant is converted into an integer step size, and the expansion rate parameter of the one-dimensional causal dilation convolution kernel is precisely set accordingly. By applying a forward one-sided convolution to the approximate thermal energy sequence using the causal dilated convolution kernel, the delayed thermal response is pushed forward on the algorithm's virtual time axis to achieve spatiotemporal causal alignment. Simultaneously, the weight update mechanism of the convolutional network is activated to absorb time-delay drift errors caused by environmental disturbances. Subsequently, the spatial thermal decay factor calculated based on real-time heat transfer parameters is extracted synchronously and multiplied element-wise with the translated sequence as a dynamic physical scalar, achieving thermodynamic dissipation contraction of the waveform amplitude at the algorithm level. After the above dual physical correction of time and amplitude, it is strictly aligned with the original high-frequency electrical energy command that triggers the response on the same time section and cascaded and stitched along the channel dimension, outputting a fused feature matrix that eliminates time-delay misalignment and contains physical dissipation information.

[0059] After data alignment is completed, the system enters the dynamic construction phase of the spatiotemporal thermodynamic situational awareness map. The system graph calculation module parses the device identifiers in the fusion feature matrix, links with the underlying energy flow relation database, instantiates the node set and initial connection edge set in the graph network, and encapsulates the corresponding aligned feature slices as node initial feature vectors. For the strong nonlinear characteristics exhibited by the phase change thermal storage devices in the microgrid, a dynamic weight mapping mechanism is triggered. When the temperature of the phase change device is detected to be in the sensible heat state range, the reciprocal of the material's static thermal resistance is directly used as a constant weight. Once the temperature enters the solid-liquid phase change isothermal range (i.e., the mushy region), a preset enthalpy-temperature characteristic curve is extracted, and the first-order partial derivative of the enthalpy value at the current temperature is calculated through numerical differentiation to obtain the equivalent heat capacity peak value. To prevent gradient explosion in the graph neural network at this step point, a Gaussian smoothing function is applied to filter, limit, and normalize the peak value, transforming it into a continuous and bounded latent heat characteristic gain coefficient. The product of this coefficient and the reciprocal of the transient thermal impedance is used as a new dynamic weight, updating the adjacency matrix of the graph topology in real time. This approach perfectly reveals the hidden thermodynamic transitions within the material as dynamic fluctuations in the connectivity strength of the graph network topology.

[0060] Subsequently, an enhanced state vector extraction process embedding physical mechanisms was executed. A system of partial differential equations was constructed, including unsteady-state heat conduction, phase change latent heat source, and surface convection heat transfer terms. Specifically, based on real-time collected crop leaf area index and photosynthetic radiation, the latent heat flux of dynamic transpiration in crops was calculated and added to the above equation system as a negative heat source sink term accompanying the biological metabolic cycle. The phase change node features were substituted into the modified system of partial differential equations for finite-difference iteration to prospectively obtain the theoretical temperature evolution value and theoretical remaining heat storage capacity for the next prediction step. Through loss correction of the regularized mapping network, these theoretical physical values ​​were pressed into the hidden layer of the neural network to generate a background thermal state component with physical boundary constraints. Simultaneously, the amplitude standard deviation and the sum of the first-order absolute differences of the electrical energy detail sequence within the sliding window were calculated to extract the transient fluctuation characteristics of the power grid. The physical background thermal state component and the transient fluctuation characteristics of the power grid were concatenated to finally generate an enhanced Markov state vector that combines thermodynamic foresight and power grid pressure perception.

[0061] Entering the core decision-making phase, the architecture of the Markov state vector input deep reinforcement learning agent is enhanced through a two-layer asynchronous architecture. This architecture achieves multi-scale frequency domain decoupling in the action space: the electrical energy action generation network is controlled by a high-frequency clock cycle, independently and frequently outputting converter charging and discharging active power commands based on the state vector to smooth out transient fluctuations in the power grid; the thermal energy action generation network is controlled by a low-frequency clock cycle (whose period is a positive integer multiple of the high-frequency clock cycle), outputting action commands for water valves or phase change circulating pumps as needed. During the intervals of the low-frequency clock, the commands for thermal energy equipment are forcibly locked to constant values ​​by a zero-order hold, thereby avoiding oscillations and wear caused by high-frequency signals to hydraulic machinery.

[0062] Finally, an evaluation and model iteration process based on the thermodynamic entropy increase mechanism is executed. After environmental interaction, immediate and delayed rewards are calculated separately. After the issuance of electrical energy actions, the negative Euclidean distance mapping value of the real-time grid-connected power deviation from the baseline is immediately captured as the immediate reward; while after the issuance of thermal energy actions, the integral error of the ambient temperature deviation from the comfort target is collected only after the thermophysical response constant time has elapsed. These two types of empirical trajectories are stored in dual empirical playback data pools of different time lengths. When updating parameters during backpropagation, the standard Bellman equation is used to calculate the transient time difference error for high-frequency electrical energy pools; while for low-frequency thermal energy pools, the heat transfer loss rate of the regional energy system is innovatively calculated in real time, and its normalized reciprocal is used as a dynamic time decay factor and substituted into the second Bellman equation. This mechanism causes the delayed reward of actions that lead to high irreversible energy dissipation (high entropy increase) in the system to be severely reduced when backpropagating to the historical trajectory. Based on the transient and hysteresis time difference errors calculated above, gradient descent is used to optimize the weights of the two-layer network, ultimately outputting a globally convergent load evolution prediction result and a highly efficient optimal energy dispatch strategy. Thus, the technical solution of this invention achieves a complete closed loop from physical data perception, mathematical dimensionality reduction, mechanism-constrained prediction to intelligent strategy iteration.

[0063] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for predicting regional energy load evolution based on deep reinforcement learning, characterized in that, include: Multi-source monitoring data of the regional energy system is acquired, and wavelet transform decomposition is used to obtain the detailed electrical energy sequence and the approximate thermal energy sequence. Dynamic time delay offset coefficients are extracted based on pipeline flow velocity and spatial topological distance, and causal dilation convolution is applied to shift the approximate thermal energy sequence by time step to obtain the fused feature matrix. A graph topology is constructed based on the fusion feature matrix, and the physical phase change state and transient thermal impedance parameters of the phase change thermal storage device are mapped to the dynamic weights of the corresponding connecting edges in the graph topology to generate a situational awareness map. The node feature matrix of the situational awareness map is extracted, and the node features corresponding to the phase change thermal storage device are nonlinearly constrained and mapped using the physical evolution equation. Combined with the transient fluctuation characteristics of the power energy detail sequence, a state vector embedding the physical mechanism is generated. The state vector is input into the reinforcement learning model, and the power dispatching action and thermal dispatching action are output through a two-layer asynchronous architecture. The power dispatching action and thermal dispatching action are respectively associated with an instant reward signal based on the real-time grid-connected power exchange deviation and a delayed reward signal based on the integral of the ambient temperature deviation. By utilizing the time-difference credit allocation mechanism, the reinforcement learning model is updated based on the immediate reward signal and the delayed reward signal, and the regional load evolution prediction results and scheduling strategy are output.

2. The regional energy load evolution prediction method based on deep reinforcement learning according to claim 1, characterized in that, The process of decomposing the data into electrical energy detail sequences and thermal energy approximation sequences using wavelet transform, and then applying causal dilatation convolution to time-step shift the thermal energy approximation sequence, includes: extracting detail coefficients representing load abrupt changes from multi-source monitoring data using wavelet transform to reconstruct the electrical energy detail sequences; extracting approximation coefficients representing the slow evolution of thermophysical quantities to reconstruct the thermal energy approximation sequences; calculating dynamic time-delay offset coefficients by combining pipeline flow velocity and topological distance, and calculating spatial thermal decay factors representing energy loss based on pipeline heat transfer parameters and internal / external environmental temperature differences; setting the expansion rate of the causal dilatation convolution kernel according to the dynamic time-delay offset coefficients to time-shift the thermal energy approximation sequence; multiplying the calculated spatial thermal decay factor by the shifted sequence and concatenating it with the electrical energy detail sequences to obtain a fusion feature matrix.

3. The method for predicting regional energy load evolution based on deep reinforcement learning according to claim 1, characterized in that, The graph topology is constructed based on the fused feature matrix, including: extracting device identifiers from the fused feature matrix and retrieving the corresponding physical connection relationships in the energy flow database to determine the set of nodes and the initial set of connection edges in the device graph topology; encapsulating the power components, temperature components after translation and alignment, and flow components belonging to the same time section in the fused feature matrix into initial feature vectors corresponding to each node according to a preset feature index order; and generating an adjacency matrix representing the initial physical topology relationship of the regional energy system based on the initial feature vectors and the initial set of connection edges of each node to complete the construction of the graph topology.

4. The regional energy load evolution prediction method based on deep reinforcement learning according to claim 1, characterized in that, The process of mapping the physical phase change state and transient thermal impedance parameters of a phase change thermal storage device to the dynamic weights of the corresponding connecting edges in the graph topology includes: acquiring the material temperature monitoring values ​​of the phase change thermal storage device nodes in real time, and determining whether the material temperature monitoring values ​​are within the preset solid-liquid phase change temperature range; if the determination result is no, setting the dynamic weight between the node and its connected edge to the reciprocal of the static thermal resistance; if the determination result is yes, mapping the liquid phase volume fraction calculated based on the accumulated heat absorption to a phase change smoothing coefficient, and smoothing the partial derivative of the enthalpy value at the current temperature to a heat capacity reference coefficient; multiplying the phase change smoothing coefficient and the heat capacity reference coefficient to obtain the latent heat characteristic gain coefficient; and using the product of the latent heat characteristic gain coefficient and the reciprocal of the material transient thermal impedance value as the dynamic weight to update the adjacency matrix.

5. The regional energy load evolution prediction method based on deep reinforcement learning according to claim 1, characterized in that, The process of generating a state vector embedded with the physical mechanism is as follows: a set of partial differential equations containing unsteady heat conduction terms, phase change latent heat source terms, and surface convection heat transfer terms is pre-constructed as the physical evolution equation; the initial feature vector of the phase change heat storage device is input into the physical evolution equation to obtain the theoretical temperature evolution value and the theoretical remaining heat storage capacity of the phase change heat storage device in a single prediction step in the future; the theoretical temperature evolution value and the theoretical remaining heat storage capacity are input into a regularized mapping network to obtain the mapping output; the mapping output is used as a feature correction operator to act on the hidden layer output of the neural network to obtain the corrected background thermal state components; The amplitude standard deviation and the sum of the first-order absolute differences of the electrical energy detail sequence are calculated as transient fluctuation features. The transient fluctuation features are cascaded with the background thermal state components to output the state vector of the embedded physical mechanism.

6. The regional energy load evolution prediction method based on deep reinforcement learning according to claim 1, characterized in that, The two-layer asynchronous architecture outputs power scheduling and thermal scheduling actions, including: setting the execution frequency of the power action generation network in the deep reinforcement learning model as the first execution frequency, setting the execution frequency of the thermal action generation network as the second execution frequency, and the value of the first execution frequency being a positive integer multiple of the value of the second execution frequency; during the triggering period of the first execution frequency, the power action generation network outputs a power scheduling action containing the active power command for charging and discharging of the energy storage converter according to the state vector; during the triggering period of the second execution frequency, the thermal action generation network outputs a thermal scheduling action containing the frequency percentage of the phase change thermal storage tank circulating pump according to the state vector; at multiple first execution frequency nodes within the same second execution frequency period, the output value of the thermal scheduling action remains constant.

7. The regional energy load evolution prediction method based on deep reinforcement learning according to claim 1, characterized in that, The quantitative calculation methods for power dispatching actions and thermal dispatching actions, which are respectively associated with an immediate reward signal based on the real-time grid-connected power exchange deviation and a delayed reward signal based on the integral of the ambient temperature deviation, are as follows: The immediate reward signal is calculated as follows: When executing a power dispatching action, the Euclidean distance between the real-time grid-connected power exchange value and the preset baseline power value is calculated, and the negative mapping value of the Euclidean distance is used as the immediate reward signal returned to the power action generation network; The delayed reward signal is calculated as follows: After the following thermal dispatching action, a delayed observation timer is started synchronously. When the delayed observation timer reaches the preset thermophysical response constant time, the absolute value integral of the difference between the actual ambient temperature value and the target temperature value is calculated, and the negative scalar value of the absolute value integral of the difference is used as the delayed reward signal.

8. The method for predicting regional energy load evolution based on deep reinforcement learning according to claim 1, characterized in that, The reinforcement learning model is updated using a time-difference credit allocation mechanism based on immediate and delayed reward signals. This includes: constructing two experience replay data pools; the first pool stores quadruple trajectory data containing power dispatch actions at a first execution frequency step size, and the second pool stores quadruple trajectory data containing thermal dispatch actions at a second execution frequency step size; for the first pool, the transient time difference error is calculated using the first Bellman equation, and the power action generation network parameters are updated; for the second pool, the delayed reward signal is distributed backward along the time sequence to intermediate state nodes using the second Bellman equation, the hysteresis time difference error is calculated, and the thermal action generation network parameters are updated.

Images