A building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm

Abstract

The application provides a building energy consumption collaborative optimization strategy method based on a reinforcement learning algorithm, and belongs to the technical field of building energy consumption optimization. The application aligns the different-frequency sampling data through a multi-modal sensor fusion layer, outputs a hidden state vector through a belief state estimator, and equivalently sets a partially observable problem as a fully observable problem. The hidden state vector is input into a memory-enhanced recurrent world model, a long-time physical self-consistent prediction is realized by using an external differentiable memory matrix and an iterative physical correction cycle, a multi-step Hamilton forward deduction is executed by using a symplectic integrator to enhance Monte Carlo tree search, energy drift is eliminated, and a candidate scheduling decision is output. After being checked by a constraint projection layer, a continuous control instruction is output. Through continuous online updating of an artificial intelligence strategy network and a priority experience replay, the problem that the reinforcement learning strategy planning deviates from the physical feasible region due to the inaccurate world model prediction of the building energy system under the long-time thermal inertia working condition of the partially observable problem is solved.

Description

Technical Field

[0001] This invention belongs to the field of building energy consumption optimization technology, and specifically relates to a collaborative optimization strategy method for building energy consumption based on reinforcement learning algorithms. Background Technology

[0002] The coordinated optimization of building energy systems is a core research direction in the field of intelligent buildings. Existing technologies typically employ model predictive control or standard deep reinforcement learning methods to centrally schedule equipment such as chillers, pumps, and air conditioning terminals, achieving joint optimization of energy consumption and comfort through the construction of system dynamics models or online strategy iteration. In practical engineering deployments, these methods have been widely applied to energy management systems in large commercial buildings, data centers, and campuses, utilizing sensor networks to collect multi-source, multi-frequency data such as temperature, humidity, and electricity meter readings as system status inputs.

[0003] However, building thermal systems have thermal inertia on the order of hours, and standard recurrent neural networks struggle to capture long-term dependencies spanning hours due to gradient vanishing, leading to inaccurate predictions of pre-cooling and cold storage effects by the world model. Simultaneously, key physical quantities such as the internal thermal state of the chiller compressor cannot be directly measured, leaving the system in a partially observable state. Under these conditions, the policy generalization ability of standard reinforcement learning frameworks deteriorates significantly.

[0004] In existing technologies, due to the lack of explicit long-term memory mechanisms and physical constraint embedding in the world model, under conditions where building thermal inertia spans multiple hours and the system state is partially observable, the multi-step forward inference of Monte Carlo tree search produces spurious energy drift due to the cumulative error of Euler integrals. This causes the planned path to continuously deviate from the thermodynamically feasible region, and the control commands output by the policy network violate hard physical constraints. In other words, existing technologies suffer from the technical problem of inaccurate world model predictions leading to deviations in reinforcement learning policy planning from the physically feasible region under partially observable, long-term thermal inertia conditions. Summary of the Invention

[0005] In view of this, the present invention provides a building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm, which can solve the technical problem in the prior art where the world model prediction of the building energy system is inaccurate under partially observable and long-term thermal inertia conditions, causing the reinforcement learning strategy planning to deviate from the physical feasible region.

[0006] This invention is implemented as follows: This invention provides a building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm, including the following steps:

[0007] Collect multi-source sensor data from the building energy system, align the heterogeneous frequency sampled data through a multi-modal sensor fusion layer, input the data into the belief state estimator, and output the hidden state vector.

[0008] The hidden state vector and the preference weight vector are input together into the memory-enhanced cyclic world model to predict the future state sequence and uncertainty vector of the system, and the predicted values ​​are iteratively corrected.

[0009] Taking the future state sequence as input, perform symplectic integrator-enhanced Monte Carlo tree search, output candidate high-level scheduling decisions, decompose the high-level scheduling decisions into low-level device control quantities, and output continuous control commands after being verified by the constraint projection layer.

[0010] Based on the range of the system's physical deviation value, the resource allocation parameters of the memory-enhanced cyclic world model are dynamically adjusted, and based on the comparison results between the symplectic structure retention index and the physical consistency threshold, the number of frog-leap format iteration steps of the symplectic integrator-enhanced Monte Carlo tree search is adjusted.

[0011] After executing continuous control commands, actual energy consumption and comfort data are collected, a composite reward signal is constructed, and the parameters of the priority experience playback update strategy network and memory-enhanced cyclic world model are updated to output the final strategy for collaborative optimization of building energy consumption.

[0012] Specifically, the multimodal sensor fusion layer aligns the heterogeneous frequency sampling data by linearly interpolating the low-frequency signal to the high-frequency time axis and then splicing them together to form an observation vector with a unified time step, which serves as the input to the belief state estimator.

[0013] The belief state estimator is composed of gated recurrent units that compress the historical observation vector sequence into a hidden state vector, approximating the sufficient statistics of the system, so that the policy learning of the partially observable Markov decision process is equivalent to the policy learning under the fully observable setting.

[0014] The preference weight vector includes an energy consumption weight component and a comfort weight component, the sum of which is 1. It is updated in real time according to the electricity price period and population density, and serves as the policy condition input for the conditional reinforcement learning framework.

[0015] The memory-enhanced cyclic world model adopts a two-level memory structure, including short-term working memory and a long-term episodic memory bank. The long-term episodic memory bank is implemented using an external differentiable memory matrix. The attention read / write head retrieves historical embedding vectors from the long-term episodic memory bank based on a cosine similarity soft retrieval mechanism.

[0016] Specifically, the iterative correction of the predicted value involves substituting the initial output of the state prediction head into the thermal balance residual calculation module, using the residual gradient to correct the predicted value in reverse, and iterating until the residual is lower than the convergence threshold.

[0017] The symplectic integrator-enhanced Monte Carlo tree search expresses the building thermal network in Hamiltonian canonical equation form. In the tree expansion stage, a frog-leap symplectic integrator is used for multi-step forward inference. The node selection adopts a confidence upper bound formula that incorporates the symplectic structure retention index into the exploration term.

[0018] Specifically, the decomposition of high-level scheduling decisions and low-level device control quantities involves discretizing and grouping continuous control quantities in the parameterized action space, and then projecting them back into the continuous control space using the Wolpertinga architecture combined with K-nearest neighbor mapping.

[0019] The constraint projection layer embeds physical hard constraints into the policy network output layer, and projects the out-of-bounds continuous control vectors along the constraint normal direction to the feasible region boundary. The projection operation is transformed into a differentiable optimization subproblem through the Lagrange duality method.

[0020] The system physical deviation value is calculated as a weighted sum of the norm mean of the current batch uncertainty vector and the symplectic structure retention index. The CUDA shared memory allocation ratio of the long-term plot memory and the CUDA stream priority of short-term working memory updates are dynamically adjusted according to the range of the system physical deviation value.

[0021] Specifically, when the symplectic structure retention index is lower than the physical consistency threshold, the number of iteration steps for the frog-jump scheme is increased; when the symplectic structure retention index is higher than the physical consistency threshold, the number of iteration steps for the frog-jump scheme is decreased.

[0022] The composite reward signal uses the distance between the future temperature deviation predicted by the building thermal model and the boundary of the comfort zone as a potential function, and the difference in the potential function is supplemented to the environmental sparse reward to form the composite reward signal.

[0023] The priority experience replay sorts the samples in the experience replay buffer according to the temporal difference error, and samples with larger errors are extracted with higher probability for updating the parameters of the policy network and the memory-enhanced recurrent world model.

[0024] The writing of the long-term plot memory adopts a sparse update strategy. The current embedding vector is written to a new memory slot and the oldest memory slot is overwritten only when the cosine similarity between the current situation embedding vector and the nearest neighbor in the long-term plot memory is lower than the sparse writing threshold. The sparse writing threshold ranges from 0.85 to 0.95.

[0025] The dimensions of the hidden state vector range from 128 to 512; the discard rate of the uncertainty vector ranges from 0.1 to 0.3; and the convergence threshold ranges from [missing value]. The physical consistency threshold ranges from 0.7 to 0.9; the K value in the K-nearest neighbor mapping ranges from 3 to 10; the high threshold for the system's physical deviation value is 0.8, and the low threshold is 0.4.

[0026] This invention constructs a multimodal sensor fusion layer and a belief state estimator to convert partially observable problems into fully observable settings. It then uses the external differentiable memory matrix of the memory-enhanced cyclic world model to explicitly store the evolution laws of typical operating conditions and embeds the thermodynamic conservation equation into the iterative physical correction cycle, so that the long-term prediction of the world model has physical self-consistency.

[0027] Building upon this, the symplectic integrator enhances Monte Carlo tree search by replacing Euler integrals with a frog-leap scheme for multi-step forward inference, naturally preserving the symplectic geometric structure and eliminating spurious energy drift during the planning process, ensuring that candidate scheduling decisions always remain within the thermodynamically feasible region. The constraint projection layer embeds physical hard constraints into the policy network output layer, preventing control commands from going out of bounds through differentiable projection.

[0028] In summary, this invention solves the technical problem mentioned in the background art where the inaccurate predictions of the world model for building energy systems under partially observable, long-term thermal inertia conditions cause reinforcement learning policy planning to deviate from the physically feasible region. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention.

[0030] Figure 2 This is a graph showing the dynamic changes of the preference weight vector with electricity price periods during the 7-day test period and the corresponding distribution of control actions.

[0031] Figure 3 The distribution of retrieval frequency and cosine similarity score for the 128 memory slots in the long-term plot memory bank.

[0032] Figure 4 This is a comparison chart of the total enthalpy conservation errors of the multi-step derivation system using the frog-jump scheme symplectic integrator and the Euler integrator. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0034] like Figure 1 The diagram shown is a flowchart of a building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm provided by the present invention. This method includes the following steps:

[0035] S01. Collect multi-source sensor data of the building energy system, align the heterogeneous frequency sampling data through the multi-modal sensor fusion layer, input the belief state estimator, and output the hidden state vector.

[0036] S02. Input the hidden state vector and the preference weight vector into the memory-enhanced cyclic world model to predict the future state sequence and uncertainty vector of the system, and iteratively correct the predicted values.

[0037] S03. Taking the future state sequence as input, perform symplectic integrator-enhanced Monte Carlo tree search, output candidate high-level scheduling decisions, decompose the high-level scheduling decisions into low-level device control quantities, and output continuous control commands after verification by the constraint projection layer.

[0038] S04. Based on the range of the system's physical deviation value, dynamically adjust the resource allocation parameters of the memory-enhanced cyclic world model, and based on the comparison results between the symplectic structure retention index and the physical consistency threshold, adjust the number of frog-leap format iteration steps of the symplectic integrator-enhanced Monte Carlo tree search.

[0039] S05. After executing the continuous control command, collect actual energy consumption and comfort data, construct a composite reward signal, use priority experience playback to update the parameters of the strategy network and memory-enhanced cyclic world model, and output the final strategy for collaborative optimization of building energy consumption.

[0040] The specific function of the multimodal sensor fusion layer is as follows: the temperature sensor in the building energy system has a sampling period of 1 minute, and the electricity meter has a sampling period of 15 minutes. The fusion layer interpolates the low-frequency signal to the high-frequency time axis using linear interpolation, and then splices them into an observation vector with a unified time step to eliminate the time misalignment caused by different frequency sampling. The observation vector serves as the input of the belief state estimator.

[0041] The belief state estimator, composed of gated recurrent units, compresses the historical observation vector sequence into a hidden state vector, approximating sufficient statistics of the system. This vector handles physical quantities that cannot be directly measured, such as the internal thermal state of the chiller compressor and the refrigerant charge, making policy learning in a partially observable Markov decision process equivalent to policy learning under a fully observable setting. The dimension of the hidden state vector ranges from 128 to 512, and its specific value is determined by comparing the state prediction error on the validation set after training 50,000 steps in different dimensions within the target building simulation environment.

[0042] The preference weight vector is the multi-objective optimization preference allocation at the current moment, which includes an energy consumption weight component and a comfort weight component, the sum of which is 1. The preference weight vector is updated in real time with factors such as electricity price, time of day, and population density, and serves as the policy condition input for the conditional reinforcement learning framework, enabling a single policy network to cover the complete Pareto front.

[0043] The specific structure of the memory-enhanced recurrent world model is as follows: the network backbone adopts an improved gated recurrent unit, and the hidden state is expanded into a two-level memory structure. The first level is short-term working memory with a dimension of 256 and an update frequency of once every 5 minutes. The second level is a long-term episodic memory bank, implemented using an external differentiable memory matrix with a storage capacity of 128 memory slots. Each memory slot stores an embedding vector of a typical working condition, and the embedding vector has a dimension of 64. At each time step, the attention read / write head retrieves the historical embedding vector most similar to the current working condition from the long-term episodic memory bank based on a cosine similarity soft retrieval mechanism. The retrieval result is concatenated with the hidden state of the short-term working memory and then input into the state prediction head. The state prediction head consists of three fully connected layers with a modified linear unit as the activation function. It outputs a future state sequence and an uncertainty vector. The uncertainty vector is estimated using the Monte Carlo dropout method, with a dropout rate ranging from 0.1 to 0.3. The dropout operation is only applied to the state prediction head and not to the memory read / write module. The writing mechanism employs a sparse update strategy: the current embedding vector is written to a new memory slot and the oldest memory slot is overwritten only when the cosine similarity between the current state embedding vector and its nearest neighbor in the long-term plot memory is lower than the sparse writing threshold. The sparse writing threshold ranges from 0.85 to 0.95, determined by taking the middle value after statistically analyzing the joint distribution of memory slot utilization and prediction error using different thresholds on 7 days of historical data. The internal design incorporates an iterative physical correction loop: the initial output of the state prediction head's future state sequence is substituted into the thermal equilibrium residual calculation module, and the residual gradient is used to correct the predicted value in reverse. This process is repeated 2-3 times until the residual is lower than the convergence threshold. The convergence threshold ranges from [missing information - likely a value]. The inflection point value was selected by comparing the trade-off curves of iteration count and prediction accuracy under different thresholds on a building thermal simulation platform. The weight connection matrix between neurons in the network adopts a sparse masking strategy, with a connection sparsity rate ranging from 30% to 50%, determined by the correlation analysis of neuron activation within the layer; the feature map transmission between layers adopts a half-precision floating-point format to reduce memory usage; the batch size of data loop is dynamically adjusted according to the current memory availability, ranging from 32 to 256; in memory allocation, the long-term episodic memory matrix resides in memory to ensure consistent read and write latency; in memory allocation, the historical observation vector sequence buffer pre-allocates contiguous memory pages; in CUDA stream allocation, memory read and write operations and forward prediction calculations are allocated to independent CUDA streams to achieve asynchronous parallelism; in the allocation of different CUDA kernel functions, cosine similarity batch calculation and residual gradient calculation are allocated to independent kernel functions to improve parallel throughput; in layer allocation, the short-term working memory update layer is preferentially allocated low-latency register resources, while the long-term episodic memory retrieval layer is allocated shared memory resources.

[0044] The steps for establishing the training dataset for the memory-enhanced cyclic world model specifically include: collecting actual operating data of the target building over the past 1 to 3 years, including outdoor temperature and humidity, indoor temperature, chiller load rate, chilled water pump speed, air conditioning variable air volume terminal opening, fresh air ratio, and sub-meter readings; aligning the data at a 5-minute granularity and dividing it into a training set and a validation set with a ratio of 8:2; completing missing values ​​with linear interpolation before and after the time interval; and using a sliding window method to extract a sequence of 288 steps as samples to form a time-series sample set, where the 288 steps correspond to 24 hours.

[0045] The training steps of the memory-enhanced recurrent world model specifically include: taking a time-series sample set as input, using the real system state at the next time step as the prediction target, employing a weighted sum of mean squared error loss and physical thermal balance residual penalty as the total loss function, with weight coefficients ranging from 0.5 to 2.0, determined on the validation set through grid search; and using an adaptive moment estimation optimizer with an initial learning rate ranging from [value missing]. Cosine annealing scheduling; the number of training rounds ranges from 50 to 200, with early stopping condition being that the prediction error on the validation set does not decrease for 10 consecutive rounds; after training, the weights of the network backbone are frozen, and only the state prediction head and memory read / write head are retained to participate in subsequent online fine-tuning of reinforcement learning.

[0046] The technical advantages of the memory-enhanced recurrent world model are as follows: Building thermal systems have thermal inertia on the order of hours. Standard recurrent neural networks, affected by gradient vanishing, struggle to capture long-term dependencies spanning hours, leading to inaccurate predictions of pre-cooling and cold storage operations. This model explicitly stores the evolutionary patterns of typical operating conditions through an external differentiable memory matrix, retrieving them on demand using an attention mechanism. This allows the network to reuse existing experience when facing similar historical conditions, overcoming the short-memory limitations of recurrent networks. The iterative physical correction loop embeds thermodynamic conservation equations into the prediction process, ensuring the physical consistency of predicted values ​​and preventing non-physical predictions from arising due to out-of-generalization failures in purely data-driven models, thus improving the reliability of strategy planning.

[0047] The calculation method and dynamic adjustment rules for the system physical deviation value are as follows: The system physical deviation value is calculated by weighting the mean norm of the uncertainty vector of the current batch with the symplectic structure retention index, denoted as... At the same time, the CUDA shared memory allocation ratio of the long-term episodic memory is increased to 60%, while the priority of the CUDA stream for short-term working memory updates is reduced by one level. At the same time, the default allocation ratio is maintained, and the CUDA stream for updating short-term working memory has the same priority as the CUDA stream for updating long-term episodic memory. At that time, the CUDA shared memory allocation ratio of the long-term plot memory is reduced to 30%, and resources are released to the parallel kernel function of the state prediction head; the above thresholds of 0.4 and 0.8 are determined by quantile analysis of the distribution of the physical deviation value of the system after repeated testing on the simulation platform with 100 different working conditions, and the 33% and 67% quantiles are taken.

[0048] The principle and technical effect of the symplectic integrator enhancing Monte Carlo tree search are as follows: The state evolution of the building thermal network follows the laws of thermodynamic conservation. Standard Monte Carlo tree search uses Euler integrals for forward state extrapolation during tree expansion, and the accumulated error over multiple steps leads to spurious energy drift, causing the planned path to deviate from the physical feasible region. The algorithm expresses the building thermal network in Hamiltonian canonical equation form, with heat capacity corresponding to mass, temperature to generalized coordinates, heat flux to generalized momentum, and total enthalpy of the system to Hamiltonian. During the tree expansion stage, a leapfrog symplectic integrator is used for multi-step forward extrapolation. The leapfrog format naturally preserves the symplectic geometric structure, ensuring accurate conservation of total enthalpy in long-term planning and eliminating energy drift. Node selection employs an improved confidence upper bound formula, adding a symplectic structure retention index to the exploration term to guide the search along a physically consistent control trajectory. The backoff phase assigns higher statistical weights to leaf nodes with high symplectic retention indices, enabling physically rational paths to obtain stronger reinforcement signals in Monte Carlo statistics. This achieves physical consistency of the planned path without additional constraint projection, thereby improving the physical reliability of the strategy in long-term planning of building thermodynamic systems.

[0049] The symplectic structure retention index is a quantitative indicator that measures the degree of conservation of the total enthalpy of the system after multi-step derivation in the current branch of the symplectic integrator-enhanced Monte Carlo tree search. The value ranges from 0 to 1, and the higher the value, the more the derivation path of the branch conforms to the thermodynamic conservation law. The physical consistency threshold ranges from 0.7 to 0.9. It is determined by backtesting the actual energy consumption of the planned path under different thresholds on a building thermal simulation platform, and the threshold corresponding to the minimum backtesting error is used. When the symplectic structure retention index is lower than the physical consistency threshold, the number of iteration steps of the leapfrog format is increased; when the symplectic structure retention index is higher than the physical consistency threshold, the number of iteration steps of the leapfrog format is decreased.

[0050] Specifically, the hierarchical structure of high-level scheduling decisions and low-level equipment control quantities is as follows: high-level scheduling decisions are responsible for determining the target load allocation and operating mode of each subsystem, with a time scale of 15 minutes; low-level equipment control quantities are specific continuous control quantities such as chiller load rate, chilled water pump speed, air conditioning variable air volume terminal opening, and fresh air ratio, with a time scale of 1 minute; the two levels of decisions are executed by independent policy networks, and the parameterized action space discretizes and groups the continuous control quantities and projects them back to the continuous control space using the Wolpertinga architecture combined with K-nearest neighbor mapping.

[0051] The principle of the Wollpertinga architecture is as follows: the continuous control space is discretized into a finite set of actions. The policy network selects candidate actions from the discrete set, and then projects the discrete actions back into the original continuous control space through K-nearest neighbor mapping. This significantly reduces the effective search dimensionality while preserving the accuracy of the continuous control, alleviating the problem of policy gradient signals being overwhelmed by noise in high-dimensional continuous control spaces. The K value in the K-nearest neighbor mapping ranges from 3 to 10, and is determined by training the policy with different K values ​​in the target building control simulation environment, counting the number of policy convergence steps, and then selecting the K value corresponding to the fastest convergence.

[0052] The principle of the constraint projection layer is as follows: physical hard constraints such as the minimum supply temperature of chilled water not lower than 2℃, the minimum pressure ratio of the compressor, and the lower limit of the output frequency of the frequency converter are embedded into the output layer of the strategy network. The out-of-bounds continuous control vector output by the strategy network is projected to the boundary of the feasible region along the constraint normal direction. The projection operation is transformed into a differentiable optimization subproblem through the Lagrange dual method, which provides an effective gradient direction during backpropagation and avoids gradient distortion caused by direct pruning.

[0053] The composite reward signal is constructed by using the distance between the future temperature deviation predicted by the building thermal model and the boundary of the comfort zone as a potential function. The difference in potential function is used as an intermediate reward to supplement the sparse environmental reward calculated from the actual energy consumption and comfort data, thus forming a composite reward signal. This allows the policy network to obtain dense intermediate feedback before the arrival of the terminal reward, which is delayed by several hours, thereby alleviating the difficulty of time-series credit allocation.

[0054] The principle of the priority experience replay is as follows: the samples in the experience replay buffer are sorted according to the temporal difference error. The samples with larger errors are extracted with a higher probability for updating the parameters of the policy network and the memory-enhanced recurrent world model, so that the sparse composite reward signal samples can get more learning opportunities in training and accelerate policy convergence.

[0055] Optionally, the present invention also provides a computer-based approach to form a building energy consumption collaborative optimization strategy system based on reinforcement learning algorithms. The computer is equipped with a readable storage medium that stores program instructions, which are used to execute the above-described method when the computer is run.

[0056] The specific implementation of step S01 is as follows: The building energy system is equipped with various sensors, including temperature sensors, humidity sensors, and electricity meters. The sampling periods of these sensors are inconsistent; the temperature sensor has a sampling period of 1 minute, and the electricity meter has a sampling period of 15 minutes. The multimodal sensor fusion layer first establishes a unified high-frequency time axis, using 1 minute as the baseline time step. It then uses linear interpolation to fill in missing values ​​for low-frequency signals such as those from the electricity meter between adjacent sampling times, aligning all sensor data to the same time step and eliminating time misalignment caused by different frequency sampling. The aligned data from each channel are then concatenated into an observation vector at each time step, serving as the input to the belief state estimator. The belief state estimator consists of a gated recurrent unit that receives a sequence of historical observation vectors. Through a gating mechanism, it selectively retains or forgets historical information, compressing the historical sequence into a hidden state vector of fixed dimensions, ranging from 128 to 512. This hidden state vector approximates the sufficient statistics of the system and can implicitly encode physical quantities that cannot be directly measured, such as the internal thermal state of the refrigeration compressor and the amount of refrigerant charged. This transforms the policy learning of a partially observable Markov decision process into policy learning under a fully observable setting, providing a complete state representation for subsequent world model predictions.

[0057] The specific implementation of step S02 is as follows: The hidden state vector output in step S01 is concatenated with the preference weight vector at the current moment and then input into the memory-enhanced recurrent world model. The preference weight vector contains energy consumption weight components and comfort weight components, the sum of which is 1. It is updated in real time according to the current electricity price period and population density, serving as the policy condition input for the conditional reinforcement learning framework, enabling the single-policy network to cover the complete Pareto front. The backbone of the memory-enhanced recurrent world model adopts an improved gated recurrent unit, and the hidden state is expanded into a two-level memory structure. The first-level short-term working memory has a dimension of 256 and is updated every 5 minutes. The second-level long-term episodic memory bank is implemented using an external differentiable memory matrix, storing 128 memory slots, with each slot storing an embedding vector of a typical working condition. At each time step, the attention-based read / write head retrieves the historical embedding vector most similar to the current state from the long-term episodic memory using a cosine similarity soft retrieval mechanism. The retrieval result is concatenated with the hidden state of short-term working memory and input into the state prediction head. The state prediction head consists of three fully connected layers with modified linear units as activation functions, outputting a future state sequence and an uncertainty vector. The uncertainty vector is estimated using the Monte Carlo dropout method, with a dropout rate ranging from 0.1 to 0.3. Writing to the long-term episodic memory employs a sparse update strategy, performing writing only when the cosine similarity between the current state embedding vector and its nearest neighbor memory slot is below a sparse writing threshold, ranging from 0.85 to 0.95. The initial output of the state prediction head is fed into the thermal equilibrium residual calculation module, where the residual gradient is used to correct the predicted value. This process is repeated 2-3 times until the residual falls below a convergence threshold, ranging from [missing value]. This ensures that the output future state sequence satisfies the thermodynamic conservation constraints.

[0058] The specific implementation of step S03 is as follows: Using the future state sequence output from step S02 as input, a symplectic integrator-enhanced Monte Carlo tree search is performed. This search algorithm expresses the building thermal network in Hamiltonian canonical equation form, with heat capacity corresponding to mass, temperature to generalized coordinates, heat flux to generalized momentum, and total system enthalpy to Hamiltonian. During the tree expansion phase, a leapfrog symplectic integrator is used for multi-step forward inference. The leapfrog format naturally preserves the symplectic geometric structure, ensuring precise conservation of total system enthalpy in long-term planning and eliminating spurious energy drift caused by multi-step accumulation of Euler integrals. Node selection employs an improved confidence upper bound formula, incorporating a symplectic structure retention index into the exploration term to guide the search along a physically consistent control trajectory. In the backoff phase, leaf nodes with high symplectic structure retention indices are assigned higher statistical weights, providing stronger reinforcement signals for physically reasonable paths. After the search is completed, candidate high-level scheduling decisions are output. These decisions determine the target load allocation and operating mode for each subsystem, with a timescale of 15 minutes. High-level scheduling decisions are decomposed into low-level equipment control variables, including continuous control variables such as chiller load rate, chilled water pump speed, air conditioning variable air volume terminal opening, and fresh air ratio, with a time scale of 1 minute. The decomposition process uses a Wolpertinga architecture combined with K-nearest neighbor mapping to project discrete candidate actions back into the continuous control space, with K values ​​ranging from 3 to 10. The output continuous control variables are verified by a constraint projection layer. The constraint projection layer embeds physical hard constraints such as the minimum chilled water supply temperature, the minimum compressor pressure ratio, and the lower limit of the inverter output frequency into the policy network output layer. Out-of-bounds control variables are projected to the feasible region boundary along the constraint normal direction. The projection operation is transformed into a differentiable optimization subproblem using the Lagrange dual method, providing an effective gradient direction during backpropagation, and finally outputting continuous control commands.

[0059] The specific implementation of step S04 is as follows: The physical deviation value of the system is calculated by weighting the mean norm of the current batch uncertainty vector with the symplectic structure retention index, denoted as... .when At that time, the CUDA shared memory allocation ratio of the long-term plot memory is increased to 60%, while the priority of the CUDA stream for short-term working memory updates is reduced by one level to ensure low latency memory retrieval under high physical deviation conditions; when At the same time, maintain the default allocation ratio, and both types of CUDA streams have the same priority; when At the same time, the CUDA shared memory allocation ratio of the long-term plot memory is reduced to 30%, freeing up resources for the parallel kernel function of the state prediction head, thereby improving prediction throughput. The two thresholds of 0.4 and 0.8 were determined by performing quantile analysis on the distribution of the system's physical deviation values ​​after testing under 100 different operating conditions on a simulation platform, taking the 33% and 67% quantiles, respectively. Simultaneously, based on the comparison results between the symplectic structure retention index and the physical consistency threshold, the number of iteration steps for the leapfrog format is dynamically adjusted, with the physical consistency threshold ranging from 0.7 to 0.9. When the symplectic structure retention index is lower than the physical consistency threshold, the number of iteration steps for the leapfrog format is increased to improve integration accuracy; when the symplectic structure retention index is higher than the physical consistency threshold, the number of iteration steps for the leapfrog format is reduced to decrease computational overhead.

[0060] The specific implementation of step S05 is as follows: After executing the continuous control command output in step S03, actual energy consumption and comfort data are collected. The distance between the future temperature deviation predicted by the building thermal model and the boundary of the comfort zone is used as a potential function. The difference in the potential function is used as an intermediate reward to supplement the sparse environmental reward calculated from the actual energy consumption and comfort data, forming a composite reward signal. This allows the policy network to obtain dense intermediate feedback before the terminal reward arrives, alleviating the difficulty of temporal credit allocation. A priority experience replay mechanism is used to sort the samples in the experience replay buffer according to their temporal difference error. Samples with larger errors are extracted with a higher probability for updating the parameters of the policy network and the memory-enhanced recurrent world model. This allows the sparse composite reward signal samples to obtain more learning opportunities during training, accelerating policy convergence and ultimately outputting the final policy for collaborative optimization of building energy consumption.

[0061] It should be noted that the key technologies of this invention include: The memory-enhanced recurrent world model, through an external differentiable memory matrix and attention retrieval mechanism, overcomes the short-memory limitation of standard recurrent neural networks, enabling the world model to reuse the evolutionary patterns of typical historical operating conditions and maintain prediction accuracy even when building thermal inertia spans multiple hours; the iterative physical correction loop embeds the thermodynamic conservation equations into the prediction process, correcting the predicted values ​​in reverse through residual gradients, ensuring that the output future state sequence always satisfies thermal balance constraints, avoiding non-physical results generated by the purely data-driven model under distributed operating conditions; the symplectic integrator-enhanced Monte Carlo tree search replaces the Euler integral with a frog-leap symplectic integrator, ensuring accurate conservation of the system's total enthalpy in multi-step deductions from a geometrical perspective, and ensuring that the planned path does not deviate from the thermodynamic feasible region. These three technologies work synergistically to form a complete physical consistency guarantee chain from state representation and model prediction to planning search, enabling the reinforcement learning strategy to achieve reliable physical compliance control under complex operating conditions with partially observable and long-term thermal inertia, exhibiting stronger generalization stability compared to the independent application of each individual technology.

[0062] It should be noted that in the process of multi-objective online optimization of building energy systems, when a building simultaneously undertakes multiple usage modes, such as alternating high-density office work during the weekday day and low-load duty at night, and when the external electricity price frequently switches with peak and off-peak periods, there is a dynamic trade-off between the goal of minimizing energy consumption and the goal of ensuring indoor comfort. The optimal control strategy at different times is distributed at different positions on the Pareto front.

[0063] The above-mentioned technical problems arise because: traditional multi-objective reinforcement learning methods usually train a policy network independently for each set of preference weights. When the preference weights change continuously, it is impossible to smoothly interpolate between policies, and the parallel inference of multiple networks brings high computational overhead, making it difficult to deploy in real time on embedded energy management controllers; while the weighted summation single-objective method cannot find the true optimal solution in the non-convex region of the Pareto front, resulting in a significant degradation of control performance under specific preference weights.

[0064] The common solutions to the aforementioned technical problems are multi-strategy integration or linear weighted single-objective methods. However, the storage and inference costs of multi-strategy integration methods increase linearly with the number of preference combinations, and cannot cover all Pareto points in scenarios where preference weights change continuously. Linear weighted methods cannot approximate the concave region of the Pareto front when the objective function is non-convex, causing the strategy to systematically sacrifice a certain objective within a specific time period, resulting in unacceptable deviations in comfort or energy consumption indicators, and failing to meet the dynamic preference requirements of actual building operations.

[0065] This invention effectively solves this technical problem. It uses the preference weight vector as the policy condition input to a conditional reinforcement learning framework, inputting it along with the hidden state vector into a memory-enhanced recurrent world model and a policy network. This allows a single policy network to smoothly move along the Pareto front through continuous changes in the preference weight vector, eliminating the need to train a separate network for each set of preference weights. The preference weight vector is updated in real-time based on the current electricity price period and population density. During the training phase, the policy network covers the entire value range of the preference weights, learning a continuous mapping from preference weights to the optimal control action. Therefore, during the inference phase, only a single forward propagation is needed to respond to any combination of preferences, achieving complete coverage of the Pareto front. This meets the real-time requirements of multi-objective dynamic optimization of building energy systems without increasing inference computational overhead.

[0066] Specifically, the principle of this invention is:

[0067] The fundamental reason why this invention can solve the above-mentioned technical problems is that it eliminates the root causes of prediction inaccuracy and planning drift by working together at three levels: belief state modeling, long-term memory of world model, and physical constraint embedding.

[0068] First, for partially observable problems, traditional reinforcement learning directly uses sensor observations as state inputs. When key quantities such as the internal thermal state of the compressor are unmeasurable, Markov property is violated, and the policy gradient signal is distorted. This invention uses a belief state estimator composed of gated recurrent units to compress the historical observation sequence into a hidden state vector, approximating sufficient statistics of the system. This makes the partially observable Markov decision process equivalent to a fully observable setting at the policy learning level, fundamentally repairing the incompleteness of state representation.

[0069] Secondly, regarding prediction inaccuracies caused by long-term thermal inertia, the short-term memory of standard recurrent neural networks cannot capture cross-hour evolution patterns, while purely data-driven models are prone to producing non-physical predictions outside the training distribution. The memory-enhanced recurrent world model of this invention explicitly stores typical operating condition embedding vectors through an external differentiable memory matrix, and reuses historical experience on demand using a soft retrieval mechanism based on cosine similarity in the attention read / write head, overcoming the limitations of short-term memory in recurrent networks. The iterative physical correction loop reverses the predicted values ​​using the thermal equilibrium residual gradient, embedding the thermodynamic conservation equations into the prediction process to ensure the physical consistency of the predicted values, preventing the world model from generating non-physical states in long-term planning.

[0070] Furthermore, regarding the energy drift problem in multi-step Monte Carlo tree search, the local truncation error of the Euler integral accumulates over multiple steps, leading to a systematic shift in the total enthalpy of the system and causing the planned path to escape the feasible region. This invention expresses the building thermal network in Hamiltonian canonical equation form, replacing the Euler integral with a symplectic integrator in a leapfrog scheme. The leapfrog scheme naturally preserves the symplectic geometric structure, ensuring precise conservation of the total enthalpy in long-term planning. The symplectic structure retention index guides the statistical weights of node selection and the backoff phase, giving the physically rational path a stronger reinforcement signal in Monte Carlo statistics, achieving physical consistency of the planned path without additional constraint projection.

[0071] Finally, the constraint projection layer transforms the physical hard constraints into a differentiable optimization subproblem using the Lagrangian duality method. It projects out-of-bounds control variables along the constraint normal direction to the feasible region boundary, providing an effective gradient direction during backpropagation. This avoids gradient distortion caused by direct pruning and ensures the physical compliance of continuous control commands. These four mechanisms, progressively layered, collectively guarantee the prediction accuracy and physical compliance of the reinforcement learning strategy under partially observable, long-term thermal inertia conditions, thus solving the core technical problem at the fundamental level.

[0072] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0073] The specific implementation method of step S01 is as follows.

[0074] The multimodal sensor fusion layer collects multi-source sensor data from the building's energy system. The temperature sensor has a sampling period of 1 minute, and the electricity meter has a sampling period of 15 minutes. Low-frequency electricity meter signals are linearly interpolated to the high-frequency time axis. The interpolation formula is as follows:

[0075] ;

[0076] In the formula, for The meter signal after time interpolation, and These are adjacent low-frequency sampling times. and The meter reading, The default time is 15 minutes. After interpolation, the temperature signal and the meter signal are concatenated into an observation vector with a unified time step. ,in This represents the concatenated observation dimension. Observation vector. The input belief state estimator, which consists of gated recurrent units, is expressed by the hidden state update formula as follows:

[0077] ;

[0078] In the formula, for The hidden state vector output at time step 1. The value range is 128 to 512. This is the hidden state vector from the previous time step. These are the trainable parameters for the gated recurrent unit. This represents the forward computation function of the gated loop unit.

[0079] The specific implementation method of step S02 is as follows.

[0080] The hidden state vector With preference weight vector Common input memory-enhanced cyclic world model, in which As an energy consumption weighting component, As a weighted component for comfort, satisfying , Updated in real time according to electricity price, time of day, and population density. Short-term working memory. Updated every 5 minutes, the long-term plot memory bank is an externally differentiable memory matrix. Each row stores a typical working condition embedding vector. The attention-based read / write head uses a soft retrieval formula based on cosine similarity, as follows:

[0081] ;

[0082] In the formula, This is the query vector for the current operating condition, derived from the current hidden state vector. Obtained through linear transformation, For the first Embedding vectors of memory slots, For the first The retrieval weight of each memory slot, This is an index for traversing memory slots. The search results are... , For the weighted retrieval vector, and The concatenated input is the state prediction head, which consists of three fully connected layers with modified linear units as the activation function. The output is a sequence of future states. and uncertainty vector ,in To predict the number of time-domain steps, For the system state dimension, the uncertainty vector is estimated using the Monte Carlo dropout method, with a dropout rate ranging from 0.1 to 0.3. The sparse write strategy judgment conditions are stated as follows:

[0083] ;

[0084] In the formula, For long-term episodic memory bank and The memory slot vector with the highest cosine similarity The sparse writing threshold, ranging from 0.85 to 0.95, is determined by taking the median value after statistically analyzing the joint distribution of memory slot utilization and prediction error over 7 days of historical data. When the above conditions are met, [the following will be implemented]. Write to the new memory slot and overwrite the oldest memory slot. Iterative physical correction loops will provide an initial prediction. Substituting into the thermal balance residual calculation module, the residual formula is expressed as follows:

[0085] ;

[0086] In the formula, For the first Predicting temperature conditions step by step. This is a normalized value for temperature status reference. For time step, This is a reference value for the thermal time constant. For the first Predicting heat flow step by step, For the first Step heat loss, This is a reference normalized value for heat flow. For heat capacity, This is the normalized reference value for heat capacity. The residual is a dimensionless thermal equilibrium residual. The residual gradient is used to correct the predicted value in reverse, iterating 2-3 times until... , The range is The inflection point value was selected by comparing the trade-off curves between the number of iterations and prediction accuracy on a building thermal simulation platform. The total training loss function is expressed as follows:

[0087] ;

[0088] In the formula, For the first The true system state of a sample For the corresponding predicted value, The total number of samples in the batch. The physical residual penalty weights, ranging from 0.5 to 2.0, are determined on the validation set through grid search. An adaptive moment estimation optimizer is employed, with an initial learning rate ranging from [value missing]. The number of training rounds ranges from 50 to 200.

[0089] The specific implementation method of step S03 is as follows.

[0090] future state sequence For input, perform a symplectic integrator-enhanced Monte Carlo tree search. Write the building thermal network in Hamiltonian canonical form, with heat capacity corresponding to mass and temperature corresponding to mass. Corresponding generalized coordinates heat flow Corresponding generalized momentum With the total enthalpy of the system The forward derivation formula for the symplectic integrator in the leapfrog scheme is expressed as follows:

[0091] ,

[0092] ,

[0093] ;

[0094] In the formula, Let be the step size of the symplectic integral. This is the generalized momentum reference time normalized value. For generalized coordinate reference time normalized value, This is the reference normalized value for the Hamiltonian. For generalized coordinate reference normalized values, For generalized momentum reference normalized value, Here, represents the integration step number. All partial derivative terms are dimensionless. The node selection is expressed using the improved confidence upper bound formula as follows:

[0095] ;

[0096] In the formula, For nodes Cumulative returns In return for the reference normalized value, The empirical value for the coefficient of exploration is 1.414. This represents the number of visits to the parent node. For nodes Number of visits, The normalized value is used as a reference for the number of visits. This is the weighting coefficient for the retention of symplectic structures, with an empirical value of 0.1 to 0.5. For nodes The symplectic structure retention index for the corresponding branch; a higher value indicates that the derivation path of that branch conforms more closely to the laws of thermodynamic conservation. All three of the above terms are dimensionless. The rollback phase... Higher leaf nodes are assigned higher statistical weights, as expressed by the following formula:

[0097] ;

[0098] In the formula, This is the temperature coefficient, with an empirical value of 2.0. leaf node The backoff statistical weights, The leaf node traversal index is used. A symplectic integrator enhances the Monte Carlo tree search output, providing candidate high-level scheduling decisions. These high-level scheduling decisions determine the target load allocation and operating mode for each subsystem, with a time scale of 15 minutes. This is decomposed into low-level device control variables, including chiller load rate, chilled water pump speed, air conditioning variable air volume terminal opening, and fresh air ratio, with a time scale of 1 minute. Both levels of decisions are executed by an independent policy network, employing a Wolpertinga architecture combined with... Nearest neighbor mapping projects discrete actions back into the continuous control space. The formula for nearest neighbor mapping projection is expressed as follows:

[0099] ;

[0100] In the formula, This is the continuous control quantity after projection. These are candidate actions selected by the policy network from the set of discrete actions. For the set of discrete actions, the first... A neighboring action, This is the reference normalized value for continuous control quantities. The value ranges from 3 to 10, and is determined by different values ​​in the target building control simulation environment. After training and counting the convergence steps of the policies, the value corresponding to the fastest convergence is selected. The constrained projection layer will control the out-of-bounds variables. Projecting along the constraint normal direction, the projection formula is as follows:

[0101] ;

[0102] In the formula, This represents the feasible controllable quantity after projection. The feasible region is defined by physical constraints such as the minimum supply temperature of chilled water not being lower than 2℃. The projection operation is transformed into a differentiable optimization subproblem using the Lagrange duality method to ensure gradient effectiveness.

[0103] The specific implementation method of step S04 is as follows.

[0104] The formula for calculating the system physical deviation value is expressed as follows:

[0105] ;

[0106] In the formula, Let be the norm mean of the uncertainty vector for the current batch. This is a reference normalized value for the uncertainty. The average retention index of the symmetric structure in the current batch is calculated using the following formula: ,in This represents the number of search branches in the current batch. For the first The symplectic structure retention index of the branch, and For the weighting coefficients, satisfying Each experience point is set to 0.5. This represents the physical deviation value of the dimensionless system. At the same time, the shared memory allocation ratio of the long-term episodic memory bank is increased to 60%, while the priority of the short-term working memory update stream is reduced by one level. At the same time, maintain the default allocation ratio and ensure that both streams have the same priority. At that time, the shared memory allocation ratio of the long-term plot memory is reduced to 30%, releasing resources to the parallel kernel function of the state prediction head. The above threshold is determined by testing 100 different operating conditions on a simulation platform. Quantile analysis was performed on the distribution, using the 33% and 67% quantiles for determination. The mean value of the symmetric structure retention index... Below the physical consistency threshold When, increase the number of iterations for the frog-jump format; when Higher than At the same time, reduce the number of iterations in the frog-leap format. The range is 0.7 to 0.9. The threshold is determined by backtesting the actual energy consumption of the planned path under different thresholds on a building thermal simulation platform, and the threshold corresponding to the minimum backtesting error is used.

[0107] The specific implementation method of step S05 is as follows.

[0108] After executing continuous control commands, actual energy consumption and comfort data are collected to construct a composite reward signal. The distance between the future temperature deviation predicted by the building thermal model and the boundary of the comfort zone is used as the potential function. The formula for calculating the potential function is: In the formula for Predicted indoor temperature at any time The center temperature of the comfort range. The width is half of the comfort zone; all three values ​​are in degrees Celsius (°C). Dimensionless. The potential function difference is used to supplement the sparse reward of the environment. The formula for the composite reward signal is as follows:

[0109] ;

[0110] In the formula, for Constantly combine reward signals, The environmental sparsity reward is calculated from actual energy consumption and comfort data. This is the scaling factor for the potential function difference, with an empirical value of 0.1 to 1.0. These represent the potential function differences between adjacent time points; all three terms are dimensionless. Prioritize empirical playback by applying temporal difference errors to buffer samples. Sort, number The probability formula for selecting a sample is expressed as follows:

[0111] ;

[0112] In the formula, For the first The temporal difference error of each sample, This is the reference normalized value for timing difference error. To prevent a small constant with a probability of zero, an empirical value of 0.01 is used. The priority index, ranging from 0.5 to 1.0, indicates that samples with larger errors are selected with higher probability. This index is used to update the parameters of the policy network and the memory-enhanced recurrent world model, and outputs the final policy for the collaborative optimization of building energy consumption.

[0113] To better understand and implement this invention, the following is a specific application scenario of the invention, Example 2: To verify the effectiveness of the invention, technicians built a test environment and deployed the method of the invention on the energy management simulation platform of a certain complex building. Using the operating data of 7 consecutive days under typical summer conditions as the test scenario, the prediction accuracy and planning physical compliance of the building energy consumption collaborative optimization strategy were comprehensively evaluated.

[0114] The building has a total floor area of ​​approximately 42,000 square meters. The air conditioning system consists of two centrifugal chillers, three chilled water pumps, and several variable air volume (VAV) terminals. The energy management simulation platform operates at a 5-minute granularity, with temperature sensors sampling every 1 minute and electricity meters every 15 minutes. Technicians collected actual operating data from the building over the past two years, including outdoor temperature and humidity, indoor temperature, chiller load rate, chilled water pump speed, VAV terminal opening, fresh air ratio, and sub-meter readings. After aligning to a 5-minute granularity, the data was divided into training and validation sets at an 8:2 ratio. Missing values ​​were filled in using linear interpolation before and after the specified time. A 288-step sequence was extracted using a sliding window method to form the time-series sample set.

[0115] The multimodal sensor fusion layer linearly interpolates the low-frequency signal from the electricity meter to a 1-minute high-frequency time axis, and concatenates it with high-frequency sensor data such as temperature data to form an observation vector with a unified time step, eliminating time misalignment caused by inter-frequency sampling. The belief state estimator uses a gated recurrent unit with a hidden state dimension of 256 to compress the historical 288-step observation vector sequence, outputting the hidden state vector as input to the subsequent model. The memory-enhanced recurrent world model uses a weighted sum of mean squared error loss and physical thermal balance residual penalty as the total loss function during the training phase. The weight coefficients are determined to be 1.2 through grid search. An adaptive moment estimation optimizer is used, with an initial learning rate of... Cosine annealing scheduling is used. After 100 training rounds, training is terminated early if the prediction error on the validation set does not decrease for 10 consecutive rounds. After training, the weights of the network backbone are frozen, and only the state prediction head and memory read / write head are retained to participate in subsequent online fine-tuning of reinforcement learning.

[0116] The long-term plot memory bank consists of 128 memory slots, each with an embedding vector dimension of 64. A sparse write threshold of 0.90 was set. The joint distribution of memory slot utilization and prediction error under this threshold was verified using 7 days of historical data, confirming that this value represents a trade-off between utilization and accuracy. The uncertainty vector was estimated using the Monte Carlo dropout method, with a dropout rate set at 0.2. The convergence threshold for the iterative physical correction loop was set to... After 2 to 3 iterations, the residuals can converge, ensuring the physical consistency of the predicted values.

[0117] The symplectic integrator-enhanced Monte Carlo tree search is based on the Hamiltonian canonical equations of the building thermal network. During the tree expansion stage, a leapfrog symplectic integrator is used for multi-step forward inference. The physical consistency threshold is set to 0.80. When the symplectic structure retention index is below 0.80, the number of leapfrog iterations is increased; when it is above 0.80, the number of iterations is decreased, dynamically balancing integration accuracy and computational cost. The decomposition of low-level equipment control variables adopts the Wolpertinga architecture. The K-nearest neighbor mapping K-value is set to 5. Verification in the target building control simulation environment shows that K=5 results in the shortest strategy convergence steps. The constraint projection layer embeds physical hard constraints such as the minimum chilled water supply temperature of 2℃, the minimum compressor pressure ratio, and the lower limit of the inverter output frequency into the strategy network output layer. Out-of-bounds control variables are projected using the Lagrange dual method for differentiable projection.

[0118] During the 7-day continuous testing period, the technicians recorded the state prediction error, symplectic structure retention index, system physical deviation value, and strategy convergence steps of the world model under different operating conditions, as shown in Table 1.

[0119] Table 1. Statistics of key indicators during the 7-day test period

[0120]

[0121] As shown in Table 1, when a higher system physical deviation value occurs on the 5th day, the symplectic structure retention index decreases accordingly. The system automatically increases the number of iteration steps of the frog-jump format, and the constraint out-of-bounds rate remains at 0%, verifying the effectiveness of the dynamic adjustment mechanism.

[0122] The preference weight vector adjusts the energy consumption weight component to 0.75 during peak electricity price periods and to 0.30 during off-peak electricity price periods, as follows: Figure 2 As shown, the dynamic curves of the preference weight vector and the corresponding control action distribution during the 7-day test period are illustrated. The system can directly output the corresponding optimal control command from a single policy network under different preference weights without switching policy networks, verifying the ability of the conditional reinforcement learning framework to cover the complete Pareto front.

[0123] The utilization of memory slots in the long-term plot memory bank during the 7-day test period is as follows: Figure 3 As shown, the distribution of the number of times each slot in the 128 memory slots was retrieved, as well as the score distribution of the cosine similarity soft retrieval, are presented. This indicates that the memory bank can accurately retrieve historical similar working conditions under typical working conditions. The sparse writing mechanism effectively avoids the memory slots being repeatedly covered by homogeneous working conditions, thus maintaining the diversity of the memory bank.

[0124] In the construction of the composite reward signal, the potential function is calculated by the distance of the indoor temperature from the boundary of the comfort zone. The temporal difference error distribution of the policy network during the test is shown in Table 2. High error samples are extracted with higher probability through the priority experience replay mechanism, which accelerates the policy convergence in the sparse reward scenario.

[0125] Table 2. Statistical Table of Timing Differential Error Distribution for Priority Experience Playback

[0126]

[0127] The physical consistency verification results of the planned path for the symplectic integrator-enhanced Monte Carlo tree search are as follows: Figure 4 As shown, the comparison curves of the symplectic integrator and the standard Euler integrator under different planning steps are presented, which intuitively illustrate that the leapfrog scheme effectively suppresses energy drift in multi-step derivation.

[0128] This invention brings the following technological advancements compared to traditional methods. Traditional world models rely on standard recurrent neural networks to capture system dynamics, but are unable to effectively model the thermal inertia process across hours due to gradient vanishing. This invention explicitly stores the evolutionary patterns of typical operating conditions through an external differentiable memory matrix and reuses historical experience on demand using an attention retrieval mechanism, breaking through the time span limitation of standard recurrent networks at the memory structure level. Traditional Monte Carlo tree search uses Euler integrals for multi-step deduction, and the accumulation of errors in multiple steps leads to energy drift, causing the planned path to deviate from the physical feasible region. This invention uses a leapfrog symplectic integrator to ensure accurate conservation of total enthalpy at the geometric structure level, achieving thermodynamic consistency of the planned path without additional constraint projection. Traditional constraint processing methods directly prune out-of-bounds control variables, resulting in gradient signal distortion. This invention preserves the effective gradient direction through differentiable Lagrange dual projection, ensuring the smooth differentiability of physical constraints during policy optimization.

[0129] It should be noted that the variables involved in this invention are explained in detail in Tables 3 and 4.

[0130] Table 3. Variable Explanation Table (Part 1)

[0131]

[0132] Table 4. Variable Explanation Table (Part Two)

[0133]

[0134] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A collaborative optimization strategy for building energy consumption based on reinforcement learning algorithms, characterized in that, Includes the following steps: Collect multi-source sensor data from the building energy system, align the heterogeneous frequency sampled data through a multi-modal sensor fusion layer, input the data into the belief state estimator, and output the hidden state vector. The hidden state vector and the preference weight vector are input together into the memory-enhanced cyclic world model to predict the future state sequence and uncertainty vector of the system, and the predicted values ​​are iteratively corrected. Taking the future state sequence as input, perform symplectic integrator-enhanced Monte Carlo tree search, output candidate high-level scheduling decisions, decompose the high-level scheduling decisions into low-level device control quantities, and output continuous control commands after being verified by the constraint projection layer. Based on the range of the system's physical deviation value, the resource allocation parameters of the memory-enhanced cyclic world model are dynamically adjusted, and based on the comparison results between the symplectic structure retention index and the physical consistency threshold, the number of frog-leap format iteration steps of the symplectic integrator-enhanced Monte Carlo tree search is adjusted. After executing continuous control commands, actual energy consumption and comfort data are collected, a composite reward signal is constructed, and the parameters of the priority experience playback update strategy network and memory-enhanced cyclic world model are updated to output the final strategy for collaborative optimization of building energy consumption.

2. The building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm according to claim 1, characterized in that, The multimodal sensor fusion layer aligns the heterogeneous frequency sampling data by linearly interpolating the low-frequency signal to the high-frequency time axis and then splicing them together to form an observation vector with a unified time step, which serves as the input to the belief state estimator.

3. The building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm according to claim 2, characterized in that, The belief state estimator is composed of gated recurrent units that compress the historical observation vector sequence into a hidden state vector, approximating the sufficient statistics of the system, so that the policy learning of the partially observable Markov decision process is equivalent to the policy learning under the fully observable setting.

4. The building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm according to claim 3, characterized in that, The preference weight vector includes an energy consumption weight component and a comfort weight component, the sum of which is 1. It is updated in real time according to the electricity price period and population density, and serves as the policy condition input for the conditional reinforcement learning framework.

5. The building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm according to claim 4, characterized in that, The memory-enhanced cyclic world model adopts a two-level memory structure, including short-term working memory and a long-term episodic memory bank. The long-term episodic memory bank is implemented using an external differentiable memory matrix. The attention-based read / write head retrieves historical embedding vectors from the long-term episodic memory bank based on a cosine similarity soft retrieval mechanism.

6. The building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm according to claim 5, characterized in that, The iterative correction of the predicted value specifically involves substituting the initial output of the state prediction head into the thermal balance residual calculation module, using the residual gradient to correct the predicted value in reverse, and iterating until the residual is lower than the convergence threshold.

7. The building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm according to claim 6, characterized in that, The symplectic integrator-enhanced Monte Carlo tree search expresses the building thermal network in Hamiltonian canonical equation form. During the tree expansion stage, a frog-leap symplectic integrator is used for multi-step forward inference. Node selection adopts a confidence upper bound formula that incorporates a symplectic structure retention index into the exploration term.

8. The building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm according to claim 7, characterized in that, The decomposition of high-level scheduling decisions and low-level device control quantities specifically involves discretizing and grouping continuous control quantities in the parameterized action space, and then projecting them back into the continuous control space using the Wolpertinga architecture combined with K-nearest neighbor mapping.

9. The building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm according to claim 8, characterized in that, The constraint projection layer embeds physical hard constraints into the policy network output layer, and projects the out-of-bounds continuous control vectors along the constraint normal direction to the feasible region boundary. The projection operation is transformed into a differentiable optimization subproblem through the Lagrange duality method.

10. The building energy consumption collaborative optimization strategy method based on reinforcement learning algorithm according to claim 9, characterized in that, The system physical deviation value is calculated as a weighted sum of the norm mean of the current batch uncertainty vector and the symplectic structure retention index. Based on the range of the system physical deviation value, the CUDA shared memory allocation ratio of the long-term plot memory and the CUDA stream priority of short-term working memory updates are dynamically adjusted.

Images