Physical information security reinforcement learning control method and device, electronic equipment and medium

Abstract

The application provides a physical information security reinforcement learning control method and device, electronic equipment and medium. The method comprises the following steps: collecting operation data of a ground source heat pump system and a service area thereof, and constructing an operation state vector. The operation state vector is input into a policy network based on maximum entropy reinforcement learning, a normalized control action is obtained through reparameterization sampling, and the control action is issued as an actual control instruction of the ground source heat pump system after physical mapping. Based on energy consumption, comfort constraints, safety constraints, action changes and ground temperature drift, a reward and constraint cost are constructed. The training mechanism of the policy network and the double critic network is updated by using a preset mechanism, and the preset mechanism comprises constraint awareness entropy scheduling, a physical information critic and a Lagrange soft constraint. The trained policy network is deployed to an upper controller or a building control system of the ground source heat pump system to output control instructions in real time in a cycle. The operation energy consumption of the ground source heat pump system is reduced, and the comfort control quality is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent control technology for building heating, ventilation and air conditioning, and in particular to a physical information security reinforcement learning control method, device, electronic device and medium. Background Technology

[0002] Ground source heat pump systems utilize shallow geothermal energy for building cooling and heating, offering advantages such as high energy efficiency and stable operation. They have been widely applied in large public buildings (including but not limited to high-speed railway stations, airport terminals, commercial complexes, hospitals, and office buildings) for cooling and heating. The actual load of ground source heat pump systems is typically characterized by high occupant mobility, significant time-of-day variations, and substantial weather influences. Their operation and control require simultaneous consideration of energy consumption, comfort, equipment safety, and the thermal balance of the buried pipes, placing high demands on the multi-objective coordination and robustness of the control strategy.

[0003] Commonly used ground source heat pump control schemes in existing engineering projects include proportional-integral-derivative (PID) control, rule-based control, and model predictive control (MPC). Among them, PID control and rule-based control are simple in structure and easy to implement, but their control parameters rely on manual experience for adjustment, making it difficult to achieve global coordinated optimization of multiple objectives such as energy consumption, comfort, and safety under varying operating conditions. MPC solves the optimal control sequence for several future steps through rolling optimization, and can take into account future load and equipment constraints, but its online optimization performance is highly dependent on the accuracy of the building and equipment models, and its control performance is prone to degradation under actual engineering conditions such as model parameter mismatch, sensor noise, and sudden load changes.

[0004] Therefore, there is an urgent need for an intelligent control method that can simultaneously take into account the physical consistency of safety exploration and value assessment, the dynamic balance of multiple constraints, and adapt to the physical characteristics of ground source heat pump systems. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of related technologies and provide a physical information security reinforcement learning control method, device, electronic device and medium.

[0006] In a first aspect, embodiments of this application provide a physical information security reinforcement learning control method, comprising: Collect operational data of the ground source heat pump system and its service area to construct an operational status vector; the operational status vector includes indoor average temperature, outdoor temperature, equivalent ground temperature of buried pipes, indoor humidity, personnel load ratio, and hourly code; The operating state vector is input into a policy network based on maximum entropy reinforcement learning, and normalized control actions are obtained through reparameterized sampling. The control actions are then physically mapped and issued as actual control commands for the ground source heat pump system. The control commands include water supply temperature, circulation flow rate, and humidity regulation. Based on energy consumption, comfort constraints, safety constraints, motion changes, and ground temperature drift, a reward and constraint cost are constructed. The training mechanism of the policy network and the dual critic network is updated collaboratively using a preset mechanism, which includes: constraint-aware entropy scheduling, physical information critic, and Lagrange soft constraints. The trained strategy network is deployed to the supervisory controller or building control system of the ground source heat pump system to output control commands in real time on a periodic basis.

[0007] Secondly, embodiments of this application provide a physical information security reinforcement learning control device, comprising: The data acquisition module is used to collect operational data of the ground source heat pump system and its service area; The state construction module is used to construct an operating state vector based on the operating data; the operating state vector includes indoor average temperature, outdoor temperature, equivalent ground temperature of buried pipes, indoor humidity, personnel load ratio, and hourly code. The decision module includes a policy network based on maximum entropy reinforcement learning and a dual critic network. The decision module is used to input the running state vector into the policy network, obtain normalized control actions through reparameterized sampling, and issue the control actions as actual control commands for the ground source heat pump system after physical mapping. The control commands include water supply temperature, circulation flow rate, and humidity adjustment. The reward calculation module is used to construct rewards and constraint costs based on energy consumption, comfort constraints, safety constraints, motion changes, and ground temperature drift. The constraint update module is used to collaboratively update the training mechanism of the policy network and the dual critic network using a preset mechanism, which includes: constraint-aware entropy scheduling, physical information critic, and Lagrange soft constraints. The control output module is used to deploy the trained strategy network to the upper controller of the ground source heat pump system or the building control system to output control commands in real time on a periodic basis.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the control method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method described in the first aspect.

[0010] The technical solution provided in this application enables the controller to actively adjust the exploration intensity according to the system's safety status during operation without relying on a precise online optimization model. It also achieves coordinated optimization of energy consumption, comfort, and safety through physical information commentators and Lagrange soft constraints. This reduces the operating energy consumption of ground source heat pump systems and improves the quality of comfort control. It can be applied to intelligent HVAC control in large public buildings (including but not limited to high-speed railway stations, airport terminals, commercial complexes, hospitals, and office buildings).

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0013] Figure 1 This is a flowchart of the reinforcement learning control method for physical information security in this application.

[0014] Figure 2 This is a schematic diagram of the module structure of the physical information security reinforcement learning control device of this application.

[0015] Figure 3 This is a schematic diagram of the module structure of a physical information security reinforcement learning control device according to another embodiment of this application.

[0016] Figure 4 This is a flowchart illustrating the scheduling method of the constraint-aware entropy scheduling module in this application.

[0017] Figure 5 This is a flowchart illustrating the training methods for the Physical Information Critic Training Module and the Lagrange Soft Constraint Training Module of this application. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0019] In recent years, deep reinforcement learning has been introduced into the field of HVAC control because it can learn multi-objective optimal control strategies online through interaction with the environment. However, directly applying general deep reinforcement learning methods to ground source heat pump systems has the following significant drawbacks: (1) General maximum entropy reinforcement learning methods need to maintain a high exploration entropy in the policy training stage to improve policy diversity. However, when the system state is close to the lower limit of water supply temperature, the upper limit of compressor load, or dangerous critical states such as indoor overheating or overcooling, it still maintains a high random exploration intensity, which may produce dangerous control actions and lead to unsafe equipment operation. (2) Existing reinforcement learning methods that introduce physical constraints mostly use action layer projection or fixed physical regularization terms, which only passively constrain the action end and fail to embed thermodynamic consistency constraints in the value evaluation stage of the critic network. Therefore, the critic network may still give a high value estimate for state-action pairs that do not conform to thermodynamic relations, resulting in slow convergence and poor stability of policy learning. (3) In existing methods, the weights among energy consumption, comfort, and safety items are usually fixed values, which need to be manually set by domain experts before training. When the operating conditions change, the seasons change, or the electricity price changes, the fixed weights are difficult to adapt in real time, often resulting in a multi-objective imbalance such as "good energy saving but poor comfort" or "good comfort but high energy consumption", which requires manual readjustment for several days to several weeks; (4) Existing methods do not adequately support the physical characteristics of ground source heat pump systems, such as the equivalent ground temperature of buried pipes, the nonlinear coupling between the coefficient of performance (COP) and flow rate, and the long-term thermal imbalance caused by ground temperature drift. Therefore, it is difficult to guarantee the equipment safety and energy balance of ground source heat pump systems under long-term operation. (5) Existing methods are not robust enough to uncertainties in engineering practice such as sensor noise, building model parameter mismatch and sudden internal load disturbances. Strategies that perform well in simulation environment often experience performance degradation when deployed in real engineering.

[0020] To address the shortcomings of the aforementioned related technologies, this application proposes a physical information security reinforcement learning control method, device, electronic device, and medium. This application aims to combine physical information constraints with maximum entropy reinforcement learning for intelligent control of ground source heat pump systems. Security / physical / constraint mechanisms are simultaneously embedded at the strategy exploration, value assessment, and target weighting stages, enabling the controller to automatically find the optimal balance between energy consumption, comfort, and safety.

[0021] To better understand the technical solution of this application, the physical information security reinforcement learning control method, apparatus, electronic device, and medium of this application will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the features in the following embodiments and implementations can be combined with each other.

[0022] See Figure 1 As shown, embodiments of this application provide a physical information security reinforcement learning control method for ground source heat pump systems, including: Step S1: Collect operational data of the ground source heat pump system and its service area, and construct an operational state vector. The operational state vector... Including average indoor temperature Outdoor temperature equivalent geothermal temperature of buried pipes Indoor humidity The personnel load ratio Occ (range [0,1]) and hour code h (i.e., the current hour, (0-23)). Among them, Let t be the running state vector at time t.

[0023] The collection of operational data from ground source heat pump systems and their service areas can be achieved through the following methods: The indoor temperature is collected by multiple indoor temperature sensors installed in the service area. The arithmetic mean of the indoor temperature at each measuring point is calculated as the indoor average temperature, which directly reflects the thermal comfort level of the service area and is a core indicator for measuring the cooling / heating effect of the ground source heat pump system.

[0024] The outdoor temperature, collected by temperature sensors installed outdoors in the service area, is a key external disturbance factor affecting the heat load of the building envelope and has important guiding significance for system load prediction.

[0025] Temperature sensors buried in the soil around the underground pipe collect the soil temperature, and the equivalent ground temperature of the underground pipe is calculated based on the soil temperature around the underground pipe. This reflects the heat exchange capacity of the ground source side and is an important parameter for judging the soil heat balance state and the long-term sustainability of the system.

[0026] The indoor humidity is collected by humidity sensors installed in the service area, which affects the thermal comfort of the human body and the latent heat load of the building, and together with the temperature, constitutes a complete description of the indoor thermal environment.

[0027] The ratio of real-time number of people in the service area to the designed capacity is used to determine the personnel load ratio, which is the main source of heat disturbance inside the building and directly affects the real-time load demand of the system.

[0028] The hourly code is assigned a value based on the hourly interval in which the data is collected, reflecting the periodic pattern of building energy consumption (such as weekday / non-weekday, daytime / nighttime, etc.), which helps the model learn the temporal distribution characteristics of the load.

[0029] Step S2, the running state vector Input a policy network based on maximum entropy reinforcement learning, and obtain normalized control actions through reparameterization sampling. The control action After physical mapping, the commands are issued as actual control instructions for the ground source heat pump system. These control instructions include supply water temperature, circulation flow rate, and humidity regulation. The supply water temperature control instruction is used to adjust the unit's outlet water temperature; the circulation flow rate control instruction is used to adjust the pump speed to change the circulation flow rate; and the humidity regulation control instruction is used to adjust the indoor humidity of the service area. Optionally, the policy network can be a Gaussian policy network. .in, Let be the normalized control action vector at time t. The normalized control action component for water supply temperature, with a value range of [-1, 1]; The normalized control action component for the circulating flow rate, with a value range of [-1, 1]; This is the normalized control action component for humidity regulation, with a value range of [-1, 1]. Let θ be a Gaussian policy network, θ be the policy network parameters, μ be the action mean vector, and σ be the action standard deviation vector.

[0030] Understandably, the operating state vector is input into a pre-trained policy network based on maximum entropy reinforcement learning. The policy network outputs an action probability distribution according to the current operating state. Normalized control actions are obtained by sampling from the action probability distribution through a reparameterization sampling method. The normalized control actions are then converted into actual control quantities through physical mapping, generating actual control commands for the ground source heat pump system and sending them to the actuators.

[0031] Step S3: Construct a reward based on energy consumption, comfort constraints, safety constraints, motion changes, and ground temperature drift. With constraint cost .in, The instantaneous reward at time t. Let be the constraint cost vector at time t. The consequences of compromising one's rights for comfort are accumulated. Accumulated costs for safety violations.

[0032] Understandably, rewards and constraints are constructed based on energy consumption, comfort constraints, safety constraints, action changes, and ground temperature drift. The cumulative cost of comfort violations is used to evaluate whether the indoor thermal and humidity environment of the service area meets the comfort requirements of personnel. The cumulative cost of safety violations is used to ensure that all equipment in the ground source heat pump system operates within a safe operating range, avoiding equipment damage or system failure due to improper control. The cost of action changes is used to suppress frequent fluctuations in control actions, avoiding increased equipment wear and energy consumption caused by frequent actuator movements. The cost of ground temperature drift is used to constrain the soil thermal imbalance problem in the long-term operation of the ground source heat pump system. Under long-term cooling or heating conditions, if the heat extraction and heat dissipation are unbalanced, the soil temperature around the buried pipes will continuously rise or fall, ultimately affecting the system's heat exchange efficiency and long-term operational sustainability.

[0033] In this way, a comprehensive reward value can be calculated based on the real-time operating status and control actions of the ground source heat pump system. The comprehensive reward value is used to evaluate the quality of the current control actions and guide the updating of the reinforcement learning strategy network.

[0034] Step S4: The training mechanism of the policy network and the dual critic network is updated collaboratively using a preset mechanism, which includes: constraint-aware entropy scheduling, physical information critic, and Lagrange soft constraints.

[0035] Understandably, the constraint-aware entropy scheduling described herein is a cross-disciplinary technology paradigm combining constraint-awareness and entropy scheduling in reinforcement learning, multi-agent systems, and resource optimization. It is used to dynamically adjust the entropy regularization coefficient based on the degree to which the current policy's constraints are satisfied, adaptively balancing the exploratory nature of the policy with constraint satisfaction. In the standard SAC (Soft Actor-Critic) algorithm, the effective entropy coefficient (also known as the entropy regularization coefficient or temperature parameter) is typically a fixed hyperparameter, requiring extensive manual tuning. In the dynamic environment of ground source heat pump system control, a fixed effective entropy coefficient is insufficient to simultaneously adapt to the exploration and utilization needs under different operating conditions. This implementation proposes a constraint-aware entropy scheduling mechanism that can dynamically adjust the effective entropy coefficient based on the degree to which the current policy satisfies the constraints.

[0036] The physical information critic, a module in the interdisciplinary field of artificial intelligence and scientific computing, is used to evaluate the physical consistency of generated results. Commonly found in Physics-Informed Machine Learning frameworks, it embeds the physical mechanisms of a ground source heat pump system into the critic network's objective function, guiding the network's value estimation with a physical model and improving sample efficiency and estimation accuracy. Ground source heat pump systems operate according to well-defined physical laws, including energy conservation, heat transfer equations, and buried pipe thermal response models. Traditional reinforcement learning methods rely entirely on data-driven approaches, ignoring these known physical priors, leading to low sample efficiency and potential value estimations that violate physical laws. This implementation proposes a physical information critic mechanism that embeds the physical mechanisms of the ground source heat pump system into the critic network's objective function.

[0037] The Lagrange soft constraint refers to a relaxation strategy that uses the Lagrange multiplier method to transform "hard constraints" in the optimization problem into penalty terms in the objective function, thereby allowing the constraints to be violated at a certain cost. This strategy converts the safety constraint cost and geothermal drift cost constructed in step S3 into Lagrange multiplier terms. The penalty weights of the constraints are adaptively adjusted during the strategy optimization process using the Lagrange duality method, achieving soft constraint satisfaction. Ground source heat pump system control involves multiple constraints (safety constraints, geothermal drift constraints, etc.). Traditional hard constraint handling methods (such as action pruning and projection) may disrupt the continuity of actions or lead to suboptimal solutions in the continuous action space. This implementation uses the Lagrange method to transform hard constraints into soft constraints.

[0038] The synergistic relationship among the three mechanisms is as follows: Updating the policy network using the constraint-aware entropy scheduling mechanism dynamically adjusts the effective entropy coefficient based on the current constraint satisfaction (constraint cost information fed back by Lagrange multipliers), influencing the exploration-exploitation balance of the policy network. Updating the policy network using the physics information critic mechanism provides a value estimate consistent with physical laws, serving as a target signal for policy network optimization. Updating the policy network using the Lagrange soft constraint mechanism transforms the constraint cost into a penalty term for policy optimization through Lagrange multipliers. The feedback loop reflects the policy network update results in the constraint cost of the next round, thus affecting the adjustment of the effective entropy coefficient in the constraint-aware entropy scheduling and the updating of the Lagrange multipliers.

[0039] Step S5: Deploy the trained strategy network to the host controller or building control system of the ground source heat pump system to output control commands in real time on a periodic basis.

[0040] Understandably, at the beginning of each control cycle, the supervisory controller or building control system reads real-time operating data of the ground source heat pump system and its service area through the communication gateway. The collected real-time operating data is constructed into an operating state vector according to the format defined in step S1. If any sensor data acquisition fails or exceeds a reasonable range, valid data from the previous moment is used as a replacement, and the abnormal event is recorded in the system log. The constructed operating state vector is then input into the deployed strategy network for forward computation. The strategy network outputs the probability distribution parameters of the normalized control action, and the normalized control action is obtained through reparameterization sampling. The normalized control action is then converted into actual control commands according to the physical mapping rules defined in step S2, including water supply temperature, circulation flow rate, and humidity adjustment. The control commands can be, for example, actual control quantities, such as water supply temperature setpoint, circulation flow rate setpoint, and humidity adjustment quantity. The obtained actual control quantities are encapsulated into control commands conforming to the communication protocol and sent to the corresponding field actuators through the communication bus. After completing the above steps, the supervisory controller or building control system enters a waiting state until the trigger time of the next control cycle arrives, and the above steps are repeated.

[0041] Through the above technical solution, this application enables the controller to actively adjust the exploration intensity according to the system safety status during operation without relying on a precise online optimization model. It also achieves synergistic optimization of energy consumption, comfort, and safety through physical information commentators and Lagrange soft constraints. This can reduce the operating energy consumption of ground source heat pump systems and improve the quality of comfort control. It can be applied to intelligent HVAC control in large public buildings (including but not limited to high-speed railway stations, airport terminals, commercial complexes, hospitals, and office buildings).

[0042] In some alternative implementations, the hour encoding includes hour sine encoding. And hour cosine encoding During the online deployment phase, the operating state vector is a seven-dimensional core state vector, including indoor average temperature, outdoor temperature, equivalent ground temperature of buried pipes, indoor humidity, personnel load ratio, hourly sine coding, and hourly cosine coding. During the offline training phase, the operating state vector, in addition to the seven-dimensional core state vector, also includes at least one of the following extended dimensions: temperature difference between regions, outdoor temperature prediction for the next few hours, time-of-use electricity price segment, and the number of hours between peak and off-peak transitions. The physical significance of the time-of-use electricity price extended dimension is that it can guide the strategy network to proactively increase energy consumption during off-peak hours (such as pre-cooling / pre-heating buildings, storing energy in buried pipes), and reduce energy consumption or utilize building thermal inertia to maintain the indoor environment during peak hours, thereby minimizing electricity costs. Furthermore, the feature of the number of hours between peak and off-peak transitions allows the strategy network to be forward-looking, enabling it to proactively adjust control strategies before electricity price transitions (such as increasing energy storage before the end of the off-peak period), achieving cross-period economic optimization.

[0043] Understandably, the online deployment phase is constrained by the cost of sensor deployment, communication bandwidth, and real-time inference requirements in practical engineering. Using a simplified seven-dimensional core state vector can reduce system complexity and maintenance costs. The offline training phase, however, can utilize historical data and weather forecast data for batch replay training, unrestricted by real-time sensor deployment. Therefore, it can include more extended dimensions in the running state vector, enabling the agent to learn richer environmental features and improving the generalization and cost-effectiveness of the policy.

[0044] Among them, the temperature difference between different areas can refer to the deviation of the temperature in each area from the average temperature of the entire building. This is used to characterize the unevenness of heat distribution inside a building; outdoor temperature forecasts for the next few hours can be forecasts for outdoor temperatures at 1 hour, 3 hours, 6 hours, etc., to provide the strategy network with forward-looking meteorological change information. Time-of-use pricing segments and the number of hours from peak-valley transitions are used to guide the strategy network to increase energy consumption during off-peak hours and reduce energy consumption during peak hours, thereby reducing operating electricity costs.

[0045] Based on the above technical solution, this implementation adopts a two-stage differentiated state vector design strategy of "dimensionality increase during training and dimensionality reduction during deployment". After training, the policy network can adapt to the complete state space including the extended dimensions, and is also robust to the core state subspace - by randomly discarding some extended dimensions during training, the policy network can still output reasonable control actions by relying only on the core state vectors during deployment.

[0046] In some optional implementations, the physical mapping satisfies the following relationship: the calculation of the water supply temperature control command is as follows: ; The calculation of the flow control command is as follows: ; The calculation of the control command for the adjustment is as follows: ; in, This is a water supply temperature control command. This is the lower limit of the water supply temperature, which is 5℃ under cooling conditions. It is the minimum allowable water supply temperature under cooling conditions. Below this temperature, there is a risk of evaporator frosting or pipe freezing. This is the upper limit of the water supply temperature, which is 18℃ under cooling conditions. It is the highest effective water supply temperature under cooling conditions. If the temperature is higher than this, the building will not be able to effectively bear the cooling load. The normalized control action component for water supply temperature, with a value range of [-1, 1]; This is a flow ratio control command for the circulating water pump, with a value range of [0,1]. The normalized control action component for the circulating flow rate, with a value range of [-1, 1]; This is a humidity control command. This is the preset lower limit for humidity adjustment. This is the preset upper limit for humidity adjustment. This is the normalized control action component for humidity regulation, with a value range of [-1, 1].

[0047] Understandably, after reparameterization sampling, the policy network outputs a three-dimensional normalized control action vector. The three components correspond to water supply temperature regulation, circulating water pump flow regulation, and humidity regulation, respectively, with each component taking values ​​in the range [-1, 1]. Each of the three components is independently converted into a physically meaningful control command through its respective physical mapping function. After completing the physical mapping, the three control commands are encapsulated into a control command conforming to the communication protocol and sent to the corresponding field actuators.

[0048] The physical mapping relationship of the water supply temperature control command can be as follows: when When =-1, =5 + (-1 + 1) × 0.5 × (18 - 5) = 5℃ (full load cooling); when When = -0.5, =5 + (-0.5 + 1) × 0.5 × (18 - 5) = 8.25℃; when When =0, =5 + (0 + 1) × 0.5 × (18 - 5) = 11.5℃ (intermediate load); when When =0.5, =5 + (0.5 + 1) × 0.5 × (18 - 5) = 14.75℃; when When =1, =5+(1+1)×0.5×(18-5)=18℃(Minimum cooling / shutdown edge).

[0049] The physical mapping relationship of the cyclic flow control command can be as follows: when When =-1, =0 (the water pump is stopped or running at the minimum permissible frequency); when When =0, =0.5 (the water pump operates at 50% of its rated flow); when When =1, =1 (the water pump operates at 100% of the design flow rate / rated frequency).

[0050] and This is a preset humidity setpoint boundary (unit: %RH). Based on human comfort requirements, =40%RH =60%RH, the physical mapping relationship of humidity regulation control commands can be as follows: when When =-1, = 40%RH (Minimum humidity setting, biased towards dehumidification); when When =0, = 50%RH (intermediate humidity setting); when When =1, = 60%RH (maximum humidity setting, tends to humidify).

[0051] In some alternative implementations, the training mechanism employs a Gaussian policy network. With the dual critic network (i.e., the dual-Q critic network); the policy network outputs the action mean vector μ and the log-standard deviation. And use calculation formula , Continuous control actions are sampled using reparameterization techniques; the target value of the dual critic network is adopted. and The method of taking the minimum value and using the calculation formula. Superimposed strategy entropy; Where θ is the policy network parameter, s is the running state vector, μ is the action mean vector, and σ is the action standard deviation vector; For Gaussian policy networks, and For two independent Q-commenter networks, φ1 and φ2 are their network parameters; ε is the standard normal sampling noise. y is a standard multivariate normal distribution; y is the temporal difference target value of the dual critic network; r is the instantaneous reward; γ is the discount factor, with a value range of [0,1]. For the corresponding target commentator network (i=1,2), Polyak Averaging is used for soft updates; s′ is the state at the next time step, and a′ is the action at the next time step; This represents the effective entropy coefficient. Polyak averaging is a trajectory smoothing technique for iterative optimization algorithms such as stochastic gradient descent, which can significantly improve the generalization ability and stability of deep learning models.

[0052] Understandably, the policy network takes the running state vector as input and outputs the action mean and logarithmic standard deviation, avoiding the constraint that the standard deviation must be positive, improving numerical stability, and preventing gradient saturation. Continuous control action samples are then obtained from a Gaussian distribution using a reparameterization technique. The dual critic networks independently estimate the state-action value. When calculating the temporal difference objective, a minimum double Q-value approach is used to suppress positive bias in value estimation, and a policy entropy term is superimposed to achieve the maximum entropy reinforcement learning objective. Independent parameter initialization of the two critic networks (using different random seeds) ensures that they produce differentiated value estimates during training, thus providing an effective basis for the "minimum double Q-value" policy.

[0053] In some implementations, the training mechanism for constraint-aware entropy scheduling includes: obtaining a smoothed temperature by performing an exponential moving average on the indoor temperature; calculating the distance between the smoothed temperature and the safe temperature boundary, as well as the undercooling or overheating deviation relative to the set temperature; obtaining a safe scaling factor and a comfort scaling factor through a Sigmoid function, respectively; taking the smaller value between the safe scaling factor and the comfort scaling factor as the entropy scaling factor and limiting it within a preset range; and obtaining the effective entropy coefficient by multiplying the base entropy coefficient by the entropy scaling factor.

[0054] Understandably, the steps of the constraint-aware entropy scheduling include: Regarding the indoor temperature according to Perform an exponential moving average; Through calculation formula Calculate the distance between the smoothed temperature and the safe temperature boundary; Calculation formula under refrigeration conditions Calculate the subcooling deviation using the formula under heating conditions. Calculate the overheating deviation; According to the calculation formula , Calculate the safety scaling factor and the comfort scaling factor separately; According to the calculation formula Take the smaller value and limit it to a preset range; According to the calculation formula The effective entropy coefficient is obtained.

[0055] Where d is the distance between the smoothing temperature and the safety boundary, and β is the moving average coefficient (range [0.8, 0.99]). The average indoor temperature. This is the exponential moving average of the indoor temperature. The exponential moving average of the safety boundary temperature is given, where g is the deviation from supercooling or superheating. To set the temperature, For safety scaling factor, For comfort scaling factor, For the Sigmoid function, Here, m is the slope parameter for the safety term, and m is the safety margin parameter. For the slope parameter of the comfort term, Based on comfort standards, This is the final entropy scaling factor. The effective entropy coefficient, It is the basic entropy coefficient.

[0056] Optionally, the Sigmoid function is calculated as follows: Both the safety scaling factor and the comfort scaling factor are implemented using piecewise monotonic functions; where, Let x be the independent variable and e be the natural constant. Thus, by calculating the safety scaling factor and the comfort scaling factor using the Sigmoid function, the discrete "constraint satisfied / violated" binary judgment is transformed into a continuous scaling value within the [0,1] interval. Compared to the traditional reinforcement learning scheduling method where the entropy coefficient remains constant or undergoes hard switching at constraint boundaries, the smooth mapping using the Sigmoid function in this application allows the entropy coefficient to change continuously with the proximity of the constraint, avoiding policy oscillations and training non-convergence caused by abrupt changes in the entropy coefficient. Simultaneously, by using the formula for calculating the final entropy scaling factor to impose upper and lower bounds on the scaling factor, it effectively prevents the entropy coefficient from approaching zero in extreme cases (leading to a complete loss of policy exploration) or exceeding a reasonable range (leading to excessive policy randomness), ensuring the long-term stability and convergence of the training process.

[0057] Through the above technical solution, this application decouples the equipment safety constraints and indoor thermal comfort constraints of a ground source heat pump system at the entropy scheduling level by separately calculating a safety scaling factor and a comfort scaling factor. The safety scaling factor is calculated based on the deviation between the exponential moving average of the indoor temperature and the safe temperature threshold, independently reflecting the "danger distance" of the system from the safety boundary; the comfort scaling factor is calculated based on the undercooling / overheating deviation between the indoor temperature and the set temperature, independently reflecting the "uncomfortable distance" of the system from the comfort boundary. The two scaling factors measure the "stress level" of the system's current state from two different dimensions of safety and comfort, and by taking the smaller value, the final effective entropy coefficient is always dominated by the more pressing constraint under the current state. This mechanism realizes structured coordination of entropy scheduling under multiple constraints, avoiding scheduling imbalance problems caused by a single constraint dominating or coupling conflicts between constraints.

[0058] Furthermore, before calculating safety deviations and overcooling / overheating deviations, the indoor temperature is first processed using an exponential moving average. This preprocessing step effectively filters out temperature spikes and random fluctuations caused by measurement noise, local airflow disturbances, or instantaneous human activity from the indoor temperature sensor, enabling the entropy scheduling mechanism to make decisions based on "temperature trends" rather than "instantaneous temperature values." Compared to existing schemes that directly use raw temperature measurements to constrain sensing, the moving average processing in this application effectively prevents misadjustment of the entropy coefficient due to accidental sensor anomalies, avoids frequent switching of the exploration-exploitation mode due to the strategy's oversensitivity to noise, and significantly enhances the robustness of the entropy scheduling mechanism to environmental noise and measurement uncertainties.

[0059] In some implementations, the training mechanism for the physical information critic includes: constructing the physical information critic loss based on the degree of mismatch between cooling demand and cooling effort, and weighting the physical residuals to the temporal difference loss of the dual critic network; and setting the cooling demand to 0 when the indoor temperature is lower than the set temperature, so that the dual critic network reduces the value estimation of ineffective cooling actions.

[0060] Understandably, this application proposes a physical information critic training mechanism. Its loss function is constructed based on the degree of mismatch between cooling demand and cooling effort, and the physical residuals are weighted and superimposed onto the temporal difference loss of the dual critic network. Here, "cooling demand" represents the building's objective need for cooling capacity under the current thermal environment (determined by the deviation between indoor temperature and set temperature), and "cooling effort" represents the cooling cost incurred by the current control actions of the ground source heat pump system (determined by action variables such as supply water temperature and circulation flow rate). When the cooling demand is small but the cooling effort is large (similar to "overkill"), the physical residual is positive, imposing an additional penalty on the critic network; conversely, when the cooling demand is large but the cooling effort is insufficient (similar to "underkill"), the physical residual also penalizes the critic network, guiding it to reduce its value estimation of "overkill" actions.

[0061] Therefore, the introduction of the physical residual term makes the value estimation of the commentator network no longer solely dependent on the data-driven temporal difference error, but simultaneously constrained by the conservation of energy and the fundamental laws of thermodynamics. This fundamentally avoids the "physically intuitive" value function that pure data-driven methods may learn, enabling the commentator network to output physically reasonable and engineering-credible value estimates even in regions where training data is sparse or insufficiently covered.

[0062] Furthermore, when the indoor temperature is lower than the set temperature, the cooling demand is set to 0, causing the dual-critic network to reduce its value estimation of ineffective cooling actions. The technical effect of this mechanism is that the critic network will reduce the selection of actions that excessively cool when cooling is not needed during subsequent policy evaluations, and the policy network will gradually learn to avoid outputting ineffective cooling actions under overly cold conditions during optimization. Compared to the traditional reinforcement learning reward function that only indirectly suppresses excessive cooling through energy consumption costs, the "cooling demand zeroing + physical residual penalty" mechanism of this application directly constrains value based on thermodynamic causal logic, resulting in faster convergence speed, stronger physical interpretability, and effectively eliminating ineffective cooling energy consumption caused by control lag or blind policy exploration, significantly reducing energy waste in ground source heat pump systems under partial load conditions.

[0063] In some implementations, the cooling requirement is calculated as follows: The formula for calculating the cooling effort is: The formula for calculating the physical residual is as follows: The formula for calculating the total loss of the dual-critic network is as follows: ;in , , ; Where D represents cooling requirements. Indoor temperature, To set the temperature, k is the temperature normalization coefficient; E is the cooling effort. For the normalized action component of the water supply temperature, For the normalized motion components of the cyclic flow, The normalized control action component for the original water supply temperature, with a value range of [-1, 1]; The normalized control action component of the original circulating flow, with a value range of [-1, 1]; For physical residuals, The total loss for the dual-critic network, For time-series difference loss, For the commentator network, s is the state, a is the action, y is the temporal difference target value, and w is the physical residual weight.

[0064] Understandably, the formula for calculating cooling demand in this application constructs a normalized metric for cooling demand with clear physical meaning, achieving a standardized characterization of thermal comfort deviation. Its physical meaning is: ( The value represents the degree and direction of the deviation of the current indoor temperature from the set temperature. Dividing it by k achieves dimensionlessness. clip(·, 0, 1) limits the cooling demand to the interval [0, 1] - when the indoor temperature is lower than or equal to the set temperature, D = 0 (no cooling demand), when the indoor temperature is higher than the set temperature by more than k, D = 1 (full cooling demand), and when it is between the two, D changes continuously in the interval (0, 1). Compared to existing technologies that directly use the absolute value of temperature deviation as a reward or penalty, the normalized cooling demand proposed in this application has the following advantages: First, the cooling demand is uniformly mapped to the standard range of [0, 1], which is naturally aligned with the numerical range of cooling effort, facilitating direct comparison of physical residuals and error calculation. Second, the k-value, as an adjustable parameter, allows engineers to adjust the sensitivity of the cooling demand to temperature deviation according to the building's thermal inertia and control precision requirements, exhibiting good engineering adaptability. Third, the clip operation ensures that the cooling demand will not exceed the physically reasonable range due to extreme high temperatures, avoiding gradient instability caused by numerical dimension imbalance during training.

[0065] The formula for calculating cooling effort constructs a measure of cooling effort that couples supply water temperature and circulation flow rate, accurately characterizing the comprehensive cost of cooling behavior. Its physical meaning is: This characterizes the degree of decrease in water supply temperature; the lower the water supply temperature ( The smaller the size, the greater the cooling effort. Characterizes the level of investment in circulating flow; the greater the circulating flow ( The larger the supply water temperature, the greater the cooling effort; the product of the two reflects the combined physical cost of achieving high cooling effort by simultaneously paying for "lower supply water temperature" and "greater circulation flow".

[0066] The physical residual is calculated as the square of the difference between cooling effort and cooling demand. Constructing the physical residual in squared form achieves symmetric smooth penalty for supply-demand mismatch, and its technical effect is as follows: the squared form ensures that a positive physical residual penalty is generated regardless of whether E>D (supply exceeds demand, over-cooling) or E<D (demand exceeds supply, insufficient cooling), and the penalty increases quadratically with the increase of the supply-demand deviation. This symmetric penalty mechanism enables the critic network to be effectively physically constrained in both situations of "oversupply" and "undersupply": when supply exceeds demand, the value estimation of excessive cooling actions is reduced to save energy consumption; when demand exceeds supply, the value estimation of cooling actions that cannot meet the demand is reduced to avoid invalid operation.

[0067] The calculation formula for the total loss of the dual critic network integrates physical residuals through regularized superposition, realizing collaborative optimization of physical guidance and data driving, and its beneficial effects are as follows: the physical residual term is integrated into the total critic loss in the form of an additional regularization term rather than a substitution term, and the temporal difference loss is still the main learning objective of the critic network (driving the critic network to fit the real cumulative return), and the physical residual term only serves as an auxiliary constraint to guide the critic network to learn in a direction that conforms to the laws of thermodynamics.

[0068] During the training process, the gradient magnitudes of the temporal difference loss and the physical residual loss are relatively balanced, and there will be no situation where one gradient dominates and the other gradient is submerged, thereby avoiding the problems of training instability or slow convergence caused by dimensional imbalance. The parameter update of the critic network can simultaneously receive effective gradient signals from two directions of "reward prediction error" and "physical mismatch error" at each step, realizing "physics-data dual-driven" collaborative training in the true sense.

[0069] In some embodiments, the physical residual is an energy balance residual calculated based on envelope heat transfer, internal heat load, water supply side cooling capacity and coefficient of performance COP, and the calculation formula of the physical residual is ; wherein, is the energy balance residual, is the net heat gain on the building side calculated from the envelope heat transfer and the internal heat load; is the net cooling capacity on the air conditioning side calculated from the water supply side cooling capacity and the coefficient of performance COP; is the square of the 2-norm.

[0070] Understandably, compared to the aforementioned physical residual calculation formula based on the matching degree between cooling demand and cooling effort, the energy balance residual in this implementation is directly rooted in the first law of thermodynamics (the law of conservation of energy), possessing a stronger physical theoretical foundation. An energy balance residual based on the first law of thermodynamics is constructed, achieving global consistency of physical constraints. By introducing heat exchange in the building envelope, physical modeling of the building's thermal characteristics is achieved. Indirect measurement of the system's actual cooling capacity is achieved using COP and cooling capacity on the water supply side, avoiding the engineering difficulties of direct measurement. It can also complement the physical residual of the previous implementation, allowing for flexible selection or parallel use depending on data availability. This enhances the applicability of physical information commentators in new buildings or data-scarce scenarios.

[0071] In some implementations, the Lagrange soft constraint includes a comfort constraint multiplier, a safety constraint multiplier, and a policy loss. The training mechanism for the Lagrange soft constraint includes updating the comfort constraint multiplier and the safety constraint multiplier according to a Lagrange soft constraint update rule, wherein the update rule for the comfort constraint multiplier is as follows: The formula for calculating the strategy loss is: ; in, For comfort-constrained multipliers, To constrain the multipliers, and These are the learning rates for the comfort-constrained multipliers and the safety-constrained multipliers, respectively. and These are the threshold values ​​for the comfort constraint multipliers and the safety constraint multipliers, respectively. The consequences of compromising one's rights for comfort are accumulated. Accumulated costs for safety violations; To constrain the upper limit of the multipliers for comfort, To ensure safety, an upper limit is imposed on the multipliers; For strategic losses, The effective entropy coefficient, For Gaussian policy networks, For the commentator network, s represents the state and a represents the action. Optionally, the Lagrange soft constraint includes setting upper limits for the comfort constraint multiplier and the safety constraint multiplier respectively, to prevent the comfort constraint multiplier and the safety constraint multiplier from increasing infinitely during training and causing policy distortion.

[0072] Understandably, by employing the aforementioned Lagrange soft-constraint training mechanism, the comfort constraint multiplier and the safety constraint multiplier adjust independently during the update process without interfering with each other. The two constraint multipliers can evolve independently at their respective suitable time scales, achieving adaptive coordinated optimization of multi-time-scale constraints. This achieves decoupled and independent adjustment of comfort and safety constraints, overcoming the problem of multi-constraint coupling conflicts.

[0073] This application imposes upper bound constraints on both Lagrange multipliers, preventing policy instability caused by excessive penalties. This ensures that even in extreme cases where constraints cannot be immediately satisfied, the Lagrange penalty term will not dominate the policy loss, allowing the policy network to still consider other optimization objectives and output smooth, safe, and physically feasible control actions. Simultaneously, the clip operation guarantees that the multipliers are always non-negative and do not exceed the upper bound, mathematically guaranteeing the boundedness and well-posedness of the policy optimization problem. Furthermore, this application introduces a constraint threshold into the multiplier update rule. and By designing constraint thresholds, the "soft boundary" constraint is satisfied, thus avoiding rigidity issues.

[0074] The formula for calculating policy loss allows for a dynamic adaptive balancing mechanism that integrates constraints "softly" into the policy optimization process, rather than imposing them on the policy network as hard constraints or fixed penalties. This gives the policy network the ability to make flexible decisions near the constraint boundaries, avoiding the problems of "over-constraint" (the policy is too conservative) or "under-constraint" (constraints are frequently violated) caused by fixed penalty weights. It achieves a dynamic adaptive balance between constraint costs and the main optimization objective.

[0075] This application configures independent learning rate parameters, constraint thresholds, and upper limits for the comfort constraint multiplier and the safety constraint multiplier, respectively. The ability to configure multi-dimensional parameters independently enables the Ben Lagrange soft constraint mechanism to have high engineering adaptability and scenario flexibility, and can be widely applied to ground source heat pump system control scenarios in different types of buildings such as office buildings, shopping malls, hospitals, and data centers. The differentiated configuration of comfort constraint and safety constraint multipliers gives engineering flexibility and adjustability.

[0076] The Lagrange soft constraint mechanism, together with the aforementioned physical information commentator and constraint-aware entropy scheduling, forms a collaborative framework: the physical information commentator ensures, from the value estimation level, that the policy network tends to choose physically rational and energy-efficient actions, providing a "physical value benchmark"; the Lagrange soft constraint adaptively adjusts the penalty weight of constraints in policy optimization from a macroscopic level based on the long-term constraint satisfaction, providing "constraint adaptive adjustment"; the constraint-aware entropy scheduling adjusts the exploration intensity of the policy in real time from a microscopic level based on the constraint tension of the current state, providing "real-time exploration inhibition"; the three mechanisms jointly constrain the behavioral space of the policy network from the three dimensions of value benchmark, long-term constraints, and real-time exploration, respectively. Through the above collaborative optimization, the reinforcement learning control framework simultaneously achieves the physical rationality, constraint satisfaction, training stability, and operational economy of the control strategy in the safety-critical application scenario of ground source heat pump systems, constituting a complete technical barrier that distinguishes it from existing reinforcement learning control schemes.

[0077] In some implementations, the rewards include at least two of the following: energy consumption, time-of-use electricity pricing, temperature deviation, overcooling penalty, humidity violation, safety violation, action change penalty, and ground temperature drift penalty.

[0078] In this embodiment, the reward includes energy consumption, time-of-use electricity price, temperature deviation, overcooling penalty, humidity violation, safety violation, action change penalty, and ground temperature drift penalty. The reward is obtained by summing all of the above items. When the indoor temperature is higher than the set temperature and the energy consumption of the ground source heat pump system is lower than the preset energy consumption threshold (i.e., when...), the reward is awarded. and (At that time), the reward also includes items that are not considered as penalties; The formula for calculating the reward is: The calculation formula for items not considered as penalties is as follows: ; in, The instantaneous reward at time t. For energy consumption coefficients, Let be the energy consumption at time t. Let be the time-of-use electricity price coefficient at time t. This is the coefficient for the temperature deviation term. Indoor temperature, To set the temperature, This is the coefficient for the overcooling penalty term; For indicator functions; For safety violation coefficients, The consequences of violating safety regulations; The coefficient for the action variation term. and Let be the control action vectors at time t and the previous time, respectively. It is the square of the L2 norm; This is the coefficient for the geothermal drift term. This represents the change in equivalent ground temperature between adjacent periods for buried pipes. Rewards for free time at night; As a non-punishment item, The coefficient for non-performing penalty items; Let be the energy consumption at time t. This is the preset energy consumption threshold.

[0079] Understandably, this application constructs a multi-dimensional, physically interpretable reward signal, realizing a structured trade-off between energy consumption and comfort. Each cost component corresponds to different optimization objectives in the operation of the ground source heat pump system: energy consumption corresponds to operational economy, temperature deviation corresponds to thermal comfort, overcooling penalty corresponds to ineffective cooling suppression, safety violation corresponds to equipment safety, action change corresponds to actuator lifespan, and ground temperature drift corresponds to long-term soil thermal balance. By using a weighted summation method, the multi-objective optimization problem is transformed into a single reward signal, enabling the reinforcement learning agent to optimize strategies under the guidance of a unified value function. All six key factors affecting the operational quality of the ground source heat pump system are incorporated into the reward framework. Each cost component has a clear physical meaning and its weight can be independently adjusted, allowing engineers to flexibly configure the optimization focus according to different operating stages and seasonal characteristics, achieving a leap from "single-objective optimality" to "multi-objective Pareto optimality."

[0080] By incorporating an overcooling penalty term into the reward function and penalizing overcooling through an indicator function with a dead zone, energy waste from ineffective cooling is eliminated. The penalty is activated only when the indoor temperature falls below the set temperature by more than 1°C (a dead zone). The physical significance and technical effect of this design are as follows: The introduction of a 1°C dead zone ensures that the system is not penalized for overcooling when the indoor temperature is slightly below the set temperature (within 1°C), allowing for natural buffering due to building thermal inertia and avoiding actuator wear and energy waste caused by frequent adjustments due to small temperature fluctuations. Once the indoor temperature falls below the set temperature by more than 1°C, the indicator function is activated, and the penalty term grows rapidly as the degree of overcooling increases in a quadratic function form. This allows the policy network to quickly learn to avoid significant overcooling during training, fundamentally eliminating energy waste caused by overcooling, especially significantly suppressing ineffective cooling behavior during transitional seasons and partial load conditions.

[0081] By implementing a penalty mechanism for inaction, the "slacking off" behavior of the policy network was completely suppressed. This was achieved by setting a conditional penalty for inaction, which is applied when the inaction occurs (when...). and (Time-based) Dynamically added penalty items for inaction. These penalty items are proportional to the square of the temperature deviation; that is, the higher the indoor temperature and the further it deviates from the set temperature, the heavier the penalty for inaction, with the penalty increasing quadratically with temperature deterioration. This mechanism fundamentally blocks the speculative path of the strategy network to "save energy" through "inaction," ensuring that the strategy network always fulfills its basic obligation to maintain the indoor thermal environment while pursuing energy conservation. This is a key mechanism design to guarantee the dual objectives of "energy saving and comfort" of the control method in this application.

[0082] The conditional penalty using indicator functions enables precise localization and directional optimization of constraint violations. Indicator functions are employed in the overcooling penalty. This implements conditional triggering, where the penalty is always zero if the condition is not met, and does not affect the policy network with any gradient.

[0083] When the indoor temperature is higher than the set temperature ( Temperature deviation item When the indoor temperature is […], the driving strategy network increases cooling to lower the temperature; , Within the comfort zone, both the temperature deviation term and the overcooling penalty term are small or zero, giving the policy network ample freedom to optimize freely according to the principle of minimizing energy consumption within this "comfort dead zone"; when the indoor temperature is below... At this time, the overcooling penalty term increases rapidly, driving the policy network to reduce cooling to restore temperature. The optimization objectives under the three operating conditions are clearly distinguished and do not interfere with each other, enabling the policy network to focus on different optimization sub-objectives in different temperature ranges. This achieves regionalized and precise optimization of control behavior, significantly improving the training efficiency and final control performance of the policy network.

[0084] The system's basic operating state is maintained through nighttime idle rewards, which are set as nighttime idle rewards in the reward function. This constant reward provides positive incentives to the policy network during periods of low system load or no operation at night, keeping it in a basic operating state rather than causing a complete shutdown, thereby avoiding equipment wear and tear caused by frequent start-ups and shutdowns and response delays when starting up the next day.

[0085] In some implementations, the coefficient of performance (COP) of the ground source heat pump system is estimated by the difference between the equivalent ground temperature of the buried pipe and the indoor temperature. The total energy consumption of the ground source heat pump system is calculated based on the cooling capacity, COP, and circulating water pump power, and ground temperature drift is incorporated as a long-term constraint as a reward. The formula for calculating the energy efficiency coefficient is: The formula for calculating the total energy consumption is: The ratio of the power to the flow rate of the circulating water pump satisfies the calculation formula. The formula for calculating the geothermal drift penalty is: ,in ; Wherein, COP is the coefficient of performance. For the equivalent geothermal temperature of the buried pipe, Let f(·) be the indoor temperature, and f(·) be the empirical function that maps the equivalent ground temperature of the buried pipe to the indoor temperature as COP. Let be the total energy consumption at time t. Let be the cooling capacity at time t. To control the cycle duration, For the power of the circulating water pump, This refers to the proportion of traffic. This is a penalty for geothermal drift. This is the coefficient for the geothermal drift term. The difference between the equivalent ground temperature of the buried pipe at time t and the previous time is given.

[0086] Understandably, this application estimates the coefficient of performance (COP) based on the difference between the equivalent ground temperature of the buried pipe and the indoor temperature, constructing a physical mechanism-driven energy efficiency characterization. The COP of the ground source heat pump system is constructed as a function of the difference between the equivalent ground temperature of the buried pipe and the indoor temperature. This temperature difference-driven COP estimation incorporates both the ground source temperature (reflecting the long-term operating status of the buried pipe heat exchanger and the daily heat dissipation / extraction conditions) and the indoor temperature (reflecting the terminal load demand and water supply temperature setting), resulting in stronger physical rationality and adaptability to operating conditions. This model can estimate the COP in real time using existing state vector information without additional sensors, significantly improving the accuracy and dynamic response capability of energy efficiency estimation while maintaining computational lightweightness.

[0087] The total energy consumption calculation formula uses the reciprocal of COP to construct a total energy consumption model, accurately depicting the energy efficiency degradation under partial load conditions. When the building is operating under partial load ( (Smaller), if the COP is at a low level due to excessive temperature difference, the unit's energy consumption item / COP does not decrease proportionally (low COP means high power consumption per unit of cooling capacity), thus imposing a reasonable energy consumption penalty on low-load, low-efficiency operating conditions in the reward function, effectively suppressing the low-efficiency operating mode of the strategy network in partial load conditions of "turning on large machines to meet small loads".

[0088] By modeling pump power using the cubic relationship of flow rate, the nonlinear energy consumption characteristics of pump operation with variable speed are accurately captured. The formula that the ratio of circulating pump power to flow rate satisfies is that pump power is proportional to the cube of flow rate. The cubic model accurately describes the actual energy consumption characteristics of the variable speed pump, making the energy consumption term in the reward function more realistically reflect the physical energy consumption of the system. This ensures that the "energy-saving strategy" obtained by the strategy network optimization is highly consistent with the energy-saving potential of the actual system, avoiding strategy deviation caused by distortion of the energy consumption model.

[0089] The calculation formula for the geothermal drift penalty term, which is the absolute value of the first-order difference between the equivalent geothermal temperature of the buried pipe at the current moment and the previous moment, is used to construct the geothermal drift penalty in the form of a first-order difference, realizing online real-time constraints on soil thermal balance. The physical significance of this first-order difference penalty is that rapid changes in geothermal temperature often correspond to drastic adjustments in system operation strategies or sudden switching of cooling / heating modes (such as switching from intermittent operation to continuous full-load operation). Such rapid fluctuations in geothermal temperature are not only detrimental to the efficient and stable operation of the system, but may also reflect short-term aggressive behavior in the control commands output by the strategy network. By immediately penalizing the rate of geothermal change in the reward function, the strategy network can perceive the impact of the current action on the geothermal trend at each decision step, thereby proactively avoiding control strategies that lead to rapid deterioration of geothermal temperature.

[0090] In some implementations, the training phase of the policy network employs a building thermodynamics simulation environment driven by real or typical meteorological year data; the training process alternates between preset challenge scenarios, which include at least one of sensor noise, model parameter mismatch, and sudden internal load disturbances.

[0091] Understandably, simulation training driven by real / typical meteorological year data achieves full coverage of year-round operating conditions. The policy network can learn optimal control strategies for different seasons and climatic conditions during the training phase, eliminating the need for trial and error to adapt to unfamiliar conditions during actual operation, significantly reducing the risks and energy costs of the online exploration phase. The training samples cover the entire year's climate space, enabling the policy network to possess pre-learned coping capabilities when facing any climatic conditions after deployment, significantly improving the generalization performance and climate adaptability of the control strategy. Using typical meteorological year data ensures the repeatability and comparability of training results across different years, providing a scientific basis for standardized training and evaluation of the policy network.

[0092] The building thermodynamics simulation environment replaces the real system for offline training of policy networks, enabling reinforcement learning agents to conduct large-scale policy exploration and trial-and-error learning in the simulation environment, providing a zero-risk, high-efficiency policy exploration platform.

[0093] Training with sensor noise injection enhances the robustness of the policy network to measurement uncertainties. By continuously injecting noise matching the characteristics of real sensors during training, the policy network is forced to learn control strategies that can still make reasonable decisions under "imperfect observation" conditions. This training method gives the policy network a natural tolerance to sensor measurement errors, environmental interference, and signal fluctuations after deployment. The control actions do not change drastically due to small random fluctuations in sensor readings, significantly improving the stability and reliability of the control strategy in real-world engineering environments.

[0094] By training with model parameter mismatch, the adaptability of the policy network to changes in system characteristics is enhanced. After deployment, even if there are certain deviations between the thermal characteristics of the real system and the training simulator, or if the system performance gradually degrades over time, the policy network can still output reasonable and effective control commands based on the current observation state, without the need for retraining due to changes in system parameters. This feature significantly extends the effective lifespan of the control policy and reduces the frequency and cost of system maintenance and policy updates.

[0095] Through training with sudden internal load disturbances, the policy network's ability to respond quickly to abnormal operating conditions was enhanced. After deployment, when faced with various sudden load changes occurring in real buildings, the policy network can identify anomalies within several control cycles and automatically adjust control variables such as water supply temperature and circulation flow rate, quickly bringing the indoor temperature back to the comfortable range. This significantly shortens the response delay time required by traditional control methods due to a lack of prior knowledge, improving the system's robustness in dealing with emergencies and enhancing user comfort.

[0096] This application alternates between three challenging scenarios—sensor noise, model parameter mismatch, and sudden internal load disturbances—during training, rather than completing all training in a single fixed environment. This multi-scenario alternating training strategy achieves a balance between robustness and optimization. Through the combined effect of the above multi-scenario robust training strategy, this application fundamentally narrows the "simulation-reality gap" between the simulation training environment and the real deployment environment.

[0097] In some implementations, the constraint-aware entropy scheduling, physical information commentator, and Lagrange soft constraint mechanism are also applicable to any of the following HVAC systems: air-source heat pump systems, water-source heat pump systems, and electric chiller-cooled tower systems. For the other HVAC systems, the equivalent ground temperature of the buried pipe is... Circulating water pump power The ground temperature drift penalty term is replaced as follows: In an air source heat pump system, the equivalent ground temperature of the buried pipe is replaced with the equivalent air temperature on the condenser side, the circulating water pump power is replaced with the compressor and fan power, and the weight of the ground temperature drift penalty term is reset to zero; in a water source heat pump system, the equivalent ground temperature of the buried pipe is replaced with the equivalent water temperature on the source side; in an electric chiller unit with a cooling tower system, the equivalent ground temperature of the buried pipe is replaced with the cooling tower outlet water temperature, and the weight of the ground temperature drift penalty term is reset to zero.

[0098] Through the above technical solution, this application provides a physical information security reinforcement learning control method for ground source heat pump systems. It constructs an operating state vector by collecting indoor temperature, outdoor temperature, equivalent ground temperature of buried pipes, indoor humidity, personnel load ratio, and time period coding. This operating state vector is then input into a policy network based on maximum entropy reinforcement learning, and outputs control commands for water supply temperature, circulation flow rate, and humidity adjustment via physical mapping. During training, the effective entropy coefficient is dynamically adjusted based on the safety boundary distance and set temperature deviation through constrained perceptual entropy scheduling. The physical residuals of cooling demand and cooling effort are embedded into the loss of the dual-commentator network through a physical information commentator, constraining the thermodynamic consistency of the value function. Energy consumption, comfort, and safety are dynamically balanced using comfort and safety Lagrange multipliers. Without relying on an exact online optimization model, the controller can proactively adjust the exploration intensity based on the system's safety status during operation, and achieves coordinated optimization of energy consumption, comfort, and safety through physical information commentators and Lagrange soft constraints. It can reduce the operating energy consumption of ground source heat pump systems, improve the quality of comfort control, and maintain the safe operation of equipment under conditions of sensor noise, model parameter mismatch and sudden load disturbance. It can be applied to intelligent HVAC control of large public buildings (including but not limited to high-speed railway stations, airport terminals, commercial complexes, hospitals and office buildings).

[0099] See Figure 2 As shown, an embodiment of this application provides a physical information security reinforcement learning control device, which can be used to implement the control method described in the above embodiments and implementation methods. The control device includes: a data acquisition module 10, a state construction module 20, a decision-making module 30, a reward calculation module 40, a constraint update module 50, and a control output module 60.

[0100] The data acquisition module 10 is used to collect operational data of the ground source heat pump system and its service area. The data acquisition module 10 can be used to perform the "collecting operational data of the ground source heat pump system and its service area" step in step S1 of the above control method.

[0101] In each control cycle The data acquisition module 10 synchronously collects operational data of the ground source heat pump system and its service area through sensors, building management system (BMS), and meteorological interface, including at least: average indoor temperature. Outdoor temperature equivalent geothermal temperature of buried pipes Indoor relative humidity Personnel workload ratio and the current hour In this embodiment of the application, the control period It is 1 hour. For example, suppose the operating data collected in the current control cycle is: , , , , , .

[0102] The state construction module 20 is used to construct an operating state vector based on the operating data; the operating state vector includes indoor average temperature, outdoor temperature, equivalent ground temperature of buried pipes, indoor humidity, personnel load ratio, and hourly code. The state construction module 20 can be used to execute the "construct operating state vector" step in step S1 of the above control method.

[0103] The state construction module 20 constructs an operating state vector based on the operating data. During the online deployment phase, the running state vector is a seven-dimensional core state vector: ;in and Used to characterize the cyclical operating pattern within a day. During the offline training phase, the operating state vector further includes at least one set of the following extended dimensions: (i) the deviation of the temperature in each area from the average temperature of the entire building. (4 dimensions in total); (ii) Outdoor temperature forecast for the next 1 hour, 3 hours, and 6 hours. (3 dimensions in total); (iii) Time-of-use electricity pricing segment The number of hours between peak and valley transitions (2 dimensions in total). This extended dimension is used only during the training phase to improve the policy's generalization ability to complex operating conditions. During online deployment, inference is still performed using the seven-dimensional core state vector, thereby reducing reliance on the station building engineering data acquisition system. The deviation of the temperature in the four zones from the average temperature of the entire building; For the outdoor temperature prediction in the next Δh hours, Δh is taken as 1, 3, and 6 hours respectively; This represents the time-of-use electricity price segment at time t. For example, the state construction module 20 receives the operational data collected by the data acquisition module 10 and constructs a seven-dimensional core state vector as follows: .

[0104] The decision module 30 includes a policy network based on maximum entropy reinforcement learning and a dual-critic network. The decision module 30 is used for online inference of control actions. It inputs the running state vector into the policy network and obtains normalized control actions through reparameterization sampling. The control actions are then physically mapped and issued as actual control commands for the ground source heat pump system. The control commands include water supply temperature, circulation flow rate, and humidity adjustment. The decision module 30 can be used to execute step S2 of the above control method. Optionally, the decision module 30 can be a PI-SAC (Predictive Information Soft Actor-Critic) decision module. PI-SAC is a representation learning method that combines predictive information with maximum entropy reinforcement learning (Soft Actor-Critic, SAC), aiming to improve the sample efficiency and policy generalization ability of the agent in complex environments.

[0105] Decision module 30 includes a Gaussian policy network. With DoubleQ Critics Network The policy network uses a multilayer perceptron as its backbone and outputs the average action value. and logarithmic standard deviation And through reparameterized sampling ( Normalized continuous control actions are obtained. ;in Corresponding water supply temperature, Corresponding circulation flow Adjust humidity accordingly.

[0106] The dual-Q critic network outputs state-action value respectively. and It also has a network of targeted critics. , According to Polyak average ( Soft update is used to calculate the stable temporal difference objective. Here, Q_φ1 and Q_φ2 are online double-Q commentator networks, with φ1 and φ2 being their network parameters; Q_bar_φ1 and Q_bar_φ2 are the corresponding objective commentator networks, with network parameters bar_φ1 and bar_φ2 respectively; in the Polyak average soft update formula bar_φ ← ρ · bar_φ + (1 - ρ) · φ, ρ is the soft update coefficient (range [0.99, 0.999]).

[0107] The objective value of the critic network is taken as the minimum of the two Q-squares and the sum of the policy entropy: ,in As a discount factor, This is the effective entropy coefficient output by module 4.

[0108] For example, the decision module 30 receives the seven-dimensional core state vector constructed by the state construction module 20. , will the The input policy network is obtained , (correspond ); Sampling noise At that time, normalized action .

[0109] The reward calculation module 40 is used to construct rewards and constraint costs based on energy consumption, comfort constraints, safety constraints, motion changes, and ground temperature drift. The reward calculation module 40 can be used to execute step S3 of the above control method.

[0110] The reward calculation module 40 constructs a reward based on the running data and control actions. With constraint cost ,in Used for training commentator networks and policy networks. Multiplier update for module 6.

[0111] In this embodiment, the reward It consists of the following sub-items: Energy consumption item Temperature deviation item Overly cold penalty item Humidity violation Safety violations Action variation items Geothermal drift ,in Not as a penalty item Nighttime free time reward The total reward is the sum of all the individual rewards.

[0112] In this embodiment, the energy efficiency coefficient of the ground source heat pump system The energy consumption is estimated based on the difference between the equivalent ground temperature of the buried pipe and the indoor temperature. The power of the circulating water pump .

[0113] For example, the reward calculation module 40 receives data output from the data acquisition module 10, the state construction module 20, and the decision-making module 30, assuming the current period... , (flat electricity price) , , Not valid but And there were no safety violations or humidity exceeding limits; take , , , ,but , , , Other sub-items are zero, total reward .

[0114] The constraint update module 50 is used to collaboratively update the training mechanism of the policy network and the dual critic network using a preset mechanism, which includes: constraint-aware entropy scheduling, physical information critics, and Lagrange soft constraints. The constraint update module 50 can be used to execute step S4 of the above control method.

[0115] The control output module 60 is used to deploy the trained strategy network to the upper controller or building control system of the ground source heat pump system, so as to output control commands in real time on a periodic basis. That is, it is used to send the control actions to the execution end of the ground source heat pump system through physical mapping. The control output module 60 can be used to execute step S5 of the above control method.

[0116] The control output module 60 normalizes the action in each control cycle. The actual control commands are obtained through physical mapping. The data is then distributed to heat pump units, circulating water pumps, and humidification / dehumidification actuators; system feedback is then collected, and experience samples are compiled. Stored in the experience replay pool, entering the next cycle. Let be the actual control command vector issued to the execution end at time t. For water supply temperature, For the circulating water pump flow ratio, Humidity adjustment command; experience sample Stored in the experience replay pool for subsequent training.

[0117] The physical mapping satisfies: ; ; ; under refrigeration conditions , .

[0118] For example, the control output module 60 receives the data output by the above modules, and the... ,but , , The data is then distributed after linear mapping according to a preset range.

[0119] Through the above technical solution, this application enables the controller to actively adjust the exploration intensity according to the system safety status during operation without relying on a precise online optimization model. It also achieves synergistic optimization of energy consumption, comfort, and safety through physical information commentators and Lagrange soft constraints. This can reduce the operating energy consumption of ground source heat pump systems and improve the quality of comfort control. It can be applied to intelligent HVAC control in large public buildings (including but not limited to high-speed railway stations, airport terminals, commercial complexes, hospitals, and office buildings).

[0120] See Figure 2 and Figure 3 As shown, in some optional embodiments, the constraint update module 50 includes: a constraint-aware entropy scheduling module 51, a physical information critic training module 52, and a Lagrange soft constraint training module 53.

[0121] The constraint-aware entropy scheduling module 51 is used to calculate and output the effective entropy coefficient to the decision module. The constraint-aware entropy scheduling module 51, as follows... Figure 4 As shown, the effective entropy coefficient is dynamically calculated according to the following sub-steps. : S201: Based on indoor temperature Set temperature and safe temperature boundary As input; in this embodiment of the application, under refrigeration conditions, the following is taken: , .

[0122] S202: Adjust the indoor temperature according to... Perform an exponential moving average, where In this embodiment of the application, the following is taken: .

[0123] S203: Calculate smoothing temperature With the safe temperature boundary distance .

[0124] S204: Calculation of subcooling deviation under refrigeration conditions ; Calculation of superheat deviation under heating conditions .

[0125] S205: Based on the distance With preset safety margin ,according to Calculate the safety scaling factor; where For the Sigmoid function, ;when (Approaching the safety boundary) .

[0126] S206: Based on the aforementioned deviation ,according to Calculate the comfort scaling factor; where , For comfort benchmark; when When it is significantly too cold or too hot .

[0127] S207: Take the smaller value and crop: In this embodiment of the application, the following is taken: , .

[0128] S208: Basic Entropy Coefficient Updated according to the standard SAC automatic entropy adjustment method, the final effective entropy coefficient is obtained. , which serves as the policy entropy weight in maximum entropy reinforcement learning.

[0129] In this embodiment of the application, the nonlinear mapping of the constraint-aware entropy scheduling can also be implemented by a piecewise monotonic function, so that it is in When the threshold is exceeded, it transitions continuously to 0 or 1.

[0130] For example, the constraint-aware entropy scheduling module 51 receives data output from the data acquisition module 10, the state construction module 20, the decision-making module 30, and the reward calculation module 40, assuming the previous cycle... , ,but ; ; under refrigeration conditions (Not too cold); Take , ;Pick , ; ; Assumption ,but When the indoor temperature gradually rises and approaches the safety boundary (e.g.) When it reaches 28.5℃, Reduced to 0.5 ℃ , The value is further reduced to approximately 0.044, making the policy output more deterministic and thus avoiding dangerous actions caused by maximum entropy random exploration.

[0131] The Physical Information Critic Training Module 52 is used to calculate the physical residual between cooling demand and cooling effort, and participates in the calculation of the critic network loss.

[0132] Physical Information Critic Training Module 52 Figure 5 The upper branch (steps S301 to S305) is shown.

[0133] S301: Sample small batches of experience samples from the experience replay pool. In this embodiment of the application, the batch size is taken as... In the empirical sample (s, a, r, c, s′), s is the current state, a is the current action, r is the reward, c is the constraint cost, and s′ is the state at the next moment; |B| is the number of samples in a single batch.

[0134] S302: Calculate cooling requirements based on the sample ;when season This leads critics to lower their valuation of cooling actions in the "no cooling required" state.

[0135] S303: Based on normalized motion components Calculate the cooling effort, where , .

[0136] S304: Press Calculate the physical residuals.

[0137] S305: Total Loss of Constructing a Two-Critic Network ,in For time-series difference loss, the target Same as decision module 30, The weight is the physical residual; in the embodiments of this application, it is taken as... Based on the above Commentator network parameters , Perform gradient descent updates.

[0138] In this embodiment, the physical residual can also be replaced by the energy balance residual calculated based on the heat exchange of the building envelope, internal heat load, cooling capacity on the water supply side, and coefficient of performance (COP). As a physical consistency constraint in the commentator's loss, it achieves the same technical effect.

[0139] For example, the physical information critic training module 52 undertakes the normalization action. With the , ,Pick ,but ; , , ; ; Assumption , ,but .

[0140] The Lagrange soft constraint training module 53 is used to update the comfort constraint multiplier and the safety constraint multiplier according to the Lagrange soft constraint update rule.

[0141] Lagrange soft constraint training module 53, as shown Figure 5 The lower branch (steps S306 to S309) is shown.

[0142] S306: Extracting the Cost of Comfort Violations from Samples Costs of safety violations ;in This represents the cumulative amount of temperature deviation from the set value. This refers to the cumulative amount that exceeds the safety boundary.

[0143] S307: Update the Lagrange multipliers according to the following rules: ; ;in , The learning rate of the multiplier. , To constrain the threshold, , The upper limit of the multiplier is taken as the multiplier; in the embodiments of this application, it is taken as... , , , .

[0144] S308: Loss of constructing a policy network ;in Output from module 4 The updated commenter network output from Module 5.

[0145] S309: Adjust the policy network parameters based on the policy loss. Perform gradient descent updates.

[0146] For example, taking into account the aforementioned operating state, let's assume the comfort violation cost for the current batch. The consequences of safety violations And currently , ,but ; When subsequent batches of comfort violations persist, As comfort violations continue to increase, their corresponding weight in the policy loss increases, prompting the policy to update in a direction that reduces comfort violations; when comfort violations fall back to... After that, Automatic decay avoids policy distortion caused by the multiplier increasing indefinitely.

[0147] The training mechanism is jointly implemented by the constraint-aware entropy scheduling module 51, the physical information critic training module 52, and the Lagrange soft constraint training module 53: the constraint-aware entropy scheduling module 51 outputs the effective entropy coefficient. Used for policy entropy regularization; Physical Information Commentator Training Module 52 calculates physical residuals. And construct the total loss of the dual-commentator network. The Lagrange soft-constraint training module 53 updates the Lagrange multipliers using gradient ascent and constructs the policy loss. After each gradient update step, a soft update is performed on the target commentator network using Polyak averaging.

[0148] It should be noted that the control device must include the data acquisition module, state construction module, decision-making module (inference only), and control output module during the online deployment phase; the constraint perception entropy scheduling module, physical information commentator training module, and constraint update module are required during the offline training phase, but optional during the online deployment phase.

[0149] The strategy network of the control device adopts a building thermodynamic simulation environment driven by real or typical meteorological year (TMY / EPW) data during the training phase, and alternates between at least one challenging scenario among sensor noise, model parameter mismatch and sudden internal load disturbance. After training, the strategy network is deployed to the upper controller or building control system of the ground source heat pump system, and outputs control commands in real time according to the cycle, thereby realizing the safe, energy-saving and comfortable control of the ground source heat pump system.

[0150] Furthermore, the constraint-aware entropy scheduling, physical information commentator and Lagrange soft constraint mechanism, and the overall framework of the operating state vector, action space and reward function described in the embodiments of this application can also be extended to other HVAC systems such as air source heat pump systems, water source heat pump systems, and electric chiller units with cooling tower systems. Those skilled in the art can achieve this extension by referring to the following equivalent substitutions without creative effort: In air source heat pump systems, the equivalent ground temperature of the buried pipe is replaced with the equivalent air temperature on the condenser side, the circulating water pump power is replaced with the compressor and fan power, and the weight of the ground temperature drift penalty term is reset to zero; in water source heat pump systems, the equivalent ground temperature of the buried pipe is replaced with the equivalent water temperature of the source-side water body; in electric chiller units with cooling tower systems, the equivalent ground temperature of the buried pipe is replaced with the cooling tower outlet water temperature, and the ground temperature drift penalty term is treated as an optional feature. The equivalent substitution implementation does not affect the core technical solution claimed in this application.

[0151] Compared with the prior art, this application has the following beneficial effects: (1) This application introduces constraint-aware entropy scheduling on the policy entropy weights of maximum entropy reinforcement learning, so that when the system approaches the safety boundary or shows a trend of being too cold or too hot, the policy entropy weights actively decay and the policy output tends to be deterministic, thereby suppressing the device limit risk caused by the inherent random exploration of maximum entropy reinforcement learning; compared with existing methods that rely solely on action pruning or ex-post penalties, this application reduces the probability of unsafe actions from the source. (2) This application introduces physical information loss in the value evaluation stage of the critic network and directly embeds the physical residuals of cooling demand and cooling effort into the value function learning, so that the critic network gives a lower value estimate for state-action pairs that do not conform to thermodynamic relations. Compared with existing methods that only project at the action end or only use fixed physical regularization, this application constructs thermodynamic consistency constraints at the value function level, which improves the stability and sample efficiency of policy learning. (3) This application introduces two independent Lagrange multipliers for comfort and safety, and updates them online according to the discrete gradient ascent rule to achieve a dynamic trade-off between comfort constraints and safety constraints, thereby eliminating the dependence of existing methods on manual fixed weights; when the operating conditions change, the seasons change or the electricity price changes, the multipliers can be adaptively adjusted without manual readjustment. (4) Based on the physical characteristics of the ground source heat pump system, this application introduces the equivalent ground temperature of the buried pipe into the state vector, introduces the cubic relationship between the energy efficiency coefficient, the power of the circulating water pump and the flow rate into the energy consumption calculation, and adds ground temperature drift as a long-term constraint as a reward, so that the method is more suitable for the long-term safe and energy-saving operation of the ground source heat pump system. (5) The training mechanism of this application embeds constraint mechanisms at three key decision points—strategy exploration, value assessment, and target weight—through three mechanisms: constraint perception entropy scheduling, physical information commentator, and Lagrange soft constraint. This forms a closed-loop control framework with three layers of linkage: "safe exploration—thermodynamic consistency—multi-objective balance," which improves the robustness of the method under actual engineering conditions such as sensor noise, model parameter mismatch, and sudden internal load disturbances. (6) The device of this application can be deployed in software form based on building control system or host computer. During the offline training stage, policy training is completed on server or industrial PC. During the online deployment stage, only forward inference of policy network needs to be run. It has low requirements for on-site hardware resources and is easy to implement in engineering.

[0152] Embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the control method described in the above embodiments and implementation methods.

[0153] Embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method described in the above embodiments and implementation methods.

[0154] It should be noted that, in the specific implementation process, the above methods can be partially implemented by a hardware processor executing computer-executable instructions in software form stored in memory, which will not be elaborated here; the programs corresponding to the executed actions can all be stored in software form in the computer-readable storage medium of the device so that the processor can call and execute the operations corresponding to the above modules.

[0155] The computer-readable storage media mentioned above may include volatile memory, such as random access memory; may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; and may also include combinations of the above types of memory.

[0156] The processor mentioned above can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.; a general-purpose processor can be a microprocessor or any conventional processor.

[0157] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A physical information security reinforcement learning control method, characterized in that, include: Collect operational data of the ground source heat pump system and its service area to construct an operational status vector; the operational status vector includes indoor average temperature, outdoor temperature, equivalent ground temperature of buried pipes, indoor humidity, personnel load ratio, and hourly code; The operating state vector is input into a policy network based on maximum entropy reinforcement learning, and normalized control actions are obtained through reparameterized sampling. The control actions are then physically mapped and issued as actual control commands for the ground source heat pump system. The control commands include water supply temperature, circulation flow rate, and humidity regulation. Based on energy consumption, comfort constraints, safety constraints, motion changes, and ground temperature drift, a reward and constraint cost are constructed. The training mechanism of the policy network and the dual critic network is updated collaboratively using a preset mechanism, which includes: constraint-aware entropy scheduling, physical information critic, and Lagrange soft constraints. The trained strategy network is deployed to the supervisory controller or building control system of the ground source heat pump system to output control commands in real time on a periodic basis.

2. The control method according to claim 1, characterized in that, The hourly encoding includes hourly sine encoding and hourly cosine encoding; during the online deployment phase, the operating state vector is a seven-dimensional core state vector, including indoor average temperature, outdoor temperature, equivalent ground temperature of buried pipes, indoor humidity, personnel load ratio, hourly sine encoding, and hourly cosine encoding. During the offline training phase, the running state vector includes not only the seven-dimensional core state vector, but also at least one of the following extended dimensions: temperature difference in each region, outdoor temperature forecast for the next few hours, time-of-use electricity price segment, and the number of hours between peak and off-peak periods.

3. The control method according to claim 1, characterized in that, The physical mapping satisfies the following relationship: The calculation of the control command for the water supply temperature is as follows: ; The calculation of the flow control command is for ; The calculation of the control command for the adjustment is as follows: ; in, This is a water supply temperature control command. This is the lower limit of the water supply temperature; under cooling conditions, it is 5°C. This is the upper limit of the water supply temperature; under cooling conditions, it is 18°C. The normalized control action component for water supply temperature, with a value range of [-1, 1]; This is a flow ratio control command for the circulating water pump, with a value range of [0,1]. The normalized control action component for the circulating flow rate, with a value range of [-1, 1]; This is a humidity control command. This is the preset lower limit for humidity adjustment. This is the preset upper limit for humidity adjustment. This is the normalized control action component for humidity regulation, with a value range of [-1, 1].

4. The control method according to claim 1, characterized in that, The training mechanism employs a Gaussian policy network. With the dual critic network The policy network outputs the action mean vector μ and the log-standard deviation. And use calculation formula , Continuous control actions are sampled using reparameterization techniques; the target value of the dual critic network is adopted. and The method of taking the minimum value and using the calculation formula. Superimposed strategy entropy; Where θ represents the policy network parameters. Let μ be the running state vector, μ be the action mean vector, and σ be the action standard deviation vector. For Gaussian policy networks, and For two independent Q-commenter networks, φ1 and φ2 are their network parameters; ε is the standard normal sampling noise. y is a standard multivariate normal distribution; y is the temporal difference target value of the dual critic network; r is the instantaneous reward; γ is the discount factor, with a value range of [0,1]. For the corresponding target commentator network (i=1,2), Polyak average soft update is used; s′ is the state at the next time step, and a′ is the action at the next time step. This is the effective entropy coefficient.

5. The control method according to claim 1, characterized in that, The training mechanism for constraint-aware entropy scheduling includes: performing an exponential moving average on the indoor temperature to obtain a smoothed temperature; calculating the distance between the smoothed temperature and the safe temperature boundary, as well as the undercooling or overheating deviation relative to the set temperature; obtaining a safe scaling factor and a comfort scaling factor through the Sigmoid function; taking the smaller value between the safe scaling factor and the comfort scaling factor as the entropy scaling factor and limiting it within a preset range; and obtaining the effective entropy coefficient by multiplying the base entropy coefficient by the entropy scaling factor.

6. The control method according to claim 5, characterized in that, The formula for calculating the Sigmoid function is: Both the safety scaling factor and the comfort scaling factor are implemented using piecewise monotonic functions. in, Let x be the sigmoid function, x be the independent variable, and e be the natural constant.

7. The control method according to claim 1, characterized in that, The training mechanism for the physical information commentator includes: the physical information commentator loss is constructed based on the degree of mismatch between cooling demand and cooling effort, and the physical residuals are weighted and superimposed on the temporal difference loss of the dual commentator network; and when the indoor temperature is lower than the set temperature, the cooling demand is set to 0, so that the dual commentator network reduces the value estimation of ineffective cooling actions.

8. The control method according to claim 7, characterized in that, The formula for calculating the cooling requirement is: The formula for calculating the cooling effort is: The formula for calculating the physical residual is as follows: The formula for calculating the total loss of the dual-critic network is as follows: ;in , , ; Where D represents cooling requirements. Indoor temperature, To set the temperature, k is the temperature normalization coefficient; E is the cooling effort. For the normalized action component of the water supply temperature, For the normalized motion components of the cyclic flow, The normalized control action component for the original water supply temperature, with a value range of [-1, 1]; The normalized control action component of the original circulating flow, with a value range of [-1, 1]; For physical residuals, The total loss for the dual-critic network, For time-series difference loss, For the commentator network, s is the state, a is the action, y is the temporal difference target value, and w is the physical residual weight.

9. The control method according to claim 7, characterized in that, The physical residual is the energy balance residual calculated based on the heat exchange of the building envelope, internal heat load, cooling capacity on the water supply side, and coefficient of performance (COP). The formula for calculating the physical residual is as follows: ; in, For energy balance residuals, The net heat gain on the building side is calculated from the heat exchange capacity of the building envelope and the internal heat load. The net cooling capacity on the air conditioning side is calculated from the cooling capacity on the water supply side and the coefficient of performance (COP). It is the square of the second norm.

10. The control method according to claim 1, characterized in that, The Lagrange soft constraint includes comfort constraint multipliers, safety constraint multipliers, and policy loss. The training mechanism for the Lagrange soft constraint includes updating the comfort constraint multipliers and safety constraint multipliers according to the Lagrange soft constraint update rule. The update rule for the comfort constraint multipliers is as follows: The formula for calculating the strategy loss is: ; in, For comfort-constrained multipliers, To constrain the multipliers, and These are the learning rates for the comfort-constrained multipliers and the safety-constrained multipliers, respectively. and These are the threshold values ​​for the comfort constraint multipliers and the safety constraint multipliers, respectively. The consequences of compromising one's rights for comfort are accumulated. Accumulated costs for safety violations; To constrain the upper limit of the multipliers for comfort, To ensure safety, an upper limit is imposed on the multipliers; For strategic losses, The effective entropy coefficient, For Gaussian policy networks, For the commentator network, s represents the state and a represents the action.

11. The control method according to claim 10, characterized in that, The Lagrange soft constraint includes setting upper limits for the comfort constraint multiplier and the safety constraint multiplier respectively, so as to avoid the comfort constraint multiplier and the safety constraint multiplier from increasing infinitely during the training process and causing policy distortion.

12. The control method according to claim 1, characterized in that, The rewards include at least two of the following: energy consumption, time-of-use electricity pricing, temperature deviation, overcooling penalty, humidity violation, safety violation, motion change penalty, and ground temperature drift penalty.

13. The control method according to claim 12, characterized in that, The rewards include energy consumption items, time-of-use electricity price items, temperature deviation items, overcooling penalty items, humidity violation items, safety violation items, action change penalty items, and ground temperature drift penalty items. The rewards are obtained by summing up all of the above items. When the indoor temperature is higher than the set temperature and the energy consumption of the ground source heat pump system is lower than the preset energy consumption threshold, the rewards also include items that are not considered as penalties. The formula for calculating the reward is as follows: The formula for calculating the items not considered as penalties is as follows: ; in, The instantaneous reward at time t. For energy consumption coefficients, Let be the energy consumption at time t. Let be the time-of-use electricity price coefficient at time t. This is the coefficient for the temperature deviation term. Indoor temperature, To set the temperature, This is the coefficient for the overcooling penalty term; For indicator functions; For safety violation coefficients, The consequences of violating safety regulations; The coefficient for the action variation term. and Let be the control action vectors at time t and the previous time, respectively. It is the square of the L2 norm; This is the coefficient for the geothermal drift term. This represents the change in equivalent ground temperature between adjacent periods for buried pipes. Rewards for free time at night; As a penalty for non-compliance, This is the coefficient for non-penalty items.

14. The control method according to claim 1, characterized in that, The energy efficiency coefficient of the ground source heat pump system is estimated by the difference between the equivalent ground temperature of the buried pipe and the indoor temperature. The total energy consumption of the ground source heat pump system is calculated based on the cooling capacity, energy efficiency coefficient and circulating water pump power, and ground temperature drift is added as a long-term constraint as a reward. The formula for calculating the energy efficiency coefficient is: The formula for calculating the total energy consumption is: The ratio of the power to the flow rate of the circulating water pump satisfies the calculation formula. The formula for calculating the geothermal drift penalty is: ,in ; Wherein, COP is the coefficient of performance. For the equivalent geothermal temperature of the buried pipe, Let f(·) be the indoor temperature, and f(·) be the empirical function that maps the equivalent ground temperature of the buried pipe to the indoor temperature as COP. Let be the total energy consumption at time t. Let be the cooling capacity at time t. To control the cycle duration, For the power of the circulating water pump, This refers to the proportion of traffic. This is a penalty for geothermal drift. This is the coefficient for the geothermal drift term. The difference between the equivalent ground temperature of the buried pipe at time t and the previous time is given.

15. The control method according to claim 1, characterized in that, The training phase of the strategy network adopts a building thermodynamics simulation environment driven by real or typical meteorological year data; the training process is carried out alternately under preset challenge scenarios, which include at least one of sensor noise, model parameter mismatch and sudden internal load disturbance.

16. The control method according to claim 1, characterized in that, The aforementioned constraint-aware entropy scheduling, physical information commentator, and Lagrange soft constraint mechanism are also applicable to any type of HVAC system, including air source heat pump systems, water source heat pump systems, and electric chiller units with cooling tower systems.

17. A physical information security reinforcement learning control device, characterized in that, include: The data acquisition module is used to collect operational data of the ground source heat pump system and its service area; The state construction module is used to construct an operating state vector based on the operating data; the operating state vector includes indoor average temperature, outdoor temperature, equivalent ground temperature of buried pipes, indoor humidity, personnel load ratio, and hourly code. The decision module includes a policy network based on maximum entropy reinforcement learning and a dual critic network. The decision module is used to input the running state vector into the policy network, obtain normalized control actions through reparameterized sampling, and issue the control actions as actual control commands for the ground source heat pump system after physical mapping. The control commands include water supply temperature, circulation flow rate, and humidity adjustment. The reward calculation module is used to construct rewards and constraint costs based on energy consumption, comfort constraints, safety constraints, motion changes, and ground temperature drift. The constraint update module is used to collaboratively update the training mechanism of the policy network and the dual critic network using a preset mechanism, which includes: constraint-aware entropy scheduling, physical information critic, and Lagrange soft constraints. The control output module is used to deploy the trained strategy network to the upper controller of the ground source heat pump system or the building control system to output control commands in real time on a periodic basis.

18. The control device according to claim 17, characterized in that, The constraint update module includes: A constraint-aware entropy scheduling module is used to calculate and output the effective entropy coefficient to the decision module. The Physical Information Critic Training Module is used to calculate the physical residual between cooling demand and cooling effort, and participates in the calculation of the critic network loss. The Lagrange soft constraint training module is used to update the comfort constraint multiplier and the safety constraint multiplier according to the Lagrange soft constraint update rule.

19. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the control method according to any one of claims 1-16.

20. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the control method according to any one of claims 1-16.

Images