Data center cloud edge collaborative energy management method and system based on reinforcement learning

Abstract

The application discloses a data center cloud edge collaborative energy management method and system based on reinforcement learning, relates to the technical field of data center energy management and cloud computing, and performs the following on the cloud side: constructing a system model based on system historical operation data, receiving real-time operation states fed back by the edge side, and dynamically updating parameters of the system model according to the real-time operation states; in addition, new energy power generation power and load demand power are predicted, prediction data are generated, and the updated system model and the prediction data are sent to the edge side; the edge side performs the following: receiving the system model and the prediction data, adopting model predictive control, performing rolling optimization under the condition of meeting system constraints, taking minimization of comprehensive operation cost as a target, solving a control sequence and performing; and feeding back the real-time operation states to the cloud side, so that the cloud side corrects and updates the parameters of the system model, and realizes closed-loop self-updating of the system model. The application solves the problem of invalidation of an energy management strategy caused by mismatch of a system model.

Description

Technical Field

[0001] This invention relates to the fields of data center energy management and cloud computing technology, and in particular to a data center cloud-edge collaborative energy management method and system based on reinforcement learning. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of the digital economy, data centers serve as the core infrastructure supporting cloud computing and edge computing. Under the cloud-edge collaborative architecture, data centers not only need to handle massive computing tasks, but also face the severe challenge of how to provide energy efficiently and economically.

[0004] Data center integrated power supply systems are inherently complex systems with high time-varying characteristics. Traditional strategies struggle to balance system model accuracy and global optimization effects under limited computing power. In real-world dynamic operating environments, system energy management faces multiple challenges, including the availability of renewable energy and the volatility of computing tasks.

[0005] In existing research on energy management strategies for data centers, reinforcement learning is combined with model predictive control. However, reinforcement learning is only used to learn and update the weight parameters of the objective function during the optimization process. It does not solve the problem of system model mismatch caused by changes in system operating conditions or sudden changes in the parameters of the system model itself, and its system model has poor adaptability. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a data center cloud-edge collaborative energy management method and system based on reinforcement learning, aiming to solve the problem of energy management strategy failure caused by system model mismatch.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention provides a data center cloud-edge collaborative energy management method based on reinforcement learning, comprising: Cloud-side execution: Based on historical system operation data, a system model is constructed using reinforcement learning, and the real-time operation status feedback from the edge side is received. The parameters of the system model are dynamically updated according to the real-time operation status. It also predicts the power generation of new energy sources and the power demand of load based on the historical operation data of the system, generates prediction data, and sends the updated system model and the prediction data to the edge side; Edge-side execution: Receive system models and prediction data from the cloud, adopt model predictive control, and perform rolling optimization with the goal of minimizing overall operating cost while satisfying system constraints, solve for the control sequence and execute it; The real-time operating status is fed back to the cloud side, so that the cloud side can correct and update the parameters of the system model, thereby realizing the closed-loop self-updating of the system model.

[0008] A further technical solution involves constructing a system model based on historical system operation data and adjusting the model parameters using reinforcement learning. Specifically, this includes modeling the adaptive adjustment process of the system model as a Markov decision process, where the system state space, action space, state transition probabilities, and reward function are defined, and the system model parameters are updated through the interaction between the reinforcement learning agent and the environment.

[0009] In a further technical solution, the reward function is composed of a model error term and a model parameter adjustment term.

[0010] A further technical solution involves monitoring the deviation between the edge-side monitoring system model and the actual operation. When the deviation exceeds a preset threshold, feedback is triggered, and the real-time operating status is fed back to the cloud side. The cloud side corrects and updates the parameters of the system model based on an incremental learning mechanism.

[0011] Further technical solutions involve generating prediction data on the cloud side, specifically including: Multidimensional features are extracted from historical system operation data to construct a long short-term memory network prediction model; The system's historical operating data from multiple consecutive time periods are input into the Long Short-Term Memory Network prediction model to establish a source-load prediction model, and a rolling prediction method is used to generate source-load prediction sequences for multiple future time periods.

[0012] A further technical solution is that the objective function for the rolling optimization is:

[0013] in, To optimize the start time of rolling execution, To optimize the time window for scrolling, for Real-time electricity price coefficient in the electricity market for The actual electrical power purchased by the real-time data center from the main power grid. For the cost of waste heat recovery, The actual heat recovered by the absorption chiller; Costs related to the lifespan degradation of energy storage systems.

[0014] In a further technical solution, the system constraints include absorption chiller constraints, precision air conditioning constraints, energy storage device constraints, electrical load balance constraints, and cooling load balance constraints.

[0015] Secondly, the present invention provides a data center cloud-edge collaborative energy management system based on reinforcement learning, including a cloud server and an edge server, wherein the cloud server and the edge server are communicatively connected. The cloud server is configured to: construct a system model using reinforcement learning based on historical system operation data, receive real-time operation status feedback from the edge side, dynamically update the parameters of the system model according to the real-time operation status; and also predict the power generation of new energy sources and the power demand of load based on historical system operation data, generate prediction data, and send the updated system model and the prediction data to the edge server. The edge server is configured to: receive the system model and prediction data sent by the cloud server; adopt model predictive control; perform rolling optimization with the goal of minimizing the overall operating cost under the condition of satisfying system constraints; solve for and execute the control sequence; and feed back the real-time operating status to the cloud server so that the cloud server can correct and update the parameters of the system model, thereby realizing the closed-loop self-updating of the system model.

[0016] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data center cloud-edge collaborative energy management method based on reinforcement learning as described in the first aspect.

[0017] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the reinforcement learning-based data center cloud-edge collaborative energy management method described in the first aspect.

[0018] The above one or more technical solutions have the following beneficial effects: This invention utilizes reinforcement learning to construct a system model on the cloud side and dynamically updates model parameters based on real-time operational status feedback from the edge side. Simultaneously, it incorporates source-load prediction functionality distributed to the edge side, effectively solving the model mismatch problem caused by dynamic changes in system operating conditions and sudden changes in its own parameters in traditional centralized energy management. This improves the adaptability and accuracy of the system model. The edge side employs model predictive control for rolling optimization, minimizing the comprehensive operating costs comprised of electricity purchase, energy storage losses, and waste heat recovery while meeting system constraints. It can respond in real-time to fluctuations in renewable energy generation and load changes, achieving improved source-load matching and operational economy. Furthermore, the edge side feeds back real-time operational status to the cloud side, forming a closed-loop self-updating mechanism for the model, further enhancing the system's robustness and adaptability under complex operating conditions. This provides an efficient and economical energy management solution for integrated energy supply systems in data centers.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart of the data center cloud-edge collaborative energy management method based on reinforcement learning, according to an embodiment of the present invention. Figure 2 This is a structural diagram of the integrated power supply system for a data center according to an embodiment of the present invention; Figure 3 This is a diagram of the intelligent energy management architecture for data center cloud-edge collaboration in an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] Example 1 Existing technologies primarily construct integrated energy system models based on the load-energy supply-demand relationship of integrated energy equipment in the park. Then, with the goal of minimizing daily energy costs, a distributed robust optimization algorithm is used to calculate the planned daily output value of the system. A state prediction model is then built by combining the operating status, output increment, and disturbance of each unit in the park's integrated energy equipment. Subsequently, the prediction results obtained from this state prediction model are used to establish a rolling optimization objective function with the planned daily output value, and the optimal control quantity for intraday scheduling is obtained. Finally, based on this state prediction, a dual-delay deep deterministic strategy gradient algorithm is used to learn and optimize the weight parameters in the rolling optimization objective function, thereby achieving integrated energy system operation optimization.

[0026] Although this existing technology combines reinforcement learning with model predictive control, the role of reinforcement learning is only used for learning and updating the weight parameters of the objective function during the optimization process. It does not solve the problem of system model mismatch caused by changes in system operating conditions or sudden changes in the parameters of the system model itself, and its system model has poor adaptability.

[0027] To address the aforementioned issues, this invention proposes a cloud-edge collaborative energy management strategy for data center power supply systems based on reinforcement learning. This strategy employs a cloud-edge two-layer collaborative architecture to achieve adaptive updates of the system model, resolving the system model mismatch problem and improving model adaptability.

[0028] The cloud side uses reinforcement learning to build a system model based on historical system operation data, and dynamically updates the model parameters by combining the real-time operation status feedback from the edge side. At the same time, the cloud side predicts new energy power generation and load demand, providing data input for edge side rolling optimization.

[0029] The edge device receives the latest system model and prediction data from the cloud. It employs Model Predictive Control (MPC) to minimize the overall cost, including electricity purchase, energy storage losses, and waste heat recovery, while satisfying system constraints. The MPC rolling optimization principle is applied to address source-load fluctuations. When the system's model deviates significantly due to dynamic operating conditions and sudden changes in its own parameters, the edge device feeds back the real-time system operating status to the cloud. The cloud then corrects and updates the model parameters based on an incremental learning mechanism, achieving closed-loop self-updating of the system model through this feedback mechanism.

[0030] like Figure 2 This diagram illustrates the power supply system architecture for a data center, primarily including the power grid, renewable energy power generation systems such as photovoltaics, energy storage systems, and data center equipment. The energy storage system comprises precision air conditioning, PCS inverters, and energy storage devices; the data center equipment includes the data center itself. This indicates the power of the energy storage after passing through the inverter.

[0031] Based on the physical structure of the aforementioned integrated power supply system for data centers, this invention addresses the model mismatch and source-load fluctuation problems caused by variable system operating conditions and sudden changes in its own parameters. It constructs an energy management system that integrates cloud and edge collaboration, such as... Figure 3 As shown.

[0032] The cloud side undertakes the tasks of system model building and source load prediction, specifically: (1) The cloud side constructs a system model based on historical operation information and combines it with the real-time operation status and working condition data of the system fed back from the edge side. Reinforcement learning is used to incrementally update the system model parameters. By reconstructing the system model, the model mismatch problem caused by changes in working conditions and sudden changes in its own parameters is solved. This provides an accurate model foundation for the edge-side operation optimization based on MPC.

[0033] (2) Based on the historical data of system source and load, the prediction of new energy power generation and load demand is realized, providing data input for the side-side operation optimization based on MPC.

[0034] The edge component is responsible for optimization scheduling and status feedback tasks, specifically: (1) The edge uses the latest system model and prediction data issued by the cloud side, adopts the MPC method, takes minimizing the comprehensive cost of power purchase, energy storage loss and waste heat recovery as the optimization goal, and generates the optimal power output plan of the system; the source load fluctuation problem is solved by using the MPC rolling optimization principle.

[0035] (2) The edge side collects the operating status in real time during the system operation and calculates the deviation between the given value and the actual operating value. When the deviation exceeds the preset threshold, the edge side feeds back the real-time operating status and working condition information of the system to the cloud side, triggering the cloud side model retraining and incremental update, and dynamically correcting the model parameters.

[0036] By using a cloud-edge collaborative deviation feedback mechanism, the system model can be updated in a closed loop to solve the model mismatch problem in complex time-varying environments and improve the system's operational economy.

[0037] Based on the above cloud-edge collaboration mechanism and model adaptive update strategy, the overall operation and control process of this invention is as follows: Figure 1 As shown.

[0038] like Figure 1 As shown, this embodiment discloses a data center cloud-edge collaborative energy management method based on reinforcement learning, including: Cloud-side execution: Based on historical system operation data, a system model is constructed using reinforcement learning, and the real-time operation status feedback from the edge side is received. The parameters of the system model are dynamically updated according to the real-time operation status. It also predicts the power generation of new energy sources and the power demand of load based on the historical operation data of the system, generates prediction data, and sends the updated system model and the prediction data to the edge side; Edge-side execution: Receive system models and prediction data from the cloud, adopt model predictive control, and perform rolling optimization with the goal of minimizing overall operating cost while satisfying system constraints, solve for the control sequence and execute it; The real-time operating status is fed back to the cloud side, so that the cloud side can correct and update the parameters of the system model, thereby realizing the closed-loop self-updating of the system model.

[0039] (a) Cloud side Based on reinforcement learning and multi-source data-driven mechanisms, this system undertakes high-computing-power system modeling and source-load prediction tasks, providing model and data support for the adaptive energy management of the entire system. Its specific implementation plan consists of the following two functional modules: (1) System modeling and adaptive updating based on reinforcement learning The cloud side first extracts and organizes historical operational information, including power data from renewable energy sources such as photovoltaics, data center electrical load data, cooling load data, energy storage system state of charge and charging / discharging power data, main grid power purchase data, waste heat recovery data, precision air conditioning and absorption chiller operational status data, data center ambient temperature data, and historical scheduling commands and actual execution results data. Based on this, a comprehensive data center power supply system model is constructed to describe the dynamic mapping relationship between system state, control inputs, and external disturbances, providing a model foundation for edge-side MPC-based operational optimization. This process can be represented as:

[0040] in, for The first device model The predicted state at time (the next time step), for The first device model The predicted state at time (current time). To control the input values ​​for the corresponding model, For external disturbances or operating condition variables, The system model is based on historical operational information. Represents system model parameters, This represents the number of system models.

[0041] Based on the predicted state values ​​of the system model, the corresponding model output estimates can be obtained. for:

[0042] in, This is a mapping function from state to output.

[0043] From this, the system model deviation at the next moment can be obtained. for:

[0044] in, For the model in The actual output value at time (the next time).

[0045] Furthermore, system model parameters and deviation It can be represented as:

[0046]

[0047] in, This indicates transpose.

[0048] Based on this, the adaptive adjustment process of the cloud-side system model is modeled as a Markov Decision Process (MDP), defining the system state space, action space, probability transition matrix, and reward function. The system model parameters are updated through the interaction between the reinforcement learning agent and the environment. Specifically, this can be represented as:

[0049] in, Represented as a Markov decision process, For state space, For the action space, Let be the state transition probability. This is the reward function.

[0050] First, define the system state vector for reinforcement learning. ,include External disturbances or operating condition variables in the system model at any given time System model parameters and model bias :

[0051] In Markov decision-making, the agent determines the current state based on the action. Take action This allows the agent to transition from the current state to the next state. The probability of transition is:

[0052] in, For action variables.

[0053] Action variables The definition is as follows:

[0054]

[0055] in, for The system model parameters are adjusted at each moment by reinforcement learning. for System model parameters at time (next time).

[0056] To balance system model accuracy and system operational economy, the reward function is defined as consisting of a model error term and a model parameter adjustment term. The smaller the model error and the lower the system operating cost, the higher the corresponding reward value, which can be expressed as follows:

[0057] in, As a reward value, These are the weighting coefficients corresponding to the model error term. These are the weight coefficients corresponding to the model parameter adjustment terms. The system model parameters are updated through the interaction between the reinforcement learning agent and the environment.

[0058] To address model mismatch caused by physical parameter offsets, the cloud side receives real-time operational status feedback from the edge side. When the prediction bias detected by the edge side continuously exceeds a threshold, the cloud side dynamically updates the model parameters using an incremental learning method. This enables the model to achieve closed-loop self-updating.

[0059] (2) Source-load prediction function To improve the adaptability of edge-side rolling optimization to source-side power output fluctuations and load changes, the cloud side extracts multi-dimensional features from historical operating data such as meteorological information, data center ambient temperature data, photovoltaic and other new energy power data, data center electrical load data, cooling load data, and time characteristics. Based on this, a Long Short-Term Memory (LSTM) network prediction model is constructed to predict the source-side power output and load demand in the data center integrated power supply system. The prediction results are then sent to the edge side as input information for MPC.

[0060] The cloud side first performs multi-dimensional feature extraction on historical operational data, selecting photovoltaic data, data center power demand, and cooling load as... Vector information at time:

[0061]

[0062] in, for The original input vector at time step 1. for The power generation capacity of new energy sources at any given time for The power demand of the load at any given time. for The power demand for cooling load at any given time. This is the temporal feature vector obtained after multidimensional feature extraction. This indicates the extraction of multidimensional features from the vector information of the original load.

[0063] The cloud side inputs historical data from multiple consecutive time points into an LSTM network to establish a source-load prediction model. The prediction process can be represented as follows:

[0064] in, For LSTM networks in The hidden state at all times This is the LSTM state update function.

[0065] To meet the edge-side model predictive control's requirements for prediction time-domain data, the cloud side generates source load prediction sequences for multiple future time periods based on current and historical source load information using a rolling prediction method:

[0066] in, To output the mapping function, This outputs a future source load prediction sequence for multiple future time periods, representing the current output source load prediction sequence. To predict the length of the time domain.

[0067] To improve prediction accuracy, the LSTM model is trained using the error between the actual and predicted values, and its loss function is... It can be represented as:

[0068] After training, the cloud side outputs the source-load prediction sequence for the future time domain and sends it to the edge side. The edge side uses the prediction results during the MPC optimization process, thereby improving the operational economy and adaptability of the integrated energy supply system under source-load fluctuation conditions.

[0069] (ii) Edge side The system utilizes the latest cloud-based system model and source-load prediction data to perform operational optimization functions and formulate the optimal output plan for the integrated energy supply system. Simultaneously, it collects real-time system operating status data; when the optimized value exceeds the scheduling deviation threshold, the system's real-time status is fed back to the cloud to update model parameters, achieving adaptive model updates.

[0070] (1) Rolling optimization based on model predictive control Based on the latest system model and source-load prediction data from the cloud side, the edge side employs MPC to minimize the overall system operating cost and performs rolling optimization to derive the optimal system output plan. By performing rolling optimization at each time step, the system can adjust its output in real time, effectively solving the source-load mismatch problem caused by fluctuations in renewable energy output and changes in computational load.

[0071] Its objective function is defined as optimizing the rolling time window. Minimize cumulative costs within:

[0072] in, To optimize the start time of rolling execution; Optimize the time window for scrolling; for Real-time electricity price coefficient in the electricity market; for The actual electrical power purchased by the real-time data center from the main power grid; Cost of waste heat recovery; The actual heat recovered by the absorption chiller; The formula for calculating the lifespan degradation cost of an energy storage system is as follows:

[0073] in, The attenuation cost factor per unit charge / discharge power. for Real-time energy storage and charging power, for Energy storage and discharge power at all times This represents the time interval between adjacent scheduling moments.

[0074] The optimization should meet the following constraints: 1) Constraints of absorption chillers:

[0075]

[0076] in, For absorption chillers The cooling capacity provided at all times For absorption chillers Energy efficiency coefficient at any time The total heat generated by the data center This corresponds to the waste heat recovery efficiency. For absorption chillers In addition, the absorption chiller's operating range also meets the following requirements: (The text abruptly ends here, so the translation stops as well.)

[0077]

[0078] in, This represents the maximum cooling capacity of the absorption chiller. To maximize the recovery of waste heat.

[0079] 2) Precision air conditioning constraints The cooling capacity of a precision air conditioner typically satisfies an energy efficiency relationship with its power consumption:

[0080] in, For precision air conditioning The cooling capacity provided at all times. and These correspond to the characteristic temperatures of the evaporator and condenser sides of a precision air conditioner, respectively. For the power consumption of precision air conditioning, This represents the corresponding fitting coefficient. Meanwhile, the operating range of precision air conditioners should meet the following requirements:

[0081]

[0082] in, This represents the maximum power consumption of a precision air conditioner. This is the maximum cooling capacity of a precision air conditioner.

[0083] 3) Constraints on energy storage devices: To protect battery life and maintain cross-period energy dispatch capability, the state transition and power of the energy storage system must satisfy the following relationship:

[0084] Meanwhile, its charging and discharging power is subject to the following constraints:

[0085]

[0086] in, for State of charge at time t, For energy storage device capacity, and These are the discharge power and charging power of the energy storage system, respectively. and For the corresponding charge and discharge efficiency, , These are the maximum charging power and maximum discharging power of the energy storage device, respectively.

[0087] 4) Electrical load balance constraints:

[0088] in, Power output for photovoltaic and other new energy sources This indicates the server's power consumption.

[0089] 5) Cooling load balance constraints: The cooling capacity supply should not be lower than the data center's cooling load demand, while an appropriate surplus of cooling capacity can be used to mitigate cooling load fluctuations, compensate for model errors, and be absorbed through the thermal inertia of the computer room and equipment. Therefore, the cooling load balance constraint can be expressed as:

[0090] in, To meet the cooling load requirements of the data center.

[0091] (III) Cloud-Edge Collaboration (1) Deviation feedback and triggering retraining The edge-side monitoring system detects deviations between the model and actual operation. When the deviation exceeds a preset threshold, feedback is triggered, and the real-time operating status is fed back to the cloud side. The cloud side corrects and updates the parameters of the system model based on an incremental learning mechanism.

[0092] During execution, the edge side collects the system's operating status in real time and calculates the deviation between the optimized given value and the actual value. , as in the formula As shown.

[0093] To define the degree of mismatch in the current model, a bias evaluation index is defined. for:

[0094] in, For deviation, For the model in The actual output value at time 1. A tiny positive number is introduced to prevent the denominator from being zero.

[0095] When the system model deviation evaluation index exceeds the set threshold At this time, incremental learning of the model by cloud-based RL will be triggered, with the threshold defined as:

[0096] in, This means that the 95th percentile of the historical normal operation error is taken as the trigger threshold.

[0097] (2) Dynamic model distribution Once the model parameters are updated, a corrected system model matching the current operating conditions is generated in the cloud and dynamically distributed to the edge. The edge then calls upon this updated model for optimization in the next rolling optimization cycle, thereby achieving continuous iteration and online adaptive correction of the system model.

[0098] This invention's energy management method comprises a two-layer cloud and edge architecture: the cloud side constructs a system model based on historical operational data and predicts source-side output and load demand; the edge side applies the model and predicted data distributed by the cloud side, using model predictive control for rolling optimization to generate the optimal system output plan with the goal of minimizing overall operating costs. During operation, the edge side monitors model deviation in real time. When the deviation exceeds a preset threshold, the real-time status is fed back to the cloud side. The cloud side uses reinforcement learning to incrementally update the model parameters and re-distributes the updated model to the edge side, ultimately achieving closed-loop adaptive updating of the model under cloud-edge collaboration to solve the model mismatch problem, thereby improving the economic efficiency of the energy supply system.

[0099] In summary, the innovation of this invention lies in: The cloud side leverages massive global data and high computing power resources to construct a comprehensive energy supply system model for data centers and predicts the output of new energy sources and load demand. At the same time, by combining real-time status and new operating condition data from the edge side, reinforcement learning is used to update the model parameters to achieve adaptive updates of the system model, thereby improving the model accuracy.

[0100] The edge side directly calls the model and prediction information issued by the cloud side. Under the premise of meeting the system operation constraints, the MPC method is applied to solve the optimal output plan of the integrated energy supply system in real time, solve the source load fluctuation problem, and achieve the optimal economic operation of the system.

[0101] The cloud-edge closed-loop adaptive update mechanism introduces real-time deviation information from the edge into the cloud-side model update process. When the model mismatch exceeds the limit, it automatically triggers incremental learning and model reconstruction on the cloud side. The system model is updated based on the new environmental conditions and applied to edge-side optimization, forming a cloud-edge collaborative model closed-loop feedback update.

[0102] Example 2 This embodiment discloses a data center cloud-edge collaborative energy management system based on reinforcement learning, including a cloud server and an edge server, wherein the cloud server and the edge server are communicatively connected. The cloud server is configured to: construct a system model using reinforcement learning based on historical system operation data, receive real-time operation status feedback from the edge side, dynamically update the parameters of the system model according to the real-time operation status; and also predict the power generation of new energy sources and the power demand of load based on historical system operation data, generate prediction data, and send the updated system model and the prediction data to the edge server. The edge server is configured to: receive the system model and prediction data sent by the cloud server; adopt model predictive control; perform rolling optimization with the goal of minimizing the overall operating cost under the condition of satisfying system constraints; solve for and execute the control sequence; and feed back the real-time operating status to the cloud server so that the cloud server can correct and update the parameters of the system model, thereby realizing the closed-loop self-updating of the system model.

[0103] Example 3 The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 1.

[0104] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 1.

[0105] The steps and methods involved in the apparatuses of Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0106] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0107] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0108] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A data center cloud-edge collaborative energy management method based on reinforcement learning, characterized in that, include: Cloud-side execution: Based on historical system operation data, a system model is constructed using reinforcement learning, and the real-time operation status feedback from the edge side is received. The parameters of the system model are dynamically updated according to the real-time operation status. It also predicts the power generation of new energy sources and the power demand of load based on the historical operation data of the system, generates prediction data, and sends the updated system model and the prediction data to the edge side; Edge-side execution: Receive system models and prediction data from the cloud, adopt model predictive control, and perform rolling optimization with the goal of minimizing overall operating cost while satisfying system constraints, solve for the control sequence and execute it; The real-time operating status is fed back to the cloud side so that the cloud side can correct and update the parameters of the system model, thereby realizing the closed-loop self-updating of the system model. The system model is represented as follows: in, for The first device model Predicted state at any given time for The first device model Predicted state at any given time To control the input values ​​for the corresponding model, For external disturbances or operating condition variables, The system model is based on historical operational information. Represents system model parameters, The number of system models; Based on the predicted state values ​​of the system model, the corresponding model output estimates can be obtained. for: in, This is a mapping function from state to output; From this, the system model deviation at the next moment can be obtained. for: in, For the model in The actual output value at that moment; A system model is constructed based on historical system operation data, and the model parameters are adjusted using reinforcement learning. Specifically, the adaptive adjustment process of the system model is modeled as a Markov decision process, in which the system state space, action space, state transition probability and reward function are defined, and the system model parameters are updated through the interaction between the reinforcement learning agent and the environment. The system state vector in the system state space includes external disturbances or operating condition variables of the system model, system model parameters, and model bias; the reward function is composed of a model error term and a model parameter adjustment term, expressed as: in, As a reward value, These are the weighting coefficients corresponding to the model error term. The weight coefficients corresponding to the model parameter adjustment terms. for The system model parameters are adjusted at each moment by reinforcement learning. This represents model bias; The objective function for the rolling optimization is: in, To optimize the start time of rolling execution, To optimize the time window for scrolling, for Real-time electricity price coefficient in the electricity market for The actual electrical power purchased by the real-time data center from the main power grid. For the cost of waste heat recovery, The actual heat recovered by the absorption chiller; Costs related to the lifespan degradation of energy storage systems; The system constraints include absorption chiller constraints, precision air conditioning constraints, energy storage device constraints, electrical load balance constraints, and cooling load balance constraints; the cooling load balance constraint is described as follows: in, For precision air conditioning The cooling capacity provided at all times. For absorption chillers The cooling capacity provided at all times To meet the cooling load requirements of the data center; The edge-side monitoring system detects deviations between the model and actual operation. When the deviation exceeds a preset threshold, feedback is triggered, and the real-time operating status is fed back to the cloud side. The cloud side corrects and updates the parameters of the system model based on an incremental learning mechanism. During execution, the edge side collects the system's operating status in real time and calculates the deviation between the optimized given value and the actual value. To define the degree of mismatch in the current model, define the bias evaluation index. for: in, For deviation, For the model in The actual output value at time 1. Tiny positive numbers are introduced to prevent the denominator from being zero; When the system model deviation evaluation index exceeds the set threshold At this time, incremental learning of the model by cloud-based RL will be triggered, with the threshold defined as: in, This means that the 95th percentile of the historical normal operation error is taken as the trigger threshold.

2. The data center cloud-edge collaborative energy management method based on reinforcement learning as described in claim 1, characterized in that, Cloud-based prediction data generation includes: Multidimensional features are extracted from historical system operation data to construct a long short-term memory network prediction model; The system's historical operating data from multiple consecutive time periods are input into the Long Short-Term Memory Network prediction model to establish a source-load prediction model, and a rolling prediction method is used to generate source-load prediction sequences for multiple future time periods.

3. A data center cloud-edge collaborative energy management system based on reinforcement learning, employing the data center cloud-edge collaborative energy management method based on reinforcement learning as described in any one of claims 1-2, characterized in that, It includes cloud servers and edge servers, with the cloud servers and edge servers communicating with each other; The cloud server is configured to: construct a system model using reinforcement learning based on historical system operation data, receive real-time operation status feedback from the edge side, dynamically update the parameters of the system model according to the real-time operation status; and also predict the power generation of new energy sources and the power demand of load based on historical system operation data, generate prediction data, and send the updated system model and the prediction data to the edge server. The edge server is configured to: receive system models and prediction data from the cloud server, adopt model predictive control, and perform rolling optimization with the goal of minimizing overall operating costs while satisfying system constraints, solve for and execute control sequences; The real-time operating status is fed back to the cloud server, so that the cloud server can correct and update the parameters of the system model, thereby realizing the closed-loop self-updating of the system model.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the reinforcement learning-based data center cloud-edge collaborative energy management method as described in any one of claims 1-2.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the data center cloud-edge collaborative energy management method based on reinforcement learning as described in any one of claims 1-2.

Images