Unsupervised adaptive energy-saving optimization algorithm and method based on reinforcement learning

Abstract

The invention provides an unsupervised adaptive energy-saving optimization algorithm and method based on reinforcement learning, and relates to the technical field of machine learning, the algorithm comprises a state space S, an action space A, a reward function R and a Q-learning algorithm, the state space S comprises various equipment parameters and environment states of a cold source group control system; the action space A is used for adjusting control parameters according to a reinforcement learning strategy based on a current state; the reward function R is used for measuring the energy-saving effect and the equipment operation condition after the action is adjusted; the Q-learning algorithm is used for the intelligent agent to judge whether different actions are adopted in each state or not by using a Q value; according to the method, the control strategy is directly learned and optimized from environment feedback through a reinforcement learning algorithm under the condition of lacking historical data, and the method does not depend on a prior accurate model and can adapt to a dynamically changing environment.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and particularly to an unsupervised adaptive energy-saving optimization algorithm and method based on reinforcement learning. Background Art

[0002] Currently, the energy consumption management of the cold source group control system mainly relies on model-based control methods, which often require a large amount of historical data to establish an accurate energy consumption model;

[0003] However, in the early stage of the project, the lack of historical data makes it difficult to build the model, resulting in unsatisfactory control effects. In addition, these traditional methods lack adaptability in the face of environmental changes and cannot adjust control parameters in real time, thus affecting the overall energy efficiency of the system. At the same time, it is also difficult to achieve multi-objective optimization, and it is often impossible to balance equipment stability and user comfort. Therefore, the present invention proposes an unsupervised adaptive energy-saving optimization algorithm and method based on reinforcement learning to solve the problems existing in the prior art. Summary of the Invention

[0004] In view of the above problems, the present invention proposes an unsupervised adaptive energy-saving optimization algorithm and method based on reinforcement learning. Through real-time learning and dynamic adjustment of control parameters, the intelligent agent can accurately improve energy efficiency without rich historical data, while taking into account equipment stability and user comfort, thereby reducing operation and maintenance costs and promoting the intelligent development of energy-saving technologies.

[0005] To achieve the object of the present invention, the present invention is realized through the following technical solutions: An unsupervised adaptive energy-saving optimization algorithm based on reinforcement learning, including a state space S, an action space A, a reward function R, and a Q-learning algorithm. The state space S includes various device parameters and environmental states of the cold source group control system;

[0006] The action space A is used to adjust the control parameters based on the current state according to the reinforcement learning strategy;

[0007] The reward function R is used to measure the energy-saving effect and equipment operation condition after the adjustment action;

[0008] The Q-learning algorithm is used for the intelligent agent to use Q values to judge the quality of taking different actions in each state.

[0009] A further improvement lies in that: in the state space S, the input control parameters include: supply water temperature 、supply air temperature 、water pump frequency 、supply air volume 、water flow rate ,

[0010] State is represented as a set as follows

[0011] .

[0012] A further improvement lies in that: in the action space A, the action is defined as a small adjustment to the input parameter, and this set is represented as:

[0013] .

[0014] A further improvement lies in that: the reward function R consists of the following:

[0015] Reward for energy consumption reduction: By comparing the energy consumption changes before and after the adjustment, the reward value is proportional to the energy consumption reduction :

[0016]

[0017] where is the energy consumption change coefficient;

[0018] Reward for equipment operation stability: After the system is adjusted, the ability to maintain stable operation is measured by and the reward value is proportional to the system stability:

[0019]

[0020] where is the stability coefficient;

[0021] Reward for user comfort: Based on the indoor temperature after the equipment is adjusted , the reward value reflects whether the equipment adjustment maintains the comfort set value :

[0022]

[0023] where is the comfort coefficient;

[0024] The final comprehensive reward function is:

[0025] .

[0026] A further improvement lies in that: in the Q-learning algorithm, the Q value represents the sum of the expected future rewards that can be obtained by taking the action in the state :

[0027]

[0028] where is an immediate reward, is the discount factor (between 0 and 1), which is used to balance the impact of the current reward and future rewards;

[0029] After each action is executed by the agent, the Q-value is recursively updated through the following Bellman equation:

[0030]

[0031] where: is the current state, is the currently selected action, is the current reward, is the new state after the action is executed, represents the maximum Q-value that can be obtained in the new state, is the learning rate, which determines the degree of trust in new information when updating the Q-value.

[0032] The unsupervised adaptive energy-saving optimization method based on reinforcement learning includes the following steps:

[0033] S1: Initialize the Q-value table of the system ;

[0034] S2: At each time step , the reinforcement learning agent selects an action based on the current Q-value ;

[0035] S3: The selected action is executed in the cold source group control system to adjust the control parameters of the system;

[0036] S4: Obtain the adjusted state and the system operation status through energy consumption data and device logs, calculate the real-time energy consumption change, device stability, and user comfort, and then calculate the reward ;

[0037] S5: Use the reward and state transition information to update the Q-value according to the reinforcement learning algorithm;

[0038] S6: Update the system state to , and repeat the above steps until the energy-saving effect converges or reaches the set optimization goal of the system.

[0039] A further improvement lies in that: in the above S1, the initial state of the cold source group control system includes the initial values of each control parameter

[0040] A further improvement lies in that: in the above S2:

[0041]

[0042] is an adaptive parameter: adjusted according to the performance of the agent value.

[0043] A further improvement lies in that: in the S5, the update formula based on Q-learning is:

[0044]

[0045] where is the learning rate, is the discount factor.

[0046] The beneficial effects of the present invention are:

[0047] 1. Through the reinforcement learning algorithm, the present invention directly learns and optimizes the control strategy from the environmental feedback in the absence of historical data. This method does not rely on a priori accurate models and can adapt to dynamically changing environments.

[0048] 2. The agent of the present invention can dynamically adjust the system control parameters according to the real-time obtained energy consumption data, environmental data, and device status. This adaptive ability enables the system to cope with changes in the external environment and improve energy efficiency.

[0049] 3. Through the comprehensive reward function, the algorithm not only optimizes energy consumption but also considers the stability of the device and the comfort of the user, thereby achieving multi-objective optimization.

[0050] 4. The algorithm of the present invention has the ability of online learning, can continuously optimize the strategy during operation, and is adjusted through real-time feedback, improving the overall operation efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram of the method framework of the present invention, including the input control parameters (water supply temperature, air supply temperature, water pump frequency, air supply volume, and flow rate) and the real-time feedback adjustment mechanism of the agent. The agent analyzes the input parameters and operating conditions, and uses the comprehensive reward function to optimize the control strategy, taking into account the balance of energy consumption reduction, device stability, and user comfort. In this way, the system can adaptively improve the energy efficiency and user experience of the cold source group control system in the absence of historical data. DETAILED DESCRIPTION OF THE INVENTION

[0052] To deepen the understanding of the present invention, the following will further elaborate on the present invention in combination with embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0053] Embodiment 1

[0054] According to Figure 1 As shown, this embodiment proposes an unsupervised adaptive energy-saving optimization algorithm based on reinforcement learning. This algorithm mainly aims at the problem of optimizing the energy consumption of equipment lacking historical data. Especially in the early stage of the chilled water source group control system, when it is difficult to accurately model the operating parameters of the equipment, the reinforcement learning algorithm is used to adaptively adjust the operating parameters of the equipment (such as supply water temperature, supply air temperature, water pump frequency, supply air volume, water flow rate, etc.), combined with energy consumption data, environmental data and equipment feedback logs, to gradually optimize the energy consumption of the equipment and achieve the energy-saving effect.

[0055] Basic framework of the algorithm

[0056] The energy-saving optimization is realized based on the Markov decision process (MDP) of reinforcement learning, which mainly consists of the following parts: state space (S), action space (A), reward function (R) and Q-learning algorithm.

[0057] State space : It includes various equipment parameters and environmental states of the chilled water source group control system. The input control parameters include: supply water temperature , supply air temperature , water pump frequency , supply air volume , water flow rate .

[0058] State can be represented as a set, as follows

[0059]

[0060] Action space : Based on the current state, the algorithm adjusts the control parameters according to the reinforcement learning strategy. The action is defined as a small adjustment of the input parameters, and this set can be expressed as:

[0061]

[0062] Reward function : The reward function is used to measure the energy-saving effect and the equipment operation condition after the adjustment action. The reward function consists of the following parts:

[0063] Energy consumption reduction reward: By comparing the energy consumption changes before and after the adjustment, the reward value is proportional to the energy consumption reduction .

[0064]

[0065] where is the energy consumption change coefficient.

[0066] Device operation stability reward: After the system is adjusted, the ability to maintain stable operation is measured by and the reward value is proportional to the system stability.

[0067]

[0068] where is the stability coefficient.

[0069] User comfort reward: Based on the indoor temperature after the device is adjusted , the reward value reflects whether the device adjustment maintains the comfort set value ;

[0070]

[0071] where is the comfort coefficient.

[0072] The final comprehensive reward function is:

[0073]

[0074] Adaptive energy-saving optimization algorithm based on Q-learning

[0075] Q-learning is a common value-function-based reinforcement learning method. The agent optimizes the policy by learning the Q-values of each state-action pair. In this energy-saving optimization scenario, the agent uses Q-values to judge the goodness of different actions in each state.

[0076] Q-value represents the sum of the expected future rewards that can be obtained by taking action in state :

[0077]

[0078] where, is the immediate reward, is the discount factor (usually between 0 and 1), which is used to balance the influence of the current reward and the future reward.

[0079] After each time the agent executes an action, the Q-value is recursively updated through the following Bellman equation:

[0080]

[0081] where: is the current state, is the currently selected action. is the current reward. is the new state after executing the action. Represents the maximum Q-value that can be obtained in the new state. is the learning rate, which determines the degree of trust in new information when updating the Q-value.

[0082] This unsupervised adaptive energy-saving optimization algorithm is applicable to various cold source group control systems, and it performs excellently especially in the following environments:

[0083] Commercial buildings: Used for energy consumption optimization of air conditioning and HVAC systems.

[0084] Industrial facilities: Optimize the energy efficiency of refrigeration, heating, and ventilation systems.

[0085] Example Two

[0086] According to Figure 1 as shown, this example proposes an unsupervised adaptive energy-saving optimization method based on reinforcement learning, including the following processes:

[0087] Initialization: The system first initializes the Q-value table . The initial state of the cold source group control system includes the initial values of each control parameter .

[0088] Action selection: At each time step , the reinforcement learning agent selects an action according to the current Q-value : :

[0089]

[0090] is the adaptive parameter: adjusts the value according to the performance of the agent. For example, when the performance of the agent is stable, reduce exploration (decrease ); when the performance is poor, increase exploration (increase ).

[0091] Execute action: The selected action will be executed in the cold source group control system to adjust the control parameters of the system.

[0092] Feedback collection: Obtain the adjusted state and the system operation situation through energy consumption data and device logs, calculate the real-time energy consumption change, device stability, and user comfort, and then calculate the reward .

[0093] Update policy: Use the reward and state transition information to update the Q-value according to the reinforcement learning algorithm. The update formula based on Q-learning is:

[0094]

[0095] Among them, is the learning rate, is the discount factor.

[0096] State update: Update the system state to , and repeat the above steps until the energy-saving effect converges or reaches the optimization goal set by the system.

[0097] Through the reinforcement learning algorithm, in the absence of historical data, the present invention directly learns from the environmental feedback and optimizes the control strategy. This method does not rely on a priori accurate models and can adapt to dynamic environments. Moreover, the agent of the present invention can dynamically adjust the system control parameters according to the real-time obtained energy consumption data, environmental data, and device status. This adaptive ability enables the system to cope with changes in the external environment and improve energy efficiency. At the same time, through the comprehensive reward function, while optimizing energy consumption, the algorithm also considers the stability of the device and the comfort of the user, thereby achieving multi-objective optimization. In addition, the algorithm has the ability of online learning, can continuously optimize the strategy during operation, and is adjusted through real-time feedback, improving the overall operation efficiency of the system.

[0098] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art of this industry should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. An unsupervised adaptive energy-saving optimization algorithm based on reinforcement learning, including a state space S, an action space A, a reward function R, and a Q-learning algorithm, characterized in that: The state space S includes various device parameters and environmental states of the cold source group control system; The action space A is used to adjust the control parameters based on the current state according to the reinforcement learning strategy; The reward function R is used to measure the energy-saving effect and the device operation condition after the adjustment action; The Q-learning algorithm is used for the agent to use Q-values to judge the goodness or badness of taking different actions in each state.

2. The unsupervised adaptive energy-saving optimization algorithm based on reinforcement learning according to claim 1, characterized in that: In the state space S, the input control parameters include: the water supply temperature , the air supply temperature , the water pump frequency , the air supply volume , the water flow rate , Status is represented as a set as follows: 。 3. The unsupervised adaptive energy-saving optimization algorithm based on reinforcement learning according to claim 2, characterized in that: In the action space A, the action is defined as a small adjustment to the input parameter, and this set is represented as: 。 4. The unsupervised adaptive energy-saving optimization algorithm based on reinforcement learning according to claim 3, characterized in that: The reward function R consists of the following: Energy consumption reduction reward: By comparing the energy consumption changes before and after adjustment, the reward value is proportional to the energy consumption reduction is proportional to: ， wherein is the energy consumption change coefficient; Device operation stability reward: After the system is adjusted, the ability to maintain stable operation is measured by , and the reward value is directly proportional to the system stability: ， wherein is the stability coefficient; User comfort reward: based on the adjusted indoor temperature of the device , the reward value reflects whether the device adjustment maintains the comfort setpoint : ， Among them is the comfort coefficient; The final comprehensive reward function is: 。 5. The unsupervised adaptive energy-saving optimization algorithm based on reinforcement learning according to claim 4, characterized in that: In the Q-learning algorithm, the Q value represents the sum of the expected future rewards that can be obtained after taking an action in the state : ， Among them, is an immediate reward, is a discount factor (between 0 and 1) used to balance the impact of current rewards and future rewards; After each execution of an action by the agent, the Q-value is recursively updated through the following Bellman equation: ， Wherein: is the current state, is the currently selected action, is the current reward, is the new state after executing the action, represents the maximum Q-value that can be obtained in the new state, is the learning rate, which determines the degree of trust in new information when updating the Q-value.

6. An unsupervised adaptive energy-saving optimization method based on reinforcement learning, which applies the unsupervised adaptive energy-saving optimization algorithm based on reinforcement learning according to claim 5 above, is characterized in that It includes the following steps: S1: Initialize the Q-value table of the system ; S2: At each time step , the reinforcement learning agent selects an action according to the current Q value ; S3: Selected action Execute in the cold source group control system to adjust the control parameters of the system; S4: Obtain the adjusted status through energy consumption data and device logs and the system operation conditions, calculate the real-time energy consumption change, device stability and user comfort, and then calculate the rewards ; S5: Use the reward and state transition information to update the Q-value according to the reinforcement learning algorithm; S6: Update the system status to , and repeat the above steps until the energy-saving effect converges or reaches the optimization goal set by the system.

7. The unsupervised adaptive energy-saving optimization method based on reinforcement learning according to claim 6, characterized in that: In S1, the initial state of the cold source group control system includes the initial values of each control parameter .

8. The unsupervised adaptive energy-saving optimization method based on reinforcement learning according to claim 6, characterized in that: In the said S2: ， is an adaptive parameter: adjusted according to the agent's performance value.

9. The unsupervised adaptive energy-saving optimization method based on reinforcement learning according to claim 6, wherein: In the said S5, the update formula based on Q-learning is: ， Among them, is the learning rate, is the discount factor.

Images