Demand-side resource control method and system based on adaptive learning rate decay

Abstract

The demand-side resource control method and system based on adaptive learning rate decay obtains the real-time demand for electricity and heat, the operating status of each energy device, the real-time electricity price and the control instructions of each energy device; uses the acquired data and the statistical model to establish the demand-side resource control model, which includes the demand-side resource control environment, state space, action space and reward function; uses the reward function to establish the learning rate decay trigger model and the learning rate decay model; during the training of the demand-side resource control model, the learning rate of the demand-side resource control model is adjusted based on the learning rate decay trigger model and the learning rate decay model; uses the trained demand-side resource control model to obtain the demand-side resource control instructions. The adaptive learning rate is dynamically adjusted according to the characteristics of the demand-side resources, so as to improve the efficiency and reliability of the demand-side resource optimization control.

Description

Technical Field

[0001] The present invention relates to the technical field of demand-side resource optimization and regulation, and in particular to a demand-side resource regulation method and system based on adaptive learning rate decay proximal strategy optimization. Background Art

[0002] In the prior art, the optimization methods used for demand-side resource regulation include linear programming, mixed integer linear programming, etc. Although they perform well in some scenarios, these methods often face large-scale computational workloads due to the dynamics and uncertainty of demand-side resources. In recent years, the advantages of machine learning and artificial intelligence technologies, especially reinforcement learning, in quickly solving large-scale optimization and regulation problems have gradually attracted attention. In the training process of reinforcement learning, the choice of learning rate has a crucial impact on the convergence and performance of the algorithm. Under the influence of the dynamics and uncertainty of demand-side resources, the prior art often uses an excessively high learning rate to avoid the influence of demand-side resource uncertainty, but an excessively high learning rate may lead to unstable training, while lowering the learning rate to adapt to the dynamics of demand-side resources may cause the training process to be slow or fall into a local optimum, so it is impossible to regulate and optimize according to the characteristics of demand-side resources, which ultimately leads to low efficiency and inaccuracy of the optimization method, thereby affecting the efficiency and reliability of demand-side resource optimization and regulation. Summary of the invention

[0003] In order to address the deficiencies in the prior art, the present invention provides a demand-side resource control method and system based on adaptive learning rate decay, which optimizes the adaptive learning rate decay of the algorithm based on performance monitoring, dynamically adjusts the adaptive learning rate according to the characteristics of demand-side resources, and improves the efficiency and reliability of demand-side resource optimization and control.

[0004] The present invention adopts the following technical solution.

[0005] The present invention proposes a demand-side resource control method based on adaptive learning rate decay. The power grid includes multiple energy devices to generate electricity and heat, including:

[0006] Obtain the real-time demand for electricity and heat, the operating status of each energy device, the real-time electricity price, and the control instructions of each energy device;

[0007] Using the acquired data, a demand-side resource regulation model is established based on a statistical model. The demand-side resource regulation model includes a demand-side resource regulation environment, a state space, an action space, and a reward function.

[0008] Use the reward function to establish a learning rate decay trigger model and a learning rate decay model;

[0009] During the training process of the demand-side resource regulation model, the learning rate of the demand-side resource regulation model is adjusted based on the learning rate decay trigger model and the learning rate decay model; and the demand-side resource regulation instructions are obtained using the trained demand-side resource regulation model.

[0010] Preferably, based on the statistical model, a demand-side resource regulation statistical model is established, including:

[0011] Establish a demand-side resource control environment based on the operating status of each energy device;

[0012] The state space is constructed based on the real-time demand for electricity and heat, the operating status of each energy device, and the real-time electricity price;

[0013] Construct action space with control instructions of energy equipment;

[0014] Construct a reward function based on the state space and action space;

[0015] The demand-side resource regulation statistical model is constructed based on the demand-side resource regulation environment, state space, action space and reward function.

[0016] Preferably, the state space S satisfies the following relationship:

[0017] S=[i point ,t p ,t q ,s p ,s q ,p gt ,q gt ,q gb ,q tst ,p grid ,p wind ,o tst ,h tst ,ρ rtp ]

[0018] In the formula, i point It is a sign indicating whether the goal has been achieved, 1 means achieved, 0 means not achieved; t p is the difference between the total power generation and the power target; t q is the difference between the total heat generation and the heat target; s p 、s q are the distances between electricity and heat and the central point of supply and demand balance; p gt ,q gt are the electricity and heat generated by the gas turbine respectively; q gb The heat generated by the gas boiler; tst The heat generated by the thermal storage tank; grid Electricity generated for the grid; pwind Electricity generated by the wind turbine; tst is the operating status of the hot storage tank; h tst The heat stored in the thermal storage tank; ρ rtp Real-time electricity price.

[0019] Preferably, the action space A satisfies the following relationship:

[0020] A=[a gb ,a gt ,a tst ,a grid ]

[0021] In the formula, a gb 、a gt 、a tst 、a grid They are the output power of the gas turbine, the output heat of the gas boiler, the output heat of the heat storage tank and the power of the power grid; and the control instructions of each energy device have corresponding operating limits.

[0022] Preferably, the reward function is a function of state space elements and action space elements, satisfying the following relationship:

[0023] R(s,a)=-CF d -F tst +ΔR

[0024] Where R(s,a) is the reward function for state space elements and action space elements; C is the total cost of system operation; F d F is the penalty term for mismatch between electricity and heat supply and demand; tst is the penalty for overcharging or over-discharging of the thermal storage tank; ΔR is the additional reward for achieving a balance between electricity and heat supply and demand.

[0025] Preferably, the penalty term for mismatch between electricity and heat supply and demand satisfies the following relationship:

[0026]

[0027] The penalty term for overcharge and over discharge of the thermal storage tank satisfies the following relationship:

[0028] F tst =0.1*|o tst -o ini |

[0029] In the formula, o ini is the initial state of the hot storage tank;

[0030] The additional bonus for achieving a balance between electricity and heat supply and demand satisfies the following relationship:

[0031]

[0032] In the formula, F m is the set distance threshold, when F d ≤F m When G c is a counter used to track the duration of the system reaching or approaching the target; G m is the set time threshold, when G c ≥G m , the system is considered to have stably reached its goal.

[0033] Preferably, the reward function of the demand-side resource regulation model is used to establish a learning rate decay trigger model and a learning rate decay model, including:

[0034] Get the historical optimal value of the reward function;

[0035] In a learning rate adjustment cycle, if the difference between the value of the reward function and the optimal value is always less than the set threshold, the learning rate decay is triggered; conversely, if the difference between the value of the reward function and the optimal value is not less than the set threshold, or if the difference between the value of the reward function and the optimal value is less than the set threshold for less than one learning rate adjustment cycle, the learning rate decay is not triggered;

[0036] Taking the optimal value of the additional reward as the objective function, the decay rate of the learning rate is determined, and a learning rate decay control model is established.

[0037] Preferably, the historical optimal value of the reward function is obtained to satisfy the following relationship:

[0038] R * (t) = max[R * (t-1),R(t)]

[0039] In the formula, R * (t) is the optimal value of the reward function at time t, and R(t) is the value of the reward function at time t.

[0040] Preferably, the learning rate decay trigger model satisfies the following relationship:

[0041]

[0042] Where D(t) is the trigger signal of learning rate decay at time t, 1 means triggering learning rate decay, and 0 means not triggering learning rate decay; ΔR(t) is the difference between the value of the reward function at time t and the optimal value; θ is the set threshold coefficient, R * (t)·θ represents the set threshold; u(t) is the timing duration when the difference is less than the set threshold; patience is a learning rate adjustment cycle.

[0043] Preferably, when the learning rate decay is triggered, the learning rate decay model satisfies the following relationship:

[0044]

[0045] Where η(t) is the learning rate at time t; λ is the decay rate, which is a positive number less than 1.

[0046] Preferably, a proximal strategy optimization method is used to train the demand-side resource regulation model, and the training process includes:

[0047] Initialize the demand-side resource control environment according to the operating status and operating parameters of each energy device;

[0048] Initialize the state space and action space according to the initialized demand-side resource control environment and the control instructions of the energy equipment;

[0049] Establish strategic network model and value network model;

[0050] Establish control parameters and loss functions for training demand-side resource regulation models using proximal policy optimization methods;

[0051] Perform demand-side resource regulation model training.

[0052] Preferably, the demand-side resource regulation model training is performed, and multiple rounds of training are performed; in each round of training, the demand-side resource regulation model is used to obtain a reward function according to the demand-side resource regulation environment, state space, and action space; at the same time, the policy network model and the value network model are used to calculate the loss function of the demand-side resource regulation model trained by the proximal policy optimization method, and the current performance of the trained demand-side resource regulation model is evaluated based on the loss function. According to the evaluation results, the control parameters of the demand-side resource regulation model trained by the proximal policy optimization method are adjusted, including adjusting the learning rate of the demand-side resource regulation model based on the learning rate decay trigger model and the learning rate decay model; the policy network model and the value network model are optimized using the adjusted control parameters. After multiple rounds of training, a trained demand-side resource regulation model is obtained.

[0053] The present invention also proposes a demand-side resource control system based on adaptive learning rate decay, comprising:

[0054] Acquisition module, model building module, learning rate adjustment module, model training module;

[0055] The acquisition module is used to obtain the real-time demand for electricity and heat, the operating status of each energy device, the real-time electricity price, and the control instructions of each energy device;

[0056] A model building module is used to use the acquired data to build a demand-side resource regulation model based on a statistical model. The demand-side resource regulation model includes a demand-side resource regulation environment, a state space, an action space, and a reward function.

[0057] The learning rate adjustment module is used to establish a learning rate decay trigger model and a learning rate decay model using a reward function;

[0058] The model training module is used to adjust the learning rate of the demand-side resource regulation model based on the learning rate decay trigger model and the learning rate decay model during the training process of the demand-side resource regulation model; and use the trained demand-side resource regulation model to obtain demand-side resource regulation instructions.

[0059] Preferably, the model training module includes: a model initialization unit, a network model building unit, a model evaluation unit, and a model training unit;

[0060] The model initialization unit is used to initialize the demand-side resource control environment according to the operating status and operating parameters of each energy device; and initialize the state space and action space according to the initialized demand-side resource control environment and the control instructions of the energy device;

[0061] A network model building unit, used to build a strategy network model and a value network model;

[0062] A model evaluation unit, used to establish control parameters and loss functions for training the demand-side resource regulation model using a proximal policy optimization method;

[0063] The model training unit is used to perform demand-side resource regulation model training.

[0064] The beneficial effects of the present invention are that, compared with the prior art, at least the following are included: solving the problems of low optimization efficiency and long time caused by the large scale of the demand-side resource optimization and control model through adaptive learning rate decay, which can improve the optimization decision-making efficiency of the demand-side resource optimization and control; tracking the optimal performance of the demand-side resource optimization and control model training by constructing a performance monitor, decay trigger conditions and decay control model, and deciding whether to adjust the learning rate according to the model performance change; when the current performance of the training does not reach a set threshold for the optimal performance improvement, and the number of steps without performance improvement has reached a certain number of evaluation cycles, the learning rate will be decayed. Through this adaptive learning rate decay strategy, the appropriate learning rate can be adjusted according to the current performance of the algorithm training to avoid the learning rate being too large or too small, thereby improving the convergence speed and stability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1It is a flow chart of an adaptive learning rate decay optimization method for demand-side resource regulation proposed by the present invention;

[0066] Figure 2 It is a schematic diagram of the physical structure of the demand-side resource optimization and regulation environment in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only embodiments of a part of the present invention, rather than all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0068] The present invention proposes a demand-side resource control method based on adaptive learning rate decay, such as Figure 1 As shown, including:

[0069] Step 1: Obtain the real-time demand for electricity and heat, the operating status of each energy device, the real-time electricity price, and the control instructions of each energy device.

[0070] Step 2: Use the acquired data to establish a demand-side resource regulation model based on a statistical model.

[0071] For demand-side resource regulation, the operational optimization problem faced is to ensure that all equipment operates within its operating limits, guarantee supply and demand balance, and on this basis make the system operate at the lowest cost. Based on mathematical methods, the objective function of the demand-side resource regulation model is as follows:

[0072]

[0073] In the formula, C is the total cost of system operation, c e (p e ,q e ) produces electricity p for the e-th energy device e and thermal q e The operating cost, n e The total number of devices.

[0074] The constraints of the demand-side resource regulation model include but are not limited to: the operating boundary constraints of power supply and heat supply of various energy equipment, and the system supply and demand balance constraints, as follows:

[0075] p e,min ≤p e ≤p e,max

[0076] q e,min ≤qe ≤q e,max

[0077]

[0078]

[0079] In the formula, p e,min 、p e,max The power generated by the e-th energy device is p e The lower and upper limits of q e,min ,q e,max The heat q produced by the e-th energy device is e The lower and upper limits of p d 、h d are the total electrical load and thermal load of the system respectively.

[0080] In a non-limiting preferred embodiment, based on the Markov model, the demand-side resource regulation model expressed in algebraic form is converted into a demand-side resource regulation statistical model. The demand-side resource regulation statistical model includes: a demand-side resource optimization regulation environment, a state space, an action space, and a reward function.

[0081] Specifically, step 2 includes:

[0082] Step 2.1, establish a demand-side resource control environment based on the operating status of each energy device;

[0083] In a non-limiting preferred embodiment, a typical demand-side resource optimization and control environment is created, which includes a wind turbine, a gas turbine (GT), a gas source (gas), a gas boiler (GB), a thermal storage tank (TST) and a grid (Grid), and its physical structure is as follows: Figure 2 shown.

[0084] Step 2.2, construct the state space S based on the real-time demand for electricity and heat, the operating status of each energy device, and the real-time electricity price;

[0085] In a non-limiting preferred embodiment, the state space S is a 14-dimensional vector space, including but not limited to: real-time demand for electricity and heat, the operating status of each energy device, and real-time electricity prices. The dimensions of the state space are adjusted according to factors such as the type of demand-side resources and the control objectives.

[0086] The state space satisfies the following relationship:

[0087] S=[i point ,t p ,tq ,s p ,s q ,p gt ,q gt ,q gb ,q tst ,p grid ,p wind ,o tst ,h tst ,ρ rtp ]

[0088] In the formula, i point It is a sign indicating whether the goal has been achieved, 1 means achieved, 0 means not achieved; t p is the difference between the total power generation and the power target; t q is the difference between the total heat generation and the heat target; s p 、s q are the distances between electricity and heat and the central point of supply and demand balance; p gt ,q gt are the electricity and heat generated by the gas turbine respectively; q gb The heat generated by the gas boiler; tst The heat generated by the thermal storage tank; grid Electricity generated for the grid; p wind Electricity generated by the wind turbine; tst is the operating status of the hot storage tank; h tst The heat stored in the thermal storage tank; ρ rtp Real-time electricity price.

[0089] In a non-limiting preferred embodiment, the state space is normalized, including: collecting data corresponding to state space elements in the demand-side resource control environment; calculating the mean and standard deviation of each state space element data; and applying a standardization method to convert the state space element data into a distribution with a mean of 0 and a standard deviation of 1. By normalizing the state space, it is ensured that the distribution of input data is conducive to model learning, which helps to narrow the range of eigenvalues, avoid numerical instability caused by excessive differences between eigenvalues, improve the stability of algorithm training, and accelerate the training process.

[0090] Step 2.3, construct the action space A with the control instructions of the energy equipment;

[0091] In a non-limiting preferred embodiment, the action space A is a 4-dimensional vector space, including but not limited to: the output power of the gas turbine, the output heat of the gas boiler, the output heat of the heat storage tank and the power of the grid; the dimension of the action space is adjusted according to factors such as the type of demand-side resources and the regulation objectives.

[0092] The action space satisfies the following relationship:

[0093] A=[a gb ,a gt ,a tst ,a grid ]

[0094] In the formula, a gb 、a gt 、a tst 、a grid They are the output power of the gas turbine, the output heat of the gas boiler, the output heat of the heat storage tank and the power of the power grid; and the control instructions of each energy device have corresponding operating limits.

[0095] Step 2.4, construct the reward function R based on the state space and action space;

[0096] The reward function is a function of the state space elements and the action space elements, satisfying the following relationship:

[0097] R(s,a)=-CF d -F tst +ΔR

[0098] Where R(s,a) is the reward function for state space elements and action space elements; C is the total cost of system operation; F d F is the penalty term for mismatch between electricity and heat supply and demand; tst is the penalty for overcharging or over-discharging the heat storage tank to ensure that the stored heat is within a safer range; ΔR is the additional reward for achieving a balance between electricity and heat supply and demand, especially when the system reaches or approaches its target state, which helps the optimization algorithm converge to a good strategy faster;

[0099] Among them, the penalty term for mismatch between electricity and heat supply and demand satisfies the following relationship:

[0100]

[0101] In the formula, t p is the difference between the total power generation and the power target; t q is the difference between the total heat generation and the heat target;

[0102] The penalty term for overcharge and over discharge of the thermal storage tank satisfies the following relationship:

[0103] F tst =0.1*|o tst -o ini |

[0104] In the formula, o tst is the operating status of the hot storage tank; o iniis the initial state of the hot storage tank;

[0105] The additional bonus for achieving a balance between electricity and heat supply and demand satisfies the following relationship:

[0106]

[0107] In the formula, F m is the set distance threshold, when F d ≤F m When G c is a counter used to track the duration of the system reaching or approaching the target; G m is the set time threshold, when G c ≥G m , the system is considered to have stably reached its goal.

[0108] Step 2.5, construct a demand-side resource regulation model based on the demand-side resource regulation environment, state space, action space and reward function.

[0109] The demand-side resource regulation model proposed in the present invention is a statistical model established according to statistical methods. The difference from the algebraic model is that the volatility and uncertainty of the output of each energy device have been optimized and concentrated by statistical means, thereby avoiding the influence of loose data on the overall regulation efficiency and reliability. The subsequent learning rate optimization is also carried out on the basis of the statistical model. The statistical model eliminates the interference of loose data, can highlight the influence of the learning rate on the calculation efficiency of the model, and thus more accurately determine whether to trigger the learning rate decay. Moreover, the reward function in the statistical model becomes an important parameter for whether to trigger the learning rate decay and how the learning rate decays.

[0110] Step 3: Use the reward function of the demand-side resource regulation model to establish a learning rate decay trigger model and a learning rate decay model.

[0111] In the demand-side resource regulation statistical model, the reward function is related to the state space elements and the action space elements. The reward function characterizes the dynamics and uncertainty of the output of each energy device. Therefore, the learning rate of the demand-side resource regulation statistical model is adjusted based on the optimization result of the reward function, so that the learning rate can be adaptively attenuated according to the dynamics and uncertainty of the output of each energy device, thereby improving the efficiency and reliability of the demand-side resource regulation statistical model.

[0112] Specifically, step 3 includes:

[0113] Step 3.1, obtain the historical optimal value of the reward function;

[0114] In a non-limiting preferred embodiment, a performance monitor is established to obtain the historical optimal value of the reward function, satisfying the following relationship:

[0115] R * (t) = max[R * (t-1),R(t)]

[0116] In the formula, R * (t) is the optimal value of the reward function at time t, and R(t) is the value of the reward function at time t.

[0117] Step 3.2: Within a learning rate adjustment cycle, if the difference between the reward function value and the optimal value is always less than the set threshold, the learning rate decay is triggered and the process goes to step 3.3. On the contrary, if the difference between the reward function value and the optimal value is not less than the set threshold, or if the difference between the reward function value and the optimal value is less than the set threshold for less than one learning rate adjustment cycle, the learning rate decay is not triggered and the process goes back to step 3.1.

[0118] In a non-limiting preferred embodiment, the learning rate decay trigger model satisfies the following relationship:

[0119]

[0120] Where D(t) is the trigger signal of learning rate decay at time t, 1 means triggering learning rate decay, and 0 means not triggering learning rate decay; ΔR(t) is the difference between the value of the reward function at time t and the optimal value; θ is the set threshold coefficient, R * (t)·θ represents the set threshold; u(t) is the timing duration when the difference is less than the set threshold; patience is a learning rate adjustment cycle.

[0121] Step 3.3, taking the optimal value of the additional reward as the objective function, determine the decay rate of the learning rate, and establish a learning rate decay control model;

[0122] Since the additional reward reflects the balance between electricity and heat supply and demand under the volatility and uncertainty of the output of each energy device, the larger the value of the additional reward, the more it can represent that the result of demand-side resource regulation is to meet the balance between electricity and heat supply and demand under the volatility and uncertainty of the output of each energy device. Therefore, the optimal value of the additional reward is taken as the objective function, and the machine self-learning algorithm is used to obtain the attenuation rate λ of the learning rate.

[0123] When the learning rate decay is triggered, the learning rate decay model satisfies the following relationship:

[0124]

[0125] Where η(t) is the learning rate at time t; λ is the decay rate, which is a positive number less than 1.

[0126] When the learning rate decay is triggered, the learning rate decreases, that is, the learning step size is reduced.

[0127] Since the attenuation rate is determined under the condition of optimal additional reward value, the demand-side resource regulation scheme obtained by the model after adjusting the learning rate effectively ensures the balance of supply and demand of electricity and heat while meeting the volatility and uncertainty of the output of each energy equipment.

[0128] Step 4: During the training of the demand-side resource regulation model, the learning rate of the demand-side resource regulation model is adjusted based on the learning rate decay trigger model and the learning rate decay model; and the demand-side resource regulation instructions are obtained using the trained demand-side resource regulation model.

[0129] In a non-limiting preferred embodiment, a proximal strategy optimization method is used to train a demand-side resource regulation model, including:

[0130] 1) Initialize the demand-side resource control environment according to the operating status and operating parameters of each energy device;

[0131] 2) Initialize the state space and action space according to the initialized demand-side resource control environment and the control instructions of the energy equipment; in a non-limiting preferred embodiment, the initialization of the action space includes but is not limited to: determining that the dimension of the action space is 4, the elements of the action space correspond to the output power of the gas turbine, the output heat of the gas boiler, the output heat of the heat storage tank and the power of the power grid, and the control instructions of each energy device have corresponding operating limits. In a non-limiting preferred embodiment, the initialization of the state space includes but is not limited to: determining that the dimension of the state space is 14, the elements of the state space correspond to the real-time demand for electricity and heat, the operating status of each energy device, and the real-time electricity price.

[0132] 3) Establish a strategic network model;

[0133] The policy network model is used to generate the distribution of actions that should be performed in the current state. The policy network model is established, including but not limited to the input layer, hidden layer and output layer; the dimension of the input layer is the same as the dimension of the state space, that is, 14; the dimension of the output layer is the same as the dimension of the action space, that is, 4.

[0134] 4) Establish a value network model;

[0135] The value network model is used to estimate the value of the current state. The value network model is established, including but not limited to the input layer, hidden layer and output layer; the dimension of the input layer is the same as the dimension of the state space, that is, 14; the dimension of the output layer is 1, which represents the value of the current state.

[0136] When the proximal policy optimization method is used to train the demand-side resource regulation model, the established policy network model and value network model are matched with the initialized demand-side resource regulation environment, state space and action space, ensuring that the proximal policy optimization method is suitable for training the demand-side resource regulation model, so as to improve the efficiency and accuracy of the demand-side resource regulation model training.

[0137] 5) Setting control parameters for training the demand-side resource regulation model using the proximal strategy optimization method;

[0138] In a non-limiting preferred embodiment, the control parameters for training the demand-side resource regulation model using the proximal policy optimization method are used to control the convergence and stability of the demand-side resource regulation model, and the control parameters include but are not limited to: learning rate, discount factor, clipping coefficient, number of update steps, etc. Among them, the learning rate is used to control the step size of each parameter update in the demand-side resource regulation model. Although the learning rate is an internal parameter of the demand-side resource regulation model, it is introduced into the control parameters for training the demand-side resource regulation model using the proximal policy optimization method to achieve the associated control of the demand-side resource regulation model itself and the training process of the proximal policy optimization method adopted; the discount factor is the weight for balancing the demand-side resource regulation instruction and the predicted value of the demand-side resource regulation instruction; the clipping coefficient is used to limit the amplitude of the policy network model update; the number of update steps is used to balance the number of updates of the policy network model and the value network model in each round of demand-side resource regulation model training.

[0139] Under the condition that the proximal policy optimization method is applicable to training the demand-side resource regulation model, the control parameters for training the demand-side resource regulation model using the proximal policy optimization method are further set. After adjusting the learning rate of the demand-side resource regulation model based on the learning rate decay trigger model and the learning rate decay model, the demand-side resource regulation model outputs the demand-side resource regulation instruction, and uses the discount factor to correct the demand-side resource regulation instruction to balance the demand-side resource regulation instruction and the predicted value of the demand-side resource regulation instruction. At the same time, the clipping coefficient and the number of update steps are determined based on the adjusted learning rate. Therefore, in the process of training the demand-side resource regulation model using the proximal policy optimization method, the adaptive adjustment of the learning rate of the demand-side resource regulation model is realized, which can not only balance the efficiency and stability from the internal perspective of the demand-side resource regulation model, but also optimize the training process and improve the training efficiency from the external perspective of training the demand-side resource regulation model using the proximal policy optimization method.

[0140] 6) Establish a loss function for training the demand-side resource regulation model using the proximal strategy optimization method;

[0141] In a non-limiting preferred embodiment, the loss function for training the demand-side resource regulation model using the proximal policy optimization method includes but is not limited to: a policy network model loss sub-function, a value network model loss sub-function, and an entropy regularization term sub-function. Among them, the policy network model loss sub-function is used to measure the difference between the updated policy network model and the policy network model before the update. The smaller the policy network model loss sub-function, the better the stability of the policy network model update process; the value network model loss sub-function is used to evaluate the prediction error of the value network model. The smaller the value network model loss sub-function, the higher the value of the current state predicted by the value network model; the entropy regularization term sub-function is used to prevent the demand-side resource regulation model from converging to a suboptimal solution too early.

[0142] After setting the control parameters for training the demand-side resource regulation model using the proximal policy optimization method, the loss function for training the demand-side resource regulation model using the proximal policy optimization method is used to simultaneously evaluate the loss of the demand-side resource regulation model training process using the proximal policy optimization method and the loss of the demand-side resource regulation model, and the various loss amounts are fed back to the corresponding control parameters. The control parameters are used to adjust the training process of the demand-side resource regulation model using the proximal policy optimization method, thereby achieving closed-loop control of the efficiency and stability of the demand-side resource regulation model.

[0143] 7) Execute demand-side resource regulation model training;

[0144] The appropriate learning rate is adjusted according to the current performance of the demand-side resource regulation model training to avoid the learning rate being too large or too small, thereby improving the convergence speed and stability of the demand-side resource regulation model.

[0145] Specifically, the demand-side resource regulation model training includes multiple rounds of training; in each round of training, the demand-side resource regulation model is used to obtain a reward function according to the demand-side resource regulation environment, state space, and action space; at the same time, the policy network model and the value network model are used to calculate the loss function of the demand-side resource regulation model trained by the proximal policy optimization method, and the current performance of the trained demand-side resource regulation model is evaluated based on the loss function. According to the evaluation results, the control parameters of the demand-side resource regulation model trained by the proximal policy optimization method are adjusted, including adjusting the learning rate of the demand-side resource regulation model based on the learning rate decay trigger model and the learning rate decay model; the policy network model and the value network model are optimized using the adjusted control parameters. After multiple rounds of training, a trained demand-side resource regulation model is obtained.

[0146] In addition, executing demand-side resource regulation model training also includes but is not limited to: log recording and model preservation; wherein log recording is used to record key indicators in the training process, such as total reward, cost, etc.; model preservation is used to save the parameters of the strategy network model and the value network model when the total number of training times is reached.

[0147] In the embodiment of the present invention, a random scene is used to generate p in Table 1. grid ,q gb 、p wind 、h tst , rtp , as input data, and then implement the proximal strategy optimization algorithm based on adaptive learning rate decay.

[0148] In the Python environment, the same algorithm framework is built using PyTorch, and three demand-side resource optimization and control algorithms are set up for comparative analysis: the proximal policy optimization algorithm without learning rate decay strategy (PPO-NDS), the proximal policy optimization algorithm with fixed learning rate decay strategy (PPO-FDS), and the proximal policy optimization algorithm with adaptive learning rate decay strategy (PPO-ADS). The results are shown in Table 2. The effectiveness of the proposed algorithm is verified by comparing the performance of different algorithms on the training set and the test set.

[0149] Table 1 Input data

[0150] parameter <![CDATA[p grid (kW)]]> <![CDATA[q gb (kW)]]> <![CDATA[p wind (kW)]]> <![CDATA[h tst (kW)]]> <![CDATA[ρ rtp ($ / kWh)]]> Minimum 1500 5000 200 1000 0.4 maximum 7000 11000 2000 3000 0.7

[0151] Table 2 Comparison of results of three demand-side resource optimization and control methods

[0152]

[0153] Each algorithm is run with the same 5 random seeds to eliminate the randomness of the code implementation part during the simulation. The reward function R(s,a), the total cost of system operation C, and the penalty term F for the mismatch between electricity and heat supply and demand are combined into a single function. d as an evaluation indicator.

[0154] According to the simulation data analysis of the embodiment, the learning rate of PPO-ADS adaptive adjustment decreases with the increase of training steps, and is slightly adjusted during the training process. The learning rate of PPO-FDS decreases more evenly and stably, but it begins to decline continuously in the early stage of training, and is close to 0 in the middle and late stages, and fails to make adaptive adjustments in a specific period. The learning rate of PPO-NDS remains unchanged, which may lead to reduced efficiency of later training.

[0155] PPO-ADS performed best, with the fastest increase in reward value and maintained the highest reward level in the middle and late stages, and maintained high stability during training. This shows that it can better explore the environment and quickly learn useful strategies. The reward value of PPO-NDS grew second fastest, and the reward value of PPO-FDS grew the slowest and fluctuated more, which may indicate that the learning rate continued to decay from the beginning of training instead of being reduced in time, resulting in an unstable learning process.

[0156] PPO-NDS performs poorly in terms of rewards, with a median of -1.37, indicating that it is difficult for the model to obtain higher rewards when learning rate decay is not adopted. In contrast, PPO-ADS shows the best performance, with a median of -0.73, indicating that the adaptive decay strategy helps the model obtain higher rewards during testing. In addition, the performance of PPO-FDS is between the two, with a median of -1.01.

[0157] Table 3 summarizes the confidence intervals for each test metric. As can be seen from the breadth of the confidence intervals, PPO-ADS exhibits minimal volatility on all metrics, indicating better stability and reliability during training. Although PPO-NDS performs well on cost, its performance on reward and supply-demand balance is insufficient, suggesting a trade-off between different performance metrics. PPO-FDS provides an intermediate solution that exhibits balanced performance.

[0158] In summary, the test results show that the learning rate decay strategy has a significant impact on model performance. In particular, PPO-ADS shows better overall performance on the test set, providing high rewards and low supply-demand imbalance while maintaining low volatility.

[0159] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0160] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0161] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0162] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A demand-side resource control method based on adaptive learning rate decay, wherein a power grid includes multiple energy devices to generate electricity and heat, characterized in that: include: Obtain the real-time demand for electricity and heat, the operating status of each energy device, the real-time electricity price, and the control instructions of each energy device; Using the acquired data, a demand-side resource regulation model is established based on a statistical model, including: establishing a demand-side resource regulation environment according to the operating status of each energy device; constructing a state space based on the real-time demand for electricity and heat, the operating status of each energy device, and the real-time electricity price; constructing an action space based on the regulation instructions of the energy device; constructing a reward function based on the state space and the action space; and constructing a demand-side resource regulation statistical model based on the demand-side resource regulation environment, state space, action space, and reward function; The learning rate decay trigger model and the learning rate decay model are established by using the reward function; including: obtaining the historical optimal value of the reward function; within a learning rate adjustment cycle, if the difference between the value of the reward function and the optimal value is always less than the set threshold, the learning rate decay is triggered; conversely, if the difference between the value of the reward function and the optimal value is not less than the set threshold, or if the difference between the value of the reward function and the optimal value is less than the set threshold for less than one learning rate adjustment cycle, the learning rate decay is not triggered; taking the optimal additional reward value as the objective function, determining the decay rate of the learning rate, and establishing a learning rate decay control model; when the learning rate decay is triggered, the learning rate decay model satisfies the following relationship: Where η(t) is the learning rate at time t; λ is the decay rate, which is a positive number less than 1; D(t) is the learning rate decay trigger signal at time t, 1 indicates that the learning rate decay is triggered, and 0 indicates that the learning rate decay is not triggered; During the training process of the demand-side resource regulation model, the learning rate of the demand-side resource regulation model is adjusted based on the learning rate decay trigger model and the learning rate decay model; and the demand-side resource regulation instructions are obtained using the trained demand-side resource regulation model.

2. The demand-side resource control method based on adaptive learning rate decay according to claim 1 is characterized in that: The state space S satisfies the following relationship: S=[i point ,t p ,t q ,s p ,s q ,p gt ,q gt ,q gb ,q tst ,p grid ,p wind ,o tst ,h tst ,ρ rtp ] In the formula, i point It is a sign indicating whether the goal has been achieved, 1 means achieved, 0 means not achieved; t p is the difference between the total electricity generated and the electricity target; t q is the difference between the total heat generation and the heat target; s p 、s q are the distances between electricity and heat and the central point of supply and demand balance; p gt ,q gt are the electricity and heat generated by the gas turbine respectively; q gb The heat generated by the gas boiler; tst The heat generated by the thermal storage tank; grid Electricity generated for the grid; p wind Electricity generated by the wind turbine; tst is the operating status of the hot storage tank; h tst The heat stored in the thermal storage tank; ρ rtp Real-time electricity price.

3. The demand-side resource control method based on adaptive learning rate decay according to claim 2 is characterized in that: The action space A satisfies the following relationship: A=[a gb ,a gt ,a tst ,a grid ] In the formula, a gb 、a gt 、a tst 、a grid They are the output power of the gas turbine, the output heat of the gas boiler, the output heat of the heat storage tank and the power of the power grid; and the control instructions of each energy device have corresponding operating limits.

4. The demand-side resource control method based on adaptive learning rate decay according to claim 3 is characterized in that: The reward function is a function of the state space elements and the action space elements, satisfying the following relationship: R(s,a)=-C-F d -F tst +ΔR Where R(s,a) is the reward function for state space elements and action space elements; C is the total cost of system operation; F d F is the penalty term for mismatch between electricity and heat supply and demand; tst is the penalty for overcharging or over-discharging of the thermal storage tank; ΔR is the additional reward for achieving a balance between electricity and heat supply and demand.

5. The demand-side resource control method based on adaptive learning rate decay according to claim 4 is characterized in that: The penalty term for mismatch between electricity and heat supply and demand satisfies the following relationship: The penalty term for overcharge and over discharge of the thermal storage tank satisfies the following relationship: F tst =0.1*|o tst -o ini In the formula, o ini is the initial state of the hot storage tank; The additional bonus for achieving a balance between electricity and heat supply and demand satisfies the following relationship: In the formula, F m is the set distance threshold, when F d ≤F m When G c is a counter used to track the duration of time the system reaches or approaches the target; G m is the set time threshold, when G c ≥G m , the system is considered to have stably reached its goal.

6. The demand-side resource control method based on adaptive learning rate decay according to claim 1 is characterized in that: Get the historical optimal value of the reward function, satisfying the following relationship: R * (t)=max[R * (t-1),R(t)] In the formula, R * (t) is the optimal value of the reward function at time t, and R(t) is the value of the reward function at time t.

7. The demand-side resource control method based on adaptive learning rate decay according to claim 6 is characterized in that: The learning rate decay trigger model satisfies the following relationship: Where D(t) is the learning rate decay trigger signal at time t, 1 means that the learning rate decay is triggered, and 0 means that the learning rate decay is not triggered; ΔR(t) is the difference between the value of the reward function at time t and the optimal value; θ is the set threshold coefficient, R * (t)·θ represents the set threshold; u(t) is the timing duration when the difference is less than the set threshold; patience is a learning rate adjustment cycle.

8. The demand-side resource control method based on adaptive learning rate decay according to claim 1 is characterized in that: The demand-side resource regulation model is trained using the proximal strategy optimization method. The training process includes: Initialize the demand-side resource control environment according to the operating status and operating parameters of each energy device; Initialize the state space and action space according to the initialized demand-side resource control environment and the control instructions of the energy equipment; Establish strategic network model and value network model; Establish control parameters and loss functions for training demand-side resource regulation models using proximal policy optimization methods; Perform demand-side resource regulation model training.

9. The demand-side resource control method based on adaptive learning rate decay according to claim 8 is characterized in that: Execute demand-side resource regulation model training, multiple rounds of training; in each round of training, use the demand-side resource regulation model to obtain a reward function according to the demand-side resource regulation environment, state space and action space; at the same time, use the policy network model and the value network model to calculate the loss function of training the demand-side resource regulation model using the proximal policy optimization method, evaluate the current performance of the trained demand-side resource regulation model based on the loss function, and adjust the control parameters of the demand-side resource regulation model using the proximal policy optimization method according to the evaluation results, including adjusting the learning rate of the demand-side resource regulation model based on the learning rate decay trigger model and the learning rate decay model; The strategy network model and the value network model are optimized using the adjusted control parameters; after multiple rounds of training, a trained demand-side resource regulation model is obtained.

10. A demand-side resource control system based on adaptive learning rate decay, characterized in that: include: Acquisition module, model building module, learning rate adjustment module, model training module; The acquisition module is used to obtain the real-time demand for electricity and heat, the operating status of each energy device, the real-time electricity price, and the control instructions of each energy device; The model building module is used to use the acquired data to establish a demand-side resource control model based on a statistical model, including: establishing a demand-side resource control environment according to the operating status of each energy device; constructing a state space based on the real-time demand for electricity and heat, the operating status of each energy device and the real-time electricity price; constructing an action space based on the control instructions of the energy device; constructing a reward function based on the state space and the action space; and forming a demand-side resource control statistical model based on the demand-side resource control environment, the state space, the action space and the reward function; The learning rate adjustment module is used to establish a learning rate decay trigger model and a learning rate decay model using a reward function; including: obtaining the historical optimal value of the reward function; within a learning rate adjustment cycle, if the difference between the value of the reward function and the optimal value is always less than the set threshold, the learning rate decay is triggered; conversely, if the difference between the value of the reward function and the optimal value is not less than the set threshold, or if the difference between the value of the reward function and the optimal value is less than the set threshold for less than one learning rate adjustment cycle, the learning rate decay is not triggered; taking the optimal additional reward value as the objective function, determining the decay rate of the learning rate, and establishing a learning rate decay control model; when the learning rate decay is triggered, the learning rate decay model satisfies the following relationship: Where η(t) is the learning rate at time t; λ is the decay rate, which is a positive number less than 1; D(t) is the learning rate decay trigger signal at time t, 1 indicates that the learning rate decay is triggered, and 0 indicates that the learning rate decay is not triggered; The model training module is used to adjust the learning rate of the demand-side resource regulation model based on the learning rate decay trigger model and the learning rate decay model during the training process of the demand-side resource regulation model; and use the trained demand-side resource regulation model to obtain demand-side resource regulation instructions.

11. The demand-side resource control system based on adaptive learning rate decay according to claim 10, characterized in that: The model training module includes: model initialization unit, network model building unit, model evaluation unit, and model training unit; The model initialization unit is used to initialize the demand-side resource control environment according to the operating status and operating parameters of each energy device; and initialize the state space and action space according to the initialized demand-side resource control environment and the control instructions of the energy device; A network model building unit, used to build a strategy network model and a value network model; A model evaluation unit, used to establish control parameters and loss functions for training the demand-side resource regulation model using a proximal policy optimization method; The model training unit is used to perform demand-side resource regulation model training.

Images