Household electrical load optimization scheduling method and system based on MAPPO algorithm

Abstract

The invention discloses a household electrical load optimization scheduling method and system based on an MAPPO algorithm. The method comprises the following steps: (1) collecting multi-source heterogeneous data from a user side and an external environment and carrying out preprocessing; (2) estimating the future equipment use probability of the user by collecting historical equipment use data of the user, and outputting an equipment behavior prediction vector; (3) acquiring real-time electricity price information, performing short-term electricity price trend prediction by using the time sequence prediction model based on a historical electricity price sequence, and outputting an electricity price prediction sequence; (4) generating a scheduling strategy by the MAPPO network based on the equipment behavior prediction vector and the electricity price prediction sequence, and outputting a corresponding scheduling result; and (5) sending a scheduling result to the home gateway, controlling the equipment, collecting an execution feedback result in real time, updating internal parameters of the MAPPO network, and realizing strategy iteration and adaptive adjustment of the MAPPO network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of artificial intelligence, power systems and demand response management, and particularly relates to a household electricity load optimization scheduling method and system based on a MAPPO algorithm. BACKGROUND

[0002] With the continuous increase of renewable energy penetration, the proportion of new energy power generation resources such as wind power and photovoltaic power in the power system is increasing, and the system load presents increasingly significant volatility and uncertainty. This change puts higher requirements on the traditional source-side regulation method, and also promotes the transformation of the power system from "source following load" to "source-load interaction". As an important means to respond to this trend, demand side response (DR) has gradually become a key technical path to ensure the stability of the power grid, improve energy utilization efficiency and promote clean energy consumption.

[0003] Among the demand side resources, household electrical appliances, especially devices with certain thermal energy storage characteristics, such as air conditioners, water heaters, and electric heating film heating systems, are widely considered to be the most potential flexible load. Such devices not only have a large number and are widely distributed, but also have certain adjustment margin and time shifting capability, and can participate in virtual power plant operation, load peak clipping and valley filling, and auxiliary services in various application scenarios while ensuring user comfort. However, the existing scheduling control strategy of household electrical appliances still faces several significant deficiencies, which restricts the play of its flexibility and large-scale regulation value. For example, an intelligent household electrical appliance electricity flexible load optimization scheduling method and system (CN116316654A) proposes an optimization scheduling method based on photovoltaic power generation prediction, time-of-use price information, and household appliance load prediction. The method solves the running time of the electrical appliance by constructing a multi-objective optimization model (minimum electricity bill, minimum peak-to-average ratio, and minimum user dissatisfaction) and using the Gurobi optimization algorithm to realize the electricity flexible scheduling of intelligent household electrical appliances. Its design takes into account the economy and user comfort, and to some extent improves the intelligent level of household energy use. However, in this method, the household appliance scheduling decision mainly depends on static prediction and centralized modeling, which has the following deficiencies:

[0004] Firstly, the user behavior modeling is rough, only estimated by the preset time window and power demand, lacking deep modeling of the actual use habits of users, and unable to adapt to the dynamic changes of user behavior; secondly, the real-time price trend is not predicted in this scheme, only relying on the known time-of-use price information, which is difficult to cope with the market environment with frequent price fluctuations; thirdly, the optimization scheduling process completely depends on traditional solvers such as Gurobi, lacking strategy iteration and adaptive ability, and having poor scalability when facing different users or environmental changes;

[0005] The existing household power load optimization scheduling method, system, device and readable storage medium (CN112859615B) proposes a method based on user equipment use preference and time-of-use price information, which comprehensively considers user power consumption habits and energy saving goals, and optimizes the scheduling of controllable appliances (such as washing machines, electric water heaters, etc.) in the home. The method builds a multi-objective optimization function (electricity expenditure, device comfort, etc.), uses a weighted strategy and an integer programming model to solve the scheduling strategy, aiming to improve the energy efficiency of the home and reduce the peak load. Although this scheme makes some improvements in integrating user preferences and economic goals, there are still the following shortcomings in the specific implementation:

[0006] On the one hand, the method mainly uses preset weights and static parameters to build a user preference model, which cannot reflect the dynamic changes and individual differences of user behavior, and lacks real-time adaptability. On the other hand, the optimization strategy relies on traditional integer programming or heuristic algorithms, which can be solved in small-scale home scenarios, but when faced with a large number of device state combinations or environmental factor changes, the computational complexity is high, the solving efficiency is low, and it is difficult to meet the real-time requirements. In addition, the system does not introduce an external price prediction mechanism, but relies only on the pre-set time-of-use price scheme, which cannot respond to market price fluctuations or demand response signals. Finally, the scheme lacks a closed-loop feedback mechanism for scheduling execution, and cannot continuously optimize the scheduling strategy according to the actual device operation or user feedback, making it difficult to ensure the stability of the system and user satisfaction in the long run. SUMMARY

[0007] The present application aims to propose a household power load optimization scheduling method based on Multi-Agent Proximal Policy Optimization (MAPPO) algorithm, which realizes the intelligent, efficient and economical management of household power system for temperature-controlled loads and transferable loads. The method considers the individual behavior characteristics of users, the real-time price change trend of the power grid and the operation characteristics of devices, and realizes multi-objective control through the introduction of a prediction model and an intelligent scheduling algorithm.

[0008] The present application is realized by at least one of the following technical solutions.

[0009] A household power load optimization scheduling method based on MAPPO algorithm, comprising the following steps:

[0010] (1) Collecting and preprocessing multi-source heterogeneous data from the user side and the external environment;

[0011] (2) Based on the preprocessed data, using a device start probability distribution model to estimate the probability of future use of the device by the user, and outputting the device start probability;

[0012] (3) using a time series prediction model based on a historical electricity price sequence to perform short-term electricity price trend prediction, and outputting an electricity price prediction result;

[0013] (4) based on the device start probability and the electricity price prediction result, using deep reinforcement learning to realize iteration and adaptive adjustment, and outputting a scheduling result;

[0014] (5) sending the scheduling result to a home gateway to control the devices, and collecting real-time execution feedback results to update the user behavior model and the reinforcement learning state.

[0015] Further, the preprocessing includes unifying the data formats of the household appliance operating state information, the user behavior feature sequence, the electricity price sequence, the environmental context, the user comfort constraint vector, and the power grid load sequence, and processing the data.

[0016] Further, in step (2), the device start probability distribution model includes an input layer, a long short-term memory network (LSTM) layer, and a feedforward neural network layer. In the input layer, the input data is aligned by time step, and at each time step, the appliance state, behavior feature, and environmental context are concatenated to form a unified time series input sequence. Then, the time series input sequence is input into the multi-layer stacked long short-term memory network (LSTM) layer, which captures the long-term dependencies and periodic trends in historical behavior through the recursion and update of the hidden state. Finally, based on the final time hidden state output by the long short-term memory network (LSTM), the use probability prediction of each type of controllable device in the future scheduling period is generated through the multi-layer feedforward neural network layer, and the final output is a device behavior prediction vector.

[0017] Further, in step (3), the time series prediction model includes an input layer, a TCN convolutional encoding layer, a self-attention enhancement layer, and a prediction output layer. The input of the input layer is the past electricity price history sequence and external auxiliary features such as temperature and power grid load. Then, through causal convolution in the TCN convolutional encoding layer, the price change patterns at different time scales are extracted. The TCN convolutional encoding layer also introduces a dilated convolution to effectively model long-range dependencies by setting an increasing dilated rate layer by layer. Based on the output of the TCN convolutional encoding layer, a self-attention enhancement layer is introduced, and finally, the output of the self-attention enhancement layer is fed into several fully connected layers to output the future electricity price prediction sequence.

[0018] Further, in step (4), the deep reinforcement learning is based on a multi-agent proximal policy optimization algorithm, which includes:

[0019] (1) constructing a state vector, which includes user appliance operating state, electricity price prediction sequence, user behavior prediction, comfort constraint, environmental context information, and power grid historical load;

[0020] (2) Define the action space, which is a set of optional control instructions for each household appliance. The user's joint action is composed of the actions of all his / her devices.

[0021] Furthermore, a dynamic action masking mechanism is introduced into deep reinforcement learning, including:

[0022] (1) Collect equipment operating status, user behavior prediction, time period attributes and power grid constraint information;

[0023] (2) Generate action masking rules based on the collected information and form an action mask to distinguish between executable actions and actions that need to be masked;

[0024] Action masking rules include: constructing an action validity determination function based on state information. when:

[0025]

[0026] Among them, H t It represents the system's state information at time t, including factors such as user status, electricity price, and environmental context; This represents the k-th control option for device d;

[0027] Generate action mask and policy constraints: Generate action mask tensors based on the decision function results of all devices;

[0028] (3) During the sampling and execution process, selection is only allowed from valid actions;

[0029] (4) During the training phase, the action mask is used as an auxiliary input to the policy network to improve the learning effect of actionable actions.

[0030] Furthermore, the reward function of deep reinforcement learning includes a weighted combination of a primary reward and several auxiliary rewards. The primary reward includes electricity cost, load balancing, and user comfort. The auxiliary rewards are shaping terms used to improve training efficiency, and their weights are dynamically adjusted according to the policy performance.

[0031] The method for optimizing household electricity load scheduling based on the MAPPO algorithm includes:

[0032] The data acquisition and preprocessing module is responsible for collecting multi-source heterogeneous data from the user side and the external environment, and performing standardized processing to provide input data in a unified format to the subsequent user behavior prediction, electricity price prediction and optimized dispatch decision-making modules.

[0033] The user behavior prediction module estimates the probability of future device usage by collecting users' historical device usage data, resulting in a device behavior prediction vector.

[0034] An electricity price prediction module acquires real-time electricity price information and uses a time series prediction model to predict short-term electricity price trends based on a historical electricity price sequence, and outputs an electricity price prediction sequence;

[0035] An optimal scheduling decision module generates a scheduling strategy through deep reinforcement learning based on the device behavior prediction vector and the electricity price prediction sequence, and outputs a corresponding scheduling result;

[0036] A device control and feedback module sends the scheduling result to a home gateway to control the devices, and collects real-time execution feedback results to update the internal parameters of the MAPPO network.

[0037] A computer device of the present application comprises a memory and a processor, the memory and the processor are electrically connected, the memory stores a computer program, and the computer program is executed by the processor to realize the method.

[0038] A computer readable storage medium of the present application stores a computer program, and the processor realizes the method when the computer program is executed by the processor.

[0039] Compared with the prior art, the present application has the following beneficial technical effects:

[0040] 1. Enhance the dynamic modeling capability of user behavior and improve the power consumption experience. In view of the problem that the user behavior modeling in the prior art is rough and difficult to adapt to dynamic usage habits, the present application introduces an LSTM network to construct a user device start probability model, which can learn individual behavior rules based on historical usage data, improve the individualization and flexibility of the scheduling scheme, and thus enhance user acceptance and system adaptability. The introduced reward shaping mechanism considers user comfort and power consumption cost comprehensively, avoids the deterioration of user experience caused by simply saving cost, realizes the balance between energy saving and comfort, and improves user satisfaction and acceptance.

[0041] 2. Improve the scheduling stability in the electricity price fluctuation environment and realize demand response. In order to alleviate the high dependence of scheduling on load prediction accuracy, the present application constructs an electricity price prediction module to realize dynamic perception of short-term electricity price trends and improve the robustness and economic response ability of the system in the fluctuating market environment. Precise peak shaving and valley filling of power grid load and multi-device load aggregation strategy balance the utilization of the system, the system automatically reduces the power of non-emergency load during the peak period of power grid load, and reasonably increases the delayable load (such as water heater heating) during the trough period of load, realizes the intelligent transfer and smoothing of load.

[0042] 3. Adaptive learning and dynamic policy evolution capability. The system can automatically adjust according to real-time feedback data, adapt to user usage habits and changes in the external environment, and realize dynamic evolution of the scheduling policy through continuous self-optimization of the scheduling policy by deep reinforcement learning algorithm. Compared with traditional fixed rule-based or static optimization model, the present invention has stronger robustness and long-term optimization capability.

[0043] 4. Improve the safety and reliability of scheduling decisions. By introducing the dynamic action masking mechanism, avoid scheduling actions causing device conflicts or system abnormalities, enhance the stability of the overall scheduling system, reduce the risk of scheduling errors, and ensure the long-term safe operation of household appliances.

[0044] 5. Modular design, with wide application and promotion potential. The system architecture of the present invention is clear, the coupling degree between modules is low, supports flexible expansion and customized development, and is convenient for application and promotion in smart community, micro-grid and regional energy management platform and other scenarios. Its standardized data interface and general scheduling framework are suitable for a variety of household appliances and different types of user groups, greatly improving the industrial applicability and commercial value of the system. BRIEF DESCRIPTION OF DRAWINGS

[0045] Fig. 1 A flowchart of a household electricity load optimization scheduling method based on the MAPPO algorithm of an embodiment of the present invention.

[0046] Fig. 2 A MAPPO scheduling optimization flowchart used by an embodiment of the present invention.

[0047] Fig. 3 A user behavior prediction module structure diagram used by an embodiment of the present invention.

[0048] Fig. 4 A power price prediction module structure diagram used by an embodiment of the present invention.

[0049] Fig. 5 An action mask generation and execution flowchart used by an embodiment of the present invention.

[0050] Fig. 6 A reward function structure diagram used by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be described clearly and completely below with reference to the drawings in the embodiments of the present invention.

[0052] A household electricity load optimization scheduling system based on the MAPPO algorithm of the present embodiment comprises:

[0053] Data collection and preprocessing module: This module is the basic component of the system, mainly responsible for collecting multi-source heterogeneous data from the user side and the external environment, and performing standardized processing to provide input data in a unified format to the subsequent user behavior prediction, electricity price prediction and optimal scheduling decision module;

[0054] User behavior prediction module: By collecting user historical device usage data, the device startup probability distribution model is used to estimate the future device usage probability of the user, and the device behavior prediction vector is output;

[0055] Electricity price prediction module: Obtain real-time electricity price information, and use time series prediction model based on historical electricity price sequence to predict short-term electricity price trend, output electricity price prediction sequence;

[0056] Optimal scheduling decision module is the core sub-module of the system: based on the device behavior prediction vector and the electricity price prediction sequence, the MAPPO algorithm generates a scheduling strategy and outputs the corresponding scheduling result;

[0057] Device control and feedback module: send the scheduling result to the home gateway, control the household appliances (air conditioner, water heater, etc.), and real-time collect device response state and execution result, which is used for subsequent state update and user behavior modeling correction, update MAPPO algorithm internal parameters, etc.

[0058] As shown in Figs. 1-6 , the embodiment realizes a method of household electricity load optimal scheduling system based on MAPPO algorithm, comprising the following steps:

[0059] (S1), collect multi-source heterogeneous data from the user side and the external environment and perform preprocessing, including:

[0060] (S11) data format definition

[0061] a) Appliance operating state information M t :

[0062]

[0063] Wherein, indicates the start-stop state of the mth device at time t, indicates the power (unit: kW) of the mth device, m=1,2,…,n, n represents the number of devices, n d ∈{0,1} indicates the schedulable flag.

[0064] b) User behavior feature sequence A [t-k,t] :

[0065]

[0066] where A [t-k,t] represents a sequence of user behavior features from time t-k to t, which is used to characterize the user's power consumption pattern, device operation preference and historical behavior in response to scheduling instructions within a period of time. This sequence is composed of user behavior vectors a t-k ,a t-k+1 ,…,a t at each time step, where each vector represents the user's usage behavior vector of n devices at time t. represents the usage state of device n at time t. represents an n-dimensional real number space, reflecting that each behavior vector at a time step contains n different behavior dimensions.

[0067] c) Electricity price sequence B [t-k,t] :

[0068]

[0069] where B [t-k,t] represents the electricity price time sequence from time t-k to time t; b t represents the unit time electricity price at time t, with a unit of yuan / kWh; k is the review window length, representing the number of time steps traced back; represents a set of real numbers, indicating that each b t is a real scalar, and b t represents the unit time electricity price (unit: yuan / kWh).

[0070] d) Environmental context E t :

[0071]

[0072] where E t represents the environmental context information at time t, h t ∈[0,23] is the hour number corresponding to time t, used to reflect the position of the time point in a day; d t ∈{0,1} is a holiday flag; w t ∈[0,6] represents the day of the week. represents the outdoor temperature and humidity, both of which are real values.

[0073] e) User comfort constraint vector C t :

[0074] C t = [(t ac ,r ac ),(t whr wh ),δ,γ]

[0075] wherein, C t represents the comfort constraint vector of the user at time t, which is used to depict the user's preference and tolerance to the environmental control device; t ac represents the desired temperature of the air conditioner, r ac represents the temperature tolerance range of the air conditioner, t wh represents the desired temperature of the water heater, r wh represents the temperature tolerance range of the water heater, δ is the maximum allowed scheduling delay time, and γ is the user comfort weight.

[0076] f) power grid load sequence L [t-k,t] :

[0077] L [t-k,t] = [l t-k , l t-k+1 , …, l t ]

[0078] wherein, L [t-k,t] represents the power grid load time sequence from time t-k to time t, which is used to reflect the load change trend of the regional power grid within a certain time window; l t is the regional power grid load or load index at time t.

[0079] (S12) Data preprocessing

[0080] In order to ensure the consistency, stability and model generalization ability of the data, the following preprocessing operations are performed on the original data in this module:

[0081] (a) Time alignment and sliding window resampling. All data are aligned at a uniform time interval Δt (such as 15 minutes), and time series input is constructed in a sliding window manner. For non-aligned data (such as event-triggered sampling), nearest sampling retention or linear interpolation processing is used.

[0082] (b) Missing value processing. For occasional missing data, average value filling in the time neighborhood (such as mean interpolation, forward filling) is used; for serious missing fields (missing rate exceeding threshold), they can be removed or treated as unobservable state.

[0083] (c) Normalization processing:

[0084] Electricity price, appliance power, and load variables are normalized using maximum and minimum values:

[0085]

[0086] wherein, x represents the original variable value, x min and x maxrespectively represent the minimum and maximum values of the variable, denotes the variable value after max-min normalization, and the normalized result is mapped to the interval [0, 1]. The time field (such as the number of hours, the week) uses periodic feature encoding (sine and cosine encoding) to preserve periodicity:

[0087]

[0088] wherein, and respectively represent the number of hours h t The features after cosine and sine periodicity encoding are used to preserve the periodicity information of time.

[0089] (d) Category variable encoding. Discrete variables such as holiday flags and days of the week are represented by one-hot encoding, which helps to accurately distinguish different categories and improve the clarity and discriminability of feature representation.

[0090] (S2), by collecting user historical device usage data, constructing its device startup probability distribution model in different time periods, and outputting a device behavior prediction vector.

[0091] The device startup probability distribution model aims to provide prospective prior information for the optimization scheduling decision module. By modeling and predicting the probability of users using various appliances in the future scheduling period, the device startup probability distribution model helps the scheduling system to more reasonably allocate energy resources and improve response efficiency. Given the differences in life habits, appliance usage preferences, holiday arrangements, and weather and other external factors among different users, the device startup probability distribution model combines historical operation data and time context information to construct a personalized feature-oriented user electricity consumption behavior representation and prediction mechanism, serving as a key perception and decision support link in the entire scheduling system. The overall structure of the device startup probability distribution model includes: input layer, long short-term memory network LSTM layer, and feedforward neural network layer.

[0092] The input layer takes the user's appliance running state information M t , behavior feature sequence A [t-k,t] , and environmental context information E t as input, capturing the time sequence pattern and external dependency of user behavior. The overall structure takes a lightweight long short-term memory network (Long Short-Term Memory, LSTM) as the core C B, A, Bugge E S. Long short-term memory RNN [J]. arXiv preprint arXiv:2105.06756, 2021.) which realizes the probabilistic prediction of user's future electricity consumption behavior by dynamically modeling the time series characteristics. The specific process includes: first, align the input data by time step in the input layer, and splice the appliance state, behavior characteristics and environmental context at each time step to form a unified time series input sequence; then, the sequence is input into the multi-layer stacked long short-term memory network (LSTM) layer in turn, and the long-term dependence and periodic trend in the historical behavior are captured through the recursion and update of the hidden state; finally, based on the final time hidden state output by the long short-term memory network (LSTM), the use probability prediction of each type of controllable device in the future scheduling period is generated through a multi-layer feedforward neural network layer. The final output is a device behavior prediction vector

[0093]

[0094] wherein, represents the probability prediction value of the i-th device being started or used by the user in the future scheduling period, which is used to support subsequent intelligent scheduling optimization i = 1, 2,..., n, n represents the total number of controllable household appliances connected to the system.

[0095] (S3), obtaining real-time electricity price information, and using a time series prediction model to predict the short-term electricity price trend based on the historical electricity price sequence, and outputting an electricity price prediction sequence.

[0096] The time series prediction model adopts a modeling method that combines a temporal convolutional network (TCN) and a self-attention mechanism to improve the prediction accuracy and stability of future electricity price trends. This module takes the historical electricity price sequence as input to generate electricity price prediction results for future time periods, and has the characteristics of high modeling efficiency and strong context capturing ability. The overall structure of the time series prediction model includes an input layer, a TCN convolutional encoding layer, a self-attention enhancement layer, and a prediction output layer.

[0097] First, the input of the input layer is the historical price sequence B [t-k,t] In the preprocessing stage, time stamp encoding (such as hour, intra-day position, etc.) and external auxiliary features such as air temperature and grid load can be integrated. Then, through the causal convolution in the TCN convolutional encoding layer, the price change patterns at different time scales are extracted under the premise of not introducing future information when predicting. The TCN convolutional encoding layer also introduces a dilated convolution, which realizes effective modeling of long-distance dependencies by setting a layer-by-layer increasing dilated rate d, and the convolution form is:

[0098]

[0099] where y t denotes the convolutional output at the current time step t, i.e., the feature results extracted by the TCN convolutional encoding layer; k denotes the size of the convolutional kernel, i.e., the number of input historical time steps each output value depends on; w i denotes the i-th convolutional kernel weight, which is a parameter learned by the model during the training process; x t-d·i denotes the input feature of the i-th historical time step, where d is the dilation rate, which determines the sampling interval between inputs.

[0100] On the basis of the output of the TCN convolutional encoding layer, a self-attention enhancement layer is further introduced. The self-attention mechanism of the self-attention enhancement layer is used to enhance the ability to identify key historical moments. Specifically, the TCN convolutional encoding layer output is mapped to query vector Q, key vector K and value vector V through a fully connected layer, and an attention weight matrix with causal mask is constructed:

[0101]

[0102] where Z is the aggregated vector output at each time step; Softmax(·) is a normalization function that maps the elements in a vector to a set of non-negative, sum-to-one probability values; d k denotes the feature dimension of Query and Key, which is a scaling factor; M is a mask matrix used to mask information after the current moment, ensuring that the time series prediction model only makes predictions based on historical information. The self-attention enhancement layer can automatically assign different historical moments a contribution weight to the current prediction, thereby significantly improving the response ability to seasonal mutations or short-term fluctuations.

[0103] Finally, the output of the self-attention enhancement layer is fed into several fully connected layers to output the electricity price prediction sequence for the next H time steps:

[0104]

[0105] where, denotes the model's prediction of the unit time electricity price for the next H time steps at the current time t, denotes the prediction value of the electricity price at the t+h moment, h = 1, 2, …, H.

[0106] The time series prediction model considers both the parallel computing capability of TCN and the context-sensitive characteristics of self-attention mechanism in structure, and is suitable for deployment on edge nodes or scheduling centers to realize efficient real-time prediction of electricity price. The prediction result will be used as one of the key inputs of the optimization scheduling module to formulate a collaborative strategy for multi-user appliance operation, and to achieve the regulation and control targets of energy saving and peak load shifting.

[0107] (S4) Based on the device behavior prediction vector and the electricity price prediction sequence, a deep reinforcement learning is used to realize policy iteration and adaptive adjustment, and a scheduling result is output.

[0108] The deep reinforcement learning is mainly constructed based on the multi-agent proximal policy optimization algorithm MAPPO (Yu C, Velu A, Vinitsky E, et al. The surprising effectiveness of PPO in cooperative multi-agent games [J]. Advances in neural information processing systems, 2022, 35: 24611-24624.), and each user is regarded as an independent agent. The agents learn and make distributed decisions in cooperation, thereby realizing personalized and coordinated scheduling for large-scale household loads.

[0109] The design of the multi-agent proximal policy optimization algorithm MAPPO includes the following contents:

[0110] (1) Definition of state space

[0111] State vector of each user agent i At time t, the following sub-items are included but not limited to: current appliance operation state vector M t , electricity price prediction sequence User behavior prediction probability User comfort constraint vector External environment context E t , power grid load prediction L [t-k,t] .

[0112] The complete state vector is defined as:

[0113]

[0114] Among them, the state vector represents the overall system state of the i-th user at time t, represents the current connected appliance operation state information M t of the i-th user; represents the electricity price prediction sequence at time t, which is a global shared variable; represents the prediction result of the power consumption behavior of the i-th user at time t; represents the comfort constraint vector C of the i-th user t ; E t represents the environmental context information, which is a global shared variable; L [t-k,t] represents the total load history sequence of the power grid in the period [t-k, t], which is a global shared variable.

[0115] (2) Action space definition

[0116] To support unified scheduling control of various types of household appliances, the application adopts a generalized discrete action space modeling method. For each user agent i, the control action includes scheduling decisions for multiple devices, and each device action is defined as a set of optional control strategies, defined as follows:

[0117] For any device (such as air conditioners, water heaters, floor heating, washing machines, etc.), the action space is represented as:

[0118]

[0119] where D i represents the set of household appliances owned by the i-th user, d∈D i represents a specific device among them; A d represents the action space of device d, i.e., the set of control instructions available for the device; represents the k-th control option of device d, such as: on / off state, target temperature setting, heating / cooling mode, running time reservation, etc.

[0120] Then the joint action of the user agent is represented as:

[0121]

[0122] where, represents the joint control action vector of the i-th user at time t, composed of the control actions of all devices owned by the user; represents the control action of the j-th device of the user at time t, where d j ∈D i , |D i | represents the number of devices owned by user i.

[0123] (3) Dynamic action masking mechanism

[0124] In the present application, in order to avoid the reinforcement learning agent exploring invalid or unreasonable control actions during training or execution, thereby improving the effectiveness and convergence speed of the scheduling strategy, a dynamic action masking mechanism is introduced. This mechanism dynamically adjusts the set of available actions at each time by considering the current system state, device operation constraints and user behavior rules, effectively pruning the policy space. The implementation steps of this dynamic action masking mechanism are as follows:

[0125] (3a) State perception and rule screening

[0126] At each scheduling time step t, the system collects the following state information, including the current device running state (such as on / off, target temperature, remaining working time), user behavior prediction results (such as the probability of a user using a certain device, reservation preferences, etc.), current period attributes (such as late night period, holiday), power grid constraints or load limits (such as the current power upper limit reached);

[0127] (3b) Action masking rule generation

[0128] Based on the above state information, an action effectiveness judgment function is constructed as follows:

[0129]

[0130] where H t represents the state information of the system at time t, which can include user state, electricity price, environmental context, etc. Example rules include but are not limited to:

[0131] If the water temperature of the water heater has reached the upper limit, mask the "continue heating" type action;

[0132] If the air conditioner is currently in operation and the user's predicted usage probability is 0, mask the "adjust temperature" action;

[0133] If the current is the user's set sleep period, mask the "washing machine start" and other strong disturbance device operations;

[0134] If multiple devices are running simultaneously and the total power sum is close to the power meter limit, mask new high-power consumption actions.

[0135] (3c) Action mask generation and policy constraint

[0136] According to the judgment function results of all devices, an action mask tensor is generated:

[0137]

[0138] where, represents the action mask tensor of the i-th user at time t, which is used to identify the feasibility of each candidate control action; denotes the effectiveness determination result of the Kth control action of the Dth device of user i in the state denotes the effectiveness determination result of the Kth control action of the Dth device of user i in the state D D i denotes the Dth device owned by user i.

[0139] In the strategy sampling and execution stage, the sampling space is limited by the mask, and the action is sampled only from the unmasked action set, so as to avoid invalid or illegal operation.

[0140] (3d) Action mask driven strategy training.

[0141] The action mask is used as an auxiliary input to access the reinforcement learning network structure (such as the executor network), and participates in the training process, and strengthens the learning attention to the actionable actions. This mechanism does not change the original action space dimension, but dynamically clips the probability distribution support set of the strategy output.

[0142] (4) Reward function design

[0143] In order to improve the training convergence efficiency and improve the strategy quality, a reward shaping mechanism is introduced. The reward shaping mechanism introduces a series of auxiliary reward items on the basis of the main reward function, guides the executor network to gradually learn the expected behavior under the premise of not changing the optimal strategy, and effectively improves the training stability and convergence speed under the multi-objective scheduling task.

[0144] In the present application, for each user agent, the instantaneous reward function at time t is defined as the weighted combination of the original reward item and a plurality of shaping items, which is in the following form:

[0145]

[0146] Among them, denotes the total reward value obtained by the i th user at time t, which is a weighted combination of the main reward and a plurality of auxiliary reward shaping items, and is used to evaluate the overall advantages and disadvantages of the control decision at this moment. denotes the main reward related to the global optimization goal, including electricity cost, load balance and user comfort, denotes the j th auxiliary reward shaping item, λ j is the weight coefficient thereof, which can be dynamically adjusted according to the training stage or the strategy performance.

[0147] The main reward part mainly includes the following three contents:

[0148] a) Electricity cost item r cost , used to measure the economic expenditure brought by the current scheduling decision of the user, defined as:

[0149]

[0150] where, is the current device power of the user, is the predicted electricity price.

[0151] b) Grid load balancing term r grid , to guide the overall system to achieve peak shaving, defined as:

[0152]

[0153] where, L t is the total system load at the current time, is the expected load level (such as a moving average).

[0154] c) Comfort loss term r comfort , to measure the impact of device scheduling on user experience, in the form of:

[0155]

[0156] where, r ac represents the comfort reference value during air conditioning operation, reflecting the user's acceptable deviation degree for air conditioning use effect (such as temperature adjustment), which has been defined in the foregoing; r wh represents the comfort reference value during water heater use, representing the user's tolerance for hot water use temperature or availability, which has been defined in the foregoing; δ represents the allowed response delay threshold of user activity behavior, which has been defined in the foregoing; e ac , e wh , e δ represents the residual value of the expected value set by the user and the actual value, is the normalized deviation value to prevent the influence of different dimensions. w ac , w wh , w δ , w int is a weight parameter, and the weight can be set according to the importance of the device and the user preference. represents the linkage penalty of temperature deviation and scheduling delay, for example, when both are not ideal, the user discomfort is doubled. γ is the overall comfort priority weight, controlling the influence degree of the loss on the total loss.

[0157] To further improve the training efficiency of the strategy and the rationality of the behavior, the following auxiliary reward shaping terms are designed in the present application:

[0158] d) Device smoothness reward term r stable , to suppress frequent start-stop behavior of the device, in the form of:

[0159]

[0160] wherein, is an indicator function that determines whether the current action is inconsistent with the previous time, and λ1 is a weight coefficient, is the joint action of the user agent.

[0161] e) Time preference guide item r time encouraging the strategy to prioritize running the device within the user's preferred time period, defined as:

[0162]

[0163] wherein, is the set of time preferences set by user i, and λ2 is a weight coefficient for controlling its influence on the overall reward.

[0164] f) Device coordination control item r coop for avoiding local congestion or overload caused by multiple devices running simultaneously at high load, represented as:

[0165]

[0166] wherein, represents the control action of device d at time t, which takes a binary indicator value; N limit is the upper limit of simultaneous running devices, and λ3 is a weight coefficient.

[0167] g) Strategy entropy reward item r entropy for encouraging strategy diversity and exploration, especially suitable for the early training stage, defined as:

[0168] r entropy = + λ4·H[π θ (·|H t )]

[0169] wherein, H[·] represents the entropy of the strategy probability distribution, π θ () is the strategy function, and λ4 is a weight coefficient.

[0170] In specific implementation, the weight coefficients λ j of each shaping item can be dynamically adjusted using an exponential decay function to focus on exploration and behavior specification in the early training stage and focus more on optimizing the main goal in the later stage. The weight adjustment can be in the following form:

[0171]

[0172] wherein, λ j (t) represents the dynamic weight coefficient of the jth loss at time t, represents the weight value of the loss term at the initial moment, i.e., the weight setting at t = 0; a is an exponential decay coefficient, which controls the decay speed of the weight with time, and the greater the value, the faster the weight decays, and the system pays more attention to behavior guidance in the early training and gradually focuses on optimizing the main task target in the later training.

[0173] The above reward shaping mechanism significantly improves the training efficiency, behavior rationality and execution stability of the scheduling strategy in a multi-objective scenario, especially when dealing with conflicts between user preferences and grid targets, showing stronger flexibility and controllability.

[0174] The multi-agent proximal policy optimization algorithm of the embodiment adopts a centralized training (Centralized Training) and decentralized execution (Decentralized Execution) mechanism, uses global state information to improve the policy learning efficiency in the training stage, and makes independent decisions by each user agent in the execution stage, ensuring the scalability and edge deployment capability of the system.

[0175] Under the centralized training background, the Critic network of MAPPO adopts a mean square error loss function, and its loss expression is defined as follows:

[0176]

[0177] wherein, represents the loss of the value function network, and N is the number of batch samples, represents the centralized Critic network output with the global state H i and all agent observations as inputs, is the target return of the i-th sample, which is calculated by the generalized advantage estimation value. This loss function is used to minimize the difference between the Critic network estimation value and the true return, thereby improving the accuracy and stability of the value function.

[0178] Unlike the PPO algorithm, MAPPO adopts a decentralized execution mechanism, and the strategy of each agent is based on its local observation o t . The strategy loss function L actor (θ) is defined as follows:

[0179]

[0180] wherein, represents the expected operation on the sample data at time t in the training process, which is usually approximated by small batch sampling; θ represents the parameters of the current policy network, which is used to control the policy function π θ (a t |o tThe output behavior of ); clip(·) is a clipping operation used to limit the probability ratio r. t The update range of (θ) is adjusted to prevent the strategy from changing too drastically. Let be the strategy ratio, where π θ (a t |o t This indicates that, under the current policy parameters θ, the agent, based on its local observations o t Choose action a t The conditional probability; This indicates that in the old policy parameter θ old Below, the intelligent agent bases its observations on its local observations. t Choose action a t The conditional probability; is the advantage function estimate, which can be calculated by GAE (Generalized Advantage Estimation); ∈ is the policy update shearing threshold (e.g., 0.2), used to limit the KL divergence distance between the old and new policies, forcing the policy probability ratio of each update to be limited to the interval [1-∈, 1+∈], avoiding the unstable impact of large step size updates on the policy; H[.] is the policy entropy, used to encourage policy diversity and avoid prematurely falling into local optima; β is the entropy regularization coefficient. t This indicates the control action output by the optimized scheduling decision module.

[0181] (S5) Send the scheduling results to the home gateway to control the device, collect execution feedback results in real time, and update the user behavior model and reinforcement learning state. This mainly includes the following:

[0182] (S51) Dispatch instruction issued

[0183] Optimize the control action a output by the scheduling decision module t This includes the start / stop status and expected operating periods of each controllable load device (such as air conditioners, water heaters, washing machines, etc.) in a future period of time. The control actions are uniformly encoded by the system and sent to the target appliance terminal through the home energy management gateway (such as an edge device supporting communication protocols such as Zigbee, Wi-Fi, RS485, etc.) to execute the corresponding control operations.

[0184] (S52) Equipment execution feedback collection

[0185] The system transmits real-time feedback information such as the actual operating status of the devices, power data, and response latency through the home gateway, forming a feedback vector:

[0186]

[0187] Among them, f tis the execution feedback vector at time t, which contains the actual state information of each user device; n represents the total number of users accessing the system; is the actual state; represents the actual power; is the instruction response delay time (such as deviation caused by user intervention or communication delay). The system compares this feedback information with the original scheduling plan to estimate the frequency of user intervention, device response rules and other potential behavior characteristics.

[0188] (S53) Behavior model and learning state update

[0189] The system compares the difference between the device state before and after the execution of the instruction to determine whether the user manually modifies the system instruction (such as the user turning off the system turned on device), thereby correcting the preference parameters in the user behavior prediction module. Based on the feedback data, the environment state H t , the reward function r t and the state transition H t+1 can be updated to realize the closed-loop reinforcement learning process.

[0190] Through the above control and feedback closed-loop mechanism, this module realizes the automatic control and behavior-driven dynamic adjustment of household loads, effectively improves the practicality and user acceptance of the scheduling strategy, and at the same time provides online interaction data of the real environment for the reinforcement learning model, promoting the continuous self-optimization of the scheduling strategy.

[0191] As a specific embodiment, the household electricity load optimization scheduling method based on the MAPPO algorithm of the embodiment comprises the following steps:

[0192] Step S1, through the household energy management gateway and the multi-source data acquisition interface, the running state data of the electric devices such as air conditioners and water heaters in the household, environmental sensor information, user historical electricity consumption behavior data, and multi-source heterogeneous data such as power grid load and time context are collected in real time. The raw data collected after preprocessing operations such as time stamp alignment, missing value filling and normalization form a structured and standardized data set. The data set is then delivered in parallel to the user behavior modeling module and the electricity price prediction module as two core input sources.

[0193] Step S2, the user behavior prediction module uses a deep learning method to accurately predict the starting probability and running period of the device in the future scheduling period based on historical device start-stop records, user electricity consumption habit characteristics and environmental variables using a long short-term memory network. The module outputs probability distribution information reflecting the actual behavior pattern of the user, providing user preferences and behavior priors for scheduling optimization, and ensuring the rationality and user acceptance of the final scheduling decision.

[0194] Step S3, in synchronization therewith, the electricity price prediction module is based on historical electricity price data and external auxiliary information, combines a causal convolution network and a self-attention mechanism, and realizes dynamic prediction of future multi-period electricity prices. The prediction result contains the electricity price fluctuation trend of multiple time nodes, provides an economic cost reference for scheduling strategy making, enables the system to reasonably adjust the equipment operation time, avoids high-price periods, and realizes the goal of saving electricity fees.

[0195] Step S4, the output results of the user behavior prediction and the electricity price prediction module are taken as inputs and are jointly delivered to an optimized scheduling decision module. The module is based on a multi-agent proximal policy optimization algorithm, comprehensively considers factors such as device state, user behavior prediction, electricity price fluctuation, comfort constraint and power grid load, constructs a reinforcement learning framework containing a state space, an action space and a dynamic action mask mechanism. In the reasoning phase, each scheduling agent corresponds to an independent policy network (Actor), which takes the local state observation of the current time step t as input and outputs the probability distribution of each control action in the action set of the device. The system simultaneously introduces a dynamic action mask mechanism to shield actions that are illegal, uncontrollable or not allowed by user preferences, thereby ensuring that the policy output action always meets the control feasibility, safety and comfort constraints.

[0196] At the same time, in order to train better policy network parameters, the system designs a centralized Critic network, which can access the observation, action and prediction information of all devices in the training phase, including user behavior prediction, electricity price prediction results and power grid load state. The Critic network outputs the state value estimation of each Agent, and the error between the state value and the actual obtained discount reward is minimized to optimize the update of the policy network. The probability ratio loss function with a clipping term is used to control the change amplitude between the new and old policies, thereby preventing the policy from jumping sharply in the optimization process. The parameter update process of the policy network also introduces a KL divergence regularization term to suppress the degree of policy drift and avoid sudden interference to user behavior patterns and power grid scheduling plans.

[0197] The training process is based on the sampled device operation trajectory data, and uses the batch stochastic gradient descent method to optimize the above loss function, thereby synchronously updating the parameters of the policy network and the Critic network. The system further guarantees the stability of the policy evolution in the training process through the experience replay mechanism and the trust region policy based on the KL divergence constraint.

[0198] At the end of each training cycle, the updated policy network is used for the next round of scheduling inference task. To prevent policy failure or abnormal device control, the system sets up a policy drift detection mechanism and a policy rollback function. If the updated policy does not achieve performance improvement threshold on backtest data, or causes abnormal device operation in actual deployment, it will automatically roll back to the last version of policy network parameters.

[0199] In step S5, the control instructions generated by the optimization scheduling module are sent to each controllable device through the home gateway, and the device uploads the real-time feedback of the running state, power curve and user intervention to the system after execution. The feedback data is used to update the user behavior model and the environment state, forming a closed-loop adaptive mechanism to improve the prediction accuracy of the model and the robustness of the policy. At the same time, the feedback information is used as an instant reward supplement to assist the continuous optimization of the scheduling algorithm, ensuring the efficient and stable operation of the system in complex dynamic environment.

[0200] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details and do not limit the application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the specification. The specification selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and utilize the application.

Claims

1. A method for optimizing household electricity load scheduling based on the MAPPO algorithm, characterized in that, Includes the following steps: (1) Collect multi-source heterogeneous data from the user side and the external environment and preprocess it; (2) Based on the preprocessed data, the probability of a user using the device startup probability distribution model is estimated, and the device startup probability is output. (3) Based on the historical electricity price series, a time series forecasting model is used to predict the short-term electricity price trend and output the electricity price forecast results; (4) Based on the equipment start-up probability and electricity price prediction results, deep reinforcement learning is used to achieve iterative and adaptive adjustment and output scheduling results; (5) Send the scheduling results to the home gateway to control the device and collect execution feedback results in real time to update the user behavior model and reinforcement learning status.

2. The method for optimizing household electricity load scheduling based on the MAPPO algorithm according to claim 1, characterized in that, Preprocessing includes standardizing the data formats of household appliance operating status information, user behavior feature sequences, electricity price sequences, environmental context, user comfort constraint vectors, and power grid load sequences, and then processing the data.

3. The method for optimizing household electricity load scheduling based on the MAPPO algorithm according to claim 1, characterized in that, In step (2), the device start probability distribution model includes an input layer, a long short-term memory network (LSTM) layer, and a feedforward neural network layer. In the input layer, each input data is aligned according to time steps, and at each time step, the appliance status, behavioral features, and environmental context are spliced ​​together to form a unified time-series input sequence. Subsequently, the time-series input sequence is sequentially fed into a multi-layered stacked Long Short-Term Memory (LSTM) network. Through the recursion and updating of the hidden states, the long-term dependencies and periodic change trends in historical behavior are captured. Finally, based on the hidden states at the final moment output by the LSTM network, a multi-layered feedforward neural network is used to map and generate a prediction of the usage probability of each type of controllable device in the future scheduling cycle. The final output is a device behavior prediction vector.

4. The method for optimizing household electricity load scheduling based on the MAPPO algorithm according to claim 1, characterized in that, In step (3), the time series prediction model includes an input layer, a TCN convolutional coding layer, a self-attention enhancement layer, and a prediction output layer. The input of the input layer is the historical sequence of electricity prices and external auxiliary features such as temperature and grid load. Then, through causal convolution in the TCN convolutional coding layer, price change patterns at different time scales are extracted. The TCN convolutional coding layer also introduces dilated convolution. By setting an increasing dilation rate layer by layer, effective modeling of long-distance dependencies is achieved. Based on the output of the TCN convolutional coding layer, a self-attention enhancement layer is introduced. Finally, the output of the self-attention enhancement layer is fed into several fully connected layers to output the future electricity price prediction sequence.

5. The method for optimizing household electricity load scheduling based on the MAPPO algorithm according to claim 1, characterized in that, In step (4), the deep reinforcement learning is constructed based on a multi-agent proximal policy optimization algorithm, including: (1) Construct a state vector, which includes the user's home appliance operating status, electricity price prediction sequence, user behavior prediction, comfort constraints, environmental context information and historical load of the power grid; (2) Define the action space, which is a set of optional control instructions for each household appliance. The user's joint action is composed of the actions of all his / her devices.

6. The method for optimizing household electricity load scheduling based on the MAPPO algorithm according to claim 1, characterized in that, Introducing dynamic action masking mechanisms into deep reinforcement learning, including: (1) Collect equipment operating status, user behavior prediction, time period attributes and power grid constraint information; (2) Generate action masking rules based on the collected information and form an action mask to distinguish between executable actions and actions that need to be masked; Action masking rules include: constructing an action validity determination function based on state information. when: Among them, H t It represents the system's state information at time t, including factors such as user status, electricity price, and environmental context; This represents the k-th control option for device d; Generate action mask and policy constraints: Generate action mask tensors based on the decision function results of all devices; (3) During the sampling and execution process, selection is only allowed from valid actions; (4) During the training phase, the action mask is used as an auxiliary input to the policy network to improve the learning effect of actionable actions.

7. The method for optimizing household electricity load scheduling based on the MAPPO algorithm according to claim 1, characterized in that, The reward function of deep reinforcement learning includes a weighted combination of a primary reward and several auxiliary rewards. The primary reward includes electricity cost, load balancing, and user comfort. The auxiliary rewards are shaping terms used to improve training efficiency, and their weights are dynamically adjusted according to the policy performance.

8. The method for optimizing household electricity load scheduling based on the MAPPO algorithm as described in claim 1, characterized in that, include The data acquisition and preprocessing module is responsible for collecting multi-source heterogeneous data from the user side and the external environment, and performing standardized processing to provide input data in a unified format to the subsequent user behavior prediction, electricity price prediction and optimized dispatch decision-making modules. The user behavior prediction module estimates the probability of future device usage by collecting users' historical device usage data, resulting in a device behavior prediction vector. The electricity price forecasting module acquires real-time electricity price information and uses a time series forecasting model based on historical electricity price sequences to predict short-term electricity price trends, outputting an electricity price forecast sequence. The optimized scheduling decision module generates scheduling strategies based on equipment behavior prediction vectors and electricity price prediction sequences through deep reinforcement learning, and outputs the corresponding scheduling results. The device control and feedback module sends the scheduling results to the home gateway to control the devices; Collect execution feedback results in real time and update the internal parameters of the MAPPO network.

9. A computer device comprising a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, characterized in that: When the computer program is executed by the processor, it causes the processor to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor implements the method as described in any one of claims 1 to 8.

Images