An energy management method and system based on sparse federated reinforcement learning

Abstract

The application belongs to the field of home energy management, and relates to an energy management method and system based on sparse federated reinforcement learning, comprising: collecting local energy consumption information at each time step, and receiving real-time electricity price information from a power company; constructing an initial energy management model, and training the initial energy management model to obtain a final energy management model; based on the energy consumption information and the real-time electricity price information, executing an intelligent energy management algorithm through the energy management model to obtain an energy management strategy; generating control instructions for installed household devices through the energy management strategy, and managing the installed household devices through the control instructions; fundamentally solving the memory bottleneck problem of edge devices, realizing the localization deployment of complex algorithms, significantly reducing the communication overhead of federated learning, and improving the system training efficiency and stability.

Description

An energy management method and system based on sparse federated reinforcement learning Technical Field

[0001] This invention relates to the field of home energy management, and specifically discloses an energy management method and system based on sparse federated reinforcement learning. Background Technology

[0002] In recent years, with the global pursuit of sustainable development and energy efficiency, the importance of home energy management has become increasingly prominent. Home energy management systems can not only effectively reduce household energy consumption and electricity costs, but also help alleviate the pressure on the power grid and promote the integration of renewable energy into the grid. Traditional centralized management suffers from high data transmission latency and significant privacy risks, making it difficult to meet real-time control requirements. Against this backdrop, edge-side home energy management systems have become a key solution to the challenges of traditional centralized management. By offloading real-time data processing and device control tasks to edge devices, this system can provide more personalized energy-saving suggestions and efficient demand response mechanisms, achieving flexible real-time resource control. Furthermore, because sensitive data is stored locally on devices, the system can significantly improve user privacy and security. However, the core challenge of this solution lies in the fact that the limited memory, computing, and storage resources of edge devices severely restrict the local deployment and training of complex models, becoming a key bottleneck in achieving efficient and intelligent energy management.

[0003] Current research on home energy management mainly revolves around two paradigms: rule-based optimization methods and model-free reinforcement learning. While traditional optimization methods (such as model predictive control) are widely used, their strong pre-defined nature (relying on fixed objective functions and constraints) leads to decision failures in uncertain scenarios such as dynamic electricity prices, intermittent renewable energy output, and load variations. Simultaneously, the high computational complexity of solving high-dimensional optimization problems severely restricts the system's real-time response capabilities. To overcome this bottleneck, model-free reinforcement learning has emerged as an emerging solution due to its characteristic of not requiring precise environmental modeling. Through autonomous learning, it achieves near-optimal control in complex dynamic environments, exhibiting stronger adaptability and robustness. However, existing home energy management frameworks require each participating edge device to deploy and train a dense reinforcement learning model. For smart home terminals with extremely limited memory capacity, even a model of moderate complexity may exceed their hardware capacity limits, leading to deployment failure or operational crashes. Furthermore, in federated learning, frequent exchange of model parameter updates and gradients is required between the client and server. For dense models, the amount of data updated each time is enormous, consuming significant home network bandwidth and increasing communication latency.

[0004] In view of this, the present invention provides an energy management method and system based on sparse federated reinforcement learning, which fundamentally solves the memory bottleneck problem of edge devices, realizes the localized deployment of complex algorithms, significantly reduces the communication overhead of federated learning, and improves the system training efficiency and stability. Summary of the Invention

[0005] The purpose of this invention is to provide an energy management method and system based on sparse federated reinforcement learning, addressing the problems of memory bottlenecks in edge devices, enabling localized deployment of complex algorithms, significantly reducing communication overhead in federated learning, and improving system training efficiency and stability. The specific solution is as follows:

[0006] An energy management method based on sparse federated reinforcement learning includes:

[0007] Collect local energy consumption information at each time step and receive real-time electricity price information from the power company;

[0008] An initial energy management model is constructed and trained to obtain the final energy management model;

[0009] Based on energy consumption information and real-time electricity price information, an intelligent energy management algorithm is executed through an energy management model to obtain an energy management strategy;

[0010] Energy management strategies are used to generate control commands for installed home appliances, and these commands are then used to manage the installed home appliances.

[0011] Furthermore, an initial energy management model is constructed and trained to obtain the final energy management model, including:

[0012] The energy management problem is transformed into a Markov decision process; the Markov decision process includes a state space, action space, state transition probability function, and reward function;

[0013] Construct an initial energy management model; the initial energy management model includes an initial actuator network and an initial evaluator network;

[0014] Input the current observation state into the actuator network and output the optimal continuous action;

[0015] The current observed state and the optimal continuous action are concatenated and input into the evaluator network to obtain the initial discrete action, and then... The greedy strategy selects the initial discrete action to obtain the optimal discrete action;

[0016] The optimal hybrid action is constructed based on the optimal continuous action and the optimal discrete action. Based on the optimal hybrid action, the energy management model is iteratively trained using the hybrid DQN algorithm until the energy management model converges, thus obtaining the final energy management model.

[0017] Furthermore, based on the optimal hybrid action, the energy management model is iteratively trained using the hybrid DQN algorithm to obtain the final energy management model, including:

[0018] Execute the optimal hybrid action, return an immediate reward after the action is executed and transition to a new observation state, and construct SARSA experience samples;

[0019] Based on SARSA empirical samples, multi-step objectives are generated through an energy management model. The loss function of the evaluator network is constructed using the value estimate of the optimal mixed action and the expected mean square error between the multi-step objectives. The gradient descent method is used to minimize the Bellman error and complete the evaluator network update.

[0020] Based on SARSA experience samples and the updated evaluator network, the chain rule is applied to the expected value of the optimal mixed action through deterministic policy gradient to construct the loss function of the executor network. Combined with the dual time scale update rule, a single-step update is performed to complete the update of the executor network.

[0021] The parameters of the energy management model are updated using a smooth soft update method; the energy management model includes an actuator network and a judge network.

[0022] The energy management model is iteratively trained and its network parameters are updated until the model converges, resulting in the final energy management model.

[0023] Furthermore, the formula for calculating the value estimate of the optimal mixed action is as follows:

[0024] ;

[0025] in, Estimate the value of the optimal mixed action at time t; and The observation states at time t and time t+1 are respectively; and The optimal hybrid actions at time t and time t+1 are respectively; To obtain the expected value; γ is the instantaneous reward at time t; γ is the reward decay factor; max is the maximum value. The action value function given the policy π; and These are the optimal discrete actions at times t and t+1, respectively. For optimal continuous action; θ represents the actuator network parameters; θ represents the evaluator network parameters.

[0026] The loss function of the evaluator network is:

[0027] ;

[0028] in, Let be the loss function of the evaluator network; For optimal continuous action;

[0029] The loss function of the actuator network is:

[0030] ;

[0031] in, Let z be the loss function of the actuator network; z be the discrete action; and Z be the number of discrete actions.

[0032] Furthermore, an initial energy management model is constructed using sparse networks, including:

[0033] The sparse topology of the initial sparse network is constructed by allocating sparse connections between layers, resulting in the initialized sparse network. The sparse topology includes the overall sparsity of the sparse network, the total number of network layers, and the sparse connections between layers.

[0034] The sparse network is dynamically evolved using SARSA empirical samples to obtain the final sparse network.

[0035] The initial energy management model is obtained by replacing the fully connected layers of the initial actuator network and the initial evaluator network with the final sparse network.

[0036] Furthermore, the sparse network is dynamically evolved using SARSA empirical samples to obtain the final sparse network, including:

[0037] The initialized sparse network is trained using SARSA empirical samples, and the sparse network is dynamically evolved at a fixed update interval to obtain the updated sparse network.

[0038] Based on the weights of existing inter-layer sparse connections in the updated sparse network, the inter-layer sparse connections are adaptively discarded to obtain the removal set.

[0039] Based on the gradients of non-existent inter-layer sparse connections in the updated sparse network, gradient growth is performed on the inter-layer sparse connections to obtain the growth set; the number of inter-layer sparse connections in the removal set and the growth set are equal.

[0040] The sparse topology of the sparse network is updated by removing and growing sets to obtain a new sparse network. The new sparse network is then iteratively trained until it converges, and the converged sparse network is taken as the final sparse network.

[0041] Furthermore, the probability of the weights of existing inter-layer sparse connections is:

[0042] ;

[0043] in, The probability of the weights of existing sparse connections between layers; represents the weights of existing sparse connections between layers; k represents the layer variables of the sparse network. and These represent the number of neurons in the k-th and +1-th layers, respectively.

[0044] The formula for calculating overall sparsity is:

[0045] ;

[0046] in, K represents the overall sparsity; K represents the total number of layers in the sparse network.

[0047] The update ratio of inter-layer sparse connections is:

[0048] ;

[0049] in, The update ratio for sparse connections between layers; The initial update ratio is denoted by cos; cos is the cosine function; π is the mathematical constant pi; and t is the time variable. This represents the total number of iterations.

[0050] Remove the following set:

[0051] ;

[0052] in, To remove a set; ArgTopK selects the K connections with the largest absolute gradient values; The parameters are the connection weights connecting neuron i in the input layer and neuron j in the output layer; i is the input layer variable; j is the output layer variable. Let be the number of existing connections to be removed at time t in layer k.

[0053] The gradient of non-existent inter-layer sparse connections is:

[0054] ;

[0055] in, The gradient represents the non-existent sparse inter-layer connections. and These are the outputs of neuron i in layer k and neuron j in layer (K+1), respectively. Find the partial derivative with respect to the loss function L;

[0056] The growth set is:

[0057] ;

[0058] in, For the growth set; ArgTopK selects the K connections with the largest absolute gradient values; For the set of parameters excluding the set to be removed; To take the absolute value.

[0059] Furthermore, it also includes collaborative energy management learning of the energy management model through information interaction between multiple edge devices, including:

[0060] Each edge device uses SARSA experience samples in its locally stored experience replay pool to train a local sparse network and obtain the local sparse network weights for each edge device.

[0061] The server maps the weight values ​​of the local sparse network weights to the inter-layer sparse connection states of each local sparse network, and retains the non-zero inter-layer sparse connection weights based on the inter-layer sparse connection states to obtain the local effective weights.

[0062] By aggregating the effective weights of each local sparse network, the global network weights of the global network model are obtained.

[0063] Perform random singular value decomposition on the global network weights to obtain the decomposition results; the decomposition results include the left singular matrix, the singular value matrix, and the right singular matrix;

[0064] Based on the decomposition results, an approximate weight matrix is ​​obtained by approximating the weights.

[0065] Each edge device updates its local sparse network based on the approximate weight matrix to obtain a new local sparse network. The new local sparse network is then iteratively trained until the global network model converges, resulting in the final global network model.

[0066] Furthermore, the formula for calculating the global network weights is as follows:

[0067] ;

[0068] in, denoted as global network weights; k represents the layer variables of the sparse network; i represents the variables connecting the input layer; j represents the variables connecting the output layer; e represents the edge device variables; and E represents the total number of edge devices. The weights obtained from training the e-th device; To take the absolute value; sgn is the sign function;

[0069] Random singular value decomposition includes:

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] in, This is the network parameter matrix; It is a left singular matrix; It is a diagonal matrix; Let T be a right singular matrix; T is the transpose of the matrix. This is the projected network parameter matrix; for The left singular matrix; To select the L smallest values; n is the number of eigenvalues ​​to retain; L is the number of eigenvalues ​​to retain; For eigenvalues; It is a basis matrix; The distortion parameter is between 0 and 1; It is the L1 norm; This is an approximate network parameter matrix;

[0075] The formula for calculating the approximate weight matrix is:

[0076] ;

[0077] in, This is an approximate weight matrix; To preserve the left singular matrix with L eigenvalues; To preserve the L eigenvalues ​​of the diagonal matrix.

[0078] The present invention also proposes an energy management system based on sparse federated reinforcement learning, which utilizes the above-described energy management method based on sparse federated reinforcement learning, including an edge device and a server; the edge device includes an information acquisition module, a model training module, a policy generation module, and a management module;

[0079] The information acquisition module is used to collect local energy consumption information at each time step and receive real-time electricity price information from the power company;

[0080] The model training module is used to build an initial energy management model and train the initial energy management model with the server through federated learning to obtain the final energy management model;

[0081] The strategy generation module is used to generate energy management strategies by executing intelligent energy management algorithms through an energy management model based on energy consumption information and real-time electricity price information.

[0082] The management module is used to generate control commands for installed home appliances through energy management strategies, and to manage the installed home appliances through these control commands.

[0083] The present invention has the following advantages and beneficial effects:

[0084] This invention innovatively introduces a network sparsity mechanism based on topology evolution. During federated training, this mechanism dynamically and adaptively prunes low-contribution connections in the model, thereby evolving a dense model structure into a sparse one. The principle is that a sparse model only needs to store non-zero parameters and their indices, resulting in an order-of-magnitude reduction in static memory usage compared to a dense model that needs to store the entire parameter matrix. Therefore, this invention reduces the hardware resource requirements of a single node from the structural level of the model itself, enabling complex reinforcement learning-based home energy management algorithms that were previously impossible to deploy due to insufficient memory to run successfully on low-cost, resource-constrained edge devices, greatly expanding the applicability of the technology.

[0085] This invention addresses this problem through a collaborative design of sparsity reduction and communication compression. First, based on the aforementioned sparse model structure, devices only need to upload updates of non-zero parameters during communication, resulting in a significantly smaller data packet size compared to dense models. Second, this invention further integrates communication compression technology to efficiently encode these sparse update data. This dual optimization mechanism of "sparse first, then compressed" minimizes the amount of data transmitted in a single communication. The direct effect is reduced communication latency and network congestion risk, thereby accelerating the convergence process of federated aggregation and improving the training efficiency and stability of the entire distributed system in weak network environments. Attached Figure Description

[0086] Figure 1 is an exemplary flowchart of an energy management method based on sparse federated reinforcement learning proposed in this invention;

[0087] Figure 2 is an exemplary schematic diagram of the updated sparse network provided by the present invention;

[0088] Figure 3 is an exemplary schematic diagram of compressed federated learning provided by the present invention. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0090] Electricity system users all expect to reduce their electricity bills through energy management services. Typically, residential users are equipped with various distributed energy resources, including photovoltaic systems, energy storage systems, electric vehicles, and flexible appliances such as air conditioners and washing machines. Residential users can meet their electricity needs through photovoltaic power generation, energy storage discharge, and purchasing electricity from the grid. Generally, the most economical electricity consumption pattern for users is to operate appliances or charge energy storage during off-peak hours and utilize surplus electricity from energy storage systems and photovoltaic power generation during peak hours.

[0091] To manage local energy resources, a home energy management system deployed on edge devices such as smart meters can transform external information flows into device-level control flows, thereby scheduling the power flow within the home. Specifically, as shown in Figure 1, the edge device collects local energy consumption information at each time step and receives real-time electricity price information from the power company. The edge device also builds an initial energy management model and trains it to obtain a final energy management model. Subsequently, based on the energy consumption information and real-time electricity price information, the edge device executes an intelligent energy management algorithm through the energy management model to determine control commands for installed home appliances through energy management strategies. The installed home appliances are then managed through these control commands. The edge home energy management system aims to optimize energy consumption to reduce electricity bills while also considering user comfort. Therefore, neighboring home users with similar behavioral preferences can learn collaborative energy management through information interaction between multiple edge devices.

[0092] In some embodiments, an initial energy management model can be constructed using a reinforcement learning algorithm, and the initial energy management model can be trained to obtain a final energy management model, including:

[0093] The energy management problem can be formulated as a Markov decision process, which includes a state space, an action space, a state transition probability function, and a reward function. A Markov decision process can be... ,in For state space, For the action space, Let r be the state transition probability function, and r be the reward function. This is the decay factor. The system agent determines the optimal strategy by interacting with the unknown and uncertain environment. This strategy maps states to probability distributions over actions. Specifically, at time step t, the agent observes the state... And select an action The environment then returns an instant reward. And transition to the next state. The following sections will elaborate on the mathematical expressions of states, actions, rewards, state transitions, and action value functions.

[0094] Construct an initial energy management model; the initial energy management model includes an initial actuator network and an initial evaluator network; input the current observation state into the actuator network, and output the optimal continuous action; concatenate the current observation state and the optimal continuous action, and input the concatenated concatenation into the evaluator network to obtain the initial discrete action, and then... A greedy strategy selects the initial discrete actions to obtain the optimal discrete actions. Based on the optimal continuous actions and the optimal discrete actions, an optimal hybrid action is constructed. Based on the optimal hybrid action, the energy management model is iteratively trained using the hybrid DQN algorithm until the energy management model converges, thus obtaining the final energy management model.

[0095] In some embodiments, based on the optimal hybrid action, the energy management model is iteratively trained using a hybrid DQN algorithm to obtain the final energy management model. This includes: executing the optimal hybrid action, returning an immediate reward after execution, transitioning to a new observation state, and constructing SARSA empirical samples; generating multi-step objectives based on the SARSA empirical samples using the energy management model, constructing the loss function of the evaluator network using the expected mean square error between the value estimate of the optimal hybrid action and the multi-step objectives, minimizing the Bellman error using gradient descent, and updating the evaluator network; based on the SARSA empirical samples and the updated evaluator network, applying the chain rule to the expected value of the optimal hybrid action using a deterministic policy gradient to construct the loss function of the executor network, performing a single-step update using a dual-timescale update rule, and updating the executor network; updating the parameters of the energy management model using a smooth soft update method; the energy management model includes an executor network and an evaluator network; iteratively training the energy management model, updating the network parameters until the energy management model converges, and obtaining the final energy management model.

[0096] Specifically, for processing hybrid discrete-continuous action spaces This paper proposes a hybrid DQN algorithm, which combines the advantages of traditional reinforcement learning algorithms DQN and DDPG, without requiring approximation or relaxation processing. The agent updates continuous actions through deterministic policy gradients. The actuator network updates the optimal discrete action through Q-learning. The evaluator network. The deterministic action network uses the current observation state. Given the input, output the optimal continuous action. The observation status will then be... With continuous action By concatenating the input evaluator network, the optimal discrete action can be obtained. .use A greedy strategy is used for exploration, based on probability. Choose random actions. The value estimate of the optimal mixed actions can be calculated using the Bellman equation:

[0097] ;

[0098] in, Estimate the value of the optimal mixed action at time t; and The observation states at time t and time t+1 are respectively; and The optimal hybrid actions at time t and time t+1 are respectively; To obtain the expected value; γ is the instantaneous reward at time t; γ is the reward decay factor; max is the maximum value. The action value function given the policy π; and These are the optimal discrete actions at times t and t+1, respectively. For optimal continuous action; θ represents the actuator network parameters; θ represents the evaluator network parameters.

[0099] The evaluator network is updated using gradient descent to minimize the Bellman error. To avoid instability and divergence in value function estimation, a multi-step objective is generated by two independent objective networks. :

[0100] ;

[0101] in, For multi-step objectives; In the state Next action The instant reward received; For target actuator networks based on weights In state The action to be output; For state Corresponding feature representation; and These are the weight parameters for the target actuator network and the target evaluator network, respectively. The loss function of the evaluator network is defined as the expected mean square error between the optimal action value estimate and the multi-step objective:

[0102] ;

[0103] in, Let be the loss function of the evaluator network; This represents the optimal continuous action.

[0104] The loss function of the actuator network is updated through gradients of a deterministic policy. The expected value of a mixture of actions is calculated using the chain rule:

[0105] ;

[0106] in, Let z be the loss function of the actuator network; z be the discrete action; and Z be the number of discrete actions. A dual-time-scale update rule is used to achieve collaborative updating of the action network and the evaluator network: through approximation... calculate When applying gradients, only negligible single-step updates are performed. An empirical replay pool is used to eliminate sample inconsistencies caused by policy changes; its storage space follows a replacement principle (discarding the oldest sample when the space is full). The target network update uses smooth soft updates instead of direct copying.

[0107] To alleviate the memory pressure on edge devices caused by real-time variables, this invention proposes a sparse evolutionary network training method suitable for energy management systems. Inspired by biological neural networks, this method replaces conventional fully connected layers with sparse layers throughout the training process, significantly reducing the number of network parameters. In the initial stage, a random sparse network with a specific sparsity is determined. Subsequently, gradient values ​​are calculated and connection weights are updated through training. Based on the weight information, some existing connections are discarded, while an equal number of new connections are activated according to the gradient information. The sparse network constructed in this way can achieve continuous evolution through self-regulation. Sparse training includes four main stages: network initialization, network training, adaptive dropout, and gradient-guided growth, as shown in Figure 2. For ease of explanation, we will use network weights... With critics' network weight Collectively referred to as .

[0108] In some embodiments, an initial energy management model can be constructed using a sparse network, including: constructing a sparse topology of the initial sparse network by allocating sparse connections between layers to obtain an initialized sparse network; the sparse topology includes the overall sparsity of the sparse network, the total number of network layers, and sparse connections between layers; dynamically evolving the sparse network using SARSA empirical samples to obtain the final sparse network; and replacing the fully connected layers of the initial actuator network and the initial evaluator network with the final sparse network to obtain the initial energy management model.

[0109] Specifically, a sparse topology is constructed by allocating sparse connections between layers. The overall sparsity of the sparse network is defined. This represents the ratio of the number of unconnected parameters to the total number of parameters. The initial connections follow the ER (Erdös–Rényi) random graphical model. Weighted connections exist between neurons in layer k and layer (k+1). The probability is expressed as:

[0110] ;

[0111] in, The probability of the weights of existing sparse connections between layers; represents the weights of existing sparse connections between layers; k represents the layer variables of the sparse network. and These represent the number of neurons in the k-th and (k+1)-th layers, respectively. The number of connections per layer scales linearly with the sum of the input and output channels. Only the weights of the connected layers and their derived variables are computed and stored. Because... and Since most neurons do not have weighted connections, sparse training reduces the space complexity of the k-th layer from... ( · ) down to

[0112] ( + This significantly reduces computational and memory overhead. Assuming a sparse network has K layers, its overall sparsity can be expressed as:

[0113] ;

[0114] in, K represents the overall sparsity; K is the total number of layers in the sparse network; where This represents the sparsity of the k-th layer. The bias parameter has a negligible impact on the network size, and each layer still maintains dense connections.

[0115] While sparse networks are highly efficient to train, randomly generated topologies often fail to achieve ideal performance in specific energy management tasks. Therefore, this invention proposes a dynamic evolution strategy: at a fixed update interval... Within the framework, sparse topology connections are optimized with constant overall sparsity. The sparse topology connections of the sparse network are dynamically evolved using SARSA empirical samples to obtain the final sparse network. This includes: training the initialized sparse network using SARSA empirical samples, and dynamically evolving the sparse network at a fixed update interval to obtain the updated sparse network.

[0116] Based on the weights of existing inter-layer sparse connections in the updated sparse network, adaptive discarding of these connections yields a removal set. The sparse network discards the connections with the lowest importance (closest to zero), i.e., the smallest positive weight and the largest negative weight, at a discard ratio of [ratio to be inserted here]. These removed weights are not expected to significantly impact network performance. To enhance the stability of dynamic evolution, the update ratios are adaptively annealed as the convergence process progresses:

[0117] ;

[0118] in, The update ratio for sparse connections between layers; The initial update ratio is denoted by cos; cos is the cosine function; π is the mathematical constant pi; and t is the time variable. This represents the total number of iterations. The update ratio is calculated as the agent network gradually converges to the optimal solution. Approaching zero, the topology remains unchanged to focus on updating the weight parameters. For the parameter connections between input layer neuron i and output layer neuron j... Remove the set by selecting the top k connections with the lowest absolute weight.

[0119] ;

[0120] in, To remove a set; ArgTopK selects the K connections with the largest absolute gradient values; The parameters are the connection weights connecting neuron i in the input layer and neuron j in the output layer; i is the input layer variable; j is the output layer variable. Let be the number of existing connections to be removed at time t in layer k; where This represents the number of existing connections to be removed at layer k at time t, and its value depends on the update ratio. .

[0121] Based on the gradients of non-existent inter-layer sparse connections in the updated sparse network, gradient-guided growth is performed on the inter-layer sparse connections to obtain a growth set; the number of inter-layer sparse connections in the removal set and the growth set are equal. The sparse network selects the positions with the highest importance (maximum positive gradient and minimum negative gradient) from the non-existent connections and distributes them proportionally. New connections are grown. These newly generated weights are highly sensitive to reducing the loss in subsequent iterations. However, to save memory, gradient values ​​of non-existent connections are not stored during training. Therefore, we compute gradients by combining neuron outputs and their gradients using the chain rule:

[0122] ;

[0123] in, The gradient represents the non-existent sparse inter-layer connections. and These are the outputs of neuron i in layer k and neuron j in layer (K+1), respectively. Find the partial derivative of the loss function L. The growth set selects the k connections with the largest absolute gradient values ​​outside the removal set.

[0124] The growth set is:

[0125] ;

[0126] in, For the growth set; ArgTopK selects the K connections with the largest absolute gradient values; For the set of parameters excluding the set to be removed; To obtain the absolute value, the above drop-out and growth operations are applied layer by layer to evolve the network topology. The weights of newly formed connections are initialized to zero, and then the network enters an iterative training phase of updating the weight parameters.

[0127] The sparse topology of the sparse network is updated by removing and growing sets to obtain a new sparse network. The new sparse network is then iteratively trained until it converges, and the converged sparse network is taken as the final sparse network.

[0128] In some embodiments, due to the limited memory space of edge devices, the capacity of the experience replay buffer can only be set relatively small. This buffer constraint often leads to insufficient exploration of the state-action space and suboptimal utilization of historical experience, affecting algorithm performance and convergence. To address this, this invention also proposes a compressed federated learning framework, which performs collaborative energy management learning of the energy management model through information interaction among multiple edge devices, thereby utilizing the transfer data cached by distributed edge devices to train a high-performance global network. As shown in Figure 3, this framework can be viewed as... Devices share a merged memory pool without compromising privacy. The specific process is as follows:

[0129] Each edge device uses SARSA experience samples from its locally stored experience replay pool to train a local sparse network, obtaining the local sparse network weights for each edge device. The server maps the weight values ​​of the local sparse network to the inter-layer sparse connection states of each local sparse network, retaining non-zero inter-layer sparse connection weights based on the inter-layer sparse connection states to obtain the local effective weights; the server aggregates the local effective weights of each local sparse network to obtain the global network weights of the global network model.

[0130] Specifically, each edge device uses locally stored transfer data to train the sparse network. In each round, the sparse network weights are uploaded to the server to update the global network weights. :

[0131] ;

[0132] in, denoted as global network weights; k represents the layer variables of the sparse network; i represents the variables connecting the input layer; j represents the variables connecting the output layer; e represents the edge device variables; and E represents the total number of edge devices. The weights obtained from training the e-th device; To take the absolute value; sgn is the sign function. Since the discarded weights do not participate in the averaging process, the formula introduces a sign function. This method aggregates only non-zero connection weights by mapping weight values ​​to connection states. However, the high dimensionality of the weight matrix leads to significant communication overhead during the upload and distribution process. Based on the low-rank characteristic of sparse networks, this invention also introduces truncated singular value decomposition (SVD) for matrix compression during weight exchange, comprising two steps: random singular value decomposition and weight approximation.

[0133] Performing random singular value decomposition (RSD) on the global network weights yields the decomposition results, which include the left singular matrix, singular value matrix, and right singular matrix. Specifically, the random singular value decomposition involves the weight matrix... for tensor ( and (These represent the number of input / output channels, respectively). SVD (Singular Value Decomposition) can decompose a large-scale sparse matrix into several smaller dense matrices.

[0134] ;

[0135] in and It is an orthogonal matrix containing left and right singular vectors; It is a diagonal matrix, and its elements are... for The descending singular values. To reduce the computational cost of SVD, noisy singular values ​​are introduced. Gaussian random matrix Project the weight matrix onto the subspace basis matrix ( (This represents the orthogonalization operation with exponential iteration). , basis matrix The size is much smaller .pass proxy matrix It can be used for small matrices Perform SVD:

[0136] ;

[0137] in, for The left singular matrix. Original weight matrix. The left singular matrix is ​​composed of The calculation yielded the result.

[0138] Although stochastic SVD reduces the matrix size, the communication cost of transmitting the decomposed matrix remains high. Therefore, a weight approximation is performed based on the decomposition results to obtain an approximate weight matrix. Specifically, the weight approximation involves truncating the decomposition matrix for dimensionality compression: only the first few elements are retained. By taking several large singular values ​​and their corresponding singular vectors, we obtain an approximate weight matrix:

[0139] ;

[0140] in, This is an approximate weight matrix; To preserve the left singular matrix with L eigenvalues; To preserve the L eigenvalues ​​of the diagonal matrix. The weight matrix size is changed from the original... Reduced to To control approximation error, an energy threshold is used. Constraint on truncating the number of singular values:

[0141] ;

[0142] Considering The upper bound of the error of the compressed SVD algorithm can be guaranteed to be:

[0143] ;

[0144] in, This is the network parameter matrix; It is a left singular matrix; It is a diagonal matrix; Let T be a right singular matrix; T is the transpose of the matrix. This is the projected network parameter matrix; for The left singular matrix; To select the L smallest values; n is the number of eigenvalues ​​to retain; L is the number of eigenvalues ​​to retain; For eigenvalues; It is a basis matrix; The distortion parameter is between 0 and 1; It is the L1 norm; This is an approximate network parameter matrix.

[0145] Each edge device updates its local sparse network based on the approximate weight matrix to obtain a new local sparse network. The new local sparse network is then iteratively trained until the global network model converges, resulting in the final global network model.

[0146] This invention also provides an energy management system based on sparse federated reinforcement learning, including an edge device and a server. The edge device includes an information acquisition module, a model training module, a policy generation module, and a management module. The information acquisition module collects local energy consumption information at each time step and receives real-time electricity price information from the power company. The model training module constructs an initial energy management model and trains it with the server through federated learning to obtain the final energy management model. The policy generation module executes an intelligent energy management algorithm based on the energy consumption information and real-time electricity price information through the energy management model to obtain an energy management policy. The management module generates control commands for installed home appliances based on the energy management policy and manages the installed home appliances through the control commands.

[0147] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An energy management method based on sparse federated reinforcement learning, characterized in that, include: Collect local energy consumption information at each time step and receive real-time electricity price information from the power company; The process involves constructing an initial energy management model and training it to obtain a final energy management model. This includes: transforming the energy management problem into a Markov decision process (MDF); the MDF includes a state space, action space, state transition probability function, and reward function; constructing the initial energy management model using a sparse network; the initial energy management model includes an initial actuator network and an initial evaluator network, which involves: constructing the sparse topology of the initial sparse network by allocating sparse connections between layers, resulting in an initialized sparse network; the sparse topology includes the overall sparsity of the sparse network, the total number of layers, and the sparse connections between layers; and dynamically evolving the sparse network using SARSA empirical samples to obtain the final sparse network, which involves: training the initialized sparse network using SARSA empirical samples and dynamically evolving the sparse network at a fixed update interval to obtain an updated sparse network. The network is structured as follows: Based on the weights of existing inter-layer sparse connections in the updated sparse network, adaptively discarding inter-layer sparse connections to obtain a removal set; based on the gradients of non-existent inter-layer sparse connections in the updated sparse network, performing gradient growth on inter-layer sparse connections to obtain a growth set; the number of inter-layer sparse connections in the removal set and the growth set are equal; the sparse topology of the sparse network is updated based on the removal set and the growth set to obtain a new sparse network, and the new sparse network is iteratively trained until the sparse network converges, and the converged sparse network is used as the final sparse network. The fully connected layers of the initial actuator network and the initial evaluator network are replaced with the final sparse network to obtain the initial energy management model; the current observation state is input into the actuator network, and the optimal continuous action is output; the current observation state and the optimal continuous action are concatenated and input into the evaluator network to obtain the initial discrete action, and then processed... A greedy strategy selects the initial discrete action to obtain the optimal discrete action. Based on the optimal continuous action and the optimal discrete action, an optimal hybrid action is constructed. Then, based on the optimal hybrid action, the energy management model is iteratively trained using a hybrid DQN algorithm until the energy management model converges, resulting in the final energy management model. This includes: executing the optimal hybrid action, returning an immediate reward after execution, transitioning to a new observation state, and constructing SARSA empirical samples; based on the SARSA empirical samples, generating multi-step objectives through the energy management model, constructing the loss function of the evaluator network using the expected mean square error between the value estimate of the optimal hybrid action and the multi-step objective, minimizing the Bellman error using gradient descent, and completing the evaluator network update; based on the SARSA empirical samples and more... The newly redesigned evaluator network uses a deterministic policy gradient to construct the loss function of the actuator network by applying a chain rule to the expected value of the optimal mixed action. Combined with a dual-timescale update rule, it performs a single-step update to complete the actuator network update. A smooth soft update method is used to update the parameters of the energy management model. The energy management model includes both the actuator network and the evaluator network. The energy management model is iteratively trained, updating its network parameters until convergence, resulting in the final energy management model. Based on energy consumption information and real-time electricity price information, an intelligent energy management algorithm is executed through the energy management model to obtain an energy management strategy. Control commands for installed home appliances are generated using the energy management strategy, and these commands are then used to manage the installed home appliances.

2. The energy management method based on sparse federated reinforcement learning according to claim 1, characterized in that, The formula for calculating the value estimate of the optimal mixed action is: ;in, Estimate the value of the optimal mixed action at time t; and The observation states at time t and time t+1 are respectively; and The optimal hybrid actions at time t and time t+1 are respectively; To obtain the expected value; γ is the instantaneous reward at time t; γ is the reward decay factor; max is the maximum value. The action value function given the policy π; and These are the optimal discrete actions at times t and t+1, respectively. For optimal continuous action; θ represents the parameters of the actuator network; θ represents the parameters of the evaluator network; the loss function of the evaluator network is: ;in, Let be the loss function of the evaluator network; For optimal continuous actions, the loss function of the actuator network is: ;in, Let z be the loss function of the actuator network; z be the discrete action; and Z be the number of discrete actions.

3. The energy management method based on sparse federated reinforcement learning according to claim 1, characterized in that, The probability of the weights of existing inter-layer sparse connections is: ;in, The probability of the weights of existing sparse connections between layers; represents the weights of existing sparse connections between layers; k represents the layer variables of the sparse network. and These represent the number of neurons in the k-th and +1-th layers, respectively; the formula for calculating the overall sparsity is: ;in, K represents the overall sparsity; K is the total number of layers in the sparse network; the update ratio of sparse connections between layers is: ;in, The update ratio for sparse connections between layers; The initial update ratio is denoted by cos; cos is the cosine function; π is the mathematical constant pi; and t is the time variable. This represents the total number of iterations; the set to be removed is: ;in, To remove a set; ArgTopK selects the K connections with the largest absolute gradient values; The parameters are the connection weights connecting neuron i in the input layer and neuron j in the output layer; i is the input layer variable; j is the output layer variable. Let be the number of existing connections to be removed in layer k at time t; the gradient of the sparse connections between non-existent layers is: ;in, The gradient represents the non-existent sparse inter-layer connections. and These are the outputs of neuron i in layer k and neuron j in layer (K+1), respectively. To find the partial derivative of the loss function L; the growth set is: ;in, For the growth set; ArgTopK selects the K connections with the largest absolute gradient values; For the set of parameters excluding the set to be removed; To take the absolute value.

4. The energy management method based on sparse federated reinforcement learning according to claim 1, characterized in that, It also includes collaborative energy management learning of the energy management model through information interaction among multiple edge devices, including: each edge device uses SARSA experience samples in its locally stored experience replay pool to train a local sparse network and obtain the local sparse network weights corresponding to each edge device; the server maps the weight values ​​of the local sparse network weights to the inter-layer sparse connection states of each local sparse network, retains non-zero inter-layer sparse connection weights based on the inter-layer sparse connection states, and obtains the local effective weights; the local effective weights of each local sparse network are aggregated to obtain the global network weights of the global network model; random singular value decomposition is performed on the global network weights to obtain the decomposition results; the decomposition results include the left singular matrix, the singular value matrix, and the right singular matrix; weight approximation is performed based on the decomposition results to obtain the approximate weight matrix; each edge device updates its local sparse network according to the approximate weight matrix to obtain a new local sparse network, and iteratively trains the new local sparse network until the global network model converges to obtain the final global network model.

5. The energy management method based on sparse federated reinforcement learning according to claim 4, characterized in that, The formula for calculating global network weights is: ;in, denoted as global network weights; k represents the layer variables of the sparse network; i represents the variables connecting the input layer; j represents the variables connecting the output layer; e represents the edge device variables; and E represents the total number of edge devices. The weights obtained from training the e-th device; To take the absolute value; sgn is the sign function; random singular value decomposition includes: ； ； ； ;in, This is the network parameter matrix; It is a left singular matrix; It is a diagonal matrix; Let T be a right singular matrix; T is the transpose of the matrix. This is the projected network parameter matrix; for The left singular matrix; To select the L smallest values; i represents the input layer variables; L represents the number of features to retain; For eigenvalues; It is a basis matrix; The distortion parameter is between 0 and 1; It is the L1 norm; The approximate network parameter matrix is ​​given by the formula for calculating the approximate weight matrix. ;in, This is an approximate weight matrix; To preserve the left singular matrix with L eigenvalues; To preserve the L eigenvalues ​​of the diagonal matrix.

6. A system for an energy management method based on sparse federated reinforcement learning, utilizing the energy management method based on sparse federated reinforcement learning as described in any one of claims 1-5, characterized in that, It includes edge devices and servers; the edge devices include an information acquisition module, a model training module, a policy generation module, and a management module; the information acquisition module is used to collect local energy consumption information at each time step and receive real-time electricity price information from the power company; The model training module is used to build an initial energy management model and train the initial energy management model with the server through federated learning to obtain the final energy management model; The strategy generation module is used to generate energy management strategies by executing intelligent energy management algorithms through an energy management model based on energy consumption information and real-time electricity price information. The management module is used to generate control commands for installed home appliances through energy management strategies, and to manage the installed home appliances through these control commands.

Images