Energy management method and system based on sparse federal reinforcement learning

Abstract

The invention belongs to the field of household energy management, and relates to an energy management method and system based on sparse federal reinforcement learning, and the method comprises the steps: collecting local energy consumption information at each time step, and receiving real-time electricity price information from an electric power company; constructing an initial energy management model, and training the initial energy management model to obtain a final energy management model; based on the energy consumption information and the real-time electricity price information, executing an intelligent energy management algorithm through an energy management model to obtain an energy management strategy; according to the energy management strategy, a control instruction for the installed household equipment is generated, and the installed household equipment is managed according to the control instruction; the problem of memory bottleneck of edge equipment is fundamentally solved, localized deployment of a complex algorithm is realized, the communication overhead of federal learning is remarkably reduced, and the system training efficiency and stability are improved.

Description

Technical Field

[0001] This invention relates to the field of home energy management, and specifically discloses an energy management method and system based on sparse federated reinforcement learning. Background Technology

[0002] In recent years, with the global pursuit of sustainable development and energy efficiency, the importance of home energy management has become increasingly prominent. Home energy management systems can not only effectively reduce household energy consumption and electricity costs, but also help alleviate the pressure on the power grid and promote the integration of renewable energy into the grid. Traditional centralized management suffers from high data transmission latency and significant privacy risks, making it difficult to meet real-time control requirements. Against this backdrop, edge-side home energy management systems have become a key solution to the challenges of traditional centralized management. By offloading real-time data processing and device control tasks to edge devices, this system can provide more personalized energy-saving suggestions and efficient demand response mechanisms, achieving flexible real-time resource control. Furthermore, because sensitive data is stored locally on devices, the system can significantly improve user privacy and security. However, the core challenge of this solution lies in the fact that the limited memory, computing, and storage resources of edge devices severely restrict the local deployment and training of complex models, becoming a key bottleneck in achieving efficient and intelligent energy management.

[0003] Current research on home energy management mainly revolves around two paradigms: rule-based optimization methods and model-free reinforcement learning. While traditional optimization methods (such as model predictive control) are widely used, their strong pre-defined nature (relying on fixed objective functions and constraints) leads to decision failures in uncertain scenarios such as dynamic electricity prices, intermittent renewable energy output, and load variations. Simultaneously, the high computational complexity of solving high-dimensional optimization problems severely restricts the system's real-time response capabilities. To overcome this bottleneck, model-free reinforcement learning has emerged as an emerging solution due to its characteristic of not requiring precise environmental modeling. Through autonomous learning, it achieves near-optimal control in complex dynamic environments, exhibiting stronger adaptability and robustness. However, existing home energy management frameworks require each participating edge device to deploy and train a dense reinforcement learning model. For smart home terminals with extremely limited memory capacity, even a model of moderate complexity may exceed their hardware capacity limits, leading to deployment failure or operational crashes. Furthermore, in federated learning, frequent exchange of model parameter updates and gradients is required between the client and server. For dense models, the amount of data updated each time is enormous, consuming significant home network bandwidth and increasing communication latency.

[0004] In view of this, the present invention provides an energy management method and system based on sparse federated reinforcement learning, which fundamentally solves the memory bottleneck problem of edge devices, realizes the localized deployment of complex algorithms, significantly reduces the communication overhead of federated learning, and improves the system training efficiency and stability. SUMMARY

[0005] The purpose of the present application is to provide a sparse federated reinforcement learning-based energy management method and system, which solves the problem of solving the memory bottleneck problem of edge devices, realizing the localization deployment of complex algorithms, and significantly reducing the communication overhead of federated learning, improving the system training efficiency and stability; The specific scheme is as follows: A sparse federated reinforcement learning-based energy management method, comprising: Collecting local energy consumption information at each time step and receiving real-time electricity price information from the power company; Building an initial energy management model and training the initial energy management model to obtain a final energy management model; Based on the energy consumption information and real-time electricity price information, an intelligent energy management algorithm is executed through the energy management model to obtain an energy management strategy; Through the energy management strategy, control instructions for the installed household devices are generated, and the installed household devices are managed through the control instructions.

[0006] Further, the initial energy management model is built and trained to obtain the final energy management model, comprising: Convert the energy management problem into a Markov decision process; The Markov decision process includes state space, action space, state transition probability function and reward function; Building an initial energy management model; The initial energy management model includes an initial actuator network and an initial critic network; Input the current observation state into the actuator network to output the optimal continuous action; The current observation state and the optimal continuous action are spliced and input into the critic network to obtain the initial discrete action, and the optimal discrete action is selected through The greedy strategy selects the initial discrete action to obtain the optimal discrete action; Based on the optimal continuous action and the optimal discrete action, an optimal hybrid action is constructed, and based on the optimal hybrid action, the energy management model is iteratively trained through the hybrid DQN algorithm until the energy management model converges, obtaining the final energy management model.

[0007] Further, based on the optimal hybrid action, the energy management model is iteratively trained through the hybrid DQN algorithm to obtain the final energy management model, comprising: Execute the optimal hybrid action, return the immediate reward after executing the action and transition to the new observation state, and build the SARSA experience sample; Based on the SARS A experience sample, a multi-step target is generated through an energy management model, a loss function of a critic network is constructed by taking the expected mean square error between the value estimate of the optimal mixed action and the multi-step target, a Bellman error is minimized by using a gradient descent method, and critic network updating is completed; Based on the SARS A experience sample and the updated critic network, a loss function of an actor network is constructed by applying the chain rule to the expected value of the optimal mixed action through a deterministic policy gradient, a single-step update is performed in combination with a double-time-scale update rule, and actor network updating is completed; The parameters of the energy management model are updated in a smooth soft update manner; the energy management model includes an actor network and a critic network; The network parameters of the energy management model are updated through iterative training of the energy management model until the energy management model converges, and a final energy management model is obtained.

[0008] Further, the calculation formula of the value estimate of the optimal mixed action is: ; Among them, is the value estimate of the optimal mixed action at time t; and are the observed states at time t and time t+1 respectively; and are the optimal mixed actions at time t and time t+1 respectively; is to take the expected value; is the immediate reward at time t; γ is the reward decay factor; max is to take the maximum value; is the action value function of the given policy π; and are the optimal discrete actions at time t and time t+1 respectively; is the optimal continuous action; is the actor network parameter; θ is the critic network parameter; The loss function of the critic network is: ; Among them, is the loss function of the critic network; is the optimal continuous action; The loss function of the actor network is: ; Among them, is the loss function of the actor network; z is the discrete action; Z is the number of discrete actions.

[0009] Further, an initial energy management model is constructed through a sparse network, including: The sparse topology of the initial sparse network is constructed by assigning sparse connections between layers to obtain an initialized sparse network; the sparse topology includes the overall sparsity of the sparse network, the total number of layers of the network, and the sparse connections between layers; The final sparse network is obtained by dynamically evolving the sparse network through the SARSA experience sample; The fully connected layers of the initial actor network and the initial critic network are replaced by the final sparse network to obtain an initial energy management model.

[0010] Further, the final sparse network is obtained by dynamically evolving the sparse network through the SARSA experience sample, including: The initialized sparse network is trained through the SARSA experience sample, and the sparse network is dynamically evolved at a fixed update interval to obtain an updated sparse network; The existing sparse connections between layers in the updated sparse network are adaptively discarded based on the weights of the existing sparse connections between layers to obtain a removal set; The non-existing sparse connections between layers in the updated sparse network are grown based on the gradients of the non-existing sparse connections between layers to obtain a growth set; the number of sparse connections between layers in the removal set and the growth set is equal; The sparse topology of the sparse network is updated based on the removal set and the growth set to obtain a new sparse network, and the new sparse network is iteratively trained until the sparse network converges, and the converged sparse network is taken as the final sparse network.

[0011] Further, the probability of the weight of the existing sparse connection between layers is: ; Wherein, is the probability of the weight of the existing sparse connection between layers; is the weight of the existing sparse connection between layers; k is the layer variable of the sparse network; and are the number of neurons of the kth layer and the +1th layer, respectively; The calculation formula of the overall sparsity is: ; Wherein, is the overall sparsity; K is the total number of layers of the sparse network; The update ratio of the sparse connection between layers is: ; Wherein, is the update ratio of the sparse connection between layers; is the initial update ratio; cos is the cosine function; π is the circular constant; t is the time variable; is the total number of iterations; The removal set is: ; Wherein, is the removal set; ArgTopK is the top K connections with the largest absolute gradient values; is the parameter connection weight of the connection between the input layer i neuron and the output layer j neuron; i is the connection input layer variable; j is the connection output layer variable; is the number of existing connections to be removed in the kth layer at time t; The gradient of the non-existing inter-layer sparse connection is: ; Wherein, is the gradient of the non-existing inter-layer sparse connection; and are the outputs of the i neuron in the kth layer and the j neuron in the K+1th layer, respectively; is the partial derivative with respect to the loss function L; The growth set is: ; Wherein, is the growth set; ArgTopK is the top K connections with the largest absolute gradient values; is the parameter set except the removal set; is the absolute value.

[0012] Further, the energy management model is also cooperatively learned by the energy management learning through information interaction between the multiple edge devices, including: Each edge device trains the local sparse network by using the SARSA experience samples in the experience replay pool stored locally, to obtain the local sparse network weight corresponding to each edge device; The server maps the weight values of the local sparse network weight to the inter-layer sparse connection state of each local sparse network, and retains the non-zero inter-layer sparse connection weight based on the inter-layer sparse connection state, to obtain the local effective weight; The local effective weights of all local sparse networks are aggregated to obtain the global network weight of the global network model; The global network weight is subjected to random singular value decomposition to obtain a decomposition result; the decomposition result includes a left singular matrix, a singular value matrix, and a right singular matrix; Based on the decomposition result, weight approximation processing is performed to obtain an approximate weight matrix; Each edge device updates the local sparse network according to the approximate weight matrix to obtain a new local sparse network, and iteratively trains the new local sparse network until the global network model converges, to obtain the final global network model.

[0013] Further, the calculation formula of the global network weight is: ; Wherein, is the global network weight; k is the layer variable of the sparse network; i is the connection input layer variable; j is the connection output layer variable; e is the edge device variable; E is the total number of edge devices; is the weight trained by the e-th device; is the absolute value; sgn is the sign function; The random singular value decomposition includes: ; ; ; ; Wherein, is the network parameter matrix; is the left singular matrix; is the diagonal matrix; is the right singular matrix; T is the transpose of the matrix; is the projected network parameter matrix; is the left singular matrix of ; is the L smallest values; n is the reserved eigenvalue variable; L is the number of reserved eigenvalues; is the eigenvalue; is the basis matrix; is the distortion parameter between 0 and 1; is the L1 norm; is the approximate network parameter matrix; The calculation formula of the approximate weight matrix is: ; Wherein, is the approximate weight matrix; is the left singular matrix of the reserved L eigenvalues; is the diagonal matrix of the reserved L eigenvalues.

[0014] The application further provides a sparse federated reinforcement learning-based energy management method system using the sparse federated reinforcement learning-based energy management method, which comprises an edge device and a server; the edge device comprises an information acquisition module, a model training module, a policy generation module and a management module; The information acquisition module is used for collecting local energy consumption information at each time step, and receiving real-time electricity price information from a power company; The model training module is configured to construct an initial energy management model and train the initial energy management model through federated learning with the server to obtain a final energy management model. The strategy generation module is configured to execute an intelligent energy management algorithm through the energy management model based on energy consumption information and real-time electricity price information to obtain an energy management strategy. The management module is configured to generate a control instruction for the installed household device through the energy management strategy and manage the installed household device through the control instruction.

[0015] The present application has the following advantages and beneficial effects: The present application creatively introduces a network sparsification mechanism based on topology evolution. This mechanism can dynamically and adaptively prune the connections with low contribution in the model during federated training, thereby evolving a dense model structure into a sparse structure. The principle is that a sparse model only needs to store non-zero parameters and their indices, and compared with a dense model that needs to store the entire parameter matrix, the static memory occupation is reduced by orders of magnitude. Therefore, by this way, the present application reduces the requirement for single node hardware resources from the structure level of the model itself, so that the complex reinforcement learning home energy management algorithm that cannot be deployed due to insufficient memory can be successfully run on low-cost and resource-limited edge devices, greatly expanding the application scope of the technology.

[0016] The present application solves this problem through the cooperative design of sparsification and communication compression. First, based on the sparse model structure described above, the device only needs to upload the update amount of non-zero parameters when communicating, and the data packet size is naturally much smaller than that of the dense model. Second, the present application further integrates communication compression technology to efficiently encode these sparse update data. This dual optimization mechanism of "sparsification first and compression later" maximally reduces the data transmission amount of each communication. The direct effect is that the communication delay is reduced and the risk of network congestion is reduced, thereby accelerating the convergence process of federated aggregation and improving the training efficiency and stability of the entire distributed system in a weak network environment. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An exemplary flowchart of an energy management method based on sparse federated reinforcement learning is provided for the present application. Figure 2 An exemplary schematic diagram of updating a sparse network is provided for the present application. Figure 3 An exemplary schematic diagram of compressed federated learning is provided for the present application. DETAILED DESCRIPTION

[0018] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings of the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0019] Users of power systems all expect to reduce their electricity bill expenditure through energy management services. Generally, a household user is equipped with various distributed energy resources, including a photovoltaic system, an energy storage system, an electric vehicle, and flexible electrical appliances such as an air conditioner and a washing machine. The household user can meet its power demand through photovoltaic power generation, energy storage discharge, and power purchase from the power grid. Generally, for the user, the most economical mode of electricity use is to arrange the operation of electrical appliances or energy storage charging during the electricity price valley period, and to use the surplus electrical energy of the energy storage system and photovoltaic power generation to supply power during the electricity price peak period.

[0020] To manage local energy resources, a household energy management system deployed on an edge device such as a smart meter can convert external information flow into device-level control flow, thereby scheduling power flow within the household. Specifically, as shown in Figure 1 The edge device collects local energy consumption information at each time step and receives real-time electricity price information from the power company. The edge device also constructs an initial energy management model and trains the initial energy management model to obtain a final energy management model. Subsequently, the edge device executes an intelligent energy management algorithm through the energy management model based on the energy consumption information and the real-time electricity price information to determine control instructions for the installed household devices through an energy management strategy. The installed household devices are managed through the control instructions. The edge household energy management system aims to optimize energy consumption to reduce electricity bills while taking into account the comfort of the user's electricity use. Therefore, adjacent household users with similar behavioral preferences can perform collaborative energy management learning through information interaction between multiple edge devices.

[0021] In some embodiments, the initial energy management model can be constructed and trained through a reinforcement learning algorithm to obtain a final energy management model, including: The energy management problem is formulated as a Markov decision process, which includes a state space, an action space, a state transition probability function, and a reward function. The Markov decision process can be wherein is the state space, is the action space, is the state transition probability function, and r is the reward function, is a decay factor. The system agent determines the optimal strategy , which maps states to probability distributions over actions. Specifically, at time step t, the agent observes state and selects action . The environment then returns an immediate reward and transitions to the next state . The mathematical formulation of state, action, reward, state transition, and action value function will be elaborated respectively below.

[0022] constructing an initial energy management model; the initial energy management model comprising an initial actor network and an initial critic network; inputting a current observation state into the actor network to output an optimal continuous action; concatenating the current observation state and the optimal continuous action and inputting the concatenation into the critic network to obtain an initial discrete action, and selecting the initial discrete action through a greedy policy to obtain an optimal discrete action; constructing an optimal hybrid action based on the optimal continuous action and the optimal discrete action, and iteratively training the energy management model through a hybrid DQN algorithm based on the optimal hybrid action until the energy management model converges to obtain a final energy management model.

[0023] In some embodiments, iteratively training the energy management model through the hybrid DQN algorithm based on the optimal hybrid action to obtain the final energy management model comprises: executing the optimal hybrid action, returning an immediate reward and transitioning to a new observation state after executing the action, and constructing a SARSA experience sample; generating a multi-step target through the energy management model based on the SARSA experience sample, constructing a loss function of the critic network with an expected mean square error between a value estimate of the optimal hybrid action and the multi-step target, minimizing Bellman error using a gradient descent method to complete critic network update; applying a chain rule to construct a loss function of the actor network with an expected value of the optimal hybrid action based on the SARSA experience sample and the updated critic network, combining a double time scale update rule to execute a single step update to complete actor network update; updating parameters of the energy management model using a smooth soft update method; the energy management model comprising the actor network and the critic network; iteratively training the energy management model to update network parameters of the energy management model until the energy management model converges to obtain the final energy management model.

[0024] Specifically, to deal with the hybrid discrete-continuous action space , this paper proposes a hybrid DQN algorithm that combines the advantages of traditional reinforcement learning algorithms DQN and DDPG without the need for approximation or relaxation processing. The agent updates the actor network of the continuous action through the deterministic policy gradient, and updates the optimal discrete action ​the critic network. The deterministic action network takes the current observation state as input and outputs the optimal continuous action . The observation state and the continuous action are concatenated as input to the critic network to obtain the optimal discrete action . An ε-greedy policy is used for exploration, with a probability of choosing a random action. The value estimate of the optimal hybrid action can be computed by the Bellman equation: ; where is the value estimate of the optimal hybrid action at time t; and are the observation states at time t and t+1, respectively; and are the optimal hybrid actions at time t and t+1, respectively; is the expectation operator; is the immediate reward at time t; γ is the reward discount factor; max is the maximum operator; is the action-value function for a given policy π; and are the optimal discrete actions at time t and t+1, respectively; is the optimal continuous action; is the actor network parameter; θ is the critic network parameter.

[0025] The critic network is updated using gradient descent by minimizing the Bellman error. To avoid unstable and divergent value function estimates, multiple-step targets are generated from two independent target networks: ; where is the multiple-step target; is the immediate reward obtained by performing action in state ; is the action output by the target actor network based on the weight in state ; is the feature representation corresponding to state ; and are the weight parameters of the target actor network and the target critic network, respectively. The loss function of the critic network is defined as the expected mean squared error between the optimal action value estimate and the multiple-step target: ; where Loss function of critic network; Optimal continuous action.

[0026] Loss function of actor network is updated by deterministic policy gradient. Chain rule is applied to the expected value of hybrid action: ; Wherein, Loss function of actor network; z is discrete action; Z is the number of discrete actions. The collaborative update of action network and critic network is realized by double time scale update rule: the gradient of is calculated by approximating The gradient of is calculated by approximating Only negligible single-step update is performed. The inconsistency of samples caused by policy changes is eliminated by using an experience replay pool, and its storage space follows the replacement principle (the oldest sample is discarded when the space is full). The target network update adopts smooth soft update instead of direct copying.

[0027] To alleviate the memory pressure of edge devices caused by instant variables, the application provides a sparse evolutionary network training method suitable for an energy management system. Inspired by biological neural networks, the method replaces the conventional fully connected layer with a sparse layer throughout the training process, significantly reducing the number of network parameters. A random sparse network with a specific sparsity is determined in the initial stage. Then the gradient value is calculated and the connection weight is updated through training. Based on the weight information, some existing connections are discarded, and an equal amount of new connections are activated according to the gradient information. The sparse network constructed in this way can realize continuous evolution through self-regulation. The sparse training includes four main stages: network initialization, network training, adaptive dropout, and gradient-directed growth, as shown in Figure 2 For ease of expression, we collectively refer to the actor network weight And the critic network weight As .

[0028] In some embodiments, an initial energy management model can be constructed by a sparse network, including: constructing a sparse topology structure of an initial sparse network by assigning sparse connections between layers to obtain an initialized sparse network; the sparse topology structure includes the overall sparsity of the sparse network, the total number of network layers, and the sparse connections between layers; dynamically evolving the sparse network through SARSA experience samples to obtain a final sparse network; replacing the fully connected layers of the initial actor network and the initial critic network with the final sparse network to obtain the initial energy management model.

[0029] Specifically, the sparse topology structure is constructed by assigning sparse connections between layers. The overall sparsity of the sparse network is defined as This represents the ratio of the number of unconnected parameters to the total number of parameters. The initial connections follow the ER (Erdös–Rényi) random graphical model. Weighted connections exist between neurons in layer k and layer (k+1). The probability is expressed as: ; in, The probability of the weights of existing sparse connections between layers; represents the weights of existing sparse connections between layers; k represents the layer variables of the sparse network. and These represent the number of neurons in the k-th and (k+1)-th layers, respectively. The number of connections per layer scales linearly with the sum of the input and output channels. Only the weights of the connected layers and their derived variables are computed and stored. Because... and Since most neurons do not have weighted connections, sparse training reduces the space complexity of the k-th layer from... ( · ) down to ( + This significantly reduces computational and memory overhead. Assuming a sparse network has K layers, its overall sparsity can be expressed as: ; in, K represents the overall sparsity; K is the total number of layers in the sparse network; where This represents the sparsity of the k-th layer. The bias parameter has a negligible impact on the network size, and each layer still maintains dense connections.

[0030] While sparse networks are highly efficient to train, randomly generated topologies often fail to achieve ideal performance in specific energy management tasks. Therefore, this invention proposes a dynamic evolution strategy: at a fixed update interval... Within the framework, sparse topology connections are optimized with constant overall sparsity. The sparse topology connections of the sparse network are dynamically evolved using SARSA empirical samples to obtain the final sparse network. This includes: training the initialized sparse network using SARSA empirical samples, and dynamically evolving the sparse network at a fixed update interval to obtain the updated sparse network.

[0031] Based on the weights of existing inter-layer sparse connections in the updated sparse network, adaptive discarding of these connections yields a removal set. The sparse network discards the connections with the lowest importance (closest to zero), i.e., the smallest positive weight and the largest negative weight, at a discard ratio of [ratio to be inserted here]. These removed weights are expected to have little impact on the network performance. To enhance the stability of dynamic evolution, the update ratio is annealed with the convergence process: ; where, is the update ratio of inter-layer sparse connections; is the initial update ratio; cos is the cosine function; π is the constant pi; t is the time variable; is the total number of iterations. When the agent network gradually converges to the optimal solution, the update ratio tends to zero, at which time the topology remains unchanged to focus on weight parameter updates. For the parameter connection connecting the input layer i neuron and the output layer j neuron, the removal set selects the top k connections with the smallest absolute weights.

[0032] ; where, is the removal set; ArgTopK is the top K connections with the largest absolute gradient values; is the parameter connection weight connecting the input layer i neuron and the output layer j neuron; i is the input layer variable; j is the output layer variable; is the number of existing connections to be removed at the kth layer at time t; in the formula represents the number of existing connections to be removed at the kth layer at time t, which depends on the update ratio .

[0033] Based on the gradient of the non-existing inter-layer sparse connections in the updated sparse network, the inter-layer sparse connections are grown in a gradient-oriented manner to obtain a growth set; the number of inter-layer sparse connections in the removal set and the growth set is equal. The sparse network selects the position with the highest importance (the largest positive gradient and the smallest negative gradient) from the non-existing connections, and grows new connections at the same ratio . These new weights have high sensitivity to reducing subsequent iteration loss. However, to save memory, the gradient values of non-existing connections are not stored during the training process. Therefore, we calculate the gradient by the chain rule combined with the neuron output and its gradient: ; where, is the gradient of the non-existing inter-layer sparse connection; and are the outputs of the i neuron of the kth layer and the j neuron of the K+1th layer, respectively; is the partial derivative with respect to the loss function L. The growth set selects the top k connections with the largest absolute gradient values outside the removal set.

[0034] The growth set is: ; wherein, is the growth set; ArgTopK is the top K connections with the largest absolute gradient values; is the parameter set except the removal set; is the absolute value. The above dropout and growth operations are applied layer by layer to evolve the network topology. The weights of the newly generated connections are initialized to zero, and then the network enters the iterative training phase of weight parameter updating.

[0035] The sparse topology structure of the sparse network is updated based on the removal set and the growth set to obtain a new sparse network, and the new sparse network is iteratively trained until the sparse network converges, and the converged sparse network is taken as the final sparse network.

[0036] In some embodiments, due to the limited memory space in the edge device, the capacity of the experience replay buffer can only be set relatively small. Such buffer constraints usually lead to insufficient state-action space exploration and suboptimal utilization of historical experience, affecting the algorithm performance and convergence. Therefore, the present application also proposes a compressed federated learning framework, which performs collaborative energy management learning on the energy management model through information interaction among multiple edge devices to train a high-performance global network using the transition data cached by the distributed edge devices. As shown in Figure 3 , the framework can be regarded as a shared and merged memory pool for Each edge device trains a local sparse network using the SARS A experience samples in the locally stored experience replay pool, obtaining the local sparse network weight corresponding to each edge device. The server maps the weight values of the local sparse network weight to the inter-layer sparse connection state of each local sparse network, retains the non-zero inter-layer sparse connection weight based on the inter-layer sparse connection state, and obtains the local effective weight; aggregates the local effective weight of each local sparse network to obtain the global network weight of the global network model.

[0037] Specifically, each edge device trains a sparse network round, uploads the sparse network weight to the server to update the global network weight : ; wherein, is the global network weight; k is the layer variable of the sparse network; i is the connection input layer variable; j is the connection output layer variable; e is the edge device variable; E is the total number of edge devices; is the weight obtained by the e-th device training; is the absolute value; sgn is the sign function. Since the discarded weight does not participate in the averaging process, the formula By mapping the weight values to connection states, only non-zero connection weights are aggregated. However, the high dimensionality of the weight matrix leads to significant communication overhead in the uploading and distributing process. Based on the low-rank property of the sparse network, the present application also introduces a truncated singular value decomposition technique for matrix compression in the weight exchange process, including two steps of random singular value decomposition and weight approximation.

[0038] Performing random singular value decomposition on the global network weights, a decomposition result is obtained; the decomposition result includes a left singular matrix, a singular value matrix, and a right singular matrix. The random singular value decomposition is specifically, the weight matrix is decomposed into a tensor and respectively, which are the input / output channel numbers. SVD (Singular Value Decomposition) can decompose a large sparse matrix into several small dense matrices: ; where and are orthogonal matrices containing left and right singular vectors; is a diagonal matrix, and the elements are the singular values of in descending order. To reduce the SVD calculation cost, a Gaussian random matrix containing noise is introduced to project the weight matrix to the subspace basis matrix , where represents the orthogonalization operation with power iteration. Because , the size of the basis matrix is much smaller than . By proxying the matrix , SVD can be performed on the small matrix : ; where is the left singular matrix of . The left singular matrix of the original weight matrix is calculated by .

[0039] Although random SVD reduces the matrix size, the communication cost of transmitting the decomposition matrix is still high. Therefore, based on the decomposition result, weight approximation processing is performed to obtain an approximate weight matrix. The weight approximation processing is specifically, dimension compression is performed using the truncated decomposition matrix: only the first larger singular values and their corresponding singular vectors are retained to obtain the approximate weight matrix: ​ ; wherein, is an approximate weight matrix; is a left singular matrix that retains L eigenvalues; is a diagonal matrix that retains L eigenvalues. The weight matrix size is reduced from the original to To control the approximation error, the number of truncated singular values is constrained by an energy threshold ; Considering , the error upper bound of the compressed SVD algorithm can be guaranteed as: ; wherein, is a network parameter matrix; is a left singular matrix; is a diagonal matrix; is a right singular matrix; T is the transpose of the matrix; is a projected network parameter matrix; is a left singular matrix of ; is the L smallest values; n is a reserved eigenvalue variable; L is the number of reserved eigenvalues; is an eigenvalue; is a basis matrix; is a distortion parameter between 0 and 1; is an L1 norm; is an approximate network parameter matrix.

[0040] Each edge device updates the local sparse network according to the approximate weight matrix, obtains a new local sparse network, and iteratively trains the new local sparse network until the global network model converges, to obtain a final global network model.

[0041] ​This invention also provides an energy management system based on sparse federated reinforcement learning, including an edge device and a server. The edge device includes an information acquisition module, a model training module, a policy generation module, and a management module. The information acquisition module collects local energy consumption information at each time step and receives real-time electricity price information from the power company. The model training module constructs an initial energy management model and trains it with the server through federated learning to obtain the final energy management model. The policy generation module executes an intelligent energy management algorithm based on the energy consumption information and real-time electricity price information through the energy management model to obtain an energy management policy. The management module generates control commands for installed home appliances based on the energy management policy and manages the installed home appliances through the control commands.

[0042] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An energy management method based on sparse federated reinforcement learning, characterized in that, include: Collect local energy consumption information at each time step and receive real-time electricity price information from the power company; An initial energy management model is constructed and trained to obtain the final energy management model; Based on energy consumption information and real-time electricity price information, an intelligent energy management algorithm is executed through an energy management model to obtain an energy management strategy; Energy management strategies are used to generate control commands for installed home appliances, and these commands are then used to manage the installed home appliances.

2. The energy management method based on sparse federated reinforcement learning according to claim 1, characterized in that, Construct an initial energy management model and train it to obtain the final energy management model, including: The energy management problem is transformed into a Markov decision process; the Markov decision process includes a state space, action space, state transition probability function, and reward function; Construct an initial energy management model; the initial energy management model includes an initial actuator network and an initial evaluator network; Input the current observation state into the actuator network and output the optimal continuous action; The current observed state and the optimal continuous action are concatenated and input into the evaluator network to obtain the initial discrete action, and then... The greedy strategy selects the initial discrete action to obtain the optimal discrete action; The optimal hybrid action is constructed based on the optimal continuous action and the optimal discrete action. Based on the optimal hybrid action, the energy management model is iteratively trained using the hybrid DQN algorithm until the energy management model converges, thus obtaining the final energy management model.

3. The energy management method based on sparse federated reinforcement learning according to claim 2, characterized in that, Based on the optimal hybrid action, the energy management model is iteratively trained using the hybrid DQN algorithm to obtain the final energy management model, which includes: Execute the optimal hybrid action, return an immediate reward after the action is executed and transition to a new observation state, and construct SARSA experience samples; Based on SARSA empirical samples, multi-step objectives are generated through an energy management model. The loss function of the evaluator network is constructed using the value estimate of the optimal mixed action and the expected mean square error between the multi-step objectives. The gradient descent method is used to minimize the Bellman error and complete the evaluator network update. Based on SARSA experience samples and the updated evaluator network, the chain rule is applied to the expected value of the optimal mixed action through deterministic policy gradient to construct the loss function of the executor network. Combined with the dual time scale update rule, a single-step update is performed to complete the update of the executor network. The parameters of the energy management model are updated using a smooth soft update method; the energy management model includes an actuator network and a judge network. The energy management model is iteratively trained and its network parameters are updated until the model converges, resulting in the final energy management model.

4. The energy management method based on sparse federated reinforcement learning according to claim 3, characterized in that, The formula for calculating the value estimate of the optimal mixed action is: ； in, Estimate the value of the optimal mixed action at time t; and The observation states at time t and time t+1 are respectively; and The optimal hybrid actions at time t and time t+1 are respectively; To obtain the expected value; γ is the instantaneous reward at time t; γ is the reward decay factor; max is the maximum value. The action value function given the policy π; and These are the optimal discrete actions at times t and t+1, respectively. For optimal continuous action; θ represents the actuator network parameters; θ represents the evaluator network parameters. The loss function of the evaluator network is: ； in, Let be the loss function of the evaluator network; For optimal continuous action; The loss function of the actuator network is: ； in, Let z be the loss function of the actuator network; z be the discrete action; and Z be the number of discrete actions.

5. The energy management method based on sparse federated reinforcement learning according to claim 3, characterized in that, An initial energy management model is constructed using sparse networks, including: The sparse topology of the initial sparse network is constructed by allocating sparse connections between layers, resulting in the initialized sparse network. The sparse topology includes the overall sparsity of the sparse network, the total number of network layers, and the sparse connections between layers. The sparse network is dynamically evolved using SARSA empirical samples to obtain the final sparse network. The initial energy management model is obtained by replacing the fully connected layers of the initial actuator network and the initial evaluator network with the final sparse network.

6. The energy management method based on sparse federated reinforcement learning according to claim 5, characterized in that, The sparse network is dynamically evolved using SARSA empirical samples to obtain the final sparse network, including: The initialized sparse network is trained using SARSA empirical samples, and the sparse network is dynamically evolved at a fixed update interval to obtain the updated sparse network. Based on the weights of existing inter-layer sparse connections in the updated sparse network, the inter-layer sparse connections are adaptively discarded to obtain the removal set. Based on the gradients of non-existent inter-layer sparse connections in the updated sparse network, gradient growth is performed on the inter-layer sparse connections to obtain the growth set; the number of inter-layer sparse connections in the removal set and the growth set are equal. The sparse topology of the sparse network is updated by removing and growing sets to obtain a new sparse network. The new sparse network is then iteratively trained until it converges, and the converged sparse network is taken as the final sparse network.

7. The energy management method based on sparse federated reinforcement learning according to claim 6, characterized in that, The probability of the weights of existing inter-layer sparse connections is: ； in, The probability of the weights of existing sparse connections between layers; represents the weights of existing sparse connections between layers; k represents the layer variables of the sparse network. and These represent the number of neurons in the k-th and +1-th layers, respectively. The formula for calculating overall sparsity is: ； in, K represents the overall sparsity; K represents the total number of layers in the sparse network. The update ratio of inter-layer sparse connections is: ； in, The update ratio for sparse connections between layers; The initial update ratio is denoted by cos; cos is the cosine function; π is the mathematical constant pi; and t is the time variable. This represents the total number of iterations. Remove the following set: ； in, To remove a set; ArgTopK selects the K connections with the largest absolute gradient values; The parameters are the connection weights connecting neuron i in the input layer and neuron j in the output layer; i is the input layer variable; j is the output layer variable. Let be the number of existing connections to be removed at time t in layer k. The gradient of non-existent inter-layer sparse connections is: ； in, The gradient represents the non-existent sparse inter-layer connections. and These are the outputs of neuron i in layer k and neuron j in layer (K+1), respectively. Find the partial derivative with respect to the loss function L; The growth set is: ； in, For the growth set; ArgTopK selects the K connections with the largest absolute gradient values; For the set of parameters excluding the set to be removed; To take the absolute value.

8. The energy management method based on sparse federated reinforcement learning according to claim 6, characterized in that, It also includes collaborative energy management learning of the energy management model through information interaction between multiple edge devices, including: Each edge device uses SARSA experience samples in its locally stored experience replay pool to train a local sparse network and obtain the local sparse network weights for each edge device. The server maps the weight values ​​of the local sparse network weights to the inter-layer sparse connection states of each local sparse network, and retains the non-zero inter-layer sparse connection weights based on the inter-layer sparse connection states to obtain the local effective weights. By aggregating the effective weights of each local sparse network, the global network weights of the global network model are obtained. Perform random singular value decomposition on the global network weights to obtain the decomposition results; the decomposition results include the left singular matrix, the singular value matrix, and the right singular matrix; Based on the decomposition results, an approximate weight matrix is ​​obtained by approximating the weights. Each edge device updates its local sparse network based on the approximate weight matrix to obtain a new local sparse network. The new local sparse network is then iteratively trained until the global network model converges, resulting in the final global network model.

9. The energy management method based on sparse federated reinforcement learning according to claim 8, characterized in that, The formula for calculating global network weights is: ； in, denoted as global network weights; k represents the layer variables of the sparse network; i represents the variables connecting the input layer; j represents the variables connecting the output layer; e represents the edge device variables; and E represents the total number of edge devices. The weights obtained from training the e-th device; To take the absolute value; sgn is the sign function; Random singular value decomposition includes: ； ； ； ； in, This is the network parameter matrix; It is a left singular matrix; It is a diagonal matrix; Let T be a right singular matrix; T is the transpose of the matrix. This is the projected network parameter matrix; for The left singular matrix; To select the L smallest values; i represents the input layer variables; L represents the number of features to retain; For eigenvalues; It is a basis matrix; The distortion parameter is between 0 and 1; It is the L1 norm; This is an approximate network parameter matrix; The formula for calculating the approximate weight matrix is: ； in, This is an approximate weight matrix; To preserve the left singular matrix with L eigenvalues; To preserve the L eigenvalues ​​of the diagonal matrix.

10. A system for energy management based on sparse federated reinforcement learning, utilizing the energy management method based on sparse federated reinforcement learning as described in any one of claims 1-9, characterized in that, It includes edge devices and servers; the edge devices include information acquisition modules, model training modules, policy generation modules, and management modules. The information acquisition module is used to collect local energy consumption information at each time step and receive real-time electricity price information from the power company; The model training module is used to build an initial energy management model and train the initial energy management model with the server through federated learning to obtain the final energy management model; The strategy generation module is used to generate energy management strategies by executing intelligent energy management algorithms through an energy management model based on energy consumption information and real-time electricity price information. The management module is used to generate control commands for installed home appliances through energy management strategies, and to manage the installed home appliances through these control commands.

Images