Power model deployment method and device based on federated learning and knowledge distillation

Abstract

The invention discloses an electric power model deployment method and device based on federated learning and knowledge distillation, and the method comprises the steps: carrying out the data division of a common data set, and obtaining a training set; constructing a teacher model, and training the teacher model through the training set; constructing a student model, and training the student model through the training set and the teacher model; and descending the student model to a power distribution network edge node based on a federated learning framework, training the student model through a local data set, deploying the student model to the power distribution network edge node for operation, and updating the student model regularly. The method and the device provided by the invention not only can effectively reduce model parameters to realize lightweight deployment of the power distribution network, but also can ensure the security of the model during data sharing.

Description

Technical Field

[0001] The present invention relates to the field of edge computing of distribution networks, and in particular to a method and device for deploying an electric power model based on federated learning and knowledge distillation. Background Art

[0002] With the continuous development of science and technology, the edge of the distribution network is moving towards informatization and intelligence, but it also faces challenges such as a high proportion of new energy access. Among them, new energy is volatile and uncertain, and traditional optimization methods are difficult to deal with effectively. As a technical science that can simulate human intelligence, artificial intelligence technology has powerful pattern recognition and data analysis capabilities, can efficiently process complex data, and provide intelligent response solutions. Therefore, the introduction of artificial intelligence technology in the transformation of distribution services such as load forecasting and power quality monitoring can accurately monitor and analyze load fluctuations, optimize system operation conditions, and improve the efficiency and stability of distribution network operation.

[0003] However, while AI technology has brought convenience to the distribution network, it has also brought challenges in training and deployment. On the one hand, the training of AI models requires a large amount of data, but due to data security and privacy protection, it is difficult for power data owners to share data, resulting in a "data island" phenomenon. On the other hand, the computing power of edge devices is limited and it is difficult to support the operation of large parameter models. Model training needs to be completed in the cloud, and lightweight models need to be deployed on the edge. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method and device for deploying an electric power model based on federated learning and knowledge distillation, which can effectively reduce model parameters while ensuring model accuracy, realize lightweight deployment of distribution networks, and ensure the security of shared data.

[0005] In a first aspect, the present invention provides a method for deploying a power model based on federated learning and knowledge distillation, which is characterized by comprising:

[0006] Divide the public data set to obtain the training set;

[0007] Constructing a teacher model, and training the teacher model using the training set;

[0008] Constructing a student model, and training the student model using the training set and the teacher model;

[0009] Based on the federated learning framework, the student model is downlinked to the edge node of the distribution network, the student model is trained through a local data set, the student model is deployed to the edge node of the distribution network for operation, and the student model is regularly updated.

[0010] Preferably, constructing the teacher model and training the teacher model using the training set includes:

[0011] The teacher model is constructed based on a long short-term memory network, the model parameters of the teacher model are set, and the mean square error loss function and the stochastic gradient descent algorithm optimizer are used to train the teacher model through the training set until the teacher model converges.

[0012] Preferably, the mean square error loss function is the first loss function of the teacher model, and the expression includes:

[0013] L1=MSE

[0014]

[0015] Where: L1 is the first loss function of the teacher model; MSE is the mean square error loss function; y i is the actual value of the i-th sample, is the predicted value of the teacher model for the i-th sample, and n is the total number of samples.

[0016] Preferably, constructing the student model, and training the student model using the training set and the teacher model comprises:

[0017] The student model is constructed based on a long short-term memory network, a second loss function of the student model is obtained based on knowledge distillation technology, and the student model is trained through the training set and the teacher model until the student model converges.

[0018] Preferably, the second loss function of the student model includes: a prediction loss function and a knowledge distillation loss function of the student model;

[0019] The expression of the second loss function of the student model is as follows:

[0020] L2=α·MSE+(1-α)·KL(Soft Targets Teacher ,Soft Targets Student )

[0021] Where: MSE is the prediction loss function of the student model; α is a hyperparameter used to balance the weight between the prediction loss function of the student model and the knowledge distillation loss function; KL (Soft Targets Teacher ,SoftTargets Student ) is the knowledge distillation loss function; Soft Targets Teacher Soft Targets are the soft labels output by the teacher model.Student is the soft label output by the student model.

[0022] Preferably, the student model is downlinked to the edge node of the distribution network based on the federated learning framework, the student model is trained by the local data set, the student model is deployed to the edge node of the distribution network for operation, and the student model is regularly updated, including:

[0023] Downlinking the student model to the edge node of the distribution network through communication technology, performing personalized training on the student model through the local data set until the student model converges, and deploying the student model to the edge node of the distribution network for operation;

[0024] The student model is uploaded to the cloud server regularly, and the model parameters of the student model uploaded by each edge node of the distribution network are aggregated by using the federated averaging algorithm. The student model is updated by using the aggregated model parameters, and the updated student model is downloaded to the edge node of the distribution network. The updated student model is personalized trained by using the local data set, and the updated student model is downloaded and deployed to the edge node of the distribution network for operation.

[0025] Preferably, the expression of the federated averaging algorithm is as follows:

[0026]

[0027] Where: w t+1 is the model parameter of the updated student model; K is the total number of edge nodes of the distribution network participating in federated learning; n k is the size of the local data set on the edge node k of the distribution network; n is the total number of samples of all the edge nodes of the distribution network, are the local model parameters of the distribution network edge node k after the tth round of personalized training.

[0028] In a second aspect, the present invention provides a power model deployment device based on federated learning and knowledge distillation, which is characterized by comprising:

[0029] Dataset acquisition module: used to divide the public data set and obtain the training set;

[0030] The first model training module is used to construct the teacher model and train the teacher model through the training set;

[0031] The second model training module is used to construct the student model and train the student model through the training set and the teacher model;

[0032] The third model training module is used to downlink the student model to the edge node of the distribution network based on the federated learning framework, and train the student model through the local data set;

[0033] Model running module: used to deploy the student model to the edge node of the distribution network for operation and regularly update the student model.

[0034] In a third aspect, the present invention provides an electronic device, comprising a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0035] The memory stores machine-readable instructions, and the processor executes the method described in the first aspect by calling the machine-readable instructions.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium having a non-volatile program code executable by a processor, characterized in that the computer program implements the method described in the first aspect when executed by the processor.

[0037] The beneficial effects of the present invention are as follows:

[0038] Based on the federated learning framework and knowledge distillation technology, the present invention constructs a large-parameter teacher model and a small-parameter student model in the cloud, and uses a public data set to train and distill knowledge of the student model, which greatly reduces the model parameters while ensuring the accuracy of the model and realizes lightweight deployment; at the same time, relying on the federated learning framework, the student model is distributed to multiple edge nodes, and each node uses the local data set to perform personalized training on the student model, so that the model can fully learn the characteristics and patterns of the local data, and regularly upload the updated student model to the cloud for summary and aggregation, so that the model can be widely accepted without uploading local data. The training of diversified data sets improves its generalization ability.

[0039] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a flow chart of a method for deploying a power model based on federated learning and knowledge distillation provided in an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of the structure of an electric power model deployment device based on federated learning and knowledge distillation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] To facilitate understanding of this embodiment, Figure 1 The power model deployment method based on federated learning and knowledge distillation disclosed in an embodiment of the present invention is introduced in detail.

[0046] Embodiment 1:

[0047] S1: Divide the public data set to obtain the training set;

[0048] S2: Build a teacher model and train the teacher model using the training set;

[0049] S3: Build a student model and train the student model using the training set and the teacher model;

[0050] S4: Based on the federated learning framework, the student model is downlinked to the edge node of the distribution network, the student model is trained through the local data set, the student model is deployed to the edge node of the distribution network for operation, and the student model is updated regularly.

[0051] The following is a detailed description of the above steps:

[0052] S1: Divide the public data set to obtain the training set.

[0053] In this embodiment, the public data set is divided into data to obtain a training set, including:

[0054] The data in the public data set is divided, 70% of the data in the public data set is selected as training data, and a training set is obtained based on the training data.

[0055] Among them, the public data set is the data set of related businesses collected through the cloud computing data center, and the related businesses include: load forecasting, load identification, and fault detection; the data set includes load power data, load current data, photovoltaic power data, and meteorological temperature data.

[0056] Furthermore, after the above steps are implemented, the remaining data in the public data set is used as test data, and a test set is obtained based on the test data. The test set is used to verify the teacher model and / or student model after the teacher model and / or student model are trained with the training set.

[0057] S2: Build a teacher model and train the teacher model using the training set.

[0058] Among them, the teacher model is a large-parameter teacher business model.

[0059] Preferably, a teacher model is constructed and the teacher model is trained using a training set, including:

[0060] A teacher model is constructed based on the Long Short-Term Memory (LSTM) network, and the model parameters of the teacher model are set. The mean-square error (MSE) loss function and the stochastic gradient descent (SGD) optimizer are used to perform back-propagation training on the teacher model through the training set until the teacher model converges.

[0061] Furthermore, the mean square error loss function is the first loss function of the teacher model, and the expression includes:

[0062] L1=MSE

[0063]

[0064] Where: L1 is the first loss function of the teacher model; MSE is the mean square error loss function; y i is the actual value of the i-th sample, is the predicted value of the teacher model for the i-th sample, and n is the total number of samples.

[0065] In this embodiment, a teacher model is constructed based on a long short-term memory network on a cloud server, and setting the model parameters of the teacher model includes:

[0066] The forget gate weight matrix W is calculated by a random number functionf , input gate weight matrix W i , candidate memory unit weight matrix W C , output gate weight matrix W o Perform initialization settings and set the initial hidden state h 0 and the initial memory cell state C 0 All are set to zero vectors to initialize the hyperparameters, including learning rate and batch size.

[0067] Furthermore, the model of the long short-term memory network includes: forget gate output, input gate output, candidate memory unit output, memory unit state, output gate output, and hidden state of the current time step.

[0068] Among them, the forget gate output f at time t t The expression is:

[0069] f t =σ(W f ·[h {t-1} ,x t ]+b f )

[0070] In the formula, f t is the output of the forget gate at time t; W f is the forget gate weight matrix; [h {t-1} ,x t ] is the current input x t and the hidden state h at the previous time step {t-1} The series connection, b f is the forget gate bias vector.

[0071] The input gate output i at time t t The expression is:

[0072] i t =σ(W i ·[h {t-1} ,x t ]+b i )

[0073] In the formula, i t is the input gate output at time t, σ is the Sigmoid activation function; W i is the input gate weight matrix, [h {t-1} ,x t ] is the current input x t and the hidden state h at the previous time step {t-1} The series connection, b i is the input gate bias vector, where the value output range of σ is 0 to 1.

[0074] Candidate memory unit output at time t The expression is:

[0075]

[0076] In the formula, is the candidate memory unit output at time t, tanh is the hyperbolic tangent activation function, [h {t-1} ,x t ] is the current input x t and the hidden state h at the previous time step {t-1} The series connection, W C is the candidate memory unit weight matrix, b C is the candidate memory unit bias vector.

[0077] The memory cell state C at time t t The expression is:

[0078]

[0079] In the formula, f t is the output of the forget gate at time t, C {t-1} is the memory cell state at the previous time step, i t is the input gate output at time t, is the candidate memory unit output at time t.

[0080] The output gate outputs o at time t t The expression is:

[0081] o t =σ(W o ·[h {t-1} ,x t ]+b o )

[0082] Where W o is the output gate weight matrix, b o is the output gate bias vector.

[0083] The hidden state h at the current time step t The expression is:

[0084] h t =o t tanh(C t )

[0085] In the formula, o t is the output gate output at time t, tanh is the hyperbolic tangent activation function, C t is the state of the memory unit at time t.

[0086] S3: Build a student model and train the student model using the training set and the teacher model.

[0087] Among them, the student model is a small-parameter student business model.

[0088] Preferably, constructing a student model, and training the student model using a training set and a teacher model includes:

[0089] A student model is constructed based on the long short-term memory network, and the second loss function of the student model is obtained based on the knowledge distillation technology. The student model is trained through the training set and the teacher model until the student model converges.

[0090] Further, the second loss function of the student model includes: a prediction loss function and a knowledge distillation loss function of the student model;

[0091] The expression of the second loss function of the student model is as follows:

[0092] L2=α·MSE+(1-α)·KL(Soft Targets Teacher ,Soft Targets Student )

[0093] Where: MSE is the prediction loss function of the student model; α is a hyperparameter used to balance the weight between the prediction loss function of the student model and the knowledge distillation loss function; KL (Soft Targets Teacher ,Soft Targets Student ) is the knowledge distillation loss function; Soft Targets Teacher Soft Targets are the soft labels output by the teacher model. Student Soft labels output by the student model.

[0094] In this embodiment, the value range of α is [0,1]. When the value of T is higher, the output probability distribution will be smoother. The expression of the soft label is as follows:

[0095]

[0096] Where z is the label output by the model and T is the temperature parameter.

[0097] In this application, the advantage of obtaining the second loss function of the student model based on knowledge distillation technology is that knowledge distillation technology, as a model lightweight technology, can significantly reduce its complexity and deployment cost while maintaining model performance.

[0098] S4: Based on the federated learning framework, the student model is downlinked to the edge node of the distribution network, the student model is trained through the local data set, the student model is deployed to the edge node of the distribution network for operation, and the student model is updated regularly.

[0099] In this embodiment, the frequency of regular updates includes but is not limited to updating once a day. The local data set is a local private data set.

[0100] The advantage of applying the federated learning framework in this application is that, under the premise of protecting user data privacy, the student model can learn the patterns and characteristics of different data sets from multiple user private data sets, thereby enhancing the student model's ability to recognize data.

[0101] Further, in this embodiment, when the student model is trained by the local data set, the expression of the third loss function of the student model is:

[0102] L3=MSE

[0103] Where: MSE is the prediction loss function of the student model.

[0104] Preferably, the student model is downlinked to the edge node of the distribution network based on the federated learning framework, the student model is trained through the local data set, the student model is deployed to the edge node of the distribution network for operation, and the student model is regularly updated, including:

[0105] The student model is transmitted downlink to the edge node of the distribution network through communication technology, and the student model is personalized trained through the local data set until the student model converges, and the student model is deployed to the edge node of the distribution network for operation;

[0106] The student model is uploaded to the cloud server regularly, and the model parameters of the student model uploaded by each distribution network edge node are aggregated using the Federated Averaging (FedAvg) algorithm. The student model is updated using the aggregated model parameters, and the updated student model is downloaded to the distribution network edge node. The updated student model is personalized trained using the local data set, and the updated student model is downloaded and deployed to the distribution network edge node for operation.

[0107] In this embodiment, the communication technology includes Wi-Fi network technology, 5G, fiber optic communication technology, and power line carrier communication technology.

[0108] Furthermore, the expression of the federated averaging algorithm is as follows:

[0109]

[0110] Where: w t+1is the model parameter of the updated student model; K is the total number of distribution network edge nodes participating in federated learning; n k is the size of the local data set on the edge node k of the distribution network; n is the total number of samples of all the edge nodes of the distribution network, are the local model parameters of the edge node k in the distribution network after the tth round of personalized training.

[0111] Embodiment 2:

[0112] Combination Figure 2 , a lightweight deployment device of a distribution network model based on knowledge distillation disclosed in an embodiment of the present invention includes:

[0113] Dataset acquisition module: used to divide the public data set and obtain the training set;

[0114] The first model training module is used to build a teacher model and train the teacher model through the training set;

[0115] The second model training module is used to build a student model and train the student model through the training set and the teacher model;

[0116] The third model training module is used to downlink the student model to the edge node of the distribution network based on the federated learning framework and train the student model through the local data set;

[0117] Model operation module: used to deploy the student model to the edge node of the distribution network for operation and regularly update the student model.

[0118] Among them, the lightweight deployment device of the distribution network model based on knowledge distillation disclosed in this embodiment can realize all the contents of the power model deployment method based on federated learning and knowledge distillation disclosed in Example 1 of this application.

[0119] Embodiment 3:

[0120] An electronic device disclosed in an embodiment of the present invention includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected through the bus; the memory stores machine-readable instructions, and the processor can execute the entire content of the power model deployment method based on federated learning and knowledge distillation disclosed in Example 1 by calling the machine-readable instructions.

[0121] Embodiment 4:

[0122] A computer-readable storage medium having a non-volatile program code executable by a processor is disclosed in an embodiment of the present invention, wherein, when the computer program is executed by the processor, it can implement all the contents of the power model deployment method based on federated learning and knowledge distillation disclosed in Example 1 of the present application.

[0123] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. The power model deployment method based on federated learning and knowledge distillation is characterized by: include: Divide the public data set to obtain the training set; Constructing a teacher model, and training the teacher model using the training set; Constructing a student model, and training the student model using the training set and the teacher model; Based on the federated learning framework, the student model is downlinked to the edge node of the distribution network, the student model is trained through a local data set, the student model is deployed to the edge node of the distribution network for operation, and the student model is regularly updated.

2. The method for deploying a power model based on federated learning and knowledge distillation according to claim 1, characterized in that: Constructing the teacher model and training the teacher model using the training set includes: The teacher model is constructed based on a long short-term memory network, the model parameters of the teacher model are set, and the mean square error loss function and the stochastic gradient descent algorithm optimizer are used to train the teacher model through the training set until the teacher model converges.

3. The method for deploying a power model based on federated learning and knowledge distillation according to claim 2, characterized in that: The mean square error loss function is the first loss function of the teacher model, and the expression includes: L1=MSE Where: L1 is the first loss function of the teacher model; MSE is the mean square error loss function; y i is the actual value of the i-th sample, is the predicted value of the teacher model for the i-th sample, and n is the total number of samples.

4. The method for deploying a power model based on federated learning and knowledge distillation according to claim 1, characterized in that: Constructing the student model, and training the student model using the training set and the teacher model includes: The student model is constructed based on a long short-term memory network, a second loss function of the student model is obtained based on knowledge distillation technology, and the student model is trained through the training set and the teacher model until the student model converges.

5. The method for deploying a power model based on federated learning and knowledge distillation according to claim 4 is characterized in that: The second loss function of the student model includes: a prediction loss function and a knowledge distillation loss function of the student model; The expression of the second loss function of the student model is as follows: L2=α·MSE+(1-α)·KL(Soft Targets Teacher ,Soft Targets Student ) Where: MSE is the prediction loss function of the student model; α is a hyperparameter used to balance the weight between the prediction loss function of the student model and the knowledge distillation loss function; KL (Soft Targets Teacher ,Soft Targets Student ) is the knowledge distillation loss function; Soft Targets Teacher SoftTargets are the soft labels output by the teacher model. Student is the soft label output by the student model.

6. The method for deploying a power model based on federated learning and knowledge distillation according to claim 1, characterized in that: Based on the federated learning framework, the student model is downlinked to the edge node of the distribution network, the student model is trained by the local data set, the student model is deployed to the edge node of the distribution network for operation, and the student model is regularly updated, including: Downlinking the student model to the edge node of the distribution network through communication technology, performing personalized training on the student model through the local data set until the student model converges, and deploying the student model to the edge node of the distribution network for operation; The student model is uploaded to the cloud server regularly, and the model parameters of the student model uploaded by each edge node of the distribution network are aggregated by using the federated averaging algorithm. The student model is updated by using the aggregated model parameters, and the updated student model is downloaded to the edge node of the distribution network. The updated student model is personalized trained by using the local data set, and the updated student model is downloaded and deployed to the edge node of the distribution network for operation.

7. The method for deploying a power model based on federated learning and knowledge distillation according to claim 6, characterized in that: The expression of the federated averaging algorithm is as follows: Where: w t+1 is the model parameter of the updated student model; K is the total number of the distribution network edge nodes participating in federated learning; n k is the size of the local data set on the edge node k of the distribution network; n is the total number of samples of all the edge nodes of the distribution network, are the local model parameters of the distribution network edge node k after the tth round of personalized training.

8. The power model deployment device based on federated learning and knowledge distillation is characterized by: include: Dataset acquisition module: used to divide the public data set and obtain the training set; The first model training module is used to construct the teacher model and train the teacher model through the training set; The second model training module is used to construct the student model and train the student model through the training set and the teacher model; The third model training module is used to downlink the student model to the edge node of the distribution network based on the federated learning framework, and train the student model through the local data set; Model running module: used to deploy the student model to the edge node of the distribution network for operation and regularly update the student model.

9. An electronic device, characterized in that: It includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the steps of the method according to any one of claims 1 to 7 by calling the machine-readable instructions.

10. A computer-readable storage medium having a processor-executable non-volatile program code, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images