Power distribution network model lightweight deployment method and device based on knowledge distillation

Abstract

The invention provides a power distribution network model lightweight deployment method and device based on knowledge distillation, and the method comprises the steps: carrying out the data division of a first data set, and obtaining a training set; constructing a teacher model, and training the teacher model through the training set; constructing a student model, softening output labels of the teacher model and the student model, and training the student model through the training set and the teacher model; and descending the student model to the power distribution network edge node, training the student model based on the second data set, deploying the student model on the power distribution network edge node, and enabling the student model to operate on the power distribution network edge node. According to the method, the communication pressure of cloud edge interaction can be relieved, the computing power demand of an edge side computing model is reduced, and the availability of a power distribution network artificial intelligence computing model is improved.

Description

Technical Field

[0001] The present invention relates to the field of edge computing of distribution networks, and in particular to a method and device for lightweight deployment of distribution network models based on knowledge distillation. Background Art

[0002] With the development of science and technology, the scale of distribution network equipment has also developed rapidly. Among them, the measurement equipment on the edge side of the distribution network can generate a large amount of data. If this data is processed in the cloud computing data center, the measured data must be uploaded to the cloud computing data center first. This process will occupy a large amount of communication bandwidth and put a lot of pressure on communication. In addition, the distribution network has some control services, such as precise load control and distribution automation. These control services are sensitive to latency. If the above services are deployed in the cloud computing data center for processing, cloud-edge data interaction communication is required. The communication time consumed in this process may have an adverse effect on the delay-sensitive control services. For the above reasons, local business processing is required on the edge side of the distribution network.

[0003] At present, the edge computing technology of the distribution network is usually used to process and analyze the multi-source data collected by the terminal device sensors in real time through the edge nodes of the distribution network to realize local business processing. In order to better process and analyze the complex data brought by the sensors of terminal devices such as smart meters, the edge nodes of the new distribution system have gradually adopted technologies such as deep learning and data intelligent analysis to achieve high-accuracy edge-side business processing. However, although the existing edge-side artificial intelligence processing methods have a high accuracy rate, due to the large number of artificial intelligence model parameters, the model inference time is long and the model deployment communication requirements are high, making it difficult to directly deploy in actual application scenarios with high real-time requirements. Therefore, a new method is needed to meet the needs of lightweight model deployment and rapid application. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a lightweight deployment method and device for a distribution network model based on knowledge distillation, which can reduce the communication pressure of cloud-edge interaction, while reducing the computing power requirements of the edge-side computing model, thereby improving the availability of the distribution network artificial intelligence computing model.

[0005] In a first aspect, an embodiment of the present invention provides a lightweight deployment method of a distribution network model based on knowledge distillation, characterized in that it includes:

[0006] Divide the first data set into data to obtain a training set;

[0007] Constructing a teacher model, and training the teacher model using the training set;

[0008] Constructing a student model, softening the output labels of the teacher model and the student model, and training the student model using the training set and the teacher model;

[0009] Downlinking the student model to the edge node of the distribution network, and training the student model based on the second data set;

[0010] The student model is deployed on the edge node of the distribution network, so that the student model runs on the edge node of the distribution network.

[0011] Preferably, the teacher model is constructed and trained through the training set, including: constructing the teacher model based on a bidirectional long short-term memory network, initializing the model parameters of the teacher model, using the loss function of the teacher model and the adaptive moment estimation optimizer, and training the teacher model for multiple epochs through the training set until the teacher model converges.

[0012] Preferably, the loss function of the teacher model is a first cross entropy loss function L1, and the formula is as follows:

[0013]

[0014] Where: L1 is the first cross entropy loss function; y i is the true probability of the target label for the i-th sample, is the teacher model prediction probability for the i-th sample.

[0015] Preferably, the formula of the adaptive moment estimation optimizer includes:

[0016] The formula for updating the first-order moment estimate and the second-order moment estimate is:

[0017] m t =β1m t-1 +(1-β1)g t

[0018]

[0019] Where: m t 、v t are the first-order moment estimate and the second-order moment estimate at time t, respectively; β1 and β2 are hyperparameters, which are used to control the decay rates of the first-order moment estimate and the second-order moment estimate, respectively; g t is the gradient value of the model parameter at the time t;

[0020] The formula for correcting the first-order moment estimate and the second-order moment estimate is:

[0021]

[0022] Where: m t 、v t are the first-order moment estimate and the second-order moment estimate at the time t, respectively; are the first-order moment estimate and the second-order moment estimate after bias correction, respectively;

[0023] The formula for updating the model parameters of the teacher model is:

[0024]

[0025] Where: θ t is the model parameter at the time t; θ t+1 is the model parameter at time t+1; ∈ is the minimum value; η is the learning rate.

[0026] Preferably, constructing the student model, softening the output labels of the teacher model and the student model, and training the student model through the training set and the teacher model comprises the following steps:

[0027] Constructing the student model with small parameters based on long short-term memory network;

[0028] Using a Softmax function for knowledge distillation to soften the output labels of the teacher model and the student model, thereby obtaining a total loss function of the student model;

[0029] The student model is trained using the training set and the teacher model until the student model converges.

[0030] Preferably, the total loss function of the student model is equal to the weighted sum of the knowledge distillation loss function and the true label loss function;

[0031] Preferably, the formula of the total loss function L2 is:

[0032] L2=α·L KD +(1-α)·L CE

[0033] Where: L KD is the knowledge distillation loss function, that is, the second cross entropy loss function between the soft label outputs of the student model and the teacher model; L CE is the third cross entropy loss function between the hard label output and the true label output of the student model; α is used to adjust L KD and L CE A hyperparameter of the loss function weight ratio between .

[0034] Preferably, the knowledge distillation loss function L KD The formula is:

[0035] L KD =T 2 KL(Softmax(logits student / T),Softmax(logits teacher / T))

[0036] Where: KL is the KL divergence, which is used to measure the difference between the output distribution of the teacher model and the output distribution of the student model; T is the temperature parameter, Softmax(logits student / T) and Softmax(logits teacher / T) are the soft label outputs of the student model and the teacher model respectively.

[0037] In a second aspect, an embodiment of the present invention provides a lightweight deployment device for a distribution network model based on knowledge distillation, characterized in that it includes:

[0038] Data set acquisition module: used to divide the first data set into data and obtain a training set;

[0039] The first model training module is used to construct a teacher model and train the teacher model through the training set;

[0040] The second model training module is used to construct a student model, soften the output labels of the teacher model and the student model, and train the student model through the training set and the teacher model;

[0041] The third model training module is used to transmit the student model to the edge node of the distribution network and train the student model based on the second data set;

[0042] Model running module: used to deploy the student model on the edge node of the distribution network, so that the student model runs on the edge node of the distribution network.

[0043] In a third aspect, the present invention provides an electronic device, comprising a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0044] The memory stores machine-readable instructions, and the processor executes the method described in the first aspect by calling the machine-readable instructions.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having a non-volatile program code executable by a processor, characterized in that the computer program implements the method described in the first aspect when executed by the processor.

[0046] The beneficial effects brought by the present invention are as follows: the present invention constructs a teacher model and a student model in a cloud computing data center, and uses historical data sets to train the teacher model, so that it has the ability to guide the optimization of the student model; under the joint action of the historical data set and the teacher model, the student model is trained based on the knowledge distillation technology, so that the small parameter student model has performance and accuracy similar to the large parameter teacher model, and the small parameter student model is deployed to the edge node of the distribution network through the communication channel, and the local data set of the edge node is used for secondary training, which improves the student model's ability to analyze the local data set. Based on this, using a small parameter student model to perform business processing can reduce the communication pressure of cloud-edge interaction, reduce the computing power requirements of the edge-side computing model, and effectively improve the availability of the distribution network artificial intelligence computing model.

[0047] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A schematic flow chart of a method for lightweight deployment of a distribution network model based on knowledge distillation provided in an embodiment of the present invention.

[0051] Figure 2 A schematic diagram of a device for a lightweight deployment method of a distribution network model based on knowledge distillation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] To facilitate understanding of this embodiment, Figure 1 A lightweight deployment method of a distribution network model based on knowledge distillation disclosed in an embodiment of the present invention is introduced in detail.

[0054] Embodiment 1:

[0055] A lightweight deployment method of a distribution network model based on knowledge distillation disclosed in an embodiment of the present invention comprises the following specific steps:

[0056] Divide the first data set into data to obtain a training set;

[0057] Build a teacher model and train the teacher model using the training set;

[0058] Build a student model, soften the output labels of the teacher model and the student model, and train the student model using the training set and the teacher model;

[0059] Downlinking the student model to the edge node of the distribution network, and training the student model based on the second data set;

[0060] The student model is deployed on the edge nodes of the distribution network so that the student model can run on the edge nodes of the distribution network.

[0061] In combination with the above steps, this embodiment is further described:

[0062] S1: Divide the first data set into data to obtain a training set;

[0063] In this embodiment, data division is performed on the first data set to obtain a training set, including: data division is performed on the data in the first data set, 70% of the data in the first data set is selected as training data, and a training set is obtained based on the training data. The first data set is a historical data set of related services collected by a cloud computing data center, and the related services include: load forecasting, load identification, and fault detection; the historical data set includes historical load power data, historical photovoltaic power data, and historical meteorological temperature data.

[0064] Furthermore, after the above steps are implemented, the remaining data in the first data set is used as test data, and a test set is obtained based on the test data. The test set is used to verify the teacher model and / or student model after the teacher model and / or student model are trained with the training set.

[0065] S2: Build a teacher model and train the teacher model using the training set;

[0066] Preferably, a teacher model is constructed and trained through a training set, including: constructing a teacher model based on a bidirectional long short-term memory network (Bi-directional Long Short-Term Memory), initializing the model parameters of the teacher model, using the loss function of the teacher model and an adaptive moment estimation (ADAM) optimizer, and training the teacher model for multiple epochs through the training set until the teacher model converges.

[0067] In this embodiment, the teacher model is constructed based on a cloud computing data center, and initializing the parameters of the teacher model includes: setting an initial learning rate to 0.01. The teacher model can output data analysis results, which are used to observe the accuracy of the teacher model and determine the convergence status of the teacher model.

[0068] Furthermore, the loss function of the teacher model is the first cross-entropy loss function L1, and the formula is as follows:

[0069]

[0070] Where: L1 is the first cross entropy loss function; y i is the true probability of the target label for the i-th sample, is the teacher model prediction probability for the i-th sample.

[0071] Furthermore, the formula of the adaptive moment estimation optimizer includes:

[0072] (1) The formula for updating the first-order moment estimate and the second-order moment estimate is:

[0073] m t =β1m t-1 +(1-β1)g t

[0074]

[0075] Where: m t 、v tare the first-order moment estimate and the second-order moment estimate at time t, respectively; β1 and β2 are hyperparameters, which are used to control the decay rate of the first-order moment estimate and the second-order moment estimate, respectively; g t is the gradient value of the model parameters at time t;

[0076] In this embodiment, initial values ​​are set for β1 and β2, that is, β1=0.9, β2=0.999.

[0077] (2) The formula for correcting the first-order moment estimate and the second-order moment estimate is:

[0078]

[0079] Where: m t 、v t They are the first-order moment estimate and the second-order moment estimate at time t respectively; They are the first-order moment estimate and the second-order moment estimate after bias correction, respectively;

[0080] (3) The formula for updating the model parameters of the teacher model is:

[0081]

[0082] Where: θ t is the model parameter at time t; θ t+1 is the model parameter at time t+1; ∈ is the minimum value; η is the learning rate.

[0083] In this embodiment, the value of ∈ is 1e-8 and η is set to 0.01.

[0084] S3: Build a student model, soften the output labels of the teacher model and the student model, and train the student model using the training set and the teacher model;

[0085] In this embodiment, the student model is a small-parameter student model, and the student model is constructed based on a cloud computing data center.

[0086] Preferably, constructing a student model, softening the output labels of the teacher model and the student model, and training the student model through the training set and the teacher model includes the following steps:

[0087] S301: Construct a small-parameter student model based on the Long Short-Term Memory (LSTM) network;

[0088] S302: Softening the output labels of the teacher model and the student model using the Softmax function for knowledge distillation, thereby obtaining the total loss function of the student model;

[0089] S303: Train the student model using the training set and the teacher model until the student model converges.

[0090] In this embodiment, when softening the output labels of the teacher model and the student model, the distillation temperature is controlled by the temperature parameter T to adjust the softening degree of the output labels; the smaller the total loss function of the student model, the better the fitting effect between the output results obtained by the student model and the output results of the teacher model; in addition, the total loss function of the student model can promote the convergence of the student model when the student model is trained.

[0091] Furthermore, the total loss function of the student model in S302 is equal to the weighted sum of the knowledge distillation loss function and the true label loss function;

[0092] The formula for the total loss function L2 is:

[0093] L2=α·L KD +(1-α)·L CE

[0094] Where: L KD is the knowledge distillation loss function, i.e., the second cross entropy loss function between the soft label outputs of the student model and the teacher model; L CE is the third cross entropy loss function between the hard label output of the student model and the true label output; α is used to adjust L KD and L CE A hyperparameter of the loss function weight ratio between .

[0095] In this embodiment, when calculating the hard label output of the student model, the value of T is 1; the initial value of α is 0.5.

[0096] Furthermore, the knowledge distillation loss function L KD The formula is:

[0097] L KD =T 2 KL(Softmax(logits student / T),Softmax(logits teacher / T))

[0098] Where: KL is the KL (Kullback-Leibler) divergence, which is used to measure the difference between the output distribution of the teacher model and the output distribution of the student model; T is the temperature parameter, Softmax (logits student / T) and Softmax(logits teacher / T) are the soft label outputs of the student model and the teacher model respectively.

[0099] In this embodiment, the knowledge distillation loss function L KD Used to pass the soft label output Softmax (logits teacher / T) and temperature parameter T adjust the training effect of the student model.

[0100] S4: Downlink the student model to the edge node of the distribution network, and train the student model based on the second data set;

[0101] In this embodiment, the second data set is a local data set.

[0102] Based on this, the student model is downlinked to the edge node of the distribution network, and the student model is trained based on the second data set, including: using communication technology to downlink the student model to each edge node of the distribution network, and the edge node of the distribution network performs personalized training on the student model based on the local data set until the student model converges.

[0103] In this embodiment, the communication technology includes but is not limited to: optical fiber communication technology and wireless communication technology.

[0104] Among them, after personalized training, the student model only considers the hard label output part of the third cross entropy loss function, and no longer considers the knowledge distillation loss function L KD part.

[0105] S5: Deploy the student model on the edge node of the distribution network, so that the student model runs on the edge node of the distribution network.

[0106] In this embodiment, after the student model runs on the edge node of the distribution network, the input value of the relevant business is obtained, and the inference data is obtained by inference calculation based on the student model. The inference accuracy is obtained through the inference data, and the inference accuracy is tested. Among them, the relevant business data includes: load power data, photovoltaic power data, and meteorological temperature data; the inference data includes: predicted value of load forecasting and classification data of load identification.

[0107] Embodiment 2:

[0108] Combination Figure 2 , a lightweight deployment device of a distribution network model based on knowledge distillation disclosed in an embodiment of the present invention includes:

[0109] Data set acquisition module: used to divide the first data set into data and obtain a training set;

[0110] The first model training module is used to build a teacher model and train the teacher model through the training set;

[0111] The second model training module is used to build a student model, soften the output labels of the teacher model and the student model, and train the student model through the training set and the teacher model;

[0112] The third model training module is used to transmit the student model to the edge node of the distribution network and train the student model based on the second data set;

[0113] Model running module: used to deploy the student model on the edge nodes of the distribution network so that the student model can run on the edge nodes of the distribution network.

[0114] Among them, a lightweight deployment device of a distribution network model based on knowledge distillation disclosed in this embodiment can realize all the contents of a lightweight deployment method of a distribution network model based on knowledge distillation disclosed in Example 1 of this application.

[0115] Embodiment 3:

[0116] An electronic device disclosed in an embodiment of the present invention includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0117] The memory stores machine-readable instructions, and the processor can execute the entire content of a lightweight deployment method for a distribution network model based on knowledge distillation disclosed in Example 1 by calling the machine-readable instructions.

[0118] Embodiment 4:

[0119] A computer-readable storage medium having a non-volatile program code executable by a processor is disclosed in an embodiment of the present invention.

[0120] Among them, when the computer program is executed by the processor, it can implement all the contents of a lightweight deployment method of a distribution network model based on knowledge distillation disclosed in Example 1 of this application.

[0121] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A lightweight deployment method for distribution network model based on knowledge distillation, characterized in that: include: Divide the first data set into data to obtain a training set; Constructing a teacher model, and training the teacher model using the training set; Constructing a student model, softening the output labels of the teacher model and the student model, and training the student model using the training set and the teacher model; Downlinking the student model to the edge node of the distribution network, and training the student model based on the second data set; The student model is deployed on the edge node of the distribution network, so that the student model runs on the edge node of the distribution network.

2. According to claim 1, a lightweight deployment method of a distribution network model based on knowledge distillation is characterized in that: Constructing the teacher model and training the teacher model through the training set, including: constructing the teacher model based on a bidirectional long short-term memory network, initializing the model parameters of the teacher model, using the loss function of the teacher model and the adaptive moment estimation optimizer, and training the teacher model for multiple epochs through the training set until the teacher model converges.

3. According to claim 2, a lightweight deployment method of a distribution network model based on knowledge distillation is characterized in that: The loss function of the teacher model is the first cross entropy loss function L1, and the formula is as follows: Where: L1 is the first cross entropy loss function; y i is the true probability of the target label for the i-th sample, is the teacher model prediction probability for the i-th sample.

4. According to claim 2, a lightweight deployment method of a distribution network model based on knowledge distillation is characterized in that: The formula of the adaptive moment estimation optimizer includes: The formula for updating the first-order moment estimate and the second-order moment estimate is: m t =β1m t-1 +(1-β1)g t Where: m t 、v t are the first-order moment estimate and the second-order moment estimate at time t, respectively; β1 and β2 are hyperparameters, which are used to control the decay rates of the first-order moment estimate and the second-order moment estimate, respectively; g t is the gradient value of the model parameters at the time t; The formula for correcting the first-order moment estimate and the second-order moment estimate is: Where: m t 、v t are the first-order moment estimate and the second-order moment estimate at the time t, respectively; are the first-order moment estimate and the second-order moment estimate after bias correction, respectively; The formula for updating the model parameters is: Where: θ t is the model parameter at time t; θ t+1 is the model parameter at time t+1; ε is the minimum value; η is the learning rate.

5. According to a method for lightweight deployment of distribution network model based on knowledge distillation according to claim 1, it is characterized in that: Constructing the student model, softening the output labels of the teacher model and the student model, and training the student model through the training set and the teacher model, comprising the following steps: Constructing the student model with small parameters based on long short-term memory network; Using a Softmax function for knowledge distillation to soften the output labels of the teacher model and the student model, thereby obtaining a total loss function of the student model; The student model is trained using the training set and the teacher model until the student model converges.

6. A lightweight deployment method for distribution network model based on knowledge distillation according to claim 5, characterized in that: The total loss function of the student model is equal to the weighted sum of the knowledge distillation loss function and the true label loss function; The formula of the total loss function L2 is: L2=α·L KD +(1-a)·L CE Where: L KD is the knowledge distillation loss function, that is, the second cross entropy loss function between the soft label outputs of the student model and the teacher model; L CE is the third cross entropy loss function between the hard label output and the true label output of the student model; α is used to adjust L KD and L CE A hyperparameter of the loss function weight ratio between .

7. According to claim 6, a lightweight deployment method of distribution network model based on knowledge distillation is characterized in that: The knowledge distillation loss function L KD The formula is: L KD =T 2 ·KL(Softmax(logits student / T),Softmax(logits teacher / T)) Where: KL is the KL divergence, which is used to measure the difference between the output distribution of the teacher model and the output distribution of the student model; T is the temperature parameter, Softmax(logits student / T) and Softmax(logits teacher / T) are the soft label outputs of the student model and the teacher model respectively.

8. A lightweight deployment device for distribution network model based on knowledge distillation, characterized in that: include: Data set acquisition module: used to divide the first data set into data and obtain a training set; The first model training module is used to construct a teacher model and train the teacher model through the training set; The second model training module is used to construct a student model, soften the output labels of the teacher model and the student model, and train the student model through the training set and the teacher model; The third model training module is used to transmit the student model to the edge node of the distribution network and train the student model based on the second data set; Model running module: used to deploy the student model on the edge node of the distribution network, so that the student model runs on the edge node of the distribution network.

9. An electronic device, characterized in that: It includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the steps of the method according to any one of claims 1 to 7 by calling the machine-readable instructions.

10. A computer-readable storage medium having a non-volatile program code executable by a processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images