Electric power visual knowledge distillation method and system

Abstract

The invention provides an electric power visual knowledge distillation method and system, and the method comprises the steps: collecting electric power visual image data, and building an electric power visual data set; inputting the electric power visual data set into a first model for training to obtain a teacher model; constructing a second model, combining a detection framework of the teacher model with a backbone network of the second model, and constructing a student model; inputting the electric power visual data set into the teacher model and the student model at the same time to obtain output results, and calculating the difference degree between the output results of the teacher model and the student model by using a composite loss function; using a back propagation algorithm and an Adam optimizer to update parameters of the student model according to the difference degree; and enabling the difference degree to be smaller than a set value through a plurality of rounds of iteration, and outputting a student model. According to the method, on the premise that resource consumption is greatly reduced, the obtained student model has higher calculation speed, recognition precision close to that of a teacher model and smaller model size, and the electric power visual knowledge distillation system is suitable for various resource-constrained environments.

Description

Technical Field

[0001] The present invention relates to the field of electric power vision technology, and in particular to an electric power vision knowledge distillation method and system. Background Art

[0002] With the rapid development of the power industry and the continuous improvement of its intelligence level, power vision technology is becoming increasingly important in the monitoring, management and operation and maintenance of power systems. Power vision technology mainly uses image processing and computer vision algorithms to monitor and identify power equipment, lines and environments in real time to achieve functions such as fault warning, status assessment and intelligent control.

[0003] However, the current application of power vision technology faces challenges. First, power vision systems usually need to process a large amount of image and video data, which requires a large amount of computing resources. Second, in order to improve the accuracy and stability of recognition, power vision systems usually use complex deep learning models, but these models often have many parameters and large computational complexity, making it difficult to run in real time in resource-constrained environments. Summary of the invention

[0004] Purpose of the invention: The first purpose of the present invention is to provide a power vision knowledge distillation method with reduced resource consumption, high efficiency and wide application scenarios. The second purpose is to provide a power vision knowledge distillation system corresponding to the above-mentioned power vision knowledge distillation method.

[0005] Technical solution: A method for distilling power visual knowledge, comprising the following steps:

[0006] (1) Collect power visual image data and establish a power visual dataset;

[0007] (2) constructing a first model, inputting the electric power vision data set into the first model for training, until the recognition accuracy of the first model for the electric power vision data set is greater than a set value, thereby obtaining a teacher model;

[0008] (3) Constructing a second model, combining the detection framework of the teacher model with the backbone network of the second model to construct a student model;

[0009] (4) The power vision data set is input into the teacher model and the student model at the same time to obtain the output results, and the difference between the output results of the teacher model and the student model is calculated using the composite loss function;

[0010] (5) Use the back-propagation algorithm and Adam optimizer to update the parameters of the student model according to the difference;

[0011] (6) Repeat steps (4)-(5) until the difference is less than the set value through several rounds of iterations, and output the student model.

[0012] Specifically, the electric power visual image data includes image data of electric power equipment, power grid lines, and substation environments marked with labels.

[0013] Preferably, step (1) also includes: preprocessing the power visual image data, including cleaning, labeling, dividing, and enhancing operations.

[0014] Preferably, the method further comprises the following steps:

[0015] (7) Perform performance evaluation on the student model output from step (6) and output evaluation indicators.

[0016] Specifically, the evaluation indicators include recognition accuracy, calculation speed, model size, and resource consumption.

[0017] Specifically, the composite loss function is obtained by weighted calculation of the soft label loss function and the hard label loss function. The soft label loss function is the relative entropy loss function, and the calculation formula is as follows:

[0018]

[0019] Where: L soft is the soft label loss function, p is the probability distribution of the teacher model, q is the probability distribution of the student model, and N is the number of categories of the probability distribution;

[0020] The hard label loss function is the cross entropy loss function, and the calculation formula is as follows:

[0021]

[0022] Where: L hard is the hard label loss function, y i is the encoding of the hard label, q i is the predicted probability of the student model for category i, and N is the number of categories in the probability distribution;

[0023] The formula for calculating the composite loss function is as follows:

[0024] L=α·L soft +(1-α)·L hard

[0025] Where: α is the set weight coefficient.

[0026] Specifically, step (5) includes: calculating the gradient of the difference with respect to the parameters of the student model, calculating the first-order moment estimate and the second-order moment estimate of the gradient, using the Adam optimizer to automatically adjust the learning rate of the parameters of the student model, and using the back-propagation algorithm to update the parameters of the student model in the opposite direction of the gradient to reduce the difference.

[0027] Specifically, the first model includes the YOLOv5 model and the RCNN model (ReAgion-based Convolutional Neural Networks); the second model includes the MobileNet model and the YOLOv4-Tiny model.

[0028] The present invention also provides an electric power visual knowledge distillation system, comprising:

[0029] Dataset generation module: used to collect power vision image data and establish power vision dataset;

[0030] Teacher model training module: used to construct the first model, input the electric power vision data set into the first model for training, until the recognition accuracy of the first model for the electric power vision data set is greater than a set value, and obtain the teacher model;

[0031] Student model construction module: used to build the second model, combining the detection framework of the teacher model with the backbone network of the second model to build the student model;

[0032] Knowledge distillation module: used to input the power vision data set into the teacher model and the student model at the same time, obtain the output results, and use the composite loss function to calculate the difference between the output results of the teacher model and the student model;

[0033] Optimization training module: used to update the parameters of the student model according to the difference using the back propagation algorithm and the Adam optimizer;

[0034] Parameter iteration module: used to repeatedly run the knowledge distillation module and the optimization training module. After several rounds of iterations, the difference is reduced to the set value and the student model is output.

[0035] Beneficial effects: Compared with the prior art, the present invention has the following significant effects:

[0036] 1. By constructing a teacher-student model for knowledge distillation, the knowledge of large and complex power vision models can be effectively transferred to small and lightweight models, which can greatly reduce computing resources and storage space while maintaining high recognition accuracy and computing performance.

[0037] 2. The actual operating environment of the power system has problems such as limited resources and remote areas. By compressing the model scale, the applicability and deployment flexibility of the power vision system in resource-constrained environments are improved.

[0038] 3. By simplifying the model structure, the number of model parameters and the complexity of calculation are reduced, the processing speed and response time are improved, and the power system's needs for real-time monitoring and rapid decision-making are met. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the power visual knowledge distillation method according to Example 1 of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further explained below in conjunction with the accompanying drawings and specific implementation methods.

[0041] Example 1

[0042] See also Figure 1 As shown, this embodiment provides a method for distilling electric power visual knowledge, comprising the following steps:

[0043] (1) Collect power visual image data and establish a power visual dataset.

[0044] In this embodiment, image data related to power vision are collected, including image data of power equipment, power grid lines, and substation environment. The image data are preprocessed, including cleaning, labeling, division, and enhancement operations, to construct four relatively independent power vision data sets. The labeling content mainly includes the status of the staff: whether they are wearing work clothes, whether they are wearing a safety helmet, whether they are smoking, etc.; power equipment: transformers, reactors, mutual inductors, disconnectors, wires, etc.; the surrounding environment of the power system: bird nests, obstacles, equipment fireworks, etc.; monitoring equipment status: pressure gauges, ammeters, oil temperature gauges, oil level gauges, observation windows, etc.

[0045] The training set of dataset A contains 80,000 images and 310,000 labels, and the test set contains 20,000 images and 78,000 labels; the training set of dataset B contains 40,000 images and 155,000 labels, and the test set contains 10,000 images and 35,000 labels; dataset C contains 50,000 images and 210,000 labels; dataset D is a dataset based on dataset C after feature enhancement, and its data scale is 4 times that of dataset C.

[0046] (2) Construct a first model and input the electric power vision data set into the first model for training until the recognition accuracy of the first model for the electric power vision data set is greater than a set value, thereby obtaining a teacher model.

[0047] In this embodiment, the YOLOv5 (You Only Look Once version 5) model is selected as the first model for training. The YOLOv5 model is a real-time target detection algorithm that can achieve a faster detection speed while maintaining a high detection accuracy, and is suitable for application scenarios that require real-time response.

[0048] Before training, the YOLOv5 model cannot accurately identify the targets in dataset B. After sufficient pre-training on dataset A, YOLOv5 can accurately identify specific targets in dataset B with an accuracy of 92%, exceeding the set value standard. Save the trained YOLOv5 model as the teacher model.

[0049] (3) Construct a second model, combining the detection framework of the teacher model with the backbone network of the second model to construct a student model.

[0050] In this embodiment, the MobileNet model is selected as the second model, and the detection framework of YOLOv5 is combined with the backbone network of the MobileNet model to construct a MobileNet-YOLOv5 model as the student model. The MobileNet model is a lightweight convolutional neural network that can achieve efficient deep learning and feature extraction on mobile and embedded devices. It is combined with YOLOv5 to fully utilize the high efficiency of MobileNet feature extraction and the fast detection capability of YOLOv5, that is, to ensure that the structure of the student model has a certain similarity with the teacher model, so as to effectively learn from the teacher model, reduce the number of parameters and computational complexity, and maintain a certain feature extraction capability.

[0051] (4) The power vision dataset is input into the teacher model and the student model at the same time to obtain the output results, and the difference between the output results of the teacher model and the student model is calculated using the composite loss function.

[0052] In this embodiment, the data set B is input into the teacher model and the student model at the same time, and forward propagation is performed to obtain the outputs of the teacher model and the student model respectively. A composite loss function obtained by weighted calculation of the soft label loss function and the hard label loss function is introduced to measure the difference between the output results of the student model and the teacher model.

[0053] The soft label refers to the output probability distribution of the teacher model for each sample, which provides information about the relative probabilities between categories. In this embodiment, the soft label loss function is the relative entropy loss function, also known as the KL divergence (Kullback-Leibler Divergence), which is used to measure the difference between the probability distribution output by the student model and the probability distribution output by the teacher model. The calculation formula is as follows:

[0054]

[0055] Where: L soft is the soft label loss function, p is the probability distribution of the teacher model, q is the probability distribution of the student model, and N is the number of categories of the probability distribution;

[0056] The hard label refers to the true label of each sample in the data set, which is usually a discrete, definite value. In this embodiment, the hard label loss function is the cross entropy loss function, which is used to measure the accuracy of the student model's prediction of the true label. The calculation formula is as follows:

[0057]

[0058] Where: L hard is the hard label loss function, y i is the encoding of the hard label, q i is the predicted probability of the student model for category i, and N is the number of categories in the probability distribution;

[0059] The formula for calculating the composite loss function is as follows:

[0060] L=α·L soft +(1-α)·L hard

[0061] Where: L is the composite loss function, α is the set weight coefficient, and the initial value is 0.5.

[0062] Soft label loss L soft and hard label loss L hard The weighted combination of is achieved by introducing a hyperparameter α. Considering the importance of parameter α in controlling the soft label loss in the composite loss L, it is set to be dynamically changed in this embodiment: when the soft label loss converges too quickly, the α value is reduced; when the hard label loss converges too quickly, the α value is increased; the weight of the loss that converges faster is gradually reduced, and the attention to another label is increased.

[0063] The convergence speed of label loss between adjacent iterations or epochs is measured by the value of the convergence rate: convergence rate = (current loss function value - previous loss function value) / previous loss function value. If the convergence rate decreases rapidly and approaches zero at the beginning of training, it can be determined that the label loss converges too quickly.

[0064] The benefit of this combination is that it allows the student model to learn from real hard labels to ensure basic classification accuracy, and learn from the soft labels of the teacher model to obtain richer information about similarities and probability distributions between categories. In this way, the student model can inherit the generalization ability of the teacher model and has the potential to approach or even exceed the teacher model in performance while maintaining a small model size and fast inference speed.

[0065] (5) Use the back-propagation algorithm and the Adam (Adaptive Moment Estimation) optimizer to update the parameters of the student model according to the difference.

[0066] In knowledge distillation, the back-propagation algorithm is used to update the parameters of the student model based on the value of the composite loss function. Specifically, the back-propagation algorithm is based on the gradient descent method, which calculates the gradient of the composite loss function to the model parameters through the chain rule, and updates the parameters in the opposite direction of the gradient, thereby gradually reducing the value of the loss function. The gradient information indicates how to adjust the parameters to reduce the loss.

[0067] The Adam optimizer is an optimization algorithm with adaptive learning rate. It automatically adjusts the learning rate of each parameter by calculating the first-order moment estimate (mean) and the second-order moment estimate (uncentered variance) of the gradient to better adapt to different data and models. For parameters that are updated frequently, the learning rate will be reduced, while for parameters that are not updated frequently, the learning rate will be relatively large. The initial value of the learning rate is 0.001. A smaller initial learning rate helps the model learn more stably in the early stages of training and helps the student model imitate the output of the teacher model more carefully rather than roughly approximating it.

[0068] (6) Repeat steps (4)-(5) until the difference is less than the set value through several rounds of iterations, and output the student model.

[0069] (7) Perform performance evaluation on the student model outputted in step (6) and output evaluation indicators. The evaluation indicators include recognition accuracy, calculation speed, model size, and resource consumption.

[0070] In this embodiment, the performance of the student model is evaluated on independent datasets C and D, and compared with the teacher model and other benchmark models to verify the effectiveness of the method.

[0071] The three models were run under the same hardware conditions. The recognition accuracy was measured by mean average precision (mAP), the calculation speed was measured by frames per second (FPS), and the number of floating point operations (GFLOPs) was used as the measure of resource consumption. The experimental data is shown in Table 1 below:

[0072] Table 1

[0073]

[0074] It can be seen from the test results that, under the premise of greatly reducing resource consumption, the student model obtained in this embodiment has a faster calculation speed, a recognition accuracy very close to that of the teacher model, and a smaller model size, which verifies the effectiveness of this method.

[0075] Example 2

[0076] This embodiment provides a power visual knowledge distillation system, including the following modules:

[0077] Dataset generation module: used to collect power vision image data and establish power vision dataset;

[0078] Teacher model training module: used to construct the first model, input the electric power vision data set into the first model for training, until the recognition accuracy of the first model for the electric power vision data set is greater than a set value, and obtain the teacher model;

[0079] Student model construction module: used to build the second model, combining the detection framework of the teacher model with the backbone network of the second model to build the student model;

[0080] Knowledge distillation module: used to input the power vision data set into the teacher model and the student model at the same time, obtain the output results, and use the composite loss function to calculate the difference between the output results of the teacher model and the student model;

[0081] Optimization training module: used to update the parameters of the student model according to the difference using the back propagation algorithm and the Adam optimizer;

[0082] Parameter iteration module: used to repeatedly run the knowledge distillation module and the optimization training module. After several rounds of iterations, the difference is reduced to the set value and the student model is output.

[0083] The above-mentioned power vision knowledge distillation system pre-provides 4 relatively independent power vision data sets. Users can also upload power vision data sets that meet the format requirements. Users select or upload teacher models and student models through the system interface. The available teacher models in the system include YOLOv5 model and RCNN model. The available student models include MobileNet-YOLOv5 model and YOLOv4-Tiny model. Users can also upload teacher models and student models that meet the requirements.

[0084] Before starting knowledge distillation, users need to configure the relevant parameters of knowledge distillation, including the initial weight value of the composite loss function and the initial value of the learning rate. The default initial weight value of the composite loss function is 0.5, and the initial value of the learning rate is 0.001.

[0085] The strategies of knowledge distillation include offline distillation, online distillation and self-distillation. Offline distillation refers to pre-training a large model with high-precision recognition ability and rich knowledge reserves as a teacher model; the characteristic of online distillation is that the teacher model is not pre-trained in advance, but is trained together with the student model and guides the student model to learn to complete the knowledge distillation; the self-distillation teacher model and the student model are actually the same model. The model self-guides and self-learns during the training process to complete the knowledge distillation. In practical applications, the choice of distillation strategy depends on specific needs and scenarios. Offline distillation is suitable for scenarios with relatively high accuracy requirements; if a real-time, online learning system is required, then online distillation may be a better choice because it can provide immediate feedback and guidance; if resources are limited or you want to simplify the complexity of model training, then self-distillation may be more suitable. Users can choose the knowledge distillation strategy according to actual needs.

[0086] Start the knowledge distillation process and monitor various indicators during the training process, including training loss, validation loss, accuracy, gradient norm, learning rate, time and resource consumption, etc.

[0087] After training is complete, check the model evaluation report and adjust the model or parameters as needed to further optimize the Power Vision Knowledge Distillation System.

Claims

1. A method for distilling power vision knowledge, characterized in that: The following steps are involved: (1) Collect power visual image data and establish a power visual dataset; (2) constructing a first model, inputting the electric power vision data set into the first model for training, until the recognition accuracy of the first model for the electric power vision data set is greater than a set value, thereby obtaining a teacher model; (3) Constructing a second model, combining the detection framework of the teacher model with the backbone network of the second model to construct a student model; (4) The power vision data set is input into the teacher model and the student model at the same time to obtain the output results, and the difference between the output results of the teacher model and the student model is calculated using the composite loss function; (5) Use the back-propagation algorithm and Adam optimizer to update the parameters of the student model according to the difference; (6) Repeat steps (4)-(5) until the difference is less than the set value through several rounds of iterations, and output the student model.

2. The power visual knowledge distillation method according to claim 1, characterized in that: The electric power visual image data includes image data of electric power equipment, power grid lines, and substation environment marked with labels.

3. The power visual knowledge distillation method according to claim 1, characterized in that: The step (1) also includes: preprocessing the electric power visual image data, including cleaning, labeling, dividing, and enhancing operations.

4. The power visual knowledge distillation method according to claim 1, characterized in that: The following steps are also included: (7) Perform performance evaluation on the student model output from step (6) and output evaluation indicators.

5. The method for distilling electric power visual knowledge according to claim 4, characterized in that: The evaluation indicators include recognition accuracy, calculation speed, model size, and resource consumption.

6. The power visual knowledge distillation method according to claim 1, characterized in that: The composite loss function is obtained by weighted calculation of the soft label loss function and the hard label loss function.

7. The method for distilling electric power visual knowledge according to claim 6, characterized in that: The soft label loss function is a relative entropy loss function, and the calculation formula is as follows: Where: L soft is the soft label loss function, p is the probability distribution of the teacher model, q is the probability distribution of the student model, and N is the number of categories of the probability distribution; The hard label loss function is a cross entropy loss function, and the calculation formula is as follows: Where: L hard is the hard label loss function, y i is the encoding of the hard label, q i is the predicted probability of the student model for category i, and N is the number of categories in the probability distribution; The calculation formula of the composite loss function is as follows: L=α·L soft +(1-a)·L hard Where: α is the set weight coefficient.

8. The method for distilling electric power visual knowledge according to claim 1, characterized in that: The step (5) includes: calculating the gradient of the difference degree with respect to the parameters of the student model, calculating the first-order moment estimation and the second-order moment estimation of the gradient, automatically adjusting the learning rate of the parameters of the student model using the Adam optimizer, and updating the parameters of the student model along the reverse direction of the gradient using the back propagation algorithm to reduce the difference degree.

9. The method for distilling electric power visual knowledge according to claim 1, characterized in that: The first model includes a YOLOv5 model and an RCNN model; the second model includes a MobileNet model and a YOLOv4-Tiny model.

10. A power vision knowledge distillation system, characterized in that: include: Dataset generation module: used to collect power vision image data and establish power vision dataset; Teacher model training module: used to construct the first model, input the electric power vision data set into the first model for training, until the recognition accuracy of the first model for the electric power vision data set is greater than a set value, and obtain the teacher model; Student model construction module: used to build the second model, combining the detection framework of the teacher model with the backbone network of the second model to build the student model; Knowledge distillation module: used to input the power vision data set into the teacher model and the student model at the same time, obtain the output results, and use the composite loss function to calculate the difference between the output results of the teacher model and the student model; Optimization training module: used to update the parameters of the student model according to the difference using the back propagation algorithm and the Adam optimizer; Parameter iteration module: used to repeatedly run the knowledge distillation module and the optimization training module. After several rounds of iterations, the difference is reduced to the set value and the student model is output.

Images