Power equipment monitoring method and system based on knowledge distillation

Abstract

The invention provides an electrical equipment monitoring method and system based on knowledge distillation, and relates to the technical field of machine learning, and the method comprises the steps: obtaining an electrical equipment monitoring image; inputting the power equipment monitoring image into the trained student model to obtain a power equipment monitoring result; wherein the student model is obtained by jointly training a first teacher model and a second teacher model; the first teacher model is a pre-trained deep learning model, and the second teacher model is a deep learning model with randomly initialized parameters; and jointly training the student model, the first teacher model and the second teacher model by adopting the training set data, and completing training when the models converge or meet a preset condition to obtain a trained student model. The generalization ability of the student model can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a method and system for monitoring electric power equipment based on knowledge distillation. Background Art

[0002] The monitoring and fault diagnosis of power equipment are crucial, and visual recognition technology can be used to monitor power equipment in real time. In recent years, deep learning technology has made significant progress in the field of image processing. Existing visual recognition technologies mostly rely on large deep learning models, especially pre-trained visual large models (such as ResNet, EfficientNet, etc.) that perform well in multiple tasks. These models have attracted widespread attention due to their powerful feature extraction capabilities and high accuracy. However, despite their advantages in accuracy, these models still have some significant limitations, especially in terms of deployment in practical applications. Pre-trained visual large models usually have a large number of parameters, which makes them require high computing resources during reasoning, which is often difficult to meet the real-time requirements for application scenarios such as real-time monitoring. Large models have a slow reasoning speed when processing images, which affects their application effect in dynamic environments. For example, in power monitoring systems, fast response is crucial. Therefore, how to improve the efficiency of model processing and meet the real-time and accuracy requirements of real-time monitoring has become an urgent problem to be solved.

[0003] In order to deploy high-performance models in resource-constrained environments, knowledge distillation technology has emerged. By transferring the knowledge of the teacher model to the student model, knowledge distillation technology can effectively migrate the knowledge of the pre-trained large visual model to a lightweight model, achieve model compression and acceleration, and thus improve processing efficiency in real-time monitoring scenarios. However, the differences between the teacher model and the student model may result in the inability to fully or accurately transfer knowledge to the student model, thus affecting the performance and training stability of the student model. Summary of the invention

[0004] In order to address the deficiencies of the prior art, the purpose of the present invention is to provide a power equipment monitoring method and system based on knowledge distillation, which guides the training of a student model by jointly utilizing two different teacher models to enhance the knowledge distillation effect and improve the generalization ability of the student model.

[0005] In order to achieve the above object, according to some embodiments, a first aspect of the present invention provides a method for monitoring electric power equipment based on knowledge distillation, comprising:

[0006] Acquire monitoring images of power equipment;

[0007] Input the power equipment monitoring image into the trained student model to obtain the power equipment monitoring result;

[0008] Among them, the student model is obtained by jointly training the first teacher model and the second teacher model; the first teacher model is a pre-trained deep learning model, and the second teacher model is a deep learning model with randomly initialized parameters; the student model, the first teacher model and the second teacher model are jointly trained using training set data, and the training is completed when the model converges or meets the preset conditions to obtain a trained student model.

[0009] A second aspect of the present invention provides a power equipment monitoring system based on knowledge distillation, comprising:

[0010] A data acquisition module is configured to acquire monitoring images of power equipment;

[0011] The monitoring module is configured to input the power equipment monitoring image into the trained student model to obtain the power equipment monitoring result; wherein, the student model is obtained by joint training of the first teacher model and the second teacher model; the first teacher model is a pre-trained deep learning model, and the second teacher model is a deep learning model with randomly initialized parameters; the student model, the first teacher model and the second teacher model are jointly trained using the training set data, and the training is completed when the model converges or meets the preset conditions to obtain the trained student model.

[0012] A third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the above-mentioned method for monitoring power equipment based on knowledge distillation.

[0013] According to a fourth aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, complete the steps of the above-mentioned method for monitoring power equipment based on knowledge distillation.

[0014] A fifth aspect of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned power equipment monitoring method based on knowledge distillation.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The present invention provides a method and system for monitoring electric equipment based on knowledge distillation, which aims to improve the performance of student models in electric equipment monitoring and fault detection tasks, and guide the training of student models by jointly utilizing two different teacher models to achieve more efficient knowledge transfer and stronger model generalization capabilities. The two teacher models are a pre-trained teacher model and a randomly initialized teacher model. The pre-trained teacher model has rich feature extraction capabilities and can provide high-quality knowledge guidance, while the randomly initialized teacher model can start from different learning perspectives and prompt the student model to learn diversified feature representations. The joint guidance of the two teacher models enables the student model to not only obtain accurate label information from the teacher model, but also absorb potential feature information in the teacher model.

[0017] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0019] Figure 1 Schematic diagram of the model training process in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0021] Embodiment 1

[0022] Embodiment 1 of the present invention provides a method for monitoring electric power equipment based on knowledge distillation, such as Figure 1 As shown, including:

[0023] Acquire monitoring images of power equipment;

[0024] Input the power equipment monitoring image into the trained student model to obtain the power equipment monitoring result;

[0025] Among them, the student model is obtained by jointly training the first teacher model and the second teacher model; the first teacher model is a pre-trained deep learning model, and the second teacher model is a deep learning model with randomly initialized parameters; the student model, the first teacher model and the second teacher model are jointly trained using training set data, and the training is completed when the model converges or meets the preset conditions to obtain a trained student model.

[0026] In this embodiment, a pre-trained teacher model and a randomly initialized teacher model are used simultaneously to enhance the learning effect of the student model, aiming to solve the problem of huge differences between the teacher model and the student model that may occur in the existing knowledge distillation framework, thereby improving the performance and training stability of the student model.

[0027] Specifically, the first teacher model selects a deep learning model pre-trained on a large-scale image dataset, and uses the collected power image data to fine-tune the first teacher model to improve its accuracy in power image processing.

[0028] For the second teacher model, a randomly initialized deep learning model is selected and trained from scratch together with the student model.

[0029] The student model is a lightweight model that aims to improve its performance by learning the knowledge of the two teacher models, achieving performance similar to or stronger than the teacher models to meet the accuracy and real-time requirements of power equipment monitoring.

[0030] Before model training, data preparation is required, including data collection, labeling, partitioning, enhancement, and preprocessing. The performance of the model is highly dependent on the quality and format of the input data. Collect images of power equipment (such as transformers, switches, generators, etc.) to ensure that the samples cover normal and fault conditions. These image data can be obtained through field shooting or historical data sets.

[0031] Next, the collected images are annotated to record the device type and its status (normal or faulty). Annotation tools (such as LabelImg) can be used to generate corresponding label files to ensure the accuracy and consistency of data annotation. Then, the dataset is divided into training set, validation set and test set, with 70%-80% of the data used for training and 20%-30% for validation and testing, thus ensuring the generalization ability of the model.

[0032] In order to improve the robustness of the model, the training set is enhanced, including random cropping, rotation, and flipping operations to increase data diversity and prevent model overfitting. After data enhancement, the data preprocessing step is performed. Data preprocessing is also a key step. The images need to be uniformly resized and normalized to meet the input requirements of the model and ensure the consistency of the input data, thereby enhancing the training effect of the model.

[0033] After completing data preparation, enter the model building phase. The first teacher model in this embodiment uses a pre-trained ResNet50 model, which can extract rich feature knowledge. The second teacher model uses a randomly initialized ResNet50 model to provide different learning perspectives. The student model selects a lightweight MobileNetV2 model, and the model parameters are randomly initialized. The MobileNetV2 model is a lightweight model that is suitable for fast reasoning in resource-constrained environments.

[0034] When defining the student model, construct the MobileNetV2Student class, load the MobileNetV2 architecture, and add a fully connected layer (nn.Linear) for classification, outputting the probability distribution of the categories. Use the Softmax function to normalize the output of the student model for multi-class classification tasks.

[0035] During the training process, the Adam optimizer is used, and the initial learning rate is set to 0.01. The Adam optimizer combines the advantages of momentum and adaptive learning rate, and is suitable for dealing with sparse gradients and large-scale data problems.

[0036] In order to ensure the model training effect, this embodiment designs a total loss function Ltotal for the task of jointly training the student model with the first and second teacher models. The total loss function Ltotal includes classification loss, distillation loss and feature loss.

[0037] The classification loss Lclass is the cross entropy loss between the output of the student model and the true label. The cross entropy loss is used to evaluate the difference between the predicted value of the student model and the true label. The calculation formula is:

[0038]

[0039] Where N is the number of categories, yi is the one-hot encoding of the actual label, and p i is the predicted probability of the student model.

[0040] Distillation loss is used to measure the difference in probability distribution between the student model and the teacher model. The distillation loss is calculated based on the Kullback-Leibler divergence between the output of the student model and the output of the teacher model. The temperature parameter is used to control the smoothness of the soft target. The distillation loss includes the first distillation loss and the second distillation loss. The first distillation loss Ldistill1 is the Kullback-Leibler divergence between the output of the student model and the output of the pre-trained first teacher model. The calculation formula is:

[0041] Ldistill1=KL(softmax(s / T),softmax(t1 / T))

[0042] Wherein, KL represents the calculation of Kullback-Leibler divergence, s is the logits of the student model, that is, the original score output by the student model, t1 is the logits of the first teacher model, that is, the original score output by the first teacher model, and T is the temperature parameter, which is set to 2.0 in this embodiment.

[0043] The second distillation loss Ldistill2 is the Kullback-Leibler divergence between the output of the student model and the output of the randomly initialized second teacher model, and is calculated as:

[0044] Ldistill2=KL(softmax(s / T),softmax(t2 / T))

[0045] Among them, t2 is the logits of the second teacher model, that is, the original score output by the second teacher model.

[0046] Feature loss Lfeature: The features output by the teacher model and the student model after convolution and pooling are used as the intermediate layer features. The loss between the intermediate layer features of the student model and the intermediate layer features of the first teacher model and the intermediate layer features of the second teacher model is calculated as the first feature loss Lfeature. feature1 and the second feature loss L feature2 , and then weight them together. The first feature loss and the second feature loss are calculated using the L2 loss function. The formula for feature loss is:

[0047] Lfeature=a1L feature1 +a2L feature2

[0048] Among them, a1 and a2 are hyperparameters for adjusting the weights of each loss item.

[0049] The calculation formula of the total loss function is:

[0050] Ltotal=Lclass+αLdistill1+βLdistill2+γLfeature

[0051] Among them, α, β, and γ are hyperparameters for adjusting the weights of each loss term.

[0052] Next, define the DistillationTrainer class, which is responsible for managing the entire training process. The initialization part of the DistillationTrainer class receives the student model and two teacher models and defines the loss function. The distillation loss function is used to calculate the Kullback-Leibler divergence between the output of the student model and the output of the teacher model, and uses the temperature parameter (usually set to 2.0) to smooth the target distribution and improve the effectiveness of model learning.

[0053] The training function executes the training loop and sets the model to training mode. It performs forward propagation on the input of each batch, obtains the output of the three models, calculates the classification loss and distillation loss, and combines them into the total loss. It updates the parameters of the student model through backpropagation to optimize the model performance.

[0054] Finally, start model training using the prepared dataset. Create an instance of DistillationTrainer and call the train method. The training will continue for a predetermined number of rounds and output the loss value of each round to monitor the training effect.

[0055] The model training process is as follows:

[0056] Step 1: Initialize the training process: load the pre-trained first teacher network, the randomly initialized second teacher network, and the student network;

[0057] Step 2, data input: input the training data in the training set into the first teacher network, the second teacher network and the student network;

[0058] Step 3, forward propagation: calculate the output and intermediate layer features of the first teacher network; calculate the output of the second teacher network; calculate the output of the student network;

[0059] Step 4: Loss function calculation:

[0060] In the total loss calculation, the final total loss (Ltotal) is calculated as follows:

[0061] Ltotal=Lclass+αLdistill1+βLdistill2+γLfeature

[0062] Among them, α, β, and γ are hyperparameters for adjusting the weights of each loss item, which are set to 0.25 here.

[0063] Step 5, back propagation: According to the calculated total loss, the parameters of the student network and the second teacher network are updated through the back propagation algorithm;

[0064] Step 6: Iterative training: Repeat steps 2 to 5 until the model converges or reaches the preset number of training rounds to complete the training.

[0065] After the model training is completed, the validation set is used to verify the generalization ability of the student network, and the test set is used to evaluate the generalization ability of the model. After the test is completed, the student model is deployed to realize the monitoring of power equipment images.

[0066] Embodiment 2

[0067] This embodiment provides a power equipment monitoring system based on knowledge distillation, including:

[0068] A data acquisition module is configured to acquire monitoring images of power equipment;

[0069] The monitoring module is configured to input the power equipment monitoring image into the trained student model to obtain the power equipment monitoring result; wherein, the student model is obtained by joint training of the first teacher model and the second teacher model; the first teacher model is a pre-trained deep learning model, and the second teacher model is a deep learning model with randomly initialized parameters; the student model, the first teacher model and the second teacher model are jointly trained using the training set data, and the training is completed when the model converges or meets the preset conditions to obtain the trained student model.

[0070] It should be noted here that the various modules in this embodiment correspond one-to-one to the steps of the method in Example 1, and the specific implementation process is the same, which will not be repeated here.

[0071] Embodiment 3

[0072] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the method in the first embodiment.

[0073] The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor may be implemented in at least one of the following hardware forms: DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and LA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0074] The memory may include one or more computer-readable media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable medium in the memory is used to store at least one computer program, which is used to be executed by the processor to implement a power equipment monitoring method based on knowledge distillation provided in an embodiment of the present disclosure.

[0075] Those skilled in the art can understand that the electronic device provided in this embodiment can include more or fewer components, or combine certain components, or adopt different component arrangements.

[0076] Embodiment 4

[0077] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the method in the first embodiment are completed.

[0078] Embodiment 5

[0079] This embodiment provides a computer program product, including a computer program / instruction, which implements the steps of the method in Embodiment 1 when executed by a processor.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A power equipment monitoring method based on knowledge distillation, characterized in that: include: Acquire monitoring images of power equipment; Input the power equipment monitoring image into the trained student model to obtain the power equipment monitoring result; Among them, the student model is obtained by jointly training the first teacher model and the second teacher model; the first teacher model is a pre-trained deep learning model, and the second teacher model is a deep learning model with randomly initialized parameters; the student model, the first teacher model and the second teacher model are jointly trained using training set data, and the training is completed when the model converges or meets the preset conditions to obtain a trained student model.

2. The method for monitoring electric power equipment based on knowledge distillation according to claim 1, characterized in that: The total loss function during the joint training of the student model, the first teacher model, and the second teacher model is obtained by the weighted sum of classification loss, distillation loss, and feature loss.

3. The method for monitoring electric power equipment based on knowledge distillation according to claim 2, characterized in that: The classification loss is the cross entropy loss between the output of the student model and the true label.

4. The method for monitoring electric power equipment based on knowledge distillation according to claim 2, characterized in that: The distillation loss includes a first distillation loss and a second distillation loss, the first distillation loss is the Kullback-Leibler divergence between the student model output and the pre-trained first teacher model output, and the second distillation loss is the Kullback-Leibler divergence between the student model output and the randomly initialized second teacher model output.

5. The method for monitoring electric power equipment based on knowledge distillation according to claim 2, characterized in that: The feature loss is obtained by weighting the L2 loss between the intermediate layer features output by the student model and the intermediate layer features output by the first teacher model and the second teacher model.

6. The method for monitoring electric power equipment based on knowledge distillation according to claim 1, characterized in that: The first teacher model is a pre-trained ResNet50 model, the second teacher model uses a randomly initialized ResNet50 model, and the student model is a MobileNetV2 model.

7. A power equipment monitoring system based on knowledge distillation, characterized in that: include: A data acquisition module is configured to acquire monitoring images of power equipment; The monitoring module is configured to input the power equipment monitoring image into the trained student model to obtain the power equipment monitoring result; wherein, the student model is obtained by joint training of the first teacher model and the second teacher model; the first teacher model is a pre-trained deep learning model, and the second teacher model is a deep learning model with randomly initialized parameters; the student model, the first teacher model and the second teacher model are jointly trained using the training set data, and the training is completed when the model converges or meets the preset conditions to obtain the trained student model.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images