Transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning

Abstract

The invention relates to the technical field of power transformer fault diagnosis, and discloses a transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning. The method comprises the following steps: firstly, training a cloud high-performance model by using historical fault data of a transformer, lightening a large model obtained by cloud training by using a knowledge distillation mode, and deploying the large model at an edge side; secondly, constructing a cloud edge collaboration framework, performing on-site processing and identification on each piece of monitoring data on the edge side, and only sending an identification result to a cloud end to relieve the data transmission pressure; and finally, establishing a collaborative incremental learning strategy, and when the models are misaligned along with the increase of new samples, updating the cloud and edge side models by adopting a collaborative incremental learning mode based on knowledge distillation, thereby realizing synchronous updating of the cloud and edge side models, ensuring that the models can continuously and effectively learn and adapt to the new samples in an edge computing environment, and improving the accuracy of the new samples. And the fault diagnosis accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power transformer fault diagnosis, and in particular to a transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning. Background Art

[0002] In recent years, computer technology and sensor technology have developed rapidly. With the large-scale deployment of a large number of intelligent devices in the power system, the amount of real-time data collected by the power Internet of Things such as online monitoring and intelligent inspection has increased exponentially, but it also brings many challenges: First, the deployment of deep learning models requires high computing resources and is difficult to implement on the resource-constrained edge side, while lightweight models face the problems of insufficient feature extraction and limited generalization capabilities; second, transformer fault types are diverse and complex. For example, partial discharge signals are greatly affected by factors such as defect geometry, external electric field conditions, and defect location equipment materials, which may cause the newly emerging samples to differ significantly from the samples learned by the model, and the resource allocation required for retraining is difficult to meet on the edge side; in addition, transmitting a large amount of monitoring data to the cloud for centralized processing can solve the problem of computing and storage capabilities, but it will occupy a large amount of communication bandwidth during data transmission, causing the communication channel to be overloaded, and the computing resources on the edge are limited, making it difficult to cope with tasks with high computing requirements. How to effectively process and utilize these data in an edge computing environment is an urgent problem that needs to be solved.

[0003] In order to alleviate the pressure of data transmission, reasonably allocate computing resources and improve the efficiency of system operation, cloud-edge collaboration can be adopted, which refers to an operating mechanism for collaborative work between cloud computing and edge computing. The core idea is to integrate the advantages of cloud computing and edge computing to achieve more flexible and efficient data processing and computing capabilities. Under the cloud-edge collaborative architecture, the edge is responsible for real-time processing and preliminary analysis of data, and only uploads important recognition results or summary data to the cloud, reducing the pressure of data transmission and optimizing the use of communication resources.

[0004] However, pure cloud-edge collaboration still faces some problems. With the operation of the power system, new sample data that differs greatly from the model learning samples continues to increase, and the existing diagnostic model may become inaccurate. Retraining may lead to the "forgetting" of historical sample data (decreased recognition ability) or the "flooding" of a small amount of new sample data (poor learning effect). Incremental learning using a progressive correction model can effectively deal with this problem. However, due to limited computing resources, the application of traditional incremental learning methods in edge computing environments has poor learning effects on new samples. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of model updating in edge computing environment and provide a transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning. According to the technical solution of the present invention, the synchronous update of cloud and edge models is realized through the collaborative incremental learning mechanism, which ensures that the model can continuously and effectively learn and adapt to new samples in the edge computing environment, thereby improving the accuracy of fault diagnosis.

[0006] In order to achieve the above object, the present invention provides a transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning, which comprises the following steps:

[0007] (1) Use the historical transformer data collected on site to build training datasets for high-performance and lightweight models on the cloud;

[0008] (2) Deploy the lightweight model in its initial state on the edge side and wait for the cloud to send the trained model parameters;

[0009] (3) Build a pattern recognition model in the cloud and input the data set into a high-performance model for training. Then, through knowledge distillation technology, use the high-performance model to guide the training of the lightweight model. After the training is completed, save the model parameters to the cloud.

[0010] (4) Send the lightweight model weight parameters trained in the cloud to the edge side through wireless communication to achieve model deployment;

[0011] (5) After receiving the monitoring data, the edge device first processes the raw data appropriately, then inputs the data into the recognition model for fault recognition, and determines whether the highest category probability exceeds 80% based on the model output result. If it exceeds 80%, execute step (6); if it is less than 80%, go to step (7);

[0012] (6) The recognition results on the edge side are encrypted and uploaded to the cloud, and the corresponding monitoring data is stored in the local database and uploaded to the cloud as needed. At the same time, a fault type prompt is sent to the operation and maintenance personnel for subsequent maintenance and repair work;

[0013] (7) When the credibility of the output result of the edge-side model does not meet the standard, the input data and output result are encrypted and uploaded to the cloud, where the high-performance model on the cloud performs judgment. If the result meets the standard, the lightweight model is updated through incremental learning. If the result does not meet the standard, the operation and maintenance personnel judge and annotate the data, and after updating the high-performance model through incremental learning, repeat steps (3) to (5) to achieve synchronous update of the lightweight model on the edge.

[0014] Preferably, in step (1), taking partial discharge data as an example, the collected original partial discharge data can be expressed by the partial discharge signal using formula (1):

[0015] s(t)=f(t)+ε(t) (1)

[0016] Among them, s(t) is the partial discharge monitoring signal containing noise; f(t) is the effective part of the signal; ε(t) is the noise signal.

[0017] Preferably, the noisy signal s(n) is subjected to discrete wavelet transform to obtain the following formula (2):

[0018] ∫s(t)ψ j,k (t)dt=∫f(t)ψ j,k (t)dt+∫ε(t)ψ j,k (t)dt (2)

[0019] Among them, ψ j,k (t) is the wavelet basis function; j is the scale parameter of the wavelet basis function; k is the translation parameter of the wavelet basis function.

[0020] Preferably, the goal of wavelet denoising is to eliminate the influence of the noise signal ε(t) to the maximum extent, thereby retaining the effective signal f(t). The wavelet denoising process includes the following three parts:

[0021] 1) Wavelet decomposition: First, the signal S to be processed is decomposed into sub-band signals of different scales and frequencies through wavelet transform. The decomposed signals are called approximate coefficients and detail coefficients. The approximate coefficient A i is the low-frequency part of the signal. The approximate coefficient mainly reflects the overall trend of the signal, while the detail coefficient D i It reflects the details and high-frequency components of the signal;

[0022] 2) Threshold processing: In the wavelet domain, it is usually assumed that the detail coefficients of the signal mainly contain noise, while the approximate coefficients mainly contain the effective information of the signal;

[0023] 3) Reconstructing the signal: After threshold processing, the processed approximate coefficients and detail coefficients are reconstructed through inverse wavelet transform to obtain the denoised signal S`.

[0024] Preferably, in step (3), the high-performance model is trained using historical data, and the inheritance of prior knowledge is achieved by minimizing the difference in soft label outputs between the lightweight model and the high-performance model.

[0025] Preferably, the soft label is in the form of a probabilistic label with a value between 0 and 1. Compared with the binary labels of 0 and 1 output by a conventional neural network, it not only indicates the category attribute, but also contains implicit information between different categories.

[0026] Preferably, the neural network outputs the probability of each type by using a "softmax" layer, which is expressed as:

[0027]

[0028] Where: zi,zj are the Logits output after the data is input into the network; qi is the output of softmax, that is, the model prediction result.

[0029] Preferably, after introducing the distillation temperature T, the softmax expression becomes:

[0030]

[0031] As the distillation temperature T value increases, the output distribution of the model will become flatter and the distribution entropy will decrease;

[0032] Under the guidance of prior knowledge, the difference between the hard label output and the true label is minimized as much as possible. The function expression is as follows:

[0033]

[0034] Loss hard =-∑ i p i logq i (7)

[0035] Where pi is the label.

[0036] Preferably, when using knowledge distillation training, first the teacher network predicts a soft label at the distillation temperature T=t, and the soft label is passed to the student network as knowledge; then at the distillation temperature T=t, the prediction result of the student network and the soft label of the teacher network are calculated to obtain the distillation loss Losssoft.

[0037] Preferably, the classification loss Losshard is calculated by comparing the result obtained by the student network at the distillation temperature T = 1 with the true label. Since the gradients provided by the two loss functions are different during the training process, when calculating the total loss function Loss, it is necessary to multiply the distillation loss Losssoft by T before calculating it. 2 , the total loss function expression is as follows:

[0038] Loss = αL hard +(1-α)T 2 L soft (8)

[0039] In the formula, α is the weight coefficient.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] (1) Cloud-edge collaborative incremental learning: Through the cloud-edge collaborative incremental learning mechanism, the edge-side model is continuously updated and optimized, solving the problem of the traditional method of effectively processing new samples in the edge computing environment, and improving the adaptability and accuracy of the model;

[0042] (2) Hierarchical processing and decision-making mechanism: Through knowledge distillation technology, the knowledge of high-performance models in the cloud is transferred to lightweight models, so that the models deployed in the edge computing environment have both strong recognition capabilities and can meet the requirements of resource constraints. Using the hierarchical processing and decision-making mechanism, the edge-side model can independently perform preliminary fault identification and decide whether to upload the results to the cloud for further analysis based on the credibility of the results. This mechanism not only improves the real-time performance of the system, but also reduces the burden on communication bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a cloud-edge collaboration flow chart;

[0044] Figure 2 This is a schematic diagram of the incremental learning principle based on knowledge distillation;

[0045] Figure 3 The relationship between the original signal and the denoised signal is shown. DETAILED DESCRIPTION

[0046] The specific embodiments of the present invention are described in detail below. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0047] The present invention provides a transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning. First, the high-performance cloud model is trained using the transformer historical fault data, and the large model trained on the cloud is lightweighted by knowledge distillation and deployed on the edge side; secondly, a cloud-edge collaborative framework is constructed, and each monitoring data is processed and identified on the edge side, and only the identification result needs to be sent to the cloud to relieve the pressure of data transmission; finally, a collaborative incremental learning strategy is established. When the model becomes inaccurate as the number of new samples increases, the cloud and edge models are updated by collaborative incremental learning based on knowledge distillation, so as to achieve synchronous update of the cloud and edge models, ensuring that the model can continuously and effectively learn and adapt to new samples in the edge computing environment, and improve the accuracy of fault diagnosis.

[0048] Specifically, Figure 1 As shown, the transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning of the present invention includes:

[0049] (1) The historical transformer data collected on site is used to build a training data set for the cloud-based high-performance model and lightweight model. Taking partial discharge data as an example, the collected original partial discharge data can be expressed by partial discharge signal using formula (1):

[0050] s(t)=f(t)+ε(t) (1)

[0051] Where: s(t) is the partial discharge monitoring signal containing noise; f(t) is the effective part of the signal; ε(t) is the noise signal.

[0052] By performing discrete wavelet transform on the noisy signal s(n), we can get the following formula:

[0053] ∫s(t)ψ j,k (t)dt=∫f(t)ψ j,k (t)dt+∫ε(t)ψ j,k (t)dt (2)

[0054] Where: j,k (t) is the wavelet basis function; j is the scale parameter of the wavelet basis function; k is the translation parameter of the wavelet basis function.

[0055] Formula (1) can also be expressed as:

[0056] d j,k =u j,k +e j,k (3)

[0057] Where: d j,k is the wavelet coefficient of each layer after wavelet transform of the noisy signal s(t); u j,k is the wavelet transform coefficient of the effective part of the signal f(t); e j,k is the wavelet transform coefficient of the noise part of the signal ε(t)

[0058] The goal of wavelet denoising is to eliminate the influence of the noise signal ε(t) to the maximum extent possible, thereby retaining the effective signal f(t). The wavelet denoising process mainly includes the following three parts:

[0059] 1) Wavelet decomposition: First, the signal S to be processed is decomposed into sub-band signals of different scales and frequencies through wavelet transform. The decomposed signals are called approximate coefficients and detail coefficients, such as Figure 2-3 As shown, the approximate coefficient A i is the low-frequency part of the signal. The approximate coefficient mainly reflects the overall trend of the signal, while the detail coefficient D i It reflects the details and high-frequency components of the signal.

[0060] 2) Threshold processing: In the wavelet domain, it is usually assumed that the detail coefficients of the signal mainly contain noise, while the approximate coefficients mainly contain the effective information of the signal. Therefore, the detail coefficients can be threshold processed, and the detail coefficients less than a certain threshold are set to zero, thereby suppressing the noise signal.

[0061] 3) Reconstructing the signal: After threshold processing, the processed approximate coefficients and detail coefficients are reconstructed through inverse wavelet transform to obtain the denoised signal S`.

[0062] Construct a dataset using the denoised signal.

[0063] (2) Deploy a lightweight model in its initial state on the edge and wait for the cloud to send the trained model parameters.

[0064] (3) Build a pattern recognition model (including a high-performance model and a lightweight model) in the cloud, and input the data set into the high-performance model for training. Then, through knowledge distillation technology, use the high-performance model to guide the training of the lightweight model, and save the model parameters to the cloud after training is completed.

[0065] The high-performance model is trained using historical data, and the inheritance of prior knowledge is achieved by minimizing the difference in soft label output between the lightweight model and the high-performance model. Among them, the soft label is a probabilistic label with a value between 0 and 1. Compared with the binary labels of 0 and 1 (i.e. hard labels) output by conventional neural networks, it can not only indicate the category attributes, but also contain implicit information between different categories.

[0066] Neural networks usually output the probability of each type by using a "softmax" layer, which is expressed as:

[0067]

[0068] Where: zi,zj are the Logits output after the data is input into the network; qi is the output of softmax, that is, the model prediction result.

[0069] After introducing the distillation temperature T, the softmax expression becomes:

[0070]

[0071] As the distillation temperature T value increases, the distribution of the model's output results will become flatter and the distribution entropy will become smaller.

[0072] Under the guidance of prior knowledge, the difference between the hard label output and the true label is minimized (i.e., "hard" loss, classification loss Losshard). The loss function expression is as follows:

[0073]

[0074] Loss hard =-∑ i p i logq i (7)

[0075] Where pi is the label;

[0076] When using knowledge distillation training, the teacher network first predicts a soft label at the distillation temperature T = t, and the soft label is passed to the student network as knowledge. Then, at the distillation temperature T = t, the prediction result of the student network and the soft label of the teacher network are calculated to obtain the distillation loss Losssoft. The result obtained by the student network at the distillation temperature T = 1 and the true label are calculated to obtain the classification loss Losshard. Since the gradients provided by the two loss functions during the training process are different, when calculating the total loss function Loss, it is necessary to multiply the distillation loss Losssoft by T 2. The total loss function expression is as follows:

[0077] Loss = αL hard +(1-α)T 2 L soft (8)

[0078] In the formula, α is the weight coefficient.

[0079] (4) The weight parameters of the lightweight model trained in the cloud are sent to the edge side through wireless communication to realize model deployment.

[0080] (5) After receiving the monitoring data, the edge device first processes the raw data appropriately and then inputs it into the recognition model for fault identification. Based on the model output, determine whether the highest category probability exceeds 80% (this standard is determined by the mean of the highest probability output by the test set and may vary due to differences in data sets). If it exceeds 80%, execute step (6); if it is less than 80%, go to step (7).

[0081] (6) The recognition results on the edge side are encrypted and uploaded to the cloud, and the corresponding monitoring data is stored in the local database. Upload to the cloud as needed, and send fault type prompts to the operation and maintenance personnel to facilitate subsequent maintenance and repair work.

[0082] (7) When the output results of the edge model are not credible enough, the input data and output results are encrypted and uploaded to the cloud. The cloud-based high-performance model makes a judgment. If the result meets the standard, the lightweight model is updated through incremental learning based on knowledge distillation. If the result does not meet the standard, the operation and maintenance personnel judge and annotate the data, and also update the high-performance model through incremental learning based on knowledge distillation. Then, steps (3) to (5) are repeated to achieve synchronous update of the lightweight model on the edge.

[0083] Incremental learning based on knowledge distillation can share and efficiently utilize prior knowledge by repeatedly accessing a small number of representative old data, and assist in achieving self-update of the model for new data without having to repeatedly learn all the old data. The original model M is a stable deep network trained based on the old data set. The new model M′ is generally consistent with the structure of M and is initialized with the help of M. During the training phase, only M′ is updated for parameters. First, prior knowledge is transferred by constraining the soft label outputs of M and M′ for representative old data to be close to protect the previous learning; under the guidance of prior knowledge, M′ is committed to narrowing the difference between the hard label outputs of new and old data and the true labels to correct the model parameters to adapt to the new data and further consolidate the previous learning without retraining the model, which is more efficient and takes up less computing resources; finally, M′ replaces M for the next test.

[0084] The present invention combines the advantages of cloud-edge collaboration and incremental learning, which can reduce data transmission delays, reduce network bandwidth pressure, and improve the response speed and reliability of the system. When edge resources are limited, the fault recognition accuracy and real-time performance can be effectively improved through collaborative incremental learning, which solves the problem of model updating in edge computing environments. It provides an efficient and flexible solution for transformer fault diagnosis in power systems, which can be further extended to the monitoring of other equipment in power systems.

[0085] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited thereto. Within the technical concept of the present invention, the technical solution of the present invention can be subjected to a variety of simple modifications, including the combination of various technical features in any other suitable manner, and these simple modifications and combinations should also be regarded as the contents disclosed by the present invention and belong to the protection scope of the present invention.

Claims

1. A transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning, characterized in that: The method comprises the following steps: (1) Use the historical transformer data collected on site to build training datasets for high-performance and lightweight models on the cloud; (2) Deploy the lightweight model in its initial state on the edge side and wait for the cloud to send the trained model parameters; (3) Build a pattern recognition model in the cloud and input the data set into a high-performance model for training. Then, through knowledge distillation technology, use the high-performance model to guide the training of the lightweight model. After the training is completed, save the model parameters to the cloud. (4) Send the lightweight model weight parameters trained in the cloud to the edge side through wireless communication to achieve model deployment; (5) After receiving the monitoring data, the edge device first processes the raw data appropriately, then inputs the data into the recognition model for fault recognition, and determines whether the highest category probability exceeds 80% based on the model output result. If it exceeds 80%, execute step (6); if it is less than 80%, go to step (7); (6) The recognition results on the edge side are encrypted and uploaded to the cloud, and the corresponding monitoring data is stored in the local database and uploaded to the cloud as needed. At the same time, a fault type prompt is sent to the operation and maintenance personnel for subsequent maintenance and repair work; (7) When the credibility of the output result of the edge-side model does not meet the standard, the input data and output result are encrypted and uploaded to the cloud, where the high-performance model on the cloud performs judgment. If the result meets the standard, the lightweight model is updated through incremental learning. If the result does not meet the standard, the operation and maintenance personnel judge and annotate the data, and after updating the high-performance model through incremental learning, repeat steps (3) to (5) to achieve synchronous update of the lightweight model on the edge.

2. The transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning according to claim 1 is characterized in that: In step (1), taking partial discharge data as an example, the collected original partial discharge data can be expressed by partial discharge signal using formula (1): s(t)=f(t)+ε(t) (1) Among them, s(t) is the partial discharge monitoring signal containing noise; f(t) is the effective part of the signal; ε(t) is the noise signal.

3. The transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning according to claim 2 is characterized in that: Performing discrete wavelet transform on the noisy signal s(n) yields the following equation (2): ∫s(t)ψ j,k (t)dt=∫f(t)ψ j,k (t)dt+∫ε(t)ψ j,k (t)dt (2) Among them, ψ j,k (t) is the wavelet basis function; j is the scale parameter of the wavelet basis function; k is the translation parameter of the wavelet basis function.

4. The transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning according to claim 3 is characterized in that: The goal of wavelet denoising is to eliminate the influence of the noise signal ε(t) to the maximum extent, so as to retain the effective signal f(t). The wavelet denoising process includes the following three parts: 1) Wavelet decomposition: First, the signal S to be processed is decomposed into sub-band signals of different scales and frequencies through wavelet transform. The decomposed signals are called approximate coefficients and detail coefficients. The approximate coefficient A i is the low-frequency part of the signal. The approximate coefficient mainly reflects the overall trend of the signal, while the detail coefficient D i It reflects the details and high-frequency components of the signal; 2) Threshold processing: In the wavelet domain, it is usually assumed that the detail coefficients of the signal mainly contain noise, while the approximate coefficients mainly contain the effective information of the signal; 3) Reconstructing the signal: After threshold processing, the processed approximate coefficients and detail coefficients are reconstructed through inverse wavelet transform to obtain the denoised signal S`.

5. The transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning according to claim 1 is characterized in that: In step (3), the high-performance model is trained using historical data, and the inheritance of prior knowledge is achieved by minimizing the difference in soft label output between the lightweight model and the high-performance model.

6. The transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning according to claim 5 is characterized in that: Soft labels are probabilistic labels with values ​​between 0 and 1. Compared with the binary labels of 0 and 1 output by conventional neural networks, soft labels not only indicate category attributes, but also contain implicit information between different categories.

7. The transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning according to claim 6 is characterized in that: The neural network uses a "softmax" layer to output the probability of each type, which is expressed as: Where: zi,zj are the Logits output after the data is input into the network; qi is the output of softmax, that is, the model prediction result.

8. The transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning according to claim 7 is characterized in that: After introducing the distillation temperature T, the softmax expression becomes: As the distillation temperature T value increases, the output distribution of the model will become flatter and the distribution entropy will decrease; Under the guidance of prior knowledge, the difference between the hard label output and the true label is minimized as much as possible. The function expression is as follows: Loss hard =-∑ i p i logq i (7) Where pi is the label.

9. The transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning according to claim 8 is characterized in that: When using knowledge distillation training, the teacher network first predicts a soft label at the distillation temperature T = t, and the soft label is passed to the student network as knowledge; then, at the distillation temperature T = t, the prediction result of the student network and the soft label of the teacher network are calculated to obtain the distillation loss Losssoft.

10. The transformer fault diagnosis method based on knowledge distillation and collaborative incremental learning according to claim 9 is characterized in that: The classification loss Losshard is calculated by comparing the result of the student network at the distillation temperature T = 1 with the true label. Since the gradients provided by the two loss functions are different during the training process, when calculating the total loss function Loss, it is necessary to multiply the distillation loss Losssoft by T. 2 , the total loss function expression is as follows: Loss=αL hard +(1-α)T 2 L soft (8) In the formula, α is the weight coefficient.

Images