Power inspection image defect detection method based on federated learning and adaptive difference
By employing federated learning and adaptive differential techniques, the challenges of high training costs, data silos, and privacy protection in power line inspection image defect detection are addressed. This approach achieves high-precision power line inspection image defect detection, reduces communication costs, and enhances robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTH CHINA ELECTRIC POWER UNIV
- Filing Date
- 2022-05-16
- Publication Date
- 2026-07-21
AI Technical Summary
Existing methods for detecting defects in power line inspection images face challenges such as high training costs, data silos, and privacy protection. Traditional methods cannot meet the requirements for privacy protection and secure data sharing.
We employ a federated learning and adaptive differential approach, which involves collaborative training between a central server and the client. We use differential privacy technology to add noise locally to protect privacy, and a knowledge distillation strategy to accelerate model convergence and reduce the number of communications.
It achieves high-precision defect detection in power line inspection images while ensuring privacy protection and secure data sharing, reduces training costs and communication costs, and enhances the robustness of federated learning.
Smart Images
Figure CN114897837B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for detecting defects in power line inspection images, and more particularly to a method for detecting defects in power line inspection images based on federated learning and adaptive difference, belonging to the field of power inspection technology. Background Technology
[0002] With the rapid development of China's smart grid construction and artificial intelligence technology, defect detection in power grid inspection images has become a crucial task in ensuring power grid safety. The goal of power grid inspection image defect detection is to accurately identify the category and location information contained in each image. Mainstream power grid inspection image defect detection algorithms fall into two categories: traditional machine learning-based defect detection algorithms and deep learning-based defect detection algorithms. The former is quite inefficient in determining and analyzing the state of power grid inspection images, and the determination of parameter ranges and feature selection mainly relies on the experience of domain experts. Achieving good performance for deep learning-based target detection algorithms requires users to upload data to a data center for centralized training. However, inspection image data may contain national road infrastructure information and users' personal privacy information, resulting in a serious data silo phenomenon, hindering the further development of deep learning technology in the field of power grid inspection. With increasingly stringent privacy protection and data security management, and the growing prominence of the "data silo" phenomenon, the traditional method of relying on a large number of training data samples for centralized training can no longer fully comply with the State Grid's regulations on privacy protection and data security. Currently, power grid inspection image defect detection faces many challenges in its application, which can be summarized in the following three aspects:
[0003] (1) Training cost problem. Power inspection image defect detection algorithms usually adopt centralized training, that is, all labeled data samples are collected and a high-performance server is found for long-term training. However, this method has high requirements for the performance and storage of the training server, especially when facing training large-scale image data, the training time cost is very high.
[0004] (2) Data silo problem. The data silo refers to the fact that power inspection image data is often scattered across various power companies. The data is stored and maintained independently by different power companies, forming physical islands. Because power data usually has confidentiality requirements, it is difficult to share data between different departments. The data silo problem greatly limits the usable scale and value of the data.
[0005] (3) Privacy Protection Issues. During drone inspections, the image data captured by cameras contains a large amount of information about national infrastructure roads and users' homes, which can lead to the leakage of security and privacy information during target detection. With the promulgation of China's Data Security Law and the increasing awareness of privacy protection among individuals, existing target detection methods can no longer fully meet the requirements for privacy protection and secure data sharing. Summary of the Invention
[0006] The technical problem to be solved by this invention is to provide a method for detecting defects in power line inspection images based on federated learning and adaptive differential.
[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0008] A defect detection method for power line inspection images based on federated learning and adaptive difference includes the following steps:
[0009] Step 1: Construct a federated learning scenario: This includes a central server and at least K clients, where K>1; the central server and each client are bidirectionally connected; each client receives the model from the central server; the model includes an input layer, an output layer, and at least one intermediate layer; the central server sets the initial model parameters, the number of iterations t=0, and the preset detection accuracy; the central server configures the initial model parameters as the current global model parameters.
[0010] Step 2: Data preprocessing: The data includes a normal state image set and an image set of more than one fault state. Each image in each image set is labeled with a category label, the top left x-coordinate, the top left y-coordinate, the bottom right x-coordinate, and the bottom right y-coordinate.
[0011] Step 3: Central server distributes training data: The central server divides the image data into K groups according to a non-independent and identically distributed method, and distributes one group of data to K clients as local training data for each client;
[0012] Step 4: Central server distributes model parameters: The central server transmits the current model parameters and iteration count to each client as the initial model parameters for each client. K is the number of clients;
[0013] Step 5: Client-side model parameter training: Increment the training iterations by 1. Each client trains its local model using local data to obtain local model parameters. When the local model reaches the preset convergence or termination condition, local training ends, and the local model parameters are updated. It is the learning rate of client i in the t-th iteration. It is the model gradient of client i in the t-th iteration;
[0014] Step 6: Upload local model parameters: Each client adds Gaussian noise that satisfies differential privacy to its local model parameters and uploads the noisy local model parameters to the central server;
[0015] Step 7: Global Model Parameter Aggregation: The central server performs weighted aggregation of the contribution weights of the global model based on the model parameters of each client, resulting in updated global model parameters. This process consists of the following specific steps:
[0016] Step 7-1: Calculate the contribution weight of each client model parameter to the global model aggregation update;
[0017] Step 7-2: After receiving the model parameters uploaded by the client, the server calculates the contribution weights based on step 7-1, and then weights and aggregates the model parameters to obtain the updated global model parameters;
[0018] Step 8: Calculate the server's total model loss: The central server distributes the aggregated global model parameters to each client. Each client calculates its local loss under the aggregated global model parameters and uploads it to the central server. The central server calculates the server's total model loss using the following method:
[0019]
[0020] Where n i Let i be the number of data samples from the i-th client. The sum of client data samples, F i (Z i,t ) represents the local loss of the model Z in the t-th iteration of client i, H i,t The weights of the contribution of the model parameters to the global model aggregation after the local differential privacy perturbation in the t-th iteration of client i, f(Z). t ) represents the total loss of the t-th iteration of the central server aggregation model;
[0021] Determine whether the total loss of the server's model reaches the preset detection accuracy or is greater than the number of training iterations; if yes, proceed to step 9; otherwise, proceed to step 4.
[0022] Step 9: Update client model parameters: The central server sends the optimal model parameters to each client, and each client updates its local model parameters with the optimal model parameters.
[0023] Furthermore, step 6 includes the following specific steps:
[0024] Step 6-1: Each client uses its local model to extract features from a set of input images and obtain the corresponding feature matrix;
[0025] Step 6-2: Group the obtained feature matrices according to the three channels of red, green and blue, and perform ReLU activation function operation on the feature matrix of each channel to obtain the red, green and blue feature matrices;
[0026] Step 6-3: Calculate the Hadamard product matrix W from the red, green, and blue feature matrices. M Add Gaussian noise to elements greater than 0:
[0027] W′ M =W M +N'
[0028] In the formula, W′ M N is the feature matrix after adding Gaussian noise; N' is the noise matrix;
[0029] Step 6-4: The client iterates over the local model parameters to client i in the t-th round. Add Gaussian noise N:
[0030]
[0031] Among them, W′ t,L The model parameters after adding Gaussian noise to the local client, where the Gaussian noise has a mean of 0 and a covariance of (Δfσ). 2 The model parameter σ is a Gaussian distribution, where I is the identity matrix, Δf is the maximum range of variation of the model parameters, and σ is the maximum range of variation of the model parameters. The value of ∈ can be determined by the user; the smaller the value of ∈, the higher the level of privacy protection.
[0032] Compared with the prior art, the present invention has at least the following beneficial effects:
[0033] (1) Under the premise of privacy protection and secure data sharing, this invention enables collaborative training among participants to ensure high accuracy of defect detection in power inspection images;
[0034] (2) Based on federated learning, this invention introduces differential privacy technology, which transfers the data storage and model training stages of deep learning to the local area. The local user only interacts with the central server to obtain model parameters. At the same time, the client adaptively controls the addition of noise according to the data characteristics, avoiding the reverse inference of pixel-level image and label information, thereby effectively ensuring privacy protection and data security.
[0035] (3) The present invention adopts a knowledge distillation strategy to transfer the knowledge learned by the global model to the local model, thereby accelerating the convergence of the local model and effectively reducing the number of communication times and the communication cost.
[0036] (4) The present invention adaptively selects the location to add noise based on local training data samples, thereby mitigating the negative impact of noise on model accuracy;
[0037] (5) The present invention adopts an evaluation mechanism based on the contribution of client model parameters, which effectively evaluates the contribution ability of each participating client model training to the aggregation of global model parameters, and performs weighted aggregation of global model parameters based on the contribution weight of model parameters, effectively reducing the impact of the existence of malicious devices on the accuracy of federated learning aggregation model, and ensuring that federated learning is more robust. Attached Figure Description
[0038] Figure 1 This is a flowchart of the present invention;
[0039] Figure 2 This is a schematic diagram of the federated learning architecture according to an embodiment of the present invention;
[0040] Figure 3 This is a schematic diagram illustrating the addition of noise to uploaded local model parameters in an embodiment of the present invention. Detailed Implementation
[0041] Example 1
[0042] Step 1: Construct a federated learning scenario: This includes a central server and at least K clients, where K>1; the central server and each client are bidirectionally connected; each client receives the model from the central server; the model includes an input layer, an output layer, and at least one intermediate layer; the central server sets the initial model parameters, the number of iterations t=0, and the preset detection accuracy; the central server configures the initial model parameters as the current global model parameters.
[0043] Step 2: Data preprocessing: The data includes a normal state image set and an image set of more than one fault state. Each image in each image set is labeled with a category label, the top left x-coordinate, the top left y-coordinate, the bottom right x-coordinate, and the bottom right y-coordinate.
[0044] Step 3: Central server distributes training data: The central server divides the image data into K groups according to a non-independent and identically distributed method, and distributes one group of data to K clients as local training data for each client;
[0045] Step 4: Central server distributes model parameters: The central server transmits the current model parameters and iteration count to each client as the initial model parameters for each client. K is the number of clients;
[0046] Step 5: Client-side model parameter training: Increment the training iterations by 1. Each client trains its local model using local data to obtain local model parameters. When the local model reaches the preset convergence or termination condition, local training ends, and the local model parameters are updated. It is the learning rate of client i in the t-th iteration. It is the model gradient of client i in the t-th iteration;
[0047] Step 6: Upload local model parameters: Each client adds Gaussian noise that satisfies differential privacy to its local model parameters and uploads the noisy local model parameters to the central server;
[0048] Step 7: Global Model Parameter Aggregation: The central server performs weighted aggregation of the contribution weights of the global model based on the model parameters of each client, resulting in updated global model parameters. This process consists of the following specific steps:
[0049] Step 7-1: Calculate the contribution weight of each client model parameter to the global model aggregation update;
[0050] Step 7-2: After receiving the model parameters uploaded by the client, the server calculates the contribution weights based on step 7-1, and then weights and aggregates the model parameters to obtain the updated global model parameters;
[0051] Step 8: Calculate the server's total model loss: The central server distributes the aggregated global model parameters to each client. Each client calculates its local loss under the aggregated global model parameters and uploads it to the central server. The central server calculates the server's total model loss using the following method:
[0052]
[0053] Where n i Let i be the number of data samples from the i-th client. The sum of client data samples, F i (Z i,t ) represents the local loss of the model Z in the t-th iteration of client i, H i,t The weights of the contribution of the model parameters to the global model aggregation after the local differential privacy perturbation in the t-th iteration of client i, f(Z). t ) represents the total loss of the t-th iteration of the central server aggregation model;
[0054] Determine whether the total loss of the server's model reaches the preset detection accuracy or is greater than the number of training iterations; if yes, proceed to step 9; otherwise, proceed to step 4.
[0055] Step 9: Update client model parameters: The central server sends the optimal model parameters to each client, and each client updates its local model parameters with the optimal model parameters.
[0056] Furthermore, step 6 includes the following specific steps, as shown in the appendix. Figure 3 As shown:
[0057] Step 6-1: Each client uses its local model to extract features from a set of input images and obtain the corresponding feature matrix;
[0058] Step 6-2: Group the obtained feature matrices according to the three channels of red, green and blue, and perform ReLU activation function operation on the feature matrix of each channel to obtain the red, green and blue feature matrices;
[0059] Step 6-3: Calculate the Hadamard product matrix W from the red, green, and blue feature matrices. M Add Gaussian noise to elements greater than 0:
[0060] W′ M =W M +N'
[0061] In the formula, W′ M N is the feature matrix after adding Gaussian noise; N' is the noise matrix;
[0062] Step 6-4: The client iterates over the local model parameters to client i in the t-th round. Add Gaussian noise N:
[0063]
[0064] Among them, W′ t,L The model parameters after adding Gaussian noise to the local client, where the Gaussian noise has a mean of 0 and a covariance of (Δfσ). 2 The model parameter σ is a Gaussian distribution, where I is the identity matrix, Δf is the maximum range of variation of the model parameters, and σ is the maximum range of variation of the model parameters. The value of ∈ can be determined by the user. The smaller the value of ∈, the higher the level of privacy protection. In this embodiment, the value is 1.
[0065] Furthermore, step 7 includes the following specific steps:
[0066] 7-1 Calculate the contribution weight of each client model parameter to the global model aggregation update. The specific calculation method is shown in equations (4)-(9).
[0067]
[0068] In equation (4), the element [a] ji(t) ] represents the change of the j-th dimension model parameters in the i-th client during the t-th iteration, (1≤j≤m, 1≤i≤k), where m is the dimension of the model parameters and k is the number of clients participating in the federated training during the t-th iteration;
[0069]
[0070] (5) In the formula [b ji(t)Let ] represent the mean change of the j-th dimension model parameter in k groups of clients in the t-th iteration, (1≤j≤m, 1≤i≤k), where m is the dimension of the model parameter and k is the number of clients participating in federated training in the t-th iteration.
[0071] D m×k(t) =[d ji(t) ] = A m×k(t) -B m×k(t) (6)
[0072]
[0073] Equations (6)-(7) are only used for deriving and calculating equation (8)e. iz(t) It has no other specific meaning.
[0074] Assume that the initial contribution weights of the models trained and uploaded locally by each client to the global model aggregation are the same, all being... k represents the number of clients participating in the federated training round t.
[0075]
[0076] In equation (8), H i,t This represents the contribution weight of the model uploaded by client i in the t-th iteration to the global model aggregation, SUM(e) iz(t) <0 indicates whether the model parameters uploaded by client i and client z in the t-th iteration mutually restrict each other's contributions to the global model aggregation. Here, k is the number of clients participating in federated training in the t-th iteration (1≤i, z≤k and i≠z). The calculation formula is as shown in (9).
[0077]
[0078] After receiving the model parameters uploaded by the client, the server calculates the contribution weights, aggregates the model parameters using weighted methods to obtain new global model parameters, and updates the global model parameters as shown in equation (10):
[0079]
[0080] Among them, W′ t,G Here, W' represents the model parameters of the central server after the t-th iteration update, k is the number of clients participating in federated training, and W' is the number of clients participating in federated training. i,t,L Let H be the model parameters of the local client i after local differential privacy perturbation in the t-th iteration. i,t Let be the contribution weight of the model parameters of client i after local differential privacy perturbation in the t-th iteration to the global model aggregation.
[0081] The image sample annotation process involves performing a series of operations on the acquired images, generating a txt / xml file with the same name as the image. The txt / xml file records target information including: category label, top-left x-coordinate, top-left y-coordinate, bottom-right x-coordinate, and bottom-right y-coordinate. This invention utilizes a high-resolution image dataset captured by a drone during power transmission line inspection. The dataset contains 1742 sample labels across four categories. Specifically, the label for a normal bolt is ls-zc (1286), the label for a bolt missing a nut is ls-qlm (59), the label for a bolt missing a pin is ls-qxd (174), and the label for a bolt missing a washer is ls-qdp (223). The dataset is partitioned by distributing the image data to the following K clients in a non-independent, identically distributed manner.
[0082] This embodiment uses a pre-trained bolt defect detection model as the initial global model for the central server. The initial global model is then distributed to all clients participating in the training. This initial global model refers to the model trained on the central server using partially labeled images, up to the required number of iterations or the desired accuracy.
[0083] The client uses its own local power line inspection image data for training. Local training ends and the local model is updated once the model converges or reaches a set termination condition. During local training, a knowledge distillation strategy is employed to accelerate the convergence of the local model, thereby reducing the number of communication calls and effectively lowering communication costs. The training process of the knowledge distillation strategy consists of the following two steps: First, the intermediate layers of the central server model are selected as a basis, and supervised learning is performed only on the intermediate layers of the local model, enabling the local model to quickly learn the intermediate feature representation capabilities of the central server model. Second, all model layers of the local model are trained.
[0084] The client adds noise to the trained model parameters using an adaptive local differential privacy strategy and then uploads the noisy model parameters to the server.
Claims
1. A method for defect detection in power line inspection images based on federated learning and adaptive difference, characterized in that, Includes the following steps: Step 1: Construct a federated learning scenario: This includes a central server and K clients, where K>1; the central server and each client are bidirectionally connected; each client receives the model from the central server; the model includes an input layer, an output layer, and one or more intermediate layers; the central server sets the initial model parameters, the number of iterations t=0, and the preset detection accuracy; the central server configures the initial model parameters as the current global model parameters. Step 2: Data preprocessing: The data includes a normal state image set and an image set of more than one fault state. Each image in each image set is labeled with a category label, the top left x-coordinate, the top left y-coordinate, the bottom right x-coordinate, and the bottom right y-coordinate. Step 3: Central server distributes training data: The central server divides the image data into K groups according to a non-independent and identically distributed method, and distributes one group of data to each client as local training data for each client; Step 4: Central server distributes model parameters: The central server transmits the current model parameters and iteration count to each client as the initial model parameters for each client. , K is the number of clients; Step 5: Client-side model parameter training: Increment the training iterations by 1. Each client trains its local model using local data to obtain local model parameters. When the local model reaches the preset convergence or termination condition, local training ends, and the local model parameters are updated. , It is the learning rate of client i in the t-th iteration. It is the model gradient of client i in the t-th iteration; Step 6: Upload local model parameters: Each client adds Gaussian noise that satisfies differential privacy to its local model parameters and uploads the noisy local model parameters to the central server; Step 7: Global Model Parameter Aggregation: The central server performs weighted aggregation of the contribution weights of the global model based on the model parameters of each client, resulting in updated global model parameters. This process consists of the following specific steps: Step 7-1: Calculate the contribution weight of each client model parameter to the global model aggregation update; Step 7-2: After receiving the model parameters uploaded by the client, the server calculates the contribution weights based on step 7-1, and then weights and aggregates the model parameters to obtain the updated global model parameters; Step 8: Calculate the server's total model loss: The central server distributes the aggregated global model parameters to each client. Each client calculates its local loss under the aggregated global model parameters and uploads it to the central server. The central server calculates the server's total model loss using the following method: in Let i be the number of data samples from the i-th client. The sum of client data samples For client i, the local loss of model Z in the t-th iteration, The weights of the contribution of the model parameters after local differential privacy perturbation in the t-th iteration of client i to the global model aggregation. The total loss of the central server aggregation model in the t-th iteration; Determine whether the total loss of the server's model reaches the preset detection accuracy or is greater than the number of training iterations; if yes, proceed to step 9; otherwise, proceed to step 4. Step 9: Update client model parameters: The central server sends the optimal model parameters to each client, and each client updates its local model parameters with the optimal model parameters; Step 6 includes the following specific steps: Step 6-1: Each client uses its local model to extract features from a set of input images and obtain the corresponding feature matrix; Step 6-2: Group the obtained feature matrices according to the three channels of red, green and blue, and perform ReLU activation function operation on the feature matrix of each channel to obtain the red, green and blue feature matrices; Step 6-3: Calculate the Hadamard product matrix of the red, green, and blue feature matrices. Add Gaussian noise to elements greater than 0: In the formula, The feature matrix after adding Gaussian noise; This is the noise matrix; Step 6-4: The client iterates over the local model parameters to client i in the t-th round. Add Gaussian noise N: in, The model parameters after adding Gaussian noise to the local client, where the Gaussian noise has a mean of 0 and a covariance of 0. The model is distributed by a Gaussian distribution, where I is the identity matrix, Δf is the maximum range of variation of the model parameters, and σ is... The value of ∈ can be determined by the user; the smaller the value of ∈, the higher the level of privacy protection.