Method and system for identifying defects of electric power unmanned aerial vehicle routing inspection side equipment based on auto-encoder

Abstract

The invention discloses an electric power unmanned aerial vehicle inspection side equipment defect identification method and system based on an auto-encoder, and the method comprises the steps: an edge device loads an electric power unmanned aerial vehicle inspection image, and carries out the preprocessing of the image; constructing an electric power unmanned aerial vehicle inspection model which comprises a training stage model and a reasoning stage model, and reasoning the pictures; and carrying out result post-processing, judging whether the data is processed or not after post-processing, if so, ending identification, and otherwise, returning. According to the method, the idea of an auto-encoder is introduced, an original backbone network is regarded as an encoder, and a decoder is added, so that additional auxiliary training information can be added for training of the model, and the backbone part of the model can learn more sufficient and compact information. The added decoder and the network part are relatively independent and can be removed in reasoning, so that the reasoning speed of the model is increased, and the model precision is not influenced.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and more specifically, relates to a method and system for identifying defects in side equipment during inspection by an electric power drone based on an autoencoder. Background Art

[0002] With the development of smart grids, power grids are increasingly becoming integrated systems with complex networks, numerous devices, and complex technologies. In order to ensure the reliable operation and effective management of power equipment and facilities, traditional operation and maintenance methods can no longer meet the requirements. The integration of artificial intelligence and power grid application technology has effectively solved this problem. Although the inspection method of collecting data and performing detection on the edge side through drones and neural networks has made certain progress in the current power scene, the detection results of small models on the edge side are still not accurate enough, and uncertain images still require manual review, which affects the practicality of power drone inspections.

[0003] In the existing technology, data often needs to be labeled, but due to the low probability of some faults and the small number of abnormal data, the distribution of data in each category in the collected data is extremely unbalanced, with some categories of samples appearing at a very high frequency and some categories of samples appearing at a very low frequency. As a result, after the existing target detection network is trained, the recognition accuracy of the category with a high frequency of samples is higher, and the recognition accuracy of the category with a low frequency of samples is lower.

[0004] In addition, embedded devices are very suitable for deployment on the edge due to their small size, low power consumption, high reliability, and low cost. However, edge devices have limited storage and computing resources, which limits the amount of computation and size of neural network models that can be run.

[0005] Existing defect identification methods are:

[0006] The patent with authorization announcement number CN111126359A discloses a method for detecting small targets in high-definition images based on an autoencoder and the YOLO algorithm. The implementation steps are: 1) collecting high-definition images and annotating them to obtain training sets and test sets; 2) performing data expansion on the annotated training sets; 3) generating corresponding Mask data based on the annotation information; 4) building an autoencoder model; 5) training it using the training set; 6) concatenating the encoding network of the trained autoencoder with the YOLO-V3 detection network to obtain a hybrid network and training it using the training set; 7) using the trained hybrid network to perform target detection on the test set.

[0007] However, the patent first trains an autoencoder network separately, then decouples the encoder and decoder of the autoencoder network, splices the encoder part with the Yolov3 detector part, fixes the encoder network part, and then trains the newly added Yolov3 detector part. The patent is not end-to-end, but phased, which is less efficient. In addition, the patent mainly focuses on high accuracy when processing small targets. Summary of the invention

[0008] In order to solve the deficiencies in the prior art, the present invention provides a method and system for identifying defects in side equipment during inspection by an electric power drone based on an autoencoder.

[0009] The present invention adopts the following technical solution.

[0010] The first aspect of the present invention proposes a method for identifying defects in edge equipment for power drone inspection based on an autoencoder, comprising the following steps:

[0011] S1, the edge device loads the historical inspection images of the power drone and preprocesses the historical inspection images, wherein the preprocessing includes adjusting the image size and format;

[0012] S2, constructing a defect detection model for electric power drone inspection, including a training stage model and an inference stage model. The training stage model uses the image data preprocessed by S1 as training samples for training, and removes the decoder module in the trained training stage model to obtain the inference stage model;

[0013] S3, the edge device loads the real-time inspection image of the power drone, which is input into the inference stage model after preprocessing, and the result is post-processed to identify equipment defects.

[0014] Preferably, in S1, the specific content of image preprocessing is: adjust the image size to the maximum input space size (H, W) that the edge device can receive during the inference process, normalize the image pixel value to the [0, 1] interval, transpose the image format, and convert it into a tensor of (B, C, H, W), where B represents the batch size, which is set to 1; C represents the number of channels in an image, and C takes a value of 3, representing R, G, and B respectively; H represents the number of pixels in the vertical dimension of the image; W represents the number of pixels in the horizontal dimension of the image.

[0015] Preferably, in S2, the specific content of the power drone inspection defect detection model is:

[0016] The training phase model consists of four parts: backbone network, feature fusion module, detector, and decoder. The backbone network extracts features from the preprocessed image to obtain a feature map. The feature fusion module fuses the feature map obtained by the backbone network to obtain a feature map after feature fusion. The detector is used for target prediction. The decoder decodes the features generated by the backbone network to restore the input image. In the training phase, the decoding module is trained as an additional network head to increase the information learned by the model. The model is continuously optimized during the training phase.

[0017] The inference stage model consists of three parts: backbone network, feature fusion, and detector. Each part is the same as the corresponding part in the training stage. The inference stage model outputs the defect category prediction results and the defect target location prediction results.

[0018] The output of the defect category prediction result is the probability that the value at each position of the feature map is each category; the feature map is equivalent to a four-dimensional matrix, and each value in the matrix is ​​the value at each position of the feature map;

[0019] The output of the target position prediction result is the offset of the target relative to the center point of each position in the feature map in the left, top, right, and bottom directions.

[0020] Preferably, the training phase continuously optimizes the model, and the specific process is as follows:

[0021] The loss function used in training is the weighted sum of several loss functions:

[0022] l total =αl cls +βl pos +δl dfl +γl mse

[0023] Among them, α, β, δ, and γ are weights, α=0.5, β=7.5, δ=1.5, γ=1; l cls is the cross entropy loss of the category prediction result; l pos and l dfl are the intersection-over-union loss and the average data fidelity loss of the target position prediction results respectively; l mse The mean squared error loss between the decoder generated image and the original image;

[0024] Calculate the cross entropy loss l for the category prediction results cls , the cross entropy loss calculation formula for each position on the category prediction feature map is as follows:

[0025]

[0026] Where n is the number of categories, yi is the true label, if the target category corresponding to the position is the i-th class, then yi = 1, otherwise yi = 0; pi is the predicted probability that the target appearing at the position is the i-th class; the cross entropy loss of all positions in all batches of the three feature maps is summed to obtain the final category prediction loss;

[0027] Calculate the intersection-over-union loss l for the target position prediction result pos and the average fidelity loss of the data l dfl , the mean squared error loss between the decoder-generated image and the original image is calculated using the following function:

[0028]

[0029] Where W is the image width, H is the image height, and B is the batch size; is the pixel value of the i-th row and j-th column of the c-th channel of the b-th image generated by the decoder; It is the pixel value of the i-th row and j-th column of the c-th channel of the b-th original image.

[0030] Preferably, the backbone network is specifically: the backbone network includes a convolution module, a slicing layer, a Bottleneck module, a connection layer, and a pooling layer. After multiple feature extractions, feature maps feat3 and feat4 are output; the size of feature map feat3 is The size of the feature map feat4 is The size of the feature map feat5 is

[0031] Preferably, the specific content of the feature fusion is:

[0032] Feat5 is upsampled, and then connected to feat4 in the channel dimension using a connection layer. Then, the obtained feature map is fused using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain feat. 41 ;

[0033] Then on feat 41 By upsampling, the connection layer and feat3 are connected in the channel dimension, and then the feature map is fused using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain feat 31 ;

[0034] For the feature map feat 31 Use the convolution module with a convolution kernel size of 1x1 and a step size of 1 and the convolution module with a convolution kernel size of 3x3 and a step size of 1 to perform feature fusion to obtain feat32 ;

[0035] About feat 31 Perform downsampling and combine the downsampling result and feat 41 Then, the feature map is connected using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain the feat 42 ;

[0036] The feature map feat5 is processed using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain feat 51 ;

[0037] About feat 42 Use downsampling to combine the downsampling result and fe 51 Then, the feature map is connected using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain the feat 52 .

[0038] Preferably, the process of upsampling the feature map is as follows:

[0039] Upsampling, the input feature map space size is w×h, and the expected output feature map space size is 2w×2h; for each position (i, j) in the output feature map, first map the coordinates, and determine the corresponding position (i', j') of (i, j) in the input feature map according to the upsampling ratio. The formula is:

[0040]

[0041] Map it to the nearest integer coordinate, calculated as:

[0042]

[0043] The corresponding position coordinates of the four nearest neighbor pixels in the input feature map are (i', j'), (i'+1, j'), (i', j'+1) and (i'+1, j'+1). According to the values ​​of these four pixels and their weights, the value of the target pixel is calculated as follows:

[0044] I out (i,j)=w1I in (i',j')+w2I in (i'+1,j')+w3I in (i',j'+1)+w4I in (i'+1,j'+1)

[0045] Among them, w1, w2, w3, w4 correspond to the weights of the four pixels (i', j), (i'+1, j'), (i', j'+1) and (i'+1, j'+1) respectively; I out (i, j) is the feature value at the position (i, j) of the output feature map, I in (i', j') is the feature value at position (i', j') in the input feature map. Repeat the above steps until all target positions in the output feature map are filled.

[0046] Preferably, the weights of the four pixels are specifically:

[0047] w1=(i'+1-i')(j'+1-j')

[0048] w2=(i'+1-i')(j'-j')

[0049] w3=(i”-i')(j'+1-j”)

[0050] w4=(i”-i')(j”-j')

[0051] Among them, w1, w2, w3, and w4 correspond to the weights of the four pixels (i', j), (i'+1, j'), (i', j'+1), and (i'+1, j'+1), respectively.

[0052] Preferably, the specific contents of the detector are: an anchor-free detection head is used in the detector, and the detector performs target position prediction and target category prediction respectively. The target position prediction predicts the distances lt and rb from the upper left corner and lower right corner of the target to the center point of each feature position on the feature map in the original image, and the target category prediction predicts the target category closest to each feature position on the feature map.

[0053] Preferably, the specific content of the decoder is: the decoder and the backbone network maintain opposite structures, including a convolution inverse module; a slicing layer, a connection layer, a deBottleneck module, and a transposed convolution module. The input of the decoder module is the feature Feat5 from the backbone network, and the output is a picture of size (H, W).

[0054] Preferably, in S3, the post-processing is to filter the detection results using a confidence threshold, where the threshold is 0.001; to classify the detection results and perform non-maximum suppression processing; and to map the detection results back to the original image according to the scaling and translation of the image.

[0055] The second aspect of the present invention proposes a power drone inspection side equipment defect recognition system based on an autoencoder, using the method described in the first aspect of the present invention, including the following modules:

[0056] Image preprocessing module, adjusts the size and format of the image;

[0057] The training module uses the defect detection model in the training phase to train the image data, and removes the decoder module in the trained training model to obtain the inference phase model;

[0058] The equipment missing parts identification module detects defects through the inference stage model obtained by the training module to detect the type and location of defects;

[0059] The judgment module performs post-processing and determines whether the data processing is completed after post-processing.

[0060] The beneficial effect of the present invention is that, compared with the prior art,

[0061] This patent proposes a method and system for identifying defects in side equipment for power drone inspection based on autoencoders. The method adopts a lightweight design to achieve fast reasoning on edge devices. The idea of ​​autoencoders is introduced into the model, and the original backbone network is regarded as an encoder, and a decoder is added. The training goal of the image autoencoder is to minimize the reconstruction error, that is, to make the output of the decoder as close to the input data as possible. By limiting the information flow between the encoder and the decoder, the model can learn a compact representation of the input data, which usually has a higher level of abstract features than the original data, but retains rich original data information so that the decoder can restore the original data based on this information. According to the backbone network structure, a corresponding decoder network structure is designed. The added decoder and part of the network are relatively independent and can be removed during reasoning, which speeds up the reasoning speed of the model without affecting the model accuracy. The use of anchor-free detection heads in the detector can more accurately predict the bounding box of the target because it is no longer restricted by the constraints of the preset anchor box. This helps the model to show higher accuracy when dealing with targets of various shapes and sizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a detection flow chart of the present invention;

[0063] Figure 2 is the model structure in the training phase;

[0064] Figure 3 The model structure for the reasoning stage;

[0065] Figure 4 As the main network structure diagram;

[0066] Figure 5 This is the decoder structure diagram. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only embodiments of a part of the present invention, rather than all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0068] like Figure 1 As shown, Embodiment 1 of the present invention provides a method for identifying defects of side equipment during inspection by an electric power drone based on an autoencoder, comprising the following steps:

[0069] S1, the edge device loads the historical inspection images of the power drone and preprocesses the historical inspection images, wherein the preprocessing includes adjusting the image size and format;

[0070] Preferably, in S1, the specific content of image preprocessing is: adjust the image size to the maximum input space size (H, W) that the edge device can receive during the inference process, normalize the image pixel value to the [0, 1] interval, transpose the image format, and convert it into a tensor of (B, C, H, W), where B represents the batch size, which is set to 1; C represents the number of channels in an image, and C takes a value of 3, representing R, G, and B respectively; H represents the number of pixels in the vertical dimension of the image; W represents the number of pixels in the horizontal dimension of the image.

[0071] S2, constructing a defect detection model for electric power drone inspection, including a training stage model and an inference stage model. The training stage model uses the image data preprocessed by S1 as training samples for training, and removes the decoder module in the trained training stage model to obtain the inference stage model;

[0072] Preferably, in S2, the specific content of the power drone inspection defect detection model is:

[0073] like Figure 2 As shown in the figure, the training phase model consists of four parts: backbone network, feature fusion module, detector, and decoder; the backbone network extracts features from the preprocessed image to obtain a feature map; the feature fusion module performs feature fusion on the feature map obtained by the backbone network to obtain a feature map after feature fusion; the detector is used for target prediction; the decoder decodes the features generated by the backbone network to restore the input image, and the decoding module is trained as an additional network head in the training phase to increase the information learned by the model; the model is continuously optimized during the training phase;

[0074] like Figure 3As shown, the inference stage model consists of three parts: backbone network, feature fusion, and detector. Each part is the same as the corresponding part in the training stage. The inference stage model outputs the defect category prediction results and the defect target location prediction results.

[0075] The output of the defect category prediction result is the probability that the value of each position in the feature map is each category; the feature map is equivalent to a four-dimensional matrix, and each value in the matrix is ​​the value of each position in the feature map. Specifically, this embodiment has 21 categories.

[0076] The output of the target position prediction result is the offset of the target relative to the center point of each position in the feature map in the left, top, right, and bottom directions.

[0077] Preferably, the training phase continuously optimizes the model, and the specific process is as follows:

[0078] The loss function used in training is the weighted sum of several loss functions:

[0079] l total =αl cls +βl pos +δl dfl +γl mse

[0080] Among them, α, β, δ, and γ are weights, α=0.5, β=7.5, δ=1.5, γ=1; l cls is the cross entropy loss of the category prediction result; l pos and l dfl are the intersection-over-union loss and the average data fidelity loss of the target position prediction results respectively; l mse The mean squared error loss between the decoder generated image and the original image;

[0081] Calculate the cross entropy loss l for the category prediction results cls , the cross entropy loss calculation formula for each position on the category prediction feature map is as follows:

[0082]

[0083] Where n is the number of categories, yi is the true label, if the target category corresponding to the position is the i-th class, then yi = 1, otherwise yi = 0; pi is the predicted probability that the target appearing at the position is the i-th class; the cross entropy loss of all positions in all batches of the three feature maps is summed to obtain the final category prediction loss;

[0084] Calculate the intersection-over-union loss l for the target position prediction result pos and the average fidelity loss of the data l dfl, the mean squared error loss between the decoder-generated image and the original image is calculated using the following function:

[0085]

[0086] Where W is the image width, H is the image height, and B is the batch size; is the pixel value of the i-th row and j-th column of the c-th channel of the b-th image generated by the decoder; It is the pixel value of the i-th row and j-th column of the c-th channel of the b-th original image.

[0087] Preferably, the backbone network is specifically: the backbone network includes a convolution module, a slicing layer, a Bottleneck module, a connection layer, and a pooling layer. After multiple feature extractions, feature maps feat3 and feat4 are output; the size of feature map feat3 is The size of the feature map feat4 is The size of the feature map feat5 is

[0088] Specifically, the network adopts the backbone network of yolov8n, such as Figure 4 As shown, it includes convolution module, slicing layer, Bottleneck module, connection layer and pooling layer;

[0089] Specifically, it passes through two convolution modules @(3,2), one convolution module @(1,1), and one slicing layer in sequence. After the slicing layer is split, one part is output to the connection layer, and the other part is output to the Bottleneck module. The Bottleneck module is repeated n times and then output to the connection layer.

[0090] Then repeat the following steps three times, passing through a convolution module @(1,1), a convolution module @(3,2), a convolution module @(1,1), and a split layer in turn. After the split layer is split, one part is output to the connection layer and the other part is output to the Bottleneck module. The Bottleneck module is repeated n times and then output to the connection layer. The feature maps output by the first convolution module @(1,1) in the second and third steps are feat3 and feat4 respectively.

[0091] After that, it passes through two convolution modules @(1,1) in sequence and is output to the connection layer and three pooling layers respectively. The results are all output to the connection layer and then pass through another convolution module @(1,1) to output the feature map feat5.

[0092] The convolution module @(k,s) contains a convolution layer, a batch normalization layer, and a SiLU activation layer in sequence. The parameters (k,s) of the convolution module indicate that the convolution kernel size used by the convolution module is kxk and the step size is s. The slicing layer divides the feature map obtained in the previous layer of the network into two feature maps of the same size in the channel dimension. The Bottleneck module contains two convolution modules with a convolution kernel size of 3x3 and a step size of 1. The connection layer connects multiple feature maps passed in on the channel. The pooling layer uses a pooling window of 3 and a step size of 1.

[0093] Preferably, the specific content of the feature fusion is:

[0094] Feat5 is upsampled, and then connected to feat4 in the channel dimension using a connection layer. Then, the obtained feature map is fused using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain feat. 41 ;

[0095] Then on feat 41 By upsampling, the connection layer and feat3 are connected in the channel dimension, and then the feature map is fused using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain feat 31 ;

[0096] For the feature map feat 31 Use the convolution module with a convolution kernel size of 1x1 and a step size of 1 and the convolution module with a convolution kernel size of 3x3 and a step size of 1 to perform feature fusion to obtain feat 32 ;

[0097] About feat 31 Perform downsampling and combine the downsampling result and feat 41 Then, the feature map is connected using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain the feat 42 ;

[0098] The feature map feat5 is processed using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain feat 51 ;

[0099] About feat 42 Use downsampling to combine the downsampling result and fe 51Then, the feature map is connected and the convolution module with a convolution kernel size of 1x1 and a step size of 1 and the convolution module with a convolution kernel size of 3x3 and a step size of 1 are used to perform feature fusion to obtain feat. 52 .

[0100] Preferably, the process of upsampling the feature map is as follows:

[0101] Upsampling, the input feature map space size is w×h, and the expected output feature map space size is 2w×2h; for each position (i, j) in the output feature map, first map the coordinates, and determine the corresponding position (i', j') of (i, j) in the input feature map according to the upsampling ratio. The formula is:

[0102]

[0103] Map it to the nearest integer coordinate, calculated as:

[0104]

[0105] The corresponding position coordinates of the four nearest neighbor pixels in the input feature map are (i', j'), (i'+1, j'), (i', j'+1) and (i'+1, j'+1). According to the values ​​of these four pixels and their weights, the value of the target pixel is calculated as follows:

[0106] I out (i,j)=w1I in (i',j')+w2I in (i'+1,j')+w3I in (i',j'+1)+w4I in (i'+1,j'+1)

[0107] Among them, w1, w2, w3, w4 correspond to the weights of the four pixels (i', j'), (i'+1, j'), (i', j'+1) and (i'+1, j'+1) respectively; I out (i, j) is the feature value at the position (i, j) of the output feature map, I in (i', j') is the feature value of the position (i', j') in the input feature map. Repeat the above steps until all target positions in the output feature map are filled.

[0108] Preferably, the weights of the four pixels are specifically:

[0109] w1=(i'+1-i')(j'+1-j')

[0110] w2=(i'+1-i')(j'-j')

[0111] w3=(i”-i')(j'+1-j”)

[0112] w4=(i”-i')(j”-j')

[0113] Among them, w1, w2, w3, and w4 correspond to the weights of the four pixels (i', j'), (i'+1, j'), (i', j'+1), and (i'+1, j'+1), respectively.

[0114] The downsampling is performed using a maximum pooling layer with a kernel size of 3x3 and a step size of 2. The pooling process on the feature map is as follows:

[0115]

[0116] I out (i, j) is the feature value at the position (i, j) of the output feature map, I in (i', j') is the feature value of the input feature map (i', j'). The input and output are highly corresponding in these two dimensions, so the batch size dimension and channel dimension of the feature map are not considered here. When processing to the boundary of the image, fill the excess part with 0.

[0117] Preferably, the specific contents of the detector are: an anchor-free detection head is used in the detector, and the detector performs target position prediction and target category prediction respectively. The target position prediction predicts the distances lt and rb from the upper left corner and lower right corner of the target to the center point of each feature position on the feature map in the original image, and the target category prediction predicts the target category closest to each feature position on the feature map.

[0118] It should be noted that the anchor-free detection head uses the detection head of yolov8n. The anchor-free detection head can predict the bounding box of the target more accurately because it is no longer constrained by the preset anchor box. This helps the model to perform with higher accuracy when dealing with targets of various shapes and sizes. Since the setting and matching process of the anchor box is removed, the anchor-free detection head can reduce the amount of calculation during reasoning, thereby speeding up the reasoning speed. This is especially important for application scenarios with high real-time requirements. The anchor-free detection head does not rely on a specific anchor box setting, so it can better adapt to different data sets and detection tasks.

[0119] Preferably, the decoder module decoder and the backbone network maintain the opposite structure, including a convolution inverse module; a slicing layer, a connection layer, a deBottleneck module, and a transposed convolution module, whose input is the feature Feat5 from the backbone network and the output is a picture of size (H, W).

[0120] Specifically, Figure 5As shown, the specific content of the decoder is: the decoder and the backbone network basically maintain the opposite structure, and the network includes a convolution inverse module @(k, s), a slicing layer, a deBottleneck module, a connection layer, and a transposed convolution module.

[0121] The feature map feat5 passes through a convolutional inverse module @(1,1) in turn, and is output to the connection layer and three convolutional inverse modules @(3,1) respectively. The output results are all output to the connection layer and then pass through two convolutional inverse modules @(1,1);

[0122] Then repeat the following steps three times, passing through a split layer. After the split layer is split, one part is output to the connection layer, and the other part is output to the deBottleneck module. The deBottleneck module is repeated n times and then output to the connection layer, and then passes through a convolution inverse module @(1,1), a transposed convolution module, and a convolution inverse module @(1,1) in sequence.

[0123] After that, it passes through a slicing layer. After the slicing layer is split, one part is output to the connection layer, and the other part is output to the deBottleneck module. The deBottleneck module is repeated n times and then output to the connection layer. Then it passes through a convolution inverse module @(1,1) and two transposed convolution modules in sequence.

[0124] A Sigmoid activation layer is added at the end of the decoder to map the result to the (0,1) interval.

[0125] The slicing layer and the connection layer are the same as the backbone network. The convolutional inverse module contains an activation layer, a batch normalization layer, and a convolutional layer in sequence. The parameter (k, s) indicates that the convolutional kernel size of the convolutional layer is kxk and the step size is s. The deBottleneck module contains two convolutional inverse modules with a convolutional kernel size of 3x3 and a step size of 1; the transposed convolutional module contains an activation layer, a batch normalization layer, and a transposed convolutional layer in sequence. The convolutional kernel size of the convolutional layer is 3x3 and the step size is , and the transposed convolutional layer fills the excess part with 0;

[0126] The input of the decoder module is the feature Feat5 from the backbone network, and the output is an image of size (H, W).

[0127] It should be noted that the image autoencoder implements the compression and decompression process of image data through two main components: the encoder and the decoder. The encoder is responsible for compressing the input image data into a low-dimensional latent space representation, while the decoder is responsible for mapping these latent space representations back to the original input space, thereby reconstructing the original image. The training goal of the image autoencoder is to minimize the reconstruction error, that is, to make the output of the decoder as close to the input data as possible. By restricting the flow of information between the encoder and the decoder, the model is able to learn compact representations of the input data, which usually have higher-level abstract features than the original data, but retain rich original data information so that the decoder can restore the original data based on this information.

[0128] Transposed convolution is widely used in the field of deep learning to describe a special type of convolution operation, which aims to enlarge the spatial size of the input feature map while learning the upsampled filter weights. Transposed convolution can produce an output feature map that is larger than the input feature map by padding zeros around the input feature map and performing the convolution operation while the convolution kernel slides. This is actually upsampling by applying a filter to each element of the input feature map, increasing the spatial resolution.

[0129] S3, the edge device loads the real-time inspection image of the power drone, which is input into the inference stage model after preprocessing, and the result is post-processed to identify equipment defects.

[0130] Preferably, in S3, the post-processing is to filter the detection results using a confidence threshold, where the threshold is 0.001; to classify the detection results and perform non-maximum suppression processing; and to map the detection results back to the original image according to the scaling and translation of the image.

[0131] Embodiment 2 of the present invention proposes a power drone inspection side equipment defect recognition system based on an autoencoder, using the method described in Embodiment 1 of the present invention, including the following modules:

[0132] Image preprocessing module, adjusts the size and format of the image;

[0133] The training module uses the defect detection model in the training phase to train the image data, and removes the decoder module in the trained training model to obtain the inference phase model;

[0134] The equipment missing parts identification module detects defects through the inference stage model obtained by the training module to detect the type and location of defects;

[0135] The judgment module performs post-processing and determines whether the data processing is completed after post-processing.

[0136] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0137] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0138] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for identifying defects in edge equipment for power drone inspection based on autoencoder, characterized in that: The following steps are involved: S1, the edge device loads the historical inspection images of the power drone and preprocesses the historical inspection images, wherein the preprocessing includes adjusting the image size and format; S2, constructing a defect detection model for electric power drone inspection, including a training stage model and an inference stage model. The training stage model uses the image data preprocessed by S1 as training samples for training, and removes the decoder module in the trained training stage model to obtain the inference stage model; S3, the edge device loads the real-time inspection image of the power drone, which is input into the inference stage model after preprocessing, and the result is post-processed to identify equipment defects.

2. According to claim 1, a method for identifying defects in side equipment during inspection by an electric power drone based on an autoencoder is characterized in that: In S1, the specific contents of image preprocessing are: adjusting the image size to the maximum input space size (H, W) that the edge device can receive during the inference process, normalizing the image pixel value to the [0, 1] interval, transposing the image format, and converting it into a tensor of (B, C, H, W), where B represents the batch size, which is set to 1; C represents the number of channels in an image, and C takes a value of 3, representing R, G, and B respectively; H represents the number of pixels in the vertical dimension of the image; and W represents the number of pixels in the horizontal dimension of the image.

3. According to claim 2, a method for identifying defects in side equipment during inspection by an electric power drone based on an autoencoder is characterized in that: In S2, the specific contents of the power drone inspection defect detection model are: The training phase model consists of four parts: backbone network, feature fusion module, detector, and decoder. The backbone network extracts features from the preprocessed image to obtain a feature map. The feature fusion module performs feature fusion on the feature map obtained by the backbone network to obtain a feature map after feature fusion; the detector is used to perform target prediction; The decoder decodes the features generated by the backbone network and restores the input image. During the training phase, the decoding module is trained as an additional network head to increase the information learned by the model. The model is continuously optimized during the training phase; The inference stage model consists of three parts: backbone network, feature fusion, and detector. Each part is the same as the corresponding part in the training stage. The inference stage model outputs the defect category prediction results and the defect target location prediction results. The output of the defect category prediction result is the probability that the value at each position of the feature map is each category; the feature map is equivalent to a four-dimensional matrix, and each value in the matrix is ​​the value at each position of the feature map; The output of the target position prediction result is the offset of the target relative to the center point of each position in the feature map in the left, top, right, and bottom directions.

4. According to claim 3, a method for identifying defects in side equipment during inspection by an electric power drone based on an autoencoder is characterized in that: The training phase continuously optimizes the model. The specific process is as follows: The loss function used in training is the weighted sum of several loss functions: l total =αl cls +βl pos +δl dfl +γl mse Among them, α, β, δ, and γ are weights, α=0.5, β=7.5, δ=1.5, γ=1; l cls is the cross entropy loss of the category prediction result; l pos and l dfl are the intersection-over-union loss and the average data fidelity loss of the target position prediction results respectively; l mse The mean squared error loss between the decoder generated image and the original image; Calculate the cross entropy loss l for the category prediction results cls , the cross entropy loss calculation formula for each position on the category prediction feature map is as follows: Where n is the number of categories, yi is the true label, if the target category corresponding to the position is the i-th class, then yi = 1, otherwise yi = 0; pi is the predicted probability that the target appearing at the position is the i-th class; the cross entropy loss of all positions in all batches of the three feature maps is summed to obtain the final category prediction loss; Calculate the intersection-over-union loss l for the target position prediction result pos and the average fidelity loss of the data l dfl , the mean squared error loss between the decoder-generated image and the original image is calculated using the following function: Where W is the image width, H is the image height, and B is the batch size; is the pixel value of the i-th row and j-th column of the c-th channel of the b-th image generated by the decoder; It is the pixel value of the i-th row and j-th column of the c-th channel of the b-th original image.

5. According to claim 3, a method for identifying defects in side equipment during inspection by an electric power drone based on an autoencoder is characterized in that: The backbone network is specifically as follows: the backbone network includes a convolution module, a slicing layer, a Bottleneck module, a connection layer, and a pooling layer. After multiple feature extractions, feature maps feat3 and feat4 are output; the size of feature map feat3 is The size of the feature map feat4 is The size of the feature map feat5 is 6. According to claim 4, a method for identifying defects in side equipment during inspection by an electric power drone based on an autoencoder is characterized in that: The specific content of the feature fusion is: Feat5 is upsampled, and then connected to feat4 in the channel dimension using a connection layer. Then, the obtained feature map is fused using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain feat. 41 ; Then on feat 41 By upsampling, the connection layer and feat3 are connected in the channel dimension, and then the feature map is fused using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain feat 31 ; For the feature map feat 31 Use the convolution module with a convolution kernel size of 1x1 and a step size of 1 and the convolution module with a convolution kernel size of 3x3 and a step size of 1 to perform feature fusion to obtain feat 32 ; About feat 31 Perform downsampling and combine the downsampling result and feat 41 Then, the feature map is connected using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain the feat 42 ; The feature map feat5 is processed using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain feat 51 ; About feat 42 Use downsampling to combine the downsampling result and fe 51 Then, the feature map is connected using a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to obtain the feat 52 .

7. The method for identifying defects of side equipment during inspection by an electric power drone based on an autoencoder according to claim 6 is characterized in that: The process of upsampling the feature map is as follows: Upsampling, the input feature map space size is w×h, and the expected output feature map space size is 2w×2h; for each position (i, j) in the output feature map, first map the coordinates, and determine the corresponding position (i', j') of (i, j) in the input feature map according to the upsampling ratio. The formula is: Map it to the nearest integer coordinate, calculated as: The corresponding position coordinates of the four nearest neighbor pixels in the input feature map are (i', j'), (i'+1, j'), (i', j'+1) and (i'+1, j'+1). According to the values ​​of these four pixels and their weights, the value of the target pixel is calculated as follows: I out (i,j)=w1I in (i',j')+w2I in (i'+1,j')+w3I in (i',j'+1)+w4I in (i'+1,j'+1) Among them, w1, w2, w3, w4 correspond to the weights of the four pixels (i', j'), (i'+1, j'), (i', j'+1) and (i'+1, j'+1) respectively; I out (i, j) is the feature value at the position (i, j) of the output feature map, I in (i', j') is the feature value at position (i', j') in the input feature map. Repeat the above steps until all target positions in the output feature map are filled.

8. The method for identifying defects of side equipment during inspection by an electric power drone based on an autoencoder according to claim 7 is characterized in that: The weights of the four pixels are specifically: w1=(i'+1-i')(j'+1-j') w2=(i'+1-i')(j'-j') w3=(i”-i')(j'+1-j”) w4=(i”-i')(j”-j') Among them, w1, w2, w3, and w4 correspond to the weights of the four pixels (i', j'), (i'+1, j'), (i', j'+1), and (i'+1, j'+1), respectively.

9. The method for identifying defects of side equipment during inspection by an electric power drone based on an autoencoder according to claim 3 is characterized in that: The specific content of the detector is: an anchor-free detection head is used in the detector, and the detector performs target position prediction and target category prediction respectively. The target position prediction predicts the distances lt and rb from the upper left corner and lower right corner of the target to the center point of each feature position on the feature map in the original image, and the target category prediction predicts the target category closest to each feature position on the feature map.

10. The method for identifying defects of side equipment during inspection by an electric power drone based on an autoencoder according to claim 3 is characterized in that: The specific content of the decoder is: the decoder and the backbone network maintain the opposite structure, including the convolution inverse module; the slicing layer, the connection layer, the deBottleneck module, the transposed convolution module, the input of the decoder module is the feature Feat5 from the backbone network, and the output is a picture of size (H, W).

11. The method for identifying defects of side equipment during inspection by an electric power drone based on an autoencoder according to claim 1 is characterized in that: In S3, the post-processing is to filter the detection results using a confidence threshold, the threshold being 0.001; to classify the detection results and perform non-maximum suppression processing; and to map the detection results back to the original image according to the scaling and translation of the image.

12. A power drone inspection side equipment defect recognition system based on autoencoder, using the method described in any one of claims 1-11, characterized in that: Includes the following modules: Image preprocessing module, adjusts the size and format of the image; The training module uses the defect detection model in the training phase to train the image data, and removes the decoder module in the trained training model to obtain the inference phase model; The equipment missing parts identification module detects defects through the inference stage model obtained by the training module to detect the type and location of defects; The judgment module performs post-processing and determines whether the data processing is completed after post-processing.

Images