Unmanned aerial vehicle inspection defect image recognition method based on style generative adversarial network

Abstract

The application provides a UAV inspection defect image recognition method based on a style generative adversarial network, and belongs to the technical field of UAV image recognition, and comprises the following steps: S1, establishing a UAV defect image sample set; S2, constructing a style-based generative adversarial network model, and training the model by using the UAV defect image sample set established in step S1; after the training is completed, generating simulated defect images by using the generative adversarial network model, and establishing a simulated defect image sample set; S3, constructing a classification model, and training the classification model by using the UAV defect image sample set established in step S1 and the simulated defect image sample set established in step S2 until the classification model can accurately recognize defect types; the application can efficiently generate more simulated defect image samples with higher quality, thereby training and improving the classification model and improving the recognition rate of defects in UAV inspection images.

Description

Technical Field

[0001] This invention belongs to the field of UAV image recognition technology, specifically relating to a method for UAV inspection defect image recognition based on style generative adversarial networks. Background Technology

[0002] Currently, after using multi-rotor drones for detailed inspection and fault detection of power transmission lines, the massive amount of image data generated requires manual interpretation and screening to identify faults and defects. This demands familiarity with the transmission lines from the operators and significantly increases their workload. Therefore, employing deep learning methods for component detection and identification from inspection image data is of great significance for improving the efficiency of drone inspections.

[0003] In actual inspection processes, the frequency of various defects varies, leading to an imbalance in the collected defect image samples. Smaller class samples have far fewer opportunities to participate in training than larger class samples, causing the final trained model to favor detecting the larger class, resulting in significant differences in detection performance across different categories. Therefore, addressing the class imbalance problem in inspection image samples is crucial.

[0004] Chinese invention patent application CN111126446A discloses a method for augmenting defect images of industrial products using robot vision. It employs a generative adversarial network (GAN) to generate simulated images that approximate real images in order to solve the problem of augmenting small sample data. However, the GAN used is Mask-CDGAN. This method uses the image and its category together as input to train the GAN model and finally generate simulated images. The feature factors of the simulated images generated by this method are not decoupled, making it impossible to generate simulated images that are significantly different from the features of the original images while still meeting the requirements. Moreover, the training efficiency of the GAN model is low. Summary of the Invention

[0005] The purpose of this invention is to solve the problems mentioned in the background art and provide a method for identifying defect images in UAV inspections based on style generative adversarial networks. This method can efficiently generate more and higher quality simulated defect image samples, thereby training and improving the classification model and increasing the recognition rate of defects in UAV inspection images.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for defect image recognition in UAV inspection based on style generative adversarial networks includes the following steps:

[0008] S1. Establish a sample set of UAV defect images;

[0009] S2. Construct a style-based generative adversarial network model and train it using the UAV defect image sample set established in step S1; after training, use the generative adversarial network model to generate simulated defect images and establish a simulated defect image sample set.

[0010] S3. Construct a classification model and train it using the UAV defect image sample set established in step S1 and the simulated defect image sample set established in step S2 until it can accurately identify the defect type.

[0011] Preferably, step S1 includes the following sub-steps:

[0012] S11. Label the defective targets in the UAV inspection images. The labeling follows the Pascal VOC labeling rules, which include using rectangular label boxes, assigning corresponding labels to the label boxes according to the defect type, ensuring that the label boxes are close to the edge of the defective target and contain all visible pixels of the target, and that the distance between the label box and the edge of the target image is no more than 10 pixels.

[0013] S12. Adjust the annotation box to the same size as the annotation box; expand the short side of the annotation box to match the size of the long side, and then cut out the target image within the annotation box.

[0014] S13. Save the target image and classify it; save the cutout result to a uniform size and classify it according to the defect type of the standard box to form an initial UAV defect image sample set; the images in the UAV defect image sample set are the real image samples.

[0015] Preferably, in step S2, a style-based generative adversarial network model is constructed, which specifically includes a generative network and a discriminative network;

[0016] The generating network includes input noise, a mapping network, an affine layer, a Fourier feature operation layer, a convolutional layer, several network layers, and a ToRGB layer;

[0017] The input noise is random noise randomly obtained from noise that conforms to a Gaussian distribution, and is used to generate simulated image samples that are similar to real image samples.

[0018] The mapping network includes one normalization layer and eight fully connected layers, which are used to decouple the input random noise by features to obtain an intermediate vector w;

[0019] The affine layer is used to convert the intermediate vector w into global translation and rotation parameters and input them into the Fourier feature operation layer.

[0020] The Fourier feature operation layer is used to apply rotation and translation operations to the Fourier features of the global translation and rotation parameters;

[0021] The convolutional layer convolves the output of the Fourier feature operation layer, and then uses this output as the input of the first network layer.

[0022] The network layer comprises an upsampling layer, a first activation function layer, a downsampling layer, and a cropping layer. Each network layer uses the output of the previous layer and the intermediate vector w as input. The input is first upsampled by a factor of 2 by the upsampling layer to increase the resolution of the output. Then it passes through the first activation function layer, which uses the Leaky ReLU function to increase the non-linear expression of the output. After that, the downsampling layer downsamples the image by a factor of 2 to restore the resolution before upsampling. Finally, the cropping layer removes the boundary portions that have been expanded during the network layer process.

[0023] The ToRGB layer is used to generate a simulated image, including a fully connected layer, a custom convolutional layer, a second activation function layer, and a learnable parameter bias; the ToRGB layer takes the output of the last network layer and the intermediate vector w as input to generate the simulated image;

[0024] The discriminant network comprises, in sequence, a convolutional layer, a third activation function layer, six concatenated modules, a first dense fully connected layer, a fifth activation function layer, a second dense fully connected layer, and a sixth activation function layer. Each concatenated module includes a convolutional layer, a batch normalization layer, and a fourth activation function layer. The sixth activation function layer uses the Sigmoid function, while the third, fourth, and fifth activation function layers all use the Leaky ReLU function. The discriminant network is used to determine the probability that the input is real data. The convolutional layer uses convolutional kernels to extract two-dimensional features from the image data. The batch normalization layer normalizes the data output from the previous layer. The Leaky ReLU activation function adds a non-linear expression to the discriminant network. The fully connected layer aggregates and organizes all convolutional results, using the results as the input to the next layer. The Sigmoid activation function outputs a single value within the range [0,1] as the probability value of the judgment result.

[0025] Preferably, the generator network and the discriminator network each have their own loss functions;

[0026] The loss function of the generator network is:

[0027]

[0028] Where, N f This represents the number of images generated in the model; logD_a L(i) y represents the probability that the discrimination network classifies a real sample as real when inputting image i; i This represents the probability value output by the discrimination network when inputting image i, i.e.

[0029]

[0030] The loss function of the discriminant network is:

[0031]

[0032] in:

[0033]

[0034]

[0035] The loss is to classify images from the real image sample set (i.e., images from the UAV defect image sample set) as simulated images. The loss function used to classify a simulated image as a real image is N. r This indicates the number of real images in the model.

[0036] Preferably, the generative adversarial network model is trained by alternating between training the generator network and the discriminator network, with the following specific steps:

[0037] S21. Randomly set the initialization parameters for the generator network and the discriminator network;

[0038] S22. Sample m samples from the input noise as input to the generator network to obtain m simulated image samples; sample m samples from the UAV defect image sample set and use them together with the m simulated image samples as input to the discriminator network; the discriminator network judges the authenticity of the input samples, obtains the total error, and performs backpropagation to update the parameters of the discriminator network.

[0039] S23. After repeating step S22 several times, first fix the parameters of the discriminant network and train the generator network; randomly select a noise from the input noise as the input of the generator network, generate a simulated image from the generator network, input the simulated image into the discriminant network to obtain the probability that the simulated image is a real image, and backpropagate the error to the generator network to update the parameters of the generator network.

[0040] S24. Repeat steps S22 and S23 until the discrimination network can no longer determine whether the input image comes from the output of the generator network or from the UAV defect image sample set. Finally, the probability of the input image being judged as a real image is 50%, that is, random judgment.

[0041] As a preferred embodiment, the objective function for the alternating training steps of the generative adversarial network model is expressed as:

[0042]

[0043] Where, p data (x) represents the distribution of the true sample, p z(z) represents the distribution of random noise, D(x) is the discriminant network, and G(z) is the generator network; V(D,G) represents the difference between the generated simulated image sample and the defective image sample. This means first fixing the generator network G, and then training the discriminator network so that it can distinguish between real images and simulated images; This means that the generator network G is trained with the discriminant network D fixed, with the goal of minimizing the difference between the generated image samples and the real image samples, until the above equation converges.

[0044] Preferably, the constructed classification model adopts a ResNet18 network structure, which includes, in sequence, convolutional layers, batch normalization layers, max pooling layers, several residual units, average pooling layers, fully connected layers, and a seventh activation function layer; the seventh activation function layer uses the softmax function; the classification model is used to determine the defect type of the input sample; the convolutional layers use convolutional kernels to extract two-dimensional features of the image data; the batch normalization layer is used to normalize the data output from the previous layer; the max pooling layer is used to reduce the dimensionality of the information extracted by the convolutional layers; the residual units are used to further extract image feature information and add nonlinear expressions; the average pooling layer is used to transform the feature map of the final output of the residual units into a 1×1 size; the fully connected layer is used to collect and organize the results of all average pooling layers and use the results as the input of the next layer; the softmax function is used to give the probability value corresponding to each classification result.

[0045] Preferably, the residual unit includes a convolutional layer, an eighth activation function layer, another convolutional layer, and a ninth activation function layer arranged sequentially. Both the eighth and ninth activation function layers use the ReLU function to increase the nonlinear expression of the residual unit. The convolutional layer uses a convolutional kernel to extract two-dimensional features from the image data. The input of the residual unit is divided into two parts: one part is processed according to the order of the layers arranged sequentially, and the other part, together with the output before the ninth activation function layer, is used as the input of the ninth activation function layer.

[0046] Preferably, the classification model is trained on a training set before use, and the objective function of the training step can be expressed as follows:

[0047]

[0048] Where M represents the number of defect types, N c p represents the total number of defective image samples and simulated image samples that make up the training set. i,c l represents the probability that the classifier predicts defect sample i as class c. i,c This represents the label value corresponding to the defect sample i belonging to category c.

[0049] The beneficial effects of this invention are:

[0050] 1. The mapping network of the generative network maps random noise vectors to intermediate vectors w that are decoupled from features. Then, the intermediate vector w is used as the input of each network layer, enabling the generative network to generate simulated images based on the feature patterns of real images. The range of generated images is wider and the accuracy is guaranteed. It can provide more samples of small defects, thereby improving the training effect of the classification model and enabling the classification model to accurately identify defects in both small and large samples. Moreover, the simulated images generated by this method are of higher quality, which can further improve the recognition accuracy of the classification model.

[0051] 2. A separate classification model was constructed, making it independent of the generative adversarial network. This not only improves the training efficiency of the generative adversarial network but also avoids interference from image type labels, thereby improving the quality of images generated by the generative adversarial network. The improved quality of generated images provides the classification model with sufficient effective samples for training, and its classification performance is effectively improved. Moreover, training it separately from the generative adversarial network also improves the training efficiency of each model. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the structure of a generative adversarial network model;

[0053] Figure 2 This is a schematic diagram of the generative network model;

[0054] Figure 3 This is a schematic diagram of the network layer structure;

[0055] Figure 4 This is a schematic diagram of the discriminant network model;

[0056] Figure 5 This is a schematic diagram of the structure and training architecture of the classification model;

[0057] Figure 6 This is a schematic diagram of the residual unit structure. Detailed Implementation

[0058] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0059] It should be noted that the terms such as "upper", "lower", "left", "right", "front", and "back" used in the invention are only for clarity of description and are not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0060] like Figure 1-6As shown, this invention provides a method for identifying defect images in UAV inspections based on style-based generative adversarial networks. The method includes constructing a style-based generative adversarial network model and a classification model, and then training both. The generative adversarial network model is trained to generate simulated images similar to UAV inspection defect images, while the classification model is trained to identify the defect type of the input pattern.

[0061] The UAV inspection defect image recognition method based on style generative adversarial networks provided by this invention includes the following steps:

[0062] S1. Establish a sample set of UAV defect images, which includes the following steps:

[0063] S11. Label the defective targets in the UAV inspection images. The labeling follows the Pascal VOC labeling rules, which include using rectangular label boxes, assigning corresponding labels to the label boxes according to the defect type, ensuring that the label boxes are close to the edge of the defective target and contain all visible pixels of the target, and that the distance between the label box and the edge of the target image is no more than 10 pixels.

[0064] S12. Adjust the bounding boxes to the same size. Since the sizes of different types of defect targets are not consistent, the sizes of the bounding boxes also vary. If they are directly used as input to generative adversarial networks and classification models, it will affect the performance of deep learning models. Therefore, it is necessary to adjust the bounding boxes of each sample to the same size. Specifically, the method of expanding the side of the bounding box can be used to expand the short side of the defect sample bounding box to the same size as the long side, and then the image area within the bounding box is extracted.

[0065] S13. Save the target image and classify it; save the cutout result as a 128×128 image and classify it according to the defect type of the labeled box to form the original UAV defect image sample set.

[0066] S2. Construct a style-based generative adversarial network model, namely StyleGAN3;

[0067] Generative adversarial network (GAN) models consist of a generator network and a discriminator network.

[0068] like Figure 2 As shown, the generating network includes input noise, a mapping network, an affine layer, a Fourier feature operation layer, a convolutional layer, 13 network layers, and a ToRGB layer;

[0069] The input noise is random noise randomly obtained from noise that conforms to a Gaussian distribution, used to generate a simulated image similar to the real image (i.e., the image in the UAV defect image sample set);

[0070] The mapping network includes one normalization layer and eight fully connected layers. The mapping network transforms the random noise vector Z into an intermediate vector w, which can be used to decouple features and generate a vector w that does not need to follow the distribution of training data, thereby reducing feature entanglement.

[0071] The affine layer is used to convert the intermediate vector w into global translation and rotation parameters and input them into the Fourier feature operation layer.

[0072] The Fourier feature operation layer is used to apply rotation and translation operations to the Fourier features of the global translation and rotation parameters;

[0073] The convolutional layer (i.e., the Conv layer in the figure) convolves the output of the Fourier feature operation layer, and then uses this output as the input of network layer 1.

[0074] like Figure 3 As shown, the network layer includes an upsampling layer, a first activation function layer, a downsampling layer, and a cropping layer. Each network layer uses the output of the previous layer and the intermediate vector w as input. The input is first upsampled by a factor of 2 by the upsampling layer to increase the resolution of the output (i.e., the output discrete image). Then it passes through the first activation function layer, which uses the Leaky ReLU function to increase the non-linear expression of the output. After that, the downsampling layer downsamples the image by a factor of 2 to restore the resolution before upsampling. Finally, the cropping layer cuts off the boundary parts that have been expanded during the network layer process.

[0075] The ToRGB layer is used to generate a simulated image and includes a fully connected layer, a custom convolutional layer, a second activation function layer, and a learnable parameter bias. The ToRGB layer takes the output of the last network layer (i.e., network layer 13) and the intermediate vector w as input to generate the simulated image.

[0076] like Figure 4 As shown, the discriminant network includes a convolutional layer, a third activation function layer, six concatenated modules, a first dense fully connected layer, a fifth activation function layer, a second dense fully connected layer, and a sixth activation function layer arranged sequentially. The concatenated modules include a convolutional layer, a batch normalization layer, and a fourth activation function layer. The sixth activation function layer uses the Sigmoid function, and the third, fourth, and fifth activation function layers all use the Leaky ReLU function.

[0077] The input sample first passes through a convolutional layer, where the convolutional kernel extracts the two-dimensional features of the image data. Then, it passes through a non-linear activation function layer, where the Leaky ReLU function enables the network to have a non-linear expression.

[0078] Then, it goes through 6 sequential modules, each of which includes a convolutional layer, a batch normalization layer, and a fourth activation function layer. The convolutional layer still extracts two-dimensional features of the image data through convolutional kernels. The batch normalization layer (BN layer) generates a more stable distribution, which plays a role in normalizing the data output by the previous layer, which helps to prevent overfitting and speed up the training. The fourth activation function layer adds non-linear expression to the input.

[0079] Afterwards, the data obtained from all the cascaded modules are aggregated and processed by the first fully connected layer and used as the input to the next layer (i.e., the fifth activation function layer); after the fifth activation function layer adds nonlinear expression, all the results are aggregated and processed by the second fully connected layer and used as the input to the sixth activation function layer.

[0080] Finally, the sixth activation function layer uses the Sigmoid activation function to output a single value in the interval [0,1] as the probability value of the judgment result.

[0081] The simulated image samples generated by the generative network, together with the real image samples (i.e. defective image samples), serve as the input to the discriminative network. The discriminative network must distinguish between the real and fake samples, and try to differentiate between the two types of samples. The output of the discriminative network is the probability that the input belongs to the real data.

[0082] S3. Alternately train the generative adversarial network model; clearly define the loss functions for the generative and discriminative networks before training.

[0083] The loss function of the generator network is:

[0084]

[0085] Where, N f D_a represents the number of images generated in the model. L(i) For custom values, D indicates association with the discriminant network, D_a L(i) This represents the result of the discriminant network classifying a real sample as true when inputting image i, while logD_a L(i) This represents the probability that the discrimination network can produce this result (classify the real sample as real) when given input image i; y i This represents the probability value output by the discrimination network when inputting image i, i.e.

[0086]

[0087] The loss function of the discriminant network is:

[0088]

[0089] in:

[0090]

[0091]

[0092] The loss is to classify images from the real image sample set (i.e., images from the UAV defect image sample set) as simulated images. The loss function used to classify a simulated image as a real image is N. r This indicates the number of real images in the model;

[0093] After defining the loss function, the generative adversarial network model is trained alternately, specifically including the following steps:

[0094] S31. Randomly set the initialization parameters for the generator network and the discriminator network;

[0095] S32. Sample m samples from the input noise as input to the generator network to obtain m simulated image samples; sample m samples from the UAV defect image sample set and use them together with the m simulated image samples as input to the discriminator network; the discriminator network judges the authenticity of the input samples, obtains the total error, and performs backpropagation to update the parameters of the discriminator network.

[0096] S33. After repeating step S32 several times, first fix the parameters of the discriminant network and train the generator network; randomly select a noise from the input noise as the input of the generator network, generate a simulated image from the generator network, input the simulated image into the discriminant network to obtain the probability that the simulated image is a real image, and backpropagate the error to the generator network to update the parameters of the generator network.

[0097] S34. Repeat steps S32 and S33 until the discrimination network can no longer determine whether the input image comes from the output of the generator network or from the UAV defect image sample set. Finally, the probability of the input image being judged as a real image is 50%, that is, random judgment.

[0098] The objective function for the entire alternating training process described above can be expressed as:

[0099]

[0100] Where, p data (x) represents the distribution of the true sample, p z (z) represents the distribution of random noise, D(x) is the discriminant network, and G(z) is the generator network; V(D,G) represents the difference between the generated simulated image sample and the defective image sample. This means first fixing the generator network G, and then training the discriminator network so that it can distinguish between real images and simulated images; This means that the generator network G is trained with the discriminant network D fixed, with the goal of minimizing the difference between the generated image samples and the real image samples, until the above equation converges.

[0101] During the alternating training process, the batch size (i.e., the number of samples) selected for each training session was set to 32; the learning rate was set to 0.0025 when training the generator network and 0.002 when training the discriminator network; in addition, the total training epochs (i.e., the iteration period) were set to 25000, and the Adam optimizer (i.e., the adaptive moment estimator) was used to optimize the parameters of the generator network and the discriminator network.

[0102] S4. Construct a classification model;

[0103] like Figure 5 As shown, the classification model adopts a ResNet18 network structure (the part within the dashed box in the figure), which includes a convolutional layer, a batch normalization layer, a max pooling layer, 8 residual units, an average pooling layer, a fully connected layer, and a seventh activation function layer arranged sequentially; the seventh activation function layer adopts the softmax function; the classification model is used to determine the defect type of the input sample;

[0104] The classification model uses the ResNet18 network structure. This network structure first extracts features from the input samples through convolutional layers, then normalizes them through batch normalization layers to speed up training; and then performs further dimensionality reduction on the information extracted by the convolutional layers through max pooling layers to reduce the amount of computation.

[0105] Then the feature input is passed through eight consecutive residual units in series. The structure of the residual unit is as follows: Figure 6 As shown, it includes a convolutional layer, an eighth activation function layer, another convolutional layer, and a ninth activation function layer arranged sequentially. Both the eighth and ninth activation function layers use the ReLU function to increase the non-linear expression of the residual units. The convolutional layer uses convolutional kernels to extract two-dimensional features of the image data. The input of the residual units is divided into two parts: one part is processed according to the order of the residual units in each layer, and the other part, together with the output before the ninth activation function layer, is used as the input of the ninth activation function layer to alleviate the problem of information loss during transmission.

[0106] The output features from the residual unit then pass through an average pooling layer (AvgPool) to transform the feature map into a 1×1 size. The results from all the average pooling layers are then collected and processed by a fully connected layer and used as the input for the next layer (i.e., the seventh activation function layer).

[0107] Finally, the softmax function of the seventh activation layer gives the probability value corresponding to each classification result.

[0108] S5. Train the classification model; this includes the following steps:

[0109] S51. For the simulated images generated by the generative adversarial network that have been trained and meet the requirements, establish a simulated image sample set according to the same standard as establishing a UAV defect image sample set (refer to steps S11-S13).

[0110] S52. Combine the UAV defect image sample set and the simulated image sample set into a training set, and use it as the input to the classification model;

[0111] S53. Continuously input the training set into the classification model, calculate the error based on its output classification results, and backpropagate to update the parameters of the classification model until the classification model can correctly determine the type of defect.

[0112] like Figure 5 As shown, the final output of the classification model is the objective function of the classification model training steps, and the calculation formula is as follows:

[0113]

[0114] Where M represents the number of defect types, N c p represents the total number of defective image samples and simulated image samples that make up the training set. i,c l represents the probability that the classifier predicts defect sample i as class c. i,c This represents the label value corresponding to the defect sample i belonging to category c.

[0115] After the above steps, a simulated image sample set that cannot be distinguished from real images (i.e., images in the UAV defect image sample set) can be generated by a style-based generative adversarial network. This simulated image sample set, together with the UAV defect image sample set, is used as the training set for the classification model, greatly expanding the small sample set. Through repeated training on the training set, the classification model can meet the preset requirements, thereby accurately distinguishing the defects and their types in the UAV inspection images (including effectively identifying defects in small samples).

[0116] To verify that the simulated samples can improve the recognition rate of deep learning object detection, a training set of 2286 images was selected, covering four categories: ground wires, ancillary facilities, towers, and passageways. 808 samples were generated, representing 8 defect types. These were combined with the original 2000 samples to form a training set of 3808 images, divided into training and validation sets in a 3:7 ratio. In the experiment, the total number of training epochs was set to 200, and the batch size was set to 128. The initial learning rate was set to 0.001, decaying by 10 at epochs 100 and 150. Furthermore, a warm-up strategy was used to adjust the learning rate. The SGD optimizer was used to optimize the classifier parameters, with a momentum set to 0.9.

[0117] The FID score is used to reflect the difference between the real data distribution and the generated data distribution. The more realistic the generated image, the smaller the difference, indicating a stronger performance of the GAN model. The calculation formula is shown below:

[0118]

[0119] Where d represents the FID distance, m r C r Let m be the feature mean and covariance matrix of the real image. g C g To generate the feature mean and covariance matrix of the image, this invention uses the same training and test sets for both the style generative adversarial network and the traditional GAN ​​network. The test results are shown in the table below:

[0120]

[0121] mAP is used as the algorithm metric.

[0122]

[0123]

[0124] Experimental results show that, using the same Faster R-CNN algorithm, the Pseudo-Genius Network (PGN) outperforms the other two training scenarios in all three scenarios: using only the original sample set, adding samples generated by the original GAN, and adding samples generated by the Pseudo-Genius Network. This demonstrates that the simulated sample images generated by the Pseudo-Genius Network used in this invention can effectively expand the missing transmission line defect sample set and improve the recognition rate of deep learning object detection. In contrast, the test results of the original GAN ​​network are not significantly different from those using only the original sample set, indicating that the samples generated by the original GAN ​​network have low usability and cannot optimize the sample structure.

[0125] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for defect image recognition in UAV inspection based on style generative adversarial networks, characterized in that, Includes the following steps: S1. Establish a sample set of UAV defect images; S2. Construct a style-based generative adversarial network model and train it using the UAV defect image sample set established in step S1. After training, use the generative adversarial network model to generate simulated defect images and establish a simulated defect image sample set. The style-based generative adversarial network model constructed in step S2 specifically includes a generator network and a discriminator network. The generating network includes input noise, a mapping network, an affine layer, a Fourier feature operation layer, a convolutional layer, several network layers, and a ToRGB layer; The input noise is random noise randomly obtained from noise that conforms to a Gaussian distribution, and is used to generate simulated image samples that are similar to real image samples. The mapping network includes one normalization layer and eight fully connected layers, which are used to decouple the input random noise by features to obtain an intermediate vector w; The affine layer is used to convert the intermediate vector w into global translation and rotation parameters and input them into the Fourier feature operation layer. The Fourier feature operation layer is used to apply rotation and translation operations to the Fourier features of the global translation and rotation parameters; The convolutional layer convolves the output of the Fourier feature operation layer, and then uses this output as the input of the first network layer. The network layer comprises an upsampling layer, a first activation function layer, a downsampling layer, and a cropping layer. Each network layer uses the output of the previous layer and the intermediate vector w as input. The input is first upsampled by a factor of 2 by the upsampling layer to increase the resolution of the output. Then it passes through the first activation function layer, which uses the Leaky ReLU function to increase the non-linear expression of the output. After that, the downsampling layer downsamples the image by a factor of 2 to restore the resolution before upsampling. Finally, the cropping layer removes the boundary portions that have been expanded during the network layer process. The ToRGB layer is used to generate a simulated image, including a fully connected layer, a custom convolutional layer, a second activation function layer, and a learnable parameter bias; the ToRGB layer takes the output of the last network layer and the intermediate vector w as input to generate the simulated image; The discriminant network comprises, in sequence, a convolutional layer, a third activation function layer, six concatenated modules, a first dense fully connected layer, a fifth activation function layer, a second dense fully connected layer, and a sixth activation function layer. Each concatenated module includes a convolutional layer, a batch normalization layer, and a fourth activation function layer. The sixth activation function layer uses the Sigmoid function, and the third, fourth, and fifth activation function layers all use the Leaky ReLU function. The discriminant network is used to determine the probability that the input is real data. The convolutional layer uses convolutional kernels to extract two-dimensional features from the image data. The batch normalization layer is used to normalize the data output from the previous layer. The non-linear activation function Leaky ReLU is used to add a non-linear expression to the judgment network; the fully connected layer is used to collect and organize all convolution results and use the results as the input of the next layer; the sigmoid activation function is used to output a single value in the interval [0,1] as the probability value of the judgment result; S3. Construct a classification model and train it using the UAV defect image sample set established in step S1 and the simulated defect image sample set established in step S2 until it can accurately identify the defect type.

2. The method for defect image recognition in UAV inspection based on style generative adversarial networks according to claim 1, characterized in that: Step S1 includes the following sub-steps: S11. Label the defective targets in the UAV inspection images; the labeling follows the Pascal VOC labeling rules, which include using rectangular label boxes, assigning corresponding labels to the label boxes according to the defect type, ensuring that the label boxes are close to the edge of the defective target and contain all visible pixels of the target, and ensuring that the distance between the label box and the edge of the target image is no more than 10 pixels. S12. Adjust the annotation box to the same size as the annotation box; expand the short side of the annotation box to match the size of the long side, and then cut out the target image within the annotation box. S13. Save the target image and classify it; save the cutout result to a uniform size and classify it according to the defect type of the standard box to form an initial UAV defect image sample set; The images in the drone defect image sample set are the real image samples.

3. The method for defect image recognition in UAV inspection based on style generative adversarial networks according to claim 1, characterized in that: The generator network and the discriminator network each have their own loss functions; The loss function of the generator network is: in, This indicates the number of images generated in the model; This represents the probability that the discrimination network classifies a real sample as real when inputting image i. Indicates input image At that time, the probability value output by the discrimination network is determined, i.e. The loss function of the discriminant network is: in: The loss is to classify images from the real image sample set (i.e., images from the UAV defect image sample set) as simulated images. The loss that distinguishes a simulated image from a real image. This indicates the number of real images in the model.

4. The method for defect image recognition in UAV inspection based on style generative adversarial networks according to claim 3, characterized in that: The training of the generative adversarial network model employs alternating training of the generator network and the discriminator network, with the specific steps as follows: S21. Randomly set the initialization parameters for the generator network and the discriminator network; S22. Sample m samples from the input noise as input to the generator network to obtain m simulated image samples; sample m samples from the UAV defect image sample set and use them together with the m simulated image samples as input to the discriminator network; the discriminator network judges the authenticity of the input samples, obtains the total error, and performs backpropagation to update the parameters of the discriminator network. S23. After repeating step S22 several times, first fix the parameters of the discriminant network and train the generator network; randomly select a noise from the input noise as the input of the generator network, generate a simulated image from the generator network, input the simulated image into the discriminant network to obtain the probability that the simulated image is a real image, and backpropagate the error to the generator network to update the parameters of the generator network. S24. Repeat steps S22 and S23 until the discrimination network can no longer determine whether the input image comes from the output of the generator network or from the UAV defect image sample set. Finally, the probability of the input image being judged as a real image is 50%, that is, random judgment.

5. The method for defect image recognition in UAV inspection based on style generative adversarial networks according to claim 4, characterized in that: The objective function for the alternating training steps of a generative adversarial network model is expressed as: in, Represents the distribution of the real samples. Represents the distribution of random noise. To identify the network, To generate networks; This represents the difference between the generated simulated image sample and the defective image sample; This indicates that the generator network is fixed first. The discriminative network is trained to distinguish between real and simulated images; This indicates that in the discriminant network Training a generative network under fixed conditions The goal is to minimize the difference between generated image samples and real image samples until the above equation converges.

6. The method for defect image recognition in UAV inspection based on style generative adversarial networks according to claim 5, characterized in that: The constructed classification model adopts the ResNet18 network structure, which includes, in sequence, convolutional layers, batch normalization layers, max pooling layers, several residual units, average pooling layers, fully connected layers, and a seventh activation function layer; the seventh activation function layer uses the softmax function. The classification model is used to determine the defect type of the input sample. The convolutional layers use convolutional kernels to extract two-dimensional features of the image data; the batch normalization layer is used to normalize the data output from the previous layer; the max pooling layer is used to reduce the dimensionality of the information extracted by the convolutional layers; the residual units are used to further extract image feature information and add non-linear expressions; the average pooling layer is used to transform the final output feature map of the residual units into a 1×1 size; the fully connected layer is used to collect and organize the results of all average pooling layers and use the results as the input of the next layer. The softmax function is used to give the probability value corresponding to each classification result.

7. The method for defect image recognition in UAV inspection based on style generative adversarial networks according to claim 6, characterized in that: The residual unit includes a convolutional layer, an eighth activation function layer, another convolutional layer, and a ninth activation function layer arranged sequentially. The eighth and ninth activation function layers both use the ReLU function to increase the nonlinear expression of the residual unit. The convolutional layer uses convolutional kernels to extract two-dimensional features from the image data. The input of the residual unit is divided into two parts. One part is processed according to the order of each layer set by the residual unit, and the other part, together with the output before the ninth activation function layer, is used as the input of the ninth activation function layer.

8. The method for defect image recognition in UAV inspection based on style generative adversarial networks according to claim 7, characterized in that: The classification model must be trained on a training set before use. The objective function of the training step can be expressed as follows: in, Indicates the number of defect types. This represents the total number of defective image samples and simulated image samples that make up the training set; This indicates that the classifier will classify defect samples. Predicted as category The probability, Indicates defective samples Category The corresponding tag value at that time.

Images