UAV inspection image nail level defect identification method

Abstract

The application discloses a pin-level defect identification method for unmanned aerial vehicle inspection images, and comprises the following steps: S1, a multi-target detection model based on an R-FCN network is built, and a ResNet-FPN module is used as a feature extractor; S2, an inspection image is acquired and labeled, and a grid semi-overlapping splitting method is used to pre-process the inspection image to obtain a primary selected sub-image; S3, the primary selected sub-image is selected and labeled to obtain a target sub-image, and the target sub-image is used to train a multi-target detection model and a heat map generation model; S4, a heat map of a fitting is generated by using a class activation mapping method; S5, a high-definition original image and the heat map are superimposed to serve as input of the multi-target detection model, if the identification accuracy of each category is greater than a preset accuracy, it is determined that the multi-target detection model is reliable, otherwise, the data set is expanded, and S3 is re-executed; and S6, the multi-target detection model is used to identify the inspection image to obtain the category and position of a pin-level defect image. The identification accuracy of pin defects in the inspection image is significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method for identifying pin-level defects in drone inspection images. Background Art

[0002] Transmission lines are a vital component of the power grid and the arteries of the power system. Transmission line inspections are crucial to the safe and stable operation of the grid. State Grid Corporation of China invests significant manpower and resources annually to ensure power line inspections. Due to my country's vast territory and the scale of its power grid, transmission lines are long, dense, and complex. Manual inspections are subject to harsh inspection environments, heavy workloads, and the complexity of recording large amounts of data that is difficult to store, resulting in time-consuming and labor-intensive tasks. Therefore, inspections are currently primarily conducted using drones or helicopters, capturing transmission line images and transmitting them back to the backend for manual analysis to identify defects and faults and ensure their proper operation.

[0003] Pins in power transmission lines prevent the misalignment of flexible connectors. Pin fallout or dislodgement can affect the normal operation of other components on the line. Drone inspection images are large and high-resolution, and pins occupy a very small proportion of the image. Compressing the original image to the input size limit of R-FCN will lose the pin's detailed features, affecting classification. Furthermore, transmission line hardware combinations are diverse and the shapes of hardware strings are irregular, making hardware inspection a significant workload, far exceeding that of bolt inspection itself. Pin-level defect recognition targets hardware bolts with missing cotter pins. However, the variety of bolt types and subtle morphological differences in transmission line inspection images can easily lead to confusion. While drone inspection image acquisition and AI-based image analysis can significantly improve inspection efficiency, the accuracy of pin defect recognition in these images is limited. These factors hinder the development of automated and intelligent power inspection. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of low pin-level defect recognition accuracy in drone inspection images. A pin-level defect recognition method for drone inspection images is proposed. The image is gridded, labeled, and restored, and a multi-target recognition model is trained. Through a two-step target detection of hardware level and pin level, the recognition accuracy of pin defects is significantly improved.

[0005] To achieve the above technical objectives, the present invention provides a technical solution, which is a method for identifying pin-level defects in drone inspection images, comprising the following steps:

[0006] S1. Build a multi-target detection model based on the R-FCN network, using the ResNet-FPN module as the feature extractor;

[0007] S2. Obtain and annotate inspection images, and preprocess the inspection images using a grid-based semi-overlapping splitting method to obtain preliminary sub-images;

[0008] S3. Filter and label the preselected subgraphs to obtain target subgraphs, and use the target subgraphs to train the multi-target detection model and the heat map generation model.

[0009] S4. Generate heatmaps of hardware using the VGG16-CAM network through class activation mapping.

[0010] S5. Superimpose the high-definition original image and the heat map as the input of the multi-target detection model. If the recognition accuracy of each category is greater than the preset accuracy, the multi-target detection model is determined to be reliable. Otherwise, expand the data set and re-execute S3.

[0011] S6. The inspection images collected by the UAV in real time are identified through the trained multi-target detection model to obtain the category and location of the image representing the pin-level defect.

[0012] Preferably, a large-margin loss function with balanced categories is constructed as the cross-entropy loss function in the R-FCN network.

[0013] As a preference, the expression of the cross entropy loss function is:

[0014]

[0015] in:

[0016] y i is the true category of the target subgraph; c∈[0,C], C is the number of categories, n c Indicates the number of fully connected layer units whose true category is c; f y represents the output vector of the fully connected layer, represents the modulus of the fully connected layer parameter vector, ||x i || represents the modulus of the eigenvector, θ yi Represents a vector With vector x i Angle.

[0017] Preferably, S2 includes:

[0018] Split the high-definition original image into a series of preselected sub-images of different sizes according to the grid.

[0019] Among them, two adjacent preliminary selected sub-images have an overlapping area, and the width of the overlapping area is greater than the pixels of the bolt in the high-definition original image.

[0020] Preferably, in S3, deleting the pre-selected subgraphs includes:

[0021] The preliminary selected sub-images are divided into negative sample sub-images including only the background and positive sample sub-images containing pins; the negative sample sub-images are eliminated and the positive sample sub-images are used as target sub-images.

[0022] As a preference, the positive sample sub-images are further labeled with categories, including:

[0023] Category A: Hardware bolts with visible cotter pins;

[0024] Category B: Hardware bolts, cotter pins missing;

[0025] Category C: Hardware bolts, the split pin is not fully visible;

[0026] Category D: Tower bolts, no need for cotter pins.

[0027] Preferably, S4 includes:

[0028] Further extract the background information of the positive sample sub-image; generate a heat map of the pins in the image,

[0029] Use the VGG16 module to extract the feature information of the image, and the output of its last convolutional layer is 7×7×512;

[0030] Use the global average pooling layer to replace the original fully connected layer, and for each 7×7 feature map f k (x,y) average operation F k =∑ x,y f k (x,y); the final output category probability P c The expression is:

[0031]

[0032] Where k represents the feature map ordinal number, (x, y) represents the horizontal and vertical coordinates of the pixel in the feature map, c = {0, 1} represents the output category, 0 represents background, and 1 represents hardware string; Reflects the feature map f k The importance of (x,y) for category c;

[0033] The class activation map of category c is defined as:

[0034]

[0035] Based on the category information of all target sub-images, the VGG16-CAM network of the hardware string is trained. The 7×7 hardware class activation map is upsampled to the size of the input image to obtain the heat map M′(x,y).

[0036] Preferably, in S5, the high-definition original image and the heat map are superimposed as the input of the multi-target detection model, including: restoring the target sub-image after labeling to obtain the high-definition original image, superimposing the heat map M′(x, y) with the high-definition original image, and using the fourth channel as the input image of the multi-target detection model except for the three RGB channels.

[0037] The beneficial effects of the present invention are as follows: the ResNet module is used as a feature extractor, and the R-FCN network is used to realize target detection, so that the model has better recognition accuracy and faster training speed; the FPN (feature pyramid) module is added to the ResNet module, so that the model can make the neural network focus on targets of different sizes through feature fusion, solving the problem of inconsistent recognition of bolt models; the strong supervision training of the neural network is completed by refining the annotation scheme of the pin shape, enhancing the variance between samples, and solving the problem of too small difference in the shape of positive / negative samples of missing cotter pins. In view of the problem that the resolution of drone inspection images is relatively large, while the pin-level defects account for a very small proportion in the original image, directly using the original image for training is computationally huge and difficult to train, and the small target information will be lost after the original image is down-sampled. The original image is converted into high-definition sub-images through a splitting / reconstruction process. During the splitting process, overlapping portions no smaller than the bolt size are retained, background sub-images are filtered, and the remaining sub-images are identified and located within the original image. Finally, NMS (non-maximum suppression) is used to obtain the original image recognition result, solving the problems of high-resolution original images that make training difficult and information loss after downsampling the original image. Tower bolts and fitting bolts have similar shapes and appear in large numbers in drone inspection images. Tower bolts do not require cotter pins, but neural networks easily confuse them with fitting bolts without cotter pins. A dual-step target detection process, hardware-level and pin-level, is used to limit the detection area to the vicinity of line fittings. This not only eliminates interference from tower bolts that do not require cotter pins on the detection results, but also scales the image based on the known fitting dimensions so that all bolts appear similar in size, resolving the issue of target size differences caused by inconsistent shooting distances. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the pin-level defect recognition method for drone inspection images of the present invention. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific implementation method described herein is only an optimal embodiment of the present invention, which is only used to explain the present invention and does not limit the scope of protection of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] Example: Figure 1 As shown, the pin-level defect recognition method of the UAV inspection image is characterized by comprising the following steps:

[0041] S1. Build a multi-target detection model based on the R-FCN network, using the ResNet-FPN module as the feature extractor;

[0042] In S1, the R-FCN network is used for target detection, giving the model better recognition accuracy and faster training speed; the FPN (feature pyramid) module is added to the ResNet module, so that the model can focus on targets of different sizes through feature fusion, solving the problem of inconsistent recognition of bolt models.

[0043] The Softmax loss function in the R-FCN network is replaced by a class-balanced large margin loss function, which solves the problem of inter-class imbalance and enhances the model's ability to distinguish positive / negative samples with missing cotter pins.

[0044] The classification loss function of R-FCN is the cross entropy loss function, defined as

[0045]

[0046] f y Represents the output vector of the fully connected layer; j∈[0,C], C is the number of categories defined by the table, C=0 means the category is background; y i is the true category of the sample.

[0047] L i The formula is expressed as

[0048] in Define a monotonically decreasing continuous function

[0049]

[0050] Use ψ(θ yi ) replaces cos(θ in the original loss function yi ),because The neural network using this loss function has a larger inter-class interval and better ability to discriminate features. At the same time, the more difficult training prevents the model from overfitting.

[0051] When designing the large margin loss function, the difference in the number of samples of different categories is considered, and the expression is:

[0052]

[0053] y i is the true category of the target subgraph; c∈[0,C], C is the number of categories, nc Indicates the number of fully connected layer units whose true category is c; f y represents the output vector of the fully connected layer, represents the modulus of the fully connected layer parameter vector, ||x i || represents the modulus of the eigenvector, θ yi Represents a vector With vector x i Angle; coefficient 1 / n c The addition of balances the difference in the number of samples between foreground and background, and between different categories, solving the problem of inter-class imbalance.

[0054] S2. Obtain and annotate the inspection image, and preprocess the inspection image using a grid-based semi-overlapping splitting method to obtain a preliminary sub-image.

[0055] The high-definition original image is split into a series of preliminary sub-images of different sizes according to a grid; wherein, two adjacent preliminary sub-images have an overlapping area, and the width of the overlapping area is greater than the pixels of the bolt in the high-definition original image.

[0056] In this embodiment, a gridding method is employed to split the high-definition original image into a series of high-definition sub-images of varying sizes. After target detection, these sub-images are repositioned back to the original image, enabling small target detection in high-resolution images. Unlike methods that directly crop the original image, this approach, based on prior knowledge of the bolt size, leaves a certain overlap between any two adjacent grids, with a width slightly larger than the pixel size of the bolt in the original image. This partitioning method avoids grid boundaries from separating the targets, ensuring that each complete bolt-pin combination appears in at least one grid. Potentially leading to duplicate detection in overlapping regions is addressed by simply performing NMS (non-maximum suppression) after positioning the prediction box back to the original image. Furthermore, gridding effectively addresses the imbalance between positive and negative samples. Because pins occupy a small proportion of the image, negative samples (background) have a higher classification probability than positive samples (pins). After gridding, grids containing only background can be discarded during training based on their annotations, reducing computational effort while achieving a certain degree of balance between positive and negative samples. The grid division also generates a special training set of activation maps for the hardware class.

[0057] S3. Delete and label the pre-selected sub-graphs to obtain the target sub-graphs, and use the target sub-graphs to train the multi-target detection model and the heat map generation model.

[0058] In S3, the deletion of the primary subgraph includes:

[0059] The preliminary selected sub-images are divided into negative sample sub-images including only the background and positive sample sub-images containing pins; the negative sample sub-images are eliminated and the positive sample sub-images are used as target sub-images.

[0060] The positive sample sub-image is labeled as follows:

[0061] Category A: Hardware bolts with visible cotter pins;

[0062] Category B: Hardware bolts, cotter pins missing;

[0063] Category C: Hardware bolts, the split pin is not fully visible;

[0064] Category D: Tower bolts, no need for cotter pins.

[0065] S4. Generate a heat map of hardware by using the class activation mapping method;

[0066] Further extract the background information of the positive sample sub-image; generate a heat map of the pins in the image,

[0067] Use the VGG16 module to extract the feature information of the image, and the output of its last convolutional layer is 7×7×512;

[0068] Use the global average pooling layer to replace the original fully connected layer, and for each 7×7 feature map f k (x,y) average operation F k =∑ x,y f k (x,y); the final output category probability P c The expression is:

[0069]

[0070] Where k represents the feature map ordinal number, (x, y) represents the horizontal and vertical coordinates of the pixel in the feature map, c = {0, 1} represents the output category, 0 represents background, and 1 represents hardware string; Reflects the feature map f k The importance of (x,y) for category c;

[0071] The class activation map of category c is defined as:

[0072]

[0073] Based on the category information of all target sub-images, the VGG16-CAM network of the hardware string is trained. The 7×7 hardware class activation map is upsampled to the size of the input image to obtain the heat map M′(x,y).

[0074] S5. Superimpose the high-definition original image and the heat map as the input of the multi-target detection model. If the recognition accuracy of each category is greater than the preset accuracy (the accuracy is manually set according to the situation), the multi-target detection model is determined to be reliable. Otherwise, expand the data set and re-execute S3.

[0075] In S5, the high-definition original image and the heat map are superimposed as the input of the multi-target detection model, including:

[0076] The labeled target sub-image is restored to obtain the high-definition original image, and the heat map M′(x, y) is superimposed on the high-definition original image as the input image of the multi-target detection model, excluding the fourth channel in addition to the RGB three channels.

[0077] S6. The inspection images collected by the UAV in real time are identified through the trained multi-target detection model to obtain the category and location of the image representing the pin-level defect.

[0078] The specific implementation described above is a preferred implementation of the pin-level defect recognition method for drone inspection images of the present invention, and is not intended to limit the specific implementation scope of the present invention. The scope of the present invention includes but is not limited to this specific implementation. Any equivalent changes made in accordance with the shape and structure of the present invention are within the scope of protection of the present invention.

Claims

1. The pin-level defect recognition method based on UAV inspection images is characterized by: The steps include: S1. Build a multi-target detection model based on the R-FCN network, using the ResNet-FPN module as the feature extractor; S2. Obtain and annotate inspection images, and preprocess the inspection images using a grid-based semi-overlapping splitting method to obtain preliminary sub-images; S3. Filter and label the preselected subgraphs to obtain target subgraphs, and use the target subgraphs to train the multi-target detection model and the heat map generation model. S4. Generate heatmaps of hardware using the VGG16-CAM network through class activation mapping. S5. Superimpose the high-definition original image and the heat map as the input of the multi-target detection model. If the recognition accuracy of each category is greater than the preset accuracy, the multi-target detection model is determined to be reliable. Otherwise, expand the data set and re-execute S3. S6. Identify the inspection images collected by the drone in real time using the trained multi-target detection model to obtain the category and location of the image representing the pin-level defect; S4 includes: Based on the category labeling information of the pre-selected sub-images, a VGG16 network is trained to implement hardware classification. Use the VGG16 module to extract the feature information of the image, and the output of its last convolutional layer is 7×7×512; Use the global average pooling layer to replace the original fully connected layer, for each 7×7 feature map Averaging operation ; Final output category probability The expression is: ； in represents the feature map ordinal number, Represents the horizontal and vertical coordinates of the pixels in the feature map, Indicates the output category, 0 represents background, 1 represents hardware string; Reflects the feature map For categories Importance, define categories The class activation diagram is: ； Activate the hardware class image of size 7×7 After upsampling and adjusting to the size of the input image, the heat map is obtained .

2. The pin-level defect recognition method for drone inspection images according to claim 1 is characterized in that: It also includes constructing a category-balanced large margin loss function as the loss function in the R-FCN network.

3. The pin-level defect recognition method for drone inspection images according to claim 2 is characterized in that: The expression of the large margin loss function is: ； Among them: monotonically decreasing continuous function ; is the true category of the target subgraph; , is the number of categories, The true category is The number of fully connected layer units; represents the output vector of the fully connected layer, represents the modulus of the fully connected layer parameter vector, represents the modulus of the eigenvector, Represents a vector With vector Angle.

4. The pin-level defect recognition method for drone inspection images according to claim 1 is characterized in that: S2 includes: Split the high-definition original image into a series of preliminary sub-images of different sizes according to the grid; Among them, two adjacent preliminary selected sub-images have an overlapping area, and the width of the overlapping area is greater than the pixels of the bolt in the high-definition original image.

5. The pin-level defect recognition method for drone inspection images according to claim 4 is characterized in that: In S3, the deletion of the primary subgraph includes: The preliminary selected sub-images are divided into negative sample sub-images including only the background and positive sample sub-images containing pins; the negative sample sub-images are eliminated and the positive sample sub-images are used as target sub-images.

6. The pin-level defect recognition method for drone inspection images according to claim 5 is characterized in that: Label the positive sample sub-images by categories: including: Category A: Hardware bolts with visible cotter pins; Category B: Hardware bolts, cotter pins missing; Category C: Hardware bolts, the split pin is not fully visible; Category D: Tower bolts, no need for cotter pins.

7. The pin-level defect recognition method for drone inspection images according to claim 1 is characterized in that: In S5, the high-definition original image and the heat map are superimposed as the input of the multi-target detection model, including: Heat map Superimposed with the high-definition original image, it is used as the input image of the multi-target detection model, excluding the fourth channel in addition to the RGB three channels.

Images