A method, device and system for detecting power line image defects on an airborne end

Abstract

The present invention discloses a method, device and system for detecting defects in power line images for an airborne end, the method comprising: the airborne end collects an image to be detected of a power line; the image to be detected is input into a target detection model to obtain a detection result output by the target detection model, the detection result comprising a first area to be detected containing a target to be detected; according to the position information of the image to be detected collected by the airborne end, an area is extracted from the image to be detected to obtain a second area to be detected; according to the second area to be detected, the first area to be detected is corrected to obtain an image of a corrected area; the image of the corrected area is input into an improved deep convolutional neural network model to perform defect detection to obtain a defect detection result; the method can improve the accuracy of power line defect detection.

Description

Technical Field

[0001] The present invention relates to image defect detection and recognition technology, and in particular to a method, device and system for detecting image defects of power lines for airborne terminals. Background Art

[0002] In the past, line inspections were mainly carried out manually. With the development and maturity of drone technology, the use of drones to inspect transmission lines has been widely promoted and applied. Inspection drones are equipped with high-resolution cameras, remote sensing monitors, infrared thermometers and other equipment to conduct all-round photography and monitoring of power towers and power lines. Traditional defect identification mainly relies on manual review, but the main problem with manual review is that it is time-consuming, labor-intensive and inefficient.

[0003] With the development of artificial intelligence, image recognition technology is applied to defect identification in power line inspection. For example, the patent document (CN105141889A) proposes an intelligent inspection system for transmission lines based on image vision, which relates to the technical field of maintenance of power system transmission lines, including an image acquisition system, an image storage and management system, and an image intelligent processing system. The image acquisition system transmits the acquired transmission line image to be inspected to the image storage and management system, and the image storage and management system outputs the classified and marked image to the image intelligent processing system for transmission line fault detection. For another example, the patent document (CN110222683A) proposes a method for identifying and locating burst defects of insulators in power transmission lines based on a deep convolutional neural network. The inspection pictures are annotated by means of manual marking and the object detection SSD model is trained. Then, the real-time transmission line inspection images are identified. Gradient detection is performed on the identified insulator area to calculate the direction of the insulator area, and the insulator area in the image is segmented using the kmean clustering algorithm. The direction of the insulator area is corrected and the insulator foreground points are counted in the column direction. In this way, the distribution curve of the foreground points of the insulation area in the horizontal direction can be obtained, and then the burst points on the insulator can be located.

[0004] Although the above method can identify defects based on images of transmission lines, the algorithm is relatively complex, the detection accuracy is low, the reliability is low, and most areas of the power lines are hollow, such as power towers and cables. Their hollow structure makes it difficult to distinguish the background during image recognition, making the extracted detection frame inaccurate or difficult to cover the range to be detected. Summary of the invention

[0005] The present invention provides a method, device and system for detecting power line image defects on an airborne end, which can effectively improve the accuracy of power transmission line defect detection.

[0006] A method for detecting defects in power line images for an airborne terminal, comprising:

[0007] The airborne end collects the image of the power line to be inspected;

[0008] Inputting the image to be detected into a target detection model to obtain a detection result output by the target detection model, wherein the detection result includes a first region to be detected containing the target to be detected;

[0009] According to the position information of the image to be detected collected by the airborne end, performing region extraction on the image to be detected to obtain a second region to be detected;

[0010] Correcting the first area to be detected according to the second area to be detected to obtain a corrected area image;

[0011] The correction area image is input into an improved deep convolutional neural network model for defect detection to obtain a defect detection result.

[0012] Furthermore, the airborne end includes a drone and / or an inspection robot.

[0013] Further, according to the position information of the image to be detected collected by the airborne end, performing region extraction on the image to be detected to obtain a second region to be detected includes:

[0014] Inputting the position information of the image to be detected into a pre-trained prediction model, and outputting a second area to be detected;

[0015] The prediction model is obtained by training a sample set based on power line images containing targets to be detected collected by the airborne end at different locations.

[0016] Further, correcting the first area to be detected according to the second area to be detected to obtain a corrected area image includes:

[0017] Determine a first center point of the first area to be detected and a second center point of the second area to be detected;

[0018] Calculate the distance between the first center point and the second center point;

[0019] When the distance is greater than the side length of the first area to be detected and less than twice the side length of the first area to be detected, a characteristic ellipse is constructed with the first center point and the second center point as foci and twice the side length of the first area to be detected as a constant, and a circumscribed rectangle of the characteristic ellipse is used as the correction area image;

[0020] When the distance is greater than twice the side length of the first area to be detected, constructing a first characteristic circle and a second characteristic circle with the first center point and the second center point as the center of the circle respectively, and taking the circumscribed rectangle formed by the first characteristic circle and the second characteristic circle as the correction area image;

[0021] When the distance is less than or equal to the side length of the first area to be detected, the first area to be detected and the second area to be detected are merged to form the correction area image.

[0022] Furthermore, when any side of the correction region image exceeds the image to be detected, the correction region is scaled to within the region of the image to be detected.

[0023] Furthermore, the target detection model is obtained by pre-training based on a fully convolutional neural network.

[0024] Furthermore, the improved deep convolutional neural network model includes an input layer, a convolution layer, a batch normalization layer, an activation layer, a pooling layer, a fully connected layer and an output layer, wherein the convolution layer includes a standard convolution layer and a depthwise separable convolution layer.

[0025] Furthermore, before inputting the image to be detected into the target detection model, the method further includes:

[0026] The image to be detected is preprocessed.

[0027] An airborne power line image defect detection device comprises:

[0028] An image acquisition module, used for acquiring the image to be detected of the power line through the airborne terminal;

[0029] The target detection module is used to input the image to be detected into the target detection model to obtain the detection result output by the target detection model, wherein the detection result includes a first area to be detected containing the target to be detected;

[0030] An extraction module, used for performing region extraction on the image to be detected according to the position information of the image to be detected collected by the airborne end, so as to obtain a second region to be detected;

[0031] A correction module, used for correcting the first area to be detected according to the second area to be detected to obtain a correction area image;

[0032] The defect detection module is used to input the correction area image into the improved deep convolutional neural network model for defect detection to obtain defect detection results.

[0033] Furthermore, the airborne end includes a drone and / or an inspection robot.

[0034] Furthermore, the extraction module extracts a region of the image to be detected based on the position information of the image to be detected collected by the airborne end to obtain a second region to be detected, including:

[0035] Inputting the position information of the image to be detected into a pre-trained prediction model, and outputting a second area to be detected;

[0036] The prediction model is obtained by training a sample set based on power line images containing targets to be detected collected by the airborne end at different locations.

[0037] Furthermore, the correction module corrects the first area to be detected according to the second area to be detected to obtain a correction area image, including:

[0038] Determine a first center point of the first area to be detected and a second center point of the second area to be detected;

[0039] Calculate the distance between the first center point and the second center point;

[0040] When the distance is greater than the side length of the first area to be detected and less than twice the side length of the first area to be detected, a characteristic ellipse is constructed with the first center point and the second center point as foci and twice the side length of the first area to be detected as a constant, and a circumscribed rectangle of the characteristic ellipse is used as the correction area image;

[0041] When the distance is greater than twice the side length of the first area to be detected, constructing a first characteristic circle and a second characteristic circle with the first center point and the second center point as the center of the circle respectively, and taking the circumscribed rectangle formed by the first characteristic circle and the second characteristic circle as the correction area image;

[0042] When the distance is less than or equal to the side length of the first area to be detected, the first area to be detected and the second area to be detected are merged to form the correction area image.

[0043] Furthermore, when any side of the correction region image exceeds the image to be detected, the correction region is scaled to within the region of the image to be detected.

[0044] Furthermore, the target detection model is obtained by pre-training based on a fully convolutional neural network.

[0045] Furthermore, the improved deep convolutional neural network model includes an input layer, a convolution layer, a batch normalization layer, an activation layer, a pooling layer, a fully connected layer and an output layer, wherein the convolution layer includes a standard convolution layer and a depthwise separable convolution layer.

[0046] Furthermore, the device also includes a preprocessing module, which is used to preprocess the image to be detected.

[0047] A power line image defect detection system for an airborne end comprises a processor, a storage device and an airborne end, wherein the storage device stores a plurality of instructions, and the processor is used to read the instructions and execute the above method.

[0048] The method, device and system for detecting defects in power line images for airborne terminals provided by the present invention have at least the following beneficial effects:

[0049] (1) The target image to be inspected is first identified, and the identification result is corrected. The defect identification is performed on the corrected image, which can cover the target to be inspected to a greater extent and improve the accuracy of defect identification. It is particularly suitable for defect identification of hollow structures such as power lines and towers;

[0050] (2) The deep convolutional neural network is improved by using a deep separable convolutional layer instead of a common convolutional layer, which realizes the feature extraction of the spatial dimension and the fusion of the output channel information, reduces the amount of calculation, and improves the efficiency of defect recognition;

[0051] (3) The improved deep convolutional neural network model uses a batch normalization layer to solve the problem of covariate shift. In each layer, the input data is normalized according to the statistical information to avoid changes in the input distribution of the network;

[0052] (4) The image to be inspected is preprocessed in advance, which retains the natural color and rich details of the original image to be inspected, enhances the image contrast, and improves the accuracy of subsequent defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention provides a flowchart of an embodiment of the method for detecting defects in power line images for an airborne terminal.

[0054] Figure 2 The present invention provides a flowchart of an embodiment of regional correction in the method for detecting defects in power line images for airborne end.

[0055] Figure 3 The present invention provides a flowchart of an embodiment of the airborne power line image defect detection device provided by the present invention.

[0056] Figure 4 A flow chart of another embodiment of the airborne power line image defect detection device provided by the present invention.

[0057] Figure 5 The present invention provides a flowchart of an embodiment of the airborne power line image defect detection system provided by the present invention. DETAILED DESCRIPTION

[0058] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0059] refer to Figure 1 In some embodiments, a method for detecting defects in power line images for an airborne terminal is provided, comprising:

[0060] S1. Collect the image of the power line to be detected through the airborne terminal;

[0061] S2, inputting the image to be detected into a target detection model to obtain a detection result output by the target detection model, wherein the detection result includes a first region to be detected containing the target to be detected;

[0062] S3, extracting a region of the image to be detected based on the position information of the image to be detected collected by the airborne end, and obtaining a second region to be detected;

[0063] S4, correcting the first area to be detected according to the second area to be detected to obtain a corrected area image;

[0064] S5. Input the correction area image into an improved deep convolutional neural network model for defect detection to obtain a defect detection result.

[0065] Furthermore, in step S1, the airborne terminal includes but is not limited to a drone, an inspection robot, etc.

[0066] Furthermore, in step S2, the target detection model is pre-trained based on a fully convolutional neural network, and the training process is as follows:

[0067] First, multiple training samples are obtained to establish a training set. The training samples include image samples and their annotation information. For example, in this embodiment, the image samples include images of different components of power lines taken at different angles in the power corridor. Each component of the power line is labeled to form the training set. The fully convolutional neural network is trained based on the training set. The goal of the training is to minimize the difference between the detection results output by the target detection model and the labels annotated with the image samples.

[0068] Image samples can be labeled manually, but this method requires a lot of manpower and is costly. Therefore, a limited number of labeled training samples can be used to train a target detection model, and then the initially trained target detection model can be used to identify unlabeled image samples to obtain detection results. The image samples are labeled using the detection results to form pseudo-labels. After evaluating, screening and correcting the generated pseudo-labels, the initial target detection model is used to further train the image samples with pseudo-labels to obtain an updated target detection model. The updated target detection model is then used to generate new pseudo-labels, and a new round of training is performed using the new pseudo-labels. This process is repeated and multiple rounds of training iterations are performed to finally obtain a target detection model.

[0069] Further, in step S2, after the image to be detected is input into the target detection model, the detection result is output, and the detection result includes the first area to be detected of the target to be detected. For example, if the target to be detected is a power pole tower, the output detection result includes the area of ​​the power pole tower. However, in the power corridor, the transmission lines and power pole towers are hollow structures, and the hollow parts in the captured image are a large number of background images, which has a certain impact on target recognition and causes inaccurate detection results. Improving the accuracy and precision of the target detection model requires a large number of training samples and a complex model structure. To address this problem, this embodiment proposes to correct the first area to be detected contained in the detection result.

[0070] Further, in step S3, according to the position information of the image to be detected collected by the airborne end, the region of the image to be detected is extracted to obtain a second region to be detected, including:

[0071] Inputting the position information of the image to be detected into a pre-trained prediction model, and outputting a second area to be detected;

[0072] The prediction model is obtained by training a sample set based on power line images containing targets to be detected collected by the airborne end at different locations.

[0073] Specifically, the airborne end collects power line images containing targets to be detected at different positions, records their position information, marks the targets to be detected in the images, and obtains the pixel coordinate range of the targets to be detected in the images. Based on the image information obtained above, a sample set is established and the prediction model is trained until the prediction model converges. After that, the position information of the collected images to be detected is input, and the pixel coordinate range of the targets to be detected in the image can be output, that is, the second area to be detected that is most likely to contain the targets to be detected is obtained.

[0074] Furthermore, the first area to be detected and the second area to be detected are both square boxes, that is, the feature boxes output by the target detection model and the prediction model are both square boxes.

[0075] Further, refer to Figure 2 In step S4, the first area to be detected is corrected according to the second area to be detected to obtain a corrected area image, including:

[0076] S41, determining a first center point of the first to-be-detected area and a second center point of the second to-be-detected area;

[0077] S42, calculating the distance between the first center point and the second center point;

[0078] S43, when the distance is greater than the side length of the first area to be detected and less than twice the side length of the first area to be detected, construct a characteristic ellipse with the first center point and the second center point as foci and twice the side length of the first area to be detected as a constant, and use the circumscribed rectangle of the characteristic ellipse as the correction area image;

[0079] S44, when the distance is greater than twice the side length of the first area to be detected, constructing a first characteristic circle and a second characteristic circle with the first center point and the second center point as the center of the circle respectively, and taking the circumscribed rectangle formed by the first characteristic circle and the second characteristic circle as the correction area image;

[0080] S45. When the distance is less than or equal to the side length of the first area to be detected, the first area to be detected and the second area to be detected are merged to form the correction area image.

[0081] Specifically, in step S43, when the distance between the center points of the first to-be-detected area and the second to-be-detected area is greater than the side length of the first to-be-detected area and less than twice the side length of the first to-be-detected area, in this case, it is indicated that the first to-be-detected area output by the target detection model is not very accurate, and a characteristic ellipse is constructed with the first center point and the second center point as the focus and twice the side length of the first to-be-detected area as a constant. The equation of the characteristic ellipse is x 2 / a 2 +y 2 / b 2 =1(a>b>0), c 2 =a 2 -b 2 , 2c is the distance between the first center point and the second center point, and 2a is a constant, which is equal to twice the side length of the first area to be detected.

[0082] Further, in step S44, when the distance between the first center point and the second center point is greater than twice the side length of the first area to be detected, it means that the range of the target to be detected covered by the first area to be detected output by the target detection model is small, and a first characteristic circle and a second characteristic circle are constructed with the first center point and the second center point as the center of the circle, respectively, and the circumscribed rectangle formed by the first characteristic circle and the second characteristic circle is used as the correction area image.

[0083] Further, in step S45, when the distance between the first center point and the second center point is less than or equal to the side length of the first area to be detected, the first area to be detected output by the surface target detection model has a higher accuracy, and the first area to be detected and the second area to be detected are merged to form the corrected area image.

[0084] Furthermore, when any side of the correction region image exceeds the image to be detected, the correction region is scaled to within the region of the image to be detected.

[0085] After the above-mentioned regional correction, the target to be inspected can be covered more comprehensively, thereby improving the accuracy of subsequent defect identification.

[0086] Furthermore, in step S5, the improved deep convolutional neural network model includes an input layer, a convolution layer, a batch normalization layer, an activation layer, a pooling layer, a fully connected layer and an output layer, wherein the convolution layer includes a standard convolution layer and a depthwise separable convolution layer.

[0087] Specifically, the correction area image first enters the input layer, and the dimension of the input layer can be set according to actual needs. For example, the correction area image can be an RGB image of 200*200*3, then the dimension of the input layer is 200*200*3.

[0088] Furthermore, the convolution layer includes a standard convolution layer and a depthwise separable convolution layer, wherein the standard convolution layer is a 3*3 standard convolution layer, and the depthwise separable convolution layer can be multiple layers.

[0089] Specifically, the input of the standard convolutional layer is defined as D F ×D P ×M, and the output is defined as D F ×D P ×N, where D F and D P are the spatial width and height of the input feature map, M is the dimension of the input channel, and N is the dimension of the output channel.

[0090] The depth-wise separable convolution layer includes a depth-wise convolution layer and a point-wise convolution layer. The feature map output by the standard convolution layer is input to the depth-wise convolution layer. The depth-wise convolution layer extracts the feature map X(3, D F , DP ) to obtain spatially related features. Different from the conventional standard convolution layer, the deep convolution layer applies a single filter to each input channel, that is, one convolution kernel corresponds to only one input channel. In other words, one input channel is convolved by only one convolution kernel. The size of the convolution kernel can be set according to actual needs. For example, the convolution kernel size can be 2*2. The deep convolution layer uses three convolution kernels to perform convolution operations on each channel of the input feature map, expanding the channel dimension.

[0091] Furthermore, the feature map output by the deep convolution realizes the feature extraction of the spatial dimension. The feature map output by the deep convolution layer enters the point-by-point convolution layer. The point-by-point convolution layer fuses the channel information output by the deep convolution layer, and finally realizes the fusion of spatial features and channel information.

[0092] Using a depthwise separable convolutional layer instead of the traditional standard convolutional layer can effectively extract and fuse spatial and channel information, reduce the amount of calculation, and improve the efficiency of defect recognition.

[0093] Furthermore, the batch normalization layer is used to regularize the input data and keep its distribution in a stable state. Specifically, in ordinary neural networks, in order to avoid the phenomenon of gradient disappearance or gradient explosion, the data usually needs to be normalized, but this method will cause the problem of covariate shift in the hidden layer in the middle of the network, that is, the input data distribution of each layer will change with the update of the parameters of the previous layer, so each layer needs to re-learn a new normalization parameter. In this case, the training speed of the network will slow down or even fail to converge. The improved deep convolutional neural network model uses a batch normalization layer to solve this problem. It can normalize the input data in each layer according to the statistical information of the input data without changing the input distribution of the network. The calculation of the batch normalization layer is as follows:

[0094] For each batch of input data, calculate the mean and variance;

[0095] The input data of the batch is normalized using the calculated mean and variance;

[0096] Scale and translate the normalized data to get the final output;

[0097] The final output is sent to the next layer of network for calculation.

[0098] Furthermore, the activation layer processes the linear output of the previous layer through a nonlinear activation function, so as to simulate any function, thereby enhancing the representation ability of the network. As an optional implementation, the activation layer in this embodiment adopts the RELU function, which is expressed as follows:

[0099] y=0,x<0

[0100] y=x,x≥0.

[0101] Furthermore, the pooling layer is used to reduce the image size so that adjacent pixels within a specified range of the image are combined into a representative value.

[0102] Furthermore, the fully connected layer integrates the local features with category discrimination in the pooling layer through the weight matrix, and uses Softmax through the output layer to normalize the numerical vector into a probability distribution vector to obtain the prediction result.

[0103] The improved deep convolutional neural network model training process provided in this embodiment is as follows:

[0104] The collected historical image data may be an image taken by traditional ground manual collection, and the defect positions in the defect images in the sample library are marked to obtain a defect sample set;

[0105] Divide the defect sample set into a training set and a test set;

[0106] Inputting the training set into an initial deep convolutional neural network model for training, and adjusting the parameters of the deep convolutional neural network model using an asynchronous gradient descent strategy until the deep convolutional neural network model converges;

[0107] The test set is used to test the trained deep convolutional neural network model.

[0108] Among them, the asynchronous gradient descent strategy is used to adjust the parameters of the deep convolutional neural network model, including:

[0109] Set the input image size, category, learning rate, weight decay, and maximum number of iterations;

[0110] Iterate and execute the following steps until the maximum number of iterations is reached:

[0111] Randomly select defect images and defect locations in the training set and input them into the deep convolutional neural network model;

[0112] The feature map output by the depthwise separable convolutional layer is divided into grids, and a preset number of bounding boxes are predicted for each grid;

[0113] Predict the confidence of the defect target in each bounding box and give the center coordinates of each bounding box;

[0114] The confidence of the defect target and the difference value of the defect position are calculated through the loss function;

[0115] Adjusting the parameters of the deep convolutional neural network model according to the difference value, the learning rate, and the weight decay until all images in the training set are traversed;

[0116] The number of iterations increases by 1, and the above steps are executed in a loop.

[0117] Furthermore, before inputting the image to be detected into the target detection model, the method further includes:

[0118] The image to be detected is preprocessed.

[0119] In some embodiments, preprocessing of the image to be detected may include:

[0120] The three color channels of the image to be detected are stretched and mapped to the HIS space respectively, the channel representing the brightness is contrast enhanced, and gamma adaptive correction is performed, the brightness channel after contrast enhancement and gamma adaptive correction is combined with the other two channels, and converted back to the RGB space to obtain the preprocessed image to be detected.

[0121] The preprocessed image to be inspected retains the natural color and rich details of the original image to be inspected, enhances the image contrast, and improves the accuracy of subsequent defect detection.

[0122] The above-mentioned embodiment provides an airborne power line image defect detection method, which has at least the following beneficial effects:

[0123] (1) The target image to be inspected is first identified, and the identification result is corrected. The defect identification is performed on the corrected image, which can cover the target to be inspected to a greater extent and improve the accuracy of defect identification. It is particularly suitable for defect identification of hollow structures such as power lines and towers;

[0124] (2) The deep convolutional neural network is improved by using a deep separable convolutional layer instead of a common convolutional layer, which realizes the feature extraction of the spatial dimension and the fusion of the output channel information, reduces the amount of calculation, and improves the efficiency of defect recognition;

[0125] (3) The improved deep convolutional neural network model uses a batch normalization layer to solve the problem of covariate shift. In each layer, the input data is normalized according to the statistical information to avoid changes in the input distribution of the network;

[0126] (4) The image to be inspected is preprocessed in advance, which retains the natural color and rich details of the original image to be inspected, enhances the image contrast, and improves the accuracy of subsequent defect detection.

[0127] refer to Figure 3In some embodiments, a power line image defect detection device for an airborne terminal is provided, comprising:

[0128] An image acquisition module 201 is used to acquire an image of a power line to be inspected;

[0129] The target detection module 202 is used to input the image to be detected into the target detection model to obtain the detection result output by the target detection model, wherein the detection result includes a first region to be detected containing the target to be detected;

[0130] An extraction module 203 is used to extract a region of the image to be detected based on the position information of the image to be detected collected by the airborne end, so as to obtain a second region to be detected;

[0131] A correction module 204, configured to correct the first area to be detected according to the second area to be detected to obtain a corrected area image;

[0132] The defect detection module 205 is used to input the correction area image into the improved deep convolutional neural network model for defect detection to obtain a defect detection result.

[0133] Furthermore, the airborne end includes a drone and / or an inspection robot.

[0134] Furthermore, the target detection model is pre-trained based on a fully convolutional neural network, and its training process is as follows:

[0135] First, multiple training samples are obtained to establish a training set. The training samples include image samples and their annotation information. For example, in this embodiment, the image samples include images of different components of power lines taken at different angles in the power corridor. Each component of the power line is labeled to form the training set. The fully convolutional neural network is trained based on the training set. The goal of the training is to minimize the difference between the detection results output by the target detection model and the labels annotated with the image samples.

[0136] Image samples can be labeled manually, but this method requires a lot of manpower and is costly. Therefore, a limited number of labeled training samples can be used to train a target detection model, and then the initially trained target detection model can be used to detect unlabeled image samples to obtain detection results. The image samples are labeled using the detection results to form pseudo-labels. After evaluating, screening and correcting the generated pseudo-labels, the initial target detection model is used to further train the image samples with pseudo-labels to obtain an updated target detection model. The updated target detection model is then used to generate new pseudo-labels, and a new round of training is performed using the new pseudo-labels. This process is repeated and multiple rounds of training iterations are performed to finally obtain a target detection model.

[0137] Furthermore, after the image to be detected is input into the target detection model, a detection result is output, which includes a first area to be detected of the target to be detected. For example, if the target to be detected is a power pole tower, the output detection result includes the area of ​​the power pole tower. However, in the power corridor, the transmission lines and power pole towers are hollow structures, and the hollow parts in the captured image are a large number of background images, which has a certain impact on target recognition and causes inaccurate detection results. Improving the accuracy and precision of the target detection model requires a large number of training samples and a complex model structure. To address this problem, this embodiment proposes to correct the first area to be detected contained in the detection result.

[0138] Furthermore, the extraction module 203 extracts a region of the image to be detected based on the position information of the image to be detected collected by the airborne end to obtain a second region to be detected, including:

[0139] Inputting the position information of the image to be detected into a pre-trained prediction model, and outputting a second area to be detected;

[0140] The prediction model is obtained by training a sample set based on power line images containing targets to be detected collected by the airborne end at different locations.

[0141] Specifically, the airborne end collects power line images containing targets to be detected at different positions, marks the targets to be detected in the images, and obtains the pixel coordinate range of the targets to be detected in the images. A sample set is established based on the image information obtained above, and the prediction model is trained until the prediction model converges. After that, the position information of the collected images to be detected is input, and the pixel coordinate range of the targets to be detected in the image can be output, that is, the second area to be detected that is most likely to contain the targets to be detected is obtained.

[0142] Furthermore, the first area to be detected and the second area to be detected are both square boxes, that is, the feature boxes output by the target detection model and the prediction model are both square boxes.

[0143] Furthermore, the correction module 204 corrects the first area to be detected according to the second area to be detected to obtain a corrected area image, including:

[0144] Determine a first center point of the first area to be detected and a second center point of the second area to be detected;

[0145] Calculate the distance between the first center point and the second center point;

[0146] When the distance is greater than the side length of the first area to be detected and less than twice the side length of the first area to be detected, a characteristic ellipse is constructed with the first center point and the second center point as foci and twice the side length of the first area to be detected as a constant, and a circumscribed rectangle of the characteristic ellipse is used as the correction area image;

[0147] When the distance is greater than twice the side length of the first area to be detected, constructing a first characteristic circle and a second characteristic circle with the first center point and the second center point as the center of the circle respectively, and taking the circumscribed rectangle formed by the first characteristic circle and the second characteristic circle as the correction area image;

[0148] When the distance is less than or equal to the side length of the first area to be detected, the first area to be detected and the second area to be detected are merged to form the correction area image.

[0149] Specifically, when the distance between the center points of the first to-be-detected area and the second to-be-detected area is greater than the side length of the first to-be-detected area and less than twice the side length of the first to-be-detected area, in this case, it means that the first to-be-detected area output by the target detection model is not very accurate. With the first center point and the second center point as the focus and twice the side length of the first to-be-detected area as the constant, a characteristic ellipse is constructed. The equation of the characteristic ellipse is x 2 / a 2 +y 2 / b 2 =1(a>b>0), c 2 =a 2 -b 2 , 2c is the distance between the first center point and the second center point, and 2a is a constant, which is equal to twice the side length of the first area to be detected.

[0150] Furthermore, when the distance between the first center point and the second center point is greater than twice the side length of the first area to be detected, it means that the range of the target to be detected covered by the first area to be detected output by the target detection model is small, and the first characteristic circle and the second characteristic circle are constructed with the first center point and the second center point as the center of the circle respectively, and the circumscribed rectangle formed by the first characteristic circle and the second characteristic circle is used as the correction area image.

[0151] Furthermore, when the distance between the first center point and the second center point is less than or equal to the side length of the first area to be detected, the first area to be detected output by the surface target detection model is more accurate, and the first area to be detected and the second area to be detected are merged to form the correction area image.

[0152] Furthermore, when any side of the correction region image exceeds the image to be detected, the correction region is scaled to within the region of the image to be detected.

[0153] After the above-mentioned regional correction, the target to be inspected can be covered more comprehensively, thereby improving the accuracy of subsequent defect identification.

[0154] Furthermore, the improved deep convolutional neural network model includes an input layer, a convolution layer, a batch normalization layer, an activation layer, a pooling layer, a fully connected layer and an output layer, wherein the convolution layer includes a standard convolution layer and a depthwise separable convolution layer. Specifically, the correction area image first enters the input layer, and the dimension of the input layer can be set according to actual needs. For example, the correction area image can be a 200*200*3 RGB image, and the dimension of the input layer is 200*200*3.

[0155] Furthermore, the convolution layer includes a standard convolution layer and a depthwise separable convolution layer, wherein the standard convolution layer is a 3*3 standard convolution layer, and the depthwise separable convolution layer can be multiple layers.

[0156] Specifically, the input of the standard convolutional layer is defined as D F ×D P ×M, and the output is defined as D F ×D P ×N, where D F and D P are the spatial width and height of the input feature map, M is the dimension of the input channel, and N is the dimension of the output channel.

[0157] The depth-wise separable convolution layer includes a depth-wise convolution layer and a point-wise convolution layer. The feature map output by the standard convolution layer is input to the depth-wise convolution layer. The depth-wise convolution layer extracts the feature map X(3, D F , D P ) to obtain spatially related features. Different from the conventional standard convolution layer, the deep convolution layer applies a single filter to each input channel, that is, one convolution kernel corresponds to only one input channel. In other words, one input channel is convolved by only one convolution kernel. The size of the convolution kernel can be set according to actual needs. For example, the convolution kernel size can be 2*2. The deep convolution layer uses three convolution kernels to perform convolution operations on each channel of the input feature map, expanding the channel dimension.

[0158] Furthermore, the feature map output by the deep convolution realizes the feature extraction of the spatial dimension. The feature map output by the deep convolution layer enters the point-by-point convolution layer. The point-by-point convolution layer fuses the channel information output by the deep convolution layer, and finally realizes the fusion of spatial features and channel information.

[0159] Using a depthwise separable convolutional layer instead of the traditional standard convolutional layer can effectively extract and fuse spatial and channel information, reduce the amount of calculation, and improve the efficiency of defect recognition.

[0160] Furthermore, the batch normalization layer is used to regularize the input data and keep its distribution in a stable state. Specifically, in ordinary neural networks, in order to avoid the phenomenon of gradient disappearance or gradient explosion, the data usually needs to be normalized, but this method will cause the problem of covariate shift in the hidden layer in the middle of the network, that is, the input data distribution of each layer will change with the update of the parameters of the previous layer, so each layer needs to re-learn a new normalization parameter. In this case, the training speed of the network will slow down or even fail to converge. The improved deep convolutional neural network model uses a batch normalization layer to solve this problem. It can normalize the input data in each layer according to the statistical information of the input data without changing the input distribution of the network. The calculation of the batch normalization layer is as follows:

[0161] For each batch of input data, calculate the mean and variance;

[0162] The input data of the batch is normalized using the calculated mean and variance;

[0163] Scale and translate the normalized data to get the final output;

[0164] The final output is sent to the next layer of network for calculation.

[0165] Furthermore, the activation layer processes the linear output of the previous layer through a nonlinear activation function, so as to simulate any function, thereby enhancing the representation ability of the network. As an optional implementation, the activation layer in this embodiment adopts the RELU function, which is expressed as follows:

[0166] y=0,x<0

[0167] y=x,x≥0.

[0168] Furthermore, the pooling layer is used to reduce the image size so that adjacent pixels within a specified range of the image are combined into a representative value.

[0169] Furthermore, the fully connected layer integrates the local features with category discrimination in the pooling layer through the weight matrix, and uses Softmax through the output layer to normalize the numerical vector into a probability distribution vector to obtain the prediction result.

[0170] The improved deep convolutional neural network model training process provided in this embodiment is as follows:

[0171] The collected historical image data may be an image taken by traditional ground manual collection, and the defect positions in the defect images in the sample library are marked to obtain a defect sample set;

[0172] Divide the defect sample set into a training set and a test set;

[0173] Inputting the training set into an initial deep convolutional neural network model for training, and adjusting the parameters of the deep convolutional neural network model using an asynchronous gradient descent strategy until the deep convolutional neural network model converges;

[0174] The test set is used to test the trained deep convolutional neural network model.

[0175] Among them, the asynchronous gradient descent strategy is used to adjust the parameters of the deep convolutional neural network model, including:

[0176] Set the input image size, category, learning rate, weight decay, and maximum number of iterations;

[0177] Iterate and execute the following steps until the maximum number of iterations is reached:

[0178] Randomly select defect images and defect locations in the training set and input them into the deep convolutional neural network model;

[0179] The feature map output by the depthwise separable convolutional layer is divided into grids, and a preset number of bounding boxes are predicted for each grid;

[0180] Predict the confidence of the defect target in each bounding box and give the center coordinates of each bounding box;

[0181] The confidence of the defect target and the difference value of the defect position are calculated through the loss function;

[0182] Adjusting the parameters of the deep convolutional neural network model according to the difference value, the learning rate, and the weight decay until all images in the training set are traversed;

[0183] The number of iterations increases by 1, and the above steps are executed in a loop.

[0184] Further, refer to Figure 4 The device provided in this embodiment also includes a preprocessing module 200, which is used to preprocess the image to be detected.

[0185] Specifically, the preprocessing module 200 may be used to preprocess the image to be detected, including:

[0186] The three color channels of the image to be detected are stretched and mapped to the HIS space respectively, the channel representing the brightness is contrast enhanced, and gamma adaptive correction is performed, the brightness channel after contrast enhancement and gamma adaptive correction is combined with the other two channels, and converted back to the RGB space to obtain the preprocessed image to be detected.

[0187] The preprocessed image to be inspected retains the natural color and rich details of the original image to be inspected, enhances the image contrast, and improves the accuracy of subsequent defect detection.

[0188] The power line defect detection device based on image recognition provided in the above embodiment has at least the following beneficial effects:

[0189] (1) The target image to be inspected is first identified, and the identification result is corrected. The defect identification is performed on the corrected image, which can cover the target to be inspected to a greater extent and improve the accuracy of defect identification. It is particularly suitable for defect identification of hollow structures such as power lines and towers;

[0190] (2) The deep convolutional neural network is improved by using a deep separable convolutional layer instead of a common convolutional layer, which realizes the feature extraction of the spatial dimension and the fusion of the output channel information, reduces the amount of calculation, and improves the efficiency of defect recognition;

[0191] (3) The improved deep convolutional neural network model uses a batch normalization layer to solve the problem of covariate shift. In each layer, the input data is normalized according to the statistical information to avoid changes in the input distribution of the network;

[0192] (4) The image to be inspected is preprocessed in advance, which retains the natural color and rich details of the original image to be inspected, enhances the image contrast, and improves the accuracy of subsequent defect detection.

[0193] refer to Figure 5 In some embodiments, a power line image defect detection system for an airborne terminal is also provided, comprising a processor 301, a storage device 302 and an airborne terminal 303, wherein the storage device stores a plurality of instructions, and the processor 301 is used to read the instructions and execute the above method.

[0194] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for detecting defects in power line images for an airborne terminal, characterized in that: include: The airborne end collects the image of the power line to be inspected; Inputting the image to be detected into a target detection model to obtain a detection result output by the target detection model, wherein the detection result includes a first region to be detected containing the target to be detected; According to the position information of the image to be detected collected by the airborne end, performing region extraction on the image to be detected to obtain a second region to be detected; Correcting the first area to be detected according to the second area to be detected to obtain a corrected area image; Inputting the correction area image into an improved deep convolutional neural network model for defect detection to obtain a defect detection result; Correcting the first area to be detected according to the second area to be detected to obtain a corrected area image, including: Determine a first center point of the first area to be detected and a second center point of the second area to be detected; Calculate the distance between the first center point and the second center point; When the distance is greater than the side length of the first area to be detected and less than twice the side length of the first area to be detected, a characteristic ellipse is constructed with the first center point and the second center point as foci and twice the side length of the first area to be detected as a constant, and a circumscribed rectangle of the characteristic ellipse is used as the correction area image; When the distance is greater than twice the side length of the first area to be detected, constructing a first characteristic circle and a second characteristic circle with the first center point and the second center point as the center of the circle respectively, and taking the circumscribed rectangle formed by the first characteristic circle and the second characteristic circle as the correction area image; When the distance is less than or equal to the side length of the first area to be detected, the first area to be detected and the second area to be detected are merged to form the correction area image.

2. The method according to claim 1, characterized in that The airborne end includes a drone and / or an inspection robot.

3. The method according to claim 1, characterized in that According to the position information of the image to be detected collected by the airborne end, performing region extraction on the image to be detected to obtain a second region to be detected, including: Inputting the location information of the image to be detected into a pre-trained prediction model, and outputting a second area to be detected; The prediction model is obtained by training a sample set based on power line images containing targets to be detected collected by the airborne end at different locations.

4. The method according to claim 1, characterized in that: When any side of the correction region image exceeds the image to be detected, the correction region is scaled to within the region of the image to be detected.

5. The method according to claim 1, characterized in that The target detection model is obtained by pre-training based on a fully convolutional neural network.

6. The method according to claim 1, characterized in that The improved deep convolutional neural network model includes an input layer, a convolution layer, a batch normalization layer, an activation layer, a pooling layer, a fully connected layer and an output layer, wherein the convolution layer includes a standard convolution layer and a depthwise separable convolution layer.

7. The method according to claim 5, characterized in that Before inputting the image to be detected into the target detection model, the method further includes: The image to be detected is preprocessed.

8. An onboard power line image defect detection device applied to the method as claimed in any one of claims 1 to 7, characterized in that: include: An image acquisition module, used for acquiring images of power lines to be inspected; The target detection module is used to input the image to be detected into the target detection model to obtain the detection result output by the target detection model, wherein the detection result includes a first area to be detected containing the target to be detected; An extraction module, used for performing region extraction on the image to be detected according to the position information of the image to be detected collected by the airborne end, so as to obtain a second region to be detected; A correction module, used for correcting the first area to be detected according to the second area to be detected to obtain a correction area image; A defect detection module, used for inputting the correction area image into an improved deep convolutional neural network model for defect detection to obtain a defect detection result; The correction module corrects the first area to be detected according to the second area to be detected to obtain a correction area image, including: Determine a first center point of the first area to be detected and a second center point of the second area to be detected; Calculate the distance between the first center point and the second center point; When the distance is greater than the side length of the first area to be detected and less than twice the side length of the first area to be detected, a characteristic ellipse is constructed with the first center point and the second center point as foci and twice the side length of the first area to be detected as a constant, and a circumscribed rectangle of the characteristic ellipse is used as the correction area image; When the distance is greater than twice the side length of the first area to be detected, constructing a first characteristic circle and a second characteristic circle with the first center point and the second center point as the center of the circle respectively, and taking the circumscribed rectangle formed by the first characteristic circle and the second characteristic circle as the correction area image; When the distance is less than or equal to the side length of the first area to be detected, the first area to be detected and the second area to be detected are merged to form the correction area image.

9. An airborne power line image defect detection system, characterized in that: It comprises a processor, a storage device and an onboard terminal, wherein the storage device stores a plurality of instructions, and the processor is used to read the instructions and execute any method as claimed in claims 1-7.

Images