A multi-level ground wire defect identification method based on disparity-assisted semantic segmentation

Abstract

The application discloses a kind of multistage ground wire defect identification methods based on disparity auxiliary semantic segmentation, comprising: using the ground wire defect image of unmanned aerial vehicle inspection collection, the disparity of two images successively collected by unmanned aerial vehicle is divided into foreground and background area using dense inverse search (DIS) optical flow method, to reduce the influence of complex background in ground wire defect graph on defect target detection, then using the image semantic segmentation network model designed for ground wire feature to segment ground wire area from foreground area, avoid unnecessary feature map calculation by block detection, while avoiding the details of ground wire features being submerged by image downsampling, limit positive and negative samples within ground wire area, design and train ground wire target detection and identification network model for ground wire area, make ground wire target detection and identification network model specially learn the area of defect ground wire and normal ground wire, filter out the influence of complex background, realize the detection and identification of small defect area.

Description

Technical Field

[0001] The present invention belongs to the field of automated power line inspection and image processing, and particularly relates to a multi-level ground wire defect recognition method based on parallax-assisted semantic segmentation. Background Art

[0002] Drone-based power line inspection has gradually replaced manual inspections as the mainstream inspection method. Before the advent of deep learning technology, defect identification in drone-captured power line images was primarily performed manually. Traditional edge segmentation and line detection algorithms have been used to achieve a certain degree of automated defect detection for small, easily describable defects, such as ground wire defects in ground wire inspection images. However, due to the poor adaptability of these algorithms and their impact on complex environments, lighting, and angles, recognition accuracy is low, making them inadequate for practical applications. Since the introduction of deep learning methods for object detection, many studies have begun utilizing deep neural network models for ground wire detection, achieving significantly improved accuracy compared to traditional algorithm-based defect detection methods. However, current deep learning-based ground wire detection methods often utilize single-stage or two-stage target detection models for direct ground wire defect detection. This makes it difficult to identify certain small defects (e.g., a defect with an area of ​​less than 100 pixels by 100 pixels in a 5000×4000 resolution image, representing less than 0.05% of the total image) and prone to false detections, resulting in bottlenecks in detection accuracy.

[0003] After searching the literature on the existing technologies, it was found that most of the research on image defect detection focused on locating target defects based on target detection technology. In the Chinese patent "A defect detection method based on deep learning semantic segmentation" (application number: CN201910673139.5), Zhao Qing et al. proposed a defect detection method based on deep learning semantic segmentation. This method directly uses the semantic segmentation network to obtain the classification of defects, and its detection and recognition effect is equivalent to that of single-stage detection. In the Chinese patent "Insulator defect detection method based on semantic segmentation and target detection" (application number: CN202111480007.4), Wang Shengli et al. proposed a defect detection method based on semantic segmentation and target detection models. This method adopts a two-stage detection method of semantic segmentation network and target detection network, but due to the different characteristics of ground wire defect images, the algorithm is difficult to be directly applied to ground wire image defect detection. Summary of the Invention

[0004] In view of this, the present invention provides a multi-level ground wire defect recognition method based on parallax-assisted semantic segmentation, which can realize the detection and recognition of small defect areas.

[0005] The purpose of the present invention is to achieve this through the following technical solution: a multi-level ground wire defect recognition method based on parallax-assisted semantic segmentation, comprising the following steps:

[0006] A multi-level ground wire defect recognition method based on parallax-assisted semantic segmentation includes the following steps:

[0007] Perform pixel-level annotation on power line inspection conductor and ground wire images to construct a conductor and ground wire semantic segmentation dataset;

[0008] The power line inspection ground wire defect images are annotated with rectangular frames to construct a ground wire defect detection dataset;

[0009] Obtain the optical flow vector for two consecutive frames of ground wire images, and use the optical flow vector to identify the foreground area closer to the camera and the background area farther away from the camera in the image;

[0010] Use the ground wire semantic segmentation dataset to train the semantic segmentation network model;

[0011] The semantic segmentation network model is used to segment the ground wire area from the foreground area of ​​the ground wire inspection image;

[0012] Use the ground wire defect detection dataset to train the ground wire defect detection model;

[0013] The defect area is detected from the ground wire area using the ground wire defect detection and recognition network model.

[0014] Furthermore, the optical flow vector is obtained from two consecutive frames of the ground wire image, and the foreground area closer to the camera and the background area farther from the camera in the image are distinguished by using the optical flow vector, and a dense inverse search image parallax distinguishing foreground and background algorithm is used. The specific steps of the dense inverse search image parallax distinguishing foreground and background algorithm include:

[0015] Create a Gaussian pyramid for the ground wire image and calculate the optical flow vector of each layer from top to bottom;

[0016] Each layer of the image pyramid is divided into blocks. The block size and overlap ratio are determined by the image resolution. An initial vector value is assigned to each block based on the GPS position of the drone when the image was taken. The displacement vector calculated by GPS positioning is used as the initial value of the top pyramid. The next layer of the pyramid substitutes the calculation results of the upper pyramid.

[0017] Use the LK reverse search algorithm to calculate the translation vector of each block to obtain the optical flow field on the sparse grid;

[0018] Use the densification algorithm to calculate the dense optical flow field on the sparse grid;

[0019] If the current pyramid is not the bottom layer, the optical flow calculation result is multiplied by 2 and downsampled to the same resolution as the lower layer. Then it is passed to the pyramid of this layer as the initial value and transferred to the pyramid block process. Otherwise, the result is output directly.

[0020] Calculate the gradient of the optical flow vector of each pixel, calculate the gradient mean of the entire image, and mark the pixels with gradients less than the mean as background areas, and the rest as foreground areas.

[0021] Furthermore, the semantic segmentation network model is used to segment the ground wire area from the foreground area of ​​the ground wire inspection image, and the foreground area ground wire semantic segmentation algorithm is used. The foreground area ground wire semantic segmentation algorithm uses the YOLACT model as the baseline semantic segmentation network model. In terms of network structure, a semantic segmentation network model with multi-scale pooling and multi-level training is designed to improve the incomplete segmentation problem caused by the weakening of category distinguishability features while obtaining semantic information by the full convolutional neural network; in the input layer, a smaller scale input is used to downsample the original image so that the linear target features in the ground wire image can be preserved; when upsampling, a transposed convolution combined with bilinear interpolation is used to obtain a sparse feature map of the target attribute through transposed convolution, and at the same time, a dense feature map is obtained through bilinear interpolation to obtain deeper semantic features, so that the pixel segmentation result of the ground wire is more refined.

[0022] Furthermore, the ground wire defect detection and identification network model is used to detect the defect area from the ground wire area, and the ground wire area defect target detection algorithm is used. The ground wire area defect target detection algorithm selects a cascaded regional convolutional neural network as the baseline network model. In the process of selecting negative samples in the input layer, the negative samples are constrained to the ground wire area; in the feature extraction part, a spatial pyramid pooling module is added to perform multi-scale pooling on high-level features to increase the learning ability of the ground wire contour, and the underlying network is used to learn finer texture features of the ground wire; in the feature fusion part, a multi-scale feature fusion module is added. The multi-scale feature fusion module consists of a feature map fusion module and a de-aliasing module. The feature map fusion module fuses the multi-scale feature maps of different layers in the network, and then uses the de-redundant information module to perform a convolution operation on the fused feature map; in the model loss function, a focusing loss function is introduced to calculate the loss, and the shape of the cross entropy loss function is changed by adding a coefficient to the cross entropy loss function. At the same time, when calculating the loss function, a penalty term is added for defects with more samples, and the value of the penalty term is determined by a cross-validation method.

[0023] Furthermore, the focusing loss function is calculated as follows:

[0024]

[0025] y takes values ​​of 1 and -1 to represent foreground and background, and p takes values ​​of 0 to 1, indicating the probability of being predicted as foreground.

[0026] Furthermore, after the ground wire defect detection and recognition network model training is completed, the image is divided into blocks when detecting defects in the ground wire area. The block size is consistent with the network layer size, and the blocks should cover all ground wire semantic segmentation result areas, with an overlapping area of ​​5%-15% between blocks.

[0027] The beneficial effects of the present invention are:

[0028] The present invention proposes a method for identifying ground wire defects in power line inspection images. Compared with the current power line inspection defect detection and identification method using deep learning technology, the method uses the DIS optical flow method to divide the parallax of two consecutive images collected by a drone into foreground and background areas, and uses an image semantic segmentation network model to segment the ground wire area from the foreground area. The ground wire target detection and recognition network model is trained for the ground wire area, so that the ground wire target detection and recognition network model specifically learns the areas of defective ground wires and normal ground wires, filters out the influence of complex background, and realizes the detection and identification of small defective areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various other advantages and benefits of the present invention will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are provided for illustration purposes only and are not intended to limit the present invention. It is apparent that the drawings described below are merely some embodiments of the present invention, and those skilled in the art will be able to derive other drawings based on these drawings without inventive effort. The same reference numerals are used throughout the drawings to represent the same components.

[0030] In the attached figure:

[0031] Figure 1 Schematic diagram of the semantic segmentation process of the ground wire in the foreground area according to an embodiment of the present invention;

[0032] Figure 2 Schematic diagram of a spatial pyramid pooling module according to an embodiment of the present invention;

[0033] Figure 3 This is a result diagram of semantic segmentation of ground wires according to an embodiment of the present invention;

[0034] Figure 4 This is a diagram showing the detection and identification results of a ground wire defect according to an embodiment of the present invention;

[0035] Figure 5 This is a flowchart of a multi-level ground wire defect recognition method based on parallax-assisted semantic segmentation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will refer to the attached Figures 1 to 5 Specific embodiments of the present invention will now be described in greater detail. Although specific embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention may be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to facilitate a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0037] It should be noted that certain words are used in the specification and claims to refer to specific components. Those skilled in the art should understand that technicians may use different nouns to refer to the same component. This specification and claims do not use the difference in nouns as a way to distinguish components, but use the difference in the functions of the components as the criterion for distinction. As mentioned throughout the specification and claims, "including" or "comprising" is an open term, so it should be interpreted as "including but not limited to". The subsequent description of the specification is a preferred embodiment of the present invention, but the description is based on the general principles of the specification and is not intended to limit the scope of the invention. The scope of protection of the present invention shall be as defined in the attached claims.

[0038] To facilitate understanding of the embodiments of the present invention, further explanation will be given below using specific embodiments as examples in conjunction with the accompanying drawings, and the accompanying drawings do not constitute a limitation on the embodiments of the present invention.

[0039] like Figure 5 As shown, an embodiment of the present invention provides a multi-level ground wire defect recognition method based on parallax-assisted semantic segmentation, comprising the following steps:

[0040] 1) Perform pixel-level annotation on power line inspection conductor and ground wire images to construct a conductor and ground wire semantic segmentation dataset;

[0041] 2) Mark the power line inspection conductor and ground wire defect images with rectangular frames to construct a conductor and ground wire defect detection dataset;

[0042] 3) Obtain the optical flow vector for two consecutive frames of ground wire images, and use the optical flow vector to distinguish the foreground area closer to the camera and the background area farther away from the camera in the image;

[0043] 4) Using the ground wire semantic segmentation dataset to train the semantic segmentation network model;

[0044] 5) Using a semantic segmentation network model to segment the ground wire area from the foreground area of ​​the ground wire inspection image, the area is used as the input of the ground wire defect detection model;

[0045] 6) Using the ground wire defect detection dataset to train the ground wire defect detection model;

[0046] 7) Detect the defect area from the ground wire area using the ground wire defect detection model.

[0047] An embodiment of the present invention provides a multi-level ground wire defect recognition method based on disparity-assisted semantic segmentation, which includes three modules: a dense inverse search image disparity distinction foreground and background algorithm, a foreground area ground wire semantic segmentation algorithm, and a ground wire area defect target detection algorithm, which are described below respectively.

[0048] The dense inverse search image parallax algorithm for distinguishing foreground and background is described. It calculates the light loss from two consecutive ground wire images captured by the drone and divides the foreground and background areas. Specifically, it includes:

[0049] In order to consider real-time performance and calculation speed, the algorithm was optimized, retaining the reverse LK (Lukas-Kanade) search and densification steps. The relevant calculation formula is:

[0050]

[0051]

[0052]

[0053] Where x represents the center of the image, I t Represents the reference image, T represents the given template patch in the reference image, and u is the distortion vector we want to obtain. Minimizing the distortion vector means minimizing the sum of the squares of the differences between the template and the query position on the sub-window.

[0054] The specific algorithm process includes:

[0055] Create a Gaussian pyramid for the ground wire image and calculate the optical flow vector of each layer from top to bottom;

[0056] Each layer of the image pyramid is divided into blocks. The block size and overlap ratio are determined by the image resolution. An initial vector value is assigned to each block based on the GPS position of the drone when the image was taken. The displacement vector calculated by GPS positioning is used as the initial value of the top pyramid. The next layer of the pyramid substitutes the calculation results of the upper pyramid.

[0057] Use the LK (Lukas-Kanade) reverse search algorithm to calculate the translation vector of each block and obtain the optical flow field on the sparse grid;

[0058] Use the densification algorithm to calculate the dense optical flow field on the sparse grid;

[0059] If the current pyramid is not the bottom layer, the optical flow calculation result is multiplied by 2 and downsampled to the same resolution as the lower layer. Then it is passed to the pyramid at this layer as the initial value and transferred to the pyramid block process. Otherwise, the result is output directly.

[0060] The gradient of the optical flow vector for each pixel is calculated, and the mean gradient of the entire image is calculated. Pixels with gradients less than the mean gradient are marked as background, and the remaining area is marked as foreground. The foreground area is determined for the continuously captured images to be detected, and a 5*5 rectangular morphological structuring element is used to perform a morphological closing operation on the foreground area to obtain the ground wire segmentation area.

[0061] The algorithm flow for semantic segmentation of the foreground area ground wire is as follows: Figure 1 As shown in the figure, the YOLACT model is used as the baseline semantic segmentation network model. In terms of network structure, a semantic segmentation network model with multi-scale pooling and multi-level training is designed to improve the problem of incomplete segmentation caused by the weakening of category distinguishability features while obtaining semantic information in the full convolutional neural network; in the input layer, a smaller scale input is used to downsample the original image so that the linear target features in the ground conductor image can be preserved; when upsampling, a transposed convolution combined with bilinear interpolation is used to obtain a sparse feature map of the target attribute through transposed convolution, and a dense feature map is obtained through bilinear interpolation to obtain deeper semantic features, so that the pixel segmentation result of the ground conductor is more refined. The semantic segmentation result of the ground conductor is shown in the figure. Figure 3 shown.

[0062] The ground wire area defect target detection algorithm uses Cascade-RCNN (Cascaded Regional Convolutional Neural Network) as the baseline network model. Its characteristics are as follows: in the process of selecting negative samples in the input layer, the negative samples are constrained to the ground wire area. Since the original image resolution is too large, randomly selecting negative samples in the area will make the network learning too difficult. By constraining the negative samples to be in the ground wire area, the detection model can directly learn the difference between defective and non-defective ground wires, thereby improving the detection accuracy; in the feature extraction part, a spatial pyramid pooling module is added, such as Figure 2As shown in the figure, multi-scale pooling is performed on high-level features to increase the learning ability of the ground wire contour, and the underlying network is used to learn the finer texture features of the ground wire. In the feature fusion part, a multi-scale feature fusion module is added, which consists of a feature map fusion module and a de-aliasing module. The feature map fusion module fuses the multi-scale feature maps of different layers in the network, and then uses the de-redundant information module to perform a convolution operation on the fused feature maps. In the model loss function, the Focal Loss function is introduced to calculate the loss. By adding a coefficient to the cross entropy loss function, the shape of its curve is changed. At the same time, when calculating the loss function, a penalty term is added for defects with a large number of samples. The value of this penalty term can be determined by cross-validation. The Focal Loss calculation formula is as follows:

[0063]

[0064] y takes values ​​of 1 and -1 to represent foreground and background, and p takes values ​​of 0 to 1, indicating the probability of being predicted as foreground.

[0065] After the ground wire defect detection model is trained, it can be used on the ground wire defect sample images to be detected. Defects are detected and identified in the ground wire area. During the detection, the image is divided into blocks. The block size is consistent with the network layer size, and the blocks should cover all ground wire semantic segmentation result areas. There is a 5%-15% overlap between blocks. The detection and identification results of ground wire defects are as follows: Figure 4 shown.

[0066] The training process in this embodiment is as follows:

[0067] The open-source PyTorch framework was used, and the test environment was an NVIDIA A100 GPU. The operating system was Linux, CUDA version 10.1, and the compilation language was Python 3.7. The experiment used 1,000 images of ground wire data collected by drones, including 500 images with defects. All images were annotated at the pixel level and with rectangular boxes. The training and test sets were divided into a 9:1 ratio. Optical flow vectors were first used to distinguish the foreground and background regions in the image. The foreground region was then fed into a semantic segmentation network model for training to generate a ground wire region layer. This ground wire region map was then fed into an object detection network for training, and the detection model's results were measured using precision and recall. The Adam optimizer was used, and the learning rate was cosine annealing. The initial learning rate was 0.0001, and the model parameters from the last round were used as the final model.

[0068] The present invention limits the semantic segmentation of ground wires to the foreground area, reducing the false alarm of semantic segmentation. The results of the semantic segmentation of ground wires are subjected to block detection processing. On the one hand, it avoids the submergence of the detailed features of the ground wires by downsampling when the image is input into the detection network. On the other hand, subsequent detection can also use smaller network layer inputs, improving the detection ability of subtle defects. In addition, in order to force the subsequent detection recognition network model to only learn the difference between normal ground wires and defective ground wires, the regional constraint method is adopted in the training stage of the subsequent detection model. The negative samples are not randomly selected from the entire image, but are randomly selected from the segmented non-defective ground wire area to further improve the recognition accuracy of defects.

[0069] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. A multi-level ground wire defect recognition method based on parallax-assisted semantic segmentation, characterized in that: The following steps are involved: Perform pixel-level annotation on power line inspection conductor and ground wire images to construct a conductor and ground wire semantic segmentation dataset; The power line inspection ground wire defect images are annotated with rectangular frames to construct a ground wire defect detection dataset; Obtain the optical flow vector for two consecutive frames of ground wire images, and use the optical flow vector to identify the foreground area closer to the camera and the background area farther away from the camera in the image; Use the ground wire semantic segmentation dataset to train the semantic segmentation network model; The semantic segmentation network model is used to segment the ground wire area from the foreground area of ​​the ground wire inspection image; Use the ground wire defect detection dataset to train the ground wire defect detection model; Detecting a defective area from a ground conductor area using a ground conductor defect detection and recognition network model; The optical flow vector is obtained from two consecutive frames of ground wire images, and the foreground area closer to the camera and the background area farther from the camera in the image are distinguished by using the optical flow vector. A dense inverse search image parallax distinguishing foreground and background algorithm is used. The specific steps of the dense inverse search image parallax distinguishing foreground and background algorithm include: Create a Gaussian pyramid for the ground wire image and calculate the optical flow vector of each layer from top to bottom; Each layer of the image pyramid is divided into blocks. The block size and overlap ratio are determined by the image resolution. An initial vector value is assigned to each block based on the GPS position of the drone when the image was taken. The displacement vector calculated by GPS positioning is used as the initial value of the top pyramid. The next layer of the pyramid substitutes the calculation results of the upper pyramid. Use the LK reverse search algorithm to calculate the translation vector of each block to obtain the optical flow field on the sparse grid; Use the densification algorithm to calculate the dense optical flow field on the sparse grid; If the current pyramid is not the bottom layer, the optical flow calculation result is multiplied by 2 and downsampled to the same resolution as the lower layer. Then it is passed to the pyramid of this layer as the initial value and transferred to the pyramid block process. Otherwise, the result is output directly. Calculate the gradient of the optical flow vector of each pixel, calculate the gradient mean of the entire image, and mark the pixels with a gradient less than the mean as the background area, and the rest as the foreground area; The method uses a semantic segmentation network model to segment the ground wire area from the foreground area of ​​the ground wire inspection image, and adopts a foreground area ground wire semantic segmentation algorithm. The foreground area ground wire semantic segmentation algorithm uses the YOLACT model as the baseline semantic segmentation network model. In terms of network structure, a semantic segmentation network model with multi-scale pooling and multi-level training is designed to improve the incomplete segmentation problem caused by the weakening of category distinguishability features while obtaining semantic information by the full convolutional neural network; in the input layer, a small-scale input is used to downsample the original image so that the linear target features in the ground wire image can be preserved; when upsampling, a transposed convolution combined with bilinear interpolation is adopted to obtain a sparse feature map of the target attribute through transposed convolution, and at the same time obtain a dense feature map through bilinear interpolation, so as to obtain deeper semantic features and make the pixel segmentation result of the ground wire more refined.

2. The method according to claim 1, characterized in that The ground wire defect detection and recognition network model is used to detect defect areas from the ground wire area, and a ground wire area defect target detection algorithm is used. The ground wire area defect target detection algorithm uses a cascaded regional convolutional neural network as a baseline network model. During the selection of negative samples in the input layer, the negative samples are constrained to the ground wire area. In the feature extraction part, a spatial pyramid pooling module is added to perform multi-scale pooling on high-level features to increase the learning ability of the ground wire contour, and the underlying network is used to learn more detailed texture features of the ground wire. feature In the fusion part, a multi-scale feature fusion module is added. The multi-scale feature fusion module consists of a feature map fusion module and a de-aliasing module. The feature map fusion module fuses the multi-scale feature maps of different layers in the network, and then uses the de-redundant information module to perform a convolution operation on the fused feature maps. In the model loss function, a focus loss function is introduced to calculate the loss, and the shape of the cross-entropy loss function is changed by adding a coefficient to the cross-entropy loss function. At the same time, when calculating the loss function, a penalty term is added for the defect of too many samples, and the value of the penalty term is determined by the cross-validation method.

3. The method according to claim 2, characterized in that The focusing loss function is calculated as follows: ； y takes 1 and -1 to represent foreground and background, and p takes a value from 0 to 1, indicating the probability of predicting it as foreground.

4. The method according to claim 1, wherein After the ground wire defect detection and recognition network model is trained, the image is divided into blocks when detecting defects in the ground wire area. The block size is consistent with the network layer size, and the blocks should cover all ground wire semantic segmentation result areas, with an overlapping area of ​​5%-15% between blocks.

Images