Visible light image defect detection method and device for power transmission equipment, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a power transmission equipment visible light image defect detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a visible light image of to-be-detected power transmission equipment, inputting a component detection network, a classification network and a defect detection network, and carrying out the processing, the component detection network is used for detecting a component area in the power transmission equipment, cutting and zooming the detected component area to obtain a cut component image, and detecting a first defect; the classification network is used for classifying the cut part image and identifying a second defect; the defect detection network is used for carrying out defect detection on the cut part image and identifying a third defect; according to an output result of each network, displaying a defect detection result, the defect detection result including a defect position and a defect category marked in the visible light image; according to the method, the network design structure is simple, the defect detection precision is improved, and customization of defect detection categories is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of transmission equipment defect detection, and particularly to a method, device, electronic device and storage medium for visible light image defect detection of transmission equipment. Background Art

[0002] With the rapid development of the power industry, the inspection work of transmission equipment has become increasingly important. Traditional defect detection of transmission equipment mainly relies on manual inspection, but this method has many problems, such as large workload, low efficiency, high susceptibility to environmental impact, and high risk. In recent years, with the development of drone technology and image recognition technology, more and more research has begun to attempt to use visible light images taken by drones for automatic detection of transmission equipment defects.

[0003] However, most of the current image recognition methods can only detect single-type defects and are relatively sensitive to factors such as shooting conditions and background environments, making it difficult to meet the requirements for full-category and multi-type defect detection in actual power grid operation and maintenance. Summary of the Invention

[0004] The main object of the present invention is to provide a method, device, electronic device and storage medium for visible light image defect detection of transmission equipment, with a simple network design structure, which can improve the defect detection accuracy and facilitate the customization of defect detection categories.

[0005] To achieve the above object, in the first aspect of the present application, a method for visible light image defect detection of transmission equipment is provided, and the method includes:

[0006] Obtain a visible light image of the transmission equipment to be detected;

[0007] Input the visible light image into a component detection network, a classification network and a defect detection network in sequence for processing, where:

[0008] The component detection network is used to detect the component areas in the transmission equipment, cut and scale the detected component areas to obtain the cut component images, and is used to detect the first defect, where the area of the first defect is larger than a preset area;

[0009] The classification network is used to classify the cut component images and identify the second defect, where the area of the second defect is within a preset area range;

[0010] The defect detection network is used to perform defect detection on the cut component images and identify the third defect;

[0011] According to the output results of the component detection network, classification network, and defect detection network, display the defect detection results, where the defect detection results include the defect positions and defect categories marked in the visible light image, and the defect categories include the first defect, the second defect, and the third defect.

[0012] Optionally, the component detection network includes a picture feature extraction module, a feature fusion module, and a target detection head, where:

[0013] The picture feature extraction module is used to extract the features of the input picture, the feature fusion module is used to fuse the extracted features, and the target detection head is used to detect the component areas and the first defect in the power transmission equipment.

[0014] Optionally, the picture feature extraction module includes a convolutional neural network and a series of combination modules, and the combination modules include a depthwise separable convolutional network, a transformer network, a convolutional neural network, and a residual connection;

[0015] In the feature fusion module, upsampling and downsampling are performed. The upsampling is used to enlarge the low-resolution feature map and splice it with the high-resolution feature map, and then perform feature fusion and dimensionality reduction; the downsampling is used to shrink the high-resolution feature map and splice it with the low-resolution feature map, and then perform feature fusion and dimensionality reduction;

[0016] The target detection head is used to decouple the class detection and position detection, obtain the class detection result and the position detection result, and finally determine the position of the target box by using the method of taking the expectation to obtain the actual offset.

[0017] Optionally, the classification network includes a feature extraction module and a classification head. The structure of the feature extraction module is the same as that of the component detection network. The classification head processes the features through depthwise separable convolution combination and a fully connected layer, and finally normalizes the class prediction result through the Softmax function to obtain the probability of each class that is non-negative and sums to 1.

[0018] Optionally, the structure, parameter initialization, and training method of the defect detection network are the same as those of the component detection network.

[0019] Optionally, the displaying the defect detection results according to the output results of the component detection network, classification network, and defect detection network includes:

[0020] Frame the first defect in the original image according to the original coordinates and label the defect type;

[0021] Frame the second defect in the original image according to the coordinates of the detected image in the component detection network and label the defect category;

[0022] Based on the coordinates of the third defect in the cut image and the coordinates of the cut image in the original image, trace back to obtain the coordinates in the original image, and accordingly frame the third defect in the original image and label the defect category.

[0023] Optionally, the method further includes:

[0024] Specify the defect categories to be detected according to the requirements of the power supply bureau, and realize the customization of defect detection categories by setting the defect categories to be detected in the component detection network, the classification network and the defect detection network.

[0025] The second aspect of the present application provides a visible light image defect detection device for transmission equipment, including:

[0026] An acquisition module, configured to acquire a visible light image of a transmission equipment to be detected;

[0027] A component detection network module, configured to process the visible light image, detect the component area in the transmission equipment, cut and scale the detected component area to obtain a cut component image, and detect a first defect, where the area of the first defect is greater than a preset area;

[0028] A classification network module, configured to classify the cut component image and identify a second defect, where the area of the second defect is within a preset area range;

[0029] A defect detection module, configured to perform defect detection on the cut component image and identify a third defect;

[0030] A result display network module, configured to display the defect detection result according to the output results of the component detection network, the classification network and the defect detection network, where the defect detection result includes the defect positions and defect categories marked in the visible light image, and the defect categories include the first defect, the second defect and the third defect.

[0031] The third aspect of the present application provides an electronic device, including a memory and a processor. When the computer program stored in the memory is executed by the processor, the processor is caused to execute the steps of the first aspect and any one of its possible implementation manners.

[0032] The fourth aspect of the present application provides a computer-readable storage medium, storing a computer program, and when the computer program is executed by a processor, the processor is caused to execute each step in the method described in the first aspect.

[0033] The present application provides a method, device, electronic device and storage medium for defect detection of visible light images of power transmission equipment. By obtaining the visible light image of the power transmission equipment to be detected; inputting the visible light image into a component detection network, a classification network and a defect detection network in sequence for processing, where: the component detection network is used to detect the component areas in the power transmission equipment, cut and scale the detected component areas to obtain the cut component images, and is used to detect the first defect, the area of the first defect being larger than a preset area; the classification network is used to classify the cut component images and identify the second defect, the area of the second defect being within a preset area range; the defect detection network is used to detect defects in the cut component images and identify the third defect; according to the output results of the component detection network, the classification network and the defect detection network, display the defect detection results, the defect detection results including the defect positions and defect categories marked in the visible light image, the defect categories including the first defect, the second defect and the third defect; the network design structure is simple, while greatly improving the defect detection accuracy, it covers the intelligent identification of most power transmission equipment defects, and at the same time the network can increase or decrease the detection or classification categories according to the requirements of the power supply bureau to achieve customization of defect detection categories, and has high engineering practical value. Description of the Drawings

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings.

[0035] Among them:

[0036] Figure 1 It is a schematic flowchart of a method for defect detection of visible light images of power transmission equipment provided by an embodiment of the present application;

[0037] Figure 2 It is a schematic flowchart of a neural network architecture for defect detection of visible light images of power transmission equipment provided by an embodiment of the present application;

[0038] Figure 3 It is a schematic structural diagram of a device for defect detection of visible light images of power transmission equipment provided by an embodiment of the present application;

[0039] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0040] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application.

[0041] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned accompanying drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0042] Referring to "embodiment" in this context means that a specific feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0043] The embodiments of this application will be described below in conjunction with the accompanying drawings in the embodiments of this application.

[0044] Please refer to Figure 1 , which is a schematic flowchart of a method for detecting visible light image defects of a power transmission device provided by an embodiment of this application. As Figure 1 shown, the method includes:

[0045] 101. Obtain a visible light image of the power transmission device to be detected;

[0046] 102. Input the above visible light image into a component detection network, a classification network, and a defect detection network in sequence for processing, where: the above component detection network is used to detect the component area in the power transmission device, cut and scale the detected component area to obtain a cut component image, and is used to detect a first defect, and the area of the first defect is greater than a preset area; the above classification network is used to classify the above cut component image to identify a second defect, and the area of the second defect is within a preset area range; the above defect detection network is used to detect defects in the above cut component image to identify a third defect;

[0047] 103. According to the output results of the above component detection network, classification network, and defect detection network, display the defect detection results. The above defect detection results include the defect positions and defect categories marked in the above visible light image. The above defect categories include the above first defect, the above second defect, and the above third defect.

[0048] In an embodiment of the present application, the execution subject of the method may be a visible light image defect detection device for transmission equipment, and in practical applications, it may specifically be an electronic device.

[0049] The above visible light image may be an image obtained through drone inspection.

[0050] The method in the embodiment of the present application mainly consists of four parts, namely, a component detection network, a classification network, a defect detection network, and result display. Among them, the component detection network is responsible for detecting, cutting, and scaling the component areas of the pictures, thereby eliminating the influence of shooting methods and complex backgrounds and increasing the proportion of defects in the pictures; the classification network is used to detect defects with larger areas; the defect detection network is used to detect defects with relatively fixed sizes; and result display is responsible for marking the detected defect results in the original image and saving the results. The network design structure is simple. While greatly improving the defect detection accuracy, it covers the intelligent recognition of most transmission equipment defects. At the same time, the four networks can increase or decrease the detection or classification categories according to the needs of the power supply bureau to achieve customization of defect detection categories, and has high engineering practical value.

[0051] Next, the four parts involved will be specifically introduced.

[0052] I. Component Detection Network

[0053] In an optional implementation manner, the above component detection network includes a picture feature extraction module, a feature fusion module, and a target detection head, where:

[0054] The above picture feature extraction module is used to extract the features of the input picture, the above feature fusion module is used to fuse the extracted features, and the above target detection head is used to detect the component areas in the above transmission equipment and the above first defect.

[0055] The component detection network in the embodiment of the present application may include a picture feature extraction module, a feature fusion module, and a target detection head. The detection content includes all power components that may have defects, such as insulators, lightning arresters, lightning rods, hanging plates, hooks, suspension clamps, strain clamps, dampers, spacer dampers, as well as defects with larger sizes that are suitable for direct detection, such as bird nests, vines winding around the tower base, and tower collapse.

[0056] Further optionally, the above-mentioned picture feature extraction module includes a convolutional neural network and a series of combination modules, and the above-mentioned combination modules include a depthwise separable convolutional network, a transformer network, a convolutional neural network, and a residual connection;

[0057] In the above-mentioned feature fusion module, upsampling and downsampling are performed. The upsampling is used to enlarge the low-resolution feature map and splice it with the high-resolution feature map, and then perform feature fusion and dimensionality reduction; the downsampling is used to reduce the high-resolution feature map and splice it with the low-resolution feature map, and then perform feature fusion and dimensionality reduction;

[0058] The above-mentioned object detection head is used to decouple the category detection and the location detection, obtain the category detection result and the location detection result, and finally obtain the actual offset by taking the expectation to determine the position of the target box.

[0059] Specifically, the above-mentioned feature extraction module can be composed of a convolutional neural network and a series of combination modules. The convolutional neural network reduces the length and width of the input picture to half of the original, and changes the number of feature channels to 32. The combination module includes a depthwise separable convolutional network, a transformer network, a convolutional neural network, and a residual connection. Among them, the depthwise separable convolutional network extracts the local features of the image with a relatively small number of parameters, the transformer network is used to extract the global features of the image, the residual connection concatenates the original input and the output of the transformer together, and the convolutional neural network performs fusion, halves the feature size, and doubles the number of feature channels on the above-mentioned concatenated result. Assuming the original picture has 3 channels, a height of H, and a depth of D, after being processed by the feature extraction module, 32x1 / 2Hx1 / 2D (convolution 1), 64x1 / 4Hx1 / 4D (combination module 1), 128x1 / 8Hx1 / 8D (combination module 2), 256x1 / 16Hx1 / 16D (combination module 3), 512x1 / 32Hx1 / 32D (combination module 4) will be obtained. After each size reduction, feature extraction and fusion can be performed through 0-N inverted residuals, C2F, C3K2 and other structures.

[0060] The C3 module mentioned in the embodiment of the present application is composed of multiple Bottleneck Blocks, and connects the feature maps of different layers through a Concatenation (Concat) operation. This structure can capture features of different scales and enhance the feature extraction ability of the model.

[0061] The C2F module mentioned in the embodiments of this application includes a Split operation that divides the input feature map into two parts. One part directly passes through the Bottleneck Block, and the other part passes through multiple Bottleneck Blocks and then undergoes a Concat operation with the first part.

[0062] This structure allows the model to process features on different paths, which may help improve the diversity of features.

[0063] The C3K2 module mentioned in the embodiments of this application is stacked by multiple C3 modules, and each C3 module is connected through a Concat operation. This structure can further enhance the feature extraction ability of the model while maintaining computational efficiency.

[0064] In one implementation, the above-mentioned feature fusion module is divided into an upsampling process and a downsampling process. Among them:

[0065] In the upsampling process, the 512x1 / 32Hx1 / 32D feature is first passed through the torch.nn.ConvTranspose2d() function to obtain a 256x1 / 16Hx1 / 16D feature. This feature is concatenated with the feature of the same size obtained in the feature extraction module to obtain a 512x1 / 16Hx1 / 16D feature. This feature undergoes feature fusion and dimensionality reduction through C2F or C3K2 to obtain a 256x1 / 16Hx1 / 16D feature. Subsequently, the above upsampling, concatenation, and feature fusion and dimensionality reduction operations are repeated to obtain 128x1 / 8Hx1 / 8D and 64x1 / 4Hx1 / 4D features respectively. In the downsampling process, the 64x1 / 4Hx1 / 4D feature obtained by upsampling undergoes feature fusion through C2F or C3K2 to obtain a 64x1 / 4Hx1 / 4D feature. This feature is downsampled by a two-dimensional convolution with a stride of 2 to obtain a 128x1 / 8Hx1 / 8D feature. This feature is concatenated with the feature of the same size obtained by the above upsampling to obtain a 256x1 / 8Hx1 / 8D feature. Subsequently, it undergoes feature fusion and dimensionality reduction operations through C2F or C3K2 to obtain a 128x1 / 8Hx1 / 8D feature. Subsequently, the above downsampling, concatenation, and feature fusion and dimensionality reduction operations are repeated to obtain 256x1 / 16Hx1 / 16D and 512x1 / 32Hx1 / 32D features respectively.

[0066] In one implementation, the above-mentioned object detection head decouples class detection and location detection and performs them separately. Suppose the number of classes to be detected is N. The features obtained in the feature fusion module are respectively used for class detection and location detection. For example, for a feature of 512x1 / 32Hx1 / 32D, when performing class detection, it first passes through a convolution with a convolution kernel of 3x3, a stride of 1, a padding of 1, and a group of 512 to obtain a feature of 512x1 / 32Hx1 / 32D. Subsequently, it passes through a convolution with a convolution kernel of 1 and an output channel for detecting class N to obtain a feature of Nx1 / 32Hx1 / 32D. Then, it passes through a convolution with a convolution kernel of 3x3, a stride of 1, a padding of 1, and a group of N, and a convolution with a convolution kernel of 1 and an unchanged output channel, and finally obtains a feature of Nx1 / 32Hx1 / 32D. When performing location detection, it first passes through a convolution with a convolution kernel of 3x3, a stride of 1, a padding of 1, and a group of 512 to obtain a feature of 512x1 / 32Hx1 / 32D. Subsequently, it passes through a convolution with a convolution kernel of 1 and an output channel of 4xM for detection to obtain a feature of 4Mx1 / 32Hx1 / 32D. Then, it passes through a convolution with a convolution kernel of 3x3, a stride of 1, a padding of 1, and a group of 4M, and a convolution with a convolution kernel of 1 and an unchanged output channel, and finally obtains a feature of 4Mx1 / 32Hx1 / 32D. Here, M is a hyperparameter, which is an integer and its value is greater than or equal to max(1 / 40H, 1 / 40D), representing the probability distribution of the offsets in the four coordinates (left, right, up, down) of the bounding box. Finally, the actual offset is obtained by taking the expectation. For example, if M is taken as 20, within a rectangle of 1 / 32Hx1 / 32D, each 1x1 small square corresponds to 4 groups of offset probability distributions. Suppose the probability distribution of the offset in the left direction of the coordinate is (p0, p1,..., p 19 ), then the expected value of the left offset E_left = p0x0 + p1x1 + p2x2...... + p 19 x19, indicating that the detected target box is offset to the left by E_left compared to this 1x1 small square. Finally, the upper-left coordinate of the obtained target box is (E_left, E_up), and the lower-right coordinate is (E_right, E_down).

[0067] In an alternative implementation, for the above-mentioned feature extraction module, feature fusion module, and detection head, after each convolution and depthwise separable convolution operation, it is also possible to perform normalization using one of the normalization methods such as Batch Normalization, Layer Normalization, Instance Normalization, etc., and perform a non-linear transformation operation using one of the activation functions such as Mish, Leaky_relu, SiLU, etc.

[0068] Specifically, when processing image data, these modules will adopt a series of techniques to improve the efficiency and accuracy of feature extraction. The following are some key techniques and steps adopted by these modules during operation:

[0069] Convolution and Depthwise Separable Convolution:

[0070] Convolution operation is used to extract local features of images.

[0071] Depthwise Separable Convolution is an efficient convolution operation that decomposes the standard convolution into Depthwise Convolution and Pointwise Convolution, thereby reducing the amount of computation and the number of parameters.

[0072] Normalization:

[0073] After the convolution operation, normalization techniques are adopted to adjust the output of the intermediate layer of the neural network to make its distribution more stable, accelerate the training process, and improve the generalization ability of the model.

[0074] Batch Normalization: Normalize each small batch of data.

[0075] Layer Normalization: Normalize all activation values in a single sample. Compared with batch normalization, it does not depend on the batch size.

[0076] Instance Normalization: Usually used in tasks such as style transfer, normalize each single channel of each sample.

[0077] Activation Function:

[0078] Activation functions are used to introduce non-linearity so that the neural network can learn and simulate more complex function mappings.

[0079] Mish: A self-gating activation function that can provide better performance and stability.

[0080] Leaky ReLU: An improved ReLU activation function that allows non-zero gradients for inputs less than zero, thereby alleviating the "dying" problem of ReLU.

[0081] SiLU (Sigmoid Linear Unit): Combines the sigmoid function and the linear function, and can automatically adjust the scale of the activation value.

[0082] The combined use of these technologies and steps enables the feature extraction module, the feature fusion module, and the detection head to effectively extract and process features from the image, thereby improving the accuracy and robustness of defect detection. Through these operations, the component detection network can better adapt to different shooting conditions and types of equipment defects, and achieve visible light image defect detection of all categories and customizable transmission equipment.

[0083] Further optionally, for the component detection network composed of a feature extraction module, a feature fusion module, and a detection head, the loss function includes a classification loss, a bounding box regression loss, and a distribution focal loss.

[0084] Specifically, the classification loss is the cross-entropy loss function, the bounding box loss function is one of the loss functions such as Smooth L1, CIoU, GIoU, DIoU, EIoU, etc., and the distribution focal loss function is:

[0085]

[0086] where y i is the true distribution (one-hot encoding), and p i is the probability distribution predicted by the model;

[0087] The final component network loss function is the weighted value of the above three loss functions, and the weighting coefficients are learnable parameters λ1, λ2, λ3.

[0088] In an optional implementation manner, insulators, lightning arresters, lightning rods, hanging plates, hooks, suspension clamps, strain clamps, vibration dampers, spacer dampers, as well as defects with larger sizes suitable for direct detection such as bird nests, vine entanglements on the tower base, and tower collapses in the machine patrol pictures of transmission lines can be manually or semi-automatically labeled to form training data. The component detection network can initialize the parameters using the Xavier initialization or He initialization method, and input all the training pictures into the network in sequence. The results are used to calculate the error with the true values, backpropagate to calculate the gradients, and update the model parameters. Finally, a trained and optimized detection network model is obtained.

[0089] During application, the trained model can be used to detect the UAV inspection pictures of transmission lines to obtain the categories and coordinates of the targets. For defects such as bird nests, vines winding around the tower base, and tower collapse, the defect categories and their coordinates are directly passed to the result display module. For components that may contain defects, such as insulators, lightning arresters, lightning rods, hanging plates, hooks, suspension clamps, strain clamps, dampers, and spacer dampers, the targets are cut out after their sizes are enlarged. The enlargement coefficient is γ, and they are saved in different folders according to the categories. For example, if the coordinates of the upper left corner and the lower right corner of a certain component are detected as (X1, Y1) and (X2, Y2) respectively, then the enlarged coordinates are (max(X1 - 1 / 2γ(X2 - X1), X1), max(Y1 - 1 / 2γ(Y2 - Y1), Y1)), (min(X2 + 1 / 2γ(X2 - X1), X2), min(Y2 + 1 / 2γ(Y2 - Y1), Y2)).

[0090] II. Classification Network

[0091] In an alternative embodiment, the classification network includes a feature extraction module and a classification head. The structure of the feature extraction module is the same as that of the component detection network. The classification head processes the features through depthwise separable convolution combinations and fully connected layers, and finally normalizes the category prediction results through the Softmax function to obtain the probabilities of each category that are non - negative and sum to 1.

[0092] Specifically, the classification network consists of a feature extraction module and a classification head. The structure of the feature extraction module can be the same as that of the component detection network. Finally, the features of 512x1 / 32Hx1 / 32D (combination module 4) enter the classification head. The classification head consists of two depthwise separable convolution combinations and one fully connected layer. Each depthwise separable convolution combination consists of a convolution with a kernel size of 3x3, padding of 1, stride of 1, and group equal to the input feature channel, and a convolution with a kernel size of 1, stride of 1, and output channels of channel / 2, resulting in features of size 128x1 / 32Hx1 / 32D. The above features are flattened into a one - dimensional vector using the flatten() function, and then enter the fully connected layer. The output feature size is the number of categories to be classified. Finally, it enters the Softmax to normalize the category prediction results, that is, to obtain the probabilities of each category that are non - negative, sum to 1, and each value represents a category.

[0093] In an alternative embodiment, the above classification network uses the cross - entropy function as the optimization function, and the normalization function and activation function are the same as those of the component detection network;

[0094] Optionally, the above classification network uses the component images obtained by the object detection network segmentation as input for model training and prediction, and only relevant components enter the classification network when they contain defects suitable for classification, such as anti-vibration hammers, hanging plates, hooks, pole number plates, insulators, suspension wire clamps, etc.;

[0095] In an optional embodiment, the classification categories of the above-mentioned classification network include but are not limited to: deformation of shockproof hammer, rust of shockproof hammer, normal shockproof hammer, rust of hanging plate, normal hanging plate, rust of hook, normal hook, deformation of pole number plate, falling of pole number plate, blurred handwriting on pole number plate, inverted pole number plate, normal pole number plate, dirty insulator, normal insulator, rust of suspension wire clamp, normal suspension wire clamp, and other categories; among which the category "others" can be images with component detection network recognition or segmentation errors, thereby reducing the error accumulation caused by component detection errors.

[0096] In an optional implementation, the above-mentioned classification network uses Xavier initialization or He initialization to initialize parameters, and all training images are input into the network in turn to obtain the predicted probability of each category. The actual label category of the image is 1, and the labels of other categories are 0. The prediction result and the actual label are used to calculate the error, the gradient is calculated by back propagation, and the model parameters are updated. Finally, the classification network training is completed to obtain the trained and optimized detection network model, which is used for defect recognition suitable for classification. Finally, the defect category and the position of the component in the image in the original image are transmitted to the result display network.

[0097] 3. Defect Detection Network

[0098] The network structure, parameter initialization, and training method of the defect detection network in the embodiment of the present application may be the same as those of the component detection network.

[0099] The above defect detection network uses the component images obtained by the target detection network segmentation as the input for model training and prediction. The images enter the defect detection network only when the relevant component categories contain defects suitable for target detection, such as insulators, hanging plates, hooks, suspension clamps, spacers, arresters, etc.

[0100] The defect categories of the above-mentioned defect detection network include but are not limited to insulator self-explosion, insulator flashover, insulator bird droppings or paint, R pin missing, R pin abnormality, R pin rust, bolt loosening, lightning arrester damage, and lightning arrester rust.

[0101] 4. Results display network

[0102] The result display network in the embodiment of the present application is used to frame the defect location in the original image and indicate the defect category, and output the detection results at the same time. The output content may include the name of the transmission line, tower number, voltage level, tower where the defect is located, defect category, number of defects, location coordinates of the defect in the image, etc.

[0103] Specifically, the above result display network can automatically read the latitude and longitude information of the original image, calculate the location where the image was taken and the coordinates of the transmission line tower by comparing with the pre-stored latitude and longitude of the transmission line tower, find the transmission line tower closest to the shooting location, and then determine the name of the transmission line, the tower number, and the voltage level. The formula for calculating the distance using latitude and longitude is as follows:

[0104]

[0105] distance=R·c

[0106] Where:

[0107] lat1 and lon1 are the latitude and longitude of the first point;

[0108] lat2 and lon2 are the latitude and longitude of the second point;

[0109] Δlat is the difference in latitude (lat2 - lat1);

[0110] Δlon is the difference in longitude (lon2 - lon1);

[0111] R is the radius of the earth (usually taken as 6371 km);

[0112] atan2 is the arctangent function.

[0113] In an alternative embodiment, based on the output results of the above component detection network, classification network, and defect detection network, the defect detection results are displayed, including:

[0114] Frame the first defect in the original image according to the original coordinates and label the defect type;

[0115] Frame the second defect in the original image according to the coordinates of the detected image in the above component detection network and label the defect category;

[0116] Based on the coordinates of the third defect in the cut image and the coordinates of the cut image in the original image, trace back to the coordinates in the original image, and accordingly frame the third defect in the original image and label the defect category.

[0117] Specifically, in the embodiment of the present application, the defects obtained by the component detection network can be directly framed and labeled with the defect types in the original coordinates on the original picture; for the defects obtained by the classification network, they are framed and labeled with the defect categories in the original picture according to the coordinates in the component detection network of the detected picture; for the defects obtained by the defect detection network, according to the coordinates of the defect in the cut picture and the coordinates of the cut picture in the original picture, the coordinates of the defect in the original picture are traced back, and based on this, the defect is framed in the original picture and the defect category is marked. Finally, the defects detected by the three networks are framed and labeled in the original picture. The obtained pictures can be automatically saved in the statistical folder where the pictures are read, and each defective picture and the defect information are written into a table.

[0118] In an alternative embodiment, the above method further includes:

[0119] Specifying the defect categories to be detected according to the requirements of the power supply bureau, and realizing the customization of defect detection categories by setting the defect categories to be detected in the above component detection network, the above classification network, and the above defect detection network.

[0120] In the method in the embodiment of the present application, in application, the defect categories to be detected can be specified according to the requirements of the power supply bureau. Only the defect or the power component where it is located needs to be detected by the component detection network, and according to the characteristics of the defect, the cut component picture is detected for defects using the classification network or the defect detection network. If a certain type of defect does not need to be detected, only the relevant detections need to be removed in the three networks, and the customization of defect detection categories can be realized.

[0121] Figure 2 It is a flowchart of a neural network architecture for visible light image defect detection of transmission equipment provided by an embodiment of the present application. The flowchart is divided into two main parts, which detect different types of defects respectively. The following is the description of each part in the figure:

[0122] The upper part: The detection process of larger defects such as bird nests and vines winding around the tower base

[0123] Picture input: Input the image of the transmission equipment to be detected.

[0124] Feature extraction network: Extract features from the input image.

[0125] Feature fusion network: Fuse the extracted features to enhance the feature expression ability.

[0126] Target detection head: Used to detect defect targets in the image.

[0127] Target shearing and scaling: Perform shearing and scaling processing on the detected defect targets.

[0128] Defect original image display: The detected defects are marked and displayed in the original image.

[0129] Lower part: Detection process for smaller defects such as insulator explosion and hardware corrosion

[0130] Image input: Input the image of the power transmission equipment to be detected.

[0131] Component detection network: specially used to detect specific components in power transmission equipment, such as insulators, hardware, etc.

[0132] Defect Detection Network: Further analysis of the detected parts to detect whether there are defects.

[0133] Classification head: classifies the detected defects.

[0134] Classification networks: perform more detailed classification of defects, such as distinguishing different types of corrosion or damage.

[0135] Intermediate results: Display the intermediate results of the classification process.

[0136] Defect output results: Output the final defect detection results.

[0137] Result display: The detected defects are marked and displayed in the original image.

[0138] Other notes

[0139] Hardware 1: including hanging plates, hooks, suspension wire clamps, etc.

[0140] Hardware 2: including hanging plates, hooks, suspension wire clamps, shock-absorbing hammers, pole number plates, etc.

[0141] Defect categories: including insulator self-explosion, insulator flashover, insulator bird droppings or paint, R pin missing, R pin abnormality, R pin rust, bolt loosening, lightning arrester damage, lightning arrester rust, etc.

[0142] This flowchart shows a comprehensive defect detection system proposed in an embodiment of the present application, which processes different types of defects through different network modules, thereby achieving accurate detection and classification of various defects in the image of the power transmission equipment.

[0143] Based on the description of the aforementioned method embodiment, an embodiment of the present application further provides a visible light image defect detection device for power transmission equipment.

[0144] Figure 3 This is a schematic diagram of the structure of a visible light image defect detection device for power transmission equipment provided in an embodiment of the present application. Figure 3 As shown, the visible light image defect detection device 300 for power transmission equipment includes:

[0145] An acquisition module 310, configured to acquire a visible light image of a power transmission device to be detected;

[0146] A component detection network module 320, configured to process the above-mentioned visible light image, detect component areas in the power transmission device, cut and scale the detected component areas to obtain a cut component image, and detect a first defect, where the area of the first defect is greater than a preset area;

[0147] A classification network module 330, configured to classify the above-mentioned cut component image to identify a second defect, where the area of the second defect is within a preset area range;

[0148] A defect detection module 340, configured to perform defect detection on the above-mentioned cut component image to identify a third defect;

[0149] A result display network module 350, configured to display a defect detection result according to the output results of the above-mentioned component detection network, classification network, and defect detection network, where the defect detection result includes the defect positions and defect categories marked in the above-mentioned visible light image, and the defect categories include the first defect, the second defect, and the third defect.

[0150] It can be understood that the relevant content of each module involved in Figure 3 has been described in detail in the foregoing method embodiments, and specifically, reference can be made to the content in the method embodiments; that is Figure 3 A visible light image defect detection device 300 for a power transmission device provided can execute any step in the Figure 1 or Figure 2 shown embodiments, which will not be elaborated here.

[0151] Based on the description of the foregoing method embodiments, an embodiment of the present application further provides an electronic device.

[0152] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As Figure 4 shown, the electronic device 400 includes a processor 401 and a memory 402. The memory 402 stores a computer program. When the computer program is executed by the processor 401, it will execute any step in the Figure 1 or Figure 2 shown embodiments. The electronic device 400 may further include an input / output device, etc. In a specific implementation manner, the electronic device may be a terminal device, etc.

[0153] In one embodiment, a computer-readable storage medium is further provided. The computer-readable storage medium stores a computer program. When the computer program is executed by the processor 401, the processor 401 is caused to execute any of the steps in the above method embodiments.

[0154] Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0155] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0156] The above-described embodiments merely represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for detecting defects in visible light images of power transmission equipment, characterized in that, Including: Obtain the visible light image of the power transmission equipment to be detected; Input the visible light image into a component detection network, a classification network, and a defect detection network in sequence for processing, where: The component detection network is used to detect the component areas in the power transmission equipment, cut and scale the detected component areas to obtain the cut component images, and is used to detect the first defect, the area of the first defect being larger than a preset area; The classification network is used to classify the cut component images and identify the second defect, the area of the second defect being within a preset area range; The defect detection network is used to perform defect detection on the cut component images and identify the third defect; According to the output results of the component detection network, the classification network, and the defect detection network, display the defect detection results, the defect detection results including the defect positions and defect categories marked in the visible light image, the defect categories including the first defect, the second defect, and the third defect.

2. The method for defect detection of visible light images of power transmission equipment according to claim 1, wherein, The component detection network includes a picture feature extraction module, a feature fusion module, and a target detection head, where: The picture feature extraction module is used to extract the features of the input picture, the feature fusion module is used to fuse the extracted features, and the target detection head is used to detect the component areas and the first defect in the power transmission equipment.

3. The method for defect detection of visible light images of power transmission equipment according to claim 2, wherein, The picture feature extraction module includes a convolutional neural network and a series of combination modules, the combination modules including a depthwise separable convolutional network, a transformer network, a convolutional neural network, and a residual connection; In the feature fusion module, upsampling and downsampling are performed. The upsampling is used to enlarge the low-resolution feature map and splice it with the high-resolution feature map, and then perform feature fusion and dimensionality reduction; The downsampling is used to shrink the high-resolution feature map and splice it with the low-resolution feature map, and then perform feature fusion and dimensionality reduction; The target detection head is used to decouple the class detection and the position detection, obtain the class detection result and the position detection result, and finally determine the position of the target box by taking the expectation to obtain the actual offset.

4. The method for defect detection of visible light images of power transmission equipment according to claim 3, characterized in that The classification network includes a feature extraction module and a classification head. The structure of the feature extraction module is the same as that of the component detection network. The classification head processes the features through a depthwise separable convolution combination and a fully connected layer, and finally normalizes the class prediction results through the Softmax function to obtain the probabilities of each class that are non-negative and sum to 1.

5. The method for detecting defects in visible light images of power transmission equipment according to claim 1, characterized in that The structure, parameter initialization, and training method of the defect detection network are the same as those of the component detection network.

6. The method for defect detection of visible light images of power transmission equipment according to claim 1, wherein, The displaying the defect detection results according to the output results of the component detection network, the classification network, and the defect detection network includes: Frame the first defect in the original image according to the original coordinates and label the defect type; Frame the second defect in the original image according to the coordinates in the component detection network of the detected image and label the defect category; Based on the coordinates of the third defect in the cut image and the coordinates of the cut image in the original image, trace back to obtain the coordinates in the original image, and accordingly frame the third defect in the original image and label the defect category.

7. The method for defect detection of visible light images of power transmission equipment according to claim 6, wherein, The method further includes: Specifying the defect categories to be detected according to the requirements of the power supply bureau, and realizing the customization of defect detection categories by setting the defect categories to be detected in the component detection network, the classification network, and the defect detection network.

8. A visible light image defect detection device for power transmission equipment, characterized in that, It includes: An acquisition module, configured to acquire a visible light image of a transmission device to be detected; A component detection network module, configured to process the visible light image, detect the component areas in the transmission device, cut and scale the detected component areas to obtain the cut component images, and detect the first defect, where the area of the first defect is greater than a preset area; A classification network module, configured to classify the cut component images and identify the second defect, where the area of the second defect is within a preset area range; A defect detection module, configured to perform defect detection on the cut component images and identify the third defect; A result display network module, configured to display the defect detection results according to the output results of the component detection network, the classification network, and the defect detection network, where the defect detection results include the defect positions and defect categories marked in the visible light image, and the defect categories include the first defect, the second defect, and the third defect.

9. An electronic device, characterized in that, It includes a memory and a processor. When the computer program stored in the memory is executed by the processor, the processor is caused to execute the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the processor is caused to execute the steps of the method according to any one of claims 1-7.

Images