An overhead transmission line pin missing detection method, medium and system

Abstract

The present invention discloses a method, medium and system for detecting missing pins in an overhead transmission line, including: obtaining a plurality of first images of the overhead transmission line, where the overhead transmission line in the first images has pins; annotating the positions of the pins and whether the pins are missing in the plurality of first images to obtain an annotation vector of a pin annotation box, and the pin annotation box is a rotatable rectangular box; forming an input sequence by arranging the annotation vectors of the first images along the direction of the transmission line in the first images; training a target detection model based on pixel2seq using the first images and the input sequences of the first images to obtain a trained target detection model based on pixel2seq; collecting second images of the overhead transmission line in real time; after inputting the second images into the trained target detection model, outputting a prediction box of the pins, and the prediction box contains information about the position of the pins and whether the pins are missing. The present invention greatly improves the detection performance.

Description

Technical Field

[0001] The present invention relates to the technical field of overhead transmission lines, and in particular, to a method, medium and system for detecting the absence of pins in an overhead transmission line. Background Art

[0002] Pins are key components for fixing equipment such as insulators on high-voltage transmission towers. The lines in the air are always in a vibrating state, which will cause the pins, the connecting parts of the insulators thereon, to vibrate repeatedly. The cumulative effect of these vibrations will bring the risk of pin detachment. Once the pins are detached, it may cause the high-voltage line to break and fall to the ground, resulting in power outages and even endangering the safety of the area near the high-voltage tower. Timely detection of missing pins can avoid serious accidents and huge economic losses. Overhead transmission lines are mostly deployed in harsh natural environments and have a long span, often dozens or even hundreds of kilometers. Compared with machine vision detection methods, traditional manual inspections require engineers to go to the site and use telescopes to monitor section by section, with low monitoring efficiency, monitoring accuracy and personnel safety. With the development of information technology and computer vision technology, on-site collection of overhead transmission lines using drones or tower base cameras has gradually replaced the traditional manual inspection method, and has the advantages of high efficiency, speed, reliability, low cost, and being unaffected by geographical location.

[0003] Common methods for detecting the absence of pins mainly fall into traditional image processing methods, machine learning methods, and deep learning methods based on convolutional neural networks. Traditional image processing methods mainly include three steps: image preprocessing, feature selection, and classification. Among them, image preprocessing is to make the collected images have the same image gray distribution under different imaging conditions to facilitate subsequent feature extraction; in the feature extraction part, manually designed feature extractors are mainly used, such as sift operator, hog operator, template matching, etc., for feature extraction; in the classification stage, discriminative features are used for target-background classification. Machine learning methods and deep learning methods mainly improve the two steps of feature extraction and classification in the traditional method. Among them, the improvement of feature extraction by deep learning methods has greatly improved the performance of object detection and classification.

[0004] At present, certain achievements have been made in the research on circuit inspection using drones or tower base cameras. Convolutional neural networks can be used to detect screw parts containing pins in a simple background, but there are background structures similar to pins on the transmission line that interfere with the detection. At the same time, most of the existing pin defect methods use pin data labeled with horizontal frames for detection. The labeling of horizontal frames contains a large amount of background areas, resulting in the detection performance of the model for screw parts containing pins in complex backgrounds or even under occlusion needing to be improved. Summary of the Invention

[0005] An embodiment of the present invention provides a method, medium, and system for detecting missing pins in an overhead transmission line to solve the problem of poor detection effect of missing pins in an overhead transmission line in the prior art.

[0006] In a first aspect, a method for detecting missing pins in an overhead transmission line is provided, including:

[0007] Obtain a plurality of first images of the overhead transmission line, where the overhead transmission line in the first images has pins;

[0008] Annotate the positions of the pins and whether the pins are missing in the plurality of first images to obtain an annotation vector of the pin annotation box, where the pin annotation box is a rotatable rectangular box;

[0009] For each of the first images, form an input sequence by arranging the annotation vector of the first image along the direction of the transmission line in the first image;

[0010] Use the first image and the input sequence of the first image to train a target detection model based on pixel2seq to obtain the trained target detection model based on pixel2seq;

[0011] Collect second images of the overhead transmission line in real time;

[0012] After inputting the second image into the trained target detection model based on pixel2seq, output a prediction box of the pin, where the prediction box contains information about the position of the pin and whether the pin is missing.

[0013] In a second aspect, a computer-readable storage medium is provided, on which computer program instructions are stored; when the computer program instructions are executed by a processor, the method for detecting missing pins in an overhead transmission line as described in the embodiment of the first aspect above is implemented.

[0014] In a third aspect, a system for detecting missing pins in an overhead transmission line is provided, including: the computer-readable storage medium as described in the embodiment of the second aspect above.

[0015] In this way, in the embodiment of the present invention, in the case where there is interference in the detection of the background structure similar to the pin on the transmission line, and in the scenario of detecting screw parts containing pins under a complex background or even occlusion, the method of the embodiment of the present invention can greatly improve the detection performance, making the detection result accurate and the detection speed relatively fast. Description of the Drawings

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments of the present invention. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0017] Figure 1 is a flowchart of the method for detecting the missing pins of the overhead transmission line in the embodiment of the present invention;

[0018] Figure 2 are photos of non-missing pins and missing pins;

[0019] Figure 3 is a prior schematic diagram of the pins in the embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of the training of the object detection model based on pixel2seq in the embodiment of the present invention. Detailed implementation manners

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0022] The embodiment of the present invention discloses a method for detecting the missing pins of an overhead transmission line. As Figure 2 shown, they are photos of non-missing pins and missing pins respectively. Specifically, as Figure 1 shown, the method includes the following steps:

[0023] Step S1: Obtain a plurality of first images of the overhead transmission line.

[0024] Among them, there are pins on the overhead transmission line in the first image. The first image can be collected by a drone or a tower base camera.

[0025] Step S2: Mark the positions of the pins and whether the pins are missing in the plurality of first images to obtain the annotation vectors of the pin annotation frames.

[0026] In the language model, the predicted sequence is represented by an integer token sequence. When constructing the position and category of the target, the object detection model based on pixel2seq used in the embodiment of the present invention converts the continuous position information and angle information into integer tokens by means of discrete sampling.

[0027] The pin annotation box in the embodiment of the present invention is a rotatable rectangular box, that is, the target is described by the pixel coordinates and the angle in the image to represent a rotated pin target. Therefore, the annotation vector is . x represents the abscissa of the central pixel point of the pin annotation box, y represents the ordinate of the central pixel point of the pin annotation box. w represents the long side dimension of the central pixel point of the pin annotation box, h represents the short side dimension of the central pixel point of the pin annotation box. α represents the rotation angle of the pin annotation box. The rotation angle refers to the angle that the long side of the pin annotation box rotates clockwise to the positive direction of the horizontal axis of the coordinate system. p represents the state of whether the pin in the pin annotation box is missing. If the pin is missing, p it is 1. If the pin is not missing, p it is 0. For example, for a pin missing target with the central pixel point coordinates of (100, 200), the long and short side dimensions of (9, 5), and the rotation angle of 30°, the constructed annotation vector is . For a normal pin target (i.e., the pin is not missing) with the central pixel coordinates of (100, 200), the long and short side dimensions of (9, 5), and the angle of 30°, the constructed annotation vector is .

[0028] Compared with the remote sensing image field and the text recognition field, pin missing detection often needs to be calculated by a computing device mounted on a drone, and the computing power is much weaker than that of server devices. The convolutional-based rotated object detection method used in the remote sensing field often needs to perform non-maximum suppression (NMS) of the rotated prediction box to remove duplicate boxes after the prediction result. And the calculation of the intersection over union (IoU) of rotated objects is difficult to achieve parallel processing, which not only consumes computing power but also results in poor real-time performance, and there will be residual duplicate detection boxes due to inaccurate angle prediction, resulting in poor accuracy. At the same time, in order to obtain accurate angle prediction, the convolutional-based rotated object detection model often uses a rotated region of interest pooling layer to correct the features. Although this operation effectively improves feature alignment, it further consumes video memory and computing power.

[0029] Therefore, compared with the prior art, the embodiment of the present invention uses a rotatable rectangular box to annotate pin data and adds the prediction of angle information in the following training process, which can solve the problems that the horizontal annotation box in the prior art contains more background, limits the detection accuracy, and the convolutional neural network extracts rotated features with low efficiency and the post-processing operation of rotation in dense prediction has poor effect and is time-consuming, and improves the accuracy of pin missing detection.

[0030] Preferably, after step S2, the pin target data can be amplified first, specifically as follows:

[0031] Perform operations such as randomly scaling a number of first images (obtaining images of different sizes to simulate shooting distances and different camera focal lengths), cropping, and grayscale change (simulating the influence of light). In the prior art, the entire annotation sequence constructed from multiple targets in an image is disordered: the sequences of each target are randomly combined together, and an end marker is added at the end; the model trained with such a constructed training sequence has poor generalization; for example, in actual situations, when the model predicts the annotation sequence, the end identifier appears relatively late due to interference from similar targets; when encountering a target that is difficult to detect, the model prediction will prematurely appear the end identifier. To solve the above problems, a noise-adding amplification method is adopted, that is, the sequence information of noise targets is added to the input sequence, and the output sequence is invalid at the corresponding positions of the noise target sequences. In addition, for the angle parameters in the rotating targets and the different shooting angles during UAV acquisition, the images are rotationally amplified to simulate the actual situation.

[0032] Since the amplification is performed on the annotated data, subsequent image data amplification does not require secondary annotation, and the target position information can be obtained automatically.

[0033] Step S3: For each first image, form an input sequence by arranging the annotation vectors of the first image along the direction of the transmission line in the first image.

[0034] Although the detection performance of natural scenes can be improved by the aforementioned single sequence amplification method, however, the target distribution in the remote sensing scene is quite different from that in the natural scene. For example, there may be dense vehicle targets in the remote sensing scene, or a very large bridge may appear in the same scene at the same time. The encoder-decoder structure of pixel2seq discards the artificially designed sliding window in the convolutional model and extracts targets from the features of the entire image, so some targets with dense local distributions will be missed. Specifically, there is such a problem in the pin detection task: in the images taken by UAV aerial photography, the pin targets are usually very small, the pin targets do not occupy a large area in the whole image, and there is a certain overlapping relationship between two targets. Randomly combining the target sequences means that the network model has to alternately predict at various positions in the whole image. Because the small pin targets cannot ensure the same close distance as in natural images, the distance between pins is very large compared to the size of the pins, which places high requirements on feature extraction and the decoder.

[0035] Therefore, to solve this potential problem, the embodiments of the present invention add artificial prior information for pin detection in this step: pins only appear on the transmission line, and in some local areas, the pin targets have a high appearance frequency, and this is added to sequence construction and enhancement. By adding this prior information, the object detection model based on pixel2seq can predict pins along the trend of the transmission line, rather than detecting pins from random areas in the entire image.

[0036] Specifically, pins will densely appear in some parts of the transmission line and sparsely appear in some areas. To make the model's prediction more regular and efficient, when constructing the sequence, link each target sequence along the trend of the transmission line, so that the model follows the distribution law when predicting the position information of the pin target, which can reduce the prediction difficulty of the model. In addition, there may be areas with sparse pin distribution and areas with dense pin distribution along the direction of the transmission line. To further reduce the prediction difficulty of the model for pins in the dense distribution area, randomly shuffle the targets in the dense area and allow the model to make predictions in any order.

[0037] Specifically, this step includes the following process:

[0038] (1) For each first image, divide multiple pin concentration areas according to the distribution of pins in the first image.

[0039] As Figure 3 shown, divide into three pin concentration areas a, b, and c according to the distribution of pins.

[0040] (2) According to a preset standard, judge whether each pin concentration area in the first image is a dense area.

[0041] Specifically, the preset standard can be set according to experience and actual situation. In the embodiments of the present invention, the preset standard is that if there are more than 9 pins in an area of 200m * 200m, then the pin concentration area is a dense area.

[0042] (3) If the pin concentration area is a dense area, randomly arrange the annotation vectors of the pins in the pin concentration area to form the sequence of the pin concentration area.

[0043] Specifically, for the sake of easy explanation, take the Figure 3 shown area a as a dense area. Therefore, the annotation vectors of the pins in area a can be randomly arranged.

[0044] (4) If the pin concentration area is not a dense area, arrange the annotation vectors of the pins in the pin concentration area along the trend of the transmission line in the first image to form the sequence of the pin concentration area.

[0045] Specifically, for the sake of convenience in explanation, let Figure 3 the b region and the c region shown in Figure 3 not be dense regions. Then, the annotation vectors of the pins in the b region and the c region need to be arranged according to the direction of the power transmission line. There is only one annotation vector 3 for the pin in the b region, and sorting is not involved. In this figure, the direction of the power transmission line is a - b - c (or c - b - a). Then, the annotation vectors of the two pins in the c region are arranged as annotation vector 4 - annotation vector 5 according to the direction of the power transmission line (or for the corresponding c - b - a direction, the sorting is annotation vector 5 - annotation vector 4).

[0046] (5) Arrange the sequences of all pin concentration regions in the first image along the direction of the power transmission line in the first image to obtain the input sequence.

[0047] Taking Figure 3 the direction of the power transmission line as a - b - c, the input sequence is: annotation vector 1 - annotation vector 2 (since the a region is randomly arranged, it can also be annotation vector 2 - annotation vector 1) - annotation vector 3 - annotation vector 4 - annotation vector 5.

[0048] Step S4: Use the first image and the input sequence of the first image to train the object detection model based on pixel2seq to obtain the trained object detection model based on pixel2seq.

[0049] Specifically, the object detection model based on pixel2seq sequentially includes: a feature extraction module, an encoding module, a decoding module, and a prediction head module.

[0050] Specifically, as Figure 4 shown, this step includes:

[0051] (1) After inputting the first image into the feature extraction module, a feature image is output.

[0052] Among them, the feature extraction module is a Resnet50 network. The Resnet50 network is a residual network with 50 layers, which is an existing technology and will not be elaborated here.

[0053] (2) After inputting the feature image into the encoding module, an encoding sequence is output.

[0054] Among them, the encoding module is a Deformable - DETR encoder. The Deformable - DETR encoder is composed of multiple deformable self - attention modules connected in sequence.

[0055] In the self-attention module of the existing technology's Transformer, there are only three embedding layers (key, query, and value). It needs to pay attention to the information of the entire sequence length, and the attention area is very large. There is no introduction of too much prior information about the attention area, resulting in the intersection of small target detection results and slower convergence of the model.

[0056] Compared with the self-attention module in the existing technology's Transformer, the deformable self-attention module adopted in the present invention not only has the three embedding layers of query, key, and value of the self-attention module, but also has an embedding layer for predicting deformable offsets. The three embedding layers of query, key, and value respectively convert the input feature sequence into a query sequence, a key sequence, and a value sequence. Among them, the query reflects the field information of the target, the key reflects the input field information, and the value is the input content information. Each element in the sequence performs the correlation operation of key and value with each other to obtain the mutual correlation, and multiplies the correlation information with the value to obtain the output target sequence. The input of the embedding layer for predicting deformable offsets is the query sequence, and it outputs the position information concerned by this query and the attention degree of the corresponding attention position. When the self-attention module is applied, each query only pays attention to the key at the deformable offset position, and the attention degree is also determined by both the correlation degree between the key and the query and the attention degree of the deformable offset.

[0057] The function of the deformable self-attention module is to establish the mutual connection of the input feature sequence and output the processed sequence. For the encoding module, the feature map of the input image (that is, the feature map output by the Resnet50 network) is used as the input sequence, and the sequence information is encoded through multiple deformable self-attention modules, and the key and value of the output encoded sequence are transmitted to the decoder.

[0058] (3) After inputting the key and value of the encoded sequence and the input sequence into the decoding module, a decoded sequence is output.

[0059] Among them, the decoding module is a Deformable-DETR decoder. The Deformable-DETR decoder is composed of a cross-attention module and multiple deformable self-attention modules connected in sequence. The deformable self-attention module is the same as the deformable self-attention module of the encoding module, and will not be elaborated here.

[0060] The encoded sequence output by the encoder is input into the key embedding layer and value embedding layer of the cross-attention module, outputting the key and value sequences. The input sequence is input into the query embedding layer of the cross-attention module, outputting the query sequence. Among them, the sequence length of the encoder part remains unchanged, while the sequence length of the decoder part is the same as the object query, and the remaining operations are the same as those of the self-attention module.

[0061] (4) Input the decoded sequence into the prediction head module, output the confidence vector, and calculate the calculated value of the loss function through the confidence vector.

[0062] The prediction head module is a feed-forward neural network, which is also a prior art and will not be elaborated here. This confidence vector is the predicted box of the pin. The predicted box contains the position of the pin and the information on whether the pin is missing, which is in the same form as the aforementioned annotation vector and will not be elaborated here.

[0063] All the confidence vectors form a confidence vector sequence . The confidence vector obtains the final output sequence through a discrete encoding method (for example, the categorical encoding only needs y = [0,1] to represent pin defects and y = [1,0] to represent non-defective pins).

[0064] The loss function used in training is the likelihood function of the confidence vector , which can be understood as the probability size of the network under the premise of knowing the previous confidence vectors in the known image and sequence. If the current sequence position is not the end position, then the true value of the probability P is 1, otherwise it is 0.

[0065] The optimization objective of training the network in the embodiment of the present invention is to meet the requirements of the following formula:

[0066] .

[0067] Among them, x represents the input image, y and represent the input sequence and the target sequence. L is the length of the target sequence. is the pre-allocated weight of the j th annotation in the sequence. Since the prediction of the angle is more important, the weight assignment for the angle annotation in the embodiment of the present invention is 2, and the weight assignment for the prediction of the remaining annotations is 1. This loss function enables the network to predict that the next annotation is the same as the target sequence when seeing the features of the previous predicted sequence and the image.

[0068] (5) Update the network parameters by backpropagating the calculated value of the loss function.

[0069] Finally, through the above process, the trained parameters are imported into the object detection model based on pixel2seq, and the trained object detection model based on pixel2seq can be obtained.

[0070] Step S5: Collect the second image of the overhead transmission line in real time.

[0071] Similarly, the second image can be collected by a drone or a tower base camera.

[0072] Step S6: After the second image is input into the trained object detection model based on pixel2seq, the prediction box of the pin is output.

[0073] Among them, the prediction box contains the position of the pin and the information on whether the pin is missing, which is in the same form as the aforementioned annotation vector and will not be elaborated here.

[0074] Therefore, the object detection model based on pix2seq can flexibly extract features, predict a unique prediction box for each object, and reduce the operation process of non-maximum suppression.

[0075] Through the above steps, it can be seen that different from the artificial integration of the prior knowledge of the distribution of the object to be detected in the current detection model of the convolutional neural network, the object detection model based on pix2seq simply converts object detection into a language modeling task conditional on pixel input, that is, the trained network model extracts the feature map and describes the object distribution information in the figure with serialized "language". Specifically, the network obtains the discriminative features of the object and the background through convolution or other feature extraction modules in the image field, and decodes the features through the decoder in the language model. The training target of the decoding changes from "a paragraph" in the language modeling task to the required object detection result, such as "the center coordinates of object 1, the size of object 1, the angle of object 1, the category of object 1,..., the center coordinates of object n, the size of object n, the angle of object n, the category of object n". Therefore, in the case where there is interference in the detection of the background structure similar to the pin on the transmission line, and in the scenario of detecting the screw part containing the pin under a complex background or even occlusion, the method of the embodiment of the present invention can greatly improve the detection performance.

[0076] The embodiment of the present invention also discloses a computer-readable storage medium, on which computer program instructions are stored; when the computer program instructions are executed by a processor, the method for detecting pin loss of an overhead transmission line as described in the above embodiment is implemented.

[0077] The embodiment of the present invention also discloses a system for detecting pin loss of an overhead transmission line, including: the computer-readable storage medium as described in the above embodiment.

[0078] In summary, in the embodiments of the present invention, in the case where there is interference detection of background structures similar to pins on a transmission line, and in the scenario of detecting screw parts containing pins under a complex background or even occlusion, the method of the embodiments of the present invention can greatly improve the detection performance, making the detection result accurate and the detection speed relatively fast.

[0079] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for detecting the missing pins of an overhead transmission line, characterized in that, Including: Obtaining a plurality of first images of an overhead transmission line, wherein there are pins on the overhead transmission line in the first images; Annotating the positions of the pins and whether the pins are missing in the plurality of first images to obtain an annotation vector of a pin annotation box, wherein the pin annotation box is a rotatable rectangular box; For each of the first images, forming an input sequence by arranging the annotation vector of the first image along the direction of the transmission line in the first image; Training a target detection model based on pixel2seq using the first image and the input sequence of the first image to obtain the trained target detection model based on pixel2seq; Real-time collecting a second image of the overhead transmission line; After inputting the second image into the trained target detection model based on pixel2seq, outputting a prediction box of the pin, wherein the prediction box contains information about the position of the pin and whether the pin is missing; The step of forming an input sequence by arranging the annotation vector of the first image along the direction of the transmission line in the first image includes: For each of the first images, dividing a plurality of pin concentration regions according to the distribution of pins in the first image; Judging whether each of the pin concentration regions in the first image is a dense region according to a preset standard; If the pin concentration region is a dense region, randomly arranging the annotation vectors of the pins in the pin concentration region to form a sequence of the pin concentration region; If the pin concentration region is not a dense region, arranging the annotation vectors of the pins in the pin concentration region along the direction of the transmission line in the first image to form a sequence of the pin concentration region; Arranging the sequences of all the pin concentration regions in the first image along the direction of the transmission line in the first image to obtain an input sequence.

2. The method for detecting the missing pins of an overhead transmission line according to claim 1, characterized in that, The target detection model based on pixel2seq sequentially includes: a feature extraction module, an encoding module, a decoding module, and a prediction head module, wherein the feature extraction module is a Resnet50 network, the encoding module is a Deformable-DETR encoder, the decoding module is a Deformable-DETR decoder, and the prediction head module is a feed-forward neural network.

3. The method for detecting the missing pins of the overhead transmission line according to claim 2, wherein: The Deformable-DETR encoder is composed of a plurality of deformable self-attention modules connected in sequence.

4. The method for detecting the missing pins of an overhead transmission line according to claim 2, wherein: The Deformable-DETR decoder is composed of a cross-attention module and a plurality of deformable self-attention modules connected in sequence.

5. The method for detecting the missing pins of an overhead transmission line according to claim 4, wherein: The encoded sequence output by the encoder is input into the key embedding layer and the value embedding layer of the cross-attention module, and the input sequence is input into the query embedding layer of the cross-attention module.

6. The method for detecting the missing pins of an overhead transmission line according to claim 1, wherein: The loss function used in the training is the likelihood function of the confidence vector.

7. The method for detecting the missing pins of an overhead transmission line according to claim 1, characterized in that: The annotation vector is , where x represents the abscissa of the central pixel point of the pin annotation box, y represents the ordinate of the central pixel point of the pin annotation box, w represents the long side dimension of the central pixel point of the pin annotation box, and h represents the short side dimension of the central pixel point of the pin annotation box, α represents the rotation angle of the pin annotation box, p represents the state of whether the pin in the pin annotation box is missing. Among them, if the pin is missing, p is 1, and if the pin is not missing, p is 0.

8. A computer-readable storage medium, characterized in that: Computer program instructions are stored on the computer-readable storage medium; when the computer program instructions are executed by a processor, the method for detecting pin loss of an overhead transmission line according to any one of claims 1 to 7 is implemented.

9. An overhead transmission line pin missing detection system, characterized in that Including: The computer-readable storage medium according to claim 8.

Images