Method and system for detecting bolts of power equipment based on unmanned aerial vehicle inspection

Abstract

The application discloses a power equipment bolt detection method and system based on unmanned aerial vehicle inspection, wherein the method comprises the following steps: firstly, constructing a bolt data set of normal bolts, detecting the normal bolts in the bolt database by using a target detection network to obtain a bolt detection result; and constructing a bottom plate component template database according to the distribution of the bolts in the power equipment; then, judging whether the bolt detection result picture matches the component template picture in the bottom plate component template database, if the matching is successful, rotating and aligning the bolt bottom plate in the bolt detection result picture with the component template picture; finally, calculating the intersection-over-union of the bolt component real frame and the detection frame in the component template picture, if the intersection-over-union is higher than a threshold value, it is indicated that the position is a normal bolt position, otherwise, the position is a missing bolt position. The method can improve the detection efficiency and effect without relying on defect samples.

Description

Technical Field

[0001] This invention relates to the field of power line inspection technology, and in particular to a method and system for detecting bolts in power equipment based on unmanned aerial vehicle (UAV) inspection. Background Technology

[0002] Bolts play a crucial role in the connection and fixation of power transmission lines. Exposed to harsh environments such as strong winds, heavy rain, lightning, and dust, they are prone to missing bolts, creating safety hazards and seriously damaging the transmission lines. Therefore, regular bolt inspections are necessary. The traditional method is manual inspection, where inspectors check each tower along the transmission line. However, due to the complex terrain, harsh climate, and high forest coverage along most sections of the transmission lines, manual inspection is labor-intensive, inefficient, and poses safety risks. Furthermore, limitations in spatial location and detection equipment make it difficult to achieve full coverage of the line in a short time, resulting in low inspection quality and accuracy.

[0003] With the development of drone technology, the flexibility, speed, and efficiency of drones enable the rapid acquisition of numerous clear images of bolts from different shooting angles at close range. This significantly improves inspection efficiency, reduces outdoor labor intensity, and avoids the safety hazards associated with manual inspections, resulting in better safety. However, manual image-by-image inspection still suffers from high labor intensity and low efficiency.

[0004] In existing technologies, the further processing of image data mainly falls into two categories: one is to use traditional image processing algorithms to find missing bolts in the image; the other is to use deep learning methods to automatically locate bolts by training a network. For deep learning object detection networks, a large number of training samples are required. In reality, bolt missing samples are scarce, and the background of missing bolt holes is complex, making direct training of deep learning networks unable to converge. Furthermore, using traditional image processing algorithms to locate missing bolt positions suffers from high complexity in manual feature design and poor versatility. Summary of the Invention

[0005] This invention provides a method and system for detecting bolts in power equipment based on unmanned aerial vehicle (UAV) inspection, which solves or at least partially solves the technical problems of over-reliance on defect samples and poor detection results in the prior art.

[0006] To address the aforementioned technical problems, the first aspect of this invention provides a method for bolt inspection of power equipment based on unmanned aerial vehicle (UAV) inspection, comprising:

[0007] A bolt dataset is constructed using normal bolts. An object detection network is then used to detect the normal bolts in the bolt dataset to obtain bolt detection results. The bolt detection results include the location information of the detection box, confidence level, and category.

[0008] Based on the distribution of bolts in power equipment, a base plate component template database is constructed, which contains template diagrams of different types of components.

[0009] Determine whether the bolt inspection result image matches the component template image in the base plate component template database. If the match is successful, rotate and align the bolt base plate in the bolt inspection result image with the component template image.

[0010] Calculate the intersection-union ratio (IUR) of the actual bounding box and the detection box of the bolt component in the component template diagram. If the IUR is higher than the threshold, it indicates the location of a normal bolt; otherwise, it indicates the location of a missing bolt.

[0011] In one implementation, the target detection network used is YOLOv5.

[0012] In one implementation, determining whether the bolt inspection result image matches the component template image in the base plate component template database includes:

[0013] Normalized multi-order rectangular features of the outer contour are extracted from bolt inspection result images and component template images respectively, and feature vectors with shape description are constructed.

[0014] The distribution difference between two eigenvectors is solved based on Wasserstein distance;

[0015] Whether a match is successful is determined based on whether the distribution difference is within the allowable range of the threshold.

[0016] In one embodiment, the bolt base plate in the bolt inspection result image is rotated and aligned with the component template drawing, including:

[0017] Find the feature corner points of the base plate edge contour in the bolt inspection result image and the corresponding component template image;

[0018] Based on the corresponding feature corner points, determine the positional relationship between the detection result image and the bottom plate edge contour in the component template image, and rotate and align the bolt bottom plate in the bolt detection result image with the component template image according to the positional relationship.

[0019] Based on the same inventive concept, a second aspect of the present invention provides a bolt inspection system for power equipment based on unmanned aerial vehicle (UAV) inspection, comprising:

[0020] The normal bolt detection module is used to construct a bolt dataset from normal bolts, and to use a target detection network to detect normal bolts in the bolt database to obtain bolt detection results. The bolt detection results include the location information of the detection box, confidence level, and category.

[0021] The base plate component template database construction module is used to build a base plate component template database based on the distribution of bolts in power equipment. The base plate component template database contains template diagrams of different types of components.

[0022] The template matching module is used to determine whether the bolt inspection result image matches the component template image in the base plate component template database. If the match is successful, the bolt base plate in the bolt inspection result image is rotated and aligned with the component template image.

[0023] The bolt detection module is used to calculate the intersection-union ratio (IUR) of the actual bolt component frame and the detection frame in the component template diagram. If the IUR is higher than the threshold, it indicates the location of a normal bolt; otherwise, it indicates the location of a missing bolt.

[0024] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in the first aspect.

[0025] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0026] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0027] This application provides a bolt detection method for power equipment based on UAV inspection. A bolt dataset of normal bolts is constructed, and a target detection network is used to detect these normal bolts to obtain bolt detection results. Because the bolt dataset is constructed from normal bolts, the dependence on defect samples is minimal. Based on the distribution of bolts in power equipment, a base plate component template database is constructed, containing template images of different types of components. This method fully utilizes the regularity of bolt distribution on power tower components, adaptively obtaining template information for the entire component under different shooting angles and lighting conditions. Thus, it indirectly locates missing bolts using normal bolts.

[0028] Furthermore, the target detection network is YOLOv5. YOLOv5 has a strong advantage in rapid model deployment and performs better when detecting small targets such as bolts, with higher accuracy, faster running speed, and cleaner detection results with almost no overlapping bounding boxes. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of a bolt inspection method for power equipment based on unmanned aerial vehicle (UAV) inspection, provided in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the YOLOv5 network involved in the target detection module in the method provided in this embodiment of the invention;

[0032] Figure 3 This is the process of matching templates for base plate components and judging defects in the method provided in the embodiments of the present invention.

[0033] Figure 4 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0035] This invention proposes a bolt detection method for power equipment based on UAV inspection. This method detects the location of missing bolts, thereby achieving integrity detection of the power equipment nuts. It employs a combination of YOLOv5 object detection and template-driven methods for bolt location and missing bolt detection. The YOLOv5 object detection network is used to locate the positions of normal bolts to obtain detection boxes, which are then matched with component template images in a base plate component template database to obtain information about missing bolts. This method can complete the detection using only normal bolts, without relying on defect samples. Furthermore, by matching the detection results images of normal bolts with the template database, it indirectly locates missing bolts.

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1

[0038] This invention provides a method for bolt inspection of power equipment based on unmanned aerial vehicle (UAV) inspection, including:

[0039] A bolt dataset is constructed using normal bolts. An object detection network is then used to detect the normal bolts in the bolt dataset to obtain bolt detection results. The bolt detection results include the location information of the detection box, confidence level, and category.

[0040] Based on the distribution of bolts in power equipment, a base plate component template database is constructed, which contains template diagrams of different types of components.

[0041] Determine whether the bolt inspection result image matches the component template image in the base plate component template database. If the match is successful, rotate and align the bolt base plate in the bolt inspection result image with the component template image.

[0042] Calculate the intersection-union ratio (IUR) of the actual bounding box and the detection box of the bolt component in the component template diagram. If the IUR is higher than the threshold, it indicates the location of a normal bolt; otherwise, it indicates the location of a missing bolt.

[0043] Please see Figure 1 This is a flowchart of a bolt inspection method for power equipment based on unmanned aerial vehicle (UAV) inspection, provided in an embodiment of the present invention.

[0044] Specifically, the target detection network is used to detect normal bolts in the bolt database. The location of normal bolts is detected and located, and a tight detection box (containing the location information, confidence level and category of the detection box) surrounding the normal bolt can be obtained. In addition, the detection box can be clearly and intuitively visualized on the detection result image, reducing the influence of complex background of the bolt and improving the signal-to-noise ratio.

[0045] Since the distribution of bolts on power tower components is regular, and there are generally only a limited number of types of power tower bolt base plate components, the number and position of bolts on the template of the same type of base plate component are the same. Therefore, by analyzing the distribution of bolts in power equipment, we can compile images of the existing limited types of base plate component types and build a base plate component template database.

[0046] Next, the test result image is matched with the component template image in the base plate component template database. The image that matches the test result image is found in the data, and then the test result image is adjusted.

[0047] Finally, based on the intersection-union ratio of the actual bolt frame and the detection frame in the component template drawing, it is determined whether the bolt position is normal or missing. A normal bolt position indicates that the bolt at that position is installed correctly and is not missing, while a missing bolt position indicates that the bolt at that position is installed abnormally and is missing.

[0048] In one implementation, the target detection network used is YOLOv5.

[0049] Specifically, please see Figure 2 This is the YOLOv5 network model structure. The YOLOv5 network consists of four main parts: the input end, the backbone network, the feature fusion module, and the output end. First, the input layer randomly crops, scales, and stitches the input image to find the most suitable adaptive anchor box for calculation, improving the accuracy of locating small target defects like bolts. Then, the backbone network aggregates at different image fine-grained levels, extracting feature maps at three scales. Next, the feature fusion module (Neck) expands the receptive field of the feature map by adding a Spatial Pyramid Pooling (SPP) structure and using three sets of multi-scale max pooling layers. It then uses upsampling to stitch together high and low feature layers to achieve feature fusion and obtain a new feature map, improving the propagation of low-level features. Subsequently, the Path Aggregation Network (PAN) propagates features from bottom to top, from weak to strong, enabling more feature fusion at the feature layers. The combination of these two operations strengthens the network's feature fusion capability. Finally, the output layer is the network's prediction part, which uses Non-Maximum Suppression (NMS) to filter target boxes, predict image features, generate bounding boxes, and predict the category. Using the YOLOv5 network for detection results in fast processing speed and high accuracy.

[0050] In one implementation, determining whether the bolt inspection result image matches the component template image in the base plate component template database includes:

[0051] Normalized multi-order rectangular features of the outer contour are extracted from bolt inspection result images and component template images respectively, and feature vectors with shape description are constructed.

[0052] The distribution difference between two eigenvectors is solved based on Wasserstein distance;

[0053] Whether a match is successful is determined based on whether the distribution difference is within the allowable range of the threshold.

[0054] Specifically, this step mainly involves matching the shape features of the bolt base plate component.

[0055] Specifically, a shape feature matching algorithm based on high-dimensional multi-order moment feature normalization is proposed. This algorithm extracts normalized multi-order rectangular features from the outer contour of the bolt base plate component in two images to establish its shape description. These features describe the feature vectors of the bolt base plate component in both the detection result image and the component template image. The distribution difference between the two feature vectors is calculated based on the Wasserstein distance. If the difference is within a threshold range, the matching is successful. This image matching algorithm exhibits high accuracy and strong robustness.

[0056] Specifically, the bolt inspection results and component templates are described by region shape, and the description results are F1 and F, respectively. M Two feature vectors. The key to achieving region feature matching lies in solving the distribution difference between the feature vectors. This method uses Wasserstein distance to solve for the vector distribution difference.

[0057] Wasserstein distance is used to measure the distance between two distributions, and is defined as follows:

[0058]

[0059] Where Π(F1, F M ) is F1 and F M The set of all possible joint distributions obtained by combining distributions. For each possible joint distribution γ, a sample x and y can be obtained by sampling (x, y) ~ γ. The distance ||xy|| between these samples is calculated, so the expected value of the sample pair distance under this joint distribution γ can be calculated. The lower bound that this expected value can be obtained among all possible joint distributions is the Wasserstein distance, that is, the minimum cost under optimal path planning. At this point, w(F1, F M ) is denoted as σ(F1, F M The difference between two feature vectors is represented by ). If the difference satisfies the following condition, it indicates that the segmentation result image (detection result image) and the bottom template image have successfully matched:

[0060] σ(F1, F M )<δ (2)

[0061] Here, δ is the set threshold parameter, which reflects a certain degree of fault tolerance for the region.

[0062] In one embodiment, the bolt base plate in the bolt inspection result image is rotated and aligned with the component template drawing, including:

[0063] Find the feature corner points of the base plate edge contour in the bolt inspection result image and the corresponding component template image;

[0064] Based on the corresponding feature corner points, determine the positional relationship between the detection result image and the bottom plate edge contour in the component template image, and rotate and align the bolt bottom plate in the bolt detection result image with the component template image according to the positional relationship.

[0065] Specifically, after successful template matching, the feature corner points (such as corner points and their positional relationships) of the base plate edge contour in the detection result image and the corresponding template are found to determine the positional relationship of the base plate edge contour in the two images. The base plate component in the detection image is rotated and aligned with the template to ensure that the two roughly coincide.

[0066] Please see Figure 3 This is the process of matching templates for base plate components and judging defects in the method provided in the embodiments of the present invention.

[0067] After a successful match, the Intersection over Union (IoU) between the actual bounding box and the detection box of the bolt component in the template is calculated. If the IoU is higher than a threshold, it represents the location of a normal bolt; if the calculated value is lower than the threshold, it represents the location of a missing bolt. This is how the location of a missing bolt is detected.

[0068] The beneficial effects of the method proposed in this invention include:

[0069] ① It has very little dependence on defect samples and is mainly based on inferences from a large number of normal bolt sample data.

[0070] ② By fully utilizing the regularity of bolt distribution on power tower components, the template information of the entire component can be adaptively obtained under different shooting angles and lighting conditions, thereby achieving the effect of indirectly locating missing bolts using normal bolts.

[0071] ③ The YOLOv5 network has a strong advantage in rapid model deployment and performs better when detecting small targets such as bolts. It has higher accuracy, faster running speed, cleaner detection results, and almost no overlapping borders.

[0072] Example 2

[0073] Based on the same inventive concept, this embodiment provides a bolt inspection system for power equipment based on drone inspection, including:

[0074] The normal bolt detection module is used to construct a bolt dataset from normal bolts, and to use a target detection network to detect normal bolts in the bolt database to obtain bolt detection results. The bolt detection results include the location information of the detection box, confidence level, and category.

[0075] The base plate component template database construction module is used to build a base plate component template database based on the distribution of bolts in power equipment. The base plate component template database contains template diagrams of different types of components.

[0076] The template matching module is used to determine whether the bolt inspection result image matches the component template image in the base plate component template database. If the match is successful, the bolt base plate in the bolt inspection result image is rotated and aligned with the component template image.

[0077] The bolt detection module is used to calculate the intersection-union ratio (IUR) of the actual bolt component frame and the detection frame in the component template diagram. If the IUR is higher than the threshold, it indicates the location of a normal bolt; otherwise, it indicates the location of a missing bolt.

[0078] Since the system described in Embodiment 2 of this invention is the same system used in implementing the UAV-based bolt inspection method for power equipment in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this system based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All systems used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0079] Example 3

[0080] Based on the same inventive concept, please refer to Figure 4 The present invention also provides a computer-readable storage medium 300 having a computer program 311 stored thereon, which, when executed, implements the method described in Embodiment 1.

[0081] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in the method for detecting bolts in power equipment based on UAV inspection in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0082] Example 4

[0083] Based on the same inventive concept, this application also provides a computer device, such as... Figure 5 As shown, it includes a memory 401, a processor 402, and a computer program 403 stored in the memory and executable on the processor. When the processor executes the program, it implements the method in Embodiment 1.

[0084] Since the computer equipment described in Embodiment 4 of this invention is the same computer equipment used in implementing the UAV-based bolt inspection method for power equipment in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer equipment based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer equipment used in the method of Embodiment 1 of this invention falls within the scope of protection of this invention.

[0085] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0087] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0088] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for detecting bolts of power equipment based on unmanned aerial vehicle inspection, characterized in that, include: A bolt dataset is constructed using normal bolts. An object detection network is then used to detect the normal bolts in the bolt dataset to obtain bolt detection results. The bolt detection results include the location information of the detection box, confidence level, and category. Based on the distribution of bolts in power equipment, a base plate component template database is constructed, which contains template diagrams of different types of components. Determine whether the bolt inspection result image matches the component template image in the base plate component template database. If the match is successful, rotate and align the bolt base plate in the bolt inspection result image with the component template image. Calculate the intersection-union ratio (IUR) of the actual bounding box and the detection box of the bolt component in the component template diagram. If the IUR is higher than the threshold, it indicates the location of a normal bolt; otherwise, it indicates the location of a missing bolt. The process of determining whether the bolt inspection result image matches the component template image in the base plate component template database includes: Normalized multi-order rectangular features of the outer contour are extracted from bolt inspection result images and component template images respectively, and feature vectors with shape description are constructed. The distribution difference between two eigenvectors is solved based on Wasserstein distance; Whether a match is successful is determined based on whether the distribution difference is within the allowable range of the threshold; The bolt base plate in the bolt inspection result image has been rotated and aligned with the component template drawing, including: Find the feature corner points of the base plate edge contour in the bolt inspection result image and the corresponding component template image; Based on the corresponding feature corner points, determine the positional relationship between the detection result image and the bottom plate edge contour in the component template image, and rotate and align the bolt bottom plate in the bolt detection result image with the component template image according to the positional relationship. 2.The unmanned aerial vehicle inspection based power equipment bolt detection method of claim 1, wherein, The object detection network used is YOLOv5.

3. A power equipment bolt detection system based on unmanned aerial vehicle inspection, characterized in that, Based on the method described in claim 1, the system includes: The normal bolt detection module is used to construct a bolt dataset from normal bolts, and to use a target detection network to detect normal bolts in the bolt database to obtain bolt detection results. The bolt detection results include the location information of the detection box, confidence level, and category. The base plate component template database construction module is used to build a base plate component template database based on the distribution of bolts in power equipment. The base plate component template database contains template diagrams of different types of components. The template matching module is used to determine whether the bolt inspection result image matches the component template image in the base plate component template database. If the match is successful, the bolt base plate in the bolt inspection result image is rotated and aligned with the component template image. The bolt detection module is used to calculate the intersection-union ratio (IUR) of the actual bolt component frame and the detection frame in the component template diagram. If the IUR is higher than the threshold, it indicates the location of a normal bolt; otherwise, it indicates the location of a missing bolt.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the program is executed, it implements the method as described in any one of claims 1 to 2.

5. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 2.

Images