Unmanned aerial vehicle automatic overhead transmission line inspection method and system based on front-end target identification

Abstract

The application provides a kind of unmanned aerial vehicle automation overhead transmission line inspection method and system based on front-end target identification, and relates to unmanned aerial vehicle transmission inspection field.No need to plan GPS inspection path and inspection point in advance, it can realize the automatic inspection of overhead transmission line, can effectively reduce the work of planning GPS path planning and setting cruise point, improve the inspection efficiency, reduce the labor cost.The application can automatically determine the inspection point according to the size of the target tower during the inspection process, and can cover the tower target after determining the inspection point, and filter the image after shooting, remove the invalid background image, reduce the data traffic, and improve the endurance of unmanned aerial vehicle.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) power transmission line inspection, and in particular to an automated UAV method and system for inspecting overhead power transmission lines based on front-end target recognition. Background Technology

[0002] After the infrastructure of overhead transmission lines is completed, inspection and acceptance work is required. In addition, regular inspections are also necessary during daily use. Currently, inspection methods mainly include manual tower climbing, image capture using high-magnification lenses, and drone-based tower inspection. Since overhead transmission line towers are generally over 20 meters high, manual tower climbing for inspection is time-consuming, difficult, and dangerous. Therefore, there is a growing trend towards using drones to capture detailed tower images for acceptance. However, manual drone operation for acceptance also presents challenges such as high operational difficulty, long inspection times, and a large amount of invalid data, affecting acceptance efficiency.

[0003] Existing drone-based overhead power line inspection systems typically require pre-planning of GPS inspection routes and setting up numerous observation points. They cannot autonomously inspect routes and points during the inspection process, resulting in low drone inspection efficiency. Furthermore, due to terrain and GPS signal limitations, pre-set inspection points may deviate significantly from actual inspection points, leading to missed inspections of some locations by the drone. Summary of the Invention

[0004] Purpose of the invention: To propose an automated overhead power transmission line inspection method and system based on front-end target recognition using unmanned aerial vehicles (UAVs), in order to overcome the aforementioned problems existing in the prior art.

[0005] The technical solution of this invention is implemented as follows:

[0006] Firstly, a method for automated overhead power transmission line inspection using unmanned aerial vehicles (UAVs) based on front-end target recognition is proposed. This method includes:

[0007] S1. Specify the starting tower, establish the starting position for the UAV inspection, and record the waypoint information of the starting position. S ;

[0008] The starting position must meet at least the following conditions: the starting tower is located within the field of view of the drone's camera;

[0009] S2. Establish an inspection mission. Starting from the initial position, the drone collects the current video stream in real time and tracks four targets of the tower: the tower head, the entire tower, the tower top, and the tower base.

[0010] During the tracking process, the detection positions of the tower head and the entire tower are kept within the first predetermined position of the lens's field of view; then the drone flies upward, so that the upper edge of the detection frame of the tower head is located within the second predetermined position of the lens's field of view;

[0011] S3. The drone flies toward the tower, detects the top of the tower and identifies key points on the top of the tower during the flight; when the top of the tower is detected and key points on the top of the tower are identified at the same time, it is determined that the drone is currently close to the top of the tower and sends the first adjustment command.

[0012] The first adjustment command includes: controlling the gimbal of the drone to adjust the lens's pitch angle to a vertically downward direction, and controlling the speed of the drone in steps until the overlap between the center of the tower top detection frame and the center of the image is within a threshold range. At this point, it is determined that the drone is currently directly above the tower top, and the drone's waypoint information G is recorded. fd ;

[0013] S4. Once the drone is directly above the top of the tower, it begins detecting the top of the tower;

[0014] S5. The drone maintains the heading angle and gimbal top angle of S4 to perform insulator detection with angle regression and calculate the inlet and outlet angles of the conductors.

[0015] S6. The drone maintains the heading from S5, adjusts the gimbal to eye level, and begins to reverse along the negative Y-axis, maintaining detection of the entire tower during the reversal. After reversing to the third predetermined position, the drone begins to descend, maintaining detection of the tower head during the descent. When the detection frame of the tower head is located in the center of the image, the drone stops descending, and the waypoint information G of the drone's current position is recorded. ftt ;

[0016] S7: The drone maintains the gimbal's eye level on the S7, retreats a predetermined distance, begins to descend, and stops after descending the predetermined distance; the drone automatically detects the tower body during the descent.

[0017] S8. Adjust the drone's gimbal to a 45-degree downward angle, and control the drone to descend again. During the descent, maintain detection of the tower base. When the tower base's detection frame is completely within the image, stop the drone's descent; record the drone's waypoint information G at this moment. ff ;

[0018] S9. After completing the frontal inspection of the tower, the drone flies back to G. fd Waypoints, with G fd The inspection of the right side of the tower is completed starting from the waypoint;

[0019] S10. Adjust the drone's attitude and complete the inspection of the back and left sides of the tower in sequence. At this point, the inspection of the current tower is completed, and the first completion message is output.

[0020] S11, the drone flies back to waypoint G fd After adjusting its attitude, it flies to the next level tower and repeats S4 to S10. After completing the inspection of the remaining N towers, the inspection task ends.

[0021] In a further embodiment of the first aspect, the first predetermined position is defined as follows: the detection position of the entire tower and the tower head is located at the middle position in the horizontal direction of the video stream image;

[0022] The second predetermined position is defined as follows: the upper edge of the detection frame of the tower head is located at the middle position in the vertical direction of the video stream image;

[0023] The third predetermined position is defined as follows: the position when the entire tower is captured in the video stream image of the drone.

[0024] In a further embodiment of the first aspect, the starting position in step S1 at least meets the following conditions: the starting tower is located within the field of view of the UAV's camera; when multiple towers exist within the field of view, the tower with the largest imaging area within the field of view is selected as the initial tower for inspection. Alternatively, the operator can manually set the starting tower for inspection.

[0025] In a further embodiment of the first aspect, the waypoint information G fd Including the current distance D of the drone from the ground fdg The distance D between the drone and the top of the tower at the current moment. fdt The GPS information of the drone's current location;

[0026] According to the waypoint information G fd Preliminary calculation of tower height H t :

[0027] H t =D fdg -D fdt .

[0028] In a further embodiment of the first aspect, step S2 includes: constructing an object detection model to track four targets—the tower head, the entire tower, the tower top, and the tower base—in real time. Here, to ensure the detection model can run efficiently on the UAV platform, the weights of the object detection model are quantized to further accelerate the model's operation.

[0029] The process of building an object detection model is as follows: collect video data of overhead transmission line poles, mark the rectangular box information of the tower head, the entire tower, the tower top, and the tower base, and mark the key point information of the tower head, the tower top, and the tower base.

[0030] The detection model used in this application is an improvement based on the YOLOv5s model, with the following four main improvements:

[0031] a. Decouple the detection and classification heads in the original YOLOv5s model and use two branches to predict detection and classification respectively;

[0032] b. The model classification branch is trained using focal loss for the first 50 epochs and varifocal loss for the remaining epochs.

[0033] c. Since the target of this application is a man-made building with a fixed aspect ratio, the aspect ratio information of the object is used in the anchor setting;

[0034] d. SimOTA was used for anchor assignment. After these changes, the model's detection performance was significantly improved.

[0035] In a further embodiment of the first aspect, the process of detecting the key points at the top of the tower in step S3 includes:

[0036] Construct a key point detection model at the top of the tower;

[0037] The tower tops detected by the object detection model are input into the tower top keypoint detection model. To ensure efficient operation of the tower top keypoint detection model on the UAV platform, the skeleton network of the model is simplified, retaining only the most basic structure, which greatly improves the network's operating speed.

[0038] Specifically, this application uses HRNet to identify key points at the top, top, and base of a tower. HRNet contains multiple parallel feature map branches with different resolutions, and the interaction between these branches is achieved through upsampling and summation operations, thereby improving model performance. This application makes two improvements to HRNet during experiments:

[0039] a. During training, the keypoint detection targets (tower head, tower top, tower base) are randomly cropped centered on the bounding box. If a keypoint is outside the cropping area, it is made invisible. This is done to simulate the jittering of the detection bounding box output by the detection model.

[0040] b. The parallel subnetworks and repeated feature fusion in HRNet result in a large number of model parameters, which is not conducive to deployment on UAV platforms. Therefore, this application replaces the ordinary convolution operation in HRNet with depthwise separable convolution to reduce the memory usage during model runtime.

[0041] In a further embodiment of the first aspect, step S4 includes:

[0042] Adjust the rotation angle of the drone's gimbal according to the ratio of the long and short sides of the detection frame at the top of the tower. When the short side of the rectangle no longer gets shorter, it means that the drone's heading angle is now orthogonal to the large or small side of the tower.

[0043] In a further embodiment of the first aspect, the process of performing angle regression detection on the insulator in step S5 and calculating the entry and exit angles of the conductors further includes:

[0044] S51. Taking the center of the rectangular frame at the top of the tower as the origin, the horizontal centerline of the tower top detection frame as the x-axis, and the vertical centerline of the tower top detection frame as the y-axis, determine the positive direction of the y-axis; this direction ensures that the angle formed by the GPS position of the top of the previous tower, the GPS position of the top of this tower, and the positive direction of the y-axis is greater than 90 degrees.

[0045] S52. Establish a spatial coordinate system X with the positive direction to the right of the positive y-axis as the positive direction of the x-axis. d At this moment, the drone's gimbal is facing the positive y-axis.

[0046] S53. Use RotatedRetinaNet to detect insulators on transmission lines and output the position of the insulator center point, the width of the insulator, the height of the insulator, and the direction of the insulator.

[0047] Oriented object detection models need to determine the object's rotation direction in addition to its position, meaning they need to simultaneously determine the target's center point, dimensions, and rotation angle. This is common in target detection in top-down views, such as remote sensing image target detection and aerial image target detection. This application uses RotatedRetinaNet for oriented insulator detection. Two improvements were made to RotatedRetinaNet during the experiments: a. To reduce resource consumption and adapt to UAV platforms, all ordinary convolutions were replaced with depthwise separable convolutions; b. To further improve model performance, the loss function in RotatedRetinaNet was replaced with GWD Loss. The construction process of the oriented object detection model is as follows: 1. Using the labelme annotation tool, polygon annotations were performed on the insulators in the UAV aerial data, and the minimum bounding rectangle of the polygon annotation region was calculated to generate training data; 2. The oriented insulator detection model was trained based on the modified RotatedRetinaNet model; 3. The model was quantized and deployed to the UAV.

[0048] S54. Obtain the direction of the conductor based on the direction of the insulator;

[0049] S55. Insulators located in the first and second quadrants are used to calculate the conductor outgoing angle θ. c Insulators located in the third and fourth quadrants are used to calculate the conductor entry angle θ. r At this point, the x-axis is the angle bisector of the angle between the incoming and outgoing conductors; the incoming and outgoing angles are used to search for the next level of tower.

[0050] In a further embodiment of the first aspect, step S7 further includes:

[0051] Adjust the drone gimbal to a head-up view and use the object detection model to obtain the position of the tower head and the position of the tower base;

[0052] Construct key point detection models for the tower head and the tower base respectively, and obtain the coordinates of key points at the tower head and the tower base.

[0053] Two key points at the connection between the lowest crossbeam in the tower head and the tower body are taken as the top coordinate points of the tower body; two key points at the connection between the top left and right sides of the tower base and the tower body are taken as the bottom coordinate points of the tower body.

[0054] The position of the tower in the image is determined using four key points at the top and bottom of the tower, while also considering camera intrinsic and extrinsic parameters and the tower height H. t The drone's retreat distance N and descent distance M are used to automatically calculate the locations for tower-coverage photography.

[0055] After acquiring the locations for comprehensive image capture, the gimbal is invoked to sequentially traverse each shooting location. During image capture, ResNet18 is used to evaluate the image content. If tower structures are present in the image, the image is saved and transmitted back; otherwise, it is not transmitted and the image is deleted. This application made two improvements to ResNet18 during the experiment: a. To reduce resource consumption and adapt to the drone platform, all ordinary convolutions were replaced with depthwise separable convolutions; b. To further improve model performance, an attention mechanism (SE Attention) was added to the model.

[0056] Secondly, an automated overhead power transmission line inspection system using unmanned aerial vehicles (UAVs) is proposed. This system includes a UAV and electronic equipment. The electronic equipment can be embedded directly within the UAV or run in the cloud. When running in the cloud, control commands can be transmitted to the UAV via a wireless network. The electronic equipment includes a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other via the communication bus. The memory stores at least one executable instruction, which causes the processor to execute the power transmission line inspection method described in the first aspect.

[0057] Beneficial Effects: The automated overhead power transmission line inspection method and system proposed in this invention eliminates the need for pre-planning GPS inspection routes and points, enabling automated inspection of overhead power transmission lines. This effectively reduces the workload of GPS route planning and patrol point setting, improving inspection efficiency and lowering labor costs. During the inspection process, this invention automatically determines inspection points based on the size of the target towers. After determining the inspection points, it can take comprehensive photos of the target towers and filter the images to remove invalid background images, reducing data upload traffic and improving the drone's endurance. Attached Figure Description

[0058] Figure 1 This is a flowchart of an automated overhead power line inspection system using unmanned aerial vehicles (UAVs).

[0059] Figure 2 This is a flowchart of the inspection process for the front of the tower.

[0060] Figure 3 This is a schematic diagram of the top of the tower.

[0061] Figure 4 This is a schematic diagram of the front of the tower.

[0062] Figure 5 This is a schematic diagram of the insulator markings on the top of the tower.

[0063] Figure 6 This is a diagram illustrating the course adjustment of a drone.

[0064] Figure 7 This is a diagram illustrating a drone flying upwards.

[0065] Figure 8 This is a schematic diagram of a spatial coordinate system.

[0066] Figure 9 This is a schematic diagram of a tower-head coverage photography setup.

[0067] Figure 10 This is a schematic diagram of the tower body area.

[0068] Figure 11 This is a schematic diagram of the tower base area. Detailed Implementation

[0069] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0070] After the infrastructure of overhead transmission lines is completed, inspection and acceptance work is required. In addition, regular inspections are also necessary during daily use. Currently, the main inspection methods include manual tower climbing, image capture using high-magnification lenses, and drone photography of towers. Since overhead transmission line towers are generally over 20 meters high, manual tower climbing for inspection is time-consuming, difficult, and dangerous. Therefore, the focus is gradually shifting to using drones to capture detailed images of the towers for acceptance purposes.

[0071] The applicant's research found that manual drone inspection also suffers from problems such as high operational difficulty, long inspection time, and a large amount of invalid data, affecting inspection efficiency. Existing drone inspection systems require drone operators to receive flight modeling data in advance to determine the GPS location information of each inspection point, and cannot automatically plan inspection routes during the inspection process, resulting in low drone inspection efficiency. Furthermore, while there is some research on fully autonomous drone navigation, it still requires pre-planning of GPS inspection paths and relies on GPS location information and numerous other sensors; secondly, the aforementioned research mainly focuses on the drone's navigation process, lacking research on how to complete subsequent inspection actions after reaching waypoints, thus failing to effectively improve inspection quality.

[0072] To address this, the applicant proposes an automated overhead power transmission line inspection method and system based on front-end target recognition using unmanned aerial vehicles (UAVs). This method enables automated inspection of overhead power transmission lines without the need for pre-planning GPS inspection routes and locations, effectively reducing the workload of GPS route planning and patrol point setting, improving inspection efficiency, and lowering labor costs. The invention can automatically determine inspection points based on the size of the target towers during the inspection process. After determining the inspection points, it can take comprehensive photos of the target towers and filter the images to remove invalid background images, reducing data upload traffic and improving the UAV's endurance.

[0073] Figure 1 The system flowchart of the UAV-based automated overhead power line inspection technology based on front-end target recognition is shown.

[0074] Figure 2 The inspection process for the front part of the inspection tower was demonstrated.

[0075] like Figure 2 As shown, the inspection of the front of the tower includes three parts: inspection of the tower head, inspection of the tower body, and inspection of the tower base. The inspection process for the right side, left side, and back of the tower is the same as the inspection process for the front, and also includes these three parts.

[0076] The following describes the UAV-based automated overhead power line inspection method of this application from one perspective, with specific implementation details as follows:

[0077] Data collection and model training

[0078] (1) The drone pilot controls the drone to fly along the power transmission line to collect data on the towers and channels. When collecting tower data, the drone needs to be controlled to fly around the tower to collect images of the tower at different angles and in different postures. At the same time, data also needs to be collected for each type of tower.

[0079] (2) Perform frame extraction on the acquired video to generate an image dataset.

[0080] (3) Label the data. First, label the tower head, tower top, the entire tower, and the tower base. Then, label the key points of the tower top, tower head, and tower base. Finally, label the insulator strings in the diagram. The labeling effect is as follows: Figure 3 , Figure 4 and Figure 5 As shown.

[0081] Figure 3 The solid lines on the outer sides indicate the entire tower. Figure 3 The dashed box at the top center is the label for the tower head. The points within the standard box of the tower head are the key points of the tower head. There are a total of 16 key points labeled in the tower head section. Figure 3 The key points are indicated by white dots. The key points of the tower head are the two outermost vertices on the left and right sides of each crossarm and the intersection of the crossarm and the tower column. These 16 key points are named in order from top to bottom and left to right as follows: TL, TLC, TRC, TR, UL, ULC, URC, UR, ML, MLC, MRC, MR, BL, BLC, BRC, BR. Figure 3 The dotted circle at the bottom center marks the base of the tower. The points within the standard frame of the base are the key points of the base. There are a total of 5 key points in the base. Figure 3 The five key points are represented by white triangles. These are the left and right joints p_bl and p_br of the bottom diagonal member and the foundation; the intersection p_tm of the two bottom diagonal members; and the intersection p_tl and p_tr of the horizontal support and the left and right main members where this intersection is located.

[0082] Figure 4 The box in the image represents the label box at the top of the tower. The points within this label box are the key points of the top section of the tower, represented by triangles. The key points at the top of the tower are arranged in a top-to-bottom, left-to-right order and named t1, t2, tl, tr, bl, br, b1, b2.

[0083] Figure 5 The box in the middle is the label box for the tower top insulator; here we use a rotated rectangle for labeling.

[0084] (4) Train the YOLOv5s detection model using bounding boxes for the tower head, the entire tower, the tower base, and the tower top. Train an HRNet-based keypoint detection model using keypoints at the tower head, tower top, and tower base. Note that three keypoint detection models need to be trained here to detect insulators in different locations. Train a tower top insulator detection model based on the RotatedRetinaNet model. Since all models run on the front-end UAV platform, we have simplified all the above network models to improve the running speed.

[0085] (5) Quantize the above models, which requires the use of operator fusion, perceptual quantization and other techniques. Finally, deploy these models on the UAV platform.

[0086] Pole inspection process

[0087] (1) The drone operator points the drone camera at the starting tower of the inspection route of the overhead power transmission line, starts the flight control app to begin the inspection mission, and records the waypoint information G when the drone takes off. SNote that when starting the inspection, the target tower must be within the drone's field of view, and ideally there should only be one tower within the field of view. If there are multiple towers within the field of view, the system will default to selecting the tower with the larger imaging area as the initial tower for inspection. Alternatively, the drone operator can manually rotate the initial inspection tower.

[0088] (2) After the UAV takes off, the YOLOv5s model is used to detect towers in the UAV video stream. Based on the detection results, the UAV's heading is adjusted so that the target tower is located in the center of the captured image. For example... Figure 6 As shown in the figure, the tower in the center is the starting tower for this patrol mission.

[0089] (3) After takeoff, the UAV detects the tower, including four targets: the tower top, the entire tower, the top of the tower, and the base. Based on the positions of the entire tower and the tower top in the image, the UAV's yaw angle is adjusted so that the detection positions of the entire tower and the tower top are located in the middle of the horizontal direction of the image. The UAV then flies upwards, so that the upper edge of the detection frame for the tower top is located slightly above the middle of the vertical direction of the image. Figure 7 As shown in the image, the drone has reached the location of the inspection tower.

[0090] (4) The drone flies towards the tower, continuously detecting the entire tower, the top of the tower, the top of the tower, and key points on the top of the tower during flight. The top of the tower is used to fine-tune the drone's heading, and the adjustment method is the same as the previous step. The detection results of the top of the tower and the key points on the top of the tower are used to determine whether the drone has approached directly above the tower. When the top of the tower and the key points on the top of the tower can be detected simultaneously, it is determined that the drone has approached directly above the top of the tower. At this time, the drone gimbal is adjusted to set the camera's pitch angle to a vertical 90-degree downward direction, and the drone speed is controlled and adjusted step by step until the center of the top of the tower detection frame and the center of the image are basically aligned. At this time, the drone has flown directly above the top of the tower. Record the drone's waypoint information G. fd Waypoint information G here fd Including the distance D of the drone from the ground fdg The distance D between the drone and the top of the tower fdt The GPS information of the drone's current location. Estimate the tower height H at this time. t H t =D fdg -D fdt Here, HRNet is used to detect keypoints at the top of the tower. The input for keypoint detection is the tower top detected by the YOLOv5s model. Figure 8 As shown, at this time, both the tower top detection model and the tower top key point detection model detected the corresponding target, indicating that the drone has reached the top of the inspection tower.

[0091] (5) When the drone is directly above the top of the tower, the top of the tower is detected, and the rotation angle of the drone's gimbal is adjusted according to the ratio of the long and short sides of the detection frame at the top of the tower. When the short side of the rectangle no longer shortens, it means that the drone's heading angle is now orthogonal to the large or small side of the tower.

[0092] (6) Keep the UAV heading angle and gimbal viewing angle unchanged, and perform angle regression detection on the insulator to calculate the inlet and outlet angles of the conductor.

[0093] At this point, with the center of the rectangular frame at the top of the tower as the origin, the horizontal centerline of the tower top detection frame as the x-axis, and the vertical centerline of the tower top detection frame as the y-axis, the positive direction of the y-axis is determined. This direction ensures that the angle formed by the GPS position of the top of the previous tower, the GPS position of the top of this tower, and the positive direction of the y-axis is greater than 90 degrees. Finally, with the direction to the right of the positive direction of the y-axis as the positive direction of the x-axis, a spatial coordinate system X is established. d At this point, the drone is facing the positive y-axis. RotatedRetinaNet is used to inspect the insulators on the transmission line. RotatedRetinaNet can output not only the position of the insulator's center point, the width and height of the insulator, but also the insulator's orientation. The orientation of the conductor can be obtained from the insulator's orientation.

[0094] Insulators in the first and second quadrants are used to calculate the conductor exit angle θ. c Insulators in the third and fourth quadrants are used to calculate the conductor entry angle θ. r At this point, the x-axis should be the angle bisector of the angle between the incoming and outgoing lines of the conductor. For example... Figure 8 As shown, coordinate system X d The insulators are marked with dotted lines, and the directional insulators are represented by a rotating rectangle. The flat-angled dotted arrow in the directional insulator detection frame indicates the direction of the conductor exit.

[0095] (7) The UAV maintains its current heading, adjusts the gimbal to eye level, and begins to reverse along the negative y-axis. During the reversal, it maintains detection of the entire tower. The UAV stops reversing when the YOLOv5s detection model can capture the complete tower from the UAV's video stream, and the reversal distance N meters is recorded. At this point, the UAV begins to descend, maintaining detection of the tower head during the descent. The UAV stops descending when the detection frame of the tower head is in the center of the image, and the descent distance M meters is recorded. The waypoint information G of the UAV's location is recorded. ftt This includes the distance D between the drone and the ground. ftg And the GPS information at the top of the tower at the current location. Based on the camera's internal and external parameters, the current waypoint information G of the drone. ftt When the drone is at the top of the tower, waypoint information G fdThe system automatically calculates the location of the tower's position in the image for coverage photography. The coverage photography of the tower's position is as follows: Figure 9 As shown. The number of grids for overlay photography is automatically determined based on the image size and shooting granularity, rather than a fixed number. After obtaining the points for overlay photography, the gimbal is invoked to traverse each shooting point in turn. When taking a picture, ResNet18 is used to judge the content of the image. If tower materials appear in the image, the image is saved and transmitted back; otherwise, the image is not transmitted back and is deleted after shooting.

[0096] (8) Adjust the UAV gimbal to a level view and use the detection model to obtain the tower head and base positions. Then, use the tower head key point detection model and the tower base key point detection model to obtain the coordinates of the tower head key points and the tower base key points. Use the tower head key points BLC and BRC and the tower base key points p_tl and p_tr to determine the range of the tower body, and perform a coverage photograph of the tower body. The method for the coverage photograph of the tower body is the same as that for the tower head. The range of the tower body is as follows: Figure 10 As shown, the polygonal area formed by the double black lines in the image represents the front of the tower. The method for photographing the entire tower body is the same as that for the tower head.

[0097] (9) Adjust the drone gimbal to a 45° downward angle and control the drone to descend. Maintain detection of the tower base during descent; stop the drone descent when the tower base's detection frame is completely within the image. Record the waypoint information G of the drone's location. fj Using the location of the tower base in the image, and based on the camera's intrinsic and extrinsic parameters, the system automatically calculates the points for tower base coverage photography. Photos are taken at each point, and the resulting images are then filtered using the model. For example... Figure 11 As shown in the figure, the dotted rectangle represents the base area of ​​the tower.

[0098] (10) After the drone completes the frontal inspection of the tower, the drone flies back to G. fd Return to spatial coordinate system X d From the origin position, adjust the gimbal to eye level, oriented towards the positive X-axis. Then, repeat step 7 to record the drone's waypoint information (G). rt Based on the internal and external parameters of the drone's camera, the tower detection position, and the drone's waypoint information G... rt and G ftt Take a comprehensive photo of the right side of the tower head and then filter the image.

[0099] The drone's gimbal was then adjusted to a level view, and the YOLOv5s model was used to inspect the tower base, the entire tower, and the tower top. A key point detection model for the tower base was used to detect key points. Based on the location of the key points at the top of the tower base, the lower edge of the detection frame at the top of the tower, and the position of the detection frame across the entire tower, the location of the right side of the tower was determined. After determining the location of the right side of the tower, a comprehensive photograph of the tower was taken, and the image was filtered using the model. After completing the inspection of the right side of the tower, the drone began its descent to inspect the right side of the tower base. The inspection method for the right side of the tower base was the same as that for the front tower base. The reason why the key points at the tower top were not used to determine the location of the upper half of the tower during the side inspection was because the key point features at the side of the tower top were not obvious, resulting in poor detection performance; therefore, other methods were used to indirectly determine the location of the tower body.

[0100] (11) After the UAV completed the inspection of the front and right sides of the tower, it began to inspect the back and left sides of the tower. The inspection method of the back of the tower is the same as that of the front of the tower, and the inspection method of the left side of the tower is the same as that of the right side of the tower.

[0101] (12) After completing the inspection of the current tower, the UAV will fly back to waypoint G. fd Adjust the drone's gimbal to eye level and adjust its heading so that it faces the positive direction of the Y-axis in the spatial coordinate system. The drone should travel along the guide wire at an angle θ. c During flight, the drone maintains its detection of the next-level tower, controlling its flight direction to keep the next-level tower centered in the frame. As the drone flies, the area of ​​the next-level tower in the frame increases until it exceeds the drone's frame range. At this point, the tower inspection begins, using the same method as above. If no outgoing insulator is found during the directional insulator detection in step (6), then this tower is the last level tower. After completing the inspection of this tower, the inspection task ends.

[0102] In summary, this application enables automated inspection of overhead transmission lines, eliminating the need for pre-planning drone patrol paths or configuring gimbals for image capture and positioning, thus significantly improving inspection efficiency and reducing labor costs. Firstly, it acquires the positions of various structural components of the inspected towers, using a YOLOv5s detection model to detect the entire tower, tower head, tower top, and tower base. The RotatedRetinaNet-based insulator detection model, which not only outputs the insulator's position but also its orientation, provides further information. Hrnet-based tower top key point detection can determine whether the drone has reached the designated location. Hrnet-based tower head and base key point detection models effectively acquire the positions of key points at the tower head and base, more accurately locating the edges of the tower body and base. Based on the detection frame position, the gimbal's internal and external parameters, and the drone's waypoint information, it enables comprehensive image capture of the target object. This comprehensive image capture automatically determines the shooting points, allowing for tower inspection by photographing these points. After completing the inspection of the current tower, the drone autonomously plans its flight path based on the direction of the outgoing line insulators and the inspection results of the next-level tower. This application enables automated inspection of transmission lines, greatly improving inspection efficiency.

[0103] Although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for automated overhead power line inspection using unmanned aerial vehicles (UAVs) based on front-end target recognition, characterized in that, include: S1. Specify the starting tower, establish the starting position for the UAV inspection, and record the waypoint information of the starting position. ; The starting position must meet at least the following conditions: the starting tower is located within the field of view of the drone's camera; S2. Establish an inspection mission. Starting from the initial position, the drone collects the current video stream in real time and tracks four targets of the tower: the tower head, the entire tower, the tower top, and the tower base. During the tracking process, the detection positions of the tower head and the entire tower are kept within the first predetermined position of the lens's field of view; then the drone flies upward, so that the upper edge of the detection frame of the tower head is located within the second predetermined position of the lens's field of view; S3. The drone flies toward the tower, detects the top of the tower and identifies key points on the top of the tower during the flight; when the top of the tower is detected and key points on the top of the tower are identified at the same time, it is determined that the drone is currently close to the top of the tower and sends the first adjustment command. The first adjustment command includes: controlling the gimbal of the drone to adjust the lens's pitch angle to a vertically downward direction, and controlling the speed of the drone in steps until the overlap between the center of the tower top detection frame and the center of the image is within a threshold range. At this point, it is determined that the drone is currently directly above the tower top, and the waypoint information of the drone at this moment is recorded. ; S4. Once the drone is directly above the top of the tower, it begins detecting the top of the tower; S5. The drone maintains the heading angle and gimbal view angle of S4, performs insulator detection with angle regression, and calculates the inlet and outlet angles of the conductors, specifically including: S51. Taking the center of the rectangular frame at the top of the tower as the origin, the horizontal centerline of the tower top detection frame as the x-axis, and the vertical centerline of the tower top detection frame as the y-axis, determine the positive direction of the y-axis; this direction ensures that the angle formed by the GPS position of the top of the previous tower, the GPS position of the top of this tower, and the positive direction of the y-axis is greater than 90 degrees. S52. Establish a spatial coordinate system with the positive direction to the right of the positive y-axis as the positive direction of the x-axis. At this moment, the drone's gimbal is facing the positive y-axis. S53. Use the orientation object detection model to detect insulators on the transmission line and output the position of the insulator center point, the width of the insulator, the height of the insulator, and the orientation of the insulator. S54. Obtain the direction of the conductor based on the direction of the insulator; S55. Insulators located in the first and second quadrants are used to calculate the conductor outgoing angle. Insulators located in the third and fourth quadrants are used to calculate the conductor entry angle. At this point, the x-axis is the angle bisector of the angle between the incoming and outgoing conductors; the incoming and outgoing angles are used to search for the next level of tower. S6. The drone maintains the heading from S5, adjusts the gimbal to eye level, and begins to reverse along the negative Y-axis, maintaining detection of the entire tower during the reversal. After reversing to the third predetermined position, the drone begins to descend, maintaining detection of the tower head during the descent. When the detection frame of the tower head is located in the center of the image, the drone stops descending, and the waypoint information of the drone's current position is recorded. ; S7: The drone maintains the gimbal's eye level on the S7, retreats a predetermined distance, begins to descend, and stops after descending the predetermined distance; the drone automatically detects the tower body during the descent. S8. Adjust the drone's gimbal to a 45-degree downward angle, and control the drone to descend again. During the descent, maintain detection of the tower base. When the tower base's detection frame is completely within the image, stop the drone's descent; record the waypoint information of the drone's current location. ; S9. After completing the frontal inspection of the tower, the drone flew back to its original position. waypoints, The inspection of the right side of the tower is completed starting from the waypoint; S10. Adjust the drone's attitude and complete the inspection of the back and left sides of the tower in sequence. At this point, the inspection of the current tower is completed, and the first completion message is output. S11, the drone flies back to its waypoint. After adjusting its attitude, it flies to the next level tower and repeats S4 to S10. After completing the inspection of the remaining N towers, the inspection task ends.

2. The transmission line inspection method according to claim 1, characterized in that, When the detection position of the entire tower and the tower head is located at the middle position in the horizontal direction of the video stream image, this moment is defined as the first predetermined position; When the upper edge of the detection frame of the tower head is located at the middle position in the vertical direction of the video stream image, this moment is defined as the second predetermined position; When the complete tower is captured in the video stream image of the drone, this moment is defined as the third predetermined position.

3. The transmission line inspection method according to claim 1, characterized in that, The starting position in step S1 must meet at least the following conditions: the starting tower is located within the field of view of the UAV's camera; when there are multiple towers within the field of view of the camera, the tower with the largest imaging area within the field of view of the camera is selected as the initial tower for inspection.

4. The transmission line inspection method according to claim 1, characterized in that, The waypoint information Including the current distance of the drone from the ground The current distance between the drone and the top of the tower. The GPS information of the drone's current location; Based on the waypoint information Preliminary calculation of tower height : 。 5. The transmission line inspection method according to claim 1, characterized in that, Step S2 includes: constructing an object detection model to track four targets in real time: tower head, the entire tower, the top of the tower, and the base of the tower.

6. The transmission line inspection method according to claim 5, characterized in that, Step S3, which involves detecting the key points at the top of the tower, includes: Construct a key point detection model at the top of the tower; The tower top detected by the object detection model is input into the tower top key point detection model.

7. The transmission line inspection method according to claim 1, characterized in that, Step S4 includes: Adjust the rotation angle of the drone's gimbal according to the ratio of the long and short sides of the detection frame at the top of the tower. When the short side of the rectangle no longer gets shorter, it means that the drone's heading angle is now orthogonal to the large or small side of the tower.

8. The transmission line inspection method according to claim 5, characterized in that, Step S7 further includes: Adjust the drone gimbal to a head-up view and use the object detection model to obtain the position of the tower head and the position of the tower base; Construct key point detection models for the tower head and the tower base respectively, and obtain the coordinates of key points at the tower head and the tower base. Two key points at the connection between the lowest crossbeam in the tower head and the tower body are taken as the top coordinate points of the tower body; two key points at the connection between the top left and right sides of the tower base and the tower body are taken as the bottom coordinate points of the tower body. The position of the tower in the image is determined using four key points at the top and bottom of the tower, while also taking into account camera intrinsic and extrinsic parameters and the tower height. Drone retreat distance Drone descent distance It automatically calculates the locations for taking photos covering the entire tower. After obtaining the locations for coverage photography, the gimbal is invoked to traverse each shooting location in turn. When taking a picture, ResNet18 is used to judge the content of the picture. If tower materials appear in the picture, the picture is saved and sent back. Otherwise, the picture is not sent back and the image is deleted after taking the picture.

9. An automated overhead power transmission line inspection system using unmanned aerial vehicles (UAVs), characterized in that: include: Drones; An electronic device is embedded in the drone or located in the cloud and communicates with the drone; the electronic device includes a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform the transmission line inspection method as described in any one of claims 1 to 8.

Images