Defect Analysis Method for UAV Inspection and Target Detection of Transmission Tower with One Tower and Six Cameras

Abstract

The present invention relates to the technical field of transmission tower inspection, and discloses a method for inspecting a transmission tower with six cameras by an unmanned aerial vehicle and analyzing defects in target detection. By simply marking in the three-dimensional tower point cloud, the flight points for six cameras on one tower are calculated and generated; the flight points are connected in sequence to generate the inspection route of the unmanned aerial vehicle for six cameras on one tower; it is judged whether the route is safe, and turning points are automatically added to the unsafe route, so as to obtain the final inspection route of the tower unmanned aerial vehicle; the actual flight inspection is carried out, and the flight points are optimized: the yolov5 or Cornernet-Compare target detection algorithm is adopted to intelligently identify potential hazards in the inspection photos. The present invention can quickly generate the route for six cameras on one tower, improve the intelligent recognition algorithm, quickly identify defects and potential hazards in the inspection photos, and can find out defects and potential hazards faster and more accurately, improving the quality of data processing.

Description

Technical Field

[0001] The present invention belongs to the technical field of transmission line inspection, and particularly relates to a method for inspecting a transmission tower with six cameras by an unmanned aerial vehicle and defect analysis of target detection. Background Art

[0002] The unmanned aerial vehicle (UAV) automatic driving inspection technology is widely used in power inspection. The automatic fine inspection of transmission line towers can greatly improve the inspection efficiency, reduce the requirements for the UAV operation quality of inspectors, greatly reduce the inspection cost, and ensure the safety of power grid inspection operations.

[0003] For the automatic driving of UAV power inspection, a safe inspection route is required, and there are mainly two methods for generating it. One is to use machine learning, where experienced pilots manually control the UAV for fine inspection. The operator controls the UAV to perform fine acquisition of tower components according to the standard inspection operation guide, automatically records, and details the inspection information, and saves it to generate a fine inspection automatic driving route for the tower. The other is to automatically generate a route based on the extraction of a three-dimensional point cloud model. After obtaining the inspection shooting points of the tower body through puncture points, a method for automatically generating an inspection route according to the inspection point information is more efficient than generating a route by collecting tower inspection points through machine learning.

[0004] The six-shot inspection of a transmission tower usually refers to the full view of the tower, the tower head, the tower body, the pole number plate, the tower base, and the access roads on the large and small side. When performing fine inspection of the tower, it is necessary to take pictures at these shooting points. At present, there is a problem with automatically generating a route based on the extraction of components from a three-dimensional point cloud model. There is a lack of a method for generating flight points for the six-shot inspection of the tower, the degree of intelligence is low, the generated six-shot flight points are generally unsafe, and there are deviations in the shooting targets, resulting in poor inspection quality. With the rapid development of current artificial intelligence technology, the flight points of the six-shot inspection can be considered to be optimized by combining front-end intelligent recognition technology to improve the safety of the route. The inspection photos of the six-shot inspection can also be optimized for the model to better identify the targets. Currently, there is a lack of research on the method for inspecting a transmission tower with six cameras by an unmanned aerial vehicle in the current power inspection industry.

[0005] Therefore, there is a need for a simple, safe and reliable method that can automatically and quickly generate safe and accurate six-shot flight points according to the inspection requirements of the six-shot inspection of the tower, perform automatic safety verification of the flight route, and combine with the front-end intelligent recognition of the UAV to optimize the flight points during actual flight, save them as a safe route for re-flight automatic driving, and perform intelligent recognition of defect hazards on the inspection images of the six-shot inspection, further improving the automation level and inspection efficiency of UAV fine inspection and reducing the operation difficulty. Summary of the Invention

[0006] To solve the problems pointed out in the above background technology, improve the automation level of the one-pole six-shot inspection of transmission lines, and reduce the operation difficulty, the present invention provides a method for one-pole six-shot UAV inspection and target detection defect analysis of transmission towers. Through the three-dimensional tower point cloud and the line trend, it can quickly generate the whole tower view, tower head, tower body, pole number plate, tower base, large-side channel, and small-side channel waypoints of the transmission tower. The generated waypoints meet the requirements of the UAV inspection image shooting guidance for overhead transmission lines. When conducting the first flight inspection of the flight route, combined with the front-end intelligent recognition technology of the UAV, the waypoints are detected and optimized to generate a safe flight route, and for the inspection photos of one-pole six-shot, intelligent identification of defect hazards is carried out, making the one-pole six-shot inspection of transmission towers faster, more standard, and safer, reducing the operation difficulty, and improving the level of intelligent data processing.

[0007] To achieve the above object, the present invention adopts the following technical solutions: A method for one-pole six-shot UAV inspection and target detection defect analysis of transmission towers, the steps are as follows:

[0008] Step S1: Use a UAV equipped with RTK positioning to inspect the transmission line towers, collect three-dimensional high-precision laser point cloud data, and load and present the three-dimensional tower point cloud;

[0009] Step S2: Automatically calculate and generate one-pole six-shot waypoints by simply marking in the three-dimensional tower point cloud, including waypoint information of the whole tower view, tower head, tower body, pole number plate, tower base, large-side channel, and small-side channel; The waypoint information includes waypoint coordinates, UAV gimbal angle, and aircraft head direction;

[0010] Step S3: Connect the one-pole six-shot waypoints generated in Step S2 in the order: whole tower view waypoint, tower head waypoint, tower body waypoint, pole number plate waypoint, tower base waypoint, large-side channel waypoint, small-side channel waypoint, to generate a one-pole six-shot UAV inspection flight route for the transmission tower;

[0011] Step S4: Route safety verification: Judge whether the route is safe, and automatically add turning points to the unsafe route to obtain the final UAV inspection route for the transmission tower;

[0012] Step S5: The UAV conducts one-pole six-shot actual flight inspection and optimizes the waypoints: Based on the generated inspection route, the UAV performs the first flight inspection, combines the built-in front-end intelligent recognition device of the UAV, identifies and detects the shooting target, optimizes the flight waypoints, takes pictures, and saves the adjusted waypoints;

[0013] Step S6, Intelligent Analysis of Inspection Photos for One Pole with Six Photos: Use the yolov5 or Cornernet-Compare object detection algorithm to perform intelligent identification of potential hazards in the inspection photos; The Cornernet-Compare object detection algorithm is a neural network model of the Y-shaped structure for defect component detection that, based on the idea of the Cornernet object detection algorithm and targeting the characteristics of the inspection photos for one pole with six photos, uses Heatmap labels to mark the regions of interest and then realizes efficient feature localization and recognition by calculating the differences in features in the regions of interest.

[0014] Further preferably, the specific process of step S6 is as follows:

[0015] S61. Export the taken inspection photos for one pole with six photos, classify the photos, perform distance matching based on the longitude and latitude of the line pole tower and the longitude and latitude of the inspection photos, and automatically classify and bind the inspection photos to the nearest line pole tower.

[0016] S62. Rename the taken inspection photos, automatically rename the inspection photos according to the shooting order, and the naming format is: voltage level + line name + pole tower number + waypoint name + shooting date.

[0017] S63. Call the object detection algorithm model to detect potential hazards in the inspection photos. If the inspection photos for one pole with six photos of the pole tower are taken for the first time, call the trained yolov5 object detection algorithm for identification. If it is a resumption of flight inspection and there are original normal inspection photos, call the Cornernet-Compare object detection algorithm for detection.

[0018] S64. Through intelligent identification, detect whether there are construction machinery in the overall view of the tower, whether there are foreign objects such as bird nests in the tower head, whether the tower body is tilted, whether the information on the pole number plate is blurred, whether the tower base has sunk, and whether there are construction machinery, tower cranes, and abnormal smoke and fire in the channels on both the large and small side. If any abnormalities are detected, record the results, provide query results and lists, return the inspection hazard information for one pole with six photos, and support the export of the inspection hazard information for one pole with six photos.

[0019] Further preferably, the Cornernet-Compare object detection algorithm deletes the Offset and Height / Width outputs of Cornernet, only retains the Heatmap output, and adds a set of original normal inspection photos as control feature inputs, connecting the Heatmap features of the resumption of flight inspection photos and the original normal inspection photos to achieve difference differentiation.

[0020] During the forward propagation process, the go-around inspection photos and the original normal inspection photos are input simultaneously. Then, feature information is extracted through the backbone network respectively, and the Heatmap features of the go-around inspection photos and the original normal inspection photos are obtained through the deconvolution structure of the hourglass network, namely HeatmapC and HeatmapH respectively. HeatmapC and HeatmapH are added together to obtain HeatmapD, and HeatmapC, HeatmapD, and HeatmapH are output simultaneously. Among them, the key defect information is obtained by decoding HeatmapD.

[0021] During the backpropagation process, the loss calculation formula is as follows:

[0022] L output = k H L H + k D L D + k C L C

[0023] Where L output is the output loss, L H is the sum of squares between the output value of HeatmapH and the manually labeled sample Ground Truth, and k H is the sum of squares coefficient between the output value of HeatmapH and the manually labeled sample Ground Truth; L D is the sum of squares between the output value of HeatmapD and the manually labeled sample Ground Truth, and k D is the sum of squares coefficient between the output value of HeatmapD and the manually labeled sample Ground Truth; L C is the sum of squares between the output value of HeatmapC and the manually labeled sample Ground Truth, and k C is the sum of squares coefficient between the output value of HeatmapC and the manually labeled sample Ground Truth.

[0024] Further preferably, the specific process of step S2 is as follows:

[0025] S21. Obtain the position coordinates of the middle point at the top of the tower: In the three-dimensional tower point cloud, with the direction from the small side to the large side of the tower as the reference, mark the position coordinates of the left ground wire hanging point A(x1, y1, z1) and the right ground wire hanging point B(x2, y2, z2) from left to right respectively. The position coordinates of the middle point at the top of the tower are the middle point position connecting the left ground wire and the right ground wire, and the position coordinates of the middle point at the top of the tower C(x3, y3, z3) are calculated; x represents longitude, y represents latitude; z represents altitude;

[0026] S22. Obtain the position coordinates of the tower call sign, tower plate number, and tower base. Mark the position coordinates of the tower call sign D(x4, y4, z4), the position coordinates of the tower plate number E(x5, y5, z5), and the position coordinates of the tower base F(x6, y6, z6) from the three-dimensional tower point cloud. Among them, the shooting point position of the tower plate number is the marked position of the tower plate number; the shooting point position of the tower base is the marked position of the tower call sign;

[0027] S23. Calculate the position coordinates of the tower head shooting point G(x7, y7, z7). The tower head shooting point is at the middle position between the middle point of the tower top and the tower call sign;

[0028] S24. Calculate the position coordinates of the tower body shooting point H(x8, y8, z8). The tower body shooting point is at the middle position between the tower call sign position and the tower base position;

[0029] S25. Calculate the position coordinates of the full tower shooting point I(x9, y9, z9). The full tower shooting point is at the middle position between the middle point of the tower top and the tower base position;

[0030] S26. Through the above steps S22 to S25, the position coordinates of the shooting points of the full tower, tower head, tower body, tower plate number, and tower base have been obtained. Use these position coordinates of the shooting points to calculate the waypoint coordinates of the full tower, tower head, tower body, tower plate number, and tower base respectively.

[0031] Further preferably, the waypoint coordinate calculation method is as follows:

[0032] Assume the waypoint coordinates are: L(x, y, z); configure the flight angle α and the pan-tilt angle β of the waypoint. The flight angle refers to the horizontal angle between the vector with the tower top as the origin and from the tower top to the left side of the tower and the vector from the tower top to the shooting point ; configure the horizontal distance h and the vertical distance v; the horizontal distance refers to the planar distance between the shooting point and the tower top; when the vertical distance is positive, it represents the distance above the tower top, and when it is negative, it represents the distance above the tower base;

[0033] The process of calculating the waypoint is: calculate the planar coordinates of the waypoint through the horizontal distance h, that is, the distance dis on the extension line from the tower top to the left side of the tower. The formula is as follows:

[0034]

[0035] Get The coordinate K of the point on the extension line at a distance equal to the horizontal distance h from the tower top:

[0036]

[0037] Get the waypoint vector L:

[0038]

[0039] Then, create a rotation matrix M that rotates by the flight angle α around the z-axis:

[0040]

[0041] After the waypoint vector L is transformed by the rotation matrix M, it becomes:

[0042] L' = M * L;

[0043] L′ is the waypoint vector after being transformed by the rotation matrix;

[0044] Then, calculate the coordinates of the final waypoint through the gimbal angle. Assume the coordinates of the shooting point are (x0, y0, z0), and calculate the planar distance from the waypoint to the shooting point:

[0045]

[0046] Calculate the z coordinate of the waypoint. When the vertical distance v = 0, z = z0 - DH × tanβ; when the vertical distance v > 0, z = z3 + v; when the vertical distance v < 0, z = z6 – v; finally, obtain the waypoint coordinates L(x, y, z).

[0047] Through the above calculation process, calculate the waypoint coordinates of the entire tower, tower head, tower body, pole number plate, and tower base respectively. The waypoint positions of the large-size side channel and small-size side channel are unified above the tower top, and the waypoint coordinates of the large-size side channel and small-size side channel are L(x, y, z) = (x3, y3, z3 + v);

[0048] After obtaining the waypoint coordinates of one pole with six photos, calculate the nose direction angle γ of the UAV. The calculation method is as follows: Given the vector from the tower top to the left side of the tower From To The horizontal rotation angle is the flight angle α, then the horizontal rotation angle of the waypoint coordinates L(x, y, z) from being perpendicular to To Is 90° - α. The horizontal angle θ between the vector from the waypoint to the tower top And the vector from the waypoint to the shooting point Follows the following formula:

[0049]

[0050] First, find the dot product If dot<0, θ>0, the horizontal angle is 180° - θ; if dot<0, θ<0, the horizontal angle is -180° - θ; where

[0051] The nose direction angle γ = -(90° - α) + θ;

[0052] Through the above calculation process, the nose direction angles of the navigation points of the overall tower view, tower head, tower body, pole number plate, tower foundation, large-side channel, and small-side channel are calculated respectively.

[0053] Further preferably, in step S4, according to the order of the navigation points, it is judged whether the connection line between adjacent navigation points 1 and 2 is safe. The judgment method is as follows: Design an inspection function to judge whether the points on the connection line between navigation points 1 and 2 are inside the polygon. The polygon generation rule is: taking the pole tower as the center, generate a spatial bounding box that completely surrounds the entire pole tower and project it onto the ground to form a polygon; take the maximum height of the bounding box as h max ; judge whether the connection line between navigation points 1 and 2 is safe according to the following rules;

[0054] In the first case, neither navigation point 1 nor navigation point 2 is inside the polygon and their heights are both higher than the maximum height h of the bounding box max , then it is safe and no turning point needs to be added;

[0055] In the second case, neither navigation point 1 nor navigation point 2 is inside the polygon and there is a situation where the height is lower than the maximum height h of the bounding box max , then there may be insecurity and a turning point needs to be added; the coordinates of the turning point = the coordinates of navigation point 2, and the height is the maximum value of the height of navigation point 1 and the maximum height h of the bounding box max ;

[0056] In the third case, one of navigation points 1 and 2 is inside the polygon and the heights of both navigation points are greater than the maximum height h of the bounding box max , then it is safe and no turning point needs to be added;

[0057] In the fourth case, one of navigation points 1 and 2 is inside the polygon and the heights of both navigation points are less than the maximum height h of the bounding box max , then judge whether the distance between the two navigation points is less than the set safety distance of 3 meters. If so, it is safe and no turning point needs to be added; if not, a turning point needs to be added. The addition principle is as follows: Let d1 = the distance from navigation point 1 to the pole tower, d2 = the distance from navigation point 2 to the pole tower, then the coordinates of the turning point are equal to the coordinates of the navigation point corresponding to max{d1, d2}, and the height is equal to the height of the navigation point corresponding to min{d1, d2};

[0058] In the fifth case, both navigation points 1 and 2 are inside the polygon. If the heights are both greater than the maximum height h of the bounding box max , then it is safe and no turning point needs to be added;

[0059] In the sixth case, both navigation points 1 and 2 are inside the polygon. If the heights are both less than the maximum height h of the bounding box max, it is safe and no turning point needs to be added; otherwise, it is the same as the fourth case;

[0060] After adding turning points, connect them in sequence to generate a verified one-pole, six-license inspection route.

[0061] Further preferably, the specific process of step S5 is as follows:

[0062] S51, the drone loads the one-pole-six-photograph route of the pole tower, and performs automatic driving according to the waypoint. When the drone reaches the waypoint, the front-end intelligent recognition device built into the drone uses the yolov5 target detection algorithm to identify the patrol shooting target in the real-time image transmission screen of the drone, and obtains the bounding box surrounding the shooting target. According to the bounding box, the relative position of the shooting target and the shooting photo is calculated to determine whether it is in the center of the shooting. If so, take a photo, otherwise proceed to the next step;

[0063] S52, if the center of the shooting target is not at the center of the photograph, the gimbal angle is fine-tuned up and down, and the front-end intelligent recognition device performs a detection every time the gimbal angle is fine-tuned by 0.5°, wherein the gimbal angle is limited to be adjusted up and down within plus or minus 15°. Once it is detected that the center of the shooting target is at the center of the photograph, the adjustment is locked, the adjustment information is recorded, and the photograph is taken, otherwise, the next step is entered;

[0064] S53, when the gimbal angle cannot be locked at the center of the shooting target by fine-tuning the gimbal angle, the gimbal angle set at the waypoint is restored, and the direction of the drone head is adjusted left and right. The front-end intelligent recognition device performs a detection every time the angle is fine-tuned by 0.5°, wherein the left and right adjustment of the head direction is limited to no more than plus or minus 15°. Once it is detected that the center of the shooting target is at the center of the photo, the adjustment is locked, the adjustment information is recorded, and the photo is taken, otherwise, the next step is entered;

[0065] S54, when the automatic fine-tuning of the gimbal angle and the direction of the aircraft head cannot lock the shooting target at the shooting center, a prompt will pop up for manual confirmation, and manual adjustment of waypoints and taking photos are supported;

[0066] S55. According to steps S51-S54, the front-end intelligent recognition detection and fine-tuning are performed on different waypoints of the one-pole six-photograph system in turn, photos are taken, the waypoint adjustment records are saved, and the route is updated to a safe route, which can be used for the next automatic inspection of the go-around.

[0067] The present invention covers the entire process of one-pole six-photo inspection of transmission towers, including automatic driving route planning, route verification, front-end intelligent recognition and optimization of waypoints, and intelligent recognition and hidden danger analysis of inspection photos, and has the following advantages:

[0068] (1) High precision: The drone conducts patrols based on RTK high-precision positioning, collects and processes point cloud data, and the elevation result error is within ±10cm, which can provide professional accuracy guarantee.

[0069] (2) Technology maturity: Based on the three-dimensional route planning system of drone autonomous driving, this method further improves product functionality and ease of use

[0070] (3) Reliable results: Based on the tower top, pole number plate, tower base, and tower location points, the generated waypoints are more reliable, avoiding the need for a large amount of manual modulation of the fully automatically generated waypoints.

[0071] (4) Safety: The generated one-pole six-photon waypoints are automatically generated to avoid potential safety hazards caused by human operation of the drone. The generated route is highly accurate and the flight is safer

[0072] (5) Simple operation: The operation process is simplified so that team members can quickly master the method and complete the inspection of one transmission line tower with six lights.

[0073] (6) High degree of automation: Through front-end intelligent recognition technology, a round of one-pole six-photo inspection has been completed while optimizing waypoints.

[0074] (7) High operating efficiency: The Cornernet-Compare algorithm model improved by this method can identify defects and hidden dangers faster and more accurately, improve data processing quality, and save data processing time. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a schematic diagram of the neural network structure of the Cornernet-Compare target detection algorithm. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in further detail below.

[0077] A method for defect analysis of a transmission tower with one pole and six lights using a drone and target detection, the steps are as follows:

[0078] Step S1: Use a drone equipped with RTK positioning to inspect the transmission line towers, collect three-dimensional high-precision laser point cloud data, and load and present the three-dimensional tower point cloud.

[0079] Step S2: By simply marking in the three-dimensional tower point cloud, the one-pole six-photo waypoints are automatically calculated and generated, including the waypoint information of the tower, tower head, tower body, pole number plate, tower base, large side channel, and small side channel; the waypoint information includes the waypoint coordinates, the drone gimbal angle, and the nose direction; the detailed steps are as follows:

[0080] S21. Obtain the position coordinates of the middle point at the top of the tower. In the 3D tower point cloud, with the direction from the smaller side to the larger side of the tower as the reference, mark the position coordinates of the left ground wire hanging point A(x1, y1, z1) and the right ground wire hanging point B(x2, y2, z2) from left to right. The position coordinates of the middle point at the top of the tower are the middle point position connecting the left and right ground wires, and calculate the position coordinates of the middle point at the top of the tower C(x3, y3, z3); x represents longitude, y represents latitude; z represents altitude, and the calculation formula is as follows:

[0081] C(x3, y3, z3) = ((x1 + x2) / 2, (y1 + y2) / 2, (z1 + z2) / 2);

[0082] S22. Obtain the position coordinates of the tower call sign, pole number plate, and tower base. Mark the position coordinates of the tower call sign D(x4, y4, z4), the tower number plate position coordinates E(x5, y5, z5), and the tower base position coordinates F(x6, y6, z6) from the 3D tower point cloud. The shooting point position of the pole number plate is the marked tower number plate position; the shooting point position of the tower base is the marked tower call sign position.

[0083] S23. Calculate the position coordinates of the shooting point of the tower head G(x7, y7, z7). The shooting point position of the tower head is at the middle position between the middle point at the top of the tower and the tower call sign, and the calculation formula is as follows:

[0084] G(x7, y7, z7) = ((x3 + x4) / 2, (y3 + y4) / 2, (z3 + z4) / 2);

[0085] S24. Calculate the position coordinates of the shooting point of the tower body H(x8, y8, z8). The shooting point position of the tower body is at the middle position between the tower call sign position and the tower base position, and the calculation formula is as follows:

[0086] H(x8, y8, z8) = ((x4 + x6) / 2, (y4 + y6) / 2, (z4 + z6) / 2);

[0087] S25. Calculate the position coordinates of the shooting point of the overall tower view I(x9, y9, z9). The shooting point position of the overall tower view is at the middle position between the middle point at the top of the tower and the tower base position, and the calculation formula is as follows:

[0088] I(x9, y9, z9) = ((x3 + x6) / 2, (y3 + y6) / 2, (z3 + z6) / 2;

[0089] S26. Through the above steps S22 to S25, the position coordinates of the shooting points of the overall tower view, tower head, tower body, pole number plate, and tower base have been obtained. Use these position coordinates of the shooting points to calculate the waypoint coordinates of the overall tower view, tower head, tower body, pole number plate, and tower base respectively. The calculation method is as follows:

[0090] Assume that the waypoint coordinates are: L(x, y, z); the flight angle α, which is the horizontal angle between the vector from the top of the tower to the left side of the tower (with the top of the tower as the origin) and the vector from the top of the tower to the shooting point, and can be configured; the pan-tilt angle β of the waypoint can be configured. According to actual tests, the recommended optimal pan-tilt angles for the waypoints of the full view of the tower, tower head, tower body, pole number plate, tower base, large-side channel, and small-side channel are: -45°, -30°, -60°, -60°, -60°, -10°, -10° respectively. and the vector from the top of the tower to the shooting point The horizontal distance h can be configured, which is the planar distance between the shooting point and the top of the tower; the vertical distance v

[0091] can be configured. When it is positive, it represents the distance above the top of the tower, and when it is negative, it represents the distance above the tower base.

[0092]

[0093] The process of calculating the waypoint is as follows: Calculate the planar coordinates of the waypoint through the horizontal distance h, that is, the distance dis on the extension line from the top of the tower to the left side of the tower. The formula is as follows:

[0094]

[0095] Get The coordinate K of the point on the extension line whose distance from the top of the tower is equal to the horizontal distance h:

[0096]

[0097] Get the waypoint vector L:

[0098]

[0099] Then, create a rotation matrix M that rotates by the flight angle α around the z-axis:

[0100]

[0101] After the waypoint vector L is transformed by the rotation matrix M:

[0102] L' = M * L;

[0103] L′ is the waypoint vector after being transformed by the rotation matrix;

[0104] Then calculate the coordinates of the final waypoint through the pan-tilt angle. Assume that the coordinates of the shooting point are (x0, y0, z0), and the planar distance from the waypoint to the shooting point can be calculated:

[0105]

[0106] ​Calculate the z - coordinate of the waypoint. When the vertical distance v = 0, z = z0 - DH×tanβ; when the vertical distance v>0, z = z3 + v; when the vertical distance v<0, z = z6 - v. Finally, obtain the waypoint coordinates L(x, y, z).

[0107] Through the above calculation process, calculate the waypoint coordinates of the overall tower view, tower head, tower body, pole number plate, tower base respectively. The waypoint positions of the large - side channel and small - side channel are unified above the tower top. The waypoint coordinates of the large - side channel and small - side channel are L(x, y, z)=(x3, y3, z3 + v).

[0108] After obtaining the waypoint coordinates of one tower with six views, calculate the nose - direction angle γ of the UAV. The calculation method is as follows: Given the vector from the tower top to the left side of the tower From To The horizontal rotation angle is the flight angle α. Then the horizontal rotation angle of the waypoint coordinates L(x, y, z) from perpendicular to To Is 90° - α. The horizontal angle θ between the vector from the waypoint to the tower top And the vector from the waypoint to the shooting point Follows the following formula:

[0109]

[0110] First, calculate the dot product If dot<0, θ>0, the horizontal angle is 180° - θ; if dot<0, θ<0, the horizontal angle is - 180° - θ; where The nose - direction angle γ=-(90° - α)+θ.

[0111] Through the above calculation process, calculate the nose - direction angles of the waypoints of the overall tower view, tower head, tower body, pole number plate, tower base, large - side channel, and small - side channel respectively.

[0112] So far, the position coordinates, pan - tilt angles, and nose - direction of the waypoints of the overall tower view, tower head, tower body, pole number plate, tower base, large - side channel, and small - side channel have been obtained.

[0113] Step S3: Connect the waypoints of one tower with six views generated in step S2 in the order: overall tower - view waypoint, tower - head waypoint, tower - body waypoint, pole - number - plate waypoint, tower - base waypoint, large - side - channel waypoint, small - side - channel waypoint, to generate the UAV inspection route for one tower with six views of the pole tower.

[0114] Step S4, Route Safety Check. When the drone is flying, it flies in a straight line between waypoints, so the connected route may be unsafe. The present invention uses the following method to determine whether the route is safe and automatically adds turning points to the unsafe route to obtain the final route for the tower inspection by the drone.

[0115] In the order of waypoints, determine whether the connection line between adjacent waypoint 1 and waypoint 2 is safe. The determination method is as follows: Design an inspection function to determine whether the points on the connection line between waypoint 1 and waypoint 2 are inside the polygon. The polygon generation rule is: With the tower as the center, generate a spatial bounding box to completely enclose the entire tower and project it onto the ground to form a polygon; Take the maximum height of the bounding box as h max . Determine whether the connection line between waypoint 1 and waypoint 2 is safe according to the following rules.

[0116] In the first case, both waypoint 1 and waypoint 2 are not inside the polygon and their heights are both higher than the maximum height h of the bounding box max , then it is safe and no turning point needs to be added;

[0117] In the second case, both waypoint 1 and waypoint 2 are not inside the polygon and there is a situation where the height is lower than the maximum height h of the bounding box max (taking waypoint 2 being lower than h max as an example), then there may be insecurity and a turning point needs to be added; The coordinates of the turning point = the coordinates of waypoint 2, and the height is the maximum value of the height of waypoint 1 and the maximum height h of the bounding box max ;

[0118] In the third case, one of waypoint 1 and waypoint 2 is inside the polygon and the heights of both waypoints are greater than the maximum height h of the bounding box max , then it is safe and no turning point needs to be added;

[0119] In the fourth case, one of waypoint 1 and waypoint 2 is inside the polygon and the heights of both waypoints are less than the maximum height h of the bounding box max , then determine whether the distance between the two waypoints is less than the set safety distance of 3 meters. If so, it is safe and no turning point needs to be added; If not, a turning point needs to be added. The addition principle is as follows: Take d1 = the distance from waypoint 1 to the tower, d2 = the distance from waypoint 2 to the tower, then the coordinates of the turning point are equal to the coordinates of the waypoint corresponding to max{d1, d2}, and the height is equal to the height of the waypoint corresponding to min{d1, d2};

[0120] In the fifth case, both waypoint 1 and waypoint 2 are inside the polygon. If the heights are both greater than the maximum height h of the bounding box max , then it is safe and no turning point needs to be added;

[0121] In the sixth case, both waypoint 1 and waypoint 2 are inside the polygon. If the heights are both less than the maximum height h of the bounding box, max it is safe and no turning points need to be added; otherwise, it is the same as the fourth case;

[0122] After adding turning points according to the above method, connect them in sequence to generate a verified six-photo inspection route for one pole.

[0123] Step S5: The drone conducts a six-photo real-flight inspection for one pole and optimizes the waypoints. Based on the generated inspection route, the drone performs the first flight inspection. Combining with the built-in front-end intelligent recognition device of the drone, it identifies and detects the shooting target, optimizes the flight waypoints, takes pictures, and saves the adjusted waypoints. The specific steps are as follows:

[0124] S51: The drone loads the six-photo route for one pole and conducts automatic flight according to the waypoints. When the drone reaches a waypoint, the built-in front-end intelligent recognition device of the drone uses the yolov5 object detection algorithm to identify the inspection shooting target in the real-time image transmission screen of the drone, obtains the bounding box surrounding the shooting target, calculates the relative position of the shooting target to the taken photo according to the bounding box, and judges whether it is at the center of the photo. If so, take a picture; otherwise, go to the next step.

[0125] S52: If the center of the shooting target is not at the center of the taken photo, finely adjust the pan-tilt angle up and down. Each time the pan-tilt angle is finely adjusted by 0.5°, the front-end intelligent recognition device conducts a detection. The up and down adjustment of the pan-tilt angle is restricted not to exceed ±15°. Once it is detected that the center of the shooting target is at the center of the taken photo, lock the adjustment, record the adjustment information, and take a picture; otherwise, go to the next step.

[0126] S53: When the shooting target cannot be locked at the center position by finely adjusting the pan-tilt angle, restore the pan-tilt angle set by the waypoint and finely adjust the heading direction of the drone left and right. Each time the angle is finely adjusted by 0.5°, the front-end intelligent recognition device conducts a detection. The left and right adjustment of the heading direction is restricted not to exceed ±15°. Once it is detected that the center of the shooting target is at the center of the taken photo, lock the adjustment, record the adjustment information, and take a picture; otherwise, go to the next step.

[0127] S54: When the shooting target cannot be locked at the center of the photo by automatically finely adjusting the pan-tilt angle and the heading direction, a prompt is popped up for manual confirmation, and manual adjustment of the waypoints and taking pictures are supported.

[0128] S55: Conduct front-end intelligent recognition detection and fine adjustment in sequence for different waypoints of the six-photo inspection for one pole according to steps S51 - S54, perform taking pictures, save the waypoint adjustment records, and update the route to a safe route, which can be used for the next automatic inspection during the next re-flight.

[0129] Step S6, Intelligent Analysis of Inspection Photos for One Pole with Six Inspections: Using the yolov5 or Cornernet-Compare object detection algorithm, perform intelligent identification of potential hazards on the inspection photos. The steps are as follows:

[0130] S61. Export the inspection photos of one pole with six inspections, classify the inspection photos, and perform distance matching based on the longitude and latitude of the line pole tower and the longitude and latitude of the inspection photos, and automatically classify and bind the inspection photos to the nearest line pole tower;

[0131] S62. Rename the taken inspection photos, and automatically rename the inspection photos according to the shooting order. The naming format is: voltage level + line name + pole tower number + flight point name + shooting date.

[0132] S63. Call the object detection algorithm model to detect defects and potential hazards in the inspection photos. If it is the first inspection and photo shooting of one pole with six inspections, call the trained yolov5 object detection algorithm for identification. If it is a re-flight inspection and there are original normal inspection photos, call the improved Cornernet-Compare object detection algorithm for detection.

[0133] The Cornernet-Compare object detection algorithm is a neural network model of the Y structure that, based on the characteristics of the inspection photos of one pole with six inspections and the idea of the Cornernet object detection algorithm, uses Heatmap tags to mark the regions of interest, and then realizes efficient feature localization and recognition by calculating the differences in features in the regions of interest to complete the detection of defect components.

[0134] As Figure 1 shown, in the neural network structure part, the Cornernet-Compare object detection algorithm deletes the Offset and Height / Width outputs of Cornernet, only retains the Heatmap output, and adds a set of original normal inspection photos as control feature inputs, connecting the Heatmap features of the re-flight inspection photos and the original normal inspection photos to achieve difference discrimination.

[0135] In the forward propagation process, after inputting the re-flight inspection photos and the original normal inspection photos simultaneously, extract feature information through the backbone network respectively, and then obtain the Heatmap features of the re-flight inspection photos and the original normal inspection photos, namely HeatmapC and HeatmapH, through the deconvolution structure of the hourglass network respectively. HeatmapC and HeatmapH are added to get HeatmapD, and HeatmapC, HeatmapD, and HeatmapH are output simultaneously. Among them, the key defect information is obtained by decoding HeatmapD.

[0136] During the backpropagation process, the loss calculation formula of the original Cornernet object detection algorithm is as follows:

[0137] L output = L det + λL push + μL offset

[0138] where L output is the output loss, L det is the sum of squares between the output Heatmap features and the manually annotated sample GroundTruth, L push is the sum of squares of the sizes between the predicted bounding box and the true bounding box, and λ is the coefficient of the sum of squares of the sizes between the predicted bounding box and the true bounding box; L offset is the sum of squares of the ratios between the predicted bounding box and the true bounding box, and μ is the coefficient of the sum of squares of the ratios between the predicted bounding box and the true bounding box.

[0139] In the Cornernet-Compare object detection algorithm, the modified loss calculation formula is as follows:

[0140] L output = k H L H + k D L D + k C L C

[0141] where L output is the output loss, L H is the sum of squares between the output value of HeatmapH and the manually annotated sample Ground Truth, k H is the coefficient of the sum of squares between the output value of HeatmapH and the manually annotated sample Ground Truth, with a value of 0.2; L D is the sum of squares between the output value of HeatmapD and the manually annotated sample Ground Truth, k D is the coefficient of the sum of squares between the output value of HeatmapD and the manually annotated sample Ground Truth, with a value of 0.6; L C is the sum of squares between the output value of HeatmapC and the manually annotated sample Ground Truth, k C is the coefficient of the sum of squares between the output value of HeatmapH and the manually annotated sample Ground Truth, with a value of 0.2.

[0142] The Cornernet-Compare object detection algorithm can directly output the defect heatmap for each category. Compared with the original Cornernet object detection algorithm, the Cornernet-Compare object detection algorithm adds the extraction and comparison of the features of the sample inspection photos, can extract the differences between the feature samples in two inspection photos. After actual testing, the improved algorithm can accurately and quickly locate and identify features, find out potential defect hazards, and mark the positions.

[0143] S64. Through intelligent identification, detect whether there are construction machinery in the overall view of the tower, whether there are bird's nest foreign objects on the tower head, whether the tower body is tilted, whether the information of the pole number plate is blurred, whether the tower foundation has sunk, and whether there are construction machinery, tower cranes, and abnormal smoke and fire in the side channels on both the large and small number sides. If any abnormality is detected, record the results, provide query results and lists, return the hidden danger information of the one-pole six-photo inspection, and support the export of the hidden danger information of the one-pole six-photo inspection.

Claims

1. A method for defect analysis of six - camera UAV inspection and target detection for a transmission tower pole, characterized in that, The steps are as follows: Step S1: Use a drone equipped with RTK positioning to inspect transmission line towers, collect three-dimensional high-precision laser point cloud data, and load and present the three-dimensional tower point cloud; Step S2: Automatically calculate and generate six flight points for one tower by simply marking in the three-dimensional tower point cloud, including flight point information for the overall tower view, tower head, tower body, pole number plate, tower base, large-side channel, and small-side channel; the flight point information includes flight point coordinates, drone gimbal angle, and aircraft nose direction; Step S3: Connect the six flight points for one tower generated in Step S2 in the order: overall tower view flight point, tower head flight point, tower body flight point, pole number plate flight point, tower base flight point, large-side channel flight point, and small-side channel flight point to generate a six-flight-point drone inspection route for the tower; Step S4: Route safety verification: Judge whether the route is safe, and automatically add turning points to the unsafe route to obtain the final drone inspection route for the tower; Step S5: Drone six-flight-point actual flight inspection and optimize flight points: Based on the generated inspection route, the drone performs the first flight inspection, combines with the built-in front-end intelligent recognition device of the drone, identifies and detects the shooting target, optimizes the flight points, takes pictures, and saves the adjusted flight points; Step S6: Intelligent analysis of six-flight-point inspection photos: Use the yolov5 or Cornernet-Compare object detection algorithm to perform intelligent identification of potential hazards in the inspection photos; the Cornernet-Compare object detection algorithm is a neural network model of the Y structure that is tailored to the characteristics of six-flight-point inspection photos, based on the Cornernet object detection algorithm, uses Heatmap labels to mark the regions of interest, and then realizes feature localization and recognition by calculating the differences in features in the regions of interest to complete the detection of defect components; 2. A method for inspecting a transmission tower with six cameras by an unmanned aerial vehicle and analyzing defects in target detection according to claim 1, characterized in that The specific process of Step S6 is as follows: S61: Export the six-flight-point inspection photos taken, classify the photos, and perform distance matching based on the longitude and latitude of the line tower and the longitude and latitude of the inspection photos, and automatically classify and bind the inspection photos to the nearest line tower; S62: Rename the taken inspection photos, and automatically rename the inspection photos according to the shooting order. The naming format is: voltage level + line name + tower number + flight point name + shooting date; S63: Call the object detection algorithm model to detect potential hazards in the inspection photos. If it is the first inspection and photo shooting for the six-flight-point tower, call the trained yolov5 object detection algorithm for identification. If it is a re-flight inspection and there are original normal inspection photos, call the Cornernet-Compare object detection algorithm for detection; S64: Through intelligent identification, detect whether there are construction machinery in the overall tower view, whether there are bird nests or foreign objects in the tower head, whether the tower body is tilted, whether the pole number plate information is blurred, whether the tower base has sunk, and whether there are construction machinery, tower cranes, or abnormal smoke and fire in the large and small side channels. If any abnormalities are detected, record the results, provide query results and lists, return the six-flight-point inspection potential hazard information, and support the export of six-flight-point inspection potential hazard information.

3. A method for inspecting a transmission tower with six cameras by an unmanned aerial vehicle and analyzing defects in target detection according to claim 1, characterized in that The Cornernet-Compare object detection algorithm deletes the Offset and Height / Width outputs of Cornernet, only retains the Heatmap output, and adds a set of original normal inspection photos as control feature inputs, connecting the Heatmap features of the go-around inspection photos and the original normal inspection photos to achieve difference discrimination; In the forward propagation process, after simultaneously inputting the go-around inspection photos and the original normal inspection photos, the feature information is extracted through the backbone network respectively, and then the Heatmap features of the go-around inspection photos and the original normal inspection photos are obtained through the deconvolution structure of the hourglass network, namely HeatmapC and HeatmapH. HeatmapC and HeatmapH are added to get HeatmapD, and HeatmapC, HeatmapD, and HeatmapH are output simultaneously. Among them, the key defect information is obtained by decoding HeatmapD; In the backpropagation process, the loss calculation formula is as follows: L output = k H L H + k D L D + k C L C ; Among them, L output is the output loss, and L H is the sum of squares between the output value of HeatmapH and the manually annotated sample Ground Truth, and k H is the sum of squares coefficient between the output value of HeatmapH and the manually annotated sample Ground Truth; L D is the sum of squares between the output value of HeatmapD and the manually annotated sample Ground Truth, and k D is the sum of squares coefficient between the output value of HeatmapD and the manually annotated sample Ground Truth; L C is the sum of squares between the output value of HeatmapC and the manually annotated sample Ground Truth, and k C is the sum of squares coefficient between the output value of HeatmapH and the manually annotated sample Ground Truth.

4. A method for inspecting a transmission tower with six cameras by an unmanned aerial vehicle and analyzing defects in target detection according to claim 1, characterized in that The specific process of step S2 is as follows: S21. Obtain the position coordinates of the middle point of the tower top: In the three-dimensional tower pole point cloud, with the direction from the small side to the large side of the tower pole as the reference, mark the position coordinates of the left ground wire hanging point A(x1, y1, z1) and the right ground wire hanging point B(x2, y2, z2) from left to right. The position coordinates of the middle point of the tower top are the middle point position connecting the left ground wire and the right ground wire, and calculate the position coordinates of the middle point of the tower top C(x3, y3, z3); x represents longitude, y represents latitude; z represents altitude; S22. Obtain the position coordinates of the tower nameplate, pole number plate, and tower base. Mark the position coordinates of the tower nameplate D(x4, y4, z4), the position coordinates of the tower number plate E(x5, y5, z5), and the position coordinates of the tower base F(x6, y6, z6) from the three-dimensional tower pole point cloud. Among them, the shooting point position of the pole number plate is the marked position of the tower number plate; the shooting point position of the tower base is the marked position of the tower nameplate; S23. Calculate the position coordinates of the tower head shooting point G(x7, y7, z7). The tower head shooting point is at the middle position between the tower top middle point and the tower nameplate; S24. Calculate the position coordinates of the tower body shooting point H(x8, y8, z8). The tower body shooting point is at the middle position between the tower nameplate position and the tower base position; S25. Calculate the position coordinates of the full view shooting point of the tower I(x9, y9, z9). The full view shooting point of the tower is at the middle position between the tower top middle point and the tower base position; S26. Through the above steps S22 to S25, the position coordinates of the shooting points of the full view of the tower, the tower head, the tower body, the pole number plate, and the tower base have been obtained. Use these shooting point position coordinates to calculate the waypoint coordinates of the full view of the tower, the tower head, the tower body, the pole number plate, and the tower base respectively.

5. A method for inspecting a transmission tower with six cameras by an unmanned aerial vehicle and analyzing defects in target detection according to claim 4, characterized in that, The waypoint coordinate calculation method is as follows: Suppose the waypoint coordinates are: L(x, y, z); configure the flight angle α and the gimbal angle β of the waypoint. The flight angle refers to the horizontal angle between the vector with the tower top as the origin and from the tower top to the left side of the tower , and the vector from the tower top to the shooting point ; configure the horizontal distance h and the vertical distance v; the horizontal distance refers to the planar distance between the shooting point and the tower top; when the vertical distance is positive, it represents the distance above the tower top, and when it is negative, it represents the distance above the tower base; The process of calculating the waypoint is: Calculate the plane coordinates of the waypoint through the horizontal distance h, that is, the distance dis on the extension line from the tower top to the left side of the tower. The formula is as follows: ； Obtain Coordinate K at a distance equal to the horizontal distance h from the top of the tower on the extension line: ； Get the waypoint vector L: ； Then, create a rotation matrix M that rotates the flight angle α around the z-axis: ； The waypoint vector L after transformation by the rotation matrix M is: ； is the waypoint vector after rotation matrix transformation; Then, calculate the coordinates of the final waypoint through the pan-tilt angle. Assume the coordinates of the shooting point are (x0, y0, z0), and calculate the planar distance from the waypoint to the shooting point: ； Calculate the z coordinate of the waypoint. When the vertical distance v = 0, z = z0 - DH × tanβ; when the vertical distance v > 0, z = z3 + v; when the vertical distance v < 0, z = z6 - v; finally, obtain the waypoint coordinates L(x, y, z); Through the above calculation process, calculate the waypoint coordinates of the overall tower view, tower head, tower body, pole number plate, and tower base respectively; The waypoint positions of the large-side channel and small-side channel are both above the tower top. The waypoint coordinates L(x, y, z) of the large-side channel and small-side channel are (x3, y3, z3 + v); After obtaining the waypoint coordinates of a six-shot pole, calculate the nose direction angle γ of the UAV. The calculation method is as follows: Given the vector from the top of the tower to the left side of the tower , from to the horizontal rotation angle is the flight angle α. Then the horizontal rotation angle of the waypoint coordinates L(x, y, z) from perpendicular to to is 90° - α. The horizontal angle θ between the vector from the waypoint to the top of the tower and the vector from the waypoint to the shooting point follows the following formula: ； First, calculate the dot product , if dot < 0 and θ > 0, the horizontal angle is 180° - θ; if dot < 0 and θ < 0, the horizontal angle is -180° - θ; where ; the nose direction angle γ = -(90° - α) + θ; Through the above calculation process, calculate the nose direction angles of the waypoints of the overall tower view, tower head, tower body, pole number plate, tower base, large-side channel, and small-side channel respectively.

6. A method for inspecting a transmission tower with six cameras by an unmanned aerial vehicle and analyzing defects in target detection according to claim 1, characterized in that, In step S4, in the order of waypoints, it is determined whether the connection line between adjacent waypoint 1 and waypoint 2 is safe. The determination method is as follows: A check function is designed to determine whether the points on the connection line between waypoint 1 and waypoint 2 are inside the polygon. The polygon generation rule is: Taking the tower as the center, a spatial bounding box is generated to completely enclose the entire tower and projected onto the ground to form a polygon; the maximum height of the bounding box is taken as h max ; Judge whether the line connecting waypoint 1 and waypoint 2 is safe according to the following rules; In the first case, neither waypoint 1 nor waypoint 2 is inside the polygon and their heights are both higher than the maximum height h of the bounding box max , then it is safe and no turning points need to be added; In the second case, neither waypoint 1 nor waypoint 2 is inside the polygon and there is a height lower than the maximum height h of the bounding box max In this case, there may be an insecurity and turning points need to be added; the coordinates of the turning point = the coordinates of waypoint 2, and the height is the maximum value of the height of waypoint 1 and the maximum height h of the bounding box max ; In the third case, one of waypoint 1 and waypoint 2 is inside the polygon, and the heights of both waypoints are greater than the maximum height h of the bounding box max , then it is safe and no turning point needs to be added; In the fourth case, one of waypoint 1 and waypoint 2 is inside the polygon, and the heights of both waypoints are less than the maximum height h of the bounding box max , then determine whether the distance between the two waypoints is less than the set safety distance of 3 meters. If it is, it is safe and no turning point needs to be added; if not, a turning point needs to be added. The addition principle is as follows: let d1 be the distance from waypoint 1 to the tower pole, and d2 be the distance from waypoint 2 to the tower pole. Then the coordinates of the turning point are equal to the coordinates of the waypoint corresponding to max{d1, d2}, and the height is equal to the height of the waypoint corresponding to min{d1, d2}; In the fifth case, both waypoint 1 and waypoint 2 are inside the polygon. If their heights are both greater than the maximum height h of the bounding box max , then it is safe and no turning points need to be added; In the sixth case, both waypoint 1 and waypoint 2 are inside the polygon. If their heights are both less than the maximum height h of the bounding box max , then it is safe and no turning points need to be added; otherwise, it is the same as the fourth case; After adding turning points, connect them in sequence to generate a verified one-pole six-shot inspection route.

7. A method for inspecting a transmission tower with six cameras by an unmanned aerial vehicle and analyzing defects in target detection according to claim 1, characterized in that, The specific process of step S5 is as follows: S51. The drone loads the one-pole six-shot route of the tower and performs autopilot according to the waypoints. When the drone reaches a waypoint, the built-in front-end intelligent recognition device in the drone uses the yolov5 target detection algorithm to recognize the inspection shooting target in the real-time image transmission screen of the drone, obtain the bounding box surrounding the shooting target, calculate the relative position of the shooting target to the taken photo according to the bounding box, and judge whether it is at the center of the shooting. If so, take a photo; otherwise, go to the next step; S52. If the center of the shooting target is not at the center of the taken photo, finely adjust the pan-tilt angle up and down. Each time the pan-tilt angle is finely adjusted by 0.5°, the front-end intelligent recognition device performs a detection. The up and down adjustment of the pan-tilt angle is restricted not to exceed plus or minus 15°. Once it is detected that the center of the shooting target is at the center of the taken photo, lock the adjustment, record the adjustment information, and take a photo; otherwise, go to the next step; S53. When the shooting target cannot be locked at the center position by finely adjusting the pan-tilt angle, restore the pan-tilt angle set by the waypoint and finely adjust the nose direction of the drone left and right. Each time the angle is finely adjusted by 0.5°, the front-end intelligent recognition device performs a detection. The left and right adjustment of the nose direction is restricted not to exceed plus or minus 15°. Once it is detected that the center of the shooting target is at the center of the taken photo, lock the adjustment, record the adjustment information, and take a photo; otherwise, go to the next step; S54. When the shooting target cannot be locked at the center of the shooting by automatically fine-tuning the pan-tilt angle and the nose direction, a prompt is popped up for manual confirmation, and manual adjustment of the waypoint and taking photos are supported; S55. Perform front-end intelligent recognition detection and fine-tuning in sequence for different waypoints of one-pole six-shot according to steps S51 - S54, execute taking photos, save the waypoint adjustment records, and update the route to a safe route. This route is used for automatic inspection during the next takeoff.

Images