An automatic image matching and archiving method based on UAV waypoint and ground wire spatial location information

Abstract

This invention provides an automatic image matching and archiving method based on the spatial location information of UAV waypoints and conductor ground wires. The method includes: point cloud classification; extraction of the tower's point cloud; calculation of the coordinates of the tower's crossarm center point, treating it as the tower's center point; extraction of the tower's center point's GPS information; statistical analysis of the GPS information of all waypoints photographed in the flight path file used in a conductor ground wire inspection operation; calculation of the horizontal distance from the smaller tower to the projected waypoint after projection onto the line connecting the center points of adjacent towers, based on the extracted GPS information of the tower's center point and waypoints; mapping waypoints and photos one-to-one according to their coordinate positions and the order of photographing, completing photo archiving; and naming the photos according to the obtained horizontal distance of the waypoints from the smaller towers and the correspondence between photos and waypoints, and naming all photos sequentially. This invention can effectively correlate inspection operation images with specific conductor ground wires, while reducing the workload of operators.

Description

Technical Field

[0001] This invention relates to the field of automatic archiving of UAV inspection images of guide and ground lines, and proposes an automatic image matching and archiving method based on the spatial location information of UAV waypoints and guide and ground lines. Background Technology

[0002] Overhead conductor and ground wire breaks have always been a significant threat to the safe and stable operation of power transmission networks, and such incidents occur frequently. For example, conductors may break under normal operating conditions due to steel core corrosion caused by loose strands. Before a conductor or ground wire breaks, there are usually defects such as broken strands, loose strands, and corrosion to varying degrees. Broken strands and loose strands are often more common defects in conductors and ground wires, and the causes are varied.

[0003] During routine inspections, frontline teams occasionally discover broken or scattered strands in conductors using traditional methods. However, these discoveries are often accidental, and when they are found, they are usually critical issues involving broken or scattered strands. Many other broken or scattered strand problems are missed by traditional inspection methods, and numerous minor issues go undetected. Therefore, conducting close-range, high-precision autonomous inspections of conductors and ground wires using drones is of significant practical importance. This allows for parallel, equidistant, efficient, complete, and standardized image acquisition of conductors and ground wires, supporting the optimization of intelligent image recognition models and preventing conductor breakage accidents.

[0004] Conductor and ground wire inspections, whether manually operated or autonomously flown, generate a large number of photos. Because drone photos are named with sequential numbers, it is difficult, if not impossible, for staff to match the photos with the corresponding power lines. Therefore, renaming the photos and assigning them to the power lines between the photographed towers is particularly important. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic image matching and archiving method based on the spatial location information of UAV waypoints and ground wires, which can effectively match the inspection operation image results with specific ground wires, while reducing the workload of operators.

[0006] An automatic image matching and archiving method based on the spatial location information of UAV waypoints and ground wires includes the following steps:

[0007] Step 1: Point cloud classification, extract point cloud of the tower body, calculate the coordinates of the center point of the tower crossarm and regard it as the center point of the tower, and extract the GPS information of the center point of the tower.

[0008] Step 2: Compile GPS information for all photographed waypoints in the route file used in a particular conductor and ground wire inspection operation;

[0009] Step 3: Based on the GPS information of the tower center point and waypoint extracted in Step 1 and Step 2, calculate the horizontal distance of the waypoint from the smaller tower after it is projected onto the line connecting the center points of its adjacent towers.

[0010] Step 4: Based on the time sequence of the photos, match the waypoints with the photos according to their coordinates and the order in which they were taken, and complete the photo archiving;

[0011] Step 5: Based on the horizontal distance between the waypoint and the small tower obtained in Step 3 and the correspondence between the photos and waypoints in Step 4, set them as the corresponding photo names, and name all photos in sequence.

[0012] Further, in step 1, point cloud classification is performed, the point cloud of the tower body is extracted, the coordinates of the center point of the tower crossarm are calculated and regarded as the center point of the tower, and its GPS information is extracted, specifically including:

[0013] First, the point cloud data is projected onto the bottom surface to achieve orientation correction, and then the tower image is generated by projecting onto the front and side surfaces.

[0014] Secondly, the collected tower images were used to form a dataset to train a VGG16-structured convolutional autoencoder network, and the encoded part of the network was extracted.

[0015] Next, the tower diagram is input into the coding network to obtain feature vectors. The feature vectors are then input into the EM classifier to achieve automatic classification. After classification, the latitude and longitude of the tower crossarm center point are calculated and regarded as the tower center point.

[0016] Furthermore, the GPS information of the waypoints collected in step 2 includes the waypoint name, longitude, latitude, altitude, gimbal pitch angle, gimbal deflection angle, and effective focal length.

[0017] Furthermore, step 3, based on the GPS information of the tower center point and waypoint extracted in steps 1 and 2, calculates the horizontal distance L from the smaller tower after the waypoint is projected onto the line connecting the center points of its adjacent towers. Specifically, this includes:

[0018] Establish the centerline l based on the coordinates of the center point O1 of the smaller tower and the center point O2 of the larger tower. 1，2 The linear equation is obtained, and the coefficients A0, B0, and C0 are found:

[0019] A0x+B0y+C0=0

[0020] Then calculate waypoint P. i to the center line of the tower l 1，2 Distance d:

[0021]

[0022] Finally, based on the calculated waypoint P i to the center line of the tower l 1，2 The distance d, the tower center point O1, and the waypoint P i Find the coordinates of Pi The distance s to O1 i , and the waypoint P is obtained according to the Pythagorean theorem i The projection distance L to the center line l of the tower pole 1，2 i :

[0023]

[0024] The projection distance L i is stored sequentially.

[0025] Furthermore, in step 4, according to the time sequence of taking pictures, the waypoints are corresponded to the photos one by one to complete photo filing, which specifically includes:

[0026] The UAV flies in sequence according to the waypoints and performs shooting tasks at each waypoint. Therefore, the time sequence of taking pictures is the sequence of the waypoints. The photos are stored according to the data directory and the output directory. Among them, the data directory includes the original photo folder, the flight route file and the tower pole file; the output directory includes the renamed photo folder, the json marking file and the xls table file.

[0027] Furthermore, in step 5, combining the horizontal distance of the waypoint from the small-sized tower pole obtained in step 3 and the corresponding result of the photo and the waypoint in step 4, set them as the corresponding photo names and name all the photos sequentially, which specifically includes:

[0028] All the waypoints in the flight route are processed in step 3 sequentially. All the photos are sorted according to the time sequence of taking pictures, and the waypoints and the photos are corresponded to each other according to the sequence of the waypoints. At this time, the i-th photo corresponds to the waypoint P i , and name the photo with "distance to the small-sized tower L i meters" as the one corresponding to P i to complete the batch renaming of the photos in the flight route.

[0029] Through the GPS and gimbal information of the waypoints of the UAV taking pictures, the relative position between the UAV and the shooting target during the operation is obtained. Since the UAV flies in sequence according to the waypoints and performs shooting tasks at each waypoint, the time sequence of taking pictures is the sequence of the waypoints, and only one photo is taken at each waypoint. By assigning the horizontal distance L of the projection of the waypoint on the connecting line of the center points of its adjacent tower poles to each photo in sequence, the batch renaming of the photos can be completed. The process is clear, the response is fast, and the effect is good. Brief Description of the Drawings

[0030] Figure 1 It is a schematic diagram of the GPS information of the waypoints in the embodiment of the present invention;

[0031] Figure 2 ​This is a schematic diagram showing the horizontal distance from the smaller tower to the projected waypoint on the line connecting the center points of its adjacent towers, according to an embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram of the data catalog of an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the output directory of an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of the photo archiving results of an embodiment of the present invention;

[0035] Figure 6 This is a flowchart illustrating an automatic image matching and archiving method based on the spatial location information of UAV waypoints and ground wires, according to an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] like Figure 6 As shown, this embodiment of the invention provides an automatic image matching and archiving method based on the spatial location information of UAV waypoints and ground wires, including the following steps:

[0038] Step 1: Point cloud classification, extract point cloud of the tower body, calculate the coordinates of the center point of the tower crossarm and regard it as the center point of the tower, and extract the GPS information of the center point of the tower.

[0039] This invention utilizes a convolutional autoencoder neural network and the EM clustering algorithm to automate the classification of pole tower point cloud data. Step 1 specifically includes:

[0040] First, the point cloud data is projected onto the bottom plane to achieve orientation correction. Then, front and side projections are used to generate tower images. Since the LiDAR cannot guarantee that the scanning plane is always parallel to the front of the tower when scanning, there is an angle between the tower point cloud and the world coordinate system. Therefore, the coordinates of the tower point cloud need to be rotated and translated so that the axes of the circumscribed ellipsoid of the tower point cloud are approximately parallel to the axes of the world coordinate system. The projection formula from the world coordinate system to the image coordinate system for the tower point cloud data is as follows:

[0041]

[0042] Where x, y, and v represent the width, height, and pixel value in the image coordinate system, respectively, and X, Y, and Z represent the coordinates of the point cloud data in the world coordinate system.

[0043] Secondly, the collected tower images are used to form a dataset to train a VGG16 convolutional autoencoder network. The encoded part of the network is extracted as a feature encoder to complete the mapping from image to vector, that is, image feature extraction.

[0044] Next, the tower diagram is input into the coding network to obtain feature vectors. The feature vectors are then input into the EM classifier to achieve automatic classification. After classification, the latitude and longitude of the tower crossarm center point are calculated and regarded as the tower center point.

[0045] Step 2: Compile GPS information for all waypoints photographed in the route file used in a certain conductor and ground wire inspection operation; the GPS information includes the waypoint number, name, longitude, latitude, altitude, route name, tower number, conductor or ground wire, left and right phase, etc.

[0046] Step 3: Based on the GPS information of the tower center point and waypoint extracted in Step 1 and Step 2, calculate the horizontal distance from the smaller tower after the waypoint is projected onto the line connecting the center points of its adjacent towers; Step 3 specifically includes:

[0047] Based on the coordinates of the center points O1 of the smaller tower and O2 of the larger tower, establish the linear equation of the center line l1, 2, and obtain the coefficients A0, B0, and C0:

[0048] A0x+B0y+C0=0

[0049] Then calculate waypoint P. i to the center line of the tower l 1，2 Distance d:

[0050]

[0051] Finally, based on the calculated waypoint P i to the center line of the tower l 1，2 The distance d, the tower center point O1, and the waypoint P i Find the coordinates of P i Distance s to O1 i The waypoint P is obtained using the Pythagorean theorem. i to the center line of the tower l 1，2 Projection distance L i :

[0052]

[0053] Projection distance L i Store sequentially.

[0054] Step 4: According to the chronological order of taking photos, correspond the waypoints and photos one by one according to the coordinate positions and the order of taking photos to complete photo archiving. Step 4 specifically includes:

[0055] The drone flies in sequence according to the waypoints and performs shooting tasks at each waypoint. Therefore, the chronological order of taking photos is the sequence of waypoints. Store the photos according to the data directory and the output directory. Among them, the data directory includes the original photo folder, the flight route file, and the tower file; the output directory includes the renamed photo folder, the json marker file, and the xls table file.

[0056] Step 5: According to the horizontal distance from the waypoint to the small-numbered tower obtained in Step 3 and the corresponding result of the photo and the waypoint in Step 4, set it as the corresponding photo name and name all the photos in sequence. Step 5 specifically includes:

[0057] Complete the processing in Step 3 for all the waypoints in the flight route in sequence, sort all the photos according to the chronological order of taking photos, and correspond the waypoints and photos one by one according to the sequence of waypoints. At this time, the i-th photo corresponds to the waypoint P i , and name the photo corresponding to P i with "L meters away from the small-numbered tower" as P i corresponding, and batch-rename the photos of the flight route will be completed.

[0058] Take the right-phase power line of the 17-18th tower of a certain line as an example to perform drone operations and automatic archiving and renaming of the operation results.

[0059] First, classify the sample point cloud data to obtain the point cloud data of the tower body, and record the tower information in the tower file (coordinates of the 17th tower (114°45’51”E, 30°31’22”N), coordinates of the 18th tower (114°45’41”E, 30°31’34”N)),

[0060] Complete the tower flight route planning for this operation, and record the waypoint information in the flight route as Figure 1 shown:

[0061] As Figure 2 shown, establish the linear equation of the center line l 1，2 of according to the center point coordinates of the two towers,

[0062] A0x + B0y + C0 = 0

[0063] Then calculate the distance d1 from the waypoint P1 to the tower center line l 1，2 :

[0064]

[0065] And obtain P according to the coordinates of the center point O1 of the small-numbered tower and the waypoint P1 iThe distance s1 to O1;

[0066] Finally, using the Pythagorean theorem, the distance l from waypoint P1 to the center line of the tower is obtained. 1，2 The projection distance L1.

[0067]

[0068] Sort all photos according to the order in which they were taken, and then match waypoints with photos one by one according to the order in which waypoints were taken. At this point, the first photo corresponds to waypoint P1, so the photo corresponding to P1 is named "4.6 meters away from the small tower".

[0069] Store photos according to a data directory and an output directory. The data directory ( Figure 3 It contains the original photo folder, flight path files, and tower files, and the output directory ( Figure 4 It includes renaming photo folders, JSON tag files, and XLS spreadsheet files.

[0070] The folder naming convention is as follows:

[0071] xx line (line name) #yy (small pole number) - xx line (line name) #zz (large pole number) - mm line (conductor or ground wire) - n phase (left phase or right phase)

[0072] Repeat the above steps in sequence, as follows: Figure 5 The image shows the completion of archiving all waypoint photos.

[0073] This invention obtains the relative position of the drone and the target during operation by using GPS and gimbal information of the drone's waypoints. Since the drone flies sequentially according to the waypoints and performs the shooting task at each waypoint, the time order of taking pictures is the order of the waypoints. Each waypoint only takes one picture. The horizontal distance L from the smaller tower to the projection of the waypoint onto the line connecting the center points of its adjacent towers is assigned to each picture in turn, which can complete the batch renaming of pictures. The process is clear, the response is fast and the effect is good.

[0074] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An automatic picture matching and archiving method based on unmanned aerial vehicle waypoint and ground wire spatial position information, characterized in that, Comprising the following steps: Step 1: point cloud classification, extracting the tower body point cloud, calculating the tower cross arm center point coordinates, and taking the tower center point as the tower center point, extracting the GPS information of the tower center point; Step 2: statistics of the GPS information of all photographing waypoints in the flight path file used in a ground wire inspection operation; Step 3: according to the GPS information of the tower center point and the waypoint extracted in steps 1 and 2, the horizontal distance of the waypoint from the small tower after projection on the adjacent tower center point line is calculated; Step 4: according to the time sequence of photographing, the waypoint and the photo are one-to-one corresponding according to the coordinate position and the photographing order, and the photo archiving is completed; Step 5: according to the horizontal distance of the waypoint from the small tower obtained in step 3 and the corresponding result of the photo and the waypoint in step 4, set it as the corresponding photo name, and name all photos in turn.

2. The method of claim 1, wherein the method further comprises: Step 1: point cloud classification, extracting the tower body point cloud, calculating the tower cross arm center point coordinates, and taking the tower center point as the tower center point, extracting the GPS information, including: Firstly, the point cloud data is projected on the bottom surface to realize direction correction, and then the tower image is generated by front and side projection; Secondly, the collected tower images are composed into a data set to train the convolutional auto-encoding network with VGG16 structure, and the encoding part in the network is extracted; Thirdly, the tower image is input into the encoding network to obtain the feature vector, which is input into the EM classifier to realize automatic classification. After classification, the tower cross arm center point longitude and latitude are calculated and taken as the tower center point.

3. The method of claim 1, wherein: The GPS information of the photographing waypoint in step 2 includes waypoint name, longitude, latitude, height, gimbal pitch angle, gimbal yaw angle and effective focal length.

4. The method of claim 1, wherein: Step 3: according to the GPS information of the tower center point and the waypoint extracted in steps 1 and 2, the horizontal distance L of the waypoint from the small tower after projection on the adjacent tower center point line is calculated, including: The linear equation of the center line l is established according to the coordinates of the small tower center point O1 and the large tower center point O2, and the coefficients A0, B0 and C0 are obtained. 1，2 The linear equation of the center line l is established according to the coordinates of the small tower center point O1 and the large tower center point O2, and the coefficients A0, B0 and C0 are obtained. ； Then the waypoint P is calculated i to the tower centerline l 1，2 :​ ； Finally, based on the calculated waypoint P i to the center line of the tower l 1，2 distance And the tower center point O1 and waypoint P i Find the coordinates of P i Distance to O1 The waypoint P is obtained using the Pythagorean theorem. i to the center line of the tower l 1，2 Projection distance : ； The projection distance are stored sequentially.

5. The method of claim 1, wherein: Step 4: according to the time sequence of photographing, the waypoint and the photo are one-to-one corresponding, and the photo archiving is completed, including: The UAV flies according to the waypoints in turn and performs the shooting task at each waypoint. Therefore, the time sequence of photographing is the sequence of the waypoints. The photos are stored according to the data directory and the output directory, wherein the data directory includes the original photo folder, the flight path file and the tower file; the output directory includes the renamed photo folder, the json mark file and the xls table file.

6. The method of claim 4, wherein: Step 5: combining the horizontal distance of the waypoint from the small tower obtained in step 3 and the corresponding result of the photo and the waypoint in step 4, setting it as the corresponding photo name, and naming all photos in turn, including: All the steps in step 3 are completed in sequence for all the waypoints in the route, all the photos are sorted according to the photo shooting time sequence, and the waypoints are one-to-one corresponding to the photos according to the waypoint sequence, at this time the i-th photo corresponds to the waypoint P i "Distance small tower L i m" is P i Corresponding photo naming, complete the batch renaming of the route photos.

Images