Unmanned aerial vehicle picture-based heat pipe rendering correction method and device, and medium

Abstract

The application discloses a heat pipe rendering and rectification method and device based on a UAV picture, and a medium, and belongs to the technical field of heat pipe inspection, and is used for solving the technical problem that the picture information and the UAV and the suspended state information are not synchronized when the existing UAV inspects a heat pipe, the position of the underground heat pipe in the picture is deviated, and the inspection effect is affected. The method comprises the following steps: identifying and matching key points of a front frame image and a rear frame image shot by the UAV, and determining a perspective transformation matrix based on the key points; mapping and processing pixel coordinates selected from the rear frame image on the front frame image according to the perspective transformation matrix, to obtain front frame GIS coordinates of the front frame image; performing correction processing on the flight state of the UAV under the factors of distance and angle in all directions, to obtain UAV parameters; and performing image drawing on the position of the pipe, to obtain a final pipe position image.

Description

Technical Field

[0001] This application relates to the field of thermal pipeline inspection, and in particular to a method, equipment and medium for thermal pipeline rendering and correction based on UAV images. Background Technology

[0002] Currently, most pipeline real-time rendering products on the market mainly utilize the intrinsic parameters of cameras mounted on drones, as well as information such as the drone's latitude, longitude, and altitude, to convert GIS points into pixel positions in the image.

[0003] During the construction of heating pipelines, welding is performed every 12 meters, and GIS information is marked at the weld points for subsequent maintenance. Heating pipelines are typically laid more than 1 meter underground, and may pass through areas including roads, fields, bridges, and waterways, making ground inspections difficult. Drones are usually used for inspections to ensure pipeline safety. During inspections, drone operators often rely on memory and experience to determine the pipeline's location, which can easily lead to errors.

[0004] By utilizing the intrinsic parameters of the camera mounted on the drone, as well as the drone's latitude, longitude, altitude, and other information, GIS points can be converted into pixel coordinates in the image. By connecting the GIS points of pipelines near the drone into a line, the location of the pipelines can be plotted in the image.

[0005] During drone flight, the mounted camera continuously outputs video footage, while the flight control system simultaneously acquires real-time status information of the drone and its mounted equipment. For example, at 25 FPS, the footage is updated at 25 frames per second. However, the drone and its mounted equipment's status information is typically updated much less frequently, causing a desynchronization between the video feed and the drone and its mounted equipment's status information. This results in the underground pipeline appearing out of position in the footage, ultimately affecting the effectiveness of the inspection. Summary of the Invention

[0006] This application provides a method, device, and medium for correcting the rendering of thermal pipelines based on drone footage, which addresses the following technical problem: When existing drones inspect thermal pipelines, the image information and the drone and its mounting status information are easily out of sync, causing the underground thermal pipelines to shift in position in the image and affecting the inspection effect.

[0007] The embodiments of this application adopt the following technical solutions:

[0008] On one hand, this application provides a method for correcting the rendering of thermal pipelines based on UAV images, including: identifying and matching key points in a previous frame image and a subsequent frame image captured by the UAV to determine a perspective transformation matrix based on key point pairs; mapping the pixel coordinates selected in the subsequent frame image onto the previous frame image according to the perspective transformation matrix to obtain the previous frame GIS coordinates of the previous frame image; correcting the UAV flight state under all-round distance and angle factors according to the pixel coordinates in the subsequent frame image and the previous frame GIS coordinates to obtain UAV parameters; wherein, the UAV parameters include: mounted pitch angle, mounted yaw angle, vertical field of view angle, horizontal field of view angle, UAV longitude, and UAV latitude; and drawing the pipeline position image using the UAV parameters and pre-acquired pipeline GIS information to obtain the final pipeline position image.

[0009] This application's embodiments, by identifying and matching key points, can more accurately determine the relative positional relationships between images, thereby improving the accuracy of perspective transformation. By using a perspective transformation matrix to map the pixel coordinates of the subsequent frame image to the GIS coordinates of the previous frame image, precise coordinate conversion is achieved, improving the accuracy of UAV parameters. Simultaneously, comprehensive measurement of UAV parameters, including pitch angle, yaw angle, field of view, latitude and longitude, provides a complete data foundation for subsequent data processing. Combining UAV parameters with pre-acquired GIS information enhances the accuracy of pipeline location mapping. Furthermore, the automated data processing workflow reduces manual intervention and improves work efficiency. Precise image matching and flight status correction also reduce errors in data processing. This further improves the synchronization level between image information and UAV and payload status information, reducing the problem of underground pipelines shifting in the image and enhancing the inspection effect of heating pipelines.

[0010] In one feasible implementation, key point identification and matching are performed on the previous frame image and the subsequent frame image captured by the UAV to determine a perspective transformation matrix based on key point pairs. Specifically, this includes: using the UAV to inspect and photograph underground heating pipes, and selecting images of any adjacent frames to obtain the previous frame image and the subsequent frame image; identifying and extracting key points from the previous frame image to obtain the previous frame key points; identifying and extracting key points from the subsequent frame image to obtain the subsequent frame key points; matching the previous frame key points and the subsequent frame key points using a preset Brute-Force technique, and sorting the matched key points to determine sorted key point pairs; and performing matrix transformation on the key point pairs to obtain the perspective transformation matrix.

[0011] In one feasible implementation, according to the perspective transformation matrix, the pixel coordinates selected in the subsequent frame image are mapped onto the preceding frame image to obtain the preceding frame GIS coordinates of the preceding frame image. Specifically, this includes: dividing the subsequent frame image into nine equal parts based on the resolution of the subsequent frame image, and determining the corresponding pixel coordinates of the center point of each divided pixel region; wherein the number of pixel coordinates is nine; mapping the pixel coordinates in the subsequent frame image onto the preceding frame image according to the perspective transformation matrix to determine the mapped pixel coordinates in the preceding frame image; wherein the number of mapped pixel coordinates is nine; and calculating the relevant GIS coordinates based on the UAV status information corresponding to the preceding frame image to obtain the preceding frame GIS coordinates of the preceding frame image.

[0012] In one feasible implementation, based on the pixel coordinates in the subsequent frame image and the GIS coordinates of the preceding frame, the UAV flight state is corrected under all-round distance and angle factors to obtain UAV parameters. Specifically, this includes: determining the subsequent frame GIS coordinates of the subsequent frame image based on the matching correction of the preceding frame GIS coordinates; extracting the pixel coordinates of the subsequent frame image; calculating the distance between each pixel coordinate based on the optical center position of the UAV and the subsequent frame GIS coordinates to obtain the pixel spacing; performing ground distance transmission processing on the pixel coordinates based on the optical center position to obtain the ground projection distance of the pixel coordinates; and using a preset trigonometric function and... The field of view (FOV) parameters of the UAV are obtained by calculating the field of view of the pixel spacing and the ground projection distance, based on the included angle of the optical center position, the pixel spacing, and the ground projection distance. The FAV parameters include the mounted pitch angle, the vertical field of view, and the horizontal field of view. The latitude and longitude parameters of the UAV are calculated based on the latitude and longitude differences corresponding to the pixel coordinates. These parameters include the mounted yaw angle, the UAV longitude, and the UAV latitude. The UAV's flight state is corrected using the FAV parameters and the latitude and longitude parameters, and based on the matching correction between the previous frame GIS coordinates and the subsequent frame GIS coordinates, resulting in the corrected UAV parameters.

[0013] In one feasible implementation, the latitude and longitude parameters of the UAV are calculated based on the latitude and longitude difference corresponding to the pixel coordinates. Specifically, this includes: obtaining the latitude difference and longitude difference of the pixel coordinates at both ends of the edge of the subsequent frame image based on the latitude and longitude values ​​of the pixel coordinates at both ends of the edge; calculating the payload yaw angle of the UAV based on the latitude difference and the longitude difference; calculating the center latitude and longitude difference between the optical center position and the center pixel coordinates, and comparing the center latitude and longitude difference with the edge latitude and longitude difference of the pixel coordinates at both ends of the edge; wherein the center pixel coordinates and the optical center position are perpendicular; if the edge latitude and longitude difference is greater than the center latitude and longitude difference, the latitude and longitude parameters of the UAV are determined as the latitude and longitude of the center pixel coordinates minus the center latitude and longitude difference; if the edge latitude and longitude difference is less than the center latitude and longitude difference, the latitude and longitude parameters of the UAV are determined as the latitude and longitude of the center pixel coordinates plus the center latitude and longitude difference.

[0014] In one feasible implementation, the previous frame GIS coordinates can be used to correct the coordinates of the UAV status information corresponding to the subsequent frame image.

[0015] In one feasible implementation, before drawing the pipeline location image using the UAV parameters and pre-acquired pipeline GIS information to obtain the final pipeline location image, the method further includes: based on the UAV parameters, performing ground projection on four corner points in the captured current frame image to obtain the latitude and longitude of the four corner points; constructing a straight line function based on the latitude and longitude of the four corner points to obtain an edge straight line function based on the four edges of the current frame image; performing information list sliding processing on the pre-acquired pipeline GIS information through a preset sliding window to obtain two adjacent thermal pipeline welding points; wherein, the pipeline GIS information includes the point information of the thermal pipeline welding points; and constructing an actual welding straight line function based on the two adjacent thermal pipeline welding points.

[0016] In one feasible implementation, the pipeline location is imaged using the UAV parameters and pre-acquired pipeline GIS information to obtain a final pipeline location image. Specifically, this includes: calculating the intersection of the actual welding line function and the edge line function to obtain two intersection points within the current frame image range; if two adjacent welding points of the thermal pipeline are both located in the current frame image, the coordinates corresponding to the welding points are determined as pipeline location endpoints; if two adjacent welding points of the thermal pipeline are not both located in the current frame image, the coordinates corresponding to the two intersection points are determined as pipeline location endpoints; based on the pipeline location endpoints and using the camera mounted on the UAV, the pipeline location is rendered to obtain the final pipeline location image.

[0017] Secondly, embodiments of this application also provide a thermal pipe rendering correction device based on UAV images. The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to execute a thermal pipe rendering correction method based on UAV images as described in any of the above embodiments.

[0018] Thirdly, embodiments of this application also provide a non-volatile computer storage medium, characterized in that the storage medium is a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium stores at least one program, each program including instructions, and when the instructions are executed by a terminal, the terminal executes a thermal pipeline rendering correction method based on UAV images as described in any of the above embodiments.

[0019] This application provides a method, device, and medium for rendering and correcting thermal pipelines based on UAV images. Compared with the prior art, the embodiments of this application have the following beneficial technical effects:

[0020] 1. By identifying and matching key points, the relative positional relationships between images can be determined more accurately, thereby improving the accuracy of perspective transformation.

[0021] 2. By using a perspective transformation matrix, the pixel coordinates of the subsequent frame image are mapped to the GIS coordinates of the previous frame image, thus achieving accurate coordinate transformation.

[0022] 3. By correcting for distance and angle factors in all aspects of the drone's flight status, the accuracy of the drone's parameters has been improved.

[0023] 4. Comprehensive measurement of UAV parameters, including pitch angle, yaw angle, field of view, latitude and longitude, provides a complete data foundation for subsequent data processing.

[0024] 5. Combining UAV parameters with pre-acquired GIS information enhances the accuracy of pipeline location mapping.

[0025] 6. By using image rendering technology, the location of pipes is visualized, making it easier for users to intuitively identify and locate pipes.

[0026] 7. Automated data processing reduces manual intervention and improves work efficiency.

[0027] 8. Through precise image matching and flight status correction, errors in data processing are reduced.

[0028] 9. The application of this method can enhance the practicality of thermal pipeline inspection and maintenance, especially in complex environments.

[0029] 10. Automated and precise processing methods may reduce reliance on manual inspections, thereby lowering related costs. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0031] Figure 1 A flowchart of a thermal pipeline rendering correction method based on UAV images provided in this application embodiment;

[0032] Figure 2 A pixel coordinate distribution map provided in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram illustrating the distance between pixel coordinates provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of an optical center position mapping method provided in an embodiment of this application;

[0035] Figure 5 A schematic diagram of a vertical field of view provided for an embodiment of this application;

[0036] Figure 6 A schematic diagram of a horizontal field of view provided for an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of a thermal pipeline rendering and correction device based on drone footage, provided as an embodiment of this application. Detailed Implementation

[0038] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0039] This application provides a method for correcting the rendering of thermal pipes based on drone footage, such as... Figure 1As shown, the thermal pipeline rendering correction method based on UAV images specifically includes steps S101-S104:

[0040] S101. Identify and match key points in the previous frame image and the next frame image captured by the drone to determine the perspective transformation matrix based on the key point pairs.

[0041] Specifically, the first step is to use drones to inspect and photograph the underground heating pipes, and then select images from any adjacent frames to obtain the previous and next frame images.

[0042] Furthermore, key points are identified and extracted from the previous frame image to obtain the key points of the previous frame. Similarly, key points are identified and extracted from the subsequent frame image to obtain the key points of the subsequent frame.

[0043] Furthermore, using a pre-defined Brute-Force technique, keypoints from the previous frame are matched with keypoints from the next frame, and the matched keypoints are then sorted to determine the sorted keypoint pairs. Finally, the keypoint pairs are transformed into a matrix to obtain the perspective transformation matrix.

[0044] As a feasible implementation method, for two consecutive frames of images captured by a drone, key points are first identified and extracted. Then, the Brute-Force method is used to match the key points between the two frames. After sorting the matched key points, perspective transformation matrix operations are performed on the sorted key point pairs. A perspective transformation matrix is ​​a transformation matrix that maps objects or scenes in three-dimensional space onto a two-dimensional plane, primarily used for 3D graphics rendering and transformation. Through the perspective transformation matrix, points in three-dimensional space can be mapped onto a two-dimensional plane while preserving the distance and perspective effects of objects. The perspective transformation matrix is ​​determined by a series of parameters, including the projection center point, the view plane normal vector, the perspective width, and the height.

[0045] S102. Based on the perspective transformation matrix, the pixel coordinates selected in the subsequent frame image are mapped onto the previous frame image to obtain the previous frame GIS coordinates of the previous frame image.

[0046] Specifically, the first step is to divide the subsequent frame image into nine equal parts based on its resolution, and then determine the corresponding pixel coordinates for the center point of each divided pixel region. The number of pixel coordinates is nine.

[0047] Furthermore, based on the perspective transformation matrix, the pixel coordinates in the subsequent frame image are mapped to the previous frame image, thus determining the mapped pixel coordinates in the previous frame image. The number of mapped pixel coordinates is nine.

[0048] Furthermore, by using the UAV status information corresponding to the previous frame image, the GIS coordinates of the mapped pixel coordinates are calculated to obtain the previous frame GIS coordinates of the previous frame image.

[0049] In one embodiment, Figure 2 A pixel coordinate distribution map provided for an embodiment of this application, such as Figure 2 As shown, when a drone is flying, the images in consecutive frames are different, and the corresponding pixel coordinate systems need to be transformed and unified. Using the coordinate system of the previous frame (the image before the main frame) as an anchor point, the coordinates of the subsequent frame are transformed to the coordinate system of the previous frame. First, nine points need to be selected from the subsequent frame. Taking a 1920*1080 resolution as an example: select 0-(320,180), 1-(960,180), 2-(1600,180), 3-(320,540), 4-(960,540), 5-(1600,540), 6-(320,900), 7-(960,900), and 8-(1600,900). Then, based on the perspective transformation matrix output above, the nine points selected in the subsequent frame are mapped back to the previous frame, resulting in nine points. Based on the UAV status information in the previous frame image, the GIS coordinates of these 9 points in the previous frame image are calculated. These 9 previous frame GIS coordinates can then be used to correct the flight status information in the subsequent frame.

[0050] S103. Based on the pixel coordinates in the subsequent frame image and the GIS coordinates in the previous frame, perform comprehensive distance and angle correction processing on the UAV flight status to obtain UAV parameters. Among them, the UAV parameters include: payload pitch angle, payload yaw angle, vertical field of view, horizontal field of view, UAV longitude, and UAV latitude.

[0051] Specifically, after matching and correcting the GIS coordinates of the subsequent frame image using the GIS coordinates of the previous frame, the GIS coordinates of the subsequent frame image are determined. Then, the pixel coordinates of the subsequent frame image are extracted. Based on the UAV's optical center position and the subsequent frame GIS coordinates, the distance between each pixel coordinate is calculated to obtain the pixel spacing. The previous frame GIS coordinates can be used to correct the coordinates of the UAV's state information corresponding to the subsequent frame image.

[0052] In one embodiment, Figure 3 This is a schematic diagram illustrating the distance between pixel coordinates provided in an embodiment of this application; Figure 4 This is a schematic diagram of an optical center position mapping method provided in an embodiment of this application, as shown below. Figure 3 as well as Figure 4 As shown, firstly, a trigonometric function relationship is established between the coordinates of 9 pixels in the subsequent frame image and the optical center. Figure 4(The numbers 1, 2...8, 9 in the image represent the corresponding pixel coordinates). Then, let d1 be the GIS distance between point 1 and point 4, d2 be the GIS distance between point 4 and point 7, d3 be the GIS distance between point 7 and the drone on the ground, alt be the height of the drone above the ground, pitch be the angle between the center of the image and the vertical line, alpha be the angle between point 1, the optical center, and point 4, beta be the distance between point 3, the optical center, and point 4, up be the distance between points 0 and 1 projected onto the ground, mid be the distance between points 3 and 4 projected onto the ground, down be the distance between points 6 and 7 projected onto the ground, dist1C be the distance between point 1 projected onto the ground and the optical center, dist4C be the distance between point 4 projected onto the ground and the optical center, dist7C be the distance between point 7 projected onto the ground and the optical center, and let L be the focal length.

[0053] Furthermore, based on the optical center position, the pixel coordinates are subjected to transmission processing related to the ground distance to obtain the ground projection distance of the pixel coordinates.

[0054] Furthermore, by using preset trigonometric functions and the angle between the optical center positions, the field of view is calculated for the pixel spacing and ground projection distance to obtain the UAV's field of view parameters. These parameters include: mount pitch angle, vertical field of view, and horizontal field of view.

[0055] In one embodiment, Figure 5 A schematic diagram of a vertical field of view provided for an embodiment of this application; Figure 6 A schematic diagram of a horizontal field of view provided for an embodiment of this application, such as... Figure 5 as well as Figure 6 As shown:

[0056] On the imaging plane: tan(∠0C1)=L*tan(beta) / (L / cos(alpha)), which simplifies to: tan(∠0C1)=tan(beta)*cos(alpha);

[0057] On the ground: tan(∠0C1)=up / dist1C;

[0058] On the imaging plane: tan(∠6C7)=tan(∠0C1);

[0059] On the ground: tan(∠6C7) = down / dist7C;

[0060] On the ground: dist1C = (d1 + d2 + d3) * sin(pitch + alpha);

[0061] On the ground: dist4C = (d2 + d3) * sin(pitch);

[0062] On the ground: dist7C = d3*sin(pitch-alpha);

[0063] On the ground: tan(pitch-alpha) = d³ / alt;

[0064] In the above 8 equations, the known quantities are alt, up, down, d1, and d2, the intermediate quantities are d3, tan(∠0C1), dist1C, dist4C, and dist7C, and the quantities to be solved are pitch, alpha, and beta.

[0065] Solving for intermediate quantities using a system of multiple equations is denoted as:

[0066] D = (d1 + d2) / alt;

[0067] t = (up - down) / (up + down);

[0068] T = t * t;

[0069] a = 2;

[0070] b = D * TD;

[0071] c = 2 * T;

[0072] Find: tan(pitch)=(up-down) / ((up+down)*

[0073] tan(alpha)); tan(beta)=mid*cos(pitch) / alt

[0074] Further, the field of view parameters are obtained:

[0075] 1) Vertical field of view: fovV = 2*arctan(3*tan(alpha) / 2);

[0076] 2) Horizontal field of view: fovH=2*arctan(3*tan(beta) / 2);

[0077] 3) Mounted pitch angle: pitch = arctan((up-down) / ((up+down)*tan(alpha)));

[0078] Furthermore, the latitude and longitude parameters of the UAV need to be calculated based on the latitude and longitude difference corresponding to the pixel coordinates. These parameters include: payload yaw angle, UAV longitude, and UAV latitude.

[0079] Furthermore, based on the latitude and longitude values ​​of the pixel coordinates at both ends of the edge of the subsequent frame image, the latitude difference and longitude difference of the pixel coordinates at both ends of the edge are obtained. Then, based on the latitude and longitude differences, the payload yaw angle of the UAV is calculated.

[0080] Furthermore, the difference in latitude and longitude between the optical center position and the center pixel coordinates is calculated, and this difference is compared with the difference in latitude and longitude between the pixel coordinates at both ends of the edge. The center pixel coordinates and the optical center position are perpendicular to each other.

[0081] If the latitude and longitude difference at the edge is greater than the latitude and longitude difference at the center, then the latitude and longitude parameters of the UAV will be determined as the latitude and longitude of the center pixel coordinates minus the latitude and longitude difference at the center.

[0082] If the latitude and longitude difference at the edge is less than the latitude and longitude difference at the center, then the latitude and longitude parameters of the UAV will be determined as the latitude and longitude of the center pixel coordinates plus the latitude and longitude difference at the center.

[0083] In one embodiment, such as Figure 4 As shown, the latitude and longitude differences between points 1 and 7 on the ground are longitude_17 and latitude_17 (edge ​​latitude and longitude differences), respectively, and the yaw angle of the payload is obtained as yaw =

[0084] arctan(longitude_17 / latitude_17). On the ground, the distance from point 4 to the drone is: d2 + d3 = alt * tan(pitch). Therefore, the differences in latitude and longitude between the drone and point 4 (the difference in center latitude and longitude) are:

[0085] longitude_diff=|(d2+d3)*sin(yaw)|

[0086] latitude_diff=|(d2+d3)*cos(yaw)|

[0087] Then, compare the latitude and longitude of point 1 and point 4:

[0088] 1) If the longitude of point 1 is large (i.e., the difference between the latitude and longitude of the edge is greater than the difference between the latitude and longitude of the center), then the longitude of the UAV is longitude = longitude_4 - longitude_diff. If the longitude of point 1 is small (i.e., the difference between the latitude and longitude of the edge is less than the difference between the latitude and longitude of the center), then the longitude of the UAV is longitude = longitude_4 + longitude_diff.

[0089] 2) If the latitude of point 1 is large (i.e., the difference between the latitude and longitude of the edge is greater than the difference between the latitude and longitude of the center), then the latitude of the UAV is latitude = latitude_4 - latitude_diff. If the latitude of point 1 is small (i.e., the difference between the latitude and longitude of the edge is less than the difference between the latitude and longitude of the center), then the latitude of the UAV is latitude = latitude_4 + latitude_diff.

[0090] Finally, the latitude and longitude parameters of the drone were obtained.

[0091] Furthermore, by using the field of view angle parameters and latitude and longitude parameters, and based on the matching correction between the GIS coordinates of the previous frame and the GIS coordinates of the next frame, the flight state of the UAV is corrected, and finally the corrected UAV parameters are obtained.

[0092] S104. Using UAV parameters and pre-acquired pipeline GIS information, the pipeline location is drawn into an image to obtain the final pipeline location image.

[0093] Specifically, by using the matched and corrected UAV parameters, the four corner points in the captured current frame image are projected onto the ground to obtain the latitude and longitude of the four corner points.

[0094] Furthermore, it is necessary to construct straight line functions for the latitude and longitude of the four corner points to obtain edge straight line functions based on the four edges of the current frame image.

[0095] Furthermore, the pre-acquired pipeline GIS information is processed through a sliding window to obtain two adjacent thermal pipeline welding points; the pipeline GIS information includes point information of the thermal pipeline welding points. Based on the two adjacent thermal pipeline welding points, an actual welding line function is constructed.

[0096] Furthermore, the intersection points of the actual welding line function and the edge line function are calculated to obtain two intersection points within the current frame image range.

[0097] If two adjacent welding points of the thermal pipes are both located in the current frame image, then the coordinates corresponding to the welding points of the thermal pipes are determined as the endpoints of the pipes.

[0098] If two adjacent welding points of thermal pipes are not both located in the current frame image, the coordinates corresponding to the two intersection points are determined as the pipe position endpoints.

[0099] Furthermore, based on the endpoints of the pipeline, and using the camera mounted on the drone, the pipeline location is rendered and drawn to obtain the final pipeline location image.

[0100] In one embodiment, pipeline GIS information is typically point information, such as... Figure 4As shown, based on the point sorting and the point GIS data, pipelines can be drawn on the map. After the drone parameters are corrected, the pipeline location can be rendered on the screen based on the drone parameters and pipeline GIS information. The specific process is as follows:

[0101] 1) Calculate the latitude and longitude of the four corner points projected onto the ground in the current frame image to obtain the straight line equations (edge ​​line functions) of the four sides.

[0102] 2) Using a window of length 2, slide through the pipeline GIS information list to obtain any two adjacent welding points of the thermal pipeline, denoted as p0 and p1, which is to calculate the actual welding line function line_p for these two points.

[0103] 3) Calculate the intersection points of the actual welding line function line_p under p0 and p1 and the four edge line functions in step 1), and retain the two intersection points located at the edges within the current frame image range, denoted as point_list.

[0104] 4) In point_list, there are 2 points. Then, it is determined whether p0 and p1 are in the current frame image. If they are in the current frame image, the coordinates in the current frame image are taken as the pipe position endpoints. Otherwise, the corresponding points in point_list are taken as the pipe position endpoints, a straight line is drawn, and then the pipe position image is rendered to obtain the final pipe position image.

[0105] In addition, this application also provides a thermal pipeline rendering correction device based on drone footage, such as... Figure 7 As shown, the thermal pipeline rendering and correction device based on drone footage specifically includes 700 components:

[0106] At least one processor 701; and a memory 702 communicatively connected to the at least one processor 701; wherein the memory 702 stores instructions executable by the at least one processor 701 to enable the at least one processor 701 to execute:

[0107] The key points of the previous frame image and the next frame image captured by the drone are identified and matched to determine the perspective transformation matrix based on the key point pairs.

[0108] Based on the perspective transformation matrix, the pixel coordinates selected in the subsequent frame image are mapped onto the previous frame image to obtain the previous frame GIS coordinates of the previous frame image.

[0109] Based on the pixel coordinates in the subsequent frame image and the GIS coordinates in the previous frame, the UAV flight status is corrected under all distance and angle factors to obtain UAV parameters. The UAV parameters include: payload pitch angle, payload yaw angle, vertical field of view, horizontal field of view, UAV longitude, and UAV latitude.

[0110] Using drone parameters and pre-acquired pipeline GIS information, the pipeline location is imaged to obtain the final pipeline location image.

[0111] This application's embodiments, by identifying and matching key points, can more accurately determine the relative positional relationships between images, thereby improving the accuracy of perspective transformation. By using a perspective transformation matrix to map the pixel coordinates of the subsequent frame image to the GIS coordinates of the previous frame image, precise coordinate conversion is achieved, improving the accuracy of UAV parameters. Simultaneously, comprehensive measurement of UAV parameters, including pitch angle, yaw angle, field of view, latitude and longitude, provides a complete data foundation for subsequent data processing. Combining UAV parameters with pre-acquired GIS information enhances the accuracy of pipeline location mapping. Furthermore, the automated data processing workflow reduces manual intervention and improves work efficiency. Precise image matching and flight status correction also reduce errors in data processing. This further improves the synchronization level between image information and UAV and payload status information, reducing the problem of underground pipelines shifting in the image and enhancing the inspection effect of heating pipelines.

[0112] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0113] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0114] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0119] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0120] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0121] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0122] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of this specification.

Claims

1. A method for rendering and correcting thermal pipes based on UAV images, characterized in that, The method includes: The key points of the previous frame image and the next frame image captured by the drone are identified and matched to determine the perspective transformation matrix based on the key point pairs. Based on the perspective transformation matrix, the pixel coordinates selected in the subsequent frame image are mapped onto the previous frame image to obtain the previous frame GIS coordinates of the previous frame image; Based on the pixel coordinates in the subsequent frame image and the GIS coordinates in the previous frame, the UAV flight status is corrected under all distance and angle factors to obtain UAV parameters, specifically including: Based on the matching and correction of the previous frame GIS coordinates, the subsequent frame GIS coordinates of the subsequent frame image are determined; and the pixel coordinates of the subsequent frame image are extracted. Based on the optical center position of the UAV and the GIS coordinates of the subsequent frame, the distance between each pixel coordinate is calculated to obtain the pixel spacing; Based on the optical center position, the pixel coordinates are subjected to transmission processing related to ground distance to obtain the ground projection distance of the pixel coordinates; The field of view parameters of the UAV are obtained by calculating the pixel spacing and the ground projection distance using preset trigonometric functions and the included angles related to the optical center position; wherein, the field of view parameters include: mounted pitch angle, vertical field of view angle and horizontal field of view angle. Based on the latitude and longitude difference corresponding to the pixel coordinates, the latitude and longitude parameters of the UAV are calculated; wherein, the latitude and longitude parameters include: payload yaw angle, UAV longitude, and UAV latitude, specifically including: Based on the latitude and longitude values ​​of the pixel coordinates at both ends of the edge of the subsequent frame image, the latitude difference and longitude difference of the pixel coordinates at both ends of the edge are obtained; and the payload yaw angle of the UAV is calculated based on the latitude difference and the longitude difference. Calculate the center latitude and longitude difference between the optical center position and the center pixel coordinates, and compare the center latitude and longitude difference with the edge latitude and longitude difference of the pixel coordinates at both ends of the edge; wherein, the center pixel coordinates and the optical center position are perpendicular to each other; If the latitude and longitude difference at the edge is greater than the latitude and longitude difference at the center, then the latitude and longitude parameters of the UAV are determined as the latitude and longitude of the center pixel coordinates minus the latitude and longitude difference at the center. If the latitude and longitude difference at the edge is less than the latitude and longitude difference at the center, then the latitude and longitude parameters of the UAV are determined as the latitude and longitude of the center pixel coordinates plus the latitude and longitude difference at the center. The UAV flight state is corrected using the field of view parameters and the latitude and longitude parameters, and based on the matching correction between the previous frame GIS coordinates and the subsequent frame GIS coordinates, to obtain the corrected UAV parameters. The UAV parameters include: payload pitch angle, payload yaw angle, vertical field of view angle, horizontal field of view angle, UAV longitude, and UAV latitude. The previous frame GIS coordinates can correct the coordinates of the UAV state information corresponding to the subsequent frame image. Based on the UAV parameters, the four corner points in the captured current frame image are projected onto the ground to obtain the latitude and longitude of the four corner points; A straight line function is constructed based on the latitude and longitude of the four corner points to obtain the edge straight line function based on the four edges of the current frame image; By using a preset sliding window, the pre-acquired pipeline GIS information is processed through a list sliding process to obtain two adjacent thermal pipeline welding points; wherein, the pipeline GIS information includes the point information of the thermal pipeline welding points; Based on two adjacent welding points of the aforementioned thermal pipes, an actual welding linear function is constructed; Using the drone parameters and pre-acquired pipeline GIS information, the pipeline location is imaged to obtain the final pipeline location image, specifically including: The intersection points of the actual welding line function and the edge line function are calculated to obtain two intersection points located within the current frame image range; If two adjacent welding points of the thermal pipe are both located in the current frame image, then the coordinates corresponding to the welding points of the thermal pipe are determined as the endpoints of the pipe positions. If two adjacent welding points of the thermal pipes are not both located in the current frame image, then the coordinates corresponding to the two intersection points are determined as the pipe position endpoints; Based on the endpoint of the pipeline, and using the camera mounted on the drone, the pipeline location is rendered and drawn to obtain the final pipeline location image.

2. The method for rendering and correcting thermal pipes based on UAV images according to claim 1, characterized in that, Keypoint identification and matching are performed on the previous and subsequent frames captured by the drone to determine the perspective transformation matrix based on the keypoint pairs, specifically including: The drone is used to inspect and photograph underground heating pipes, and images of any adjacent frames are selected to obtain the previous frame image and the next frame image. Key points are identified and extracted from the previous frame image to obtain the previous frame key points; and key points are identified and extracted from the subsequent frame image to obtain the subsequent frame key points. Using the preset Brute-Force technology, the key points of the previous frame are matched with the key points of the next frame, and the matched key points are sorted to determine the sorted key point pairs. The key point pairs are transformed into a matrix to obtain the perspective transformation matrix.

3. The method for rendering and correcting thermal pipes based on UAV images according to claim 1, characterized in that, Based on the perspective transformation matrix, the pixel coordinates selected in the subsequent frame image are mapped onto the preceding frame image to obtain the preceding frame GIS coordinates of the preceding frame image, specifically including: Based on the resolution of the subsequent frame image, the subsequent frame image is divided into nine equal parts for pixel regions, and the center point of each divided pixel region is determined to have corresponding pixel coordinates; wherein, the number of pixel coordinates is nine. Based on the perspective transformation matrix, the pixel coordinates in the subsequent frame image are mapped to the previous frame image to determine the mapped pixel coordinates in the previous frame image; wherein, the number of mapped pixel coordinates is 9; By using the UAV status information corresponding to the previous frame image, the GIS coordinates of the mapped pixel coordinates are calculated to obtain the previous frame GIS coordinates of the previous frame image.

4. A thermal pipeline rendering and correction device based on UAV images, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, enabling the at least one processor to execute a thermal pipeline rendering correction method based on UAV footage according to any one of claims 1-3.

5. A non-volatile computer storage medium, characterized in that, The storage medium is a non-volatile computer-readable storage medium, which stores at least one program, each program including instructions, which, when executed by a terminal, cause the terminal to perform a thermal pipeline rendering correction method based on UAV footage according to any one of claims 1-3.

Images