Positioning and inspection control method for multi-sensor-fusion pipeline unmanned aerial vehicle

Abstract

The present invention relates to a positioning and inspection control method for a multi-sensor-fusion pipeline unmanned aerial vehicle. The method comprises: acquiring a video stream image frame of real-time pipeline inspection by means of a binocular camera; determining the darkness degree of the image frame, and if the image frame does not meet a brightness requirement, performing real-time low-light enhancement on the image frame; determining the blur degree of the image frame, and if the image is blurry, performing real-time deblurring processing on the image frame; performing pose calculation on the image frame; and outputting positioning data and pose data of an unmanned aerial vehicle at a real-time location point, and a pipeline inspection image. The present invention realizes safe and effective positioning of an unmanned aerial vehicle in a dark environment where a GPS signal is lost, and realizes safe and normal inspection of the unmanned aerial vehicle in environments such as drainage channels and pipelines on the basis of the positioning of the unmanned aerial vehicle.

Description

Pipeline UAV positioning and inspection control method based on multi-sensor fusion Technical Field

[0001] The present invention belongs to the field of hydropower station safety monitoring, and specifically relates to a pipeline unmanned aerial vehicle positioning and inspection control method based on multi-sensor fusion. Background Art

[0002] Underground, enclosed, dark pipe spaces are common in industrial facilities such as large power plants, and are used for drainage, purification, and decontamination. These environments typically lack good lighting, and because they are underground, GPS signals are often lost.

[0003] Take the drainage tunnels of hydropower projects as an example. These tunnels are located on both sides of the dam, with concrete walls and a total length of over 2,000 meters. These tunnels are semi-enclosed spaces characterized by low light, humidity, lack of GPS signals, and interference with geomagnetic signals. The water volume in these tunnels varies year-round, drying up during the dry season and flooding during the flood season. This constant operation poses risks of erosion, collapse, and accumulation. Inspections using vehicles and unmanned boats are limited in adaptability, inefficiency, and risk. Due to the constant accumulation of silt and other waterborne debris, the inner walls have a rich textured texture. Given the complex environment within the tunnels, after accurately positioning drones using appropriate positioning methods, the next challenge is to ensure stable, safe, and efficient drone inspections.

[0004] In a pipeline environment similar to a drainage hole, the technical problems that need to be solved by multi-sensor fusion pipeline drone positioning and inspection control are mainly divided into the following three parts: First, how to realize the drone's positioning function with the assistance of sensors in a dark underground pipeline environment where GPS signals are missing; Second, during the pipeline inspection process, the drone will inevitably encounter scenarios with changing pipeline environments, such as branch holes and gaps. How to ensure the normal flight of the drone in such environments and how to set the priority of its flight selection; Third, for any drone flight, reducing the loss of the drone in the event of abnormal sensor data is a key requirement. Therefore, it is necessary to consider the drone's backup strategy and the implementation of operations such as hovering and landing under the backup strategy.

[0005] Summary of the Invention

[0006] The purpose of the present invention is to address the above-mentioned problems and provide a multi-sensor fusion pipeline drone positioning and inspection control method, which improves the flight safety of the drone in pipelines with missing signals through the backup redundancy design of the drone; and performs dark light enhancement and deblurring on the images obtained by the drone to improve the quality of the pipeline inspection images output by the drone.

[0007] The technical solution of the present invention is a pipeline drone positioning and inspection control method based on multi-sensor fusion, which includes the following steps:

[0008] S1: Obtain real-time pipeline inspection video stream image frames through a binocular camera;

[0009] S2: Determine the darkness level of the image frame obtained in step S1. If the image frame does not meet the brightness requirement, perform real-time dark light enhancement on the image frame and execute step S3. If the brightness requirement is met, execute step S3 directly.

[0010] S3: Determine the blur level of the image frame obtained in step S2. If the image is blurred, perform real-time deblurring on the image frame and execute step S4. If the image is not blurred, execute step S4 directly.

[0011] S4: performing pose calculation on the image frame obtained in step S3;

[0012] S5: Output the real-time positioning data and posture data of the UAV and pipeline inspection images.

[0013] Preferably, the pipeline UAV positioning inspection control method further includes determining the validity of the posture data output by the UAV. If the validity is unqualified, a redundant backup system of the UAV including a monocular camera and a single-line laser radar is activated to obtain speed estimates of various aspects of the UAV using the redundant backup system. Specifically,

[0014] 1) Determine whether the error of the posture data output by the drone meets the accuracy requirements. If it does, end; if it does not, execute step 2);

[0015] 2) Activate the monocular camera on the drone and use the video stream features obtained by the monocular camera to estimate the front-to-back and left-to-right speed of the drone;

[0016] 3) Activate the single-line laser radar on the UAV and use the laser point cloud obtained by the single-line laser radar to estimate the UAV's up and down and left and right speed;

[0017] 4) Fusing the up-down, forward-backward, and left-right velocity estimates obtained in steps 2) and 3) to obtain velocity estimates for each direction of the drone;

[0018] 5) Realize the flight control of the UAV according to the target point release.

[0019] Furthermore, the pipeline drone positioning and inspection control method also includes judging the validity of the output data of the drone's redundant backup system. If the output data of the drone's redundant backup system meets the accuracy requirements, the drone's flight control is realized according to the target point release; otherwise, the drone is controlled to land on the spot or return or hover.

[0020] Preferably, in step S3, a generative adversarial network (GAN) is used to perform real-time deblurring processing on the image frame.

[0021] Preferably, in step S5, a simultaneous localization and mapping (SLAM) method is used to realize the posture solution of the UAV.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1) The present invention realizes the safe and effective positioning of drones in dark environments and those without GPS signals. Based on the positioning of drones, the invention also realizes the safe and normal inspection of drones in environments such as drains and pipelines.

[0024] 2) The present invention improves the quality of pipeline inspection images through image processing methods such as dark light enhancement and real-time deblurring.

[0025] 3) The present invention realizes high-speed, stable and fully automatic inspection flight of the UAV in the corridor.

[0026] 4) The present invention realizes rapid inspection of corridors, obtains video images of corridors, improves corridor inspection efficiency, and reduces the intensity of manual inspections.

[0027] 5) The backup redundancy design enables the drone to be effectively controlled to hover, land or return when the sensor data fails, thereby improving the safety of the inspection drone. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will be further described below with reference to the accompanying drawings and examples.

[0029] FIG1 is a schematic diagram of a flow chart of image processing of a video stream image frame of a drone according to an embodiment of the present invention.

[0030] FIG2 is a flow chart showing the process of obtaining the UAV speed estimation using the UAV redundant backup system.

[0031] FIG3 is a schematic diagram of an imaging relationship according to an embodiment of the present invention.

[0032] FIG4 is a schematic diagram of a point cloud acquired by a laser radar according to an embodiment of the present invention.

[0033] FIG5 is a schematic diagram of the positions of the UAV at time t and time t-1 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The embodiment mainly includes the following three parts: validity judgment and processing of video stream images in low-light environments; positioning backup means of multi-sensor fusion; and validity judgment of positioning data, and based on this judgment, the drone can continue to fly or hover or land.

[0035] Using binocular cameras to locate drones in low-light environments, or even in darkness, is challenging. This invention uses both physical and software methods to achieve drone positioning in dark environments and to determine the validity of the binocular camera's video stream.

[0036] Using a binocular camera to position a drone in dark environments allows for safe and stable inspections, provided the binocular camera is functioning properly. However, this approach presents a potential risk: assuming the binocular camera is always functioning properly. However, the proper functioning of a binocular camera is highly dependent on the quality of its image frames and the effectiveness of the environment. If the wall environment lacks specific features during flight, or if the drone experiences rapid deflection or stall, the binocular camera's positioning may fail.

[0037] This invention, independent of the binocular camera itself, designs a backup speed control system based on a single-line laser radar and a monocular camera. The key concept is to determine the validity of the binocular camera's data output. When the binocular camera's position output error is excessive, backup speed control is employed, thereby improving the drone's safety. Of course, when the binocular camera is operating normally, the real-time output of the binocular camera remains the primary focus.

[0038] The single-line laser radar performs cross-sectional scanning perpendicular to the drone's nose's direction of travel, while the monocular camera, mounted directly below the drone, estimates its velocity in both the forward and backward, and left and right directions.

[0039] As shown in Figure 1, the multi-sensor fusion pipeline drone positioning and inspection control method includes:

[0040] S1: Obtain real-time pipeline inspection video stream image frames through a binocular camera;

[0041] S2: Determine the darkness level of the image frame obtained in step S1. If the image frame does not meet the brightness requirement, perform real-time dark light enhancement on the image frame and execute step S3. If the brightness requirement is met, execute step S3 directly.

[0042] S3: Determine the blur level of the image frame obtained in step S2. If the image is blurred, perform real-time deblurring on the image frame and execute step S4. If the image is not blurred, execute step S4 directly.

[0043] S4: performing pose calculation on the image frame obtained in step S3;

[0044] S5: Output the real-time positioning data and posture data of the UAV and pipeline inspection images.

[0045] As shown in Figure 2, the redundant backup system of the drone is used to obtain the speed estimates of various aspects of the drone, including:

[0046] 1) Determine whether the error of the posture data output by the drone meets the accuracy requirements. If it does, end; if it does not, execute step 2);

[0047] 2) Activate the monocular camera on the drone and use the video stream features obtained by the monocular camera to estimate the front-to-back and left-to-right speed of the drone;

[0048] 3) Activate the single-line laser radar on the UAV and use the laser point cloud obtained by the single-line laser radar to estimate the UAV's up and down and left and right speed;

[0049] 4) Fusing the up-down, forward-backward, and left-right velocity estimates obtained in steps 2) and 3) to obtain velocity estimates for each direction of the drone;

[0050] 5) Realize the flight control of the UAV according to the target point release.

[0051] The first problem to be solved is how to estimate the actual movement distance of the drone based on the blur degree. Assuming that the blur length of the image is known, according to the imaging relationship shown in Figure 3, within the exposure time t2 range, the actual movement distance is:

[0052] Where: L represents the actual movement distance; H represents the height from the drone lens to the wall; l represents the blur length of the image; and h represents the focal length.

[0053] After obtaining the exposure time and distance L, based on the performance of the drone hardware, we know that the exposure time t2 is approximately a constant value. However, between two frames, in addition to the exposure time, there is also a non-exposure time t1. Both the exposure time and the non-exposure time are related only to the hardware itself. Therefore, the exposure time ratio r is:

[0054] Then the actual displacement distance L between the two frames of pictures is r It can be expressed as: r =L / r

[0055] In the axial direction, the velocity estimation can be expressed as:

[0056] In practical applications, the distance to the pipe wall can be acquired using other sensors such as 2D LiDAR. The camera focal length is a parameter set before takeoff. The length and ratio of the exposure time to the non-exposure time can be acquired through hardware. Once the blur length is obtained, the motion length can be estimated.

[0057] The laser point cloud generated by a single-line UAV LiDAR is used to estimate vertical and horizontal velocity. Taking a circular tunnel as an example, the x-axis is the x-axis, the z-axis is the vertical axis, and the y-axis is the horizontal axis. The 2D LiDAR is mounted with the drone's nose perpendicular to the x-axis. A single 2D LiDAR point cloud is acquired approximately in the same YOZ plane. The point cloud is first segmented in real time for each frame, the mean of the point cloud in each direction is calculated, and the centroid of the cross section is fitted. Finally, the real-time velocity of the UAV is estimated. The point cloud acquired by the LiDAR in real time is shown in Figure 4.

[0058] The cross-sectional structures of pipelines are mainly square and circular. The centroids of both shapes are located at the intersection of the upper and lower symmetry lines and the left and right symmetry lines. This property can be used to simplify the centroid calculation steps to reduce time consumption.

[0059] Assume that each frame of point cloud is N points, with (y i , z i ), 1≤i≤N represents a point in the YOZ plane, then the real-time local coordinates of the centroid for:

[0060] When the number of point cloud data is large, the calculation time is long. Considering the special characteristics, it is considered to use local point cloud instead of global point cloud to simplify the calculation.

[0061] According to the angle information of each point cloud of the two-dimensional laser radar, the positive direction of z is the 0° direction of the laser radar, and the point cloud is divided into the point cloud P above u (345°~15°), right point cloud P r (75°~105°), point cloud P directly below b (165°~195°) and the left point cloud P l (255°~285°). Due to the scanning symmetry of the laser point cloud itself, it can be seen that the number of point clouds in the four intervals is the same. Therefore, the real-time local position estimation of the centroid can be expressed as:

[0062] Where: y r Indicates the point cloud P on the right r The y-direction mean of l Indicates the point cloud P on the left l The y-direction mean; z u Indicates the point cloud P directly above u The z-direction mean value; b Indicates the point cloud P directly below b The z-direction mean.

[0063] This method reduces the number of point clouds involved in the calculation. Furthermore, for the left and right point clouds, z-direction information does not need to be calculated, and similarly for the top and bottom points. This improves computational speed, ensuring the frequency of real-time drone pose releases and further ensuring the drone's real-time safety.

[0064] After obtaining the real-time local centroid position, the real-time motion velocity is calculated based on the centroid position obtained from the point clouds of the two preceding and succeeding frames. Considering that sudden wall changes are unlikely in real engineering environments and that the cross-sections are typically approximately the same throughout the entire flight, it can be assumed that the drone's actual movement occurs within a similar cross-sectional environment. In the event of a sudden change in the cross-sectional environment, the real-time width and height must be calculated by combining the four point clouds to confirm the presence of a sudden change, and appropriate control measures must be taken to ensure stable flight.

[0065] In the cross-section approximation environment, time t and time t-1 represent the current frame time and the previous frame time, respectively, as shown in Figure 5. and They represent the projection of the current global position of the UAV and the global position of the previous moment on the current section respectively.

[0066] in:

[0067] When observing along the axis x, the x information of the two frames before and after can be ignored, that is, only the YOZ plane is analyzed.

[0068] When the global coordinates are known, based on the position of the drone in the previous and next two frames, the speed of the drone in the y and z directions can be expressed as:

[0069] The speed obtained by the two-dimensional lidar is fused with the global positioning speed of the binocular camera and other sensor information. On the one hand, this ensures data redundancy to avoid the loss of control of the drone due to failure of a certain sensor. On the other hand, multi-sensor data fusion can ensure the accuracy and effectiveness of speed estimation.

[0070] In the case of global information loss, that is, considering the possible positioning failure of binocular camera visual positioning, it is necessary to estimate the real-time speed of the UAV through real-time local coordinates.

[0071] In the laser radar's own coordinate system, the two-dimensional laser radar itself is always at the origin (0,0), which is shown in Figure 5. and The positions are all (0,0), and the lidar itself cannot obtain information in the x direction.

[0072] The only valid information at this time is the real-time centroid position. In this case, the velocity of the drone is estimated to be:

[0073] Since the drone's global positioning is lost, local velocity estimation is often used for emergency control in this situation, or for relatively low-precision control of the drone when global positioning is not required. Therefore, the accuracy of the velocity estimation can tolerate a small range of error.

[0074] Regardless of whether global positioning information is available or not, drone safety is always the primary consideration. Speed ​​control is particularly important for drones, and achieving real-time speed estimation is the primary guarantee for drone safety.

[0075] In this embodiment, a simultaneous localization and mapping SLAM method is used to calculate the pose of the drone. The core of visual SLAM technology is to obtain the common feature points of two consecutive frames and match them to determine the motion relationship between the two frames, thereby obtaining the global pose. After obtaining enough feature points from the image, the feature points between the two frames need to be matched to calculate the motion relationship between the two frames. In this embodiment, the shortest Hamming distance is used as the optimal matching method.

[0076] The feature-based visual SLAM method processes the images output by the camera and filters the pixels and their neighborhoods with strong characteristics in the image instead of calculating all the pixels. This reduces the amount of calculation, improves the efficiency of the algorithm, and ensures the real-time performance of the normal flight of the UAV.

[0077] In summary, this embodiment achieves safe and effective positioning of drones in darkness and in the absence of GPS signals. Based on this positioning, the drone can safely and properly inspect sewers, pipes, and other environments. Furthermore, the drone's real-time sensor data is evaluated and its flight strategy is adjusted in real time based on the data's validity. Furthermore, in the event of sensor data failure, a redundant backup design enables the drone to perform required operations such as hovering, landing, and returning home.

Claims

1. A multi-sensor fusion pipeline drone positioning inspection control method, characterized in that: The following steps are involved: S1: Obtain real-time pipeline inspection video stream image frames through a binocular camera; S2: Determine the darkness of the image frame obtained in step S1. If the image frame does not meet the brightness requirement, perform real-time dark light enhancement on the image frame and execute step S3; If the brightness requirement is met, directly execute step S3; S3: Determine the blur degree of the image frame obtained in step S2. If the image is blurred, perform real-time deblurring on the image frame and execute step S4; if the image is not blurred, directly execute step S4; S4: performing pose calculation on the image frame obtained in step S3; S5: Output the real-time positioning data and posture data of the drone and the pipeline inspection image.

2. The multi-sensor fusion pipeline drone positioning inspection control method according to claim 1 is characterized in that: The pipeline UAV positioning inspection control method also includes judging the validity of the posture data output by the UAV. If the validity is unqualified, the UAV redundant backup system including a monocular camera and a single-line laser radar is activated, and the UAV redundant backup system is used to obtain the speed estimation of various aspects of the UAV, specifically: 1) Determine whether the error of the posture data output by the drone meets the accuracy requirements. If so, end; if not, execute step 2); 2) Enable the monocular camera on the drone and use the video stream features obtained by the monocular camera to estimate the front and back and left and right speed of the drone; 3) Enable the single-line laser radar on the drone and use the laser point cloud obtained by the single-line laser radar to estimate the up and down and left and right speed of the drone; 4) Fusing the up-down, front-back, left-right velocity estimates obtained in step 2) and step 3) to obtain velocity estimates in all directions of the drone; 5) Realize the flight control of the UAV according to the target point release.

3. The multi-sensor fusion pipeline drone positioning inspection control method according to claim 2 is characterized in that: The pipeline UAV positioning inspection control method also includes judging the validity of the output data of the redundant backup system of the UAV. If the output data of the redundant backup system of the UAV meets the accuracy requirement, the flight control of the UAV is realized according to the target point release; otherwise, the UAV is controlled to land on the spot or return or hover.

4. The multi-sensor fusion pipeline drone positioning inspection control method according to claim 1, 2 or 3, characterized in that: In step S3, a generative adversarial network (GAN) is used to perform real-time deblurring processing on the image frame.

5. The multi-sensor fusion pipeline drone positioning inspection control method according to claim 4 is characterized in that: In step S5, a simultaneous localization and mapping (SLAM) method is used to achieve the posture calculation of the UAV.

Images