Unmanned aerial vehicle automatic flight control method and system for underground cable trench inspection

Abstract

The invention relates to the field of unmanned aerial vehicle inspection, in particular to an unmanned aerial vehicle automatic flight control method and system for underground cable trench inspection, and the system comprises an environment sensing subsystem and an underground tunnel navigation subsystem. The visual sensor module obtains image data, the multi-sensor joint calibration module processes laser radar and camera data, the multi-layer fusion algorithm module fuses the calibration data, the underground tunnel navigation subsystem solves the problem of navigation obstacle avoidance, map construction of a tunnel environment and accurate estimation of the pose of an unmanned aerial vehicle are achieved, and an inspection route is planned and optimized. All the modules are integrated to achieve efficient and safe inspection of the underground cable trench, the inspection safety and efficiency are remarkably improved, the defects of a traditional manual inspection method are overcome, wide application prospects are achieved, safe operation of the underground cable can be effectively guaranteed through implementation of the system, and powerful support is provided for stability of an electric power system.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle inspection, and in particular to an automatic flight control method and system for unmanned aerial vehicle inspection of underground cable trenches. Background Art

[0002] With the continuous advancement of science and technology, the inspection of underground cable trenches has become an important part of ensuring the stable operation of the power system. The traditional manual inspection method is not only labor-intensive, but also has safety hazards due to the complex tunnel environment, and it is difficult to fully detect the potential dangers of the cable. To meet this challenge, in recent years, inspection drones have gradually become an effective alternative.

[0003] In the existing technology, due to the small space and complex structure in the cable trench, traditional drone navigation technology such as GPS cannot be effectively used in underground environments, and navigation is easily interfered by obstacles.

[0004] Therefore, it is necessary to propose an automatic flight control method and system for UAVs for underground cable trench inspection to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide an automatic flight control method and system for unmanned aerial vehicles for underground cable trench inspection, so as to solve the problem that traditional navigation technologies of unmanned aerial vehicles, such as GPS, cannot be effectively applied in underground environments and are easily interfered by obstacles during navigation.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An automatic flight control method and system for a drone for underground cable trench inspection, the method comprising:

[0008] Environmental perception and sensor calibration are performed through LiDAR and visual sensors. LiDAR emits laser beams and receives reflected signals to calculate the distance to surrounding objects and generate a high-precision 3D point cloud map of the tunnel. Visual sensors obtain tunnel image data and capture tunnel features.

[0009] Record the acquisition time of the laser radar and visual sensor, calculate the time difference, and establish the fusion model after time synchronization;

[0010] The depth camera is used to acquire data, extract and match feature points, and the Voxblox algorithm is used to fuse the camera pose data and the generated point cloud data to build a TSDF map. After optimization, underground tunnel navigation and map construction are achieved.

[0011] Preferably, when performing spatial calibration of the laser radar and the visual sensor, the laser radar and the visual sensor are fixed on a drone, wherein the fields of view of the laser radar and the visual sensor partially overlap, and the relative position and posture of the two sensors in three-dimensional space are obtained by illuminating or photographing a calibration plate or a known target point, and a coordinate mapping relationship between the two is established through a calibration matrix, so that the visual sensor camera and the laser radar data are aligned in space.

[0012] Preferably, the calibration matrix is ​​constructed as follows: the laser radar and the visual sensor are in different coordinate systems, and a conversion relationship is required so that the points in the laser radar coordinate system can be accurately mapped to the camera coordinate system, and vice versa. The coordinate mapping relationship is established through the calibration matrix. Assuming that n pairs of corresponding points are obtained through the laser radar and the visual sensor, for the i-th pair of corresponding points, the corresponding point cloud point coordinates in the laser radar are The coordinates of the corresponding points in the visual sensor image are According to the spatial transformation relationship, the following equation can be established for each pair of corresponding points:

[0013]

[0014] Among them, λ i is the scale factor, K is the internal parameter matrix of the visual sensor camera, R is the rotation matrix, T is the translation vector, and [R|T] is a 3×4 matrix formed by the combination of the rotation matrix R and the translation vector T;

[0015] and They are the homogeneous coordinate representations of the corresponding points in the laser radar coordinate system and the visual sensor camera coordinate system, respectively, to facilitate rotation and translation linear transformation operations. After expanding the equation, λ is eliminated. i , the obtained set of linear equations is transformed into the matrix form of Ax = b, where A is the coefficient matrix and x is [R 11 R 12 R 13 T1 R 21 R 22 R 23 T2 R 31 R 32 R 33 T3] T , T is represented as the transposed matrix, b is the constant term vector, and the least squares method is used to obtain the values ​​of the rotation matrix R and the translation vector T. According to the rotation matrix R and the translation vector T, assuming that the calibration matrix is ​​M, then M is

[0016] Preferably, let the coordinates of a point in the laser radar coordinate system be P lidar =(xlidar, y lidar, z lidar ) T , the corresponding point coordinates in the visual sensor are P camera =(x camera ,y camera ,z camera ) T ,By constructing and solving a set of linear equations, according to the calibration matrix M, the mapping relationship from the laser radar to the visual sensor camera coordinate system is obtained as follows: Realize the mapping from the visual sensor camera to the lidar coordinate system, using the inverse matrix M of the calibration matrix -1 , assuming that the coordinates of the corresponding point in the visual sensor camera are Q camera =(x camera ,y camera ,z camera ) T , the corresponding point coordinates in the laser radar coordinate system are Q lidar =(x lidar ,y lidar ,z lidar ) T , then the mapping relationship is In constructing the calibration matrix, the rotation operation is first performed to rotate the point in the lidar coordinate system through the rotation matrix R so that its direction is consistent with the direction of the visual sensor camera coordinate system. Then the translation operation is performed to move the rotated point to the correct position in the camera coordinate system through the translation vector T.

[0017] Preferably, the multi-layer fusion algorithm module includes time synchronization, data synchronization processing, and fusion model establishment. Time synchronization can ensure that the laser radar and visual sensor data are collected at the same time, and the data of the two are calibrated through timestamps. Assuming that the timestamp of the laser radar data collection is t1, and the timestamp of the visual sensor data collection is t2, the time difference Δt=t1-t2 is calculated. If Δt≠0, the visual sensor data is adjusted in time. If Δt>0, the collection time of the visual sensor data is delayed by Δt, so that the laser radar and visual sensor data are aligned on the time axis, eliminating the asynchronous problem. The drone uses the synchronization module it carries to ensure that the two sensors work synchronously during the inspection process.

[0018] Preferably, the data obtained after data synchronization processing dynamically adjusts the fusion ratio of lidar and visual data through the method of data fusion strategy. In narrow environments, lidar data is used preferentially for three-dimensional modeling. In scenes with good lighting and rich details, the weight of visual data is increased to obtain more accurate scene understanding. The fusion model is established according to the characteristics of the inspection environment. The system will adjust the weight distribution of lidar and visual data in real time. According to the complexity of the scene and the positioning accuracy requirements, Bayesian estimation or deep learning model is used to fuse lidar point cloud data with image data. Through this adaptive model, the perception ability of the drone in complex environments is enhanced, and the reliability and accuracy of data fusion are improved.

[0019] Preferably, in the underground tunnel navigation subsystem, RGBD data is acquired by a D435I camera, the camera pose data and the generated point cloud data are fused using a Voxblox algorithm to construct a TSDF map, and then an ESDF map is generated based on the TSDF, and a grid is used to visualize the map. The feature points are detected using an adaptive threshold FAST algorithm, and the ORB feature is homogenized using a Qtree data structure to improve the accuracy of feature matching. The feature pre-matching is performed using a FLANN algorithm, and the feature matching results are optimized in combination with a RANSAC algorithm. The camera pose is initially estimated using an EPnP algorithm, and a key frame selection strategy is designed to improve the estimation accuracy. A binary dictionary is generated using a bag-of-words model, loop detection is performed, and the pose estimation is optimized using a least squares method and a bundle adjustment method to reduce errors.

[0020] Preferably, the starting point and target point of the inspection are set on the map of the tunnel environment, and a suitable global path planning algorithm is selected to generate an initial path from the starting point to the target point. The global path is optimized according to the actual constraints of the cable trench to generate a path that can be used for navigation. Finally, local replanning is performed to ensure that the drone can avoid moving obstacles, thereby realizing autonomous inspection path planning from the starting point to the target point.

[0021] Preferably, the system includes: an environment perception subsystem and an underground tunnel navigation subsystem, wherein the environment perception subsystem is used to solve the complex perception problem in the underground cable trench environment, and the underground tunnel navigation subsystem is used to solve the problem that the UAV is difficult to stably navigate and avoid obstacles in a complex tunnel environment due to the obstruction or lack of traditional navigation;

[0022] The environmental perception subsystem includes a laser radar module, which is used for three-dimensional spatial mapping. It calculates the distance to surrounding objects by emitting laser beams and receiving reflected signals, thereby generating a high-precision three-dimensional point cloud map of the tunnel.

[0023] The environmental perception subsystem also includes a visual sensor module, which can obtain image data of the tunnel environment, capture the appearance of the cable, the surface features of the tunnel wall, and some possible signs, providing the drone with a perception method different from that of lidar.

[0024] The environment perception subsystem also includes a multi-sensor joint calibration module, which can perform spatial calibration and data format conversion and unification on the lidar and camera. The data format conversion and unification is to convert the lidar data into the same coordinate system and resolution as the camera data.

[0025] The environment perception subsystem also includes a multi-layer fusion algorithm module, which is used to fuse the calibration data of the laser radar and the camera;

[0026] The underground tunnel navigation subsystem includes a depth camera module, which is used to obtain RGBD data stream;

[0027] The underground tunnel navigation subsystem includes a SLAM algorithm module, which is used to extract and match feature points based on the data obtained by the depth camera module, thereby realizing the construction of a map of the tunnel environment and the accurate estimation of the drone's position and posture;

[0028] The underground tunnel navigation subsystem also includes a path planning module, which can reasonably plan the inspection route and optimize the inspection path according to the actual constraints of the cable trench.

[0029] Preferably, after processing the time and space synchronization problems of heterogeneous data through a multi-layer fusion algorithm, an adaptive data fusion mechanism is established. The specific contents of the data fusion mechanism include:

[0030] The positioning information module uses the positioning information fused by the laser radar and visual sensors to build a three-dimensional map of the underground corridor and locate the drone in real time. By fusing the three-dimensional point cloud data of the laser radar and the feature points in the visual image, the relative position and flight path of the drone in the corridor are determined;

[0031] Operation control information module: During the inspection process, the drone automatically adjusts its flight attitude and speed through the integrated positioning data and environmental perception data to ensure that it follows the predetermined inspection route. The system dynamically adjusts the flight altitude, speed and attitude angle according to the complexity of the underground pipeline corridor, so that the drone can safely avoid obstacles and complete the inspection task;

[0032] In the detection information module, during the inspection process, the system will detect abnormal situations in real time and compare them with the preset inspection standards. Through the data fusion mechanism, the drone can accurately locate the abnormal points in the tunnel and provide real-time feedback to the remote server. In general, the system identifies structural abnormalities in the tunnel by processing the fused image data and lidar point cloud data, and sends the abnormality detection information to the main controller, which automatically plans the next inspection task based on the detection results.

[0033] Technical effects and advantages of the present invention:

[0034] 1. The environmental perception subsystem integrates laser radar and visual sensors, and realizes the effective fusion of the two sensor data through the multi-sensor joint calibration module and the multi-layer fusion algorithm module. The laser radar provides three-dimensional spatial mapping information, and the visual sensor provides image detail information. The two complement each other, enabling the drone to fully perceive the complex environment of the underground cable trench, including the appearance of the cable, tunnel wall features and signs, etc.;

[0035] 2. The multi-sensor joint calibration module ensures that the lidar and visual sensor data are spatially aligned and have a unified coordinate system and resolution through specific spatial calibration and data format conversion methods. The multi-layer fusion algorithm module further solves the problem of heterogeneous and asynchronous data fusion, improves the accuracy and reliability of data fusion, and thus enhances the drone's ability to perceive the environment;

[0036] 3. Through a series of advanced feature point extraction, matching and optimization algorithms, the accuracy of map construction and drone positioning is improved, so that drones can accurately locate in complex tunnel environments without GPS signals, optimize inspection paths and avoid obstacles, improve inspection efficiency, and ensure that drones can complete inspection tasks safely and efficiently;

[0037] 4. The multi-layer fusion algorithm module and data fusion mechanism can be adaptively adjusted according to different environmental scenarios and task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the automatic flight control method of the present invention.

[0039] Figure 2 Schematic diagram of the structure of the environment perception subsystem of the present invention.

[0040] Figure 3 The present invention is a flow chart of a multi-sensor data fusion method for enhancing the perception capability of a UAV in a complex environment.

[0041] Figure 4 It is a schematic diagram of the structure of the underground tunnel navigation subsystem of the present invention.

[0042] Figure 5 This is a UAV positioning and high-precision map construction solution based on the D435I camera of the present invention.

[0043] Figure 6 This is the cable trench autonomous inspection path planning solution of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The present invention provides Figure 1 An automatic flight control method for a drone for underground cable trench inspection is shown, the method comprising:

[0046] S100: Environmental perception and sensor calibration are performed through lidar and visual sensors. The lidar emits laser beams and receives reflected signals to calculate the distance to surrounding objects and generate a high-precision 3D point cloud map of the tunnel. The visual sensor obtains tunnel image data and captures tunnel features.

[0047] S200: Record the acquisition time of the laser radar and visual sensor, calculate the time difference, and establish the fusion model after time synchronization;

[0048] S300: acquires data through a depth camera, extracts and matches feature points, uses the Voxblox algorithm to fuse camera pose data and generated point cloud data, builds a TSDF map, and realizes underground tunnel navigation and map construction after optimization processing;

[0049] When performing spatial calibration of the laser radar and the visual sensor, the laser radar and the visual sensor are fixed on the drone, wherein the field of view of the laser radar and the visual sensor partially overlap. The relative position and posture of the two sensors in the three-dimensional space are obtained by illuminating or photographing the calibration plate or known target points. The coordinate mapping relationship between the two is established through the calibration matrix, so that the visual sensor camera and the laser radar data are aligned in space;

[0050] Furthermore, the installation angles and positions of the lidar and visual sensor are adjusted so that their fields of view partially overlap. This setting enables the two sensors to observe part of the same area at the same time, providing the necessary conditions for obtaining corresponding points. In actual operation, the angles of the two can be accurately controlled through pre-designed mounting brackets and adjustment devices so that the overlapping area covers the key parts of the cable trench that need to be sensed, such as cable connections, turns, etc.

[0051] The calibration matrix construction method is as follows: the laser radar and the visual sensor are in different coordinate systems, and a conversion relationship is required so that the points in the laser radar coordinate system can be accurately mapped to the camera coordinate system, and vice versa. The coordinate mapping relationship is established through the calibration matrix. Assuming that n pairs of corresponding points are obtained through the laser radar and the visual sensor, for the i-th pair of corresponding points, the coordinates of the corresponding point cloud points in the laser radar are The coordinates of the corresponding points in the visual sensor image are According to the spatial transformation relationship, the following equation can be established for each pair of corresponding points:

[0052]

[0053] Among them, λ i is the scale factor, K is the internal parameter matrix of the visual sensor camera, R is the rotation matrix, T is the translation vector, and [R|T] is a 3×4 matrix formed by the combination of the rotation matrix R and the translation vector T;

[0054] and They are the homogeneous coordinate representations of the corresponding points in the laser radar coordinate system and the visual sensor camera coordinate system, respectively, to facilitate rotation and translation linear transformation operations. After expanding the equation, λ is eliminated. i , the obtained set of linear equations is transformed into the matrix form of Ax = b, where A is the coefficient matrix and x is [R 11 R 12 R 13 T1 R 21 R 22 R 23 T2 R 31 R 32 R 33 T3] T , T is represented as the transposed matrix, b is the constant term vector, and the least squares method is used to obtain the values ​​of the rotation matrix R and the translation vector T. According to the rotation matrix R and the translation vector T, assuming that the calibration matrix is ​​M, then M is

[0055] Suppose the coordinates of a point in the laser radar coordinate system are P lidar =(x lidar, ylidar, z lidar ) T , the corresponding point coordinates in the visual sensor are P camera =(x camera ,y camera ,z camera ) T ,By constructing and solving a set of linear equations, according to the calibration matrix M, the mapping relationship from the laser radar to the visual sensor camera coordinate system is obtained as follows: Realize the mapping from the visual sensor camera to the lidar coordinate system, using the inverse matrix M of the calibration matrix -1 , assuming that the coordinates of the corresponding point in the visual sensor camera are Q camera =(x camera ,y camera ,z camera ) T , the corresponding point coordinates in the laser radar coordinate system are Q lidar =(x lidar ,y lidar ,z lidar ) T , then the mapping relationship is In constructing the calibration matrix, the rotation operation is first performed to rotate the point in the laser radar coordinate system through the rotation matrix R so that its direction is consistent with the direction of the visual sensor camera coordinate system. Then the translation operation is performed to move the rotated point to the correct position in the camera coordinate system through the translation vector T.

[0056] Furthermore, for each pair of corresponding points, an equation is established based on the spatial transformation relationship of the different coordinate systems of the laser radar and the visual sensor. i is the scale factor, K is the internal parameter matrix of the visual sensor camera (including the focal length, principal point position and other internal parameters of the camera, which are usually calibrated when the camera leaves the factory or can be predetermined by a specific calibration method), R is the rotation matrix, T is the translation vector, and [R|T] is a 3×4 matrix formed by the combination of the rotation matrix R and the translation vector T; and They are the homogeneous coordinate representations of corresponding points in the laser radar coordinate system and the visual sensor camera coordinate system respectively. The introduction of homogeneous coordinates enables linear transformation operations such as rotation and translation to be unified in the form of matrix multiplication, which facilitates subsequent calculations.

[0057] Furthermore, the above equation is expanded and λ is eliminated. i, get a set of linear equations, and convert them into the matrix form of Ax=b, then use the least squares method to solve Ax=b to get the values ​​of the rotation matrix R and the translation vector T. The principle of the least squares method is to minimize the sum of squares of errors. In this case, it is to minimize the error between the coordinate mapping relationship obtained by calculation and the coordinates of the corresponding points actually measured, so as to obtain the rotation matrix and translation vector closest to the actual situation.

[0058] Furthermore, according to the obtained rotation matrix R and translation vector T, the calibration matrix M is determined in a set manner. The calibration matrix M is the key transformation matrix that accurately maps points in the lidar coordinate system to the camera coordinate system. It integrates the rotation and translation information and realizes the coordinate mapping relationship between the two coordinate systems. Through the calibration matrix, the coordinates of any point in the lidar coordinate system can be converted to the coordinates in the visual sensor camera coordinate system, and vice versa, so that the visual sensor camera and lidar data are spatially aligned, laying the foundation for subsequent data fusion and environmental perception.

[0059] The multi-layer fusion algorithm module includes time synchronization, data synchronization processing, and fusion model establishment. Time synchronization can ensure that the laser radar and visual sensor data are collected at the same time. The data of the two are calibrated through timestamps. Assuming that the timestamp of the laser radar data collection is t1, and the timestamp of the visual sensor data collection is t2, the time difference Δt=t1-t2 is calculated. If Δt≠0, the visual sensor data is adjusted in time. If Δt>0, the collection time of the visual sensor data is delayed by Δt, so that the laser radar and visual sensor data are aligned on the time axis, eliminating the asynchronous problem. The drone uses the synchronization module to ensure that the two sensors work synchronously during the inspection process.

[0060] The data obtained after data synchronization processing dynamically adjusts the fusion ratio of LiDAR and visual data through data fusion strategy. In narrow environments, LiDAR data is used first for 3D modeling. In scenes with good light and rich details, the weight of visual data is increased to obtain more accurate scene understanding. The fusion model is established according to the characteristics of the inspection environment. The system will adjust the weight distribution of LiDAR and visual data in real time. According to the complexity of the scene and the positioning accuracy requirements, Bayesian estimation or deep learning model is used to fuse LiDAR point cloud data with image data. Through this adaptive model, the perception ability of drones in complex environments is enhanced, and the reliability and accuracy of data fusion are improved.

[0061] Furthermore, the multi-layer fusion algorithm module plays a vital role in the entire system. Its time synchronization function is achieved by marking the timestamps t1 and t2 for the data collected by the lidar and visual sensors. After calculating the time difference between the two, the visual sensor data is accurately adjusted according to the difference. When the time difference meets specific conditions, the visual sensor data is moved forward or delayed to ensure that the data of the two are perfectly aligned on the time axis, thereby eliminating the asynchronous problem. The synchronization module carried by the drone is like a command center, continuously monitoring and coordinating the work of the two sensors to ensure that they are always synchronized throughout the inspection process, providing a solid foundation for time consistency for subsequent data fusion processing, so that the system can perceive the underground cable trench environment more accurately and stably, and effectively improve the reliability and accuracy of the inspection task.

[0062] In the underground tunnel navigation subsystem, RGBD data is obtained through the D435I camera, and the camera pose data and the generated point cloud data are fused using the Voxblox algorithm to construct a TSDF map. Then, the ESDF map is generated based on the TSDF, and the map is visualized using a grid. The FAST algorithm with an adaptive threshold is used to detect feature points, and the Qtree data structure is used to homogenize the ORB features to improve the accuracy of feature matching. The FLANN algorithm is used for feature pre-matching, and the feature matching results are optimized in combination with the RANSAC algorithm. The camera pose is initially estimated using the EPnP algorithm, and a key frame selection strategy is designed to improve the estimation accuracy. The bag-of-words model is used to generate a binary dictionary, and loop detection is performed. The least squares method and bundle adjustment method are used to optimize the pose estimation to reduce errors.

[0063] Set the inspection start and target points on the map of the tunnel environment, select a suitable global path planning algorithm to generate an initial path from the start to the target, optimize the global path according to the actual constraints of the cable trench, generate a path that can be used for navigation, and finally perform local replanning to ensure that the drone can avoid moving obstacles, thereby realizing autonomous inspection path planning from the start to the target;

[0064] Furthermore, in the underground tunnel navigation subsystem, the D435I camera is used to obtain RGBD data, which covers color information (RGB) and depth information, providing a rich source of information for subsequent processing. Then, the Voxblox algorithm is used to merge the camera pose data with the generated point cloud data to construct a TSDF map, and then the ESDF map is generated based on the TSDF. The map is visualized with the help of a grid, so that the tunnel environment is presented in an intuitive form. The feature points are accurately detected by the FAST algorithm with an adaptive threshold, and the ORB feature is homogenized by using the Qtree data structure, which effectively improves the accuracy of feature matching. The FLANN algorithm is used to carry out feature pre-matching, and the matching results are optimized in combination with the RANSAC algorithm. Then, the EPnP algorithm is used to preliminarily estimate the camera pose, and the key frame selection strategy is carefully designed to further improve the accuracy of pose estimation. The bag-of-words model is used to generate a binary dictionary for loop detection, and the least squares method and bundle adjustment method are used to optimize the pose estimation to minimize the error. After setting the starting point and target point of the inspection on the tunnel environment map, select an appropriate global path planning algorithm to generate an initial path, then optimize the path according to the actual constraints of the cable trench to obtain a navigable path, and finally perform local replanning to ensure that the drone can flexibly avoid moving obstacles during the inspection process, and ultimately successfully achieve autonomous inspection path planning from the starting point to the target point, ensuring that the entire inspection task is completed efficiently, safely and accurately.

[0065] The present invention provides Figure 2-6 An automatic flight control system for an unmanned aerial vehicle for underground cable trench inspection, the system comprising: an environment perception subsystem and an underground tunnel navigation subsystem, the environment perception subsystem is used to solve the complex perception problem in the underground cable trench environment, and the underground tunnel navigation subsystem is used to solve the problem that the unmanned aerial vehicle is difficult to stably navigate and avoid obstacles in a complex tunnel environment due to the obstruction or lack of traditional navigation;

[0066] The environmental perception subsystem includes a laser radar module, which is used for three-dimensional spatial mapping. It calculates the distance to surrounding objects by emitting laser beams and receiving reflected signals, thereby generating a high-precision three-dimensional point cloud map of the tunnel.

[0067] The environmental perception subsystem also includes a visual sensor module, which can obtain image data of the tunnel environment, capture the appearance of the cable, the surface features of the tunnel wall, and some possible signs, providing the drone with a perception method different from that of lidar.

[0068] The environment perception subsystem also includes a multi-sensor joint calibration module, which can perform spatial calibration and data format conversion and unification on the lidar and camera. The data format conversion and unification is to convert the lidar data into the same coordinate system and resolution as the camera data.

[0069] The environment perception subsystem also includes a multi-layer fusion algorithm module, which is used to fuse the calibration data of the laser radar and the camera;

[0070] The underground tunnel navigation subsystem includes a depth camera module, which is used to obtain RGBD data stream;

[0071] The underground tunnel navigation subsystem includes a SLAM algorithm module, which is used to extract and match feature points based on the data obtained by the depth camera module, thereby realizing the construction of a map of the tunnel environment and the accurate estimation of the drone's position and posture;

[0072] The underground tunnel navigation subsystem also includes a path planning module, which can reasonably plan the inspection route and optimize the inspection path according to the actual constraints of the cable trench;

[0073] After processing the time and space synchronization issues of heterogeneous data through multi-layer fusion algorithms, an adaptive data fusion mechanism is established. The specific contents of the data fusion mechanism include:

[0074] The positioning information module uses the positioning information fused by the laser radar and visual sensors to build a three-dimensional map of the underground corridor and locate the drone in real time. By fusing the three-dimensional point cloud data of the laser radar and the feature points in the visual image, the relative position and flight path of the drone in the corridor are determined;

[0075] Operation control information module: During the inspection process, the drone automatically adjusts its flight attitude and speed through the integrated positioning data and environmental perception data to ensure that it follows the predetermined inspection route. The system dynamically adjusts the flight altitude, speed and attitude angle according to the complexity of the underground pipeline corridor, so that the drone can safely avoid obstacles and complete the inspection task;

[0076] In the detection information module, during the inspection process, the system will detect abnormal situations in real time and compare them with the preset inspection standards. Through the data fusion mechanism, the drone can accurately locate the abnormal points in the tunnel and provide real-time feedback to the remote server. In general, the system identifies structural abnormalities in the tunnel by processing the fused image data and lidar point cloud data, and sends the abnormality detection information to the main controller, which automatically plans the next inspection task based on the detection results.

[0077] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for automatic flight control of a drone for underground cable trench inspection, characterized in that: The method comprises: Environmental perception and sensor calibration are performed through LiDAR and visual sensors. LiDAR emits laser beams and receives reflected signals to calculate the distance to surrounding objects and generate a high-precision 3D point cloud map of the tunnel. Visual sensors obtain tunnel image data and capture tunnel features. Record the acquisition time of the laser radar and visual sensor, calculate the time difference, and establish the fusion model after time synchronization; The depth camera is used to acquire data, extract and match feature points, and the Voxblox algorithm is used to fuse the camera pose data and the generated point cloud data to build a TSDF map. After optimization, underground tunnel navigation and map construction are achieved.

2. The automatic flight control method of a drone for underground cable trench inspection according to claim 1 is characterized by: When performing spatial calibration of the LiDAR and vision sensor, the LiDAR and vision sensor are fixed on the UAV, where the fields of view of the LiDAR and vision sensor partially overlap. The relative position and posture of the two sensors in three-dimensional space are obtained by illuminating or photographing a calibration plate or known target point. A coordinate mapping relationship between the two is established through the calibration matrix, so that the vision sensor camera and LiDAR data are aligned in space.

3. The automatic flight control method of a drone for underground cable trench inspection according to claim 1 is characterized in that: The construction method of the calibration matrix is ​​as follows: the laser radar and the visual sensor are in different coordinate systems, and a conversion relationship is required so that the points in the laser radar coordinate system can be accurately mapped to the camera coordinate system. Conversely, the coordinate mapping relationship is established through the calibration matrix. Assuming that n pairs of corresponding points are obtained through the laser radar and the visual sensor, for the i-th pair of corresponding points, the coordinates of the corresponding point cloud points in the laser radar are The coordinates of the corresponding points in the visual sensor image are According to the spatial transformation relationship, the following equation can be established for each pair of corresponding points: Among them, λ i is the scale factor, K is the internal parameter matrix of the visual sensor camera, R is the rotation matrix, T is the translation vector, and [R|T] is a 3×4 matrix formed by the combination of the rotation matrix R and the translation vector T; and They are the homogeneous coordinate representations of the corresponding points in the laser radar coordinate system and the visual sensor camera coordinate system, respectively, to facilitate rotation and translation linear transformation operations. After expanding the equation, λ is eliminated. i , the obtained set of linear equations is transformed into the matrix form of Ax = b, where A is the coefficient matrix and x is [R 11 R 12 R 13 T1 R 21 R 22 R 23 T2 R 31 R 32 R 33 T3] T , T is represented as the transposed matrix, b is the constant term vector, and the least squares method is used to obtain the values ​​of the rotation matrix R and the translation vector T. According to the rotation matrix R and the translation vector T, assuming that the calibration matrix is ​​M, then M is 4. The automatic flight control method of a drone for underground cable trench inspection according to claim 3 is characterized by: Suppose the coordinates of a point in the laser radar coordinate system are P lidar =(x lidar, y lidar, z lidar ) T , the corresponding point coordinates in the visual sensor are P camera =(x camera ,y camera ,z camera ) T ,By constructing and solving a set of linear equations, according to the calibration matrix M, the mapping relationship from the laser radar to the visual sensor camera coordinate system is obtained as follows: Realize the mapping from the visual sensor camera to the lidar coordinate system, using the inverse matrix M of the calibration matrix -1 , assuming that the coordinates of the corresponding point in the visual sensor camera are Q camera =(x camera ,y camera ,z camera ) T , the corresponding point coordinates in the laser radar coordinate system are Q lidar =(x lidar ,y lidar ,z lidar ) T , then the mapping relationship is In constructing the calibration matrix, the rotation operation is first performed to rotate the point in the lidar coordinate system through the rotation matrix R so that its direction is consistent with the direction of the visual sensor camera coordinate system. Then the translation operation is performed to move the rotated point to the correct position in the camera coordinate system through the translation vector T.

5. The automatic flight control method of a drone for underground cable trench inspection according to claim 1 is characterized by: The multi-layer fusion algorithm module includes time synchronization, data synchronization processing, and fusion model establishment. Time synchronization can ensure that the laser radar and visual sensor data are collected at the same time, and the data of the two are calibrated through timestamps. Assuming that the timestamp of the laser radar data collection is t1, and the timestamp of the visual sensor data collection is t2, the time difference Δt=t1-t2 is calculated. If Δt≠0, the time of the visual sensor data is adjusted. If Δt>0, the collection time of the visual sensor data is delayed by Δt, so that the laser radar and visual sensor data are aligned on the time axis, eliminating the asynchronous problem. The drone uses the synchronization module to ensure that the two sensors work synchronously during the inspection process.

6. The automatic flight control method of a drone for underground cable trench inspection according to claim 5 is characterized by: The data obtained after data synchronization processing dynamically adjusts the fusion ratio of lidar and visual data through data fusion strategy. In narrow environments, lidar data is used for 3D modeling first. In scenes with good lighting and rich details, the weight of visual data is increased to obtain more accurate scene understanding. The fusion model is established according to the characteristics of the inspection environment. The system will adjust the weight distribution of lidar and visual data in real time. According to the complexity of the scene and the positioning accuracy requirements, Bayesian estimation or deep learning model is used to fuse lidar point cloud data with image data. Through this adaptive model, the perception ability of drones in complex environments is enhanced, and the reliability and accuracy of data fusion are improved.

7. The automatic flight control method of a drone for underground cable trench inspection according to claim 1 is characterized by: In the underground tunnel navigation subsystem, RGBD data is obtained through the D435I camera, and the camera pose data and the generated point cloud data are fused using the Voxblox algorithm to construct a TSDF map. Then, the ESDF map is generated based on the TSDF, and the grid is used to visualize the map. The FAST algorithm with adaptive threshold is used to detect feature points, and the Qtree data structure is used to homogenize the ORB features to improve the accuracy of feature matching. The FLANN algorithm is used for feature pre-matching, and the feature matching results are optimized in combination with the RANSAC algorithm. The camera pose is initially estimated by the EPnP algorithm, and a key frame selection strategy is designed to improve the estimation accuracy. The bag-of-words model is used to generate a binary dictionary, and loop detection is performed. The least squares method and bundle adjustment method are used to optimize the pose estimation to reduce errors.

8. The automatic flight control method and system of a drone for underground cable trench inspection according to claim 1, characterized in that: The starting point and target point of the inspection are set on the map of the tunnel environment, and an appropriate global path planning algorithm is selected to generate the initial path from the starting point to the target point. The global path is optimized according to the actual constraints of the cable trench to generate a path that can be used for navigation. Finally, local replanning is performed to ensure that the drone can avoid moving obstacles, thereby realizing autonomous inspection path planning from the starting point to the target point.

9. An automatic flight control system for unmanned aerial vehicles for underground cable trench inspection, characterized in that: The system includes: an environment perception subsystem and an underground tunnel navigation subsystem. The environment perception subsystem is used to solve the complex perception problem in the underground cable trench environment. The underground tunnel navigation subsystem is used to solve the problem that it is difficult for the UAV to stably navigate and avoid obstacles in a complex tunnel environment due to the obstruction or lack of traditional navigation. The environmental perception subsystem includes a laser radar module, which is used for three-dimensional spatial mapping. It calculates the distance to surrounding objects by emitting laser beams and receiving reflected signals, thereby generating a high-precision three-dimensional point cloud map of the tunnel. The environmental perception subsystem also includes a visual sensor module, which can obtain image data of the tunnel environment, capture the appearance of the cable, the surface features of the tunnel wall, and some possible signs, providing the drone with a perception method different from that of lidar. The environment perception subsystem also includes a multi-sensor joint calibration module, which can perform spatial calibration and data format conversion and unification on the lidar and camera. The data format conversion and unification is to convert the lidar data into the same coordinate system and resolution as the camera data. The environment perception subsystem also includes a multi-layer fusion algorithm module, which is used to fuse the calibration data of the laser radar and the camera; The underground tunnel navigation subsystem includes a depth camera module, which is used to obtain RGBD data stream; The underground tunnel navigation subsystem includes a SLAM algorithm module, which is used to extract and match feature points based on the data obtained by the depth camera module, thereby realizing the construction of a map of the tunnel environment and the accurate estimation of the drone's position and posture; The underground tunnel navigation subsystem also includes a path planning module, which can reasonably plan the inspection route and optimize the inspection path according to the actual constraints of the cable trench.

10. The automatic flight control system of a drone for underground cable trench inspection according to claim 9, characterized in that: After processing the time and space synchronization issues of heterogeneous data through multi-layer fusion algorithms, an adaptive data fusion mechanism is established. The specific contents of the data fusion mechanism include: The positioning information module uses the positioning information fused by the laser radar and visual sensors to build a three-dimensional map of the underground corridor and locate the drone in real time. By fusing the three-dimensional point cloud data of the laser radar and the feature points in the visual image, the relative position and flight path of the drone in the corridor are determined; Operation control information module: During the inspection process, the drone automatically adjusts its flight attitude and speed through the integrated positioning data and environmental perception data to ensure that it follows the predetermined inspection route. The system dynamically adjusts the flight altitude, speed and attitude angle according to the complexity of the underground pipeline corridor, so that the drone can safely avoid obstacles and complete the inspection task; In the detection information module, during the inspection process, the system will detect abnormal situations in real time and compare them with the preset inspection standards. Through the data fusion mechanism, the drone can accurately locate the abnormal points in the tunnel and provide real-time feedback to the remote server. In general, the system identifies structural abnormalities in the tunnel by processing the fused image data and lidar point cloud data, and sends the abnormality detection information to the main controller, which automatically plans the next inspection task based on the detection results.

Images