Unmanned aerial vehicle autonomous flight and inspection system based on multi-sensor fusion technology

Through multi-sensor fusion technology, visual, lidar and IMU data are integrated to generate high-precision three-dimensional environmental maps and position estimation, solving the problem of difficult positioning and inspection in complex environments in traditional drone inspection systems, and achieving high-precision autonomous flight and inspection.

CN119987412APending Publication Date: 2025-05-13SHANXI JIAOKE INFORMATION SYST ENG CO LTD +1
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202411960908.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-12-30
Publication Date
2025-05-13

AI Technical Summary

Technical Problem

Traditional UAV patrol systems are difficult to achieve high-precision positioning and patrol in complex environments, especially when there is no GNSS signal, more occlusions or insufficient light, a single sensor is difficult to meet the needs of high-precision patrol.

Method used

Using multi-sensor fusion technology, data from sensors such as vision cameras, lidars, inertial measurement units (IMUs) are integrated, and through improved adaptive extended Kalman filtering algorithms and synchronous positioning and mapping algorithms, high-precision three-dimensional environmental maps and pose estimation are generated to achieve autonomous path planning and intelligent obstacle avoidance.

Benefits of technology

It improves the inspection capability and positioning accuracy of the drone in complex environments, enhances the environmental perception capability, realizes high-precision autonomous flight and patrol, and significantly improves the accuracy and stability of the patrol.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119987412A_ABST
    Figure CN119987412A_ABST
Patent Text Reader

Abstract

The invention relates to an unmanned aerial vehicle autonomous flight and inspection system based on a multi-sensor fusion technology, which integrates various sensors such as a visual camera, a laser radar, an inertial measurement unit and a GPS (Global Positioning System), and realizes hierarchical fusion processing of multi-sensor data based on improved adaptive extended Kalman filtering. And a high-precision three-dimensional environment map and pose estimation of the unmanned aerial vehicle are generated in combination with a synchronous positioning and mapping algorithm. And then, the path planning module generates an optimal flight path according to the inspection task area and the environment structure, and automatically calls an obstacle avoidance algorithm to perform dynamic path adjustment when encountering an obstacle. And then, the visual detection module analyzes, identifies and marks the real-time image data through a deep learning algorithm, and locates abnormal conditions in the target area. The system has remarkable advantages in the aspects of inspection precision, stability and adaptability, and is suitable for multiple fields such as geological inspection, road inspection, power inspection, traffic facility detection and disaster monitoring.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to an unmanned aerial vehicle autonomous flight and inspection system based on multi-sensor fusion technology, which can adapt to inspection tasks in complex environments. Background Art

[0002] With the rapid development of drone technology, drones have been widely used in many industries, including agriculture, energy, transportation, security and infrastructure inspection. In inspection tasks, drones have gradually become an important means to replace traditional human inspections with their flexibility, maneuverability and efficiency. However, traditional drone inspection systems usually rely on a single sensor, such as GPS, vision or lidar, and still face some technical challenges in practical applications, especially in complex environments, such as no GNSS signal, many obstructions or insufficient light. The information provided by a single sensor is often difficult to meet the needs of high-precision inspections, and even cannot work effectively. Therefore, multi-sensor fusion technology has become an inevitable trend in the development of drone inspection systems. By combining multiple sensors such as vision, lidar, inertial measurement unit (IMU), the environmental perception ability of drones can be enhanced to achieve accurate positioning and intelligent inspections. Summary of the invention

[0003] The main purpose of the present invention is to provide a UAV autonomous flight and inspection system based on multi-sensor fusion technology, which improves the inspection capability and positioning accuracy of UAVs in complex environments by integrating information from multiple sensors, and solves the problem that UAV inspection systems in the prior art are easily restricted in specific environments.

[0004] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] An autonomous flight and inspection system for unmanned aerial vehicles based on multi-sensor fusion technology includes: an unmanned aerial vehicle platform, a multi-sensor layered fusion module, an inspection task module, an image recognition and detection module, a data transmission and storage module, and a ground control station. The working process includes the following steps:

[0006] 1) During the inspection process, the visual camera, lidar and IMU collect data in real time and upload it to the onboard computing platform;

[0007] 2) The multi-sensor hierarchical fusion module integrates vision, LiDAR and IMU data to generate high-precision 3D environment maps and pose estimates based on the SLAM algorithm;

[0008] 3) The path planning algorithm in the inspection task module generates the optimal inspection path according to the inspection task area and environmental structure; when encountering obstacles, the obstacle avoidance algorithm is automatically called to correct the path;

[0009] 4) The visual detection algorithm in the image recognition and detection module analyzes the visual data, identifies and marks abnormal conditions of the inspection targets, and transmits the UAV inspection data to the ground station in real time through the wireless communication module for monitoring and subsequent analysis.

[0010] Furthermore, the multi-sensor hierarchical fusion module integrates the depth information of the lidar, the image information of the binocular camera, the attitude data provided by the IMU and the GPS data in real time based on the improved adaptive extended Kalman filter algorithm to improve the positioning accuracy and inspection stability of the drone.

[0011] Furthermore, the working process of the multi-sensor hierarchical fusion module includes the following steps:

[0012] 1) Collect the sensor data carried by the drone, and store the collected data in the storage space opened up by the algorithm;

[0013] 2) Multi-sensor spatiotemporal calibration: Spatiotemporal calibration includes coordinate transformation and data alignment. The sensor data is converted to the same coordinate system through coordinate transformation. LiDAR is selected as the main sensor, and its frequency is used to determine the time step of the sliding window. The timestamp of the LiDAR data is selected as the time frame to synchronize the multi-sensor data.

[0014] 3) Preprocessing of sensor data: The raw data obtained by IMU are acceleration and angular velocity. Pre-integration is performed based on the IMU pre-integration algorithm to obtain the rotation and translation transformation between two frames. After obtaining the IMU data, it is incrementally combined with the barometer data to obtain the relative position and posture of the drone relative to the starting state.

[0015] 4) The data of IMU and barometer, and the data of visual camera and LiDAR are fused and calculated respectively, and the two groups of estimators constitute the first level of sensor fusion; the IMU and barometer data fusion algorithm is based on the improved adaptive extended Kalman filter algorithm, and the visual camera and LiDAR data fusion is based on the synchronous positioning and mapping algorithm. The two estimators obtain a group of UAV pose estimates respectively;

[0016] 5) The two sets of pose estimates obtained by the two estimators of the first level are fused to provide a global optimal estimate: the covariance crossover algorithm is used to fuse and jointly optimize the estimates with unknown correlations, and the final pose estimation result is output.

[0017] Furthermore, the image recognition and detection module uses a deep learning algorithm to perform real-time processing and analysis on the images collected by the binocular camera, identify abnormal conditions of the inspection targets, including defects, cracks and rust, and achieve accurate detection and classification of the inspection targets by training a deep neural network model.

[0018] Furthermore, the data transmission and storage module transmits the inspection data to the ground station in real time through wireless communication equipment, and performs data backup locally on the drone. The inspection data includes images, point cloud data, and flight trajectories, which can be used for subsequent analysis and troubleshooting.

[0019] Compared with the prior art, the UAV inspection system based on multi-sensor fusion proposed in the present invention integrates and integrates multiple sensor information such as visual cameras, LiDAR, IMU, GPS, etc., realizes high-precision positioning, three-dimensional mapping, autonomous path planning and intelligent obstacle avoidance of UAVs in complex environments, improves the accuracy and stability of inspections, and shows significant advantages in complex or dynamic environments. Through the fusion of multi-sensor data, the present invention has strong robustness in precise positioning, obstacle identification and path planning, and provides reliable guarantee for the efficient completion of inspection tasks. The system innovatively combines deep learning algorithms to analyze visual data in real time, realizes the identification of abnormal conditions of inspection targets, such as cracks, damage, wear, etc., and has high-precision automatic labeling and classification functions. In addition, based on the multi-source data hierarchical fusion algorithm, the system can optimize the real-time posture and position of the UAV, significantly improving the data accuracy and system stability during the inspection process. By fusing sensor information, the adaptability of the system in terms of flight altitude, posture and environmental perception far exceeds that of traditional single-sensor inspection systems. Compared with the traditional UAV inspection system, the present invention has lower cost and wider coverage, and is suitable for large-scale and diversified inspection tasks. In addition, data analysis based on multi-sensor fusion provides detailed deformation laws, which provides important technical support and scientific basis for geological disaster warning and infrastructure inspection. Therefore, the system has broad application potential and promotion value in the inspection fields of roads, railways, bridges, power lines, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is the structural diagram of the UAV platform based on multi-sensor fusion.

[0021] Figure 2 This is the flow chart of multi-sensor hierarchical fusion algorithm

[0022] Figure 3 It is a flow chart of the autonomous flight and inspection algorithm of UAV based on multi-sensor fusion technology. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solution of the present invention, the drone inspection system based on multi-sensor fusion provided by the present invention is described in detail below in conjunction with the embodiments. The following embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0024] An autonomous flight and inspection system for unmanned aerial vehicles based on multi-sensor fusion technology. It aims to improve the autonomous positioning, path planning, obstacle avoidance and inspection capabilities of unmanned aerial vehicles in complex environments. First, the system integrates multiple sensors such as visual cameras, lidar, inertial measurement units and GPS, and realizes hierarchical fusion processing of multi-sensor data based on the improved adaptive extended Kalman filter, and combines the synchronous positioning and mapping algorithm to generate a high-precision three-dimensional environmental map and the pose estimation of the unmanned aerial vehicle. Then, the system uses the path planning module to generate the optimal flight path according to the inspection task area and environmental structure, and automatically calls the obstacle avoidance algorithm to adjust the dynamic path when encountering obstacles, ensuring the safety and efficiency of the unmanned aerial vehicle inspection. Next, the visual detection module analyzes the real-time image data through the deep learning algorithm, identifies, marks and locates the abnormal situation in the target area. Finally, the detection results are transmitted to the ground station through the wireless communication module for monitoring personnel to view and subsequent data analysis. Compared with the existing technology, this system has significant advantages in the accuracy, stability and adaptability of inspection, and is suitable for geological inspection, highway inspection, power inspection, transportation facility inspection, disaster monitoring and other fields.

[0025] An autonomous flight and inspection system for unmanned aerial vehicles (UAVs) based on multi-sensor fusion technology comprises: an unmanned aerial vehicle platform, a multi-sensor layered fusion module, an inspection task module, an image recognition and detection module, a data transmission and storage module, and a ground control station.

[0026] The drone platform is used to carry the sensors required to perform the task, including but not limited to visual cameras, laser radars, inertial measurement units (IMUs) and GPS modules. The drone has autonomous flight capabilities and can automatically adjust its flight trajectory during the inspection process.

[0027] The multi-sensor hierarchical fusion positioning module is used to fuse the data from each sensor. Based on the improved adaptive extended Kalman filter (EKF) algorithm, the depth information of the laser radar, the image information of the binocular camera, the attitude data provided by the IMU and the GPS data are integrated in real time to improve the positioning accuracy and inspection stability of the drone.

[0028] The inspection task module is used to receive and process the parameters of the inspection task, including the inspection area, target detection requirements, etc. The drone automatically plans the path according to the preset task, and uses multi-sensor data to adjust the flight route in real time to ensure coverage of the designated inspection area.

[0029] The image recognition and detection module uses a deep learning algorithm to process and analyze the images collected by the binocular camera in real time to identify abnormal conditions of the inspection target, such as defects, cracks, rust, etc. By training a deep neural network model, accurate detection and classification of the inspection target can be achieved.

[0030] The data transmission and storage module transmits the inspection data to the ground station in real time through wireless communication equipment, and performs data backup locally on the drone. The inspection data includes images, point cloud data, and flight trajectory, which can be used for subsequent analysis and troubleshooting.

[0031] The ground control station is used to receive real-time data from the drone, display information such as inspection progress, drone status, abnormal conditions, etc., and can intervene in or adjust the flight of the drone through remote control.

[0032] Example:

[0033] like Figure 1 As shown, the UAV platform based on multi-sensor fusion includes: a UAV frame, a power kit, a flight controller, an airborne computing platform and multiple sensors.

[0034] The drone frame is made of high-strength carbon fiber material, which is lightweight and high-strength, suitable for complex inspection environments. The drone adopts a four-rotor structure, 300mm long, 300mm wide, and 165mm high to ensure compact structure and flight stability.

[0035] The power kit is composed of a motor, an ESC and propellers. The motor is connected to the ESC to provide continuous and stable power output, so that the UAV can maintain stable flight during inspection tasks.

[0036] The flight controller uses Pixhawk firmware, and the flight controller is connected to a galvanometer, an electric regulator, a GPS, a downward-looking laser sensor, and a wireless communication module. The downward-looking laser sensor is used to measure the flight altitude of the drone in real time to improve the accuracy of autonomous positioning; the wireless communication module realizes real-time communication between the drone and the ground station for data transmission and remote control; the galvanometer is used to monitor the power of the drone for easy battery management; the GPS module is located on the top of the drone, and when there is a GNSS signal, the GPS provides auxiliary position estimation and provides reference information for multi-sensor data fusion.

[0037] The onboard computing platform runs the Linux system and is responsible for data collection, preprocessing and analysis. The computing platform has high-performance computing capabilities and can process visual, lidar and IMU data simultaneously, ensuring real-time processing of multi-source data and algorithm operations.

[0038] The multi-sensor mainly includes a visual camera, a laser radar (LiDAR), an inertial navigation sensor (IMU), a barometer and a downward-looking laser sensor. Among them, the downward-looking laser sensor is connected to the flight controller to provide the relative height information between the UAV and the ground; the visual camera, LiDAR and IMU data are transmitted to the onboard computer, and the sensor data is fused through a multi-sensor fusion algorithm, combined with a simultaneous localization and mapping (SLAM) algorithm to estimate the position and attitude of the UAV, and the target is identified and detected based on the visual detection algorithm.

[0039] The multi-sensor hierarchical fusion algorithm performs hierarchical fusion processing on multiple sensor data based on the improved extended Kalman filter (EKF) algorithm. Through the improved EKF algorithm, the system combines the attitude information of the IMU, the barometer data information, and the data of the visual camera and LiDAR to eliminate the error accumulation of a single sensor. The fusion of multi-sensor hierarchical data not only improves the positioning accuracy of the system, but also enhances its stability in complex environments, especially in the absence of GPS signals or weak signals, relying on visual and LiDAR information to achieve high-precision positioning.

[0040] Figure 2 This is the algorithm flow chart of the multi-sensor hierarchical fusion algorithm. The specific steps are as follows:

[0041] The first step is to collect the sensor data carried by the drone, and store the collected data in the storage space opened up by the algorithm.

[0042] Step 2: Multi-sensor spatiotemporal calibration. Spatiotemporal calibration includes coordinate transformation and data alignment. Multi-sensor involves the mutual transformation of multiple coordinate systems. The sensor data is converted to the same coordinate system through coordinate transformation so that they are aligned in space. In addition, since the frequency of each sensor is different, it is necessary to synchronize the data from different sensors. LiDAR is selected as the main sensor, and its frequency is used to determine the time step of the sliding window. The timestamp of the LiDAR data is selected as the time frame to synchronize the multi-sensor data.

[0043] Step 3: Preprocess the sensor data. The raw data obtained by the IMU are acceleration and angular velocity. Pre-integration is performed based on the IMU pre-integration algorithm to obtain the rotation and translation transformation between the two frames. In addition, after obtaining the IMU data, it is necessary to incrementally combine it with the barometer data to infer the relative position of the drone relative to the starting state.

[0044] Step 4: The data of IMU and barometer, and the data of visual camera and LiDAR are fused and calculated respectively, and the two groups of estimators form the first level of sensor fusion. The IMU and barometer data fusion algorithm is based on the improved adaptive extended Kalman filter algorithm, and the visual camera and LiDAR data fusion is based on the synchronous positioning and mapping algorithm. The two estimators obtain a set of drone pose estimates respectively.

[0045] Step 5: The two sets of pose estimates obtained by the two estimators in the first stage are fused to provide a global optimal estimate. The covariance crossover algorithm is used to fuse and jointly optimize the estimates with unknown correlations and output the final pose estimation result.

[0046] The sensor data preprocessing algorithm includes an IMU pre-integration algorithm and an incremental combination algorithm, which preliminarily combines the IMU data and the barometer data to prepare for subsequent data fusion.

[0047] The IMU pre-integration algorithm converts the acceleration of the IMU at consecutive moments into and angular velocity Data integration obtains the rotation and translation transformation between two frames of data.

[0048]

[0049] in

[0050]

[0051]

[0052] Among them, b a is the acceleration bias, b ω is the gyroscope bias, n is the noise, is the rotation matrix from the nth frame to the tth frame, g n is the gravity vector in the navigation frame, p is the position vector, v is the velocity vector, and q is the rotation matrix.

[0053]

[0054] The incremental combination algorithm uses the attitude and heading information obtained by IMU pre-integration to update the state, concentrates both sides of the IMU on the sliding window, and then pre-integrates the measured manifold. The IMU pre-integration based on the continuous quaternion is used to derive the updated state, and the barometer data is combined as the supplementary sensor information to update the position. The dynamic differential equation can be expressed as

[0055]

[0056] Using the trapezoidal integration method, we can get

[0057]

[0058] Based on the above formula, the incremental combined pose estimation and iterative update are performed.

[0059] The improved adaptive Kalman filter algorithm is used to estimate the attitude, speed and position of the UAV. The state model and observation model are established according to the following formula:

[0060]

[0061] Where F is the state transfer matrix, defined as

[0062]

[0063] Where Q1 = -(ω rn ×), Q2 and Q3 are defined as

[0064]

[0065] Estimating Noise State Using Adaptive Kalman Filtering Algorithm The structure is as follows

[0066]

[0067] Where R k is the variance of the measurement noise, it can be constructed as

[0068]

[0069] Based on the above calculations, a local optimal estimate can be obtained.

[0070] The covariance cross-fusion algorithm is used to fuse two estimates with unknown correlation to obtain the fusion weight factor β n .set up is the covariance matrix of the two estimators, is the final estimated covariance of the joint estimator, then

[0071]

[0072] This optimal estimation problem is solved by minimizing the trace of the covariance matrix, namely

[0073]

[0074] get

[0075]

[0076] get

[0077]

[0078] This is the final fusion covariance matrix.

[0079] The Simultaneous Localization and Mapping (SLAM) algorithm uses a visual camera and LiDAR to simultaneously estimate the position and map the environment. The SLAM algorithm can generate a three-dimensional map of the environment during the flight of the drone and correct the drone's position in real time. The SLAM algorithm also combines loop detection technology to ensure that error accumulation is eliminated during long-term inspection tasks and ensure mapping accuracy. The SLAM method is combined with a path planning algorithm to achieve autonomous positioning and navigation of the drone.

[0080] The path planning algorithm integrates global planning algorithms and local planning algorithms, enabling the drone to plan the optimal path in the mission area. The path planning module designs the inspection path of the drone based on the requirements of the inspection task and terrain information, ensuring that the inspection area is covered while avoiding flight duplication and improving efficiency. In addition, based on the real-time environmental perception of laser radar and visual cameras, the system also integrates an obstacle avoidance algorithm to detect the location of obstacles in real time and calculate a safe path.

[0081] The visual detection algorithm is based on deep learning technology and is used to perform real-time analysis of visual data collected by drones to identify and mark abnormalities of inspection targets. First, the images collected by the visual camera are preprocessed, including image enhancement, denoising and brightness adjustment, to improve image quality and enhance details; based on the flight path and mission area of ​​the drone, the image data is cropped into specific areas to focus on the parts that need to be inspected; the abnormalities in the inspection target are identified through the target detection algorithm, and the detected abnormal points are classified (such as defects, cracks, rust, etc.), and marked on the image with different colors, and the confidence score of the abnormal type is generated. Based on the drone's own positioning and image annotation information, the target position is determined and the target is tracked.

[0082] Figure 3 The following is the workflow diagram of the overall system. The specific steps are as follows:

[0083] Step 1: During the inspection process, the visual camera, lidar and IMU collect data in real time and upload it to the onboard computing platform.

[0084] Step 2: Integrate vision, LiDAR, IMU and other data through a multi-sensor hierarchical fusion algorithm, and generate high-precision three-dimensional environment maps and pose estimates based on the SLAM algorithm.

[0085] Step 3: The path planning module generates the optimal inspection path based on the inspection task area and environmental structure; when encountering obstacles, the system automatically calls the obstacle avoidance algorithm to correct the path.

[0086] Step 4: The visual detection algorithm analyzes the visual data, identifies and marks abnormal conditions of the inspection target, and transmits the drone inspection data to the ground station in real time through the wireless communication module for monitoring and subsequent analysis.

[0087] In summary, the present invention provides an autonomous flight and inspection system for unmanned aerial vehicles based on multi-sensor fusion technology, including: unmanned aerial vehicles, multi-sensor data fusion, synchronous positioning and mapping, autonomous path planning obstacle avoidance and visual detection. The inspection system consists of an unmanned aerial vehicle body, a multi-sensor data hierarchical fusion algorithm, a synchronous positioning and mapping algorithm, an autonomous path planning obstacle avoidance algorithm, and a visual detection algorithm. The unmanned aerial vehicle body is made of carbon fiber material, includes a modularly designed frame, has detachable parts for easy transportation and maintenance, and is equipped with a multi-functional bracket for installing various sensor devices. The multi-sensor data hierarchical fusion algorithm is based on Kalman filtering and Bayesian reasoning models, and the data of IMU, LiDAR, visual sensors, etc. are integrated in real time to provide high-precision position, attitude and environmental information. The synchronous positioning and mapping algorithm uses a combination of vision and LiDAR to achieve real-time positioning and three-dimensional map construction in complex environments. The autonomous path planning obstacle avoidance algorithm is based on the path planning algorithm, combined with the dynamic window method and stereo geometry, to ensure the optimal path while avoiding obstacles. The visual inspection algorithm is based on deep learning technology and the Yolo model, which can detect defects, cracks, rust and other abnormal conditions in the inspection area in real time. It also has target tracking and positioning functions, and dynamically tracks the inspection target through the target's location information. The drone inspection system is also equipped with a remote data transmission module, which can transmit the inspection data in real time to the ground control center for analysis and storage.

[0088] The examples of the present invention are described in detail above in conjunction with the embodiments, but the present invention is not limited to the above examples. Various changes can be made within the knowledge scope of ordinary technicians in the field without departing from the purpose of the present invention, and should also be regarded as the protection scope of the present invention.

Claims

1. An autonomous flight and inspection system for unmanned aerial vehicles based on multi-sensor fusion technology, characterized in that: include: The UAV platform, multi-sensor layered fusion module, inspection task module, image recognition and detection module, data transmission and storage module, and ground control station, the working process includes the following steps: 1) During the inspection process, the visual camera, lidar and IMU collect data in real time and upload it to the onboard computing platform; 2) The multi-sensor hierarchical fusion module integrates vision, LiDAR and IMU data to generate high-precision 3D environment maps and pose estimates based on the SLAM algorithm; 3) The path planning algorithm in the inspection task module generates the optimal inspection path according to the inspection task area and environmental structure; when encountering obstacles, the obstacle avoidance algorithm is automatically called to correct the path; 4) The visual detection algorithm in the image recognition and detection module analyzes the visual data, identifies and marks abnormal conditions of the inspection targets, and transmits the UAV inspection data to the ground station in real time through the wireless communication module for monitoring and subsequent analysis.

2. The UAV autonomous flight and inspection system based on multi-sensor fusion technology according to claim 1 is characterized in that: The multi-sensor hierarchical fusion module is based on an improved adaptive extended Kalman filter algorithm, which integrates the depth information of the lidar, the image information of the binocular camera, the attitude data provided by the IMU and the GPS data in real time to improve the positioning accuracy and inspection stability of the UAV.

3. The UAV autonomous flight and inspection system based on multi-sensor fusion technology according to claim 2 is characterized in that: The working process of the multi-sensor hierarchical fusion module includes the following steps: 1) Collect the sensor data carried by the drone, and store the collected data in the storage space opened up by the algorithm; 2) Multi-sensor spatiotemporal calibration: Spatiotemporal calibration includes coordinate transformation and data alignment. The sensor data is converted to the same coordinate system through coordinate transformation. LiDAR is selected as the main sensor, and its frequency is used to determine the time step of the sliding window. The timestamp of the LiDAR data is selected as the time frame to synchronize the multi-sensor data. 3) Preprocessing of sensor data: The raw data obtained by IMU are acceleration and angular velocity. Pre-integration is performed based on the IMU pre-integration algorithm to obtain the rotation and translation transformation between two frames. After obtaining the IMU data, it is incrementally combined with the barometer data to obtain the relative position and posture of the drone relative to the starting state. 4) The data of IMU and barometer, and the data of visual camera and LiDAR are fused and calculated respectively, and the two groups of estimators constitute the first level of sensor fusion; the IMU and barometer data fusion algorithm is based on the improved adaptive extended Kalman filter algorithm, and the visual camera and LiDAR data fusion is based on the synchronous positioning and mapping algorithm. The two estimators obtain a group of UAV pose estimates respectively; 5) The two sets of pose estimates obtained by the two estimators of the first level are fused to provide a global optimal estimate: the covariance crossover algorithm is used to fuse and jointly optimize the estimates with unknown correlations, and the final pose estimation result is output.

4. The UAV autonomous flight and inspection system based on multi-sensor fusion technology according to claim 3 is characterized in that: The image recognition and detection module uses a deep learning algorithm to process and analyze the images collected by the binocular camera in real time, identify abnormal conditions of the inspection targets, including defects, cracks and rust, and achieve accurate detection and classification of the inspection targets by training a deep neural network model.

5. The UAV autonomous flight and inspection system based on multi-sensor fusion technology according to any one of claims 1 to 4, characterized in that: The data transmission and storage module transmits the inspection data to the ground station in real time through wireless communication equipment, and performs data backup locally on the UAV. The inspection data includes images, point cloud data and flight trajectory for subsequent analysis and troubleshooting.

Citation Information

Cited By

  • Patrol route self-adjusting method and system based on unmanned aerial vehicle inspection

    CN120276483A

  • Intelligent unmanned aerial vehicle surveying and mapping and environmental protection inspection method

    CN120595821A

  • Unmanned aerial vehicle line simulation method and system based on image recognition algorithm

    CN120631046A

  • Intelligent thermal power generation facility inspection method and system based on unmanned aerial vehicle

    CN120803044A

  • Unmanned aerial vehicle dynamic path real-time optimization method based on multi-sensor fusion

    CN120907557A