Unmanned aerial vehicle multi-mode measurement data fusion method

Abstract

The invention discloses a multi-modal measurement data fusion method for an unmanned aerial vehicle. The method comprises the following steps: synchronously acquiring multi-modal measurement data through a plurality of sensors carried on the unmanned aerial vehicle; performing preprocessing including time synchronization and space alignment on the data; based on the environmental reliability evaluation model, dynamically analyzing reliability parameters of each sensor in the current environment, and generating a dynamic weight for each sensor data; and utilizing a fusion module to carry out adaptive weighted fusion on the preprocessed data according to the dynamic weight, and generating a fusion result containing unmanned aerial vehicle global pose estimation and an environment map. The device comprises functional modules corresponding to the steps. The core of the method is that through dynamic reliability evaluation and weight generation, the weight is introduced into the Kalman filter as an observation noise covariance adjustment factor, adaptive optimization of the fusion strategy to the environment is realized, and the pose estimation precision, the system robustness and the long-term operation stability of the unmanned aerial vehicle in a complex scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) navigation and perception technology, specifically to a method for fusing multimodal measurement data from UAVs. Background Technology

[0002] With the widespread application of drones in autonomous navigation, logistics delivery, geographic mapping, and emergency rescue, extremely high demands are placed on their ability to operate stably in complex environments such as those without GPS signals, sudden changes in lighting, and dynamic obstacles. Single sensors (such as cameras, LiDAR, and IMUs) have inherent physical limitations, making it difficult to provide reliable data in all scenarios. For example, visual sensors are prone to failure in low-light or texture-deficient environments; LiDAR performance degrades in fog, rain, and smoke, and its detection capability for transparent objects such as glass is weak; while IMUs can provide high-frequency information about their own motion, their measurement errors accumulate and diverge over time (i.e., drift).

[0003] To address the aforementioned issues, multi-sensor fusion technology has become a standard solution in the UAV field. In existing technologies, the most common method is to use filtering algorithms (such as the Kalman filter family) or optimization algorithms (such as graph optimization) to fuse data from different sensors. However, most existing fusion schemes have the following limitations:

[0004] 1. Many solutions pre-determine fixed weights or confidence levels for different sensors. For example, GPS is trusted in open areas, while vision / laser is trusted indoors. This method cannot adapt to dynamic changes in the environment. When the performance of a sensor drops sharply due to sudden interference (such as instantaneous strong light or temporary obstruction), static weights can lead to large errors in the fusion results, or even cause the system to crash.

[0005] 2. While some solutions attempt dynamic adjustment, their evaluation parameters are singular or their models are simplistic, failing to comprehensively and quantitatively reflect the sensor's true reliability in the current specific environment. For example, some solutions evaluate the camera solely based on image brightness, ignoring crucial factors such as the number of feature points and motion blur.

[0006] 3. Calibration of extrinsic parameters (i.e., coordinate transformation relationships) between sensors is fundamental to data fusion. However, in practical applications, extrinsic parameters may change due to vibration, collisions, or temperature variations, leading to the failure of the fusion prerequisite. Most existing solutions rely on ground calibration before flight and lack the ability to perform online calibration and compensation during flight.

[0007] Therefore, there is an urgent need in this field for a data fusion solution that can assess sensor reliability in real time and accurately, perform adaptive fusion accordingly, and have online calibration capabilities, so as to comprehensively improve the autonomy and safety of UAVs in complex scenarios. Summary of the Invention

[0008] The present invention aims to solve the above-mentioned technical problems by providing a method for fusing multimodal measurement data from unmanned aerial vehicles (UAVs).

[0009] To solve the above-mentioned technical problems, the technical solution provided by the present invention is: a method for fusing multimodal measurement data from unmanned aerial vehicles, comprising the following steps: S1. Simultaneously acquire multimodal measurement data using multiple sensors mounted on the drone; S2. Preprocess the multimodal measurement data, the preprocessing including time synchronization and spatial alignment; S3. Based on the environmental reliability assessment model, dynamically assess the reliability of each sensor in the current environment and generate a dynamic weight for each type of sensor data; S4. Using the fusion module, adaptive weighted fusion is performed on the preprocessed multimodal measurement data according to the dynamic weights to generate a fusion result that includes UAV global pose estimation and environmental map.

[0010] Furthermore, the multiple sensors in step S1 include at least a visual sensor, a lidar sensor, an inertial measurement unit, and a global navigation satellite system receiver.

[0011] Furthermore, the environmental reliability assessment model in step S3 determines the reliability of each sensor by analyzing at least one of the following parameters: For visual sensors, the light intensity, the number of image feature points, or the image clarity are considered. For point cloud density or effective measurement distance of lidar sensors; Static deviations of the accelerometer or gyroscope in the inertial measurement unit; This refers to the signal-to-noise ratio or the number of satellites for a global navigation satellite system receiver.

[0012] Furthermore, the fusion module in step S4 is implemented using an extended Kalman filter or an unscented Kalman filter, and the dynamic weights are input into the filter as adjustment factors of the observation noise covariance matrix.

[0013] Furthermore, step S5 is included: using the fused global pose estimation results to calibrate the external parameters of the visual sensor and / or lidar sensor relative to the UAV body online.

[0014] Furthermore, the fused global pose estimation and environmental map are used for the UAV's autonomous navigation, obstacle avoidance, or 3D reconstruction tasks.

[0015] Furthermore, the spatial alignment in step S2 specifically involves: based on pre-calibrated extrinsic parameters, converting data from different sensor coordinate systems to a unified body coordinate system or world coordinate system.

[0016] A UAV multimodal measurement data fusion device, comprising: The data acquisition module is used to simultaneously acquire multimodal measurement data through multiple sensors mounted on the drone; A preprocessing module, connected to the data acquisition module, is used to perform time synchronization and spatial alignment of the multimodal measurement data; The reliability assessment module is used to dynamically assess the reliability of each sensor in the current environment based on the environmental reliability assessment model, and to generate a dynamic weight for each type of sensor data. An adaptive fusion module, connected to the preprocessing module and the reliability assessment module, is used to adaptively weight and fuse the preprocessed multimodal measurement data according to the dynamic weights to generate a fusion result that includes UAV global pose estimation and environmental map.

[0017] Furthermore, the data acquisition module includes at least a visual sensor, a lidar sensor, an inertial measurement unit, and a global navigation satellite system receiver. The advantages of this invention compared to the prior art are: By constructing an environmental reliability assessment model, the reliability of each sensor under the current environment is quantified in real time and from multiple dimensions, and dynamic weights are generated. This enables the fusion system to automatically reduce the contribution of failed sensors in sudden severe environments, significantly improving the system's robustness and survivability in complex scenarios such as sudden changes in illumination and signal obstruction.

[0018] By introducing dynamic weights into the observation noise covariance matrix of the Kalman filter, the uncertainty of different sensor observations is described more accurately from a probabilistic model. This makes the state estimation more inclined to trust the sensors with high reliability, thereby obtaining a more accurate global pose and environment map.

[0019] The added online extrinsic parameter calibration function can automatically correct extrinsic parameter deviations caused by mechanical vibration, ensuring the long-term stability of the multi-sensor fusion foundation, reducing maintenance frequency, and improving the long-term operational reliability of the UAV. These advantages work together to provide a higher-quality data foundation for advanced tasks such as autonomous navigation, obstacle avoidance, and 3D reconstruction of UAVs. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of a UAV multimodal measurement data fusion method according to the present invention. Detailed Implementation

[0022] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0023] This embodiment provides a method and apparatus for fusing multimodal measurement data from an unmanned aerial vehicle (UAV). (Reference) Figure 1 The method in this embodiment specifically includes the following steps:

[0024] Step S1: Multimodal data synchronization acquisition. The UAV is equipped with visual sensors (such as a global shutter camera), lidar (such as a 16-line or 32-line LiDAR), an inertial measurement unit (IMU), and a GNSS receiver. These sensors achieve microsecond-level time synchronization through hardware triggering or software timestamps, ensuring that different data streams are aligned in time.

[0025] Step S2: Data Preprocessing. Preprocessing includes time synchronization and spatial alignment. Time synchronization was completed in S1. Spatial alignment utilizes the extrinsic parameter matrices (such as the transformation matrix Timu cam from camera to IMU, or the transformation matrix Timu lidar from LiDAR to IMU) calibrated on the ground before the UAV leaves the factory or before flight, to uniformly transform the observation data from all sensors to the IMU coordinate system or a fixed world coordinate system. For example, transforming a visual feature point P... cam Transformation from camera coordinate system to body coordinate system: P body =Tbody cam·P cam .

[0026] Step S3: Dynamic Reliability Assessment and Weight Generation. This step is the core innovation of this invention. Its basic principle is that the reliability of a sensor is strongly correlated with the quality of its directly observed quantities, and these quantities can be quantified through a series of calculable indicators.

[0027] Visual sensor reliability R cam Calculate the average brightness L of the current image. If L is lower than the threshold T... low or higher than the threshold T high If this happens, reliability decreases. Simultaneously, feature extraction algorithms (such as FAST, ORB) are used to calculate the number N of feature points in the image. feat The more data points, the higher the reliability. Furthermore, image sharpness C can be evaluated by calculating the image's gradient information or Laplacian variance, avoiding errors caused by motion blur. Ultimately, R... cam It can be L, N feat The result after normalization of the weighted sum or product form of C.

[0028] LiDAR reliability R lidar The point cloud density D is calculated as the number of points within a unit solid angle. A density that is too low indicates that the detected object is too far away or that an absorbing medium is present. Simultaneously, the distance distribution of the effective echoes is statistically analyzed. If most of the point cloud is concentrated near the maximum range, the reliability R is considered high. lidar reduce.

[0029] IMU reliability imu While the IMU remains a critical sensor during dynamic motion, the stability of its bias is an important indicator. When the drone is briefly stationary, its current static bias (B) can be estimated using the variance of the accelerometer and gyroscope outputs. A larger bias or more drastic changes indicate that the IMU is more susceptible to temperature or vibration, affecting its long-term integral reliability (R). imu Adjust accordingly.

[0030] GNSS reliability R gnss : Read the signal carrier-to-noise ratio (CN0) and the number of visible satellites N directly from the GNSS receiver sat The lower the average value of CN0, the more N sat The less, the more R gnss The lower.

[0031] Finally, the reliability score R of all sensors was calculated. i Normalization is performed to generate the corresponding dynamic weights w. i =R i / ∑R j .

[0032] Step S4: Adaptive Weighted Fusion. This embodiment uses an Extended Kalman Filter (EKF) as the fusion module. Its core principle is that the EKF estimates the system state (e.g., position, attitude, velocity) through two steps: prediction and update. In the update step, it depends on the observation model and the observation noise covariance matrix R. = To fuse observational data. The greater the observational noise, the lower the filter's confidence in that observational data.

[0033] The dynamic weight w generated in step S3 of this invention i This serves as an adjustment factor for the observation noise covariance matrix R. Specifically, for a given sensor i, its original observation noise covariance matrix is ​​R. i0 (Given a pre-defined constant matrix), the actual observed noise covariance used is R. i =R i0 / w i This means:

[0034] When the sensor has high reliability (wi When R is large, i As the value decreases, EKF will "trust" the sensor's observations more when updating the state, giving them greater correction weight.

[0035] When sensor reliability is low (w i (hours) R i If the value increases, EKF will consider the observation to be highly uncertain, thus almost ignoring its impact and avoiding erroneous data from contaminating the fusion results.

[0036] In this way, the fusion process achieves fully adaptive weighting, and its theoretical basis is firmly established within the probabilistic framework of Kalman filtering.

[0037] Step S5: Online Extrinsic Parameter Calibration. After a long flight, the fused pose estimation sequence has formed a high-precision trajectory. The principle is: using this high-precision trajectory as a reference "true value," the optimal coordinate transformation relationship (i.e., extrinsic parameters) between the observation data of each sensor can be deduced. For example, by using the hand-eye calibration method (Ax=XB), the motion of feature points observed by the camera is correlated with the fused body motion to solve for the latest camera extrinsic parameter Tbodycam. This process can be triggered periodically or when a sudden increase in pose estimation residuals is detected, thereby correcting extrinsic parameter errors in real time and ensuring the accuracy of spatial alignment in S2.

[0038] The apparatus for implementing the above method includes: a data acquisition module, a preprocessing module, a reliability assessment module, and an adaptive fusion module. These modules can be implemented in hardware (such as a dedicated processing chip integrated into the UAV flight control system) or in software (such as a program running on an onboard computer).

[0039] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for unmanned aerial vehicle multi-modal measurement data fusion, characterized in that: Includes the following steps: S1. Simultaneously acquire multimodal measurement data using multiple sensors mounted on the drone; S2. Preprocess the multimodal measurement data, the preprocessing including time synchronization and spatial alignment; S3. Based on the environmental reliability assessment model, dynamically assess the reliability of each sensor in the current environment and generate a dynamic weight for each type of sensor data; S4. Using the fusion module, adaptive weighted fusion is performed on the preprocessed multimodal measurement data according to the dynamic weights to generate a fusion result that includes UAV global pose estimation and environmental map. 2.The unmanned aerial vehicle multi-modal measurement data fusion method of claim 1, wherein: The multiple sensors in step S1 include at least a visual sensor, a lidar sensor, an inertial measurement unit, and a global navigation satellite system receiver. 3.The unmanned aerial vehicle multi-modal measurement data fusion method of claim 1, wherein: The environmental reliability assessment model in step S3 determines the reliability of each sensor by analyzing at least one of the following parameters: For visual sensors, the light intensity, the number of image feature points, or the image clarity are considered. For point cloud density or effective measurement distance of lidar sensors; Static deviations of the accelerometer or gyroscope in the inertial measurement unit; This refers to the signal-to-noise ratio or the number of satellites for a global navigation satellite system receiver.

4. The unmanned aerial vehicle multi-modal measurement data fusion method of claim 1, wherein: The fusion module in step S4 is implemented using an extended Kalman filter or an unscented Kalman filter, and the dynamic weights are input into the filter as adjustment factors for the observation noise covariance matrix.

5. The unmanned aerial vehicle multi-modal measurement data fusion method of claim 1, wherein: It also includes step S5: using the fused global pose estimation results to calibrate the external parameters of the visual sensor and / or lidar sensor relative to the UAV body online.

6. The unmanned aerial vehicle multi-modal measurement data fusion method of claim 1, wherein, The fused global pose estimation and environmental map are used for the UAV's autonomous navigation, obstacle avoidance, or 3D reconstruction tasks.

7. The unmanned aerial vehicle multi-modal measurement data fusion method of claim 1, wherein, The spatial alignment in step S2 specifically involves: based on pre-calibrated extrinsic parameters, converting data from different sensor coordinate systems to a unified body coordinate system or world coordinate system.

8. An unmanned aerial vehicle multi-modal measurement data fusion device, characterized in that, include: The data acquisition module is used to simultaneously acquire multimodal measurement data through multiple sensors mounted on the drone; A preprocessing module, connected to the data acquisition module, is used to perform time synchronization and spatial alignment of the multimodal measurement data; The reliability assessment module is used to dynamically assess the reliability of each sensor in the current environment based on the environmental reliability assessment model, and to generate a dynamic weight for each type of sensor data. An adaptive fusion module, connected to the preprocessing module and the reliability assessment module, is used to adaptively weight and fuse the preprocessed multimodal measurement data according to the dynamic weights to generate a fusion result that includes UAV global pose estimation and environmental map.

9. The unmanned aerial vehicle multi-modal measurement data fusion device according to claim 8, characterized in that: The data acquisition module includes at least one visual sensor, one lidar sensor, one inertial measurement unit, and one global navigation satellite system receiver.

Images