A UAV power inspection multimodal data fusion system and UAV

Abstract

The present invention discloses a multimodal data fusion system for UAV power inspection and a UAV, which relates to the field of UAV control technology. The system effectively improves the accuracy and robustness of multimodal sensor data fusion through dynamic weight calculation and adaptive adjustment. In the prior art, sensor data fusion is mostly static mode, and once the weight is set, it cannot be adjusted according to environmental changes, which makes the system face the problem of reduced accuracy of fusion data under complex or unforeseen environmental conditions. By introducing dynamic weight calculation based on a large model, the weight can change with the real-time quality of sensor data, ensuring the optimal fusion effect of sensor data in different environments, thereby significantly improving the flight accuracy and stability of the UAV.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a UAV power inspection multimodal data fusion system and a UAV. Background Art

[0002] With the development of unmanned aerial vehicle (UAV) technology, drones have been gradually applied to power line inspections, carrying out inspections of power lines and equipment in high-risk areas. Traditional power line inspections rely heavily on manual labor, but drone inspections not only significantly improve inspection efficiency but also replace manual labor in hazardous environments, significantly reducing labor costs and safety risks. In recent years, advances in sensor technology have enabled drones to be equipped with a variety of sensors (such as high-definition cameras, infrared thermal imagers, and lidar), enabling more comprehensive inspections of power equipment.

[0003] In existing technologies, multimodal data fusion is used in drone fault detection and identification tasks. By fusing visible light images, infrared imaging, and lidar data, it can overcome the shortcomings of a single data source and significantly improve the comprehensiveness and accuracy of detection. However, the current static weighting model, which combines data from different sensors using a fixed weight distribution method, performs well in certain specific environments, but cannot adapt to dynamic environmental changes, resulting in limited detection results in complex environments.

[0004] For example, the data collection performance of various sensors can vary significantly during the transition between day and night, inclement weather, or during drastic temperature fluctuations. Visible light images are more effective during the day, but the effectiveness of this data source decreases significantly at night. Infrared imaging data can be subject to noise interference at higher temperatures. Furthermore, environmental factors such as wind speed and humidity can also affect the accuracy of lidar. To address these issues, some adaptive weighting methods have emerged in existing technologies, using techniques such as deep learning to dynamically adjust data weights to adapt to environmental changes. However, these methods have yet to be systematically applied to drone-based power inspections.

[0005] The prior art provides a method for measuring the altitude of unmanned aerial vehicle (UAV) power inspections based on multi-sensor fusion. This method uses a binocular camera, ultrasonic sensor, inertial measurement unit (IMU) module, and barometer to measure the UAV's flight altitude, addressing the issue of inaccurate altitude measurement in complex terrain. The existing solution uses sensors installed on the UAV, including a binocular camera, ultrasonic sensor, IMU module, and barometer. The system first uses the initial values ​​of the IMU and barometer as the benchmark for altitude measurement. During the UAV's takeoff and landing process, the IMU module and barometer capture changes in values. By collecting this data in real time, the system can make a preliminary assessment of the UAV's altitude information.

[0006] During drone flight, altitude measurement relies not only on the IMU and barometer, but also on data from the binocular camera and ultrasonic sensors for supplementary correction. The binocular camera obtains altitude information by calculating image parallax, while the ultrasonic sensor estimates altitude based on the propagation time of sound waves. This data serves as auxiliary information to correct the initial altitude measurement results from the IMU and barometer, achieving multi-sensor information fusion for altitude measurement and improving overall measurement accuracy. During the multi-sensor data fusion process, the system adopts a static fusion strategy, assigning fixed weights to different sensors and fusing their data in a certain proportion to obtain the final altitude measurement.

[0007] Furthermore, to address the impact of errors caused by complex environments, the system also includes real-time state updates and error calculation steps. Through state updates, the system adjusts altitude measurements based on changes in sensor values, calculates altitude error gains, and continuously updates state covariance, effectively reducing errors caused by environmental disturbances or sensor drift.

[0008] The multi-sensor fusion system's workflow proceeds smoothly, from initial baseline establishment to sensor data collection, fusion, and correction, ultimately outputting altitude information. Within the system's hardware architecture, the processor executes programs to process data from each sensor, while the memory stores computer programs for integrating and processing sensor data. This integrated hardware and software architecture enables the system to process various sensor data in real time throughout the drone's flight, ensuring accurate altitude measurement.

[0009] As can be seen, existing technical solutions use a static weight fusion method for altitude measurement, that is, the fusion weights of different sensors are fixed and do not dynamically adjust with changes in the external environment. This static fusion method is effective in some cases, but it may lead to reduced measurement accuracy in complex environments such as mountainous terrain, areas with low surface reflectivity, or conditions with large airflow disturbances. Because in these situations, the measurement reliability of individual sensors can vary significantly, but static fusion fails to fully account for this and cannot dynamically and adaptively adjust the weights of each sensor.

[0010] The main drawback of existing technologies is that their static weight fusion strategies are insufficiently adaptable to complex and changing environments. Existing solutions use fixed weights to fuse data from binocular cameras, ultrasonic sensors, IMU modules, and barometers. This, to a certain extent, ignores the varying reliability of different sensors under varying environmental conditions. Specifically, this fixed weighting can lead to a decrease in altitude measurement accuracy under certain conditions, as detailed below.

[0011] First, in complex mountainous terrain, the surface reflectivity can be low, significantly impacting the measurement accuracy of binocular cameras and ultrasonic sensors. Binocular cameras rely on image parallax for distance measurement. When the surface reflectivity is low or there are obstructions, the accuracy of the image parallax calculations is affected, leading to errors in height measurements. Ultrasonic sensors are susceptible to surface irregularities and environmental noise, resulting in inaccurate measurements. However, existing technologies do not specifically address the particularities of these environments, and static fusion weights are unable to mitigate the impact of unreliable sensor measurements on the final height measurement.

[0012] Secondly, the barometer altitude measurement used in existing solutions is very sensitive to airflow disturbances, especially in mountainous environments, where frequent airflow changes can cause significant fluctuations in barometer measurements. However, because the barometer weight in existing solutions is fixed, the system cannot effectively reduce the influence of barometer instability on the final fusion result, which can lead to large deviations in altitude measurements.

[0013] Furthermore, due to the presence of cumulative errors, the IMU module's measurement accuracy gradually decreases over time. This is especially true during long-duration UAV flights, where IMU drift can lead to cumulative measurement errors. However, existing technologies do not dynamically adjust the IMU weights to address this characteristic, instead maintaining fixed weights. This results in the IMU's influence on the fusion results gradually increasing over long-duration flights, thereby affecting the final altitude measurement accuracy.

[0014] Therefore, the static fusion strategy employed in existing solutions makes the entire system inflexible to environmental changes and sensor performance fluctuations. This can lead to a significant decrease in measurement accuracy in complex environments, such as mountainous areas, low-reflectivity surfaces, and areas with significant airflow disturbances. Compared to the dynamic weighted adaptive fusion method of the present invention, existing technologies lack the ability to dynamically adjust sensor weights based on real-time environmental changes, failing to fully leverage the advantages of multi-sensor fusion. Consequently, measurement accuracy is low and robustness is insufficient in changing environments. Summary of the Invention

[0015] In view of the above problems, the present invention provides a UAV power inspection multimodal data fusion system and a UAV for overcoming the above problems or at least partially solving the above problems.

[0016] The present invention provides the following solutions:

[0017] A multimodal data fusion system for UAV power inspection, comprising:

[0018] A multimodal sensor unit, which is provided on the main body of the power inspection drone and is used to obtain the drone's altitude data and environmental parameters in real time;

[0019] A computing and processing unit, the computing and processing unit being provided on the main body of the power inspection drone, the multimodal sensor unit being communicatively connected to the computing and processing unit;

[0020] The computing processing unit is configured to perform the following operations:

[0021] Acquiring sensor data corresponding to each sensor collected by the multimodal sensor unit;

[0022] Adopting an adaptive deep learning model to extract features from each of the sensor data using a deep neural network, calculating the confidence of each sensor in the current environment, and dynamically adjusting the weight of each of the sensor data in the fusion process based on the environmental data and sensor feedback information, so as to obtain a target weight corresponding to each of the sensor data;

[0023] fusing the sensor data using the target weights corresponding to the sensors to obtain target height information and target attitude information;

[0024] The target height information and the target attitude information are sent to the UAV flight controller so that the UAV flight controller can adjust the flight path and height according to the target height information and the target attitude information.

[0025] Preferably, the target weight is calculated by the following formula:

[0026] w i (t) = exp(a i (t)) / ∑exp(a i (t))

[0027] Where: w i (t) represents the dynamic weight of the i-th sensor at time t, a i (t) represents the score calculated by the deep learning model, which represents the relative importance of the i-th sensor in the current environment at time t.

[0028] Preferably: i (t) is calculated by the following formula:

[0029] a i (t) = f(H sensor (t), E env (t), w i (t-1))

[0030] Where: H sensor (t) represents the data of the i-th sensor at time t, E env (t) represents the current environment state, w i (t-1) represents the weight at the previous moment.

[0031] Preferably: the multimodal sensor unit includes at least a binocular camera, an ultrasonic sensor, an inertial measurement unit and a barometer;

[0032] The binocular camera is used to obtain ground visual information to calculate the height and posture of the drone relative to the ground;

[0033] The ultrasonic sensor is used to detect the distance between the drone and the ground;

[0034] The inertial measurement unit is used to detect the attitude changes of the drone in real time;

[0035] The barometer is used to obtain the relative altitude information of the drone.

[0036] Preferably, dynamically adjusting the weight of each sensor data in the fusion process according to the environmental data and the feedback information of the sensor includes:

[0037] When the drone takes off, the barometer is used to roughly estimate the initial altitude, the inertial measurement unit is used to compensate for dynamic changes in the barometer, and the binocular camera and the ultrasonic sensor provide accurate altitude measurement when approaching the ground;

[0038] During flight, the adaptive deep learning model dynamically adjusts the weights of each sensor based on changes in terrain and environment. When the drone flies over water or highly reflective ground, the weights of the barometer and the inertial measurement unit increase, while the weights of the ultrasonic sensor and the binocular camera decrease.

[0039] During landing, the ultrasonic sensor and the binocular camera are used to accurately locate the ground, with the inertial measurement unit and the barometer serving as auxiliary.

[0040] Preferably, in a mountainous environment, the adaptive deep learning model is used to reduce the weight of the ultrasonic sensor and increase the weight of the inertial measurement unit and the barometer;

[0041] When the airflow disturbance is large, the adaptive deep learning model is used to identify the airflow disturbance according to the data changes of the inertial measurement unit, and the weight of the barometer in the fusion is reduced, while the weights of the ultrasound and the binocular camera are increased.

[0042] Preferably, the sensor data are fused using the following formula:

[0043] H fused (t)=w1(t)H camera (t)+w2(t)H ultrasonic (t)+w3(t)H IMU (t)

[0044] +w4(t)H barometer (t)

[0045] Where: H fused (t) represents the final result after data fusion at time t, H camera 、H ultrasonic 、H IMU 、H barometer They are the height information of binocular camera, ultrasonic sensor, IMU and barometer respectively, and w1, w2, w3 and w4 are the dynamic weights of each sensor.

[0046] Preferably, the computing and processing unit includes an embedded processor.

[0047] A UAV comprises a housing, a bracket, a propeller, a motor, a power system, a battery module, and the above-mentioned UAV power inspection multimodal data fusion system.

[0048] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0049] The embodiment of the present application provides a multimodal data fusion system and a drone for UAV power inspection. The system effectively improves the accuracy and robustness of multimodal sensor data fusion through dynamic weight calculation and adaptive adjustment. In the prior art, sensor data fusion is mostly static mode. Once the weight is set, it cannot be adjusted according to environmental changes. This makes the system face the problem of reduced accuracy of fusion data under complex or unforeseen environmental conditions. By introducing dynamic weight calculation based on a large model, the weight can change with the real-time quality of the sensor data, ensuring the optimal fusion effect of sensor data in different environments, thereby significantly improving the flight accuracy and stability of the drone.

[0050] The adaptive data fusion mechanism has a high degree of anti-interference capability. During flight, some sensors may be affected by environmental interference or exhibit abnormal signals. For example, barometer data is prone to fluctuations or noise in strong winds, while laser sensor measurement accuracy may decrease in low-reflectivity terrain. By using a large model to monitor and dynamically evaluate sensor status in real time, anomalous data sources can be effectively detected, downgraded, or temporarily excluded from participation. This reduces the impact of a single sensor anomaly on the overall system and enhances the system's stability and fault tolerance in complex environments.

[0051] Furthermore, in the preferred embodiment, multimodal fusion enables the drone to maintain consistent flight accuracy across diverse terrains and altitudes. The coordinated integration of sensor data from binocular cameras, ultrasonic sensors, IMUs, and barometers enables the drone to obtain precise altitude and environmental information at varying altitudes and in diverse terrains. This flexible multimodal fusion strategy overcomes the limitations of a single sensor under specific conditions, enabling the system to more reliably perform inspections in complex terrains such as mountainous areas and forests.

[0052] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0054] Figure 1 This is a structural diagram of a multimodal data fusion system for UAV power inspection provided by an embodiment of the present invention;

[0055] Figure 2 This is a workflow diagram of a multimodal data fusion system for UAV power inspection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0057] See also Figure 1 , is a multimodal data fusion system for UAV power inspection provided by an embodiment of the present invention, such as Figure 1 As shown, the system may include:

[0058] A multimodal sensor unit, which is provided on the main body of the power inspection drone and is used to obtain the drone's altitude data and environmental parameters in real time. In a specific implementation, the embodiment of the present application may provide that the multimodal sensor unit includes at least a binocular camera, an ultrasonic sensor, an inertial measurement unit, and a barometer;

[0059] The binocular camera is used to obtain ground visual information to calculate the height and posture of the drone relative to the ground;

[0060] The ultrasonic sensor is used to detect the distance between the drone and the ground;

[0061] The inertial measurement unit is used to detect the attitude changes of the drone in real time;

[0062] The barometer is used to obtain the relative altitude information of the drone.

[0063] A computing and processing unit is provided on the main body of the power inspection drone, and the multimodal sensor unit is communicatively connected to the computing and processing unit; in specific implementation, the embodiment of the present application can provide that the computing and processing unit includes an embedded processor.

[0064] The computing processing unit is configured to perform the following operations:

[0065] Acquiring sensor data corresponding to each sensor collected by the multimodal sensor unit;

[0066] An adaptive deep learning model is used to extract features from each of the sensor data using a deep neural network, calculate the confidence of each sensor in the current environment, and dynamically adjust the weight of each of the sensor data in the fusion process based on the environmental data and the feedback information of the sensor, so as to obtain the target weight corresponding to each of the sensor data; in specific implementation, the embodiment of the present application can provide that the target weight is obtained by calculating the following formula:

[0067] w i (t) = exp(a i (t)) / ∑exp(a i (t))

[0068] Where: w i (t) represents the dynamic weight of the i-th sensor at time t, a i (t) represents the score calculated by the deep learning model, which represents the relative importance of the i-th sensor in the current environment at time t.

[0069] Furthermore, the a i (t) is calculated by the following formula:

[0070] a i (t) = f(H sensor (t), E env (t), w i (t-1))

[0071] Where: H sensor (t) represents the data of the i-th sensor at time t, Eenv (t) represents the current environment state, w i (t-1) represents the weight at the previous moment.

[0072] When the drone takes off, the barometer is used to roughly estimate the initial altitude, the inertial measurement unit is used to compensate for dynamic changes in the barometer, and the binocular camera and the ultrasonic sensor provide accurate altitude measurement when approaching the ground;

[0073] During flight, the adaptive deep learning model dynamically adjusts the weights of each sensor based on changes in terrain and environment. When the drone flies over water or highly reflective ground, the weights of the barometer and the inertial measurement unit increase, while the weights of the ultrasonic sensor and the binocular camera decrease.

[0074] During landing, the ultrasonic sensor and the binocular camera are used to accurately locate the ground, with the inertial measurement unit and the barometer serving as auxiliary.

[0075] In a mountainous environment, the adaptive deep learning model is used to reduce the weight of the ultrasonic sensor and increase the weight of the inertial measurement unit and the barometer;

[0076] When the airflow disturbance is large, the adaptive deep learning model is used to identify the airflow disturbance according to the data changes of the inertial measurement unit, and the weight of the barometer in the fusion is reduced, while the weights of the ultrasound and the binocular camera are increased.

[0077] The target weights corresponding to the sensors are used to fuse the sensor data to obtain target height information and target attitude information. In a specific implementation, the embodiment of the present application can provide the following formula to fuse the sensor data:

[0078] H fused (t)=w1(t)H camera (t)+w2(t)H ultrasonic (t)+w3(t)H IMU (t)

[0079] +w4(t)H barometer (t)

[0080] Where: H fused (t) represents the final result after data fusion at time t, H camera 、H ultrasonic 、H IMU 、H barometer They are the height information of binocular camera, ultrasonic sensor, IMU and barometer respectively, and w1, w2, w3 and w4 are the dynamic weights of each sensor.

[0081] The target height information and the target attitude information are sent to the UAV flight controller so that the UAV flight controller can adjust the flight path and height according to the target height information and the target attitude information.

[0082] The multimodal data fusion system for UAV power inspections provided in the embodiments of this application addresses the problems of low multimodal data fusion accuracy, poor environmental adaptability, and insufficient data reliability in existing UAV power inspection technologies. In existing UAV power inspections, static fusion methods are often used to integrate multi-sensor data. However, because static weights cannot be adaptively adjusted based on environmental changes, sensor measurement errors increase in complex inspection environments, especially in mountainous areas with variable terrain and frequent airflow disturbances, making it difficult to ensure measurement accuracy and stability during the inspection process.

[0083] First, existing technologies often use static weights in multi-sensor data fusion, which means that the weight ratios between different sensors are pre-set, and it is impossible to dynamically adjust the importance of each sensor according to the actual scenario. For example, when the flight altitude changes or the ambient light changes, the measurement accuracy of sensors such as binocular cameras, ultrasonic sensors, and barometers will vary, and fixed static weights cannot effectively cope with these changes, thereby affecting the accuracy of altitude information measurement. The system provided in this application introduces a dynamic weight adaptive fusion strategy, which aims to dynamically adjust the weights of each sensor according to the real-time environment and sensor characteristics to improve the accuracy of data fusion.

[0084] Secondly, the sensor fusion methods of the prior art have obvious adaptability problems when facing complex terrain (such as mountains, slopes, etc.) and adverse environmental conditions (such as airflow disturbances and poor surface reflectivity). This lack of adaptability makes it difficult for drones to obtain stable altitude data during flight, affecting the smooth completion of inspection tasks. The system provided in this application solves this adaptability problem by using a large model to extract features and optimize the fusion strategy of multimodal sensor data, so that the fusion results can be adaptively adjusted to ensure that stable and reliable altitude measurements can still be provided in different environments. In addition, the prior art lacks the ability to effectively detect and process sensor data anomalies. Especially under complex airflow conditions in mountainous areas, noise or outliers often appear in sensor data. Traditional static fusion methods find it difficult to identify and correct these data in a timely manner, thereby reducing the accuracy and reliability of the overall measurement. The system provided in this application uses a large model to perform intelligent anomaly detection and correction on sensor data, ensuring that outliers can be effectively removed during the fusion process, thereby further improving the credibility of the fused data.

[0085] As can be seen, the main purpose of the system provided by this application is to overcome the limitations of static fusion in existing technologies and solve the problems of low multimodal sensor fusion accuracy, poor environmental adaptability, and insufficient data anomaly processing capabilities. By introducing a dynamic weighted adaptive fusion method, the present invention can significantly improve the accuracy and stability of UAV power inspections in complex environments, providing more accurate and reliable technical support for power system inspections.

[0086] Dynamic weight adaptive fusion method driven by adaptive deep learning model based on deep learning: The model itself performs real-time analysis on the data of each sensor (binocular camera, ultrasound, IMU, barometer), dynamically calculates and allocates sensor weights to cope with fluctuations in sensor signal quality caused by environmental changes, and ensures the reliability and accuracy of data fusion results.

[0087] Adaptive multimodal data fusion algorithm: Adopts adaptive fusion algorithms such as weighted averaging and Bayesian estimation to effectively integrate multi-sensor data, achieving highly accurate measurement in complex terrain or special environments. It is particularly suitable for precise altitude control of UAVs in low-altitude flight.

[0088] Closed-loop feedback control and optimization mechanism: The system is designed with a closed-loop feedback structure, which uses the accuracy of multimodal data fusion results to monitor and provide feedback on the execution of flight control commands in real time, ensuring control accuracy. At the same time, it has the ability of continuous optimization and self-learning, thereby improving flight stability and adaptability.

[0089] Environmental status assessment and sensor anomaly handling strategy: Use deep learning models to evaluate the status of sensors in different environments, detect data noise and signal interference in real time, automatically exclude abnormal data or reduce the contribution of unreliable data, and ensure the overall stability and high credibility of fused data.

[0090] Multi-level collaborative optimization data fusion mechanism: Combining the rough estimation of the barometer and IMU with the fine measurement of the binocular camera and ultrasound, a multi-level, multi-precision sensor data fusion system is constructed to optimize the advantages of different types of sensors and achieve dynamic fusion from coarse to precise, thereby enhancing the system's multi-scenario adaptability.

[0091] High-precision data fusion technology for UAV power inspections: Multimodal data fusion technology, particularly suitable for power inspection tasks, improves inspection accuracy by enhancing the data adaptability of UAVs in low-altitude and complex terrain conditions, ensuring that UAVs have higher environmental adaptability and mission reliability in power inspection environments.

[0092] The system provided by this application is introduced in detail below.

[0093] The system is based on a control process for multi-sensor data acquisition and fusion, including real-time data acquisition, dynamic weight calculation, fusion processing, and anomaly detection, to ensure the accuracy and reliability of UAV power inspections. The adaptive deep learning model provided in the embodiment of the present application collects sensor data in real time and performs feature extraction and fusion strategy optimization on multimodal sensor data. First, the system uses a deep learning algorithm to dynamically adjust the weight of each sensor based on the environmental feedback information of each sensor. The model extracts features from the input data, identifies the impact of environmental changes on the accuracy of each sensor, and thus optimizes the data fusion process.

[0094] The model runs through the following steps:

[0095] 1. Data Acquisition and Preprocessing: Each sensor collects data in real time, and a preprocessing module removes noise and outliers. A combination of sensors, including binocular cameras, ultrasonic sensors, inertial sensors (IMU modules), and barometers, is used to obtain real-time drone altitude data and other environmental parameters.

[0096] 2. Feature Extraction and Weight Calculation: The adaptive deep learning model uses a deep neural network to extract features from each sensor data, calculate the confidence level of each sensor in the current environment, and dynamically assign weights. This dynamic weighting method adjusts the fusion weights of each sensor in real time based on the confidence level of the sensor data and environmental changes, improving data fusion accuracy and system adaptability.

[0097] 3. Data fusion: Based on the calculated weights, the data from each sensor is fused to obtain the final key information such as height and attitude.

[0098] 4. Control and feedback: The fused data is used in the drone’s control system to achieve precise adjustments to the flight path and altitude.

[0099] The model's primary purpose is to improve the accuracy and stability of drone inspections in complex environments. By dynamically adjusting the fusion weights of each sensor, the adaptive deep learning model optimizes data fusion results under varying environmental conditions, ensuring the system can provide stable and reliable measurement and control information in a variety of complex situations.

[0100] The present application provides a multimodal data dynamic weight adaptive fusion system for drone power inspections. This system utilizes multiple sensors through fusion technology to improve the accuracy of drone altitude and posture measurements, and uses an adaptive deep learning model to dynamically calculate the weight of sensor data to cope with the complexity of various inspection environments. Specific implementation methods are as follows:

[0101] Drone body: The drone body provided in the embodiments of the present application includes a housing, a bracket, propellers, a motor, a power system, a battery module, etc. These components are interconnected to form a stable platform for carrying various sensors and computing units.

[0102] The multimodal sensor unit includes:

[0103] Binocular camera: Installed on the front of the drone, used to obtain ground visual information to calculate the drone's altitude and attitude relative to the ground.

[0104] Ultrasonic sensor: Installed on the bottom of the drone, it is mainly used to detect the distance between the drone and the ground and is suitable for accurate distance measurement at medium and low altitudes.

[0105] Inertial Measurement Unit (IMU): Consists of a three-axis accelerometer and a three-axis gyroscope, used to detect the drone's attitude changes in real time, including information such as acceleration and angular velocity.

[0106] Barometer: Used to obtain the relative altitude information of the drone, especially suitable for rough altitude estimation over a large altitude range.

[0107] Computing Processing Unit: This unit uses an embedded processor (such as an ARM Cortex series chip) to collect data from each sensor and perform real-time fusion processing. This processing unit integrates an adaptive deep learning model algorithm to dynamically calculate the weight of each sensor.

[0108] like Figure 2 As shown, the system functions and processes are described as follows:

[0109] 1. Data collection and transmission.

[0110] Each sensor collects data at a fixed frequency (50 Hz) and transmits the data to the computing processing unit through a communication interface (I2C or SPI).

[0111] The goal of data collection is to obtain sufficiently detailed height and posture information to provide a basis for subsequent fusion and weight calculation.

[0112] 2. Dynamic weight calculation module.

[0113] Initial weight setting: When the system is initialized, the initial weight is set according to the working principle of each sensor and historical environmental data. The sum of the weight values ​​of all sensors is 1.

[0114] Dynamic Weight Adjustment: During flight, the adaptive deep learning model algorithm in the embedded processor adjusts the weights of each sensor in real time based on real-time environmental data and sensor feedback. This adjustment is based on factors such as ground reflectivity, ambient airflow disturbances, and terrain fluctuations, all of which can affect sensor accuracy and reliability.

[0115] The dynamic weight calculation formula is based on the feature extraction results of the deep learning model. It can identify the impact of environmental changes on the accuracy of each sensor in real time and optimize the weight adjustment according to environmental conditions to obtain the target weight. The target weight is calculated by the following formula:

[0116] w i (t) = exp(a i (t)) / ∑exp(a i (t))

[0117] Where: w i (t) represents the dynamic weight of the i-th sensor at time t, a i (t) represents the score calculated by the deep learning model, which represents the relative importance of the i-th sensor in the current environment at time t.

[0118] The a i (t) is calculated by the following formula:

[0119] a i (t) = f(H sensor (t), E env (t), w i (t-1))

[0120] Where: H sensor (t) represents the data of the i-th sensor at time t, E env (t) represents the current environment state, w i (t-1) represents the weight at the previous moment.

[0121] 3. Data fusion model.

[0122] Multimodal data fusion: The computing unit receives real-time data from the binocular camera, ultrasonic sensor, IMU, and barometer, and uses dynamic weights to perform a weighted average of the measurements of each sensor to obtain the optimal height and attitude estimation. The data from each sensor is fused using the following formula:

[0123] H fused (t)=w1(t)H camera (t)+w2(t)H ultrasonic (t)+w3(t)H IMU (t)

[0124] +w4(t)H barometer (t)

[0125] Where: H fused (t) represents the final result after data fusion at time t, H camera 、H ultrasonic 、H IMU 、H barometer They are the height information of binocular camera, ultrasonic sensor, IMU and barometer respectively, and w1, w2, w3 and w4 are the dynamic weights of each sensor.

[0126] 4. Flight control and feedback.

[0127] Flight Status Adjustment: The fused altitude and attitude information is used to control the drone's flight system. The embedded processing unit feeds the fusion results back to the flight controller, which adjusts the motor speed and drone's attitude in real time to maintain stable flight.

[0128] Feedback link: The control system adjusts the flight path in real time through the PID control algorithm based on the fused sensor information and the preset flight trajectory to ensure that the drone flies stably along the power inspection line.

[0129] Dynamic fusion of altitude measurements includes:

[0130] Takeoff: During takeoff, the barometer provides a rough estimate of the initial altitude, while the IMU compensates for dynamic changes in the barometer. The binocular camera and ultrasonic sensor play a primary role near the ground, providing precise altitude measurements.

[0131] During flight inspection, the adaptive deep learning model dynamically adjusts the weights of each sensor based on the terrain and environmental changes. For example, when the drone flies over water or highly reflective surfaces, the weights of the barometer and IMU increase, while those of the ultrasonic sensor and binocular camera decrease.

[0132] Landing phase: During landing, the ultrasonic sensor and binocular camera are used to accurately locate the ground, and the IMU and barometer serve as auxiliary to ensure the safe landing of the drone.

[0133] Environmental adaptability analysis includes:

[0134] Mountainous environments: Due to the uneven terrain in mountainous areas, ultrasonic sensors may experience reduced reflection accuracy. In these cases, the large model analyzes the terrain features and reduces the weight of the ultrasonic sensor while increasing the weight of the IMU and barometer to reduce measurement errors.

[0135] Airflow disturbances: When airflow disturbances are large, barometer measurements are easily affected. The adaptive deep learning model identifies airflow disturbances based on changes in IMU data, reduces the weight of the barometer in the fusion, and increases the weight of the ultrasonic and binocular cameras to improve measurement accuracy.

[0136] In summary, the multimodal data fusion system for UAV power inspection provided by this application effectively improves the accuracy and robustness of multimodal sensor data fusion through dynamic weight calculation and adaptive adjustment. In the prior art, sensor data fusion is mostly static mode, and once the weight is set, it cannot be adjusted according to environmental changes. This makes the system face the problem of reduced accuracy of fusion data under complex or unforeseen environmental conditions. By introducing dynamic weight calculation based on a large model, the weight can change with the real-time quality of sensor data, ensuring the optimal fusion effect of sensor data in different environments, thereby significantly improving the flight accuracy and stability of the UAV.

[0137] The adaptive data fusion mechanism has a high degree of anti-interference capability. During flight, some sensors may be affected by environmental interference or exhibit abnormal signals. For example, barometer data is prone to fluctuations or noise in strong winds, while laser sensor measurement accuracy may decrease in low-reflectivity terrain. By using a large model to monitor and dynamically evaluate sensor status in real time, anomalous data sources can be effectively detected, downgraded, or temporarily excluded from participation. This reduces the impact of a single sensor anomaly on the overall system and enhances the system's stability and fault tolerance in complex environments.

[0138] Multimodal fusion enables drones to maintain consistent flight accuracy across diverse terrains and altitudes. The coordinated integration of sensor data from binocular cameras, ultrasonic sensors, IMUs, and barometers enables the drone to obtain precise altitude and environmental information at varying altitudes and in diverse terrains. This flexible multimodal fusion strategy overcomes the limitations of a single sensor under specific conditions, enabling the system to more reliably perform inspections in complex terrains such as mountainous areas and forests.

[0139] The system's adaptability is further enhanced through a closed-loop feedback mechanism. By continuously monitoring the effectiveness of flight control commands and integrating sensor feedback, the computing unit dynamically adjusts the weighting and fusion strategies of each sensor, making the drone's flight control more precise and efficient. This closed-loop feedback control mechanism imbues the system with a certain degree of learning capability, allowing it to continuously optimize flight strategies based on actual flight conditions, thereby achieving long-term self-improvement and ensuring the reliability of inspection missions.

[0140] Building on existing technologies, this system addresses the limited accuracy and poor anti-interference capabilities of static fusion solutions in complex environments through the dynamic, weighted adaptive fusion of multimodal sensor data. It also incorporates a closed-loop feedback mechanism to achieve self-optimization. As a result, the system significantly improves the flight accuracy, data reliability, and anti-interference capabilities of UAV power inspection systems in complex environments, offering significant application value and advantages.

[0141] An embodiment of the present application may also provide a drone, including a housing, a bracket, a propeller, a motor, a power system, a battery module, and the above-mentioned drone power inspection multimodal data fusion system.

[0142] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0143] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0144] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A multimodal data fusion system for UAV power inspection, characterized by: include: A multimodal sensor unit, which is provided on the main body of the power inspection drone and is used to obtain the drone's altitude data and environmental parameters in real time; A computing and processing unit, the computing and processing unit being provided on the main body of the power inspection drone, the multimodal sensor unit being communicatively connected to the computing and processing unit; The computing processing unit is configured to perform the following operations: Acquiring sensor data corresponding to each sensor collected by the multimodal sensor unit; An adaptive deep learning model is used to extract features from each sensor data using a deep neural network, calculate the confidence of each sensor in the current environment, and dynamically adjust the weight of each sensor data in the fusion process based on the environmental data and sensor feedback information to obtain the target weight corresponding to each sensor data; the target weight is calculated by the following formula: w i (t)=exp(a i (t)) / ∑exp(a i (t)) Where: w i (t) represents the dynamic weight of the i-th sensor at time t, a i (t) represents the score calculated by the deep learning model, which represents the relative importance of the i-th sensor in the current environment at time t; fusing the sensor data using the target weights corresponding to the sensors to obtain target height information and target attitude information; The target height information and the target attitude information are sent to the UAV flight controller so that the UAV flight controller can adjust the flight path and height according to the target height information and the target attitude information.

2. The multimodal data fusion system for UAV power inspection according to claim 1 is characterized in that: The a i (t) is calculated by the following formula: a i (t)=f(H sensor (t),E env (t),w i (t-1)) Where: H sensor (t) represents the data of the i-th sensor at time t, E env (t) represents the current environment state, w i (t-1) represents the weight at the previous moment.

3. The UAV power inspection multimodal data fusion system according to claim 1 is characterized in that: The multimodal sensor unit includes at least a binocular camera, an ultrasonic sensor, an inertial measurement unit, and a barometer; The binocular camera is used to obtain ground visual information to calculate the height and posture of the drone relative to the ground; The ultrasonic sensor is used to detect the distance between the drone and the ground; The inertial measurement unit is used to detect the attitude changes of the drone in real time; The barometer is used to obtain the relative altitude information of the drone.

4. The multimodal data fusion system for UAV power inspection according to claim 3 is characterized in that: Dynamically adjusting the weights of the sensor data in the fusion process based on environmental data and sensor feedback information includes: When the drone takes off, the barometer is used to roughly estimate the initial altitude, the inertial measurement unit is used to compensate for dynamic changes in the barometer, and the binocular camera and the ultrasonic sensor provide accurate altitude measurement when approaching the ground; During flight, the adaptive deep learning model dynamically adjusts the weights of each sensor based on changes in terrain and environment. When the drone flies over water or highly reflective ground, the weights of the barometer and the inertial measurement unit increase, while the weights of the ultrasonic sensor and the binocular camera decrease. During landing, the ultrasonic sensor and the binocular camera are used to accurately locate the ground, with the inertial measurement unit and the barometer serving as auxiliary.

5. The multimodal data fusion system for UAV power inspection according to claim 3 is characterized in that: In a mountainous environment, the adaptive deep learning model is used to reduce the weight of the ultrasonic sensor and increase the weight of the inertial measurement unit and the barometer; When the airflow disturbance is large, the adaptive deep learning model is used to identify the airflow disturbance according to the data changes of the inertial measurement unit, and the weight of the barometer in the fusion is reduced, while the weights of the ultrasound and the binocular camera are increased.

6. The multimodal data fusion system for UAV power inspection according to claim 3 is characterized in that: The sensor data are fused using the following formula: H fused (t)=w1(t)H camera (t)+w2(t)H ultrasonic (t)+w3(t)H IMU (t)+w4(t)H barometer (t) Where: H fused (t) represents the final result after data fusion at time t, H camera 、H ultrasonic 、H IMU 、H barometer They are the height information of binocular camera, ultrasonic sensor, IMU and barometer respectively, and w1, w2, w3 and w4 are the dynamic weights of each sensor.

7. The multimodal data fusion system for UAV power inspection according to claim 1 is characterized in that: The computing processing unit includes an embedded processor.

8. A drone, characterized in that: The invention comprises a housing, a bracket, a propeller, a motor, a power system, a battery module and the multimodal data fusion system for power inspection of a UAV according to any one of claims 1 to 7.

Images