Unmanned aerial vehicle electric power inspection multi-modal data fusion system and unmanned aerial vehicle

Abstract

The invention discloses an unmanned aerial vehicle electric power inspection multi-modal data fusion system and an unmanned aerial vehicle, relates to the technical field of unmanned aerial vehicle control, and effectively improves the accuracy and robustness of multi-modal sensor data fusion through dynamic weight calculation and adaptive adjustment. In the prior art, data fusion of a sensor is mostly in a static mode, and the weight cannot be adjusted according to environmental changes once being set, so that a system possibly faces the problem that the fusion data precision is reduced under a complex or unpredictable environmental condition. By introducing the dynamic weight calculation based on the large model, the weight can change along with the real-time quality of the sensor data, the optimal fusion effect of the sensor data in different environments is ensured, and the flight precision and stability of the unmanned aerial vehicle are further remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) control, and particularly to a multi-modal data fusion system for UAV power inspection and a UAV. Background Art

[0002] With the development of unmanned aerial vehicle (UAV) technology, UAVs have been gradually applied to the field of power inspection, undertaking the detection work of power lines and equipment in high-risk areas. Traditional power inspection mostly relies on manual labor, while UAV inspection can not only significantly improve the inspection efficiency, but also replace manual work in dangerous environments, greatly reducing the labor cost and safety risk. In recent years, with the progress of sensor technology, UAVs have been able to carry a variety of sensors (such as high-definition cameras, infrared thermal imagers, lidar, etc.) to achieve a more comprehensive detection of power equipment.

[0003] In the prior art, multi-modal data fusion is applied to the fault detection and identification tasks of UAVs. By fusing data such as visible light images, infrared imaging, and lidar, the deficiencies of a single data source can be made up for, and the comprehensiveness and accuracy of detection can be significantly improved. However, the current static weighting mode, that is, a fixed weight allocation method is set to merge the data of different sensors. This static weighting mode performs well in some specific environments, but it cannot adapt to the dynamic changes of the environment, resulting in limited detection effects in complex environments.

[0004] For example, in the case of day-night alternation, bad weather, or drastic temperature fluctuations, the acquisition effects of each sensor vary greatly. The visible light image has a good effect during the day, but the effectiveness of this data source significantly decreases in the night environment; while at higher temperatures, noise interference may occur in the infrared imaging data. In addition, environmental factors such as wind speed and humidity will also affect the accuracy of lidar. To address these problems, some adaptive weighting methods have emerged in the prior art, which dynamically adjust the data weights through technologies such as deep learning to adapt to environmental changes, but these methods have not been systematically applied to the field of UAV power inspection.

[0005] The prior art provides a method for measuring the altitude of a UAV for power inspection based on multi-sensor fusion, which uses a binocular camera, an ultrasonic sensor, an inertial measurement unit (IMU) module, and a barometer to measure the flight altitude of the UAV to solve the problem of inaccurate altitude measurement under complex terrain conditions. The sensors of the existing solution are installed on the UAV, including a binocular camera, an ultrasonic sensor, an IMU module, and a barometer. The system first uses the values of the IMU and the barometer in the initial state as the benchmark for altitude measurement. During the takeoff and landing of the UAV, the IMU module and the barometer will capture the numerical changes. Through the real-time acquisition of these data, the system can make a preliminary judgment on the altitude information of the UAV.

[0006] During the flight of the drone, altitude measurement not only relies on the IMU and barometer, but also on the data from the binocular camera and ultrasonic sensor for auxiliary correction. The binocular camera obtains altitude information by calculating the image parallax, while the ultrasonic sensor estimates the altitude through the sound wave propagation time. These data are used as auxiliary information to correct the preliminary altitude measurement results obtained by the IMU and barometer, realizing altitude measurement with multi-sensor information fusion and improving the overall measurement accuracy. In the process of multi-sensor data fusion, the system adopts a static fusion strategy, that is, by assigning fixed weights to different sensors and fusing their data in a certain proportion to obtain the final altitude measurement value.

[0007] In addition, to cope with the error effects brought by complex environments, the system also includes steps for real-time state update and error calculation. Through state update, the system can adjust the altitude measurement results according to the changes in sensor values, calculate the altitude error gain, and continuously update the state covariance, so that the fusion results can effectively reduce the errors caused by environmental disturbances or sensor drifts.

[0008] The working process of the multi-sensor fusion system proceeds orderly from the benchmark establishment in the initial stage to sensor data acquisition, fusion, correction, and finally to the output of the final altitude information. In the overall hardware architecture of the system, the processor is responsible for executing relevant programs to process the data of each sensor, and the computer program for realizing the fusion and processing of sensor data is stored in the memory. Through this architecture combining hardware and software, the system can process various sensor data during the flight of the drone in real time to ensure accurate measurement of altitude.

[0009] It can be seen that the existing technical solutions adopt the method of static weight fusion in altitude measurement, that is, the fusion weights of different sensors are fixed and will not be dynamically adjusted with the change of the external environment. This static fusion method is effective in some cases, but in complex environments such as mountainous terrains, places with low surface reflectivity, or under conditions of large airflow disturbances, it may lead to a decrease in measurement accuracy. Because in these cases, the measurement reliability of a single sensor will change significantly, but the static fusion fails to fully consider this point and cannot dynamically and adaptively adjust the weights of each sensor.

[0010] The main drawback of the existing technology lies in the insufficient adaptability of its static weight fusion strategy to complex and changeable environments. The existing solutions use fixed weights to fuse the data of the binocular camera, ultrasonic sensor, IMU module, and barometer, which to a certain extent ignores the reliability changes of different sensors under different environmental conditions. Specifically, this fixed weight setting may lead to a decrease in altitude measurement accuracy under certain specific conditions, and the following is a detailed analysis.

[0011] First, in complex mountainous terrains, the surface reflectivity may be low, and the measurement accuracies of binocular cameras and ultrasonic sensors are significantly affected under these conditions. Binocular cameras rely on image parallax for distance measurement. When the surface reflectivity is low or there are obstructions, the accuracy of image parallax calculation is affected, leading to errors in height measurement. Ultrasonic sensors, on the other hand, are easily affected by irregular surfaces and environmental noise, making the measurement results inaccurate. However, the particularities of such environments are not specifically addressed in the prior art, and static fusion weights cannot reduce the impact of unreliable sensor measurement results on the final height measurement result.

[0012] Second, the barometric height measurement in the existing solutions is very sensitive to airflow disturbances. Especially in mountainous environments, due to frequent airflow changes, the measured values of the barometer may fluctuate significantly. However, since the weight of the barometer in the existing solutions is fixed, the system cannot effectively reduce its impact on the final fusion result when the barometer is unstable, which may lead to large deviations in height measurement.

[0013] In addition, due to the existence of cumulative errors, the measurement accuracy of the IMU module gradually decreases over time. Especially in the case of long - term flight of the unmanned aerial vehicle, the drift of the IMU will cause the accumulation of measurement errors. However, the prior art does not dynamically adjust the weight of the IMU according to this characteristic, but keeps a fixed weight. This makes the impact of the IMU on the fusion result gradually increase during long - term flight, thereby affecting the final height measurement accuracy.

[0014] Therefore, the static fusion strategy adopted in the prior art solutions makes the entire system lack adaptability to environmental changes and sensor performance fluctuations. This results in a significant decrease in measurement accuracy under complex environments, such as mountainous areas, low - reflectivity surfaces, and conditions with large airflow disturbances. Compared with the dynamic weight adaptive fusion method of the present invention, the prior art lacks the ability to dynamically adjust sensor weights according to real - time environmental changes, cannot fully utilize the advantages of multi - sensor fusion, and thus has low measurement accuracy and insufficient robustness in variable environments. Summary of the Invention

[0015] In view of the above problems, the present invention provides an unmanned aerial vehicle (UAV) power inspection multi - modal data fusion system and a UAV for overcoming the above problems or at least partially solving the above problems.

[0016] The present invention provides the following solutions:

[0017] An unmanned aerial vehicle power inspection multi - modal data fusion system, comprising:

[0018] A multi-modal sensor unit, which is arranged on the main body of the power inspection unmanned aerial vehicle, and is used to obtain the altitude data and environmental parameters of the unmanned aerial vehicle in real time;

[0019] A calculation and processing unit, which is arranged on the main body of the power inspection unmanned aerial vehicle, and the multi-modal sensor unit is communicably connected to the calculation and processing unit;

[0020] The calculation and processing unit is used to perform the following operations:

[0021] Obtain the sensor data corresponding to each sensor collected by the multi-modal sensor unit;

[0022] Use an adaptive deep learning model to extract features from each of the sensor data by using a deep neural network, calculate the confidence of each sensor in the current environment, and dynamically adjust the weights of each of the sensor data in the fusion process according to the environmental data and the feedback information of the sensor, so as to obtain the target weight corresponding to each of the sensor data;

[0023] Fuse each of the sensor data by using the target weights corresponding to each of the sensors to obtain target altitude information and target attitude information;

[0024] Send the target altitude information and the target attitude information to the unmanned aerial vehicle flight controller, so that the unmanned aerial vehicle flight controller can adjust the flight path and altitude according to the target altitude information and the target attitude information.

[0025] Preferably: The target weight is calculated by the following formula:

[0026] w i (t) = exp(a i (t)) / ∑exp(a i (t))

[0027] In the formula: w i (t) represents the dynamic weight of the i-th sensor at time t, and a i (t) represents the score calculated by the deep learning model, representing the relative importance of the i-th sensor in the current environment at time t.

[0028] Preferably: The a i (t) is calculated by the following formula:

[0029] a i (t) = f(H sensor (t)), E env (t), w i (t - 1)

[0030] Where: H sensor (t) represents the data of the i-th sensor at time t, and E env (t) represents the current environmental state, and w i (t - 1) represents the weight at the previous moment.

[0031] Preferably: The multi-modal sensor unit at least includes a binocular camera, an ultrasonic sensor, an inertial measurement unit, and a barometer;

[0032] The binocular camera is used to obtain ground visual information for calculating the height of the drone and its attitude relative to the ground;

[0033] The ultrasonic sensor is used to detect the distance between the drone and the ground;

[0034] The inertial measurement unit is used to detect the attitude change of the drone in real time;

[0035] The barometer is used to obtain the relative height information of the drone.

[0036] Preferably: Dynamically adjusting the weights of the data of each of the sensors in the fusion process according to the environmental data and the feedback information of the sensors includes:

[0037] When the drone takes off, the barometer is used to roughly estimate the initial height, the inertial measurement unit is used to compensate for the dynamic changes of the barometer, and the binocular camera and the ultrasonic sensor provide accurate height measurements when approaching the ground;

[0038] During the flight, according to the changes in different terrains and environments, the adaptive deep learning model dynamically adjusts the weights of the sensors; when the drone flies over water or a ground with strong reflectivity, the weights of the barometer and the inertial measurement unit will increase, and the weights of the ultrasonic sensor and the binocular camera will decrease;

[0039] When landing, the ultrasonic sensor and the binocular camera are used to accurately locate the ground, and the inertial measurement unit and the barometer are used as aids.

[0040] Preferably: In a mountainous environment, the weight of the ultrasonic sensor is reduced and the weights of the inertial measurement unit and the barometer are increased through the adaptive deep learning model;

[0041] When the airflow disturbance is large, the adaptive deep learning model identifies the airflow disturbance situation according to the data change of the inertial measurement unit, and reduces the weight of the barometer in the fusion and increases the weights of the ultrasonic and the binocular camera.

[0042] Preferably: The data of each of the sensors are fused by the following formula:

[0043] H fused H(t) = w 1 H(t) camera + w 2 H(t) ultrasonic + w 3 H(t) IMU H(t)

[0044] + w 4 H(t) barometer H(t)

[0045] Where: H fused H(t) represents the final result after data fusion at time t, H camera 、H ultrasonic 、H IMU 、H barometer are the height information of the binocular camera, ultrasonic sensor, IMU, and barometer respectively, and w 1 、w 2 、w 3 、w 4 are the dynamic weights of each sensor.

[0046] Preferably: The calculation and processing unit includes an embedded processor.

[0047] A drone, comprising a housing, a bracket, a propeller, a motor, a power system, a battery module, and the above-mentioned multi-modal data fusion system for drone power inspection.

[0048] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0049] A multi-modal data fusion system for drone power inspection and a drone provided by an embodiment of the present application. Through the calculation and adaptive adjustment of dynamic weights, the accuracy and robustness of multi-modal sensor data fusion are effectively improved. In the prior art, the data fusion of sensors is mostly in a static mode, and once the weights are set, they cannot be adjusted according to environmental changes, which may cause the problem of decreased accuracy of fused data in complex or unforeseeable environmental conditions. By introducing the dynamic weight calculation based on a large model, the weights can change with the real-time quality of sensor data, ensuring the optimal fusion effect of sensor data in different environments, and thus significantly improving the flight accuracy and stability of the drone.

[0050] The adaptive data fusion mechanism has a high anti-interference ability. During flight, some sensors may be affected by environmental interference or abnormal signals. For example, in a strong wind environment, the barometer data is prone to fluctuations or noise, and in a terrain with low reflectivity, the measurement accuracy of the laser sensor may decrease. By using a large model to monitor and dynamically evaluate the status of sensors in real time, abnormal data sources can be effectively detected, and their weights can be reduced or the data can be temporarily excluded from participation, thereby reducing the impact of a single sensor anomaly on the overall system and enhancing the stability and fault tolerance of the system in complex environments.

[0051] In addition, in the preferred embodiment, multi-modal fusion enables the drone to maintain consistent flight accuracy at different terrains and different flight altitudes. The collaborative fusion of sensor data such as binocular cameras, ultrasonic sensors, IMUs, and barometers enables the drone to obtain accurate altitude and environmental information at different altitudes in the air or when facing diverse terrains. This flexible multi-modal fusion strategy compensates for the limitations of single sensors under specific conditions, enabling the system to more reliably perform inspection tasks under complex terrain conditions such as mountains and forests.

[0052] Of course, it is not necessary for any product implementing the present invention to simultaneously achieve all the above-mentioned advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0054] Figure 1 is a schematic structural diagram of a multi-modal data fusion system for drone power inspection provided by an embodiment of the present invention;

[0055] Figure 2 is a working flow chart of a multi-modal data fusion system for drone power inspection provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0057] See Figure 1, a multi-modal data fusion system for UAV power inspection provided by an embodiment of the present invention, as Figure 1 shown, the system may include:

[0058] A multi-modal sensor unit, which is arranged on the main body of the UAV for power inspection. The multi-modal sensor unit is used to obtain the altitude data and environmental parameters of the UAV in real time. Specifically, in the implementation, the embodiment of the present application may provide that the multi-modal sensor unit at least includes a binocular camera, an ultrasonic sensor, an inertial measurement unit, and a barometer;

[0059] The binocular camera is used to obtain ground visual information for calculating the altitude of the UAV and its attitude relative to the ground;

[0060] The ultrasonic sensor is used to detect the distance between the UAV and the ground;

[0061] The inertial measurement unit is used to detect the attitude change of the UAV in real time;

[0062] The barometer is used to obtain the relative altitude information of the UAV.

[0063] A calculation and processing unit, which is arranged on the main body of the UAV for power inspection. The multi-modal sensor unit is communicatively connected to the calculation and processing unit. Specifically, in the implementation, the embodiment of the present application may provide that the calculation and processing unit includes an embedded processor.

[0064] The calculation and processing unit is used to perform the following operations:

[0065] Obtain the sensor data corresponding to each sensor collected by the multi-modal sensor unit;

[0066] Use an adaptive deep learning model to extract features from each of the sensor data using a deep neural network, calculate the confidence of each sensor in the current environment, and dynamically adjust the weights of each of the sensor data in the fusion process according to the environmental data and the feedback information of the sensors, so as to obtain the target weight corresponding to each of the sensor data. Specifically, in the implementation, the embodiment of the present application may provide that the target weight is calculated by the following formula:

[0067] w i (t) = exp(a i (t)) / ∑exp(a i (t))

[0068] In the formula: w i (t) represents the dynamic weight of the i-th sensor at time t, a i(t) represents the score calculated by the deep learning model, representing the relative importance of the i-th sensor at time t in the current environment.

[0069] Further, the a i (t) is calculated by the following formula:

[0070] a i (t) = f(H sensor (t)), E env (t), w i (t - 1)

[0071] In the formula: H sensor (t) represents the data of the i-th sensor at time t, E env (t) represents the current environmental state, w i (t - 1) represents the weight at the previous moment.

[0072] When the drone takes off, the barometer is used to roughly estimate the initial height, the inertial measurement unit is used to compensate for the dynamic changes of the barometer, and the binocular camera and the ultrasonic sensor provide accurate height measurements when approaching the ground;

[0073] During flight, according to the changes in different terrains and environments, the adaptive deep learning model dynamically adjusts the weights of each sensor; when the drone flies over water or a ground with strong reflectivity, the weights of the barometer and the inertial measurement unit will increase, and the weights of the ultrasonic sensor and the binocular camera will decrease;

[0074] When landing, the ultrasonic sensor and the binocular camera are used to accurately locate the ground, and the inertial measurement unit and the barometer are used as aids.

[0075] In a mountainous environment, the weight of the ultrasonic sensor is reduced and the weights of the inertial measurement unit and the barometer are increased through the adaptive deep learning model;

[0076] When the airflow disturbance is large, the adaptive deep learning model identifies the airflow disturbance situation according to the data change of the inertial measurement unit, and reduces the weight of the barometer in the fusion and increases the weights of the ultrasonic and the binocular camera.

[0077] The target weight corresponding to each sensor is used to fuse the data of each sensor to obtain the target height information and the target attitude information; specifically, in implementation, the embodiments of the present application can provide the fusion of the data of each sensor through the following formula:

[0078] H fused (t) = w 1 (t)H camera(t) + w 2 (t)H ultrasonic (t) + w 3 (t)H IMU (t)

[0079] +w 4 (t)H barometer (t)

[0080] Where: H fused (t) represents the final result after data fusion at time t, H camera 、H ultrasonic 、H IMU 、H barometer are the height information of the binocular camera, ultrasonic sensor, IMU, and barometer respectively, and w 1 、w 2 、w 3 、w 4 are the dynamic weights of each sensor.

[0081] Send the target height information and the target attitude information to the UAV flight controller, so that the UAV flight controller can adjust the flight path and height according to the target height information and the target attitude information.

[0082] The UAV power inspection multi-modal data fusion system provided by the embodiments of the present application solves the problems of low multi-modal data fusion accuracy, poor environmental adaptability, and insufficient data reliability in the existing UAV power inspection technology. In the existing UAV power inspection process, a static fusion method is often used to integrate multi-sensor data. However, since the static weights cannot be adaptively adjusted according to environmental changes, in complex inspection environments, especially in mountainous areas with variable terrain and frequent airflow disturbances, the measurement errors of sensors increase, making it difficult to ensure the measurement accuracy and stability during the inspection process.

[0083] First of all, the existing technology often uses static weights in multi-sensor data fusion, which means that the weight ratio between different sensors is preset and cannot dynamically adjust the importance of each sensor according to the actual scenario. For example, when the flight height changes or the environmental light changes, the measurement accuracies of sensors such as the binocular camera, ultrasonic sensor, and barometer will be different, and the fixed static weights cannot effectively cope with these changes, thus affecting the accuracy of height information measurement. The system provided by the present application introduces a dynamic weight adaptive fusion strategy, aiming to dynamically adjust the weights of each sensor according to real-time environment and sensor characteristic changes to improve the accuracy of data fusion.

[0084] Secondly, the sensor fusion methods of the prior art have obvious adaptability problems when facing complex terrains (such as mountains, slopes, etc.) and adverse environmental conditions (such as airflow disturbance, poor surface reflection). This lack of adaptability makes it difficult for drones to obtain stable altitude data during flight, affecting the smooth completion of inspection tasks. The system provided by this application solves this adaptability problem by using a large model to extract features from multi-modal sensor data and optimize the fusion strategy, enabling the fusion result to be adaptively adjusted to ensure stable and reliable altitude measurement in different environments. In addition, the prior art lacks the ability to effectively detect and process abnormal sensor data. Especially under the complex airflow conditions in mountainous areas, noise or outliers often appear in sensor data, and traditional static fusion methods are difficult to identify and correct these data in a timely manner, thus reducing the accuracy and reliability of overall measurement. The system provided by this application performs intelligent anomaly detection and correction on sensor data through a large model to ensure that outliers can be effectively removed during the fusion process, thereby further improving the credibility of the fused data.

[0085] It can be seen that the main purpose of the system provided by this application is to overcome the limitations of static fusion in the prior art and solve the problems of low accuracy of multi-modal sensor fusion, poor environmental adaptability, and insufficient data anomaly processing ability. By introducing the dynamic weight adaptive fusion method, the present invention can greatly improve the power inspection accuracy and stability of drones in complex environments, providing more accurate and reliable technical support for the inspection of power systems.

[0086] Dynamic weight adaptive fusion method driven by an adaptive deep learning model based on deep learning: The model itself analyzes each sensor data (binocular camera, ultrasonic, IMU, barometer) in real time, dynamically calculates and assigns sensor weights to cope with the fluctuations in sensor signal quality caused by environmental changes, ensuring the reliability and accuracy of the data fusion result.

[0087] Adaptive multi-modal data fusion algorithm: Using adaptive fusion algorithms such as weighted average and Bayesian estimation to effectively integrate multi-sensor data, achieving highly accurate altitude measurement in complex terrains or special environments, especially suitable for precise altitude control of drones during low-altitude flight.

[0088] Closed-loop feedback control and optimization mechanism: The system designs a closed-loop feedback structure, utilizes the accuracy of the multi-modal data fusion result to monitor and feedback the execution of flight control commands in real time, ensures control accuracy, and at the same time has the self-learning ability of continuous optimization, thereby improving flight stability and adaptability.

[0089] Environmental Status Assessment and Sensor Abnormality Handling Strategy: Evaluate the status of sensors in different environments through a deep learning model, detect data noise and signal interference in real time, automatically exclude abnormal data or reduce the contribution of unreliable data, and ensure the overall stability and high credibility of the fused data.

[0090] Multi-level Collaborative Optimization Data Fusion Mechanism: Combine the rough estimates of the barometer and IMU with the fine measurements of the binocular camera and ultrasonic wave to construct a multi-level and multi-precision sensor data fusion system, optimize the advantages of different types of sensors, and achieve dynamic fusion from rough to precise to enhance the multi-scenario adaptability of the system.

[0091] High-precision Data Fusion Technology for UAV Power Inspection: A multi-modal data fusion technology specially applicable to power inspection tasks, which enhances the data adaptability of UAVs under low-altitude and complex terrain conditions, improves inspection accuracy, and ensures that UAVs have higher environmental adaptability and task reliability in the power inspection environment.

[0092] The following provides a detailed introduction to the system provided in this application.

[0093] The system is based on a control process of multi-sensor data acquisition and fusion, including links such as real-time data acquisition, dynamic weight calculation, fusion processing, and anomaly detection, to ensure the accuracy and reliability of UAV power inspection. The adaptive deep learning model provided in the embodiments of this application extracts features from multi-modal sensor data and optimizes the fusion strategy by real-time collecting sensor data. First, the system dynamically adjusts the weights of each sensor according to the environmental feedback information of each sensor using deep learning algorithms. The model extracts features from the input data, identifies the impact of environmental changes on the accuracy of each sensor, and then optimizes the data fusion process.

[0094] The operation of this model includes the following steps:

[0095] 1. Data Acquisition and Preprocessing: Each sensor collects data in real time, and the preprocessing module removes noise and outliers. Multiple sensors such as binocular cameras, ultrasonic sensors, inertial sensors (IMU modules), and barometers are fused to obtain the altitude data of the UAV and other environmental parameters in real time.

[0096] 2. Feature Extraction and Weight Calculation: The adaptive deep learning model uses a deep neural network to extract features from the data of each sensor, calculates the confidence of each sensor in the current environment, and dynamically assigns weights. Through a dynamic weighting method, the fusion weights of each sensor are adjusted in real time based on the confidence of the sensor data and environmental changes, improving the data fusion accuracy and system adaptability.

[0097] 3. Data fusion: Based on the calculated weights, the data of each sensor are fused to obtain key information such as the final height and attitude.

[0098] 4. Control and feedback: The fused data are used in the control system of the UAV to achieve precise adjustment of the flight path and height.

[0099] The main function of this model is to improve the inspection accuracy and stability of the UAV in complex environments. By dynamically adjusting the fusion weights of each sensor, the adaptive deep learning model can optimize the results of data fusion under different environmental conditions, ensuring that the system can still provide stable and reliable measurement and control information in various complex situations.

[0100] A multi-modal data dynamic weight adaptive fusion system for UAV power inspection provided by an embodiment of this application uses multiple sensors to improve the accuracy of measuring the height and pose of the UAV through fusion technology, and calculates the dynamic weights of sensor data with the help of an adaptive deep learning model to cope with the complexity of various inspection environments. The specific implementation methods are as follows:

[0101] UAV body: The UAV body provided by an embodiment of this application includes a housing, a bracket, propellers, motors, a power system, a battery module, etc. These components are connected to each other to form a stable platform for carrying multiple sensors and computing units.

[0102] The multi-modal sensor unit includes:

[0103] Binocular camera: Installed at the front of the UAV, it is used to obtain ground visual information to calculate the height of the UAV and its attitude relative to the ground.

[0104] Ultrasonic sensor: Installed at the bottom of the UAV, it is mainly used to detect the distance between the UAV and the ground, and is suitable for precise distance measurement at medium and low altitudes.

[0105] Inertial measurement unit (IMU): Consisting of a three-axis accelerometer and a three-axis gyroscope, it is used to detect the attitude changes of the UAV in real time, including information such as acceleration and angular velocity.

[0106] Barometer: Used to obtain the relative height information of the UAV, and is especially suitable for rough height estimation in a large height range.

[0107] Computing and processing unit: An embedded processor (such as an ARM Cortex series chip) is used to collect the data of each sensor and perform real-time fusion processing. This processing unit integrates the adaptive deep learning model algorithm for dynamically calculating the weights of each sensor.

[0108] As Figure 2 shown, the system functions and processes are described as follows:

[0109] 1. Data acquisition and transmission.

[0110] Each sensor collects data at a fixed frequency (50Hz) and transmits the data to the computing and processing unit through a communication interface (I2C or SPI).

[0111] The goal of data acquisition is to obtain sufficiently detailed altitude and attitude information to provide a basis for subsequent fusion and weight calculation.

[0112] 2. Dynamic weight calculation module.

[0113] Initial weight setting: When the system is initialized, the initial weights are set according to the working principles of each sensor and historical environmental data, and the sum of the weight values of all sensors is 1.

[0114] Dynamic weight adjustment: During the flight of the UAV, the adaptive deep learning model algorithm in the embedded processor will adjust the weights of each sensor in real time according to the real-time environmental data and sensor feedback information. The specific adjustment basis includes: factors such as ground reflection characteristics, environmental airflow disturbances, and terrain undulations. These environmental changes will affect the accuracy and reliability of the sensors.

[0115] The calculation formula of the dynamic weight is based on the feature extraction results of the deep learning model, which can identify the impact of environmental changes on the accuracy of each sensor in real time, and optimize the weight adjustment according to the environmental conditions to obtain the target weight. The target weight is calculated by the following formula:

[0116] w i (t) = exp(a i (t)) / ∑exp(a i (t))

[0117] In the formula: w i (t) represents the dynamic weight of the i-th sensor at time t, and a i (t) represents the score calculated by the deep learning model, representing the relative importance of the i-th sensor in the current environment at time t.

[0118] The a i (t) is calculated by the following formula:

[0119] a i (t) = f(H sensor (t)), E env (t), w i (t - 1)

[0120] In the formula: H sensor (t) represents the data of the i-th sensor at time t, E env (t) represents the current environmental state, wi (t - 1) represents the weight at the previous moment.

[0121] 3. Data fusion module.

[0122] Multi-modal data fusion: The computing unit receives real-time data from the binocular camera, ultrasonic sensor, IMU, and barometer, and uses dynamic weights to perform weighted averaging on the measurement values of each sensor to obtain optimal height and attitude estimates. The data of each of the said sensors is fused through the following formula:

[0123] H fused (t) = w 1 (t)H camera (t) + w 2 (t)H ultrasonic (t) + w 3 (t)H IMU (t)

[0124] + w 4 (t)H barometer (t)

[0125] In the formula: H fused (t) represents the final result after data fusion at time t, H camera , H ultrasonic , H IMU , H barometer are the height information of the binocular camera, ultrasonic sensor, IMU, and barometer respectively, and w 1 , w 2 , w 3 , w 4 are the dynamic weights of each sensor.

[0126] 4. Flight control and feedback.

[0127] Flight state adjustment: The fused height and attitude information is used to control the flight system of the drone. The embedded processing unit feeds back the fusion result to the flight controller to adjust the rotation speed of the motor and the attitude of the drone in real time, so that the drone maintains stable flight.

[0128] Feedback link: The control system adjusts the flight path in real time according to the fused sensor information and the preset flight trajectory through the PID control algorithm to ensure that the drone flies stably along the power inspection line.

[0129] The dynamic fusion of height measurement includes:

[0130] Takeoff stage: When the drone takes off, the barometer is used for a rough estimate of the initial height, and the IMU is used to compensate for the dynamic changes of the barometer. The binocular camera and ultrasonic sensor play a major role at heights close to the ground, providing accurate height measurements.

[0131] Flight inspection phase: During flight, according to the changes in different terrains and environments, the adaptive deep learning model dynamically adjusts the weights of each sensor. For example, when the drone flies over water or a ground with strong reflectivity, the weights of the barometer and IMU will increase, while the weights of the ultrasonic sensor and binocular camera will decrease.

[0132] Landing phase: During landing, the ultrasonic sensor and binocular camera are used for precise ground positioning, and the IMU and barometer are used as aids to ensure the safe landing of the drone.

[0133] Environmental adaptability analysis includes:

[0134] Mountainous environment: In a mountainous environment, due to the uneven ground, the ultrasonic sensor may have a problem of decreased reflection accuracy. At this time, after the large model analyzes the terrain features, it will reduce the weight of the ultrasonic sensor and increase the weights of the IMU and barometer to reduce measurement errors.

[0135] In the case of airflow disturbance: When the airflow disturbance is large, the measurement data of the barometer is easily affected. The adaptive deep learning model identifies the airflow disturbance situation based on the data changes of the IMU, and reduces the weight of the barometer in the fusion and increases the weights of the ultrasonic and binocular cameras to improve the measurement accuracy.

[0136] In summary, the multi-modal data fusion system for drone power inspection provided by this application effectively improves the accuracy and robustness of multi-modal sensor data fusion through the calculation and adaptive adjustment of dynamic weights. In the prior art, the data fusion of sensors is mostly in a static mode, and once the weights are set, they cannot be adjusted according to environmental changes, which may cause the problem of decreased accuracy of the fused data in complex or unforeseen environmental conditions. By introducing the dynamic weight calculation based on the large model, the weights can change with the real-time quality of the sensor data, ensuring the optimal fusion effect of the sensor data in different environments, and thus significantly improving the flight accuracy and stability of the drone.

[0137] The adaptive data fusion mechanism has high anti-interference ability. During flight, some sensors may be affected by environmental interference or abnormal signals. For example, in a strong wind environment, the barometer data is prone to fluctuations or noise, and in a terrain with low reflectivity, the measurement accuracy of the laser sensor may decrease. By using the large model to monitor and dynamically evaluate the state of the sensors in real time, it can effectively detect abnormal data sources, reduce their weights or temporarily exclude the participation of this data, thereby reducing the impact of a single sensor anomaly on the overall system and enhancing the stability and fault tolerance of the system in complex environments.

[0138] Multi-modal fusion enables the drone to maintain consistent flight accuracy at different terrains and flight altitudes. The collaborative fusion of sensor data such as binocular cameras, ultrasonic sensors, IMUs, and barometers enables the drone to obtain accurate altitude and environmental information at different altitudes in the air or when facing diverse terrains. This flexible multi-modal fusion strategy compensates for the limitations of a single sensor under specific conditions, enabling the system to perform inspection tasks more reliably under complex terrain conditions such as mountains and forests.

[0139] The adaptability of the system can also be further enhanced through a closed-loop feedback mechanism. By continuously monitoring the execution effect of flight control commands and combining sensor feedback information, the calculation unit can dynamically adjust the weights and fusion strategies of each sensor, making the flight control of the drone more accurate and efficient. This closed-loop feedback control mechanism enables the system to have a certain learning ability, continuously optimize the flight strategy according to the actual flight situation, and thus achieve self-improvement during long-term operation to ensure the reliability of the inspection task.

[0140] Based on the existing technology, this system solves the problems of insufficient accuracy and poor anti-interference ability of the static fusion scheme in complex environments through the dynamic weight adaptive fusion of multi-modal sensor data, and at the same time introduces a closed-loop feedback mechanism to achieve self-optimization of the system. Therefore, this system greatly improves the flight accuracy, data reliability, and anti-interference ability of the drone power inspection system in complex environments, and has significant application value and advantages.

[0141] An embodiment of this application can also provide a drone, including a housing, a bracket, propellers, motors, a power system, a battery module, and the above-mentioned multi-modal data fusion system for drone power inspection.

[0142] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0143] As can be seen from the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of this application, in essence, or the part that makes a contribution to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0144] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for a system or a system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiment. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0145] The above is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A multi-modal data fusion system for UAV power inspection, characterized in that: include: A multimodal sensor unit, which is arranged on the main body of the power inspection UAV and is used to obtain the altitude data and environmental parameters of the UAV in real time; A computing and processing unit, wherein the computing and processing unit is disposed on the main body of the power inspection drone, and the multimodal sensor unit is communicatively connected to the computing and processing unit; The computing processing unit is used to perform the following operations: Acquire sensor data corresponding to each sensor collected by the multimodal sensor unit; Adopting an adaptive deep learning model to extract features from each of the sensor data using a deep neural network, calculating the confidence of each of the sensors in the current environment, and dynamically adjusting the weight of each of the sensor data in the fusion process according to the environmental data and the feedback information of the sensor, so as to obtain the target weight corresponding to each of the sensor data; fusing the data of each sensor using the target weight corresponding to each sensor to obtain target height information and target posture information; The target height information and the target attitude information are sent to the UAV flight controller so that the UAV flight controller can adjust the flight path and height according to the target height information and the target attitude information.

2. The UAV power inspection multimodal data fusion system according to claim 1 is characterized in that: The target weight is calculated by the following formula: w i (t)=exp(a i (t)) / ∑exp(a i (t)) Where: w i (t) represents the dynamic weight of the i-th sensor at time t, a i (t) represents the score calculated by the deep learning model, which represents the relative importance of the i-th sensor in the current environment at time t.

3. The UAV power inspection multimodal data fusion system according to claim 2 is characterized in that: The a i (t) is calculated by the following formula: a i (t)=f(H sensor (t)),E env (t),w i (t-1) Where: H sensor (t) represents the data of the i-th sensor at time t, E env (t) represents the current environment state, w i (t-1) represents the weight at the previous moment.

4. The UAV power inspection multimodal data fusion system according to claim 1 is characterized in that: The multimodal sensor unit includes at least a binocular camera, an ultrasonic sensor, an inertial measurement unit and a barometer; The binocular camera is used to obtain ground visual information to calculate the height of the drone and its posture relative to the ground; The ultrasonic sensor is used to detect the distance between the drone and the ground; The inertial measurement unit is used to detect the attitude changes of the drone in real time; The barometer is used to obtain the relative altitude information of the drone.

5. The UAV power inspection multimodal data fusion system according to claim 4 is characterized in that: Dynamically adjusting the weight of each sensor data in the fusion process according to the environmental data and the feedback information of the sensor includes: When the drone takes off, the barometer is used to roughly estimate the initial altitude, the inertial measurement unit is used to compensate for the dynamic changes of the barometer, and the binocular camera and the ultrasonic sensor provide accurate altitude measurement when approaching the ground; During the flight, the adaptive deep learning model dynamically adjusts the weight of each sensor according to the changes in different terrains and environments; when the drone flies over water or highly reflective ground, the weight of the barometer and the inertial measurement unit will increase, and the weight of the ultrasonic sensor and the binocular camera will decrease; During landing, the ultrasonic sensor and the binocular camera are used to accurately locate the ground, and the inertial measurement unit and the barometer are used as auxiliary.

6. The UAV power inspection multimodal data fusion system according to claim 4 is characterized in that: In a mountainous environment, the adaptive deep learning model is used to reduce the weight of the ultrasonic sensor and increase the weight of the inertial measurement unit and the barometer; When the airflow disturbance is large, the adaptive deep learning model is used to identify the airflow disturbance according to the data changes of the inertial measurement unit, and the weight of the barometer in the fusion is reduced, and the weights of the ultrasonic wave and the binocular camera are increased.

7. The UAV power inspection multimodal data fusion system according to claim 4 is characterized in that: The sensor data are fused by the following formula: H fused (t)=w1(t)H camera (t)+w2(t)H ultrasonic (t)+w3(t)H IMU (t) +w4(t)H barometer (t) Where: H fused (t) represents the final result after data fusion at time t, H camera , H ultrasonic , H IMU , H barometer They are the height information of binocular camera, ultrasonic sensor, IMU and barometer respectively. w1, w2, w3 and w4 are the dynamic weights of each sensor.

8. The UAV power inspection multimodal data fusion system according to claim 1 is characterized in that: The computing processing unit includes an embedded processor.

9. A drone, characterized in that: It comprises a shell, a bracket, a propeller, a motor, a power system, a battery module and the unmanned aerial vehicle power inspection multimodal data fusion system according to any one of claims 1 to 8.

Images