Unmanned aerial vehicle aiot global perception sensing module based on multi-source heterogeneous fusion

Abstract

The application discloses a multi-source heterogeneous fusion-based AIoT global perception sensor module of a UAV, which comprises a perception layer composed of image, ranging and environmental physical quantity sensors; a data fusion processing unit based on an FPGA+ARM heterogeneous architecture, which is used for time-space synchronization, feature-level fusion and semantic analysis of multi-source data; a communication and interface unit; an isolated power management unit and a protective shell.The module completes pixel-level registration, RGB-D generation and lightweight AI inference at an edge end, outputs a global semantic map containing an obstacle category, depth information and environmental parameters, and transmits the global semantic map to a flight control and a ground station in real time through a CAN bus and wireless data transmission.The design realizes millisecond-level close coupling fusion of multi-source data, solves the problems of poor real-time performance, asynchronous conflict and bandwidth bottleneck in a traditional scheme, and significantly improves all-weather autonomous perception and obstacle avoidance capability of the UAV.

Description

Technical Field

[0001] This application relates to the fields of UAV environmental perception and AIoT technology, specifically a UAV AIoT global perception sensing module based on multi-source heterogeneous fusion. Background Technology

[0002] In recent years, drone technology has been widely and deeply applied in many fields such as power line inspection, agricultural plant protection, emergency rescue, and urban security. As application scenarios evolve from simple line-of-sight flight to complex, beyond-line-of-sight, and fully autonomous operations, unprecedented demands are being placed on the environmental perception capabilities of drones. They need to acquire multi-dimensional information, including visible light, infrared, distance, environmental parameters, and even multispectral data, on a miniaturized, low-power platform to cope with complex challenges such as changing lighting conditions, severe weather, and dense obstacles.

[0003] Currently, mainstream drone perception solutions on the market typically adopt a functionally separate architecture, meaning each sensor carries an independent camera, ultrasonic sensor, lidar, etc., and the data is simply aggregated through a flight control system. For example, an existing patent discloses a multi-sensor mount device for drones, which only achieves physical fixation of multiple sensors and independent data transmission. Existing technologies attempt to fuse data from different sensors through software at the backend ground station, but real-time performance is limited by communication link bandwidth and latency, failing to meet the millisecond-level response requirements for autonomous obstacle avoidance in the air.

[0004] The shortcomings and problems in existing technologies are mainly reflected in the following aspects: First, the poor time synchronization and spatial registration accuracy between heterogeneous sensors leads to ghosting or information contradictions in the fused data, reducing the reliability of perception. Second, sensor data fusion processing heavily relies on high-performance computing resources in the cloud or ground station, making it impossible to achieve low-latency, high-throughput fusion processing at the local edge of the module, resulting in delayed autonomous decision-making. Third, the power management and thermal design of the module are not optimized for multi-source high-power sensors (such as LiDAR and thermal imagers), making it prone to power supply noise interference and thermal failure within the compact size of the UAV. Therefore, there is an urgent need to design a dedicated UAV sensing module that can achieve tight-coupled fusion from the hardware layer to the algorithm layer and has full-domain perception capabilities. Summary of the Invention

[0005] This application provides a drone AIoT global perception sensing module based on multi-source heterogeneous fusion. Through the tightly coupled layout of heterogeneous sensors at the hardware level, chip-level acceleration of dedicated data fusion units, and optimized power supply and structural thermal design, it can effectively solve the problems of spatiotemporal asynchrony of multi-source data, large fusion processing delay, and poor environmental adaptability in the background technology.

[0006] To achieve the above objectives, this application provides the following technical solution: a UAV AIoT global perception sensing module based on multi-source heterogeneous fusion, comprising a perception layer component (including three or more types of sensors: image, ranging, and environmental physical quantities), a data fusion processing unit (based on FPGA+ARM heterogeneous computing), a communication and interface unit, a power management unit, and a housing. During operation, the data fusion processing unit performs pixel-level real-time registration and preprocessing of image and ranging data using an FPGA parallel pipeline. Subsequently, the ARM core runs a lightweight AI model for semantic-level fusion, ultimately generating a unified global perception data frame with timestamps and confidence levels, which is then sent to the UAV flight control system via redundant communication interfaces. This technical solution achieves integrated, low-latency, and highly reliable front-end processing from raw multi-source data to decision-making information.

[0007] Furthermore, the image acquisition unit includes at least two of the following: a visible light camera module, an infrared thermal imaging module, and a low-light night vision module; the ranging unit is a solid-state lidar or a millimeter-wave radar; and the environmental physical quantity sensing unit includes a temperature and humidity sensor, a barometric pressure sensor, and a multispectral sensor. This multimodal configuration enables all-weather, multi-spectral environmental information capture, significantly improving the perception robustness in complex scenarios.

[0008] Furthermore, the core of the data fusion processing unit is an embedded heterogeneous computing chip, including an FPGA logic gate array and an ARM core processor. The FPGA is used to implement pixel-level real-time correction of the image acquisition unit and point cloud preprocessing of the ranging unit, while the ARM core processor runs a lightweight deep learning model to perform target recognition and semantic segmentation. This hardware heterogeneous architecture combines the low-latency parallel processing capabilities of the FPGA with the complex logic control advantages of the ARM, completing multi-source data fusion in milliseconds.

[0009] Furthermore, the communication and interface unit includes a CAN bus interface, a UART serial port, and a wireless data transmission module; the wireless data transmission module supports 4G / 5G or LoRa protocols and is used to transmit critical perception data when the UAV is outside its line-of-sight range. This redundant communication design ensures highly reliable internal interaction with the flight control system and a long-distance data backup link with the ground station.

[0010] Furthermore, the power management unit includes an EMI filter circuit, an isolated DC-DC converter, and a multi-channel LDO regulator; the isolated DC-DC converter independently powers the image acquisition unit, the ranging unit, and the data fusion processing unit. This isolated power supply architecture effectively suppresses power supply noise introduced by high-power devices such as motors, ensuring the signal-to-noise ratio of analog sensors (such as the ranging unit).

[0011] Furthermore, the housing is made of aerospace-grade aluminum alloy or carbon fiber composite material, with heat dissipation fins on its bottom and sides, and the interior of the housing is filled with thermally conductive potting compound. This structural design improves the module's heat dissipation capacity and environmental adaptability while reducing weight, making it suitable for the thermal management requirements of high-intensity flights.

[0012] Furthermore, it also includes an inertial measurement unit electrically connected to the data fusion processing unit, used to acquire the module's own angular velocity and acceleration data; the data fusion processing unit uses the data from the inertial measurement unit to perform motion compensation on the sensing data from the image acquisition unit and the ranging unit. This design effectively eliminates the distortion effects of UAV flight vibration and attitude changes on sensor data.

[0013] Furthermore, the data fusion processing unit internally runs a multi-source sensor spatiotemporal synchronization algorithm based on extended Kalman filtering or graph optimization, performing pixel-by-pixel registration between the two-dimensional pixel coordinates of the image acquisition unit and the three-dimensional point cloud coordinates of the ranging unit to generate an RGB-D data stream that integrates color and depth information. This feature-level fusion directly provides a dense environmental depth map for UAV obstacle avoidance and autonomous navigation.

[0014] Furthermore, the data fusion processing unit embeds a lightweight neural network model based on knowledge distillation, which performs semantic-level fusion of data from the image acquisition unit and multispectral sensors locally, outputting a global semantic map containing obstacle category and vegetation health labels. This edge computing capability significantly reduces the computational load on the UAV flight control system and lowers the data transmission bandwidth pressure.

[0015] Furthermore, the outer surface of the housing is equipped with a quick-release dust and rain cover, and optically coated glass is embedded in the corresponding sensor windows of the quick-release dust and rain cover. This design enables the module to maintain its sensing function under adverse weather conditions such as rain, fog, and dust, expanding the range of environments in which the UAV can operate.

[0016] Compared with the prior art, the beneficial effects of this application are: 1. This application achieves millisecond-level spatiotemporal synchronization and feature-level fusion of multi-source heterogeneous data such as images, point clouds, and environmental physical quantities at the module level through an independently designed data fusion processing unit (FPGA+ARM heterogeneous architecture). This solves the problems of poor real-time performance and bandwidth bottleneck caused by relying on ground station processing in traditional solutions, and significantly improves the response speed and safety of UAV autonomous flight.

[0017] 2. This application integrates various heterogeneous sensors such as visible light, infrared, radar, and multispectral sensors, and combines them with motion compensation algorithms to achieve all-weather (day / night / rain / fog) and all-dimensional (2D vision / 3D space / environmental parameters) all-domain perception capabilities, making up for the functional blind spots of single sensors in specific environments and greatly expanding the operational scenarios of UAVs.

[0018] 3. This application effectively solves the electromagnetic compatibility and heat dissipation problems of multiple high-power sensors coexisting by adopting isolated power supply partitioning, thermal potting and heat dissipation fin integrated shell design, and ensures the long-term working stability and reliability of the module in the confined space and high-intensity flight conditions of UAV. Attached Figure Description

[0019] Figure 1 This is a cross-sectional view of the overall structure of the UAV AIoT global perception sensing module based on multi-source heterogeneous fusion in this application.

[0020] Figure 2 This is an internal logic block diagram of the data fusion processing unit in this application.

[0021] Figure 3 This is a schematic diagram showing the module of this application installed under the fuselage of a drone in its usage state.

[0022] In the diagram: 1. Image acquisition unit; 11. Visible light camera module; 12. Infrared thermal imaging module; 13. Low-light night vision module; 2. Ranging unit; 3. Environmental physical quantity sensing unit; 31. Temperature and humidity sensor; 32. Barometric pressure sensor; 33. Multispectral sensor; 4. Data fusion processing unit; 41. FPGA logic gate array; 42. ARM core processor; 5. Communication and interface unit; 51. CAN bus interface; 52. UART serial port; 53. Wireless data transmission module; 6. Power management unit; 61. EMI filter circuit; 62. Isolated DC-DC converter; 63. LDO regulator; 7. Housing; 71. Heat sink; 72. Quick-release dustproof and rainproof cover; 8. Inertial measurement unit. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In the description of this application, any descriptions of orientation, such as up, down, front, back, left, right, etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are merely for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. When a feature is referred to as being set, fixed, or connected to another feature, it can be directly set, fixed, or connected to the other feature, or it can be indirectly set, fixed, or connected to the other feature.

[0025] Please see Figures 1 to 3 This application provides the following technical solution: a multi-source heterogeneous fusion-based UAV AIOT global perception sensing module, including: a perception layer component, specifically including an image acquisition unit 1 integrating a visible light camera module 11 (model: SONY IMX415, 4K resolution), an infrared thermal imaging module 12 (resolution 640512, uncooled type), and a low-light night vision module 13 (starlight-level CMOS); a solid-state lidar based on MEMS micromirror technology as a ranging unit 2 (detection distance 0.1-50m, accuracy 2cm); and an environmental physical quantity sensing unit 3 integrating a temperature and humidity sensor 31 (SHT30), a barometric pressure sensor 32 (BMP388), and a multispectral sensor 33 (6 channels, 450nm-850nm). A data fusion processing unit 4 with a Xilinx Zynq UltraScale+ series chip (integrating an FPGA logic gate array 41 and a quad-core ARM Cortex-A53 processor 42) as its core, whose input terminals are electrically connected to all the above sensors. A communication and interface unit 5, comprising a CAN bus interface 51 (for communication with the flight controller), a UART serial port 52 (for debugging and backup), and a 4G wireless data transmission module 53 (EC20), is connected to the output of the data fusion processing unit 4. A power management unit 6, including an EMI filter circuit 61, an isolated DC-DC converter 62 (12V input, isolated 5V, 3.3V, and 1.8V outputs), and multiple low-noise LDO regulators 63, provides independent power to each unit. A housing 7, made of 6061 aerospace-grade aluminum alloy, has heat sink fins 71 at the bottom and is filled with thermally conductive potting compound. Furthermore, the inertial measurement unit 8 (ICM-42688) is also soldered onto the same circuit board as the data fusion processing unit 4 and communicates with it via the SPI bus.

[0026] In use: First, fix this module to the bottom or front of the UAV body through the mechanical interface of the housing 7, and connect the 12V DC power supply and CAN bus interface 51 provided by the UAV. After power-on, the isolated DC-DC converter 62 of the power management unit 6 starts up, and establishes a stable and clean power supply for the lidar (range measuring unit 2), thermal imaging module (12) and FPGA+ARM core in sequence.

[0027] After the module performs a self-test upon startup, it enters normal operating mode. Data Acquisition: The visible light, infrared, and night vision modules of image acquisition unit 1 output video streams at a frame rate of 30fps; the solid-state LiDAR outputs 3D point clouds at a scanning frequency of 20Hz; the environmental physical quantity sensing unit 3 outputs temperature, humidity, air pressure, and multispectral reflectance data at a rate of 100Hz; and the inertial measurement unit 8 outputs acceleration and angular velocity at a rate of 400Hz. All of these data are accompanied by hardware timestamps from the data fusion processing unit 4.

[0028] Low-level preprocessing: The FPGA logic gate array 41 receives all data streams in parallel. On one hand, it uses the data from the inertial measurement unit 8 to remove motion distortion from the lidar point cloud using a lookup table method; on the other hand, it performs pixel-level processing such as bad pixel correction and gamma correction on visible light and infrared images, and uses pre-defined calibration parameters to geometrically register the pixel coordinates of the two-dimensional image with the coordinates of the three-dimensional point cloud to generate each point cloud with color information.

[0029] Mid-layer fusion: The ARM core processor 42 runs a lightweight YOLOv5s model (accelerated by TensorRT) to perform target detection on the fused RGB-D data stream, identifying obstacles such as power lines, tree branches, and birds within 10 meters ahead, and providing their bounding boxes and depth information. Simultaneously, the ARM core reads data from the multispectral sensor 33 and performs spectral analysis with vegetation areas in the visible light image to calculate the Normalized Difference Vegetation Index (NDVI).

[0030] Data Output: The data fusion processing unit 4 encapsulates the above processing results into a custom UAV perception data frame. This frame includes: a synchronized timestamp, a dense depth map (obtained from the fusion of LiDAR and vision), an obstacle list (category, distance, azimuth), and the region's NDVI value. This data frame is sent to the UAV flight control system at a rate of 500kbps via the CAN bus interface 51. Simultaneously, key frames and abnormal alarms (such as the detection of high-temperature points or obstacles less than 5 meters away) are backed up and uploaded to the ground monitoring station via a 4G network through the wireless data transmission module 53.

[0031] The entire processing flow, from the input of raw sensor data to the generation of the final perception data frame, has a total latency controlled within 50ms, meeting the real-time requirements of the UAV for emergency obstacle avoidance at a speed of 10m / s.

[0032] It is worth noting that the component selections disclosed in this embodiment are merely examples and are not intended to limit the scope of this application. For example, the solid-state lidar can be either the Beowing CE30 or the OusterOS0-32, and its detection range and scan line count can be adjusted according to the UAV's payload capacity and operational scenario; the core chip of the data fusion processing unit 4 can also be an NVIDIA Jetson OrinNX, paired with an independent FPGA coprocessor; the inertial measurement unit 8 can also be a Bosch BMI088. The isolation voltage of the power management unit 6 is recommended to be no less than 1500VDC. The data fusion and deep learning algorithms running on the control core (ARM processor 42) are implemented using the commonly used Caffe / TensorRT inference framework and EKF fusion library. The shell material can be replaced with carbon fiber to further reduce weight without affecting the electromagnetic shielding effect.

[0033] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A drone AIoT global perception sensing module based on multi-source heterogeneous fusion, characterized in that, include: The perception layer component includes at least one image acquisition unit (1), a ranging unit (2), and an environmental physical quantity sensing unit (3). The data fusion processing unit (4) has its input terminals electrically connected to the image acquisition unit (1), the ranging unit (2) and the environmental physical quantity sensing unit (3), respectively, and is used to receive and perform spatiotemporal synchronization, feature-level fusion and heterogeneous data alignment of multi-source data; The communication and interface unit (5) is electrically connected to the output of the data fusion processing unit (4) and is used to send the fused global perception data to the UAV flight control system or ground station. The power management unit (6) is electrically connected to the sensing layer component, the data fusion processing unit (4) and the communication and interface unit (5) respectively, and is used to provide a stable and isolated power supply; The housing (7) contains and fixes the above-mentioned units, and is provided with a mechanical interface for docking with the UAV fuselage.

2. The UAV AIoT global perception sensing module based on multi-source heterogeneous fusion according to claim 1, characterized in that: The image acquisition unit (1) includes at least two of the following: a visible light camera module (11), an infrared thermal imaging module (12), and a low-light night vision module (13); the ranging unit (2) is a solid-state lidar or a millimeter-wave radar; the environmental physical quantity sensing unit (3) includes a temperature and humidity sensor (31), a barometric pressure sensor (32), and a multispectral sensor (33). This multimodal configuration enables all-weather, multi-spectral environmental information capture, significantly improving the perception robustness in complex scenarios.

3. The UAV AIoT global perception sensing module based on multi-source heterogeneous fusion according to claim 1, characterized in that: The core of the data fusion processing unit (4) is an embedded heterogeneous computing chip, including an FPGA logic gate array (41) and an ARM core processor (42). The FPGA is used to realize the pixel-level real-time correction of the image acquisition unit (1) and the point cloud preprocessing of the ranging unit (2). The ARM core processor (42) runs a lightweight deep learning model and performs target recognition and semantic segmentation. This hardware heterogeneous architecture combines the low-latency parallel processing capability of the FPGA with the complex logic control advantages of the ARM, and completes multi-source data fusion in milliseconds.

4. The UAV AIoT global perception sensing module based on multi-source heterogeneous fusion according to claim 1, characterized in that: The communication and interface unit (5) includes a CAN bus interface (51), a UART serial port (52), and a wireless data transmission module (53). The wireless data transmission module (53) supports 4G / 5G or LoRa protocols and is used to transmit key perception data when the UAV is out of line of sight. This redundant communication design ensures highly reliable internal interaction with the flight control system and long-distance data backup links with the ground station.

5. The UAV AIoT global perception sensing module based on multi-source heterogeneous fusion according to claim 1, characterized in that: The power management unit (6) includes an EMI filter circuit (61), an isolated DC-DC converter (62), and a multi-channel LDO regulator (63). The isolated DC-DC converter (62) provides independent power to the image acquisition unit (1), the ranging unit (2), and the data fusion processing unit (4). This isolated power supply architecture effectively suppresses power noise introduced by high-power devices such as motors, and ensures the signal-to-noise ratio of analog sensors (such as the ranging unit).

6. The UAV AIoT global perception sensing module based on multi-source heterogeneous fusion according to claim 1, characterized in that: The housing (7) is made of aviation aluminum alloy or carbon fiber composite material. It has heat dissipation fins (71) on its bottom and sides, and the interior of the housing (7) is filled with thermally conductive potting compound. This structural design improves the heat dissipation capacity and environmental adaptability of the module while reducing weight, and is suitable for the thermal management requirements of high-intensity flight.

7. The UAV AIoT global perception sensing module based on multi-source heterogeneous fusion according to claim 1, characterized in that: It also includes an inertial measurement unit (8) electrically connected to the data fusion processing unit (4) for acquiring the module’s own angular velocity and acceleration data; the data fusion processing unit (4) uses the data from the inertial measurement unit (8) to perform motion compensation on the sensing data of the image acquisition unit (1) and the ranging unit (2). This design effectively eliminates the distortion effect of UAV flight vibration and attitude change on sensor data.

8. The UAV AIoT global perception sensing module based on multi-source heterogeneous fusion according to claim 1, characterized in that: The data fusion processing unit (4) runs a multi-source sensor spatiotemporal synchronization algorithm based on extended Kalman filtering or graph optimization. It performs pixel-by-pixel registration of the two-dimensional pixel coordinates of the image acquisition unit (1) and the three-dimensional point cloud coordinates of the ranging unit (2) to generate an RGB-D data stream that integrates color information and depth information. This feature-level fusion directly provides a dense environmental depth map for UAV obstacle avoidance and autonomous navigation.

9. The UAV AIoT global perception sensing module based on multi-source heterogeneous fusion according to claim 1, characterized in that: The data fusion processing unit (4) is embedded with a lightweight neural network model based on knowledge distillation. It performs semantic-level fusion of data from the image acquisition unit (1) and the multispectral sensor (33) locally on the module and outputs a global semantic map containing obstacle categories (such as trees, wires, and buildings) and vegetation health labels. This edge computing capability greatly reduces the computational load of the UAV flight control system and reduces the data transmission bandwidth pressure.

10. The UAV AIoT global perception sensing module based on multi-source heterogeneous fusion according to claim 1, characterized in that: The outer shell (7) is provided with a quick-release dust and rain cover (72). The quick-release dust and rain cover (72) is embedded with optical coated glass at each sensor window. This design enables the module to maintain its sensing function under adverse weather conditions such as rain, fog, and dust, thus expanding the range of environments in which the UAV can operate.

Images