Unmanned aerial vehicle indoor three-dimensional reconstruction device and system fusing multi-modal data

Abstract

The utility model discloses an unmanned aerial vehicle indoor three-dimensional reconstruction device and system fusing multi-modal data, the device comprises a fuselage body, and a multi-modal data acquisition module, a main control module and a flight control module which are arranged on the fuselage body, the multi-modal data acquisition module comprises a laser radar, a binocular vision camera and an event camera, the laser radar, the binocular vision camera, the event camera and the flight control module are respectively connected with the main control module, and the main control module is used for realizing indoor scene three-dimensional reconstruction based on indoor scene multi-modal data acquired by the multi-modal data acquisition module and unmanned aerial vehicle pose data provided by the flight control module. The indoor scene three-dimensional reconstruction system is simple and convenient to operate and low in cost, can realize high-precision data acquisition and fusion, and can effectively improve the precision and efficiency of indoor scene three-dimensional reconstruction.

Description

Technical Field

[0001] The utility model relates to the technical field of unmanned aerial vehicle (UAV) three-dimensional reconstruction, and in particular to a device and system for indoor three-dimensional reconstruction of UAVs by fusing multimodal data. Background Art

[0002] With rapid advances in computer vision, sensor technology, and artificial intelligence, drone technology has demonstrated unprecedented potential in numerous fields. This is particularly true for 3D reconstruction of indoor environments. With the growing demand for high-precision indoor 3D models in industries like architecture, interior design, and virtual reality, drones, with their flexibility and efficiency, have become a crucial tool for rapid and accurate spatial mapping.

[0003] However, the complexity and diversity of indoor environments present numerous challenges. Compared to outdoor environments, indoor spaces often suffer from poor lighting, complex structures, and numerous obstructions. These factors significantly impact the quality of data collected by drone-mounted visual sensors. Furthermore, data from a single sensor often fails to fully and accurately capture the three-dimensional information of an indoor space, limiting the accuracy and completeness of the reconstructed model. Therefore, designing a drone device that can efficiently fuse multimodal data to achieve high-precision three-dimensional reconstruction of indoor scenes has become a pressing technological need. Utility Model Content

[0004] This utility model aims to address, at least to some extent, one of the technical problems in the related art. To this end, the first objective of this utility model is to provide a drone-based indoor 3D reconstruction device that fuses multimodal data. This device integrates a laser radar, a binocular vision camera, and an event camera, achieving high-precision data acquisition and fusion. It is easy to operate and low-cost, effectively improving the accuracy and efficiency of 3D reconstruction of indoor scenes.

[0005] The second object of the present invention is to provide a UAV indoor three-dimensional reconstruction system that integrates multimodal data.

[0006] In order to achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions:

[0007] A device for indoor 3D reconstruction of unmanned aerial vehicles (UAVs) by fusing multimodal data, comprising:

[0008] A fuselage body and a multimodal data acquisition module, a main control module and a flight control module arranged on the fuselage body, wherein the multimodal data acquisition module includes a laser radar, a binocular vision camera and an event camera, and the laser radar, binocular vision camera, event camera and flight control module are respectively connected to the main control module, wherein the main control module is used to realize three-dimensional reconstruction of indoor scenes based on the multimodal data of indoor scenes collected by the multimodal data acquisition module and the drone posture data provided by the flight control module.

[0009] Preferably, the fuselage body includes a UAV frame, a bottom fixed platform, a top fixed platform and a plurality of isolation columns. The UAV frame is a planar structure formed by staggered connection of a plurality of first connecting rods. The bottom fixed platform is arranged on the lower surface of the UAV frame, and the top fixed platform is arranged on the upper surface of the UAV frame. A plurality of isolation columns penetrate the bottom fixed platform and the top fixed platform to achieve fixation of the bottom fixed platform and the top fixed platform.

[0010] Preferably, the extension sections of the plurality of isolation columns extending from above the top fixed platform form a cavity, and the main control module is fixed to the upper surface of the top fixed platform and confined within the cavity.

[0011] Preferably, the laser radar is arranged on the upper surface of the main control module and is confined within the cavity.

[0012] Preferably, a bracket is provided on one side of the top fixed platform, and the event camera is mounted on a side wall of the bracket.

[0013] Preferably, the binocular vision camera is arranged on the lower surface of the drone frame on a side close to the event camera.

[0014] Preferably, the flight control module is arranged on the upper surface of the bottom fixed platform, the device also includes an electronic speed regulator, the electronic speed regulator is arranged between the bottom fixed platform and the flight control module, and a battery is fixed on the lower surface of the bottom fixed platform.

[0015] Preferably, rubber pads are provided on the installation sides of the laser radar, binocular vision camera and event camera.

[0016] Preferably, the ends of the extension sections of the multiple isolation columns extending from above the top fixed platform are connected by multiple second connecting rods to form a fixed frame, and the device also includes an RTK antenna, which is arranged on the fixed frame and connected to the main control module.

[0017] To achieve the above objectives, the second aspect of the present invention provides a UAV indoor 3D reconstruction system integrating multimodal data, comprising:

[0018] Ground station host computer;

[0019] Ground receiving device; and

[0020] The above-mentioned UAV indoor 3D reconstruction device integrating multimodal data; the ground station host computer and the ground receiving device are respectively connected to the main control module in the UAV indoor 3D reconstruction device.

[0021] The utility model has at least the following technical effects:

[0022] The utility model provides a drone indoor 3D reconstruction device and system that fuses multimodal data. The drone indoor 3D reconstruction device has a sophisticated design, convenient operation, and high cost-effectiveness. By integrating lidar, binocular vision sensors and event cameras, it can realize multimodal data fusion, efficient data processing, and can generate high-precision indoor 3D reconstruction models in real time.

[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will become apparent from the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of the structure of a UAV indoor 3D reconstruction device that integrates multimodal data according to an embodiment of the present invention.

[0025] Figure 2 This is a diagram showing the working principle of the UAV indoor 3D reconstruction system integrating multimodal data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present embodiment is described in detail below. Examples of the embodiment are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0027] The following describes the apparatus and system for indoor 3D reconstruction of a drone by fusing multimodal data according to this embodiment with reference to the accompanying drawings.

[0028] Figure 1 This is a schematic diagram of the structure of the UAV indoor 3D reconstruction device that integrates multimodal data according to an embodiment of the present invention. Figure 1As shown, the UAV indoor 3D reconstruction device integrating multimodal data includes a fuselage body and a multimodal data acquisition module, a main control module 7 and a flight control module 2 arranged on the fuselage body. The multimodal data acquisition module includes a laser radar 1, a binocular vision camera 3 and an event camera 6. The laser radar 1, the binocular vision camera 3, the event camera 6 and the flight control module 2 are respectively connected to the main control module 7, wherein the main control module 7 is used to realize 3D reconstruction of the indoor scene based on the multimodal data of the indoor scene collected by the multimodal data acquisition module and the UAV posture data provided by the flight control module 2.

[0029] In this embodiment, the laser radar 1, binocular vision camera 3 and event camera 6 can be used to obtain multimodal data of indoor scenes, and then the main control module 7 performs data processing, feature matching and pose estimation based on the obtained multimodal data of indoor scenes, thereby realizing high-precision mapping of the three-dimensional structured model of the indoor scene.

[0030] Please continue to refer to Figure 1 The fuselage body includes a drone frame 8, a bottom fixed platform 9, a top fixed platform 10 and a plurality of isolation columns 12. The drone frame 8 is a four-rotor drone frame, and the isolation columns 12 are double-pass circular isolation columns. The drone frame 8 is a planar structure formed by staggered connection of multiple first connecting rods. The bottom fixed platform 9 is arranged on the lower surface of the drone frame 8, and the top fixed platform 10 is arranged on the upper surface of the drone frame 8. Multiple isolation columns 12 penetrate the bottom fixed platform 9 and the top fixed platform 10 to achieve the fixation of the bottom fixed platform 9 and the top fixed platform 10.

[0031] In this embodiment, the fuselage body is made of lightweight high-strength carbon fiber material, which has good wind resistance and collision resistance. Reasonable wiring grooves and fixing components are provided inside the fuselage body to ensure stable connection and assembly of various components.

[0032] Furthermore, the extension of multiple isolation columns 12 from the top fixed platform 10 forms a cavity, within which a main control module 7 (model RK3588 main control core board) is bolted. Specifically, main control module 7 is fixed to the upper surface of the top fixed platform 10 and confined within this cavity. Main control module 7 is primarily responsible for processing and fusing multimodal data of the indoor scene and performing 3D reconstruction.

[0033] In this embodiment, the main control module 7 uses the highly integrated RK3588 main control core board, effectively controlling the size and weight of the drone. Furthermore, after the multimodal data of the indoor scene is transmitted to the main control module 7, it can be transmitted via the AXI bus (a bus protocol) within the main control module 7, and the dual-cluster core can be used for attitude estimation and 3D mapping.

[0034] Further, such as Figure 1 As shown, the laser radar 1 is mounted on the upper surface of the main control module 7 and confined within the cavity. The laser radar 1 uses a LIVOX MID360 hybrid solid-state laser radar, secured with M3 screws through the fixture and four 5mm-deep M3 mounting holes. Additionally, the laser radar 1 is secured through a central positioning hole to improve overall positioning accuracy.

[0035] In this embodiment, the laser radar 1 uses a one-to-three-wire aviation plug to connect to the main control module 7 and the DC regulated power supply. The first end of the one-to-three-wire aviation plug is connected to the M12 aviation plug (male) of the laser radar 1, the second end is connected to the DC regulated power supply, and the third end is connected to the main control module 7. The laser radar 1 is used to scan and obtain indoor scene point cloud data and the drone's own posture data, namely IMU (inertial measurement unit) data. It can send indoor scene point cloud and IMU data to the main control module 7 via an Ethernet cable with an RJ45 connector using UDP (a data transmission protocol) protocol.

[0036] Further, such as Figure 1 As shown, a bracket (not shown) is provided on one side of the top fixing platform 10. The event camera 6 is mounted on the side wall of the bracket using four 2.5 cm screws. In this embodiment, the event camera 6 is a DAVIS346 model and has a USB 3.0 connector secured by two bolts on the left and right. The event camera 6 can transmit the collected event stream data to the main control module 7 via a USB 3.0 cable with a Micro B connector.

[0037] The event camera 6 in this embodiment is different from the mode of traditional cameras that acquire images at a certain frequency. The event camera 6 is used to asynchronously observe the real-time brightness changes of each pixel and output event information when the brightness change exceeds a certain threshold. The data format it collects is an event stream consisting of time, pixel coordinates and polarity. It has the characteristics of high dynamic range, low latency and low power consumption, and performs well in processing fast-moving objects and extreme lighting scenes.

[0038] Further, such as Figure 1 As shown, a binocular vision camera 3 is symmetrically mounted on the lower surface of the drone frame 8 near the event camera 6. The binocular vision camera 3 is primarily used to collect image data of indoor scenes. This mounting method for each module effectively prevents accidental collision damage to the mounted equipment, which includes the lidar 1, binocular vision camera 3, and event camera 6.

[0039] It should be noted that rubber pads are also installed on the mounting sides of the LiDAR 1, binocular vision camera 3, and event camera 6. For example, the contact surfaces between the LiDAR 1 and the main control module 7, the main control module 7 and the top fixed platform 10, the binocular vision camera 3 and the drone frame 8, and the event camera 6 and the bracket are all equipped with shock-absorbing materials similar to rubber pads. The use of shock-absorbing materials can reduce the impact of vibrations on the mounted equipment during flight.

[0040] Further, such as Figure 1 As shown, the upper surface of the bottom fixed platform 9 is provided with a flight control module 2. The flight control module 2 includes a flight controller, an IMU sensor, and a compass. The flight controller is connected to the main control module 7. The flight control module 2 can obtain the UAV's position information, such as speed and direction, through the built-in IMU sensor and compass. The acquired position information is sent to the main control module 7 via the flight controller for data preprocessing in the main control module 7.

[0041] Further, such as Figure 1 As shown, an electronic speed regulator 11 is also provided between the bottom fixing platform 9 and the flight control module 2. A lithium battery 4 is also fixed to the lower surface of the bottom fixing platform 9 by means of a binding strap, which facilitates battery replacement. In this embodiment, the electronic speed regulator 11 is a key component in the UAV power system. It is mainly responsible for converting the DC power provided by the power supply into AC power and accurately controlling the speed of the motor, thereby achieving precise control of the UAV's flight state. In this embodiment, the flight control module 2 and the electronic speed regulator 11 constitute a flight control system, wherein the flight control system also includes a remote control and ground station software for achieving active flight control of the UAV.

[0042] The above-mentioned mounting equipment also includes an RTK (antenna for implementing real-time dynamic positioning technology) antenna 5. Above the laser radar 1 is the RTK antenna 5, which can receive GPS (Global Positioning System) satellite signals and provide basic data for positioning. Specifically, the ends of the extension sections of multiple isolation columns 12 extending from above the top fixed platform 10 are connected by multiple second connecting rods to form a fixed frame, and the RTK antenna 5 is arranged on the fixed frame and connected to the main control module 7. The above-mentioned mounting equipment is installed in the above-mentioned spatial position in a manner that the center of gravity is horizontally concentrated, which can reduce the interference and shaking caused by the center of gravity when the drone is self-balancing.

[0043] Furthermore, the present invention provides a UAV indoor 3D reconstruction system that integrates multimodal data. The system includes a ground station host computer, a ground receiving device, and the aforementioned UAV indoor 3D reconstruction system that integrates multimodal data. The ground station host computer and the ground receiving device are each connected to a main control module 7 in the UAV indoor 3D reconstruction system.

[0044] Figure 2 This is a diagram showing the working principle of the UAV indoor 3D reconstruction system integrating multimodal data according to an embodiment of the present invention.

[0045] like Figure 2 As shown, the LiDAR 1 scans and acquires indoor scene point cloud data and the drone's own posture data, namely IMU (Inertial Measurement Unit) data. It then sends the indoor scene point cloud and IMU data to the main control module 7 via an Ethernet cable using the UDP protocol. The flight control module 2 further acquires the drone's posture data through the IMU sensor and compass, and sends it to the main control module 7 via the flight controller. Simultaneously, the RTK antenna 5 sends real-time drone position data to the main control module 7. After receiving the IMU data sent by the LiDAR 1, the drone's posture data sent by the flight control module 2, and the drone's position data sent by the RTK antenna 5, the main control module 7 performs data preprocessing operations, then estimates the drone's posture and motion state, and sends the drone's posture estimation data to the ground station host computer via WIFI (Wireless Fidelity) communication, so that the ground station host computer can view the drone's real-time dynamics and basic parameters.

[0046] At the same time, the main control module 7 can also obtain the indoor scene point cloud data sent by the lidar 1, the indoor scene image data sent by the binocular vision camera 3, and the event stream data sent by the event camera 6 through Ethernet and serial ports, and reconstruct the indoor three-dimensional scene map based on these indoor scene multimodal data and the pre-processed drone attitude estimation data.

[0047] Specifically, when the device of the present invention is in use, the power switch is first turned on, and the UAV starts the power-on self-test. After the power-on self-test is completed, the UAV enters the initialization program. The flight control module 2 will display the current status through sound and light prompts. If a fault occurs, a red light will light up to alarm and wait for manual elimination of the cause of the fault. At the same time, the main control module 7 will automatically run the program after power-on, and start the relevant nodes to realize MAVLINK (a communication protocol) communication with the flight control module 2. The multimodal data acquisition module enters the initialization program and transmits data to the main control module 7 in the form of UDP broadcast. After the multimodal data acquisition module completes the initialization, the UAV will complete the data acquisition, processing and three-dimensional reconstruction tasks according to the program. The data processing and reconstruction results of the main control module 7 will be transmitted back to the ground receiving device via WIFI, so that the reconstruction and mapping of the three-dimensional indoor scene can be realized.

[0048] In order to make the three-dimensional reconstruction solution clearer, the principles of this part are described in detail below. However, it should be noted that the algorithms and methods involved are not the improvements of this application. The improvements of this application are the above-mentioned structural parts.

[0049] Specifically, before the system is run, the event camera, binocular vision camera and lidar will be manually calibrated for external parameters to ensure that the data space of each sensor is aligned. The calibrated external parameter matrix (rotation matrix and displacement vector) will be used in subsequent data processing. After the data collected by multiple sensors are transmitted to the main control module through the serial port in real time, since the event stream is continuous and the images taken by the binocular vision camera are discrete, data alignment is required first. The time window is divided according to equal event intervals (usually 30Hz), and the timestamps of all event points in the time window are aligned to the moment at the end of the window, and compressed into an event frame, which has the same two-dimensional data structure as the image frame. The two types of data are simultaneously input into the FE-Net neural network built based on the indoor scene dataset for fusion processing and feature extraction to obtain a visual feature image that highlights edges, corners or other significant features; then, the feature information extracted in the image is used to narrow down the points. The feature search range in the cloud is determined, and geometric feature points are extracted from the laser point cloud data. ISS (image descriptor) feature points are used to describe its local geometric structure. Furthermore, matching feature search is performed based on the feature description of the image and point cloud. The RANSAC (image stitching and feature point matching algorithm) geometric constraint algorithm is used to eliminate false matches, and the features in the laser point cloud data are verified using image color and texture features. Finally, a high-precision dense 3D point cloud model is output. Afterwards, the relative position of each frame is calculated according to the drone pose estimation information, and combined with the real-time feature matching results, a smooth connection between the reconstruction results of different frames is achieved to generate a coherent 3D model.

[0050] In summary, the utility model provides a drone indoor 3D reconstruction device and system that integrates multimodal data. The drone indoor 3D reconstruction device has a sophisticated design, convenient operation, and high cost-effectiveness. By integrating lidar, binocular vision sensors and event cameras, it can realize multimodal data fusion, efficient data processing, and can generate high-precision indoor 3D reconstruction models in real time.

[0051] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0052] Although the present invention has been described in detail through the above preferred embodiments, it should be understood that the above description should not be considered as limiting the present invention. After reading the above description, various modifications and alternatives to the present invention will be readily apparent to those skilled in the art. Therefore, the scope of protection of the present invention shall be defined by the appended claims.

Claims

1. A UAV indoor 3D reconstruction device integrating multimodal data, characterized in that: include: A fuselage body and a multimodal data acquisition module, a main control module and a flight control module arranged on the fuselage body, wherein the multimodal data acquisition module includes a laser radar, a binocular vision camera and an event camera, and the laser radar, binocular vision camera, event camera and flight control module are respectively connected to the main control module, wherein the main control module is used to realize three-dimensional reconstruction of indoor scenes based on the multimodal data of indoor scenes collected by the multimodal data acquisition module and the drone posture data provided by the flight control module.

2. The device for indoor 3D reconstruction of an unmanned aerial vehicle by fusing multimodal data according to claim 1, wherein: The fuselage body includes a drone frame, a bottom fixed platform, a top fixed platform and a plurality of isolation columns. The drone frame is a planar structure formed by staggered connection of a plurality of first connecting rods. The bottom fixed platform is arranged on the lower surface of the drone frame, and the top fixed platform is arranged on the upper surface of the drone frame. A plurality of isolation columns penetrate the bottom fixed platform and the top fixed platform to achieve the fixation of the bottom fixed platform and the top fixed platform.

3. The device for indoor 3D reconstruction of unmanned aerial vehicles by fusing multimodal data according to claim 2, wherein: A cavity is formed by extending sections of a plurality of isolation columns extending from above the top fixing platform. The main control module is fixed on the upper surface of the top fixing platform and is confined within the cavity.

4. The device for indoor 3D reconstruction of unmanned aerial vehicles by fusing multimodal data according to claim 3, characterized in that: The laser radar is arranged on the upper surface of the main control module and is confined within the cavity.

5. The device for indoor 3D reconstruction of unmanned aerial vehicles by fusing multimodal data according to claim 2, wherein: A bracket is provided on one side of the top fixed platform, and the event camera is mounted on the side wall of the bracket.

6. The device for indoor 3D reconstruction of an unmanned aerial vehicle by fusing multimodal data according to claim 5, characterized in that: The binocular vision camera is arranged on the lower surface of the drone frame on a side close to the event camera.

7. The device for indoor 3D reconstruction of an unmanned aerial vehicle by fusing multimodal data according to claim 2, wherein: The flight control module is arranged on the upper surface of the bottom fixed platform. The device also includes an electronic speed regulator, which is arranged between the bottom fixed platform and the flight control module. A battery is fixed on the lower surface of the bottom fixed platform.

8. The device for indoor 3D reconstruction of an unmanned aerial vehicle by fusing multimodal data according to claim 1, wherein: The installation sides of the laser radar, binocular vision camera and event camera are provided with rubber pads.

9. The device for indoor 3D reconstruction of an unmanned aerial vehicle by fusing multimodal data according to claim 3, wherein: The ends of the extension sections of the multiple isolation columns extending from above the top fixed platform are connected through multiple second connecting rods to form a fixing frame. The device also includes an RTK antenna, which is arranged on the fixing frame and connected to the main control module.

10. An indoor 3D reconstruction system for unmanned aerial vehicles integrating multimodal data, characterized in that: include: Ground station host computer; Ground receiving device; as well as The UAV indoor 3D reconstruction device for fusing multimodal data according to any one of claims 1 to 9; the ground station host computer and the ground receiving device are respectively connected to the main control module in the UAV indoor 3D reconstruction device.

Images