Light field reconstruction method based on unmanned aerial vehicle system
Through the drone system equipped with high-definition cameras, combined with automatic control technology and ROS system, high-resolution light field image acquisition and stitching are achieved, solving the resolution loss and scene limitation problems in existing light field acquisition technologies. It provides flexible and multi-scenario light field information acquisition capabilities.
Patent Information
- Application Number
- CN202411079436.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-03
AI Technical Summary
The existing light field acquisition technology has problems such as resolution loss, high cost, difficulty in hardware synchronization and limited scenes, especially the flexibility and mobility of the light field acquisition scheme based on microlens arrays and camera arrays.
Light field reconstruction is carried out using a drone system equipped with a high-definition camera. Through the single drone nine-grid running method, ring running method or array acquisition technology, combined with automatic control technology and ROS system, high-resolution light field image acquisition and stitching , use the ground control station to perform data processing.
It realizes flexible light field information acquisition in multiple scenes without sacrificing resolution, with the advantages of large baseline, high flexibility, and no loss of viewpoint resolution, and is suitable for aerial and near-earth scenarios.
Smart Images

Figure CN119068111A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned aerial vehicles and light field reconstruction, and in particular to a light field reconstruction method based on an unmanned aerial vehicle system. The present invention adopts automatic control technology to realize image acquisition, temporarily stores data through a mobile carrier or transmits the data to a ground end for processing in real time, and realizes light field reconstruction using image data. Background Art
[0002] The concept of light field proposed by Alexander Gershun is relative to the electromagnetic field, and it is believed that light radiation can be expressed as three-dimensional space. Later, Edward H. Adelson used a full light function to represent the light field. The full light function is a 7-dimensional mathematical model, including the position coordinates of the three-dimensional space, the angle at which the light enters the human eye, the wavelength of the light, and the moment when the light enters the human eye. Now, the academic community generally adopts a light field 4D model proposed by Marc Levoy. At present, light field acquisition schemes are mainly divided into light field acquisition schemes based on microlens arrays and light field acquisition schemes based on camera arrays. For example, the light field camera based on microlens arrays of Lytro in the United States focuses the light emitted by the object on the microlens array plane through the main lens, and then uses the microlens array to separate the light and record it with a sensor, so that the direction and intensity of the light are recorded at the same time. The light field acquisition based on the camera array is essentially a lot of sensors arranged together to obtain light at the same time, and each sensor is a viewpoint. Generally speaking, the light field acquisition scheme based on the microlens array is small in size and low in cost, but due to the limitation of lens accuracy, the acquisition method based on the microlens array will cause serious resolution loss. The camera array-based light field acquisition solution has the advantages of a large baseline and no loss of viewpoint resolution, but it is expensive and has the disadvantages of difficult hardware synchronization. In addition, the camera array is large and bulky, inconvenient to move, and its application scenarios are very limited. There are also some fusion methods, such as the mask-based light field acquisition solution. Mask-based light field acquisition removes the redundancy of the light field by learning the light field. Specifically, a translucent coding mask is used to reconstruct from a single image using a learnable light field dictionary, so as to collect less data to reconstruct the complete light field. However, the light field quality of mask-based light field acquisition is poor and the signal-to-noise ratio is low.
[0003] In recent years, drones have appeared frequently and have been widely used in the fields of agricultural plant protection, power inspection, surveying and mapping, aerial photography, and rescue. Thanks to the development of sensors and the maturity of drone technology, drones used in surveying and mapping have the advantages of flexibility, high speed, precision, accuracy, and low cost. In actual tasks, drones generally fly according to preset waypoints. The sensors complete the data collection and hand it over to the ground station for processing to obtain surveying and mapping data, which can also be used for three-dimensional reconstruction. Three-dimensional reconstruction is of great use in scenes such as cultural relics protection and battlefield analysis.
[0004] In the "A method for three-dimensional reconstruction of light fields based on a light field microscopy system" disclosed in the Chinese invention patent CN113850902A, the system sequentially arranges microscopic specimens, microscopes, microlens arrays, and cameras according to the propagation path of light, and determines the movement interval of the camera and the speed of the mobile platform according to the needs, thereby determining the scanning strategy on the camera side. Then, a series of preprocessing is performed on the collected images, and the objective function for three-dimensional reconstruction of multiple light field images is obtained according to the characteristics of shot noise in the microscope imaging process. Finally, the three-dimensional reconstruction of the target object is completed. This scheme uses a microlens array to collect light field information, and the microlens array sacrifices its spatial resolution for more angular resolution, which will cause resolution loss of each sub-aperture image in the light field image. In addition, the light field microscopy system uses a fixed platform to hold the camera for image acquisition, and its use scenario is limited. The platform cannot be moved freely and the objects that can be collected are also limited. Summary of the invention
[0005] In order to solve at least one of the problems existing in the prior art, the present invention provides a light field reconstruction method based on a drone system, which combines the currently mature drone technology with light field image acquisition, and uses a drone equipped with a high-definition camera to imitate the light field acquisition of a camera array. It can perform flexible and multi-scene light field information acquisition without sacrificing resolution, and has the advantages of a large baseline, high flexibility, and no loss of viewpoint resolution, and can be used for aerial and near-ground scenes.
[0006] In order to achieve the purpose of the present invention, the light field reconstruction method based on the drone system provided by the present invention comprises the following steps:
[0007] Controlling a drone equipped with a high-definition camera to approach a light field reconstruction target and collect light field images by running a light field image collection method to collect multiple images, wherein the running light field image collection method is any one of a single drone nine-square running method, a single drone circular running method, and a drone array collection;
[0008] Transmit the images collected by the drone from the onboard computer to the ground control station;
[0009] The ground control station pre-processes the multiple images sent back;
[0010] The processed images are stitched together to synthesize a multi-sub-aperture light field image.
[0011] Furthermore, the ground control station uses a computer and a 3DR data transmission radio to communicate with the UAV flight control to obtain the UAV flight status data. The computer of the ground control station is connected to the onboard computer of the UAV so that the ground control station can send instructions to the onboard computer. After receiving the instructions, the onboard computer controls the UAV to collect images.
[0012] Furthermore, the computer of the ground control station is connected to the onboard computer of the UAV via a local area network or a hotspot.
[0013] Furthermore, before running the UAV for image acquisition, the UAV control algorithm is verified in a simulation environment.
[0014] Furthermore, when operating the drone to collect images, first use the remote control to control the drone to fly near the light field reconstruction target, adjust the drone's position to a preset distance from the light field reconstruction target, and make the drone carry a high-definition camera facing the light field reconstruction target, start the drone HOLD mode to stabilize the drone's position, and then perform light field collection by collecting light field images.
[0015] Furthermore, the preprocessing includes but is not limited to any one or more of removing motion blur, color calibration, and translation.
[0016] Furthermore, the synthesized multi-subaperture light field image is stored in a ground control station.
[0017] Furthermore, the single drone nine-square grid running method is to use the drone's current position as the coordinate origin and the center of the nine-square grid, set the drone's nine-square grid waypoints, and set the spacing between adjacent waypoints in the nine-square grid and the distance between the drone and the light field reconstruction target. The drone hovers at each waypoint in sequence and takes images of the light field reconstruction target.
[0018] Furthermore, the single-UAV circular running acquisition method is that the UAV takes a preset distance above the light field reconstruction target as the initial point, and acquires images at the initial point, then flies in a circle with the initial point as the center and the preset distance as the radius, and takes multiple groups of images of the light field reconstruction target at intervals during the circular flight.
[0019] Furthermore, the drone array acquisition is to navigate all drones in the drone array to different positions in front of the light field reconstruction target, and the high-definition images carried by all drones shoot the light field reconstruction target.
[0020] Furthermore, the UAV system includes basic components: a solid frame, a stable power system, a safe control system and the ability to carry a high-definition camera. The power system includes propellers, electric speed controllers, motors and batteries, and the components of the power system are compatible with each other to ensure safe and stable power supply. The control system includes flight control, onboard computer and ground control station. The flight control interface is sufficient and compatible to ensure the connection of components including data transmission, image transmission, GPS, optical flow meter, etc., to ensure that accurate UAV flight status information can be obtained. The communication between the onboard computer and the ground control station is required to be stable and fast.
[0021] Furthermore, the drone data collection method is to send instructions to the drone by the onboard computer, use Mavros to open up communication between the drone and the computer, and use the ROS system to control the drone.
[0022] Furthermore, the drone can be navigated to the vicinity of the destination and then manually adjusted to the appropriate position by the pilot.
[0023] Furthermore, the above image acquisition methods are all based on the offboard mode of the drone, using the ROS system to send instructions to the drone to realize automatic image acquisition and control of the drone.
[0024] Compared with the prior art, the present invention can at least achieve the following beneficial effects:
[0025] 1. Use a drone equipped with a high-definition camera to collect light fields. The resolution of each sub-aperture image of the stitched and synthesized light field image is the resolution supported by the camera. In addition, the drone has a wide range of movement when collecting images, and has the advantages of a large baseline and no loss of resolution.
[0026] 2. Using drones equipped with ordinary cameras to collect light fields is highly flexible and can be performed in a variety of scenarios without being restricted by the scene. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 Schematic diagram of light field image collection by a single drone using the nine-square running method in an embodiment of the present invention.
[0028] Figure 2 This is a schematic diagram of collecting light field images by a single drone running in a circle in an embodiment of the present invention.
[0029] Figure 3 Schematic diagram of light field image acquisition by a drone array in an embodiment of the present invention.
[0030] Figure 4 This is a schematic diagram of the overall framework of the drone.
[0031] Figure 5 It is a schematic diagram of a specific process in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] The light field reconstruction method based on the UAV system provided by the present invention comprises the following steps:
[0034] Step S1: Select a drone.
[0035] The drone used can be built by yourself, or you can directly purchase an existing commercial drone.
[0036] In some embodiments of the present invention, the drone system is composed of an F450 frame, four 2216 self-locking motors, four 20A ESCs, two pairs of 9450 self-locking propellers, a 4S 3300mah battery, an open source flight controller pixhawk 6C, a pair of 3DR data transmission radios, a sports camera (the camera focal length is 35-50mm, which is suitable for most scenarios. The camera focal length used in this example is 50mm), a binocular camera (for the perception system to ensure the flight obstacle avoidance function) and a digital image transmission. The drone architecture is as follows: Figure 4 It is understandable that the frame, motor, ESC, propeller, battery, flight control, data transmission radio, sports camera, and digital image transmission are not the only ones. They can ensure reliable and safe flight, autonomous flight of the drone, and collecting pictures and transmitting data to the ground station.
[0037] In some implementations of the present invention, a mature commercial drone is directly purchased, and the present invention does not involve improvements to the drone.
[0038] Step S2: After completing the preparations for the drone to take off, debug the stable drone platform.
[0039] Download QGroundControl software on the laptop, connect the flight control to the computer using the connection cable provided by the flight control, burn the px4 firmware to the flight control, and set the flight control parameters after the firmware is successfully burned, including the calibration of the flight control sensor, the parameter setting of the GPS, the parameter setting of the optical flow sensor, and the control calibration of the remote control. After ensuring that all parameter settings are correct, you can conduct a test flight of the drone.
[0040] After the test flight is completed, the onboard computer can be set up. In some embodiments of the present invention, Intel's NUC11 is used as the onboard computer. The steps for configuring the onboard computer environment are as follows: First, install the Ubuntu system for the onboard computer. After completing the installation of Ubuntu, download and install the corresponding version of the ROS system for the onboard computer. The Ubuntu used in the example is version 20.04, and the ROS version is noetic. In addition, install the mavros toolkit for communication between the onboard computer and the flight control. After the installation is completed, test whether the onboard computer and the drone communicate successfully. Download the nomachine remote access software on both the onboard computer and the laptop. In the same local area network, the IP address can be used to realize the remote connection of the ground-side laptop to the onboard computer for subsequent control.
[0041] The ground control station uses a laptop computer and a 3DR data radio to communicate with the UAV flight control to obtain the UAV flight status data. The laptop computer of the ground control station is connected to the UAV's onboard computer through a local area network or hotspot. The ground control station sends instructions to the onboard computer, and the onboard computer receives the instructions to control the UAV to collect pictures.
[0042] Step S3: Verify the drone control algorithm on the gazebo simulation platform and build the environment model and drone model.
[0043] The drone platform is equipped with valuable equipment such as cameras and onboard computers, and accidents during actual flight may cause significant losses. Therefore, in some embodiments of the present invention, in order to reduce risks, gazebo and px4 are first used for joint simulation to experiment with the drone's collection task in a simulation environment. In the experiment, the gazebo environment is set to a breeze environment, and a cubic wooden box with a length, width, and height of 1m (in other embodiments, other values can also be used) is set as the collection object. The simulated drone carries a binocular camera for sensing the environment, and carries a high-definition camera to return the captured image. At the beginning of the experiment, the drone control script is first used to simulate the real pilot to operate the drone, and the drone is remotely controlled to fly near the collection object, and the camera carried by the drone is slightly adjusted to face the shooting object, and the drone HOLD mode is turned on to stabilize the drone position. Then the mavros toolkit is used to connect the computer to the drone communication, receive the depth information of the shooting object obtained by the binocular camera, set the radius of the drone's automatic collection path according to the depth information, start the drone's automatic collection script, and the drone automatically runs the image collection. After ensuring that the simulation environment runs correctly, the actual machine operation is performed.
[0044] The method of automatically collecting light field images by drones was tested in simulation.
[0045] Step S4: Start the onboard computer and the ground control station laptop, connect them to the same local area network, use the nomachine software to remotely connect them, and ensure that the ground control station can access the onboard computer terminal. Use data transmission to connect the drone and the ground control station laptop, start the QGroundControl software on the ground control terminal, and receive the drone flight status data. Give instructions to the drone to control the drone to approach the light field reconstruction target and start light field acquisition.
[0046] In some embodiments of the present invention, the pilot uses a remote controller to control the drone to fly near a light field reconstruction target, adjusts the drone's position to approximately 1.5-2 m from the light field reconstruction target, and makes the drone carry a high-definition camera facing the light field reconstruction target, starts the drone's HOLD mode to stabilize the drone's position, and performs light field acquisition by collecting light field images.
[0047] The light field image collection method is any one of a single drone nine-square collection method, a single drone circular collection method, and a drone array collection method.
[0048] In some embodiments of the present invention, the method of collecting light field images is based on the offboard mode of the drone, and the ROS system sends instructions to the drone to realize automatic image collection and control of the drone.
[0049] The steps of the single drone nine-square grid running method include: the drone flies according to the set nine-square grid waypoints, and collects images at each waypoint.
[0050] In some embodiments of the present invention, nomachine is used at the ground control end to start the onboard computer and the UAV's mavros communication. For the collection of the nine-square running method: please refer to Figure 1 , return the coordinates of the drone body, use the current position of the drone as the coordinate origin and the center of the nine-square grid, set the drone nine-square grid waypoints, and then set the spacing between adjacent waypoints in the nine-square grid (such as 0.5m), set the distance between the drone and the light field reconstruction target (such as 1.5m), the drone hovers at each waypoint in sequence for a preset time (such as 5s) and takes multiple images (such as 3 images), and returns the waypoint body coordinate position information to the ground control station for subsequent image processing. It can be understood that the specific numerical value given in this embodiment is only an example and does not constitute a limitation on the protection scope. In other embodiments, it can also be set to other numerical values as appropriate. This embodiment collects 9*9 images.
[0051] The single drone circular running method refers to the drone collecting light fields of light field reconstruction targets from a bird's-eye view. In some embodiments of the present invention, the drone collects images from a bird's-eye view, and takes a preset distance (such as 2m) directly above the light field reconstruction target as the initial point. After the drone hovers at this point for a preset time (such as 5s) and takes multiple images (such as 3), it moves forward in the direction of the drone's nose for a preset distance (such as 1m), and starts to fly counterclockwise in a circle with a preset radius (such as a radius of 1m) with the initial point as the center. The drone's flight speed is set to 0.2m / s, and the drone hovers for 5s to take three photos every preset flight time (such as 4s), and the image acquisition task is terminated after eight groups of photos are taken. It can be understood that the specific numerical values given in this embodiment are only an example and do not constitute a limitation on the scope of protection. In other embodiments, it can also be set to other numerical values as appropriate.
[0052] The drone array acquisition is to use the drone array to acquire the light field of the light field reconstruction target. In some embodiments of the present invention, the drone array is navigated to the light field reconstruction target by using the cluster flight control based on ROS2, and the timer is used to control the camera to acquire photos at the same time to complete the image acquisition.
[0053] Step S5: After the drone completes image acquisition, it turns on the return mode and returns to the ground. The onboard computer transmits the stored images back to the laptop computer on the ground through the local area network or hotspot connection for subsequent processing.
[0054] Step S6: The ground control station pre-processes the returned images.
[0055] The preprocessing includes: in order to remove the motion blur caused by the drone flight, the three images taken at each waypoint are median stacked to reduce the noise and outliers in the image; with the central waypoint as a reference, the other waypoint images are matched to the histogram of the central waypoint image to reduce the brightness difference.
[0056] Step S7: stitching the processed images together according to the relative positions of the shots, completing the reconstruction of the light field image of the light field reconstruction target, and synthesizing a multi-sub-aperture light field image.
[0057] It is a prior art to synthesize multiple images to obtain a light field image, and the specific synthesis process will not be described in detail here.
[0058] The synthesized multi-subaperture light field image is stored in a laptop in JPEG format. The synthesized light field image contains the light direction and intensity information of the scene, which can be used for subsequent operations such as generating depth maps, generating new viewpoint maps, and three-dimensional reconstruction.
[0059] The light field reconstruction method provided in the aforementioned embodiment of the present invention uses a drone equipped with a high-definition camera to collect images, and then stitches together a synthesized light field image. Compared with using a microlens array to collect light field information, the obtained light field image has a higher spatial resolution, and using a drone to collect images can be more flexible and can meet the needs of various scenarios such as indoor and outdoor scenes, near-ground and low-altitude scenes.
[0060] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A light field reconstruction method based on an unmanned aerial vehicle system, characterized in that: The following steps are involved: Controlling a drone equipped with a high-definition camera to approach a light field reconstruction target and collect light field images by running a light field image collection method to collect multiple images, wherein the running light field image collection method is any one of a single drone nine-square running method, a single drone circular running method, and a drone array collection; Transmit the images collected by the drone from the onboard computer to the ground control station; The ground control station pre-processes the multiple images sent back; The processed images are stitched together to synthesize a multi-sub-aperture light field image.
2. The light field reconstruction method based on the UAV system according to claim 1 is characterized in that: The ground control station uses a computer and a 3DR data radio to communicate with the UAV flight control to obtain the UAV flight status data. The computer of the ground control station is connected to the onboard computer of the UAV so that the ground control station can send instructions to the onboard computer. After receiving the instructions, the onboard computer controls the UAV to collect images.
3. The light field reconstruction method based on the UAV system according to claim 2 is characterized in that: The computer of the ground control station is connected to the onboard computer of the drone via a local area network or hotspot.
4. The light field reconstruction method based on the UAV system according to claim 1, characterized in that: Before running the drone for image acquisition, verify the drone control algorithm in a simulation environment.
5. The light field reconstruction method based on the UAV system according to claim 1, characterized in that: When running a drone to collect images, first use the remote control to control the drone to fly near the light field reconstruction target, adjust the drone's position to a preset distance from the light field reconstruction target, and make the drone carry a high-definition camera facing the light field reconstruction target, start the drone's HOLD mode to stabilize the drone's position, and then perform light field collection by collecting light field images.
6. The light field reconstruction method based on the UAV system according to claim 1, characterized in that: The preprocessing includes but is not limited to any one or more of removing motion blur, color calibration, and translation.
7. The light field reconstruction method based on the UAV system according to claim 1, characterized in that: The synthesized multi-subaperture light field image is stored in a ground control station.
8. The method for light field reconstruction based on an unmanned aerial vehicle system according to any one of claims 1 to 7, characterized in that: The single drone nine-square grid running method is to use the current position of the drone as the coordinate origin and the center of the nine-square grid, set the drone's nine-square grid waypoints, and set the spacing between adjacent waypoints in the nine-square grid and the distance between the drone and the light field reconstruction target. The drone hovers at each waypoint in sequence and takes images of the light field reconstruction target.
9. The method for light field reconstruction based on an unmanned aerial vehicle system according to any one of claims 1 to 7, characterized in that: The single-UAV circular running acquisition method is that the UAV takes a preset distance above the light field reconstruction target as the initial point, and collects images at the initial point, then takes the initial point as the center and the preset distance as the radius to fly in a circle, and takes multiple groups of images of the light field reconstruction target at intervals during the circular flight.
10. The method for light field reconstruction based on an unmanned aerial vehicle system according to any one of claims 1 to 7, characterized in that: The drone array acquisition is to navigate all drones in the drone array to different positions in front of the light field reconstruction target, and the high-definition images carried by all drones take pictures of the light field reconstruction target.
Citation Information
Patent Citations
Light field three-dimensional reconstruction method based on light field microscopic system
CN113850902A
