A distributed visual SLAM method based on UAV platform

Abstract

The present invention relates to a distributed visual SLAM method based on an unmanned aerial vehicle platform, and the steps are: the unmanned aerial vehicle flies to collect ROS data packets, parses the ROS data packets to obtain binocular image data, trains the NetVLAD weight file, runs the visual odometer front end, and finally runs the distributed visual SLAM algorithm to obtain the flight trajectory of the unmanned aerial vehicle. The present invention is aimed at the SLAM problem in a large-scale scene, which is prone to inaccurate positioning or drift. The beneficial effect of the present invention is that, for outdoor large-scale GPS-denied environments such as unmanned marine reconnaissance, a new application method of distributed visual SLAM using an unmanned aerial vehicle equipped with a binocular visual sensor is proposed. The implementation method of multi-unmanned aerial vehicle cluster exploration is adopted, and the relative posture matching between distributed terminals is performed based on the similarity of visual scenes to obtain the accurate positioning and navigation effect of distributed visual SLAM in a large-scale scene.

Description

Technical Field

[0001] The invention belongs to the research field of visual SLAM and relates to a distributed visual SLAM method based on an unmanned aerial vehicle platform. Background Art

[0002] SLAM technology is one of the key supporting technologies for the widespread application of intelligent robots in the future, and can provide robots with the ability to autonomously explore unknown environments. The current SLAM research progress has achieved relatively good application results in indoor or small outdoor environments with obvious structural information. However, for exploration in large-scale scenarios, the SLAM effect of a single machine is limited by the data scale and the performance bottleneck of the computing platform. For example, the famous gmapping algorithm will experience particle degradation of the back-end filtering algorithm in large-scale scenarios, resulting in serious positioning drift. The distributed visual SLAM method based on the unmanned aerial vehicle platform involved in the present invention is a new application method proposed for the multi-machine SLAM problem in large-scale outdoor GPS-denied environments such as unmanned marine reconnaissance. Summary of the invention

[0003] Technical issues to be solved

[0004] In order to avoid the shortcomings of the prior art, the present invention proposes a distributed visual SLAM method based on a drone platform. Under the conditions of existing experimental equipment and experimental cost, a drone is used to collect experimental data, and a multi-machine distributed SLAM method is simulated on a desktop computing platform to complete the principle verification of the invention. It provides application support for multi-machine distributed SLAM technology in large-scale scenarios.

[0005] Technical Solution

[0006] A distributed visual SLAM method based on an unmanned aerial vehicle platform is characterized by the following steps:

[0007] Step 1: Collect ROS data packets during drone flight: When the drone is flying, use the visual sensor on board to collect ROS data packets; use the ROS data packet recording method to subscribe to the binocular image data published by the camera node and record the experimental data during the drone flight;

[0008] Step 2: From the ROS data packet, use the open source ROS package image extraction program to parse the binocular image data and the timestamp data corresponding to each frame of the image. The image data can be used to train the NetVLAD weight file, and the binocular image and timestamp data are used to run the visual odometry front end to obtain the front-end VO data.

[0009] Step 3: Use image data to train NetVLAD and obtain a weight file; input the binocular image data in the extracted ROS package into the NetVLAD network. The operation method is to use the program script in Matlab to read the image data from the specified data storage directory and input it into the NetVLAD network. The training result is a NetVLAD weight file;

[0010] Step 4: Run the front-end VO algorithm ORB2-SLAM to obtain visual odometer data:

[0011] The binocular image data extracted from the ROS data packet and the corresponding timestamp data, as well as the calibrated SLAM operation configuration parameters, are used as input data for the front-end VO algorithm ORB2-SLAM to obtain visual odometer data;

[0012] The calibrated SLAM operation configuration parameters include camera dedistortion parameters and SLAM algorithm operation parameters;

[0013] Step 5: Run the open source distributed visual SLAM algorithm dslam, read the NetVLAD weight file in step 3 and the visual odometer data in step 4, as well as the trajectory log file of the drone flight in step 1, use the open source Matlab program to read its longitude and latitude data and draw it, and obtain the satellite positioning trajectory of the drone flight.

[0014] The visual sensor adopts Intel Realsense D435i.

[0015] The de-distortion parameters of the camera are obtained using a calibration plate and an open source kalibr calibration algorithm.

[0016] The operating parameters of the SLAM algorithm are set using the hardware parameters of the device and the operating configuration file template of ORB2.

[0017] Beneficial Effects

[0018] The present invention proposes a distributed visual SLAM method based on an unmanned aerial vehicle platform, the steps are: the unmanned aerial vehicle collects ROS data packets during flight, parses the ROS data packets to obtain binocular image data, trains the NetVLAD weight file, runs the visual odometer front end, and finally runs the distributed visual SLAM algorithm to obtain the flight trajectory of the unmanned aerial vehicle. The present invention is aimed at the SLAM problem in a large range of scenes, which is prone to inaccurate positioning or drift. The beneficial effect of the present invention is that, for outdoor large-scale GPS-denied environments such as unmanned marine reconnaissance, a new application method of distributed visual SLAM using unmanned aerial vehicles equipped with binocular vision sensors is proposed. The implementation method of multi-unmanned aerial vehicle cluster exploration is adopted, and the relative posture matching between distributed terminals is performed based on the similarity of visual scenes to obtain the accurate positioning and navigation effect of distributed visual SLAM in a large range of scenes.

[0019] Based on the technical requirements of the marine unmanned integrated system, the present invention proposes a distributed visual SLAM method based on an unmanned aerial vehicle platform for engineering application scenarios of island and reef reconnaissance. The problem of multi-machine SLAM in a large-scale GPS-denied environment has always been a problem that needs to be solved urgently in engineering application research. The present invention proposes a new application method of distributed visual SLAM using an unmanned aerial vehicle equipped with a binocular vision sensor, which matches the relative poses between distributed terminals based on the similarity of the visual scene to obtain the accurate positioning and navigation effect of distributed visual SLAM in a large-scale scene. The basic work of the present invention is expected to provide technical support for the subsequent complete distributed visual SLAM combined with point cloud maps, and provide new engineering application support for the reconnaissance of oceanic islands and reefs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 ROS Packet Recording

[0021] Figure 2 Experimental data collection results

[0022] Figure 3 ROS data packet analysis results

[0023] Figure 4 Image data

[0024] Figure 5 Timestamp data

[0025] Figure 6 NetVLAD network training process

[0026] Figure 7 NetVLAD network training results

[0027] Figure 8 SLAM operation configuration file

[0028] Fig. 9 Visual odometer front-end operation process

[0029] Fig.10 VO front-end operation results

[0030] Fig.11 Distributed visual SLAM algorithm operation process

[0031] Fig.12 Distributed visual SLAM algorithm operation results

[0032] Fig.13 Example 1 Satellite positioning trajectory

[0033] Fig.14 Example 1 Algorithm running trajectory

[0034] Fig.15 Example 2 Satellite positioning trajectory

[0035] Fig.16 Example 2 Algorithm running trajectory

[0036] Fig.17 Example 3 Satellite positioning trajectory

[0037] Fig.18 Example 3 Algorithm running trajectory DETAILED DESCRIPTION

[0038] The present invention will now be further described with reference to the embodiments and the accompanying drawings:

[0039] The technical solution of the present invention is as follows:

[0040] Under the existing experimental equipment and experimental cost conditions, a drone is used to collect experimental data, and a multi-machine distributed SLAM method is simulated on a desktop computing platform to complete the principle verification of the invention. The specific technical solution is as follows:

[0041] a) UAV flight to collect ROS data packets

[0042] The visual sensor carried by the drone is Intel Realsense D435i, which is a common visual perception device in the field of robotics. It has good driver support under the ROS system and uses robust hardware parameter settings under the ROS system. Therefore, the camera can be called under the ROS system, and the ROS data packet recording method can be used to subscribe to the binocular image data published by the camera node to record the experimental data during the flight of the drone. Figure 1 The experimental data collection results are shown in Figure 2 shown.

[0043] In the embodiment: using Figure 1The terminal command shown in the figure records the ROS data packets during the flight of the drone, which records the binocular image data / camera / infra1 / image_rect / raw and / camera / infra2 / image_rect / raw of the D435i camera carried by the drone. The data collection results are shown in Figure 2 As shown, the ROS data packet is a .bag file named by time. Figure 2 The .ulg file is the satellite positioning flight trajectory log file exported by the UAV flight control, and readme.txt is the appendix file that records the experimental conditions.

[0044] b) Analyze ROS data packets to obtain binocular image data

[0045] After obtaining experimental data through flight experiments, it is necessary to use the open source ROS package image extraction program to parse the binocular image data and the timestamp data corresponding to each frame of the image from the ROS data packet. Among them, the image data can be used to train the NetVLAD weight file, and the binocular image and timestamp data are used to run the visual odometry front end to obtain the front-end VO data. The parsing results are as follows Figure 3 As shown, the image data is Figure 4 As shown, the timestamp data is Figure 5 shown.

[0046] In the embodiment: the open source ROS package image extraction program is used to parse the collected .bag file, and the parsing result is as follows: Figure 3 As shown in the figure, image_0 is the left image data, image_1 is the right image data, and times.txt is the timestamp data corresponding to each image. Figure 4 As shown in the figure, it is a .png file named with a 6-digit serial number; the timestamp data is as follows Figure 5 As shown, the data describing the time is expressed in scientific notation with a fixed 6 decimal places, and the unit is seconds.

[0047] c) Train NetVLAD weight file

[0048] The NetVLAD network is used to extract visual features from image data, which are then used for scene similarity judgment of distributed terminals. The network is configured and trained in the Matlab environment to obtain a weight file. The training process is to input the image data extracted from the ROS package into the NetVLAD network for processing. The specific operation method is that the program script in Matlab goes to the specified data storage directory to read the image data and input it into the NetVLAD network. The implementation process is as follows: Figure 6 As shown, the training result is a NetVLAD weight file, such as Figure 7 shown.

[0049] In the embodiment: training NetVLAD network weight file, configuring NetVLAD network in Matlab environment, executing the program will Figure 3 The image data is read from the image_0 folder as shown in the figure. The training process is as follows Figure 6 As shown, the training result is a weight file named netvlad_feats.bin.

[0050] d) Run the visual odometry front end

[0051] The odometer data of visual SLAM is obtained by processing image data by the front-end VO, which uses the ORB2 algorithm. Its input data requirements are the binocular image data extracted from the ROS data packet in step b) and the corresponding timestamp data, as well as the calibrated SLAM operation configuration file, which specifies the camera's dedistortion parameters and the SLAM algorithm's operation parameters. The camera's calibration parameters are obtained using a calibration board and the open source kalibr calibration algorithm. The SLAM operation configuration parameters are set according to the hardware parameters of the device used and the ORB2 operation configuration file template. The SLAM operation configuration file is as follows: Figure 8 As shown in Figure 2, the visual odometer front-end operation process is as follows: Fig. 9 As shown, the VO front-end running results are as follows Fig.10 shown.

[0052] In the embodiment: Run the front-end VO algorithm ORB2-SLAM to obtain visual odometer data. Running ORB2-SLAM requires Figure 3 The binocular image data and the corresponding timestamp data, as well as the SLAM operation configuration file, are shown in Figure 8As shown, the camera's dedistortion parameters camera.p2 and others are written in. The dedistortion parameters are obtained by calibrating the D435i camera using the open source calibration program kalibr combined with the calibration board. The operating parameters of ORB2-SLAM are mainly preset parameters adjusted in the open source algorithm such as the number of feature point extractions ORBextractor.nFeatures. In the specific embodiment, the camera's resolution Camera.width, Camera.height, image data frame rate Camera.fps, the baseline coefficient Camera.bf between the camera's binoculars, and the dividing boundary coefficient TheDepth between near and far points in the image need to be adjusted in combination with the experimental equipment. After many experimental adjustments, the specific parameters used in the embodiment are Camera.width: 1280, Camera.height: 720, Camera.fps: 15.0, Camera.bf: 20, TheDepth: 360.0. Use the above parameters as input to run the visual odometer front end ORB2-SLAM. The running process is as follows: Fig. 9 The running results are as shown in Fig.10 As shown, pose_0.txt is the pose record file obtained by the front-end VO operation, times_0.txt is the corresponding timestamp file, descs_0.txt, lm_pos_0.txt and lm_obs_0.txt are additional files for evaluating algorithm performance, which have no direct relationship with the results of this patent. The files needed by the distributed SLAM algorithm dslam are mainly pose_0.txt and times_0.txt.

[0053] e) Run the distributed visual SLAM algorithm

[0054] After obtaining the visual odometer data and NetVLAD weight file, you can run the open source distributed visual SLAM algorithm dslam to verify the application effect of the distributed visual SLAM method based on the drone platform. The algorithm is executed using Matlab scripts. The running process of the distributed visual SLAM algorithm dslam is as follows: Fig.11 The running results are as shown in Fig.12 shown.

[0055] In the embodiment: the distributed SLAM algorithm dslam is run in Matlab, and the program reads the NetVLAD weight file netvlad_feats.bin generated in step c) and pose_0.txt and times_0.txt generated in step d) in the specified folder. The running process is as follows: Fig.11As shown in the figure. For the satellite positioning flight trajectory log file in step a), the open source Matlab program is used to read its latitude and longitude data and draw it. The satellite positioning trajectory of the UAV flight is shown in the figure. Fig.13 As shown in the figure, the running processing results of the distributed visual SLAM algorithm are as follows Fig.14 By comparing the two figures, it can be found that the two are basically consistent, that is, the present invention can achieve the expected engineering application effect required by the distributed visual SLAM method based on the UAV platform.

[0056] Example 2

[0057] The satellite positioning trajectory of the drone flight is as follows Fig.15 As shown in the figure, the running processing results of the distributed visual SLAM algorithm are as follows Fig.16 shown.

[0058] Example 3

[0059] The satellite positioning trajectory of the drone flight is as follows: Fig.17 As shown in the figure, the running processing results of the distributed visual SLAM algorithm are as follows Fig.18 shown.

[0060] Matters not covered by the present invention are known technologies.

[0061] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed visual SLAM method based on an unmanned aerial vehicle platform, characterized in that Here are the steps: Step 1: Collect ROS data packets during drone flight: When the drone is flying, use the visual sensor on board to collect ROS data packets; use the ROS data packet recording method to subscribe to the binocular image data published by the camera node and record the experimental data during the drone flight; Step 2: From the ROS data packet, use the open source ROS package image extraction program to parse the binocular image data and the timestamp data corresponding to each frame of the image. The image data can be used to train the NetVLAD weight file, and the binocular image and timestamp data are used to run the visual odometry front end to obtain the front-end VO data. Step 3: Use image data to train NetVLAD and obtain a weight file; input the binocular image data in the extracted ROS package into the NetVLAD network. The operation method is to use the program script in Matlab to read the image data from the specified data storage directory and input it into the NetVLAD network. The training result is a NetVLAD weight file; Step 4: Run the front-end VO algorithm ORB2-SLAM to obtain visual odometer data: The binocular image data extracted from the ROS data packet and the corresponding timestamp data, as well as the calibrated SLAM operation configuration parameters, are used as input data for the front-end VO algorithm ORB2-SLAM to obtain visual odometer data; The calibrated SLAM operation configuration parameters include camera dedistortion parameters and SLAM algorithm operation parameters; Step 5: Run the open source distributed visual SLAM algorithm dslam, read the NetVLAD weight file in step 3 and the visual odometer data in step 4, as well as the trajectory log file of the drone flight in step 1, use the open source Matlab program to read its longitude and latitude data and draw it, and obtain the satellite positioning trajectory of the drone flight.

2. The distributed visual SLAM method based on the UAV platform according to claim 1 is characterized in that: The visual sensor adopts Intel Realsense D435i.

3. The distributed visual SLAM method based on the UAV platform according to claim 1 is characterized in that: The de-distortion parameters of the camera are obtained using a calibration plate and an open source kalibr calibration algorithm.

4. The distributed visual SLAM method based on the UAV platform according to claim 1 is characterized in that: The operating parameters of the SLAM algorithm are set using the hardware parameters of the device and the operating configuration file template of ORB2.

Images