Unmanned aerial vehicle bridge and tunnel space crossing adsorption method based on slam

Abstract

The application discloses a kind of unmanned plane bridge tunnel space crossing adsorption method based on SLAM.Method includes: when unmanned plane crosses in bridge tunnel space, environment image is photographed using monocular camera, depth information is collected using laser radar;Environment image is solved to obtain initialization map after initial pose;After the pose of unmanned plane changes, correct initialization map using depth information and solve as spatial point, so as to obtain densification map;Correct mapping error of densification map, carry out plane clustering division to calibrate adsorbable area, control unmanned plane to adsorb on adsorbable area when crossing in bridge tunnel space, realize the crossing adsorption of unmanned plane bridge tunnel space.The method of the application can calibrate the adsorption area where unmanned plane can stop according to environmental conditions after SLAM mapping, and correct the map information through closed-loop similar scene detection, accumulate the pose error of unmanned plane and the parameter error of coordinate point, realize the accurate calibration of the stopping position when unmanned plane crosses.

Description

TECHNICAL FIELD

[0001] The application relates to a method for crossing and adsorbing of a UAV (Unmanned Aerial Vehicle), and relates to the field of UAV control, in particular to a method for crossing and adsorbing of a UAV bridge-tunnel space based on SLAM (Simultaneous Localization and Mapping). BACKGROUND

[0002] Large bridges represented by sea-crossing bridges are important components of traffic facilities and are one of the key links of the traffic artery. The sea-crossing bridge is located on the sea, has a poor working environment, a large volume and a complex system structure, and brings great challenges to the operation and maintenance of the bridge. The existing technical equipment is not only complex to operate, has high work intensity and high safety risks, but also has a large number of dead zones and blind zones, and cannot effectively detect and maintain the bridge, thereby bringing serious safety hazards to the long-term safe and stable operation of the bridge, and an effective bridge defect detection method needs to be established.

[0003] The flyable and adsorbable robot is a relatively novel bridge defect detection tool, and has incomparable advantages in detection efficiency, detection accuracy and other aspects compared with traditional detection methods. However, when detecting or repairing, the UAV needs to fly through and adsorb in the complex bridge-tunnel space, which puts forward new challenges to the space mapping and path finding capabilities of the UAV.

[0004] The existing UAV bridge-tunnel space crossing technology at least has the following defects: 1. In the visual SLAM (Simultaneous Localization and Mapping) mapping process, a plane extraction method in a three-dimensional scene is not provided, and a loop detection link is also lacking, so that the UAV cannot correct the map information when flying through the same area, accumulates the UAV pose error and coordinate point parameter error, and it is difficult to establish a correct map; 2. The UAV does not calibrate the adsorbable area when performing SLAM mapping in the bridge-tunnel space; 3. The existing UAV landing method is mainly aimed at the horizontal and flat ground that can be landed, and does not consider the adsorption and landing on the vertical wall surface in the bridge-tunnel space. SUMMARY

[0005] In order to solve the problems in the background art, the application provides a UAV bridge-tunnel space crossing and adsorbing method based on SLAM.

[0006] The technical scheme adopted by the application is:

[0007] The UAV bridge-tunnel space crossing and adsorbing method based on SLAM comprises:

[0008] Step one, when the UAV equipped with a monocular camera and a laser radar crosses the bridge-tunnel space, the monocular camera is used to shoot the environmental image of the bridge-tunnel space, and the laser radar is used to collect the depth information of the bridge-tunnel space.

[0009] Step two, the monocular camera shoots the environment image of the bridge tunnel space to solve the initial pose of the monocular camera, so as to obtain the initialization map.

[0010] Step three, after the pose of the unmanned aerial vehicle is changed, the 2D feature points in the initialization map are corrected by the depth information of the bridge tunnel space collected by the laser radar, and then the 2D feature points in the initialization map are calculated as 3D space points, so as to obtain the densification map.

[0011] The monocular camera collects high-resolution RGB images, which are used to obtain the initialization camera pose information and construct a sparse point cloud map. The laser radar collects depth information, couples the space collected by the laser radar with the space constructed by the camera shooting, uses the depth information to interpolate and correct the sparse point cloud, and constructs a dense point cloud space.

[0012] Step four, correct the mapping error of the densification map, then perform plane clustering division on the map space area of the densification map, so as to calibrate the adsorbable area of the unmanned aerial vehicle, control the unmanned aerial vehicle to adsorb on the adsorbable area when crossing the bridge tunnel space, and realize the crossing adsorption of the unmanned aerial vehicle in the bridge tunnel space.

[0013] In step two, the ORB (Oriented FAST and Rotated BRIEF) feature extraction and matching method is used to extract 2D feature points from the environment image of the bridge tunnel space shot by the monocular camera, and then the initial pose is solved, so as to obtain the initialization map. The ORB feature extraction and matching method defines the image matrix of the image block B with a pixel radius R. The radius of the image block B will be determined by pre-shooting. The threshold of the image distance is set to extract the feature points of the image sequence, and the feature point matching is performed between the adjacent frames. According to the requirement of the epipolar constraint solution, at least 8 pairs of matched points should be obtained.

[0014] In step three, the depth value of the 2D feature points in the initialization map is corrected by the depth information of the bridge tunnel space collected by the laser radar, and then the 2D feature points in the initialization map are calculated as 3D space points by using the nonlinear optimization PnP (Perspective-n-Point) method, the motion of 3D to 2D point pair is realized, and the densification map is obtained.

[0015] The feature point matching relationship between the initial frame and the next frame is used to solve the initial state according to the epipolar constraint, and the initial point cloud map is obtained. Then, the PnP method is used to solve the subsequent camera pose by using the obtained 3D points, and the camera motion trajectory is recorded. The laser radar correction sparse point cloud method is to construct a sparse point cloud according to the depth information of the environment directly measured by the laser radar, and correct the depth information measured by the camera shooting.

[0016] In the fourth step, the mapping error of the dense map is corrected through loop closure detection, specifically, a DBoW2 (Bags of binary words for fast place recognition in image sequence) bag-of-words model is used for correction, a plurality of bridge and tunnel space pictures are collected as a training set and input into the DBoW2 bag-of-words model for training to obtain a trained DBoW2 bag-of-words model, and then the dense map to be corrected is input into the trained DBoW2 bag-of-words model for processing to obtain a dense map with corrected mapping error.

[0017] The camera extracts and classifies features of images taken in the past, quantifies image features, and corrects map information when the unmanned aerial vehicle experiences the same area, reduces error accumulation, and completes loop closure detection.

[0018] In the fourth step, the 3D space points of the dense map are first grouped, divided into a removal group and a non-removal group, then the 3D space points in the removal group and their corresponding edges are removed, and then the map space area of the dense map is divided by an AHC (Agglomerative Hierarchical Clustering) clustering algorithm to divide the plane, so as to demarcate the adsorbable area of the unmanned aerial vehicle according to a preset adsorbable radius.

[0019] First, the space points of the dense map are grouped, then the space points are roughly divided by distance relationship, the space points are merged to form a plane rough division, the point set is further divided and the plane is extracted, and the plane division result is optimized, and the adsorbable area is demarcated according to the set adsorbable radius.

[0020] The space points in the removal group specifically include 3D space points with a mean-square error (MSE) higher than a preset MSE threshold, 3D space points without data information, 3D space points with discontinuous depth values, and 3D space points on the boundary of two planes in the dense map.

[0021] The beneficial effects of the present application are:

[0022] 1. The present application adopts a monocular camera combined with a laser radar to synchronize mapping, utilizes the characteristic that the laser radar is easy to collect depth information, and corrects the three-dimensional point cloud map established by the monocular camera.

[0023] 2. The present application provides a clustering and division method for planes in a bridge and tunnel space, so that the unmanned aerial vehicle can demarcate the adsorbable area and the obstacle area when passing through, and the obstacle avoidance ability of the unmanned aerial vehicle is strengthened.

[0024] 3、The bag-of-words model of the training bridge and tunnel space image is used for loop detection in SLAM mapping, a bridge and tunnel space image feature dictionary is established, and image feature points and image classification are easily determined.

[0025] The method can calibrate the adsorption area where the unmanned aerial vehicle can stop according to the environment after SLAM mapping, correct the map information through loop similar scene detection, accumulate the unmanned aerial vehicle pose error and coordinate point parameter error, and realize accurate calibration of the stopping position of the unmanned aerial vehicle during crossing. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The flowchart of the method of the application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail in combination with the drawings. The description introduces specific embodiments consistent with the principles of the application by way of examples but not by way of limitation, and the description of the embodiments is sufficient in detail to enable those skilled in the art to practice the application, other embodiments can be used and the structure of each element can be changed and / or replaced without departing from the scope and spirit of the application. Therefore, the following detailed description should not be understood in a limiting sense.

[0028] As shown in Figure 1 The SLAM-based unmanned aerial vehicle bridge and tunnel space crossing and adsorption method of the application is as follows:

[0029] Step one, when the unmanned aerial vehicle equipped with a monocular camera and a laser radar crosses the bridge and tunnel space, the monocular camera is used to shoot the environment image of the bridge and tunnel space, and the laser radar is used to collect the depth information of the bridge and tunnel space; in specific implementation, the unmanned aerial vehicle can be a quadcopter.

[0030] Step two, the monocular camera shoots the environment image of the bridge and tunnel space to perform initial pose solution of the monocular camera, so as to obtain an initialization map. The monocular camera shoots the environment image of the bridge and tunnel space to perform 2D feature point extraction and then initial pose solution using the ORB feature extraction and matching method, so as to obtain an initialization map. The ORB feature extraction and matching method defines the image moment of the image block B with a pixel radius R, and the radius of the image block B will be determined by pre-shooting. The threshold of the image distance is set to extract the feature points of the image sequence, the feature points are matched between adjacent frames, and at least 8 pairs of matched points are obtained according to the requirement of the solution of the epipolar constraint.

[0031] Step three, after the pose change of the unmanned aerial vehicle, the depth information of the bridge and tunnel space collected by the laser radar is used to correct the 2D feature points in the initialization map, and then the 2D feature points in the initialization map are calculated as 3D space points, so as to obtain the densification map; the monocular camera collects high-resolution RGB images to obtain the initialization camera pose information and construct a sparse point cloud map, the laser radar collects depth information, and the space collected by the laser radar is coupled with the space constructed by the camera shooting, the depth information is used to interpolate and correct the sparse point cloud, and the dense point cloud space is constructed.

[0032] The depth value of the 2D feature points in the initialization map is corrected by the depth information of the bridge and tunnel space collected by the laser radar, and then the 2D feature points in the initialization map are calculated as 3D space points by using the nonlinear optimization PnP method, the motion of 3D to 2D point pair is realized, and thus the densification map is obtained. By using the matching relationship of feature points between the initial frame and the next frame, the initial state is solved according to the epipolar constraint to obtain the initial point cloud map, and then the PnP method is used to solve the subsequent camera pose by using the obtained 3D points, and the camera motion trajectory is recorded. The laser radar correction sparse point cloud method is to construct a sparse point cloud according to the depth information of the environment directly measured by the laser radar, and correct the depth information measured by the camera shooting.

[0033] Step four, correct the mapping error of the densification map, then perform plane clustering division on the map space area of the densification map, so as to calibrate the adsorbable area of the unmanned aerial vehicle, control the unmanned aerial vehicle to adsorb on the adsorbable area when crossing the bridge and tunnel space, and realize the crossing adsorption of the unmanned aerial vehicle in the bridge and tunnel space. The mapping error of the densification map is corrected by closed loop detection, specifically, the DBoW2 bag-of-words model is used for correction, first, a plurality of bridge and tunnel space pictures are collected as a training set to input into the DBoW2 bag-of-words model for training, and a trained DBoW2 bag-of-words model is obtained, then the densification map to be corrected is input into the trained DBoW2 bag-of-words model for processing, and the densification map with corrected mapping error is obtained. The camera extracts and classifies the features of the images taken in the past, and quantizes the image features. When the unmanned aerial vehicle experiences the same area, the map information can be corrected, the error accumulation is reduced, and the closed loop detection is completed.

[0034] First, the 3D space points of the densification map are grouped into a removal group and a non-removal group, then the 3D space points in the removal group and their corresponding edges are removed, then the plane clustering division is performed on the map space area of the densification map by the AHC clustering algorithm, so as to calibrate the adsorbable area of the unmanned aerial vehicle according to the preset adsorbable radius. First, the space points of the densification map are grouped, then the space points are roughly divided by the distance relationship, the space points are merged to form a plane rough division, the point set is further divided and the plane is extracted, the plane division result is optimized, and the adsorbable area is calibrated according to the set adsorbable radius.

[0035] The space points in the removing group specifically include: 3D space points with a mean square error (MSE) higher than a preset MSE threshold, 3D space points without data information, 3D space points with discontinuous depth values, and 3D space points on the boundary of two planes in the densified map.

[0036] The specific implementation of the present application is as follows:

[0037] In step one, the method of the present application can be applied to or combined with various unmanned aerial vehicle devices, such as a bridge detection flying adsorption robot. When a visible light monocular camera and a laser radar are implemented, they are carried on a four-rotor unmanned aerial vehicle flight platform. The images under different positions, different angles, and different lighting conditions of the bridge surface can be obtained by rotating the gimbal.

[0038] In step two, the radius of the image block B is determined by pre-shooting during initialization. The specific method is to pre-shoot 10 bridge and tunnel space images, and adjust the pixel radius of the image block to at least 8 feature points per image. After obtaining the pixel radius of the image block, feature matching is performed on the initial frame and the next frame. If there are less than 8 pairs of feature points, the pixel radius of the image block is increased for re-matching. The specific method of camera pose solving is as follows:

[0039] According to the epipolar constraint, the essential matrix E and the basis Fundamental matrix F are solved:

[0040] E=t^R

[0041] F=K -T EK -1

[0042]

[0043] Where t is the displacement vector of the monocular camera, R is the pose matrix, K is the intrinsic parameter matrix of the monocular camera, x1 and x2 are the pixel coordinates of the two feature points, and p1 and p2 are the world coordinates of the two feature points.

[0044] Under the condition that the camera intrinsic parameter matrix K has been calibrated, the essential matrix E can be directly solved using the feature points:

[0045]

[0046] Where u1 and u2 are the normalized plane coordinates of the two feature points, and v1 and v2 are the normalized plane coordinates of the two feature points; e i is the i-th element in the essential matrix E, i=1, 2, …, 9.

[0047] Substitute 8 pairs of matched feature points to solve E matrix, and perform singular value SVD (Singular Value Decomposition) decomposition on the E matrix:

[0048] E = U∑V T

[0049] wherein U and V are orthogonal matrices, and ∑ is a singular value matrix; t and R are obtained through the singular value SVD decomposition, and then the 3D coordinates of the initial frame feature points are solved to obtain the 3D coordinates of the initial frame feature points as the initialized map points.

[0050] In step three, the map densification method is as follows:

[0051] After the map initialization is successful, the 3D-to-2D point motion is solved by PnP, the 2D feature points of the current frame are matched with the 3D map points of the previous frame, the 3D-to-2D point motion is solved by PnP, and the 3D-to-2D re-projection error ξ is defined * is as follows:

[0052]

[0053] wherein u i is the projection of the i th 2D feature point; s i is the projection coordinate of the i th 3D space point; ξ is the camera pose; and P i is the i th 3D space point.

[0054] The PnP pose solving is converted into an optimization problem, the error is optimized to be minimum, the pose of the current frame is solved, the 3D coordinates of the 2D feature points in the current frame are solved according to the pose data of the camera of the current frame and the 3D point coordinates of the previous frame, and the new feature points are added into the three-dimensional point cloud map to densify the map.

[0055] The laser radar obtains depth information by measuring the spatial depth information through the laser radar, and obtains 3D point coordinates, and an objective function J is established by using an iterative closest point ICP (Iterative Closest Point) method to solve the camera pose:

[0056]

[0057] wherein p i is the coordinate of the i th 3D space point, p i = Rp i ′+t, p iRi is the coordinate of the i-th 3D space point; the camera pose R and displacement vector t are obtained by solving the optimization least squares problem, and then the world coordinates of the 3D points are obtained through coordinate transformation; the sparse point cloud map constructed by the monocular camera is coupled with the map constructed by the laser radar through the synchronous time axis, the data correlation of the feature points is carried out, the depth information obtained by the laser radar is used to interpolate and fill the missing values of the sparse point cloud map, and the dense map is corrected.

[0058] In step four, the closed-loop detection method is to pre-prepare a training set containing five bridge and tunnel space photos, train the DBoW2 bag-of-words model, generate a bridge and tunnel space image feature dictionary and count a word frequency histogram, so as to classify the images; when the bag-of-words model is loaded on the unmanned aerial vehicle, the images taken by the monocular camera are classified according to the feature dictionary, and when it is identified that the unmanned aerial vehicle passes through the same area, the existing map information is compared with the previous map information to correct and reduce the cumulative error.

[0059] The space division method is to divide the point cloud into several groups of points that do not overlap according to the spatial distance in the image space as the initial coarse division, and the nodes and edges in the figure represent a group of points and their neighborhood; the bridge and tunnel space is generally composed of vertical bridge pier columns and horizontal bridge bottom surfaces, and this coarse division can preliminarily obtain the geometric distribution characteristics of the bridge and tunnel space.

[0060] The AHC clustering method is adopted to systematically merge the nodes belonging to the same plane and remove the divided groups and their corresponding edges. Until the root mean square error MSE of the plane fitting exceeds a certain threshold value, the final plane division result is obtained, denoted as P i . The plane division formed by clustering is optimized, the contour is formed by connecting the plane boundary points and the points that are not smooth are removed, the critical scale L of the adsorption plane is set according to the wheelbase of the unmanned aerial vehicle, generally L=D+30cm, D is the wheelbase of the unmanned aerial vehicle, and when the plane scale of the plane P i is greater than L, it is marked as the adsorption area of the unmanned aerial vehicle. Finally, the bridge and tunnel space traversal and adsorption of the unmanned aerial vehicle are completed through the bridge and tunnel space mapping based on SLAM.

Claims

1. A SLAM-based method for adsorption during UAV bridge and tunnel traversal, characterized in that, include: Step 1: When the drone equipped with a monocular camera and LiDAR passes through the bridge and tunnel space, the monocular camera is used to capture environmental images of the bridge and tunnel space, and the LiDAR is used to collect depth information of the bridge and tunnel space. Step 2: Perform initial pose calculation on the environmental images of the bridge and tunnel space captured by the monocular camera to obtain the initial map; Step 3: After the UAV changes its pose, the 2D feature points in the initial map are corrected by using the depth information of the bridge and tunnel space collected by the lidar. Then, the 2D feature points in the initial map are solved into 3D spatial points to obtain a denser map. Step four: Correct the mapping error of the dense map, and then perform planar clustering on the map space region of the dense map to mark the area that the UAV can attach to. Control the UAV to attach to the area that can attach when it passes through the bridge and tunnel space, so as to realize the UAV's attachment when passing through the bridge and tunnel space.

2. The SLAM-based UAV bridge and tunnel spatial crossing adsorption method according to claim 1, characterized in that: In step two, the ORB feature extraction and matching method is used to extract 2D feature points from the environmental image of the bridge and tunnel space captured by the monocular camera, and then the initial pose is calculated to obtain the initial map.

3. The SLAM-based UAV bridge and tunnel spatial crossing adsorption method according to claim 1, characterized in that: In step three, the depth information of the bridge and tunnel space collected by the lidar is used to correct the depth values ​​of the 2D feature points in the initial map. Then, the nonlinear optimization PnP method is used to solve the 2D feature points in the initial map into 3D spatial points, thereby obtaining a denser map.

4. The SLAM-based UAV bridge and tunnel spatial crossing adsorption method according to claim 1, characterized in that: In step four, the mapping error of the dense map is corrected by closed-loop detection. Specifically, the DBoW2 bag-of-words model is used for correction. First, several bridge and tunnel spatial images are collected as a training set and input into the DBoW2 bag-of-words model for training to obtain a trained DBoW2 bag-of-words model. Then, the dense map to be corrected is input into the trained DBoW2 bag-of-words model for processing to obtain a dense map with corrected mapping errors.

5. The SLAM-based UAV bridge and tunnel spatial crossing adsorption method according to claim 1, characterized in that: In step four, the 3D spatial points of the dense map are first grouped into removal groups and non-removal groups. Then, the 3D spatial points and their corresponding edges in the removal groups are removed. Then, the map spatial region of the dense map is divided into planar clusters using the AHC clustering algorithm, thereby calibrating the adsorption area of ​​the UAV according to the preset adsorption radius.

6. The SLAM-based UAV bridge and tunnel spatial crossing adsorption method according to claim 5, characterized in that: The spatial points in the removal group specifically include: 3D spatial points with a mean square error (MSE) higher than a preset MSE threshold, 3D spatial points with no data information, 3D spatial points with discontinuous depth values, and 3D spatial points on the boundary of two planes in a dense map.

Images