Unmanned aerial vehicle bridge and tunnel space crossing adsorption method based on SLAM

Abstract

The invention discloses an unmanned aerial vehicle bridge and tunnel space crossing adsorption method based on SLAM. The method comprises the steps that when the unmanned aerial vehicle passes through a bridge and tunnel space, a monocular camera is used for shooting an environment image, and a laser radar is used for collecting depth information; performing initial pose calculation on the environment image to obtain an initialized map; after the pose of the unmanned aerial vehicle is changed, correcting the initialized map through depth information and resolving the initialized map into spatial points so as to obtain a dense map; and correcting mapping errors of the dense map, performing plane clustering division to calibrate an adsorbable area, controlling the unmanned aerial vehicle to be adsorbed on the adsorbable area when the unmanned aerial vehicle passes through the bridge and tunnel space, and realizing passing adsorption of the unmanned aerial vehicle in the bridge and tunnel space. According to the method, the adsorption area where the unmanned aerial vehicle can be parked can be calibrated according to the environment condition after SLAM mapping, map information is corrected and corrected through closed-loop similar scene detection, the pose error and the coordinate point parameter error of the unmanned aerial vehicle are accumulated, and accurate calibration of the parking position when the unmanned aerial vehicle passes through is achieved.

Description

Technical Field

[0001] The invention relates to a method for adsorbing a drone passing through, and relates to the field of drone control, and in particular to a method for adsorbing a drone passing through a bridge and tunnel space based on SLAM. Background Art

[0002] Large bridges, represented by cross-sea bridges, are an important part of transportation facilities and one of the key links in the main arteries of transportation. Cross-sea bridges are located on the sea, with harsh working environments, large volumes and structures, and complex system components, which poses great challenges to the operation and maintenance of bridges. Existing technical equipment is not only complex to operate, has high work intensity, and high safety risks, but also has a large number of dead zones and blind spots, which make it impossible to effectively detect and maintain them, thus bringing serious safety hazards to the long-term safe and stable operation of bridges. It is urgent to establish an effective bridge defect detection method.

[0003] The flying and adsorbing robot is a relatively new bridge defect detection tool. Compared with traditional detection methods, it has incomparable advantages in detection efficiency, detection accuracy and other aspects. However, when performing inspection or repair operations, the drone needs to fly through and adsorb in the complex bridge and tunnel space, which poses new challenges to the drone's spatial mapping and path-finding capabilities.

[0004] The existing drone bridge and tunnel space crossing technology has at least the following defects: 1. The visual SLAM (Simultaneous Localization and Mapping) mapping process does not provide a plane extraction method in the three-dimensional scene, and lacks a closed-loop detection link, resulting in the inability to correct map information when the drone passes through the same area during flight, accumulating drone posture errors and coordinate point parameter errors, making it difficult to establish a correct map; 2. When the drone performs SLAM mapping in the bridge and tunnel space, the area where it can be adsorbed and docked is not calibrated; 3. The existing drone landing method is mainly aimed at horizontal flat ground where it can land, and does not consider adsorption and docking on the vertical wall of the bridge and tunnel space. Summary of the invention

[0005] In order to solve the problems existing in the background technology, the present invention provides a SLAM-based UAV bridge and tunnel space crossing adsorption method.

[0006] The technical solution adopted by the present invention is:

[0007] The SLAM-based UAV bridge and tunnel space crossing adsorption method of the present invention comprises:

[0008] Step 1: When a drone equipped with a monocular camera and a laser radar passes through a bridge-tunnel space, the monocular camera is used to capture environmental images of the bridge-tunnel space, and the laser radar is used to collect depth information of the bridge-tunnel space.

[0009] Step 2: The environment image of the bridge and tunnel space taken by the monocular camera is used to solve the initial pose of the monocular camera to obtain an initialization map.

[0010] Step three: After the UAV changes its posture, the 2D feature points in the initialization map are corrected through the depth information of the bridge and tunnel space collected by the lidar, and then the 2D feature points in the initialization map are solved into 3D space points to obtain a densified map.

[0011] The monocular camera collects high-resolution RGB images to obtain the initial camera pose information and construct a sparse point cloud map. The lidar collects depth information, and the space collected by the lidar is coupled with the space constructed by the camera. The depth information is used to interpolate and correct the sparse point cloud to construct a dense point cloud space.

[0012] Step 4: Correct the mapping error of the densified map, and then perform planar clustering on the map space area of ​​the densified map to calibrate the adsorbable area of ​​the drone, control the drone to adsorb on the adsorbable area when passing through the bridge and tunnel space, and realize the crossing and adsorption of the drone in the bridge and tunnel space.

[0013] In the step 2, the ORB (Oriented FAST and Rotated BRIEF) feature extraction and matching method is used to extract 2D feature points from the environment image of the bridge and tunnel space taken by the monocular camera, and then the initial pose is solved to obtain an initialized map. The ORB feature extraction and matching method defines the image moment of the image block B with a pixel radius of R, and the radius of the image block B will be measured by pre-shooting. The image distance threshold is set to extract feature points from the image sequence, and feature points are matched between adjacent frames. According to the requirements of the epipolar constraint solution, at least 8 pairs of matching points should be obtained.

[0014] In the step three, the depth information of the bridge and tunnel space collected by the lidar is used to correct the depth values ​​of the 2D feature points in the initialization map, and then the nonlinear optimization PnP (Perspective-n-Point) method is used to solve the 2D feature points in the initialization map into 3D space points to realize the movement of 3D to 2D point pairs, thereby obtaining a densified map.

[0015] Using the matching relationship between the feature points of the initial frame and the next frame, the initial state is solved according to the epipolar constraint and the initial point cloud is obtained. Then, the PnP method is used to solve the subsequent camera pose using the obtained 3D points, and the camera motion trajectory is recorded at the same time. The laser radar corrected sparse point cloud method is to construct a sparse point cloud based on the environmental depth information directly measured by the laser radar, and correct the depth information measured by the camera.

[0016] In the step 4, the mapping error of the densified map is corrected through closed-loop detection, specifically using the DBoW2 (Bags of binary words for fast place recognition in image sequence) word bag model for correction. First, a number of bridge and tunnel space pictures are collected as training sets and input into the DBoW2 word bag model for training to obtain a trained DBoW2 word bag model. Then, the densified map to be corrected is input into the trained DBoW2 word bag model for processing to obtain a densified map with corrected mapping errors.

[0017] The camera extracts and classifies features of images taken in the past, quantifies image features, and can correct map information when the drone passes through the same area, reduce error accumulation, and complete closed-loop detection.

[0018] In the step 4, the 3D spatial points of the densified map are first grouped into a removal group and a non-removal group, and then the 3D spatial points and their corresponding edges in the removal group are removed. Then, the map space area of ​​the densified map is planarly clustered by the AHC (Agglomerative Hierarchical Clustering) clustering algorithm, so as to calibrate the adsorbable area of ​​the drone according to the preset adsorbable radius.

[0019] First, the spatial points of the densified map are grouped, and then a rough division is performed based on the distance relationship of the spatial points. The spatial points are merged to form a rough plane division. The point set is further divided and the plane is extracted. The plane division result is optimized, and the adsorbable area is calibrated according to the set adsorbable radius.

[0020] The spatial points in the removal group specifically include: 3D spatial points whose mean-square error (MSE) is higher than a preset mean-square error threshold, 3D spatial points with no data information, 3D spatial points with discontinuous depth values, and 3D spatial points on the boundary of two planes in the densified map.

[0021] The beneficial effects of the present invention are:

[0022] 1. The present invention adopts a monocular camera and a laser radar to synchronously build a map, and uses the characteristic of the laser radar that it is easy to collect depth information to correct the three-dimensional point cloud map built by the monocular camera.

[0023] 2. The present invention provides a clustering method for planes in bridge and tunnel space, so that the UAV can calibrate the adsorbable area and obstacle area when crossing, thereby enhancing the obstacle avoidance capability of the UAV.

[0024] 3. The bag-of-words model of the bridge-tunnel space image trained by the present invention is used for closed-loop detection in SLAM mapping, and a feature dictionary of the bridge-tunnel space image is established, which makes it easy to determine image feature points and image classification.

[0025] The method of the invention can calibrate the adsorption area where the UAV can dock according to the environmental conditions after SLAM mapping, and correct the map information through closed-loop similar scene detection, accumulate the UAV posture error and coordinate point parameter error, and realize the accurate calibration of the docking position of the UAV when crossing. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail in conjunction with the accompanying drawings. This description introduces specific embodiments consistent with the principles of the present invention by way of example and not limitation, and the description of these embodiments is sufficiently detailed to enable those skilled in the art to practice the present invention, and other embodiments may be used and the structure of each element may be changed and / or replaced without departing from the scope and spirit of the present invention. Therefore, the following detailed description should not be understood in a restrictive sense.

[0028] like Figure 1 As shown, the SLAM-based UAV bridge and tunnel space crossing adsorption method of the present invention is as follows:

[0029] Step 1: When a drone equipped with a monocular camera and a laser radar passes through a bridge and tunnel space, the monocular camera is used to capture environmental images of the bridge and tunnel space, and the laser radar is used to collect depth information of the bridge and tunnel space. In the specific implementation, the drone can be a quad-rotor drone.

[0030] Step 2: The monocular camera is used to take the environmental image of the bridge and tunnel space and perform the initial pose solution of the monocular camera to obtain the initialization map. The ORB feature extraction and matching method is used to extract 2D feature points from the environmental image of the bridge and tunnel space taken by the monocular camera and then perform the initial pose solution to obtain the initialization map. The ORB feature extraction and matching method defines the image moment of the image block B with a pixel radius of R. The radius of the image block B will be measured by pre-shooting. The image distance threshold is set to extract feature points from the image sequence, and feature points are matched between adjacent frames. According to the requirements of the epipolar constraint solution, at least 8 pairs of matching points should be obtained.

[0031] Step three, after the UAV changes its posture, the 2D feature points in the initialization map are corrected through the depth information of the bridge and tunnel space collected by the lidar, and then the 2D feature points in the initialization map are solved as 3D space points to obtain a densified map; the monocular camera collects high-resolution RGB images to obtain the initialization camera posture information and construct a sparse point cloud map, the lidar collects depth information, the space collected by the lidar is coupled with the space constructed by the camera, and the depth information is used to interpolate and correct the sparse point cloud to construct a dense point cloud space.

[0032] The depth information of the bridge and tunnel space collected by the laser radar is used to correct the depth values ​​of the 2D feature points in the initialization map, and then the nonlinear optimization PnP method is used to solve the 2D feature points in the initialization map into 3D space points, realizing the movement of 3D to 2D point pairs, thereby obtaining a densified map. Using the matching relationship between the feature points of the initial frame and the next frame, the initial state is solved according to the epipolar constraint and the initial point cloud map is obtained. Then, the PnP method is used to solve the subsequent camera pose using the obtained 3D points, and the camera motion trajectory is recorded at the same time. The laser radar corrected sparse point cloud method is to construct a sparse point cloud based on the environmental depth information directly measured by the laser radar, and correct the depth information measured by the camera.

[0033] Step 4: Correct the mapping error of the densified map, and then perform planar clustering on the map space area of ​​the densified map to calibrate the adsorbable area of ​​the drone, control the drone to adsorb on the adsorbable area when passing through the bridge and tunnel space, and realize the crossing adsorption of the drone in the bridge and tunnel space. The mapping error of the densified map is corrected through closed-loop detection, specifically using the DBoW2 bag-of-words model for correction. First, collect a number of bridge and tunnel space pictures as training sets and input them into the DBoW2 bag-of-words model for training to obtain the trained DBoW2 bag-of-words model, and then input the densified map to be corrected into the trained DBoW2 bag-of-words model for processing to obtain a densified map with corrected mapping errors. The camera extracts and classifies the features of the images taken in the past, quantifies the image features, and can correct the map information when the drone passes through the same area, reduce error accumulation, and complete closed-loop detection.

[0034] First, the 3D spatial points of the dense map are grouped into removal groups and non-removal groups, and then the 3D spatial points and their corresponding edges in the removal group are removed. Then, the map space area of ​​the dense map is plane clustered and divided by the AHC clustering algorithm, so as to calibrate the adsorbable area of ​​the drone according to the preset adsorbable radius. First, the spatial points of the dense map are grouped, and then the coarse division is performed according to the distance relationship of the spatial points. The spatial points are merged to form a coarse plane division, and then the point set is further divided and the plane is extracted. The plane division result is optimized, and the adsorbable area is calibrated according to the set adsorbable radius.

[0035] The spatial points in the removal group specifically include: 3D spatial points whose mean square error (MSE) is higher than a preset mean square error threshold, 3D spatial points with no data information, 3D spatial points with discontinuous depth values, and 3D spatial points on the boundary of two planes in the densified map.

[0036] The specific embodiments of the present invention are as follows:

[0037] In step one, the method of the present invention can be applied or combined with a variety of UAV equipment, such as applied to a bridge inspection flying adsorption robot. The visible light monocular camera and lidar are mounted on a four-rotor UAV flight platform during specific implementation, and images of the bridge surface at different positions, angles, and lighting conditions can be obtained through the rotation of the gimbal.

[0038] In step 2, the radius of the image block B will be determined by pre-shooting during initialization. The specific method is to pre-shoot 10 bridge-tunnel space images and adjust the image block pixel radius to at least 8 feature points in each image. After obtaining the image block pixel radius, feature matching is performed on the initial frame and its next frame. If there are less than 8 pairs of feature points, the image block pixel radius is increased and re-matched. The specific method of camera pose solution is as follows:

[0039] According to the epipolar constraint, solve the essential matrix E and the fundamental matrix F:

[0040] E=t^R

[0041] F=K -T EK -1

[0042]

[0043] Among them, t is the displacement vector of the monocular camera, R is the pose matrix, K is the intrinsic parameter matrix of the monocular camera, and x 1 and x 2 are the pixel coordinates of the two feature points, p 1 and p 2 are the world coordinates of the two feature points respectively.

[0044] Under the condition that the camera intrinsic parameter matrix K has been calibrated, the essential matrix E can be directly solved using the feature points:

[0045]

[0046] Among them, u 1 and u 2 are the normalized plane coordinates of the two feature points, v 1 and v 2 are the normalized plane coordinates of the two feature points; ei is the i-th element in the essential matrix E, i = 1, 2, …, 9.

[0047] Substituting 8 pairs of matching feature points, the E matrix can be solved and the E matrix can be decomposed by singular value SVD (Singular Value Decomposition):

[0048] E=UΣV T

[0049] Among them, U and V are orthogonal matrices, and Σ is a singular value matrix. t and R are obtained by singular value SVD decomposition, and then the 3D coordinates of the feature points of the initial frame are obtained by solving, and the feature points of the initial frame are used as the initialized map points.

[0050] In step 3, the map densification method is as follows:

[0051] After the map is initialized successfully, PnP is used to solve the motion of 3D to 2D point pairs, and the 2D feature points of the current frame are matched with the 3D map points of the previous frame. The motion of 3D to 2D points is calculated by PnP, and the 3D to 2D reprojection error ξ is defined. * for:

[0052]

[0053] Among them, u i is the projection of the i-th 2D feature point; s i is the projection coordinate of the i-th 3D space point; ξ is the camera pose; P i is the i-th 3D space point.

[0054] The PnP pose solution is transformed into an optimization problem, the error is optimized to obtain the minimum value, and the pose of the current frame is obtained; according to the pose data of the current frame camera and the 3D point coordinates of the previous frame, the 3D coordinates of the 2D feature points in the current frame are calculated, and the new feature points are added to the three-dimensional point cloud map to make the map dense.

[0055] The method of obtaining depth information by laser radar is to measure the spatial depth information through laser radar and obtain the 3D point coordinates. The iterative closest point (ICP) method is used to establish the objective function J and solve the camera pose:

[0056]

[0057] Among them, p i is the coordinate of the i-th 3D space point, p i =Rp i ′+t,p i′ is the coordinate of the i-th 3D space point; the camera pose R and displacement vector t are obtained by optimizing the least squares problem, and then the world coordinates of the 3D point are obtained by coordinate transformation; the sparse point cloud map constructed by the monocular camera is coupled with the map constructed by the lidar through synchronization of the time axis, the feature points are associated with data, and the depth information obtained by the lidar is used to interpolate and fill in the missing values ​​of the sparse point cloud map to correct the dense map.

[0058] In step 4, the closed-loop detection method is to pre-prepare a training set containing five bridge and tunnel space photos, train the DBoW2 bag-of-words model, generate a bridge and tunnel space image feature dictionary and count the word frequency histogram to classify the image. When the bag-of-words model is loaded on the drone, the image taken by the monocular camera is classified according to the feature dictionary. When the drone is identified to pass through the same area, the existing map information is compared and corrected with the previous map information to reduce the cumulative error.

[0059] The spatial division method is to uniformly divide the point cloud into several non-overlapping groups of points according to the spatial distance in the image space as the initial coarse division. The nodes and edges in the graph represent a group of points and their neighborhoods. The bridge-tunnel space is generally composed of vertical piers and horizontal bridge bottom surfaces. This coarse division can preliminarily obtain the geometric distribution characteristics of the bridge-tunnel space.

[0060] The AHC clustering method is used to systematically merge nodes belonging to the same plane and remove the partitioned removal group and its corresponding edges until the root mean square error (MSE) of the plane fitting exceeds a certain threshold, and the final plane partition result is obtained, which is recorded as P i Optimize the plane segmentation formed by clustering, connect the plane boundary points to form a contour and remove the non-smooth points, set the critical scale L of the adsorption plane according to the wheelbase of the drone, generally take L = D + 30cm, D is the wheelbase of the drone, when the plane P i When the plane scale of is larger than L, it is marked as the area that the drone can adsorb. Finally, the drone's bridge and tunnel space crossing and adsorption are completed through SLAM-based bridge and tunnel space mapping.

Claims

1. A SLAM-based UAV bridge and tunnel space crossing adsorption method, characterized in that: include: Step 1: When the UAV equipped with a monocular camera and a laser radar passes through the bridge-tunnel space, the monocular camera is used to capture the environmental image of the bridge-tunnel space, and the laser radar is used to collect the depth information of the bridge-tunnel space; Step 2: The environment image of the bridge and tunnel space captured by the monocular camera is used to solve the initial pose of the monocular camera, thereby obtaining an initialization map; Step 3: After the UAV changes its posture, the 2D feature points in the initialization map are corrected by the depth information of the bridge and tunnel space collected by the lidar, and then the 2D feature points in the initialization map are solved into 3D space points to obtain a densified map; Step 4: Correct the mapping error of the densified map, and then perform planar clustering on the map space area of ​​the densified map to calibrate the adsorbable area of ​​the drone, control the drone to adsorb on the adsorbable area when passing through the bridge and tunnel space, and realize the crossing and adsorption of the drone in the bridge and tunnel space.

2. The SLAM-based UAV bridge and tunnel space crossing adsorption method according to claim 1 is characterized by: In the step 2, the ORB feature extraction and matching method is used to extract 2D feature points from the environment image of the bridge and tunnel space taken by the monocular camera and then perform initial pose calculation to obtain an initialized map.

3. The SLAM-based UAV bridge and tunnel space crossing adsorption method according to claim 1 is characterized by: In the step three, the depth information of the bridge-tunnel space collected by the lidar is used to correct the depth values ​​of the 2D feature points in the initialization map, and then the nonlinear optimization PnP method is used to solve the 2D feature points in the initialization map into 3D space points, thereby obtaining a densified map.

4. The SLAM-based UAV bridge and tunnel space crossing adsorption method according to claim 1 is characterized by: In the step 4, the mapping error of the densified map is corrected through closed-loop detection, specifically using the DBoW2 bag-of-words model for correction. First, a number of bridge and tunnel space pictures are collected as training sets and input into the DBoW2 bag-of-words model for training to obtain a trained DBoW2 bag-of-words model. Then, the densified map to be corrected is input into the trained DBoW2 bag-of-words model for processing to obtain a densified map with corrected mapping errors.

5. The SLAM-based UAV bridge and tunnel space crossing adsorption method according to claim 1 is characterized by: In the step 4, the 3D spatial points of the densified map are first grouped into a removal group and a non-removal group, and then the 3D spatial points and their corresponding edges in the removal group are removed. Then, the map space area of ​​the densified map is planarly clustered by the AHC clustering algorithm, so as to calibrate the adsorbable area of ​​the drone according to the preset adsorbable radius.

6. The SLAM-based UAV bridge and tunnel space crossing adsorption method according to claim 5 is characterized by: The spatial points in the removal group specifically include: 3D spatial points whose mean square error (MSE) is higher than a preset mean square error threshold, 3D spatial points with no data information, 3D spatial points with discontinuous depth values, and 3D spatial points on the boundary of two planes in the densified map.

Images