Data fusion and edge extraction method of monocular vision, IMU and lidar

Abstract

The present invention discloses a method for data fusion and edge extraction of monocular vision, IMU, and laser radar. The method comprises the following steps: a monocular camera collects images of a target at different angles and performs feature matching; a FAST algorithm is used to detect key points, and the BRIEF algorithm is used to convert the key points into ORB feature vectors to obtain the image feature distribution at different target angles; a FLANN matching algorithm is used to obtain feature pair matching results, and feature matching pairs with errors are eliminated using the RANSAC algorithm; point cloud data obtained by the laser radar is segmented and feature extracted, and the IMU information is pre-integrated to perform motion compensation on the point cloud data; the IMU data and keyframe poses are used to construct a fusion of the laser radar information and the visual input (VIO); and an improved Canny operator with an adaptive threshold is used for edge extraction. The present invention improves the efficiency of data fusion and edge extraction of monocular vision, IMU, and laser radar.

Claims

1. A method for data fusion and edge extraction of monocular vision, IMU and lidar, characterized in that: The following steps are involved: Step 1: Use an industrial camera to capture images of the target at different angles; Step 2: Use the FAST algorithm to collect key points in the image; Step 3: Use the rBRIEF algorithm to convert the key points collected in step 2 into feature vectors; Step 4: Perform FLANN matching on the feature vectors of the target at different angles to obtain the matching results; Step 5: Use the RANSAC algorithm based on the F matrix to remove the incorrectly matched feature points. Step 6: Perform field of view cutting, point cloud segmentation, and feature extraction on the point cloud data obtained by the lidar. Pre-integrate the IMU information and use the pre-integrated data to perform motion compensation on the point cloud data obtained by the lidar. Step 7: Use IMU data and keyframe poses to build a fusion of lidar information and VIO; Step 8: Enhance the contrast of the histogram, perform Gaussian filtering to remove noise, and use the Canny operator to extract edges, as follows: Step 8.1: Perform bilateral filtering on the collected grayscale image to obtain the filtered image. ; Step 8.2: Calculate the discrete Fourier transform DFT of the image to obtain ; Step 8.

3. Calculate the filter function and The product of , using a Gaussian function: ； The spectrum is the filter function and The product of: ； Step 8.4: Perform inverse discrete Fourier transform IDFT and get ,Pick The real part of is taken as the result image after filtering; Step 8.5: Use the Prewitt template to calculate the gradient values ​​of the result image in step 8.4 in all directions, and find the maximum gradient direction to obtain the maximum gradient image. Specifically: using Prewitt's four-direction templates and Perform convolution, and then take the largest value among the four values ​​as the gradient value of the point. The direction corresponding to the maximum value is the maximum gradient direction. After processing, the maximum gradient image is obtained. ; Step 8.6, according to Normalized to a grayscale range of 0 to 255, the approximate grayscale range is determined using a histogram, and the median grayscale value within this range is found by sorting and set as the threshold ,set up The grayscale range is [0, L-1], then the threshold Divided into 2 ranges [0,T] and [T,L-1]; Assume the grayscale in the image is The number of pixels is , the total number of pixels within the grayscale range ; The probability of each gray level appearing is ,exist The threshold is used to divide it into two categories and ; Depend on composition, Depend on Composition; then area and The probabilities of , and The average gray value is: ； ； in yes Average grayscale: ； The total variance of the two regions is: ; let exist Take the values ​​in turn, so that The largest The value is the best choice for a low threshold; similarly, Repeat the above steps to get the best choice of high threshold; Step 8.7: Using the high and low thresholds in step 8.5, obtain two threshold edge images according to the three criteria of the Canny operator. , ; Obtained by the low threshold, Obtained by a high threshold; due to The edge is basically a true edge obtained by the high threshold, but there is a break, so the edge connection is searched in the 8 areas of the break until the Connect them together to get a complete edge binary graph; Step 8.8: Finally, the edge binary image in step 8.6 is subjected to morphological opening operations using structural elements in the horizontal and vertical directions, thereby obtaining an edge binary image with the external edge burrs removed.

2. The method for data fusion and edge extraction of monocular vision, IMU and laser radar according to claim 1, characterized in that: The industrial camera is used to capture images of the target at different angles as described in step 1, as follows: Step 1.1: Install the monocular camera on the front of the unmanned vehicle, turn it on and connect it to the PC, adjust the focal length, and ensure that a clear image can be captured; install the 3D LiDAR directly above the monocular camera; Step 1.2: Use Autoware software to jointly calibrate the monocular camera and lidar. Step 1.3: Collect clear images from different angles and transfer them to the computer processing system through the image acquisition card. The processing platform is Ubuntu 18.04 + OpenCV 3.4.

2. Obtain lidar point cloud data and transfer it to the computer processing system. The processing platform is ROS.

3. The method for data fusion and edge extraction of monocular vision, IMU and laser radar according to claim 1, characterized in that: The key points in the image are collected using the FAST algorithm as described in step 2, as follows: Step 2.

1. Select any point p in the image, set its pixel brightness to lp, and draw a circle with radius r to cover M pixels around point p. Step 2.2, set the threshold value to t, and make the difference between the pixel value of N consecutive pixels on the circle and the pixel value of point p. If the absolute values ​​of these differences are greater than Bigger or bigger Small, then point p is identified as a key point; Step 2.3: Calculate the center of mass of the feature point within the radius r by using the moment. The coordinates of the feature point to the center of mass form a vector as the direction of the feature point. The moment is defined as follows: ； in, is the grayscale expression of the image; The centroid of this moment is: ； Set the corner point coordinate to O, and the angle of the vector is the direction of the feature point: 。 4. The method for data fusion and edge extraction of monocular vision, IMU and laser radar according to claim 1, characterized in that: In step 3, the key points collected in step 2 are converted into feature vectors using the rBRIEF algorithm. Specifically, the direction of the key points found in step 2 is used as the direction of BRIEF. In this way, the feature points have both direction and scale information. Step 3.1, perform Gaussian smoothing on the image. In the neighborhood of a feature point, select n pairs of pixels pi, qi, where i = 1, 2, ..., n. Let the selected point pair set be for: ； Passing angle Rotation: ； Get a new set of point pairs ; Step 3.2: Compare the grayscale values ​​of each point pair. If I(pi)>I(qi), generate a 1 in the binary string; otherwise, generate a 0. Step 3.3: Compare all the point pairs and generate a binary string of length n; the value of n is 128, 256, or 512.

5. The method for data fusion and edge extraction of monocular vision, IMU and laser radar according to claim 1, characterized in that: Performing FLANN matching on the obtained target feature vectors at different angles to obtain a matching result, which is specifically as follows: Step 4.1: Establish a priority search k-means tree, which is specifically as follows: Step 4.1.1: Use the obtained ORB feature points to establish a hierarchical k-means tree; Step 4.1.2: Take the clustering centers at each level as the nodes of the tree; Step 4.1.3: When the number of points in a certain cluster is less than K, use these data nodes as leaf nodes; Step 4.2: Search in the priority search k-means tree to obtain a matching result, which is specifically as follows: Step 4.2.1: Start retrieving from the root node N; Step 4.2.2: If N is a leaf node, add all leaf nodes at the same level to the search result, that is, count += |N|; Step 4.2.3: If N is not a leaf node, compare its child nodes with the query Q to find the nearest node Cq, and add other nodes at the same level to the priority queue; Step 4.2.4: Perform recursive search on the Cq node; Step 4.2.5: If the priority queue is not empty and count < L, take the first element of the priority queue and assign it to N, and then repeat Step 4.2.

1.

6. The method for data fusion and edge extraction of monocular vision, IMU and laser radar according to claim 1, characterized in that: Performing the operation of removing mismatched feature points on the matching result through the RANSAC algorithm based on the F matrix, which is specifically as follows: Step 5.1: Set the number of iterations to 25, that is, the program will execute at most 25 times to prevent the program from entering an infinite loop due to unreasonable threshold settings; Step 5.2: At the beginning of the program, set the set matching error threshold to 0.

05. First, calculate three pairs of point pairs arbitrarily to obtain an affine transformation matrix; then arbitrarily select three more point pairs to calculate the matching error, and iterate until the loop condition is met and exit; Step 5.3: Record the error obtained each time, select the minimum value after comparison, and substitute it as the set matching error threshold into the operation of Step 5.2 to ensure that the obtained matrix meets the actual requirements.

7. The method for data fusion and edge extraction of monocular vision, IMU and laser radar according to claim 1, characterized in that: Performing field-of-view cutting, point cloud segmentation, and feature extraction on the point cloud data obtained by the lidar, and performing motion compensation on the point cloud data obtained by the lidar by pre-integrating the information of the IMU and using the pre-integrated data, which is specifically as follows: Step 6.1: Select key frames. If there is a certain parallax between the current frame and the previous frame, then select it as a key frame; Step 6.2: After performing segmentation and identification operations on the lidar point cloud data, outliers are removed. Points with the number of point clouds in the segmented body less than a certain threshold are identified as outliers, and outliers do not participate in the subsequent operations of the subsequent algorithms; Step 6.3: After performing segmentation operations on the original lidar point cloud data, extract plane features and corner feature points from the segmented point cloud data; Step 6.4: Correct the lidar point cloud data through the data of other sensors. When the visual odometer is working properly, use VIO to correct the point cloud data; when VIO degrades, correct the point cloud data through the IMU.

8. The method for data fusion and edge extraction of monocular vision, IMU and laser radar according to claim 1, characterized in that: Step 7 uses IMU data and keyframe poses to build a fusion of lidar information and VIO, as follows: Step 7.1: The LiDAR measurement data represents the three-dimensional coordinates of environmental objects in the LiDAR body coordinate system. Since the LiDAR and IMU are fixedly connected, the LiDAR data is transformed into the corresponding VIO keyframe coordinate system. Step 7.2: Synchronize the lidar measurement data to the VIO keyframe to obtain the fusion data of lidar and VIO.

Images