Online Registration Method for Infrared and Visible Light Sensors Based on IMU and Odometry

Abstract

The present invention discloses an online registration method for infrared and visible light sensors based on IMU and odometer, which relates to the technical field of computer vision and image processing. It solves the technical problem of insufficient registration accuracy of infrared and visible light images. The key points of its technical solution are that image enhancement and gray processing are respectively performed on infrared and visible light images, improving the gray similarity of the two images, creating better conditions for subsequent extraction and matching of feature point pairs, and improving the accuracy rate; the improved FAST method is used to extract corner points, adding descriptions of scale and rotation, greatly enhancing their robustness in different image representations; according to the bidirectional search algorithm, matching feature point pairs are screened, effectively reducing the number of false matches; by using the registration strategy of constructing epipolar geometric constraints in visual odometer, the feature point pairs can be effectively utilized to calculate the accurate transformation relationship between images, thereby obtaining the latest external parameter matrix.

Description

Technical Field

[0001] This application relates to the technical field of computer vision and image processing, and particularly to an online registration method for infrared and visible light sensors based on IMU and odometer. Background Art

[0002] As one of the key technologies of driverless vehicles, intelligent perception technology directly or indirectly affects the intelligent level of vehicles and is a research hotspot in the field of intelligent driving. Multi-source information fusion perception is an inevitable way to achieve safe driving of intelligent vehicles in complex environments. The visible light imaging sensor in vehicle-mounted sensors has a high resolution and can express detailed texture information of images, but its imaging is easily affected by the natural environment and the imaging effect is not good in environments such as foggy days and nights. The main advantage of the infrared imaging sensor is that it can still image well under natural conditions such as foggy days and nights. However, the resolution of the infrared imaging sensor is generally low. Although the generated infrared image can intuitively reflect the magnitude of the thermal radiation energy of an object and is not affected by light, it will lose some appearance information such as the texture and structure of the object, and the performance of the detailed texture information of the image is not good.

[0003] Due to the different imaging mechanisms, different resolutions, and different physical imaging conditions of the two sensors, there are linear transformations such as rotation, translation, and scaling in the imaging under the two different modes, and there may even be non-linear transformations such as distortion. More and more vision systems adopt a working mode in which an infrared imaging sensor and a visible light imaging sensor cooperate with each other to supplement the single visible light imaging mode. The cooperation of multiple sensors can obtain more levels of scene information, conduct a more in-depth analysis of the image scene, and obtain richer image information. The essence of image registration is to find an optimal geometric transformation to align these two images geometrically. Fast speed, high accuracy, and good algorithm robustness have become a goal pursued by the online registration algorithm for infrared and visible light images.

[0004] Common image registration methods can be divided into two categories: region-based gray-level registration methods and feature-based registration methods. Among them, region-based gray-level registration methods usually use a certain region of the image or the entire image to estimate the transformation parameters of the images in terms of spatial geometry. Common region-based registration methods include correlation-based methods, phase correlation methods, probability measure methods, etc. The advantage of this method is that it does not require any assumptions about the relationship between the gray levels of multi-modal images and can be used for registration of almost any different modalities. However, it ignores the spatial information and result information of the images, has a long calculation time, and is not very robust to noise. Feature-based registration methods, due to their characteristics such as fast calculation speed and good robustness, have been gradually widely used. The core of it is to select common features for multi-source images and matching strategies suitable for these feature points. Typical feature-based registration algorithms include Harris corner detection, SIFT, SURF, and ORB algorithms, etc. Among them, Rublee et al. improved the matching algorithm in 2011 to address the problems of the efficiency and memory occupancy of SIFT and SURF algorithms, and proposed the ORB (Oriented FAST and Rotated BRIEF) algorithm to further improve the execution efficiency of image feature registration tasks.

[0005] In fact, during the online registration process of multi-source sensors in intelligent driving, most of them use some image registration algorithms to continuously solve the latest external parameter matrix for real-time registration. However, in this process, there may be feature point mis-matching or data calculation errors, resulting in different values of the external parameter matrix for the front and rear frames. At this time, the relative pose of the two sensors may not have changed. If the external parameter matrix is still updated in real time according to the results of the registration algorithm, it will cause "pseudo-registration".

[0006] Currently, the research status of infrared and visible light image registration still has a large gap from the requirements of intelligent driving in complex environments, and there are still many problems waiting to be solved. The online registration method of infrared and visible light for intelligent driving vehicles needs to solve the following four main problems: (1) Feature point extraction, ensuring that most feature points in the two images can correspond to the same positions in the actual scene; (2) Feature point matching, ensuring the matching accuracy between image feature points and reducing mis-matching; (3) Registration strategy, being able to effectively use feature point pairs to calculate the accurate transformation relationship between images, so as to obtain the latest external parameter matrix; (4) Registration timing, how to update the external parameter matrix when the relative pose of the two sensors changes. Summary of the Invention

[0007] This application provides an online registration method for infrared and visible light sensors based on IMU and odometer, and its technical purpose is to improve the registration accuracy of infrared and visible light images.

[0008] The above technical object of the present application is achieved by the following technical solutions:

[0009] An online registration method for infrared and visible light sensors based on IMU and odometer, comprising:

[0010] S1: Determine whether the relative pose of the front and rear frame infrared sensors and visible light sensors has changed through the IMU installed under the infrared sensor and visible light sensor. If it has changed, go to step S2;

[0011] S2: Obtain the infrared image of the infrared sensor and the visible light image of the visible light sensor in the same frame, preprocess the infrared image and the visible light image so that the sizes of the infrared image and the visible light image are the same, and obtain the first infrared image and the first visible light image;

[0012] S3: Enhance the first infrared image through the Laplace operator to obtain a second infrared image, and perform grayscale processing on the first visible light image to obtain a second visible light image;

[0013] S4: Extract the FAST corner points of the second infrared image and the second visible light image through an improved FAST corner point method;

[0014] S5: Describe the image area around the FAST corner points through the BRIEF descriptor to obtain description information, and the FAST corner points and the description information constitute feature points;

[0015] S6: Match the feature points in the second infrared image and the second visible light image according to the Hamming distance. When the Hamming distance of the matched feature points is less than twice the minimum distance, save the matched feature points to obtain a pair of matched feature points, otherwise reject the matched feature points;

[0016] S7: Screen the pair of matched feature points according to the bidirectional search algorithm to obtain the final pair of matched feature points;

[0017] S8: Use the visual odometer to construct an epipolar geometry constraint according to the final pair of matched feature points, obtain the relative pose transformation matrix of the infrared sensor and the visible light sensor according to the epipolar geometry constraint and the odometer, and register the infrared sensor and the visible light sensor according to the transformation matrix.

[0018] The beneficial effects of this application are as follows: (1) Image enhancement and grayscale processing are respectively performed on the infrared image and the visible light image, improving the grayscale similarity of the two images, creating better conditions for subsequent extraction and matching of feature point pairs, and improving the accuracy; (2) The improved FAST method is used to extract corner points, and descriptions of scale and rotation are added, greatly enhancing their robustness in different images; (3) According to the bidirectional search algorithm, matching feature point pairs are screened, effectively reducing the number of false matches; (4) Using the registration strategy of constructing epipolar geometry constraints in visual odometry, the accurate transformation relationship between images can be effectively calculated using feature point pairs, thereby obtaining the latest external parameter matrix; (5) Using the IMU can well grasp the registration timing and reduce errors. Description of Drawings

[0019] Figure 1 It is the flowchart of the method described in this application;

[0020] Figure 2 It is the flowchart of obtaining the external parameter matrix through visual odometry;

[0021] Figure 3 It is the planar projection diagram of the infrared camera and the visible light camera. Detailed Implementation Manner

[0022] The technical solution of this application will be described in detail below with reference to the drawings.

[0023] As Figure 1 shown, the online registration method of infrared and visible light sensors based on IMU and odometry described in this application includes:

[0024] S1: The IMU installed under the infrared sensor and the visible light sensor is used to judge whether the relative pose of the front and rear frame infrared sensors and visible light sensors has changed. If it has changed, go to step S2.

[0025] Specifically, the IMU generally includes an accelerometer and a gyroscope. The spatial attitude of each frame of the sensor, that is, the Euler angle, is obtained according to the acceleration and angular velocity output by the accelerometer and the gyroscope; by comparing the transformation relationship between the Euler angles of the front and rear frame infrared sensors and visible light sensors, it is judged whether the infrared sensor and the visible light sensor have drifted. If drift is detected, subsequent operations to obtain the latest external parameter matrix are performed.

[0026] S2: The infrared image of the infrared sensor and the visible light image of the visible light sensor in the same frame are acquired, and the infrared image and the visible light image are preprocessed to make the sizes of the infrared image and the visible light image the same, obtaining the first infrared image and the first visible light image.

[0027] Generally, the images obtained by the infrared camera and the visible light camera are inconsistent in terms of the number of pixels and depth. Therefore, preliminary preprocessing is required, and they are cropped to obtain two images with the same size.

[0028] S3: Enhance the first infrared image through the Laplace operator to obtain a second infrared image, and perform grayscale processing on the first visible light image to obtain a second visible light image.

[0029] Performing Laplace image enhancement and grayscale processing on the infrared image and the visible light image respectively can improve the grayscale similarity of the two images. In the subsequent processes of corner extraction and descriptor calculation, the error will be smaller, which can improve the registration accuracy of the images.

[0030] S4: Extract the FAST corners of the second infrared image and the second visible light image through the improved FAST corner method.

[0031] Specifically, step S4 includes:

[0032] S41: Select a pixel p in the second infrared image and the second visible light image respectively. The brightness of this pixel p is I p , and set a threshold T.

[0033] Specifically, construct an image pyramid for the second infrared image and the second visible light image, and select the pixel p in each layer of this image pyramid.

[0034] S42: If there are N consecutive points on the circle centered on the pixel p whose brightness is greater than I p +T or less than I p -T, then the pixel p is a FAST corner; where N≥12.

[0035] Specifically, with the pixel p as the center, select 16 pixel points on a circle with a radius of 3. If there are N consecutive points on the selected circle whose brightness is greater than I p +T or less than I p -T, then the pixel p can be considered a corner (N is usually taken as 12).

[0036] For higher efficiency, a pre-test operation is added before step S42 to quickly exclude the vast majority of pixels that are not corners. For each pixel p, if among the 1st, 5th, 9th, and 13th pixels on the circle centered on it, the brightness of three pixels is greater than I p +T or less than I p -T, perform step S42 to further detect the pixel p to determine whether it is a corner, otherwise directly exclude the possibility of the pixel p being a corner. Such a pre-test operation greatly accelerates the corner detection.

[0037] S43: Loop through steps S41 to S42 for each pixel until all FAST corner points are selected in the second infrared image and the second visible light image.

[0038] After the selection of the FAST corner points is completed, the rotation of the FAST corner points is achieved by the centroid method, and the centroid method includes:

[0039] S431: In an image block B, define the moment of the image block B as:

[0040] m pq =Σ x，y∈B x p' y q' I(x,y); p', q' = {0, 1};

[0041] where I(x, y) represents the grayscale value of each feature point;

[0042] S432: Obtain the centroid of the image block B through the moment, and the centroid is expressed as:

[0043]

[0044] where the centroid M represents the center weighted by the grayscale value of the image block B; m 10 represents the grayscale value on the X-axis, m 10 =∑ x,y∈B xI(x,y); m 01 represents the grayscale value on the Y-axis, m 01 =∑ x,y∈B yI(x,y); m 00 represents the grayscale value of the entire circle around the entire feature point, m 00 =∑ x,y∈B I(x,y);

[0045] S433: Connect the geometric center O of the image block B to the centroid M to obtain the direction vector Then the direction of the FAST corner point is expressed as:

[0046] θ=arctan(m 01 / m 10 );

[0047] S434: The FAST corner point rotates according to the direction θ to achieve the rotation invariance of the FAST corner point.

[0048] S5: Describe the image area around the FAST corner point through the BRIEF descriptor to obtain the description information, and the FAST corner point and the description information constitute the feature point.

[0049] The description vector of the BRIEF descriptor consists of 0s and 1s. If the pixel p near the FAST corner is larger than the pixel q, then 1 is taken; otherwise, 0 is taken. After describing the image area around the FAST corner, a multi-dimensional vector composed of 0s and 1s is obtained, and this multi-dimensional vector constitutes the description information.

[0050] This application uses a binary descriptor BRIEF. Its description vector consists of many 0s and 1s. Here, the 0s and 1s encode the size relationship between two pixels (such as p and q) near the corner: if p is larger than q, then 1 is taken; otherwise, 0 is taken. If we take 128 such p's and q's, we finally get a 128-dimensional vector composed of 0s and 1s.

[0051] S6: Match the feature points in the second infrared image and the second visible light image according to the Hamming distance. When the Hamming distance of the matched feature points is less than twice the minimum distance, save the matched feature points to obtain the matched feature point pairs; otherwise, eliminate the matched feature points.

[0052] The Brute Force matching algorithm is also known as brute force matching. This algorithm describes the similarity between the infrared image and the visible light image by calculating the Hamming distance, and searches for the feature point in the infrared image that is closest to the feature point in the visible light image. Although the brute force matching algorithm has relatively good robustness, when there is a certain overlapping area between the infrared and visible light images, two or more points in the infrared image match successfully with the same point in the visible light image, resulting in a poor correct matching rate. Therefore, the following two-way search algorithm is used for screening.

[0053] S7: Screen the matched feature point pairs according to the two-way search algorithm to obtain the final matched feature point pairs.

[0054] Specifically, the two-way search method includes: determining a feature point A in the second infrared image, then finding the feature point A' in the second visible light image that matches A, and then searching for the feature point in the second infrared image corresponding to the feature point A'. If it is the same point as A, then A and A' are the matched feature point pairs.

[0055] This two-way search method is expressed as:

[0056]

[0057] where, I a represents the second infrared image, and I b represents the second visible light image.

[0058] S8: Use visual odometry to construct an epipolar geometric constraint based on the final matched feature point pairs, obtain the relative pose transformation matrix of the infrared sensor and the visible light sensor according to the epipolar geometric constraint and the odometry, and register the infrared sensor and the visible light sensor according to the transformation matrix.

[0059] Specifically, as Figure 2 shown, the step S8 includes:

[0060] S71: After the extraction of the matched feature point pairs is completed, transform the homogeneous coordinates p i of the matched feature points on the second infrared image from the infrared coordinate system I to the camera coordinate system C, that is where p i =(x i , y i , 1) T , represents the rotation matrix for transforming p i from the infrared coordinate system to the visible light coordinate system; represents the translation vector for transforming p i from the infrared coordinate system to the visible light coordinate system;

[0061] S72: Construct an epipolar geometric constraint and obtain the essential matrix Obtain the external parameter matrix according to the essential matrix E

[0062] According to Figure 3 in the planar projection diagram of the infrared camera and the visible light camera, it can be seen that I and C are the optical centers of the infrared camera and the visible light camera respectively, and α and B are the normalized planes of the two cameras. p i , p c are both two-dimensional points on the image, but here they will be turned into three-dimensional vectors for calculation. The distance from the normalized plane to the centers of the infrared camera and the visible light camera is 1, so p i =(x i , y i , 1) T , p c =(x c , y c , 1) T .

[0063] The step S72 includes:

[0064] S721: Construct a vector equation, expressed as:

[0065] S722: At this time, p i and p c are not in the same coordinate system and need to be coordinate-transformed. Use the rotation matrix to transform pi Converted to the camera coordinate system with C as the origin, it is expressed as:

[0066] S723: Regarding the translation distance between the centers of the infrared camera and the visible light camera as the translation vector t, converting the vector equation into an epipolar constraint, which is expressed as:

[0067] S724: Let Then there is p c T Ep i = 0, and then E is obtained according to the pixel point position, and the latest external parameter matrix is obtained by decomposing E

[0068] The above is the exemplary embodiment of the present application, and the protection scope of the present application is defined by the claims and their equivalents.

Claims

1. An online registration method for infrared and visible light sensors based on IMU and odometer, characterized in that Including: S1: Use the IMU installed under the infrared sensor and visible light sensor to determine whether the relative pose of the infrared sensor and visible light sensor between the front and rear frames has changed. If it has changed, go to step S2; S2: Obtain the infrared image of the infrared sensor and the visible light image of the visible light sensor in the same frame, and preprocess the infrared image and the visible light image to make their sizes the same, obtaining the first infrared image and the first visible light image; S3: Enhance the first infrared image through the Laplace operator to obtain the second infrared image, and perform grayscale processing on the first visible light image to obtain the second visible light image; S4: Extract the FAST corner points of the second infrared image and the second visible light image through an improved FAST corner point method. After the selection of the FAST corner points is completed, rotate the FAST corner points through the centroid method; Among them, the centroid method includes: S431: In an image block B, define the moment of the image block B as: m pq = ∑ x，y∈B x p' y q' I(x, y); p', q' = {0, 1}; where I(x, y) represents the grayscale value of each feature point; S432: Obtain the centroid of the image block B through the moment, and the centroid is expressed as: Among them, the centroid M represents the center weighted by the gray value of the image block b; m 10 represents the gray value on the X-axis, m 10 = ∑ x,y∈B xI(x,y); m 01 represents the gray value on the Y-axis, m 01 = Σ x,y∈B yI(x,y); m 00 represents the gray value of the entire circle around the entire feature point, m 00 = ∑ x,y∈B I(x,y); S433: Connect the geometric center O of the image block B with the centroid M to obtain the direction vector Then the direction of the FAST corner point is expressed as: θ = arctan(m 01 / m 10 )); S434: The FAST corner points are rotated according to the direction θ to achieve the rotational invariance of the FAST corner points; S5: Describe the image region around the FAST corner points through the BRIEF descriptor to obtain the description information. The FAST corner points and the description information form feature points; S6: Match the feature points in the second infrared image and the second visible light image according to the Hamming distance. When the Hamming distance of the matched feature points is less than twice the minimum distance, save the matched feature points to obtain the matched feature point pairs, otherwise eliminate the matched feature points; S7: Screen the matched feature point pairs according to the bidirectional search algorithm to obtain the final matched feature point pairs; S8: Use the visual odometer to construct the epipolar geometry constraint according to the final matched feature point pairs, obtain the relative pose transformation matrix of the infrared sensor and the visible light sensor according to the epipolar geometry constraint and the odometer, and register the infrared sensor and the visible light sensor according to the transformation matrix.

2. The method according to claim 1, characterized in that In step S1, the IMU includes an accelerometer and a gyroscope. The spatial attitude of each frame of the sensor, that is, the Euler angles, is obtained according to the acceleration and angular velocity output by the accelerometer and the gyroscope. By comparing the transformation relationship between the Euler angles of the infrared sensor and the visible light sensor in the front and rear frames, it is judged whether the infrared sensor and the visible light sensor have drifted.

3. The method according to claim 1, characterized in that Step S4 includes: S41: Select a pixel p in the second infrared image and the second visible light image respectively, the brightness of the pixel p is I p , and set a threshold T; S42: If there are N consecutive points on the circle centered at pixel p with brightness greater than I p +T or less than I p -T, then pixel p is a FAST corner point; where N ≥ 12; S43: Loop steps S41 to S42 until all the FAST corner points are selected in the second infrared image and the second visible light image. After the selection of the FAST corner points is completed, rotate the FAST corner points through the centroid method.

4. The method according to claim 3, characterized in that In step S41, construct an image pyramid for the second infrared image and the second visible light image, and select a pixel p in each layer of the image pyramid.

5. The method according to claim 3 or 4, characterized in that, Before the step S42, for each pixel p, if the brightness of three pixels among the 1st, 5th, 9th, and 13th pixels on the circle centered at it is greater than I p +T or less than I p -T, step S42 is executed to further detect the pixel p to determine whether it is a corner point, otherwise the possibility of the pixel p being a corner point is directly excluded.

6. The method according to claim 1, characterized in that, In step S5, the description vector of the BRIEF descriptor consists of 0 and 1. If the pixel p near the FAST corner is larger than the pixel q, then 1 is taken; otherwise, 0 is taken. After completing the description of the image area around the FAST corner, a multi-dimensional vector composed of 0 and 1 is obtained, and this multi-dimensional vector constitutes the description information.

7. The method according to claim 1, wherein In step S7, the two-way search algorithm includes: Determine a feature point A in the second infrared image, then find a feature point A' that matches A in the second visible light image, and then search for the feature point corresponding to the feature point A' in the second infrared image If it is the same point as A, then A and A' are a pair of matching feature points; The two-way search algorithm is expressed as: Among them, I a represents the second infrared image, and I b represents the second visible light image.

8. The method according to claim 1, characterized in that, The step S8 includes: S71: After the extraction of the matching feature point pairs is completed, transform the homogeneous coordinates p of the matching feature points on the second infrared image i from the infrared coordinate system I to the camera coordinate system C, that is where p i =(x i , y i , 1) T , represents the rotation matrix for transforming p i from the infrared coordinate system to the visible light coordinate system; represents the translation vector for transforming p i from the infrared coordinate system to the visible light coordinate system; S72: Construct the epipolar geometry constraint to obtain the essential matrix Obtain the external parameter matrix based on the essential matrix E 9. The method according to claim 8, wherein The step S72 includes: S721: Construct a vector equation, expressed as: S722: Convert p i to the camera coordinate system with C as the origin, expressed as: S723: Consider the translation distance between the centers of the infrared camera and the visible light camera as the translation vector Convert the vector equation into an epipolar constraint, expressed as: S724: Let Then there is p c T Ep i = 0, and then obtain E based on the pixel point position, and decompose E to get the latest external parameter matrix

Images