Fusing inertial measurement unit information for visual odometry

Abstract

The application discloses a kind of visual odometry fusing inertial measurement unit information, it is related to odometry technical field, the visual odometry includes: camera, for real-time acquisition road video, camera two sides are respectively provided with one fill light band;Inertial measurement unit, for the heading angle of camera acquisition;Data processor, for reading video frame in road video;The picture of adjacent two frames is carried out feature point matching;According to the feature point after matching, the rotation angle increment and translation increment of camera are determined;According to the actual rotation angle increment and translation increment of each frame picture, the rotation angle and translation amount of camera are generated;The pose and pose covariance matrix of camera are obtained by filtering the rotation angle and translation amount using primary extended Kalman filtering, and the pose includes rotation angle and translation amount;The heading angle of inertial measurement unit acquisition, the pose and pose covariance matrix of camera are carried out extended Kalman filtering to obtain the final pose and pose covariance matrix.

Description

Technical Field

[0001] This invention relates to the field of odometer technology, and in particular to a visual odometer that integrates information from an inertial measurement unit. Background Technology

[0002] Visual odometry uses a camera to collect information about the surrounding environment. By analyzing the relationship between the camera and the environmental coordinate system, it calculates the positions of feature points and changes in object size in the video image, and in turn, infers changes in camera position, thus determining the vehicle's position within the environment. However, insufficient lighting or drastic changes in illumination can affect the acquisition of feature points and the calculation of distance. Therefore, the accuracy and reliability of visual odometry still need improvement. Summary of the Invention

[0003] The purpose of this invention is to provide a visual odometry that integrates information from an inertial measurement unit, thereby improving the accuracy of the visual odometry.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A visual odometry system integrating inertial measurement unit (IMU) information includes: a data processor, a camera, two supplementary lighting strips, and an IMU. The camera is fixed to the bottom of a mobile device, and the two supplementary lighting strips are located on both sides of the camera. The IMU is fixedly connected to the data processor.

[0006] The camera acquires real-time road surface video and sends the acquired road surface video to the data processor; the inertial measurement unit is used to acquire the heading angle of the camera and send the acquired heading angle of the camera to the data processor.

[0007] The data processor is used for:

[0008] Read video frames from the road surface video and record the currently read video frame as the current frame image;

[0009] Extract feature points from the current frame and calculate feature point descriptors;

[0010] Based on the feature point descriptors of the current frame and the previous frame, feature point matching is performed on the feature points of the current frame and the feature points of the previous frame.

[0011] The rotation increment and translation increment of the camera are determined based on the matched feature points; the translation increment is the pixel increment within the imaging plane of the camera.

[0012] Based on the similarity ratio coefficient, the rotation increment and translation increment of each frame are converted into actual rotation increment and actual translation increment; the similarity ratio coefficient is determined based on the actual distance between the camera and the ground measured in real time by the laser rangefinder.

[0013] The camera's rotation and translation values ​​are generated based on the actual rotation and translation increments of each frame.

[0014] The camera's rotation and translation are filtered using a primary extended Kalman filter to obtain the camera's pose and its covariance matrix, wherein the pose includes the rotation and translation.

[0015] The heading angle, camera pose, and pose covariance matrix acquired by the inertial measurement unit are subjected to extended Kalman filtering to obtain the final pose and pose covariance matrix.

[0016] Optionally, feature points are extracted from the current frame, and feature point descriptors are calculated, specifically including:

[0017] Convert the current frame to grayscale.

[0018] Median filtering is applied to the current frame after grayscale processing.

[0019] Perform contrast stretching on the current frame after median filtering;

[0020] The corner points in the current frame after contrast stretching are extracted as feature points, and feature point descriptors are calculated; the feature point descriptors include the feature point position, index number, and pixel information within the specified range of the feature point.

[0021] Optionally, corner points in the current frame after contrast stretching are extracted as feature points, specifically including:

[0022] An image edge corner feature extraction algorithm is used to extract corner points in the current frame after contrast stretching, and the extracted corner points are used as feature points.

[0023] Optionally, the camera's rotation increment and translation increment are determined based on the matched feature points, specifically including:

[0024] Using the horizontal coordinate of a feature point in the previous frame as the x-coordinate and the distance between the feature point in the previous frame and the feature point in the current frame that matches the feature point as the y-coordinate, we obtain the coordinate data of the feature point in the current frame.

[0025] The linear least squares method is used to fit the coordinate data of all matched feature points in the current frame to obtain the fitted linear equation.

[0026] The slope of the linear equation is used as the camera's rotation increment, and the intercept of the linear equation is used as the camera's translation increment.

[0027] Optionally, the motion trajectory of the mobile device is generated based on the actual rotation angle increment and actual translation increment of each frame, specifically including:

[0028] The actual rotation increment and actual translation increment of each frame are accumulated. The heading angle in the motion trajectory of the mobile device is determined based on the accumulated rotation increment, and the coordinates in the motion trajectory of the mobile device are determined based on the accumulated translation increment.

[0029] Optionally, the device also includes a housing and a glass panel, the housing including a window, the glass panel being fixed at the window, the camera being fixed inside the housing, and the camera lens being aimed at the window.

[0030] Optionally, both supplementary lighting strips are attached to the glass.

[0031] Optionally, it also includes four support studs, one end of each support stud being fixedly connected to the housing and the other end being fixedly connected to the data processor.

[0032] Optionally, the mobile device includes vehicles and robots.

[0033] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0034] This invention features two supplementary lighting strips located on either side of the camera. These strips provide supplementary lighting for the camera, reducing the impact of lighting variations on road image acquisition. Based on a real-time determined similarity ratio coefficient, the rotation and translation increments of each frame are converted into actual rotation and translation increments, improving the accuracy of the odometer. Furthermore, an extended Kalman filter algorithm is used to fuse visual odometer information with heading angle information provided by the inertial measurement unit, effectively reducing noise interference, improving positioning accuracy, and thus enhancing the accuracy of the visual odometer. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A schematic diagram of a visual odometry structure that integrates inertial measurement unit information is provided in an embodiment of the present invention;

[0037] Figure 2 for Figure 1 An exploded view of a visual odometry structure that integrates inertial measurement unit information, provided as an embodiment of the present invention.

[0038] Symbol explanation:

[0039] 1-Inertial Measurement Unit, 2-Data Processor, 3-Support Stud, 4-Monocular Camera, 5-Supplemental Light Strip, 6-Glass, 7-Housing. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] The purpose of this invention is to provide a visual odometry that integrates information from an inertial measurement unit, thereby improving the accuracy of the visual odometry.

[0042] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] like Figure 1 and Figure 2 As shown, the present invention provides a visual odometry system that integrates inertial measurement unit information, comprising: a data processor 2, a camera, two supplementary light strips 5, and an inertial measurement unit 1 (IMU sensor). The camera is fixed to the bottom of the mobile device, and the two supplementary light strips 5 are respectively located on both sides of the camera; the inertial measurement unit 1 is fixedly connected to the data processor 2.

[0044] The camera acquires real-time road surface video and sends the acquired road surface video to the data processor 2; the inertial measurement unit 1 is used to acquire the heading angle of the camera and send the acquired heading angle of the camera to the data processor 2.

[0045] The z-axis of inertial measurement unit 1 faces upward, and the y-axis faces the direction of travel of the mining rubber-tired vehicle.

[0046] The visual odometry system that integrates information from the inertial measurement unit 1 according to the present invention further includes a housing 7 and a glass 6. The housing 7 includes a window, the glass 6 is fixed at the window, and the camera is fixed inside the housing 7 with the camera lens aimed at the window. Specifically, the housing 7 is a mine explosion-proof housing.

[0047] Both supplementary light strips 5 are attached to the glass 6 to provide supplementary lighting to the outside of the housing 7.

[0048] The visual odometry that integrates information from the inertial measurement unit 1 according to the present invention also includes four support studs 3, one end of each support stud 3 being fixedly connected to the housing 7 and the other end being fixedly connected to the data processor 2.

[0049] The mobile devices include vehicles and robots.

[0050] In a specific embodiment, the entire device (visual odometer) is installed at the bottom of the mining rubber-wheeled vehicle.

[0051] The data processor 2 is used for:

[0052] Initialize the image storage matrix to store the upcoming video frames.

[0053] Read video frames from the road surface video and record the currently read video frame as the current frame.

[0054] Extract feature points from the current frame and calculate feature point descriptors.

[0055] Based on the feature point descriptors of the current frame and the previous frame, feature point matching is performed on the feature points of the current frame and the feature points of the previous frame.

[0056] The rotation increment and translation increment of the camera are determined based on the matched feature points; the translation increment is the pixel increment within the imaging plane of the camera.

[0057] Based on the similarity ratio coefficient, the rotation increment and translation increment of each frame are converted into the actual rotation increment and the actual translation increment; the similarity ratio coefficient is determined based on the actual distance between the camera and the ground measured in real time by the laser rangefinder.

[0058] The camera's rotation and translation values ​​are generated based on the actual rotation and translation increments of each frame.

[0059] The camera's rotation and translation are filtered using a primary extended Kalman filter to obtain the camera's pose and its covariance matrix, wherein the pose includes the rotation and translation.

[0060] The heading angle, camera pose, and pose covariance matrix acquired by the inertial measurement unit are subjected to extended Kalman filtering to obtain the final pose and final pose covariance matrix.

[0061] Repeat the above process to achieve mileage acquisition by the visual odometry.

[0062] In the initial extended Kalman filtering, a laser is emitted from a total station onto a vehicle equipped with the device. A high-precision value is obtained by laser ranging and used as the true value. Then, the device is used to measure the position value multiple times at the same location. The deviation between this measured value and the total station measurement value is then calculated to obtain the covariance matrix.

[0063] In this embodiment, the heading angle acquired by the inertial measurement unit and the pose after primary extended Kalman filtering are input together into the extended Kalman filtering algorithm for mutual compensation and correction, thereby achieving filtering and noise reduction and obtaining the final pose and its covariance matrix.

[0064] As a specific implementation method, feature points in the current frame are extracted, and feature point descriptors are calculated, specifically including:

[0065] The current frame is converted to grayscale to reduce the amount of data processing.

[0066] Median filtering is applied to the current frame after grayscale processing to remove noise from the image.

[0067] After median filtering, the current frame image is contrast stretched. Since the acquired image is in a dark environment, even with external lighting, the road surface texture in the image does not meet the clarity required for feature recognition. Therefore, in order to increase the contrast of the road surface texture, the contrast stretching method is used to increase the image clarity.

[0068] The corner points in the current frame after contrast stretching are extracted as feature points, and feature point descriptors are calculated; the feature point descriptors include the feature point position, index number, and pixel information within the specified range of the feature point.

[0069] As a specific implementation method, corner points in the current frame after contrast stretching are extracted as feature points, specifically including:

[0070] An image edge corner feature extraction algorithm is used to extract corner points in the current frame after contrast stretching, and the extracted corner points are used as feature points.

[0071] Specifically, the image edge corner feature extraction algorithm is the ORB (Oriented FAST and Rotated BRIEF) feature extraction algorithm.

[0072] In a specific implementation, feature point matching is performed on the feature points of the current frame and the previous frame based on the feature point descriptors. Specifically, this includes matching the feature points in the descriptors with the descriptors in the previous video frame based on the feature point location information, index number, and surrounding pixel information. Feature points in the descriptors of two frames that are close in size are considered the same feature point. If the Hamming distance between the descriptors of two frames is less than twice the minimum distance, a match is incorrect; if the Hamming distance is greater than or equal to twice the minimum distance, a match is successful.

[0073] In a specific implementation, the camera's rotation increment and translation increment are determined based on the matched feature points, specifically including:

[0074] Calculate the distance between the feature points before and after the change following the matching, as well as the horizontal coordinate (x) of the feature point pixel in the previous frame. Here, the horizontal coordinate refers to the horizontal direction in the image.

[0075] The coordinates (x, y) of a feature point in the previous frame are obtained by taking its horizontal coordinate as the x-coordinate and the distance between that feature point in the previous frame and the matching feature point in the current frame as the y-coordinate. More specifically, this step establishes a coordinate system using the horizontal coordinate of the feature point pixel as the x-coordinate and the distance before and after the feature point change as the y-coordinate, and then inputs all the data into this coordinate system.

[0076] The linear least squares method is used to fit the coordinate data of all matched feature points in the current frame to obtain the fitted linear equation.

[0077] The slope of the linear equation is used as the camera's rotation increment, and the intercept of the linear equation is used as the camera's translation increment.

[0078] As a specific implementation method, the motion trajectory of the mobile device is generated based on the actual rotation angle increment and actual translation increment of each frame, specifically including:

[0079] The actual rotation increment and actual translation increment of each frame are accumulated. The heading angle in the motion trajectory of the mobile device is determined based on the accumulated rotation increment, and the coordinates in the motion trajectory of the mobile device are determined based on the accumulated translation increment.

[0080] This invention mounts a visual odometer on the underside of a vehicle, facing downwards to collect road surface texture information. The road surface texture information is rich in feature points, and with the addition of weak supplemental lighting, the sensitivity of visual odometers to light intensity can be effectively solved. The visual odometer does not contact the ground, allowing for flexible installation; changes in tire size and slippage do not affect its measurement accuracy. Furthermore, this invention incorporates an inertial measurement unit (IMU), and improves positioning accuracy by fusing positioning information from the visual odometer and IMU using an extended Kalman filter algorithm. Mounted on the underside of the vehicle with the camera facing downwards, the vehicle blocks the shadows created by uneven streetlight illumination in the tunnel, and there is no need to collect image information in areas of changing lighting, thus reducing the impact of ambient light on the odometer. Since the shadowed areas under the vehicle are relatively dark, supplemental lighting installed within the odometer provides appropriate illumination to improve the clarity of the road surface texture information. Part of the light reflected from the road surface is collected by the camera and then processed into images. This device is mainly used in underground coal mine environments, employing a visual odometer method that identifies road surface texture to solve the slippage problem of wheeled odometers, while also addressing the sensitivity of visual odometers to light intensity.

[0081] The technical effects of this invention are as follows:

[0082] (1) This invention does not require base station positioning and is not affected by uneven lighting in the well. Appropriate supplementary lighting can improve the accuracy of feature point matching. The road surface texture information feature points are rich, making the device mileage estimation more stable.

[0083] (2) This invention is easy to install and does not require other mechanical transmission devices. Compared with wheeled odometers, this device does not contact the ground, thus solving the cumulative error caused by tire size changes and slippage.

[0084] (3) The visual algorithm in this invention adopts the inter-frame matching algorithm of feature points, which does not require the neural network deep learning algorithm and has a faster calculation speed; the inertial measurement unit provides more accurate heading angle information, and the extended Kalman filter algorithm is used to fuse the visual odometer information and the heading angle information provided by the inertial measurement unit, which effectively reduces noise interference and improves positioning accuracy.

[0085] (4) The present invention adopts a coal mine explosion-proof design and is equipped with explosion-proof features, so it can be used in underground environments.

[0086] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0087] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A visual odometry system that integrates inertial measurement unit information, characterized in that, include: The device includes a data processor, a camera, two fill light strips, and an inertial measurement unit. The camera is fixed to the bottom of the mobile device, and the two fill light strips are located on both sides of the camera. The inertial measurement unit is fixedly connected to the data processor; The camera captures real-time video of the road surface and sends the captured video to the data processor. The inertial measurement unit is used to acquire the heading angle of the camera and send the acquired heading angle of the camera to the data processor; The data processor is used for: Read video frames from the road surface video and record the currently read video frame as the current frame image; Extract feature points from the current frame and calculate feature point descriptors; Based on the feature point descriptors of the current frame and the previous frame, feature point matching is performed on the feature points of the current frame and the feature points of the previous frame. The rotation increment and translation increment of the camera are determined based on the matched feature points; the translation increment is the pixel increment within the imaging plane of the camera. Based on the similarity ratio coefficient, the rotation increment and translation increment of each frame are converted into actual rotation increment and actual translation increment; the similarity ratio coefficient is determined based on the actual distance between the camera and the ground measured in real time by the laser rangefinder. The camera's rotation and translation values ​​are generated based on the actual rotation and translation increments of each frame. The camera's rotation and translation are filtered using a primary extended Kalman filter to obtain the camera's pose and its covariance matrix, wherein the pose includes the rotation and translation. The heading angle, camera pose, and pose covariance matrix acquired by the inertial measurement unit are subjected to extended Kalman filtering to obtain the final pose and pose covariance matrix.

2. The visual odometry system that integrates inertial measurement unit information according to claim 1, characterized in that, Extract feature points from the current frame and calculate feature point descriptors, specifically including: Convert the current frame to grayscale. Median filtering is applied to the current frame after grayscale processing. Perform contrast stretching on the current frame after median filtering; The corner points in the current frame after contrast stretching are extracted as feature points, and feature point descriptors are calculated; the feature point descriptors include the feature point position, index number, and pixel information within the specified range of the feature point.

3. The visual odometry system that integrates inertial measurement unit information according to claim 2, characterized in that, The corner points in the current frame after contrast stretching are extracted as feature points, specifically including: An image edge corner feature extraction algorithm is used to extract corner points in the current frame after contrast stretching, and the extracted corner points are used as feature points.

4. The visual odometry system that integrates inertial measurement unit information according to claim 1, characterized in that, The camera's rotation and translation increments are determined based on the matched feature points, specifically including: Using the horizontal coordinate of a feature point in the previous frame as the x-coordinate and the distance between the feature point in the previous frame and the feature point in the current frame that matches the feature point as the y-coordinate, we obtain the coordinate data of the feature point in the current frame. The linear least squares method is used to fit the coordinate data of all matched feature points in the current frame to obtain the fitted linear equation. The slope of the linear equation is used as the camera's rotation increment, and the intercept of the linear equation is used as the camera's translation increment.

5. The visual odometry system that integrates inertial measurement unit information according to claim 1, characterized in that, The motion trajectory of the mobile device is generated based on the actual rotation angle increment and actual translation increment of each frame, specifically including: The actual rotation increment and actual translation increment of each frame are accumulated. The heading angle in the motion trajectory of the mobile device is determined based on the accumulated rotation increment, and the coordinates in the motion trajectory of the mobile device are determined based on the accumulated translation increment.

6. The visual odometry system that integrates inertial measurement unit information according to claim 1, characterized in that, It also includes a housing and glass, the housing having a window, the glass being fixed at the window, the camera being fixed inside the housing, and the camera's lens being aimed at the window.

7. The visual odometry system that integrates inertial measurement unit information according to claim 6, characterized in that, Both supplementary lighting strips are attached to the glass.

8. The visual odometry system that integrates inertial measurement unit information according to claim 6, characterized in that, It also includes four support studs, one end of each support stud is fixedly connected to the housing, and the other end is fixedly connected to the data processor.

9. The visual odometry system that integrates inertial measurement unit information according to claim 1, characterized in that, The mobile devices include vehicles and robots.

Images