VINS-Fusion-based robot vision inertial navigation collaborative navigation algorithm

By combining vision sensors and IMUs, using deep learning VGG-16 network for loopback detection, the problem of insufficient navigation accuracy of a single sensor in complex environments is solved, and more efficient feature point processing and map construction are achieved.

CN118913281APending Publication Date: 2024-11-08KUNMING UNIV OF SCI & TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410968422.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

A single sensor has problems of drift and inaccurate positioning when used for navigation of mobile robots, especially in complex environments, resulting in feature point matching errors and difficulty in building maps.

Method used

Using the visual inertial navigation collaborative navigation algorithm based on VINS-Fusion, image preprocessing, IMU preintegration and nonlinear optimization processing are performed through the combination of vision sensors and IMUs, and loopback detection and mapping are performed in combination with deep learning VGG-16 network to optimize Feature point extraction and matching.

Benefits of technology

It improves positioning accuracy and map construction accuracy, simplifies the calculation amount, enhances the quality of feature points, and realizes more accurate loop detection and system mapping.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118913281A_ABST
    Figure CN118913281A_ABST
Patent Text Reader

Abstract

The invention discloses an indoor robot vision inertial navigation collaborative navigation algorithm based on Vins-Fusion, and belongs to the technical field of robot vision positioning navigation. According to the algorithm, after a visual sensor is preprocessed, the visual sensor is used for collecting an image, orb feature point extraction and feature point homogenization processing are carried out, a self-screening rejection algorithm and an offset space-based local consistency purification algorithm are used for carrying out optimization processing on matching point pairs, and then an RANSAC algorithm is used for removing mismatching points. The method has the advantages that the number of feature points is effectively reduced, the quality of the feature points is improved, the system mapping calculation amount is reduced, the IMU is subjected to preprocessing and pre-integration to obtain the processed IMU data, the processed image data and the IMU data are subjected to nonlinear optimization processing in a visual inertial navigation information coupling part, and the visual inertial navigation information coupling performance is improved. VGG-16 network loopback detection based on deep learning is used for the image obtained after nonlinear optimization, loopback detection is completed more accurately, the positioning and mapping precision is effectively improved, and the method has a good effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a navigation algorithm for indoor robot vision-inertial navigation collaboration based on VINS-Fusion, and belongs to the technical field of robot vision positioning and navigation. Background Art

[0002] In today's era of rapid technological development, the rapid development of automatic mapping and intelligent technology has injected unprecedented vitality into cutting-edge fields such as mobile robots and indoor AGVs. One of the core systems of these robots is navigation. Simultaneous Localization and Mapping (SLAM) technology is an important direction of mobile robot positioning and navigation technology. As a key technology for robot automatic navigation, SLAM technology can be applied to different mobile robot platforms and has important value.

[0003] However, using only a single sensor to position the mobile robot for navigation and mapping will result in drift, inaccurate positioning, and other problems, and the true trajectory of movement cannot be obtained. This is the limitation of a single sensor, and this limitation makes single sensors not widely used in practical applications. In order to solve this limitation, solutions that utilize the characteristics of different sensors to complement each other's advantages have gradually emerged. Among many sensors, the combination of IMU and camera has good robustness and low cost. The more famous one is the Vins-Fusion algorithm that was open sourced in 2019. Because this method is used in some complex environments, small indoor spaces, or complex environments such as a large amount of accumulation, and if there is no natural light, the brightness of the image obtained by the camera will be uneven, which will greatly hinder the extraction of feature points. The messy feature points are very likely to cause feature point matching errors that affect subsequent feature matching and map construction, making it difficult to position the mobile robot.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The present invention provides a navigation algorithm for indoor robots based on VINS-Fusion visual inertial navigation collaboration, which combines visual sensors with IMUs for SLAM optimization of indoor robots.

[0006] The technical solution of the present invention is:

[0007] According to a first aspect of the present invention, there is provided an indoor navigation algorithm for indoor robot visual inertial navigation collaboration based on VINS-Fusion, comprising: preprocessing a visual sensor; acquiring an image based on the preprocessed visual sensor; preprocessing the acquired image to obtain preprocessed image data; preprocessing the IMU; performing an IMU pre-integration operation based on the preprocessed IMU; performing nonlinear optimization processing on the preprocessed image data and the data output by the IMU pre-integration operation in a sliding window to obtain a nonlinearly optimized image; performing loop closure detection and mapping based on a deep learning VGG-16 network on the image acquired after the nonlinear optimization processing.

[0008] The preprocessing of the visual sensor includes: selecting a calibration board; connecting a monocular camera as a visual sensor to an Ubuntu system; running a camera node, publishing a 200 Hz frequency topic to record a calibration package; fully rotating the calibration board to ensure that the calibration board is always in the camera field of view; and using the kalibr function package to calibrate the recorded calibration package to obtain a calibration result.

[0009] The method comprises: performing image acquisition based on a preprocessed visual sensor; preprocessing the acquired image to obtain preprocessed image data, including: performing image acquisition using a monocular camera and performing ORB feature point extraction; performing homogenization on the extracted feature points; performing feature point matching using a Hamming algorithm for feature points after homogenization of adjacent frame images; performing preliminary screening of matching points using a self-screening and elimination algorithm; performing fine screening of matching points after preliminary screening using an offset space-based local consistency purification algorithm; and removing mismatched points using a RANSAC algorithm for the finely screened matching points to obtain preprocessed image data.

[0010] The homogenization processing of the extracted feature points includes: distributing the feature points on a two-dimensional plane, dividing the plane into four sub-regions of equal size, each sub-region only containing the feature points in the region; judging the number of feature points in each divided sub-region: if the number of feature points in the sub-region is equal to 0, deleting the sub-region, if the number of feature points in the sub-region is equal to 1, storing the sub-region, if the number of feature points in the sub-region exceeds 1, continuing to divide the sub-region into four sub-regions of equal size and continuing the above judgment until the number of stored sub-regions is greater than or equal to a fixed threshold.

[0011] The self-screening and elimination matching point algorithm performs preliminary screening, and its calculation formula is as follows:

[0012] h={[h min *S,7]} -1

[0013] hx <2*h max *h

[0014] Among them, S is the scale factor; h is the parameter for determining whether the matching point can be used, and its value is determined by the minimum hamming distance h min and scale factor S, "[]" is the maximum value operation, h x is the Hamming distance between the matching points to be judged, h max is the maximum hamming distance, h min is the minimum hamming distance, and the result is used to determine whether the matching point pair is available.

[0015] The preprocessing of the IMU is specifically as follows: selecting the IMU, connecting the IMU to the same Ubuntu system as the visual sensor; running the IMU, recording the data of the topic published by the IMU to form a calibration package, and recording for a preset time; then using the kalibr function package to calibrate the IMU according to the recorded calibration package.

[0016] The method performs loop detection and mapping based on a deep learning VGG-16 network on the image obtained after nonlinear optimization processing, including: setting a similarity threshold; obtaining images of the current frame and the previous frame, and processing them using a pre-trained VGG-16 network to obtain a feature vector; using a t-SNE algorithm to project the feature vector obtained using the VGG-16 network into a low-dimensional space for processing to obtain a low-dimensional feature vector; calculating the cosine value between two low-dimensional feature vectors as the cosine similarity between the two frames; if the cosine value of the low-dimensional feature vector between the current frame and the previous frame is greater than the similarity threshold, it is judged as a loop frame, and the optimization algorithm is used to correct the trajectory to correct the accumulated error caused by error feedback; if the cosine value of the low-dimensional feature vector between the current frame and the previous frame is less than or equal to the similarity threshold, loop detection is continued; and position information is output to construct a real-time map.

[0017] According to a second aspect of the present invention, there is provided an indoor navigation system for indoor robot visual inertial navigation collaboration based on VINS-Fusion, comprising: a first module for preprocessing a visual sensor; performing image acquisition based on the preprocessed visual sensor; preprocessing the acquired image to obtain preprocessed image data; a second module for preprocessing an IMU; performing an IMU pre-integration operation based on the preprocessed IMU; a third module for performing nonlinear optimization processing on the preprocessed image data and the data output by the IMU pre-integration operation within a sliding window to obtain a nonlinearly optimized image; and a fourth module for performing loop detection and mapping based on a deep learning VGG-16 network on the image acquired after the nonlinear optimization processing.

[0018] The beneficial effects of the present invention are as follows: the present invention is a SLAM algorithm for coupling a visual sensor with an inertial navigation system, the algorithm uses a feature point optimization algorithm that adapts to homogenized feature points, and at the same time adds a method for self-screening and eliminating feature points, on this basis, the number of feature points is improved and simplified, the quality of feature points is improved, and the amount of calculation for system mapping is reduced, and an objective function for optimizing key frames and non-key frames is proposed in the joint initialization part of the visual sensor and inertial navigation system, ensuring that the variables to be estimated are in the best state, and finally a loop detection based on a deep learning VGG-16 network is proposed, and a convolutional neural network framework can be used to detect deep features, and loop detection can be completed more accurately, thereby effectively improving the accuracy of positioning and mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the algorithm of the present invention;

[0020] Figure 2 It is a schematic diagram of homogenized feature points;

[0021] Figure 3 is the camera calibration result of the present invention;

[0022] Figure 4 It is the flow chart of loop detection algorithm. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily without conflict.

[0024] Example 1: Figure 1-Figure 4 As shown, according to the first aspect of this embodiment, a VINS-Fusion-based indoor robot visual inertial navigation collaborative navigation algorithm is provided for a mobile robot, including: preprocessing a visual sensor; acquiring an image based on the preprocessed visual sensor; preprocessing the acquired image to obtain preprocessed image data; preprocessing the IMU; performing an IMU pre-integration operation based on the preprocessed IMU; performing nonlinear optimization processing on the preprocessed image data and the data output by the IMU pre-integration operation in a sliding window to obtain a nonlinearly optimized image; performing loop detection and mapping based on a deep learning VGG-16 network on the image acquired after the nonlinear optimization processing.

[0025] Furthermore, the preprocessing of the visual sensor includes: selecting a calibration board; connecting the monocular camera as a visual sensor to the Ubuntu 18.04 system; running the camera node, publishing a 200Hz frequency topic to record the calibration package; fully rotating the calibration board to ensure that the calibration board is always in the camera's field of view; using the kalibr function package to calibrate the recorded calibration package to obtain the calibration result. Exemplarily, the monocular camera uses an RGB-D camera. Figure 3 The calibration results are shown (the upper part is the calibration data display, and the lower part is the graph generated based on the data.

[0026] Further, the image is collected based on the preprocessed visual sensor; the collected image is preprocessed to obtain the preprocessed image data, including: using a monocular camera to collect the image and extracting ORB feature points; homogenizing the extracted feature points, the principle of which is as follows Figure 2 As shown in the figure; the feature points after homogenization processing of adjacent frame images are matched using the Hamming algorithm; the matching points are preliminarily screened using the self-screening and elimination algorithm; the matching points after the preliminary screening are refined using the offset space local consistency purification algorithm; for the matching points after the refined screening, the RANSAC algorithm is used to remove the mismatched points to obtain the preprocessed image data.

[0027] Specifically:

[0028] A camera is used to take images of indoor environments in real time, and an ORB algorithm is used to extract feature points (the traditional SURF and SIFT feature point extraction algorithms have a slow extraction speed, while the ORB algorithm is 100 times that of SIFT and 10 times that of SURF; therefore, the present invention adopts the ORB algorithm); ORB is improved by the FAST feature detection algorithm and the BRIEF feature description algorithm, and determines whether point p is a corner point by detecting the grayscale values ​​of the pixels around point p. The brightness of point p has been set in advance, and only the brightness of point p needs to be compared during the judgment: first, a large number of coarse features are extracted, a point p is selected from the image, and a circle with a radius of 3 pixels is drawn with point p as the center. Assuming that the brightness of point p is Ip, if the grayscale values ​​of 12 consecutive pixels on the circumference exceed the grayscale value of point p [Ip-T, Ip+T] (T=20%Ip), then point p is considered to be a feature point.

[0029] Furthermore, the homogenization processing of the extracted feature points includes: distributing the feature points on a two-dimensional plane, dividing the plane into four sub-regions of equal size, each sub-region only containing the feature points in the region; judging the number of feature points in each divided sub-region: if the number of feature points in the sub-region is equal to 0, deleting the sub-region; if the number of feature points in the sub-region is equal to 1, storing the sub-region; if the number of feature points in the sub-region exceeds 1, further dividing the sub-region into four sub-regions of equal size and continuing the above-mentioned judgment (i.e., judging whether the number of feature points in the divided sub-regions satisfies equal to 0, equal to 1 or exceeds 1) until the number of stored sub-regions is greater than or equal to a fixed threshold (in the embodiment of the present invention, the fixed threshold is set to 25, and when counting the number of sub-regions, the deleted sub-regions are not counted, i.e., the number of sub-regions equal to 1 or exceeding 1 is counted, such as Figure 2 is an example).

[0030] When the number of stored sub-regions reaches a fixed threshold, for sub-regions with more than 1 feature points, the feature point with the highest Harris response value in the sub-region is selected and retained, and the remaining feature points are deleted, thus obtaining uniform ORB feature points.

[0031] Furthermore, the self-screening and elimination matching point algorithm performs preliminary screening, and its calculation formula is as follows:

[0032] h={[h min *S,7]} -1 (1)

[0033] h x <2*h max *h(2)

[0034] Wherein, S is a scale factor, which is set to about 0.1. In the embodiment of the present invention, it is 0.1; h is a parameter for determining whether a matching point can be used, and its value is determined by the minimum hamming distance h min and scale factor S, "[]" is the maximum value operation, h x is the Hamming distance between the matching points to be judged, h max is the maximum hamming distance, h min is the minimum Hamming distance. The matching point pair is determined based on the result of formula (2). If it is satisfied, the matching point is retained, otherwise it is removed.

[0035] Furthermore, the matching point purification algorithm based on local consistency of offset space performs final screening, and the steps are as follows:

[0036] (1) Map the initial matching points to the offset space.

[0037] For the two images to be matched, quantize the coordinates of the two images to the space (0, 1, ..., N x )*(0, 1, …, M y ), where N is the maximum value of the quantized horizontal coordinate, M is the maximum value of the quantized vertical coordinate, and the quantized coordinates (xi, yi) of the matching points in the two images to be matched are obtained by rounding off and quantizing the original image coordinates of the matching points in the two images to be matched.

[0038] (2) Matching points that maintain local consistency in the offset space will fall into one or several adjacent bins in the offset space, and mismatched points with randomness and noise interference will fall into sparse bins.

[0039] Construct the corresponding offset coordinate space (-N) according to the quantized coordinates of the matching points x , -N x +1,…,N x )*(-M y , -M y +1,…,M y ), including (2N x +1)*(2M y +1) bins. A bin is a small grid divided in the offset space.

[0040] (3) Calculate the number of matching points falling into different bins

[0041] For each pair of matching points, calculate the offset of their quantized coordinates (△x, △y) = (X I -X I’ , Y I -Y I’ ), that is, the quantized coordinates of the matching point in the reference image I (X I , Y I ) minus the quantized coordinates (X I' ,Y I' ) and maps the result to the bin corresponding to the offset space;

[0042] (4) The initial matching points that fall into the bin are retained, and the mismatching points that are randomly and sparsely distributed in the bin are removed for purification.

[0043] The number of matching points falling into different bins is accumulated, and according to their distribution in the offset space, more initial matching points in the bin are retained, and matching points that are randomly and sparsely distributed in the bin are eliminated.

[0044] Furthermore, the use of the RANSAC algorithm to remove mismatched feature points includes:

[0045] Among the matching point pairs obtained in the previous steps, four non-collinear matching point pairs are randomly selected as inliers, and the remaining matching point pairs are outliers;

[0046] The transformation matrix H is calculated based on four pairs of matching points randomly selected, which is denoted as model M. The model M is:

[0047]

[0048] Let h 33 =1 to get the normalized matrix, and after calculation, k, x′, y′ are as follows:

[0049] k=h 31 x+h 32 y+1 (4)

[0050]

[0051] Among them, k is the scale parameter, x, y and x', y' are the corresponding coordinates in the image, because let h 33 =1 to obtain the normalized matrix, so the degree of freedom of the transformation matrix H is 8.

[0052] Use the estimated model M to verify whether other external points conform to the model. The points that conform to the model are called internal points, and the points that do not conform to the model are called external points. The number of internal points that conform to the model M is counted;

[0053] Continue to randomly select four non-collinear matching points to calculate the transformation matrix H, obtain the remaining external points for model estimation, count the internal points that meet the requirements of the model, and finally count a model composed of the most matching points that are considered to be internal points. Select this model as the final model and remove the external points as mismatched points.

[0054] Furthermore, the IMU is preprocessed as follows: select the IMU, connect the IMU to the same Ubuntu18.04 system as the visual sensor; run the IMU, record the data of the topic published by the IMU to form a calibration package, and the recording time is 2 hours; then use the kalibr function package to calibrate the IMU according to the recorded calibration package.

[0055] Furthermore, the steps of performing IMU pre-integration operation based on the pre-processed IMU are as follows:

[0056] The linear acceleration and angular velocity measured by the IMU are integrated over a period of time to obtain displacement, velocity and attitude information, reduce noise propagation, and improve the accuracy and robustness of the system. First, ignoring the influence of the earth's rotation, the IMU measurement model at time t is as follows:

[0057]

[0058] Among them, ^ represents the measured value, which is affected by bias and noise. b ωt 、b αt , Respectively represent the angular velocity measurement of the gyroscope in the body coordinate system at time t, the true value of the angular velocity of the gyroscope in the body coordinate system, the acceleration measurement of the accelerometer in the body coordinate system, the actual acceleration in the world coordinate system, the gravity acceleration in the world coordinate system (constant by default), the measurement noise of the gyroscope in the body coordinate system, the measurement noise of the accelerometer in the body coordinate system, and the posture converted from the world coordinate system to the body coordinate system; n ω and n α They represent the measurement value noise of the gyroscope in the body coordinate system and the measurement value noise of the accelerometer in the body coordinate system respectively.

[0059] Secondly, it is assumed that there is IMU data between two consecutive frames, and the time interval of the data is [t k , t k+1 ], then in this time interval, according to the world coordinate system and body coordinate system of IMU, combined with the data of gyroscope and accelerometer, the corresponding [t k , t k+1 ] within the displacement speed attitude As shown below:

[0060]

[0061] in, Represents the acceleration value of the accelerometer in the body coordinate system at time t.

[0062] The displacement calculated at this time speed attitude It is the value in the world coordinate system. When optimizing in the world coordinate system, only the IMU data between frames needs to be considered, and the data in the body coordinate system does not need to be considered.

[0063] By pre-integrating the IMU, the relative measurement information between frames is added as a constraint to the optimization framework, which greatly improves the optimization speed.

[0064] Furthermore, the preprocessed image data and the data output by the IMU pre-integration operation are subjected to nonlinear optimization processing in a sliding window to obtain a nonlinearly optimized processed image, including:

[0065] As the system runs, more data is received from the joint initialization of the camera and IMU. To ensure the real-time performance of the system, a sliding window nonlinear optimization with a defined sliding window size is used. The state variables in the sliding window include (position P, velocity V, attitude q, gyroscope bias, acceleration bias), external parameters between the IMU and the camera, m+1 MapPoint inverse depths, and other state variables for optimization. The state variables are as follows:

[0066]

[0067]

[0068] in, is the state variable to be optimized, χ n is the state variable to be estimated in the key frame, λ m It is the inverse of the depth of a map point observed within the sliding window; is the IMU state (position, velocity, rotation, acceleration bias, gyroscope bias) of the k-th frame image, is the IMU position, is the IMU speed, is the IMU rotation, b a is the acceleration bias, b g is the gyroscope bias; represents the camera extrinsic parameters, It represents the position from the origin of the camera coordinate system to the origin of the reference coordinate system. Represents the rotation from the origin of the camera coordinate system to the origin of the reference coordinate system.

[0069] If the acquired data is judged as a key frame and recorded as the next new frame, the key frame objective function is used for optimization calculation. The key frame objective function is as follows:

[0070] The key frame objective function is:

[0071]

[0072] Among them, ||γ P -H P X|| 2 To marginalize the prior information, is the IMU measurement residual, is the visual reprojection residual, and the optimal value of the state variable is found by solving the minimum value of the objective function.

[0073] If the acquired data is judged as a non-key frame, the visual measurement data of the next-newest frame is discarded but the IMU measurement data is retained. The IMU data between the next-newest frame and the current frame image and the image are substituted into the non-key frame objective function as a set of new data for optimization calculation. The non-key frame objective function is as follows:

[0074] The non-keyframe objective function is:

[0075]

[0076] in, Represents the IMU measurement residual between K-1 and K+1 image frames.

[0077] Continue to input the next frame image, the key frame will be optimized for the key frame objective function, and the non-key frame will be optimized for the non-key frame objective function.

[0078] Furthermore, if Figure 4 As shown, the image obtained after nonlinear optimization processing is subjected to loop detection and mapping based on deep learning VGG-16 network, including: setting a similarity threshold; obtaining the image of the current frame and the previous frame, processing them using the pre-trained VGG-16 network, and obtaining the feature vector J i =VGG-16(P i );For the feature vector J obtained using the VGG-16 network i The t-SNE algorithm is used to project it into a low-dimensional space for processing to obtain a low-dimensional feature vector J i ′=t-SNE(P i );Calculate the cosine value between two low-dimensional feature vectors K = COS (J ' i , J′ i+1 ) as the cosine similarity between the two frames; if the cosine value of the low-dimensional feature vector between the current frame and the previous frame is greater than the similarity threshold, it is judged as a loop frame, and the optimization algorithm is used to correct the trajectory and the accumulated error caused by error feedback. If the cosine value of the low-dimensional feature vector between the current frame and the previous frame is less than or equal to the similarity threshold, loop detection continues. Output pose information to build a real-time map. The optimization algorithm is a factor graph optimization algorithm.

[0079] According to a second aspect of an embodiment of the present invention, there is provided an indoor navigation system for indoor robot visual inertial navigation collaboration based on VINS-Fusion, comprising: a first module for preprocessing a visual sensor; performing image acquisition based on the preprocessed visual sensor; preprocessing the acquired image to obtain preprocessed image data; a second module for preprocessing an IMU; performing an IMU pre-integration operation based on the preprocessed IMU; a third module for performing nonlinear optimization processing on the preprocessed image data and the data output by the IMU pre-integration operation within a sliding window to obtain a nonlinearly optimized image; and a fourth module for performing loop detection and mapping based on a deep learning VGG-16 network on the image acquired after the nonlinear optimization processing.

[0080] By applying the above technical solution, it can be known that the present invention uses the visual sensor to collect images and perform orb feature point extraction and feature point homogenization processing after preprocessing the visual sensor, adopts a self-screening elimination algorithm and an offset space-based local consistency purification algorithm to optimize the matching point pairs, and then uses the RANSAC algorithm to remove the mismatched points to obtain processed image data, which effectively simplifies the number of feature points, improves the quality of feature points, and reduces the amount of system mapping calculations. At the same time, the IMU is preprocessed and pre-integrated to obtain processed IMU data, and the processed image data and IMU data are nonlinearly optimized in the visual inertial navigation information coupling part. The image obtained after nonlinear optimization uses the VGG-16 network loop detection based on deep learning to complete the loop detection more accurately, effectively improve the positioning and mapping accuracy, and has a good effect.

[0081] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An indoor navigation algorithm based on VINS-Fusion for indoor robots with visual inertial navigation, characterized in that: include: Preprocessing of visual sensors; Capture images based on pre-processed visual sensors; Preprocessing the collected images to obtain preprocessed image data; Preprocess the IMU; Perform IMU pre-integration operation based on the pre-processed IMU; The preprocessed image data and the data output by the IMU pre-integration operation are subjected to nonlinear optimization processing in a sliding window to obtain a nonlinearly optimized processed image; The images obtained after nonlinear optimization processing are subjected to loop detection and mapping based on the deep learning VGG-16 network.

2. The indoor navigation algorithm based on VINS-Fusion indoor robot visual inertial navigation collaboration according to claim 1 is characterized in that: The preprocessing of the visual sensor includes: Select the calibration board; connect the monocular camera as a visual sensor to the Ubuntu system; Run the camera node and publish a 200Hz frequency topic to record the calibration package; Rotate the calibration plate fully to ensure that it is always within the camera's field of view; Use the kalibr function package to calibrate the recorded calibration package to obtain the calibration result.

3. The indoor navigation algorithm based on VINS-Fusion indoor robot visual inertial navigation collaboration according to claim 1 is characterized in that: The image is collected by the pre-processed visual sensor; Preprocess the collected images to obtain preprocessed image data, including: Use a monocular camera to collect images and extract ORB feature points; Perform homogenization processing on the extracted feature points; The feature points of adjacent frame images after homogenization are matched using the Hamming algorithm; Use the self-screening and elimination algorithm to initially screen the matching points; Use the local consistency purification algorithm based on offset space to fine-screen the matching points after the initial screening; For the matching points after fine screening, the RANSAC algorithm is used to remove the mismatched points to obtain the preprocessed image data.

4. The indoor navigation algorithm based on VINS-Fusion for indoor robots with visual inertial navigation collaboration according to claim 3 is characterized in that: The step of homogenizing the extracted feature points comprises: Distribute the feature points on a two-dimensional plane and divide the plane into four sub-regions of equal size, each sub-region only contains the feature points in the region; The number of feature points in each divided sub-region is judged: if the number of feature points in the sub-region is equal to 0, the sub-region is deleted; if the number of feature points in the sub-region is equal to 1, the sub-region is stored; if the number of feature points in the sub-region exceeds 1, the sub-region is further divided into four sub-regions of equal size and the above judgment is continued until the number of stored sub-regions is greater than or equal to the fixed threshold.

5. The indoor navigation algorithm based on VINS-Fusion indoor robot visual inertial navigation collaboration according to claim 3 is characterized in that: The self-screening and elimination matching point algorithm performs preliminary screening, and its calculation formula is as follows: h={[h min *S,7]} -1 h x <2*h max *h Among them, S is the scale factor; h is the parameter for determining whether the matching point can be used, and its value is determined by the minimum hamming distance h min and scale factor S, "[]" is the maximum value operation, h x is the Hamming distance between the matching points to be judged, h max is the maximum hamming distance, h min is the minimum hamming distance, and the result is used to determine whether the matching point pair is available.

6. The indoor navigation algorithm based on VINS-Fusion for indoor robots and visual inertial navigation collaboration according to claim 1 is characterized in that: The IMU is preprocessed as follows: Select IMU and connect it to the same Ubuntu system as the visual sensor. Run the IMU, record the data of the topic published by the IMU to form a calibration package, and record for a preset duration; then use the kalibr function package to calibrate the IMU according to the recorded calibration package.

7. The indoor navigation algorithm based on VINS-Fusion indoor robot visual inertial navigation collaboration according to claim 1 is characterized in that: The method of performing loop detection and mapping based on a deep learning VGG-16 network on the image obtained after nonlinear optimization processing includes: Establish similarity thresholds; Get the images of the current frame and the previous frame, process them using the pre-trained VGG-16 network, and obtain the feature vector; The feature vector obtained by using the VGG-16 network is projected into a low-dimensional space using the t-SNE algorithm to obtain a low-dimensional feature vector. Calculate the cosine value between two low-dimensional feature vectors as the cosine similarity between the two frames; If the cosine value of the low-dimensional feature vector between the current frame and the previous frame is greater than the similarity threshold, it is judged as a loop frame, and the optimization algorithm is used to correct the trajectory and the accumulated error caused by error feedback. If the cosine value of the low-dimensional feature vector between the current frame and the previous frame is less than or equal to the similarity threshold, loop detection continues; Output pose information to build a real-time map.

8. An indoor navigation system based on VINS-Fusion and visual inertial navigation of indoor robots, characterized in that: include: The first module is used to pre-process the visual sensor; Capture images based on pre-processed visual sensors; Preprocessing the collected images to obtain preprocessed image data; The second module is used to pre-process the IMU; Perform IMU pre-integration operation based on the pre-processed IMU; The third module is used to perform nonlinear optimization processing on the preprocessed image data and the data output by the IMU pre-integration operation in a sliding window to obtain a nonlinearly optimized image; The fourth module is used to perform loop detection and mapping based on the deep learning VGG-16 network on the images obtained after nonlinear optimization processing.

Citation Information

Cited By

  • Nuclear environment robot monocular vision inertial navigation positioning method based on edge features

    CN120852525A