A Visual-Inertial Odometry Method Based on Unsupervised Learning

Abstract

The present invention discloses a visual inertial odometry method based on unsupervised learning, which comprises the following steps: S1: Collect input data, preprocess the image data and IMU data to synchronize the image data and IMU data; S2: Perform feature extraction to obtain the image features of the image data and the inertial navigation features of the IMU data; S3: Input the image features and inertial navigation features into a pose calculation network to output a six-degree-of-freedom pose calculation result; S4: Perform unsupervised training based on the difference between the depth map and the source image, output the training result, and obtain a trained depth network; S5: Collect new image data and input it into the trained depth network to output a positioning result. The present invention adopts a deep learning method, avoiding the complex calibration and modeling of sensors in traditional methods. At the same time, an unsupervised training method is used to solve the limitation of the data set. The whole method can make full use of the complementary information of images and inertial navigation to achieve high-precision positioning.

Description

Technical Field

[0001] The present invention relates to the technical field of visual recognition and positioning, and particularly relates to a visual inertial odometry method based on unsupervised learning. Background Art

[0002] With the progress of information science and materials science and technology, autonomous control has also flourished on this basis, and autonomous positioning technology also has more important application value in various industries. The vision-based positioning method cannot complete the positioning task in the actual application when there is a lack of light, the image is blurred or the camera collecting information is in a high-speed motion state, and the estimated pose error is large. The visual inertial odometer is a system that uses both visual information and inertial navigation information to complete the positioning task, which can make full use of the complementarity of inertial and visual information and correct each other. At the same time, since the two main types of sensors used, the camera and the inertial measurement unit, both have the characteristics of small size and low price, the visual inertial odometer system is widely used in multiple fields of autonomous positioning.

[0003] For traditional visual inertial fusion methods, the extracted features are mostly low-level features on the surface, and a large number of parameter calibrations inside and between sensors are required. Moreover, strict time synchronization is required before using image and IMU data. The noise and errors brought by data alignment will seriously affect the performance of the system. At the same time, the inertial measurement unit will be affected by factors such as sensor bias, scale factor, non-orthogonality, temperature, and noise, and its performance is greatly reduced. Therefore, complex modeling is required to eliminate the errors and noises of the IMU.

[0004] In recent years, the application of deep learning in the field of computer vision has gradually increased. Many deep learning-based models have emerged in the field of visual inertial odometry. These models can extract high-dimensional motion features of images and inertial navigation, and at the same time effectively solve problems such as difficult sensor parameter calibration, difficult extraction of motion features, and fusion of visual and inertial information. However, these methods are restricted by the training data set.

[0005] Therefore, traditional positioning methods often have problems of low accuracy and sensor errors. The positioning method based on deep learning is restricted by the training data set. The unsupervised visual inertial odometry method efficiently utilizes image and inertial navigation information through a neural network to achieve high-precision positioning, and at the same time gets rid of the restriction of the data set in an unsupervised manner. Summary of the Invention

[0006] Aiming at the above deficiencies of the prior art, the present invention provides an unsupervised training method to solve the limitation of the data set. The entire system can make full use of the complementary information of images and inertial navigation to achieve high-precision positioning, a visual inertial odometry method based on unsupervised learning.

[0007] To achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0008] Provide a visual inertial odometry method based on unsupervised learning, which includes the following steps:

[0009] S1: Collect input data, input data packet image data and IMU data, preprocess the image data and IMU data to synchronize the image data and IMU data;

[0010] S2: Extract features from the synchronized image data and IMU data to obtain the image features of the image data and the inertial navigation features of the IMU data;

[0011] S3: Input the image features and inertial navigation features into the pose calculation network, and output the pose calculation result of six degrees of freedom;

[0012] S4: Use the pose calculation result to input into the unsupervised module, construct a deep network framework, perform unsupervised training according to the difference between the depth map and the source image, output the training result, and obtain the trained deep network;

[0013] S5: Collect new image data and input it into the trained deep network to output the positioning result.

[0014] Further, the method for preprocessing the image data and IMU data in step S1 is: The camera collects adjacent two frames of images as the image data, and packs the IMU data collected during the time between collecting adjacent two frames of images together with the adjacent two frames of images as the input data to ensure the synchronization of the image data and IMU data. The synchronized IMU data is:

[0015]

[0016] where α is the acceleration on the three axes of XYZ, and ω is the angular velocity on the three axes of XYZ.

[0017] Further, step S2 includes:

[0018] S21: Input the information of adjacent two frames of synchronized images into the convolutional neural network CNN to extract image features:

[0019] Adopt the method of transfer learning to assist in building the convolutional neural network for image feature extraction. The convolutional neural network adopts the encoder structure of the FlownetSimple optical flow network, and adds a branch at the 5×5 convolutional layer of the encoder structure. The branch extracts different-level features of the network through the 7×7 spatial channel attention structure, and cascades the extracted different-level features with the features extracted by the encoder at the penultimate convolutional kernel to retain different-level features, and outputs as the extracted image features;

[0020] S22: Input the synchronized 10-frame IMU data into the long short-term memory network (LSTM). The LSTM has 2 hidden layers with 512 hidden states to complete the extraction of inertial navigation features.

[0021] Further, step S3 includes:

[0022] S31: Concatenate the extracted image features and inertial navigation features, and then analyze the data through a decoder to generate a set of weights; multiply the concatenated fused features by the weights to complete the fusion and screening of the features. The features after fusion and screening are used as the input to the pose calculation network:

[0023]

[0024] W = σ(f(wF + b))

[0025]

[0026] where F is the fused feature, F v and F i are the image feature and the inertial navigation feature respectively, is the concatenation process along the channel direction, W is the weight of the generated adjusted feature, σ is the sigmod activation function, f is the function corresponding to the decoder, w and b are both parameters learned during training, used for preliminary processing of the fused feature, and F' is the finally output screened fused feature, is the element-wise multiplication process;

[0027] S32: Use a long short-term memory network (LSTM) followed by a fully connected layer structure as the pose calculation network. Input the features after fusion and screening into the pose calculation network to output the pose calculation result of six degrees of freedom.

[0028] Further, step S4 includes:

[0029] S41: Construct a deep network. The network framework of the deep network is an encoder-decoder structure. When the camera moves in the scene, objects in adjacent images can form geometric constraints. The adjacent images respectively represent two adjacent frames of the source image I s and the target image . Use the depth map and the motion transformation matrix to construct geometric consistency:

[0030] D s (p s )K -1 p s = T t→s D t (p t )K -1 p t

[0031] Among them, D s 、D t They are the depth maps of adjacent frame images, p s , p t are the pixel points on the depth map of adjacent frames, K is the intrinsic parameter matrix of the camera, T t→s is the pose transformation matrix in the pose calculation result;

[0032] S42: Exploitation Calculate the source image I s In the target image Projection I t , by comparing the target image and Projection I t The photometric consistency constraint is constructed based on the appearance similarity of to form an unsupervised module. The loss function of the unsupervised module is:

[0033]

[0034] Among them, L p is the structural similarity loss, s is the number of the target image, and t is the number of the source image;

[0035] S43: Introduce a smoothness loss function to compensate for the photometric consistency error in textureless areas:

[0036]

[0037] Among them, L s is the smoothness loss, is a two-dimensional differential operator, D(P) and I(P) are the pixels on the depth map and source image respectively, and |·| is the absolute value of the element;

[0038] S44: Establish a total loss function based on the loss function of the unsupervised module and the smoothness loss function:

[0039] L=L p +αL s ;

[0040] Among them, α is the weighting factor;

[0041] S45: Perform unsupervised training according to the total loss function of the unsupervised module, output the training results, and obtain a trained deep network.

[0042] The beneficial effects of the present invention are as follows: the present invention adopts a deep learning approach to avoid the complex calibration and modeling of sensors in traditional methods, and adopts an unsupervised training approach to solve the limitations of the data set. The entire method can fully utilize the complementary information of images and inertial navigation to achieve high-precision positioning.

[0043] 1. By using the method of deep learning, the high-dimensional features of image and inertial navigation information are fully extracted through convolutional neural network and recurrent neural network respectively. At the same time, the long short-term memory network (LSTM) is used to effectively utilize the features to complete the pose calculation; each error index (average translation error, average rotation error) is better than the current mainstream odometer method.

[0044] 2. When the data fails, compared with the situation where other methods cannot complete the positioning task, this method can still output relatively accurate positioning results through reasonable network structure design and data screening mechanism.

[0045] 3. The network adopts an unsupervised training method. The entire network training does not require a dataset with calibrated true values, is less restricted by the dataset, and has a wide range of applications. Description of the Drawings

[0046] Figure 1 It is a schematic diagram of the visual inertial odometer method based on unsupervised learning.

[0047] Figure 2 It is a schematic diagram of the process of image feature extraction. Detailed Implementation Modes

[0048] The following describes the detailed implementation modes of the present invention to facilitate those skilled in the art of this technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the detailed implementation modes. For those of ordinary skill in the art of this technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions created using the concept of the present invention are within the scope of protection.

[0049] As Figure 1 shown, the visual inertial odometer method based on unsupervised learning of this solution includes the following steps:

[0050] S1: Collect input data, where the input data packet includes image data and IMU data, and preprocess the image data and IMU data to synchronize the image data and IMU data;

[0051] The method for preprocessing the image data and IMU data in step S1 is: The camera collects two adjacent frames of images as the image data, and packs the IMU data collected during the time between collecting two adjacent frames of images together with the two adjacent frames of images as the input data to ensure the synchronization of the image data and IMU data. The synchronized IMU data is:

[0052]

[0053] where α is the acceleration on the three axes of XYZ, and ω is the angular velocity on the three axes of XYZ.

[0054] S2: Extract features from the synchronized image data and IMU data to obtain the image features of the image data and the inertial features of the IMU data. Step S2 includes:

[0055] S21: Input two adjacent frames of synchronized image information into a convolutional neural network CNN to extract image features:

[0056] As Figure 2 shown, a transfer learning method is used to assist in building a convolutional neural network for image feature extraction. The convolutional neural network adopts the encoder structure of the FlownetSimple optical flow network, and a branch is added at the 5×5 convolutional layer of the encoder structure. The branch extracts different-level features of the network through a 7×7 spatial channel attention structure, and cascades the extracted different-level features with the features extracted by the encoder at the penultimate convolutional kernel, retains the different-level features, and outputs them as the extracted image features; expand the receptive field to enable feature extraction to extract more information.

[0057] S22: Input the synchronized 10-frame IMU data into a long short-term memory network LSTM. The long short-term memory network LSTM has a hidden layer number of 2 and contains 512 hidden states to complete the extraction of inertial features.

[0058] S3: Input the image features and inertial features into a pose calculation network to output the pose calculation results of six degrees of freedom;

[0059] Step S3 includes:

[0060] S31: Concatenate the extracted image features and inertial features, and then analyze the data through a decoder decode to generate a set of weights; multiply the fused features after concatenation with the weights to complete the fusion and screening of the features. The features after fusion and screening are used as the input of the pose calculation network:

[0061]

[0062] W = σ(f(wF + b))

[0063] [[ID=3x0]]

[0064] where F is the fused feature, F v and F i are the image feature and the inertial feature respectively, is the concatenation process along the channel direction, W is the weight of the generated adjusted feature, σ is the sigmod activation function, f is the function corresponding to the decoder decode, w and b are both parameters learned during training, used to perform preliminary processing on the fused feature, F' is the finally output screened fused feature, is the element-wise multiplication process;

[0065] S32: Use a long short - term memory network (LSTM) followed by a fully - connected layer structure as the pose calculation network. Input the feature after fusion and screening into the pose calculation network, and output the six - degree - of - freedom pose calculation result.

[0066] Selecting the long short - term memory network (LSTM) can not only effectively utilize the temporal sequence of IMU data, but also cascade the CNN and RNN networks to process image information, which can effectively mine the time information in the image sequence. The long short - term memory network (LSTM) is set to 2 layers, with 1024 parameters in each layer, and the fully - connected layer maps the 1024 - dimensional feature to a six - dimensional pose result.

[0067] S4: Input the pose calculation result into the unsupervised module, construct a deep network framework, perform unsupervised training based on the difference between the depth map and the source image, output the training result, and obtain the trained deep network;

[0068] Step S4 includes:

[0069] S41: Construct a deep network. The network framework of the deep network is an encoder - decoder structure. When the camera moves in the scene, objects in adjacent images can form geometric constraints. The adjacent images respectively represent two adjacent frames of the source image I s and the target image Use the depth map and the motion transformation matrix to construct geometric consistency:

[0070] D s (p s )K -1 p s =T t→s D t (p t )K -1 p t

[0071] where D s , D t are the depth maps of adjacent - frame images respectively, p s , p t are the pixel points on the depth maps of adjacent frames respectively, K is the internal parameter matrix of the camera, and T t→s is the pose transformation matrix in the pose calculation result;

[0072] S42: Use to calculate the projection I s of the source image I in the target image t , and by comparing the target image and the projection I tConstruct a photometric consistency constraint based on the appearance similarity to form an unsupervised module. The loss function of the unsupervised module is as follows:

[0073]

[0074] Among them, L p is the structural similarity loss, s is the number of the target image, and t is the number of the source image;

[0075] S43: Introduce a smoothness loss function to compensate for the photometric consistency error in the textureless area:

[0076]

[0077] Among them, L s is the smoothness loss, is a two-dimensional differential operator, D(P) and I(P) are the pixels on the depth map and the source image respectively, and |·| is the absolute value of the element;

[0078] S44: Establish a total loss function according to the loss function of the unsupervised module and the smoothness loss function:

[0079] L = L p + αL s ;

[0080] Among them, α is the weighting factor;

[0081] S45: Perform unsupervised training according to the total loss function of the unsupervised module, output the training result, and obtain a trained depth network.

[0082] S5: Collect new image data and input it into the trained depth network to output the positioning result.

[0083] The present invention adopts a deep learning method, avoiding the complex calibration and modeling of sensors in traditional methods. At the same time, an unsupervised training method is adopted to solve the limitation of the data set. The whole method can make full use of the complementary information of images and inertial navigation to achieve high-precision positioning.

[0084] 1. By using a deep learning method, the high-dimensional features of image and inertial navigation information are fully extracted through a convolutional neural network and a recurrent neural network respectively. At the same time, a long short-term memory network LSTM is used to effectively utilize the features to complete the pose calculation; each error index (average translation error, average rotation error) is better than the current mainstream odometry methods.

[0085] 2. When the data fails, compared with other methods that cannot complete the positioning task, the present method can still output a relatively accurate positioning result through a reasonable network structure design and data screening mechanism.

[0086] 3. The network adopts an unsupervised training method. The entire network training does not require a dataset with calibrated true values, is less restricted by the dataset, and has a wide range of applications.

Claims

1. A visual-inertial odometry method based on unsupervised learning, characterized in that It includes the following steps: S1: Collect input data, where the input data includes image data and IMU data. Preprocess the image data and IMU data to synchronize the image data and IMU data; S2: Extract features from the synchronized image data and IMU data to obtain the image features of the image data and the inertial navigation features of the IMU data; S3: Input the image features and inertial navigation features into a pose calculation network to output a six-degree-of-freedom pose calculation result; S4: Use the pose calculation result to input into an unsupervised module to construct a deep network framework. Perform unsupervised training based on the difference between the depth map and the source image, and output the training result to obtain a trained deep network; S5: Collect new image data and input it into the trained deep network to output a positioning result; The step S4 includes: S41: Construct a deep network with an encoder-decoder structure as the network framework. When the camera moves in the scene, objects in adjacent images can form geometric constraints. The adjacent images respectively represent two adjacent frames of the source image I s and the target image to construct geometric consistency by using the depth map and the motion transformation matrix: Among them, D s and D t are the depth maps of adjacent frame images respectively, p s , p t are the pixel points on the depth maps of adjacent frames respectively, K is the internal parameter matrix of the camera, is the pose transformation matrix in the pose calculation result; S42: Utilize to calculate the source image I s in the target image projection I t , by comparing the target image and the projection I t appearance similarity to construct a photometric consistency constraint, forming an unsupervised module. The loss function of the unsupervised module is: Among them, L p is the structural similarity loss, s is the number of the target image, t is the number of the source image, F v and F i are the image feature and the inertial navigation feature respectively, is the splicing process along the channel direction; S43: Introduce a smoothness loss function to compensate for the photometric consistency error in the textureless area: ； Among them, L s is the smoothness loss, is a two-dimensional differential operator, , are pixels on the depth map and the source image respectively, is to take the absolute value of the element; S44: Establish a total loss function based on the loss function of the unsupervised module and the smoothness loss function: ； Among them, is a weighting factor; S45: Perform unsupervised training according to the total loss function of the unsupervised module, output the training result, and obtain a trained deep network.

2. The visual inertial odometry method based on unsupervised learning according to claim 1, wherein The method for preprocessing the image data and IMU data in the step S1 is: The camera collects two adjacent frames of images as the image data, and packs the IMU data collected during the time between collecting two adjacent frames of images together with the two adjacent frames of images as the input data to ensure the synchronization of the image data and IMU data. The synchronized IMU data is: Among them, α are the accelerations on the X, Y, and Z axes, ω are the angular velocities on the X, Y, and Z axes.

3. The method for visual-inertial odometry based on unsupervised learning according to claim 1, wherein The step S2 includes: S21: Input the information of two adjacent synchronized frames of images into a convolutional neural network CNN to extract image features: Adopt the method of transfer learning to assist in building the convolutional neural network for image feature extraction. The convolutional neural network adopts the encoder structure of the FlownetSimple optical flow network, and adds a branch at the 5×5 convolutional layer of the encoder structure. The branch extracts different-level features of the network through a spatial channel attention structure, and cascades the extracted different-level features with the features extracted by the encoder at the 3×3 convolutional kernel to retain different-level features, and the output is used as the extracted image features; S22: Input the synchronized 10-frame IMU data into a long short-term memory network LSTM. The number of hidden layers of the long short-term memory network LSTM is 2, and it contains 512 hidden states to complete the extraction of inertial navigation features.

4. The visual-inertial odometry method based on unsupervised learning according to claim 1, wherein The step S3 includes: S31: Concatenate the extracted image features and inertial navigation features, and then analyze the data through a decoder decode to generate a set of weights; Multiply the fused features after concatenation with the weights to complete the fusion and screening of the features. The features after fusion and screening are used as the input of the pose calculation network: Among them, F is the fusion feature, F v and F i are the image feature and the inertial navigation feature respectively, is the splicing process along the channel direction, W is the weight of the generated adjusted feature, is the sigmoid activation function, f is the function corresponding to the decoder decode, w and b are both parameters learned during training, used for preliminary processing of the fusion feature, F' is the finally output filtered fusion feature, is the element-wise multiplication process; S32: Adopt a structure of a long short-term memory network LSTM followed by a fully connected layer as the pose calculation network. Input the features after fusion and screening into the pose calculation network to output a six-degree-of-freedom pose calculation result.

Images