A visual odometry calculation method based on attention convolutional neural network

Abstract

The present invention discloses a visual odometer calculation method based on an attention convolutional neural network. In view of the problem that the traditional visual odometer requires that the image contains a large amount of texture information and the solution process is relatively complicated, and the visual odometer based on the convolutional neural network has low accuracy, a visual odometer based on an attention convolutional neural network and a gated recurrent unit is proposed. The attention mechanism is used to improve the accuracy of feature extraction of the convolution module, thereby improving the accuracy of visual positioning. Compared with the previous visual odometer calculation method, it abandons the complicated solution process while ensuring accuracy, and is more suitable for practical engineering applications.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and visual positioning technology. In view of the problem that the traditional visual odometry (VO) requires that the image contains a large amount of texture information and the solution process is relatively complicated, and the visual odometry based on the convolutional neural network (CNN) has low accuracy, a visual odometry based on the attention convolutional neural network and the gated recurrent unit (GRU) is proposed. The attention mechanism is used to improve the accuracy of feature extraction of the convolution module, thereby improving the accuracy of visual positioning. Background Art

[0002] For the autonomous navigation of intelligent vehicles, the self-positioning ability of the vehicle during movement is very important. Early vehicle positioning systems usually used wheel speed encoders to calculate vehicle mileage, but this method has serious cumulative errors. With the development of computer vision technology, visual sensors are increasingly used for vehicle positioning and motion estimation. Visual sensors not only provide rich perception information, but also have the characteristics of low cost and small size. We usually call the problem of obtaining camera posture through vision as visual odometry. Related research on visual odometry began in the 1980s. In recent years, with the continuous deepening of domestic and foreign scholars' research on visual odometry, VO has gradually been applied to various fields such as robotics, autonomous driving, and pedestrian navigation.

[0003] The traditional visual odometer calculation method mainly includes steps such as image feature extraction, feature matching and pose estimation, and finally obtains the six-degree-of-freedom camera pose, namely displacement (x, y, z) and rotation (roll, pitch, yaw). The traditional visual odometer methods are mainly divided into two types: direct method and feature point method. The direct method mainly calculates the camera motion based on the grayscale information of the pixels in the image. Typical algorithms include SVO, LSD-SLAM, etc. The feature point method is considered to be the mainstream method of visual odometer. This method first extracts and matches the features of the image, and then estimates the camera motion based on the matched feature points. Depending on the camera type, ICP, epipolar geometry, PNP and other methods can be used. For example, the famous SIFT, ORB and other algorithms estimate the camera pose by extracting feature points. However, this type of method is time-consuming and computationally intensive when there are many feature points extracted. When there are few feature points extracted, there is a situation where the features are lost and the camera motion is not restored. In addition, the feature point method still faces great challenges in scenes with less texture or dynamic target motion. Although the direct method can avoid feature loss and feature calculation time, it is often used in scenes with alternating light and dark.

[0004] With the continuous development of deep learning, this method has gradually been applied to various fields, such as fault diagnosis, machine translation, image classification, etc. In recent years, this technology has also been used in visual odometry. In 2015, Kishore et al. first used convolutional neural networks to study visual odometry. He designed two different convolutional networks to learn the speed and rotation of movement respectively. Although Kishore successfully transformed the posture regression problem of visual odometry into a classification problem and designed a convolutional neural network to solve the problem, the experimental results showed that the error between the estimated trajectory and the true trajectory of this method was large. In the same year, Kendall et al. proposed the PoseNet model, which estimated the position and posture of the camera by inputting a single picture, and realized end-to-end camera posture estimation for the first time using convolutional neural networks. In 2017, Wang et al. proposed the DeepVO model, which added a recurrent neural network (RNN) to the convolutional neural network to maintain the temporal connectivity between images. The model uses adjacent image sequences in the KITTI dataset as input, first extracts image features through a convolutional neural network module, then inputs them into a recurrent neural network to learn the geometric associations of the images, and finally outputs the camera posture. Experimental results show that combining convolutional neural networks with recurrent neural networks can effectively improve the accuracy of estimated trajectories.

[0005] The accuracy of existing deep learning-based VO estimation methods is still not comparable to traditional methods. Although the latest research has greatly improved the pose accuracy by introducing optical flow, this type of method is difficult to be widely used. Most of the existing deep learning-based visual odometers are based on a single convolutional neural network, or a combination of a convolutional neural network and a recurrent neural network. The estimated trajectory accuracy is still low, so there is still a lot of room for research in model construction and other aspects. Summary of the invention

[0006] To address the above issues, we propose the ACGR (Attention Convolution and Gate Recurrent Unit) model, as Figure 1 As shown in the figure, the model incorporates the attention mechanism based on CNN and RNN, and uses the attention mechanism to improve the accuracy of feature extraction and thus improve the accuracy of estimated trajectory.

[0007] The specific steps are as follows:

[0008] 1. Feature extraction based on Attention Convolution Model.

[0009] This model introduces both spatial and channel attention mechanisms into CNN. Its structure is as follows Figure 2 shown.

[0010] ACGR designs a convolutional network module based on the implementation principle of the optical flow method in the traditional VO method, and uses this module to extract the features of the image, thereby calculating the motion features of the image and expressing them in the form of vectors.

[0011] 2. GRU-based timing modeling.

[0012] The GRU in this model has two layers, each containing 1024 hidden units. In order to maintain the original data distribution, the activation function of the original GRU is changed to the ReLU function.

[0013] 3. Fully connected layer reduces dimension and outputs pose

[0014] This model adds two fully connected layers with 128 and 6 hidden units respectively.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] First, a deep learning-based visual odometry method is implemented. This method abandons the cumbersome steps of the traditional visual odometry method and realizes an end-to-end positioning method. Second, a gated recurrent unit is added to the traditional convolutional neural network-based visual odometry method to learn the temporal correlation in image data. Third, an attention mechanism is integrated into the feature extraction module, which enables the network to learn more important geometric features in the image more intelligently and enhance the feature extraction capability of the convolutional neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a system framework diagram of the method involved in the present invention;

[0018] Figure 2 It is the structural diagram of the attention mechanism;

[0019] Figure 3 (a) is the trajectory diagram of different algorithms in sequence 03;

[0020] Figure 3 (b) is the trajectory diagram of different algorithms in sequence 04;

[0021] Figure 3 (c) is the trajectory diagram of different algorithms in sequence 09;

[0022] Figure 3 (d) is the trajectory diagram of different algorithms in sequence 10; DETAILED DESCRIPTION

[0023] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0024] The present invention utilizes attention mechanism, convolutional neural network and gated recurrent unit to realize visual odometer technology.

[0025] The specific steps include:

[0026] 1. Feature extraction based on Attention Convolution Model

[0027] (1) Convolutional Neural Network

[0028] ACGR designs a convolutional network module based on the implementation principle of the optical flow method in the traditional VO method, and uses this module to extract the features of the image, thereby calculating the motion features of the image and expressing them in the form of vectors.

[0029] The parameters of CNN are shown in Table 1. It contains 9 convolutional layers in total, and batch normalization is added after each Conv to maintain the data distribution before and after the convolution transformation, speed up the model training and avoid gradient explosion in back propagation. The size of the first convolution kernel in CNN is 7×7, the second and third layers are 5×5, and the last six layers are 3×3. ReLU is selected as the activation function. The input of this module is the image of two adjacent frames in the KITTI dataset. In order to retain the geometric features of the original data, the size of the image is modified and unified to 1280×384. The tensor size of the image after 9 layers of convolution feature extraction is 10×3×512.

[0030] Table 1 Convolutional layer parameter list

[0031] Tab.1Convolutional layer parameter list

[0032]

[0033] (2) Attention Mechanism

[0034] Embedding the attention mechanism into the convolutional network can make the network learn more important geometric features in the image more intelligently and enhance the feature extraction ability of the convolutional neural network. The calculation formula of the attention module is as follows:

[0035] M c (F)=σ(MLP(AP(F))+MLP(MP(F))) (1)

[0036]

[0037] Where MLP represents the fully connected layer, is the characteristic spectrum, is a one-dimensional channel attention feature, AP and MP represent average pooling and maximum pooling respectively. is the two-dimensional spatial attention feature. σ is the sigmoid function, f 7*7 Represents a convolution operation with a filter size of 7×7.

[0038] 2. GRU-based time series modeling

[0039] The image data used for visual odometry has a strong temporal correlation, so a recurrent neural network can be used to learn this correlation. GRU has a simpler structure and fewer parameters, so using GRU can effectively save training time.

[0040] Let the current time be t and the input be x t , the hidden state of the previous moment is represented as h t-1 , then the output of GRU and the state update equation are as follows.

[0041]

[0042] where z t represents the update gate in GRU, σ is the sigmoid activation function, which is used to control the output value between 0 and 1, and r t Represents the reset gate in GRU, the tanh activation function is used to control the output value between -1 and 1, [] indicates that two vectors are connected, and * indicates the product of the matrix. W, W z , W r Represents weights, which are randomly initialized and continuously updated during training.

[0043] The GRU in this model has two layers, each containing 1024 hidden units. The input image sequence is extracted by the Attention Convolution Model to obtain a tensor of size 10×3×512, which is then input into the first layer of GRU, and the output of this layer is input into the second layer of GRU. The output of the second layer of GRU is the output of the entire GRU module.

[0044] 3. Fully connected layer

[0045] This model has two fully connected layers, with 128 and 6 hidden units respectively. The fully connected layer reduces the dimension of the high-dimensional features output by the GRU module, and the final output six-dimensional tensor is the relative posture between the current moment and the previous moment.

[0046] 4. Loss Function

[0047] The pose estimation problem in visual odometer is expressed as a conditional probability problem. As shown in Formula 4, for a given (n+1) images:

[0048] X=(X1,X2,...,Xn+1 ) (4)

[0049] The poses between adjacent pictures can be obtained by calculation:

[0050] Y=(Y1,Y2,...,Y n ) (5)

[0051] Considering the above problem as a conditional probability problem, the calculation formula is:

[0052] p(Y|X)=p(Y1,Y2,...,Y n |X1,X2,...,X n+1 ) (6)

[0053] Solve for the optimal network parameter w * Make the probability in formula (6) maximum:

[0054]

[0055] The mean squared error (MSE) is used as the loss function, as shown in Formula 8:

[0056]

[0057] P 1i represents the displacement of the Ground Truth of the i-th pair of positive sequence inputs, Φ 1i Represents the rotation angle of GroundTruth for the i-th pair of positive sequence inputs. represents the displacement of the ith pair of samples in the positive sequence input, P represents the rotation angle of the i-th pair of samples in positive order. 2i represents the displacement of the Ground Truth of the i-th pair of reverse inputs, Φ 2i Represents the rotation angle of the Ground Truth for the i-th pair of reverse inputs. represents the displacement of the i-th pair of samples input in reverse order, represents the rotation angle of the i-th pair of samples in reverse order. M represents the number of samples, and β1 and β2 represent the scale factors of the positive and reverse order input errors, respectively.

[0058] 5. Experimental process and results

[0059] (1) Dataset Introduction

[0060] The ACGR model uses the public dataset KITTI dataset to complete the experiment. KITTI Visual Odometry is a large open source dataset that is widely used to evaluate various visual odometry models. Because only the first 11 scenes of this dataset provide ground truth, the data of the first 11 sequences are selected for the experiment. In order to meet the network's requirements for input data while retaining the geometric features of the image, all images are uniformly resized to 1280×384. In addition, sequences 03, 04, 09, and 10 have the least number of images, and the larger the number of training set images, the better the model training results, so sequences 03, 04, 09, and 10 are selected as test sets. Sequences 00, 01, 02, 05, 06, 07, and 08 contain a large number of images, so the above sequences are selected to train the model. Then, data is randomly selected from the above 7 training set sequences for verification, and the number of verification set images accounts for about one-third of the total number of training sets. The specific division is shown in Table 2.

[0061] Table 2 Training set, validation set and test set

[0062] Tab.2training set, setverification set and test set

[0063]

[0064] (2) Model training

[0065] This model is built on the deep learning framework PyTorch, and the graphics card model is Nvidia GeforceRTX2080ti. The optimizer is Adam (Adaptive Moment Estimation), and the optimization algorithm is Batch Gradient Descent (BGD).

[0066] During training, the weights of all parameters in the network are initialized using the Xavier method, and the bias is initialized using the zero method. The number of iterations is set to 100, and the initial learning rate is set to 0.01. The 100th model parameters are saved for subsequent testing.

[0067] (3) Experimental results and error analysis

[0068] In the experiment, ACGR was first trained using the 7 sequences in the training set, and then the model was tested using the above 4 sequences. The performance of ACGR was evaluated based on the test results. The estimated trajectories of test set sequences 03, 04, 09 and 10 are shown in Figure 2. Figure 3 shown.

[0069] In order to verify the model's trajectory estimation effect, this experiment compared a total of four methods, including ORBSLAM

[22] , the SFMLearner algorithm based on convolutional neural networks, the CAGR-VO (Convolution and Gate Recurrent Unit Visual Odometry) algorithm and the ACGR-VO algorithm based on convolutional and recurrent neural networks. ORBSLAM is a classic visual SLAM system that includes a loop detection module in addition to the visual odometer. In order to ensure the fairness of the comparison experiment, loop detection was not added to the ORBSLAM used in this experiment. Figure 3 It can be seen that the above four algorithms can roughly estimate the shape of the trajectory. ORBSLAM based on feature points is relatively mature. The ORBSLAM selected in the experiment does not add loop detection, so the fitting effect between the estimated trajectory and the true trajectory in the 09 sequence is poor, but the fitting effect in the 03, 04, and 10 sequences is very good, and the trajectory prediction accuracy is high. The research on visual odometry based on deep learning is still in the development stage and there is still a lot of room for research. Although the estimation effects of the other three visual odometry models in the experiment are slightly inferior to the ORB algorithm, the fitting effect of the ACGR-VO model is better than the SFMLearner model based on the convolutional neural network and the CAGR-VO model based on the convolutional and recurrent neural network. It can be seen that the accuracy of the ACGR-VO model is improved compared with other models based on convolutional neural networks, and the addition of the attention mechanism can effectively improve the accuracy of the model's trajectory prediction.

Claims

1. A visual mileage calculation method based on attention convolutional neural network, characterized in that: The specific steps include: 1) Feature extraction based on Attention Convolution Model (1) Convolutional Neural Network The parameters of CNN are shown in Table 1. It contains 9 convolutional layers in total, and batch normalization is added after each Conv. The size of the convolution kernel of the first layer in CNN is 7×7, the second and third layers are 5×5, and the last six layers are 3×3. ReLU is selected as the activation function. The input of the Attention Convolution module is the pictures of two adjacent frames in the KITTI dataset. In order to retain the geometric features of the original data, the size of the picture is modified and unified to 1280×384. The tensor size of the picture after 9 layers of convolution feature extraction is 10×3×512. Table 1 Convolutional layer parameter list (2) Attention Mechanism The attention mechanism is embedded into the convolutional network, and the calculation formula of the attention module is as follows: M c (F)=σ(MLP(AP(F))+MLP(MP(F))) (1) Where MLP represents the fully connected layer, is the characteristic spectrum, is a one-dimensional channel attention feature, AP and MP represent average pooling and maximum pooling respectively. is the two-dimensional spatial attention feature; σ is the sigmoid function, f 7*7 represents a convolution operation with a filter size of 7×7; 2) GRU-based time series modeling Let the current time be t and the input be x t , the hidden state of the previous moment is represented as h t-1 , then the output and state update equation of GRU are as follows; where z t represents the update gate in GRU, σ is the sigmoid activation function, which is used to control the output value between 0 and 1, and r t represents the reset gate in GRU, the tanh activation function is used to control the output value between -1 and 1, [] represents the connection between two vectors, and * represents the product of matrices; W, W z , W r Represents weights, which are randomly initialized and continuously updated during training; The GRU in this model has two layers, each containing 1024 hidden units. The input image sequence is extracted by the Attention Convolution Model to obtain a tensor of size 10×3×512, which is then input into the first layer of GRU. The output of this layer is input into the second layer of GRU. The output of the second layer of GRU is the output of the entire GRU module. 3) Fully connected layer There are two fully connected layers, containing 128 and 6 hidden units respectively. The fully connected layer reduces the dimension of the high-dimensional features output by the GRU module, and the final output six-dimensional tensor is the relative posture between the current moment and the previous moment. 4) Loss Function The pose estimation problem in visual odometer is expressed as a conditional probability problem; as shown in Formula 4, for a given (n+1) images: X=(X1,X2,...,X n+1 ) (4) The poses between adjacent images are obtained by calculation: Y=(Y1,Y2,...,Y n ) (5) The pose estimation problem in visual odometer is regarded as a conditional probability problem, and the calculation formula is: p(Y|X)=p(Y1,Y2,...,Y n ∣X1,X2,...,X n+1 ) (6) Solve for the optimal network parameter w * Make the probability in formula (6) maximum: The mean squared error (MSE) is used as the loss function, as shown in Formula 8: P 1i represents the displacement of the Ground Truth of the i-th pair of positive sequence inputs, Φ 1i Represents the rotation angle of GroundTruth of the i-th pair of positive sequence input; represents the displacement of the i-th pair of samples in positive sequence input, represents the rotation angle of the i-th pair of samples in positive order; P 2i represents the displacement of the Ground Truth of the i-th pair of reverse inputs, Φ 2i Represents the rotation angle of the Ground Truth of the i-th pair of reverse inputs; represents the displacement of the i-th pair of samples input in reverse order, represents the rotation angle of the i-th pair of samples in reverse order; M represents the number of samples, β1 and β2 represent the scale factors of the positive and reverse order input errors respectively; During training, the weights of all parameters in the network are initialized using the Xavier method, and the biases are initialized using the zero method; the number of iterations is set to 100, and the initial learning rate is set to 0.01; and the 100th model parameters are saved for subsequent testing.

Images