Monocular self-supervised learning scale perception visual inertial odometer method

Abstract

The invention belongs to the field of visual odometers, and particularly relates to a monocular self-supervised learning scale perception visual inertial odometer method. Comprising the following steps: collecting two adjacent frames of images in a motion scene and IMU data between the two frames of images; inputting one frame of image into a deep network for processing to obtain a depth map; inputting the two frames of images and the IMU data into a speedometer network for processing to obtain a relative pose, gravity and IMU deviation; calculating a distortion diagram according to the relative pose and the depth map; calculating luminosity consistency loss and smoothness loss according to the distortion graph and the real image; calculating pre-integration loss according to the IMU data and the relative pose; calculating conventional loss according to the IMU deviation and gravity; taking the sum of the luminosity consistency loss, the smoothness loss, the pre-integration loss and the conventional loss as the total loss; adjusting network parameters according to the total loss to obtain a trained end-to-end self-supervised network; using the trained network to obtain a positioning result; according to the invention, a more accurate positioning result can be realized.

Description

Technical Field

[0001] The present invention belongs to the field of visual odometers, and in particular relates to a scale-aware visual inertial odometer method for monocular self-supervised learning. Background Art

[0002] For practical applications with a single monocular camera, scale ambiguity is an important issue. Since self-supervised data-driven methods that do not require additional data containing scale information cannot avoid scale ambiguity, state-of-the-art deep learning-based methods address this issue by learning scale information from additional sensor measurements. In this regard, inertial measurement units (IMUs) have become a popular sensor for various mobile platforms due to their light weight and cheap price. However, unlike supervised learning where scale can be learned from ground truth information, learning scale from IMUs is challenging in a self-supervised setting. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention proposes a scale-aware visual inertial odometer method for monocular self-supervised learning, the method comprising: obtaining adjacent frame images and IMU data between two frames in a motion scene; inputting the adjacent frame images and the IMU data between two frames into a trained end-to-end self-supervised network to obtain a positioning result;

[0004] The training process of the end-to-end self-supervised network includes:

[0005] S1: Collect two adjacent frames of images in the motion scene and obtain the IMU data between the two frames of images;

[0006] S2: Input a frame of image into the deep network for processing to obtain a depth map;

[0007] S3: Input the two frames of images and IMU data into the odometer network for processing to obtain relative pose, gravity and IMU deviation;

[0008] S4: Calculate the distortion map based on the relative pose and depth map; Calculate the photometric consistency loss and smoothness loss based on the distortion map and the real image;

[0009] S5: Calculate the pre-integration loss based on IMU data and relative pose;

[0010] S6: Calculate the regular loss based on IMU bias and gravity; take the sum of photometric consistency loss, smoothness loss, pre-integration loss and regular loss as the total loss;

[0011] S7: Adjust the network parameters according to the total loss to obtain a trained end-to-end self-supervised network.

[0012] Preferably, the odometer network processes the two frames of images and the IMU data including:

[0013] Use the visual encoder to process the two frames of images to obtain image features;

[0014] The inertial encoder is used to process the IMU data to obtain the inertial navigation characteristics;

[0015] The image features and the inertial navigation features are input into the feature fusion network for fusion to obtain fusion features;

[0016] Multiple decoders are used to process the fused features to obtain relative pose, gravity and IMU deviation.

[0017] Furthermore, the inertial encoder includes three bidirectional LSTM layers.

[0018] Furthermore, the feature fusion network includes two fully connected layers, a normalization layer and a concatenation layer.

[0019] Preferably, the formula for calculating the photometric consistency loss is:

[0020]

[0021] in, represents the photometric consistency loss, N represents the number of image pixels, u represents each pixel of the image, and μ represents the per-pixel mask. Represents image I n+1 With image The difference between the n+1 Represents the current image I n The next frame image is the true value. Represents the current image I n The distortion map that warps to the next frame's view.

[0022] Preferably, the formula for calculating the smoothness loss is:

[0023]

[0024] in, represents the smoothness loss, represents the partial derivative operator with respect to i, x, y represent the row coordinate and column coordinate of pixel i in the pixel coordinate system, respectively. Represents the depth map of the current view, exp represents the exponential operation, I n Indicates the current image.

[0025] Preferably, the pre-integration loss is the sum of the rotation pre-integration loss and the speed pre-integration loss. The formula for calculating the rotation pre-integration loss and the speed pre-integration loss is:

[0026]

[0027] in, represents the rotation pre-integration loss, represents the velocity pre-integration loss, λ rot represents the rotation weight, λ vel represents the speed weight, ΔR N represents the relative rotation matrix, Represents the rotation matrix measurement value obtained using IMU data, Δv N represents the relative speed, Represents velocity measurements using IMU data.

[0028] Preferably, the conventional loss is the sum of the gravity adjustment loss, the deviation adjustment loss and the bias term.

[0029] Furthermore, the gravity adjustment loss is expressed as:

[0030]

[0031] Among them, λ grav represents the gravity adjustment weight, represents the first gravity at time N in the object coordinates, represents the second gravity at time N in the object coordinates;

[0032] The bias adjustment loss is expressed as:

[0033]

[0034] Among them, λ bdi represents the deviation angular velocity adjustment weight, λ bdiffa Indicates the deviation acceleration adjustment weight, represents the angular velocity deviation at time N, Indicates the angular velocity deviation at time 0, represents the angular velocity deviation at time N, Indicates the acceleration deviation at time 0.

[0035] The beneficial effects of the present invention are:

[0036] 1. We propose a new loss function for learning scales from IMU measurements during the training step, thereby preserving the monocular self-supervised setting of the end-to-end training framework.

[0037] 2. A network architecture for predicting gravity and IMU bias in body coordinates, as well as appropriate gravity and bias regularization functions, is proposed to train the network using a new loss function, improving the accuracy of visual inertial odometry. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall flow chart of the end-to-end self-supervisory network in the present invention;

[0039] Figure 2 This is a schematic diagram of the odometer network structure in the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The present invention proposes a monocular self-supervised learning scale-aware visual inertial odometer method, which includes the following contents:

[0042] The proposed method is derived from self-supervised monocular depth estimation and is upgraded by integrating IMU measurements to learn scale. For self-supervised monocular depth estimation, the depth map and ego motion are jointly predicted. Therefore, Figure 1 As shown, the present invention constructs two convolutional neural networks: one is a deep network that predicts a dense depth map from a single RGB image, and the other is an odometer network that predicts relative pose from a pair of continuous images.

[0043] Obtain the IMU data between adjacent frame images and two frame images in the motion scene; input the IMU data between adjacent frame images and two frame images into the trained end-to-end self-supervised network to obtain the positioning result. The training process of the end-to-end self-supervised network includes:

[0044] S1: Collect two adjacent frames of images in a motion scene and obtain the IMU data between the two frames of images.

[0045] The camera collects two adjacent frames of images in the motion scene, and the IMU measures the IMU data of the time between the two adjacent frames of images to ensure that the two adjacent frames of images are synchronized with the IMU data, and the synchronized two adjacent frames of images and IMU data are used as the input quantity of the network.

[0046] S2: Input a frame of image into the deep network for processing to obtain a depth map.

[0047] The deep network estimates the depth map from a single RGB image. The deep network used in the present invention has a U-Net structure, which is a fully convolutional encoder-decoder structure with jump connections. The deep network can learn to estimate the scale by using pre-integration loss during training. Preferably, ResNet18 is selected as the encoder. The deep network receives a single image and does not receive IMU information. A single frame image is input into the deep network for processing to obtain a depth map.

[0048] S3: Input the two frames of images and IMU data into the odometer network for processing to obtain relative pose, gravity and IMU deviation.

[0049] like Figure 2 As shown, the present invention designs an odometer network to infer the relative posture, the deviation of the inertial measurement unit and the gravity direction. The odometer network includes a visual encoder, an inertial encoder, a feature fusion network and multiple decoders.

[0050] The odometry network estimates the relative pose between two frames, the gravity direction and IMU bias from two consecutive images, and the IMU measurement between images; the odometry network processes the two frames of images and IMU data by:

[0051] The two frames of images are processed using a visual encoder to obtain image features. Preferably, ResNet18 is selected as the visual encoder, which is the same as the above-mentioned deep network encoder.

[0052] The IMU measurement values ​​are stacked between two image frames and have temporal significance. The present invention uses a bidirectional LSTM to encode the IMU measurement data. Preferably, the present invention sets the inertial encoder to be three 128-channel bidirectional LSTM layers. The inertial encoder is used to process the IMU data to obtain the inertial navigation features.

[0053] The image features and inertial navigation features are input into the feature fusion network for fusion to obtain fused features. Specifically: A single fully connected layer is added after each visual encoder and inertial encoder: the visual encoding network is 256 dimensions and the inertial encoding network is 128 dimensions. Then, each fully connected layer output feature is normalized by layer normalization to have similar amplitudes. Finally, the two features are connected to fuse into a single feature, and the fused feature is output.

[0054] Multiple decoders are used to process the fused features to obtain relative pose, gravity, and IMU bias. Specifically: Multiple decoders receive the same features from the feature fusion part and emit the final output separately. Each decoder contains 7 dense layers for decoding the fused features. The channels of each layer are 256, 256, 128, 128, 64, 64 and the dimensions of the output: the output dimension of the relative pose is a 7-dimensional vector (translation vector and quaternion), the output dimension of the gravity direction is 2-dimensional (representing the direction), and the output dimension of the IMU bias is 6-dimensional (IMU bias). Each decoder has an additional activation to correctly constrain the output. Typically, the amplitude of the input and output of the network is about 1, so the scalar hyperparameters can be multiplied by the output. In addition, if necessary, special activations will be performed on some outputs.

[0055] In order to quickly converge gravity prediction, for gravity, since the magnitude of gravity is fixed, a 2-DOF vector is regressed on the spherical coordinates. Gravity direction conversion considers that the nominal gravity direction on the robot platform is zero by initializing the network deviation. The present invention assumes that the nominal gravity direction is toward the z-axis, then the activation of the gravity direction is:

[0056] g B =||g||2[sinθcosφsinθsinφcosθ] T

[0057] Where (θ, φ) is the inclination and azimuth, which are predicted from the network. The gravitational acceleration g is 9.81m / s 2 .

[0058] S4: Calculate the distortion map based on the relative pose and depth map; Calculate the photometric consistency loss and smoothness loss based on the distortion map and the real image.

[0059] The distortion map is calculated based on the relative pose and depth map. Specifically: In order to express the stereoscopic motion, from the current view u n Point in the pixel coordinates of the next view The predicted dense depth map can be used to calculate the pixel coordinates of and the relative pose predicted in the special Euclidean group SE(3) And the camera internal parameter K is obtained, the specific formula is as follows:

[0060]

[0061] According to the above formula, the points in all pixel coordinates in the next view are calculated to obtain the distortion map.

[0062] In order to train the network, a photometric consistency loss is introduced based on stereo motion. The photometric consistency loss is formulated as the target image I from the dataset n+1(Real image) and distorted image Distance between:

[0063]

[0064] Where N is the number of image pixels and dist(x, y) is the distance between image x and image y. Specifically expressed as:

[0065] dist(x,y)=α|xy|+0.5(1-α)(1-SSIM(x,y))

[0066] Where the scalar constant α = 0.15, SSIM(x, y) represents the structural similarity index of two images:

[0067]

[0068] Among them, μ x , μ y represents the average brightness of the image, σ x , σ y represents the variance of image brightness, σ xy Represents the covariance of the two images, and C is a constant used to avoid the denominator being zero.

[0069] In order to adjust, the present invention adopts edge-aware depth smoothness image processing technology to smooth the depth map while keeping the image edge clear. The present invention also minimizes edge-aware depth smoothness and designs smoothness loss, as follows:

[0070]

[0071] in, represents the partial derivative operator with respect to i, x, y represent the row coordinate and column coordinate of pixel i in the pixel coordinate system, respectively. Represents the depth map of the current view, exp represents the exponential operation, I n Represents the current real image.

[0072] To avoid the influence of occluded or moved pixels that break photometric consistency, the per-pixel mask μ∈[0, 1] is multiplied by the distance between images to obtain a photometric consistency loss as follows:

[0073]

[0074] Wherein μ represents a mask. The present invention adopts automatic masking, and the binary mask method calculated from the source image and the target image is:

[0075]

[0076] S5: Calculate the pre-integration loss based on IMU data and relative pose.

[0077] The purpose of IMU pre-integration is to integrate the raw IMU measurements to obtain the ego-motion between two image frames. The rotation r at the ego-motion in the next frame is N , speed v N and pan p N It should be formulated using the current ego motion and the IMU measurements including acceleration a and angular velocity ω. The ego motion (pose and velocity) at the i-th frame in inertial coordinates is expressed as:

[0078]

[0079]

[0080] Among them, the superscript G represents the variable in the inertial coordinate system, R represents rotation, v represents velocity, p represents translation, and Δt k is the time from frame k to frame k+1, ω k is the unbiased angular velocity at k, a k is the unbiased acceleration at k, g G is gravity. exp represents the exponential mapping of the Lie algebra of the special orthogonal group SO(3). R0 represents the rotation at time 0, represents the speed at time 0, Represents the translation at time 0.

[0081] The pre-integration loss is used to obtain scale-aware ego-motion by integrating IMU measurements and comparing the predicted ego-motion to the obtained ego-motion. The main role of the pre-integration loss is to learn the scale from the IMU measurements by integrating them and correcting the integrated and predicted ego-motion. Since the photometric consistency loss does not include scale, only the pre-integration loss helps to learn the scale. For the predicted relative pose in object coordinates defined on se(3) (se(3) represents the Lie algebra of SE(3)), we rotate ΔR by ΔR in time N. N , speed Δv N and translation Δp N The relative form of is defined as:

[0082]

[0083] Among them, the superscript G and superscript B represent the inertial coordinates and object coordinates respectively, ΔT N is the time elapsed from time 0 to time N, is the relative attitude translation.

[0084] Using IMU pre-integration, ΔR N , Δv N , ΔpN It can be written as:

[0085]

[0086]

[0087] in is the measured angular velocity, is the measured acceleration, is the gravity in the object's coordinate system. To distinguish, ΔX N Expressed as parameters from the relative pose, Expressed as parameters from IMU measurements, where X represents one of R, v, p.

[0088] According to the above formula, three equality constraints can be generated: In order to solve these equality constraints, we also need to get the velocity From the equality of the translations p, the closed form of the velocity can be obtained as:

[0089]

[0090] Where Δv N is reformulated as:

[0091]

[0092] Because the predicted posture The speed of finite difference calculations may be noisy. Therefore, when calculating the velocity, we smooth the translation along the time axis using a moving average filter To suppress noise in the implementation.

[0093] Finally, the pre-integrated loss is obtained as the norm of the two remaining equality constraints:

[0094]

[0095] in, represents the rotation pre-integration loss, represents the velocity pre-integration loss, λ rot represents the rotation weight, λ vel represents the speed weight, ΔR N represents the relative rotation matrix, Represents the rotation matrix measurement value obtained using IMU data, Δv N represents the relative speed, represents the velocity measurement value obtained using IMU data. |||| is a norm function. In the present invention, the logarithmic hyperbolic cosine function is used as the norm function to suppress the influence of outliers to achieve rapid convergence.

[0096] S6: Calculate the regular loss based on IMU bias and gravity; take the sum of photometric consistency loss, smoothness loss, pre-integration loss and regular loss as the total loss.

[0097] In addition to scale and self-motion, predicted gravity and bias will affect the IMU pre-integration loss. If not adjusted, gravity and bias can regress freely, so the scale, gravity and bias may be estimated incorrectly. To this end, the present invention designs the adjustment loss in the direction of gravity and the zero bias of the inertial measurement unit, namely the conventional loss. The conventional loss is the sum of the gravity adjustment loss, the bias adjustment loss and the bias term. Specifically:

[0098] Gravity Adjustment Loss

[0099] The gravity in the inertial coordinate system is assumed to be constant, and the gravity in the object coordinate system is coupled to the self-motion. Therefore, a gravity adjustment loss is designed to represent the gravity in the object coordinate system using the predicted self-motion. In the object coordinate system, the gravity at time N is The gravity that can be predicted from the network at time 0 is and rotational self-motion ΔR N Estimated to be:

[0100]

[0101] Since the magnitude of gravity is constant, the geodesic distance on the Earth's surface is used as the gravity predicted by the network at time N:

[0102]

[0103] Among them, λ grav represents the gravity adjustment weight, and the symbols · and × are the inner product and outer product defined in the 3D vector space.

[0104] Bias Adjustment Loss

[0105] Traditional visual inertial navigation methods construct the zero bias model as a constant with Gaussian noise. The angular bias and linear bias are adjusted by minimizing the bias difference between adjacent frames:

[0106]

[0107] Among them, λ bdiffω represents the deviation angular velocity adjustment weight, λ bdiffa Indicates the deviation acceleration adjustment weight, represents the angular velocity deviation at time N, Indicates the angular velocity deviation at time 0, represents the angular velocity deviation at time N, Indicates the acceleration deviation at time 0.

[0108] Bias

[0109] In addition to the adjustment between adjacent frames, the present invention also adjusts the size of the bias term to avoid bias prediction being too large. The bias term is expressed as

[0110]

[0111] Among them, λ bmagω represents the angular velocity bias weight, λ bmaga Represents the acceleration bias weight.

[0112] The proposed method uses three loss functions to optimize the network: the photometric consistency loss function from the most advanced self-supervised monocular depth estimation to learn the scale-unknown depth and self-motion, the pre-integration loss function to learn the scale of self-motion from IMU measurements, and the adjustment loss function about gravity direction and bias to adjust the influence of predicted gravity direction and bias. Therefore, the sum of photometric consistency loss, smoothness loss, pre-integration loss and regular loss is taken as the total loss It is expressed as:

[0113]

[0114] S7: Adjust the network parameters according to the total loss to obtain a trained end-to-end self-supervised network.

[0115] Backward gradient propagation is performed according to the total loss and the network parameters are adjusted. When the loss function converges or the preset number of iterations is reached, the training is stopped and the optimal network parameters are saved to obtain a trained end-to-end self-supervised network.

[0116] After the training is completed, the IMU data between adjacent frame images and two frame images in the motion scene are obtained; the IMU data between adjacent frame images and two frame images are input into the trained end-to-end self-supervised network, and the relative pose and depth map are output. The relative pose and depth map are fused to obtain the positioning result.

[0117] In summary, the present invention predicts a depth map from a single image, and predicts self-motion, gravity in body coordinates, and IMU bias from two consecutive images and IMU measurements. The image is warped using the depth map and self-motion to obtain and minimize the photometric consistency loss, defined as the difference between the reference real image and the warped image. In addition, the IMU measurements are integrated using the predicted gravity and bias to obtain scale-aware self-motion. Then, the pre-integration loss defined as the difference between the integrated self-motion and the predicted self-motion is minimized. The integrated self-motion has a scale, which is predicted by minimizing the pre-integration loss. The above method can obtain an end-to-end self-supervised network with better performance, thereby obtaining more accurate positioning results.

[0118] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A monocular self-supervised learning scale-aware visual inertial odometry method, characterized in that: include: Obtain adjacent frame images and IMU data between two frame images in a motion scene; Input the IMU data between adjacent frame images and two frame images into the trained end-to-end self-supervised network to obtain the positioning result; The training process of the end-to-end self-supervised network includes: S1: Collect two adjacent frames of images in the motion scene and obtain the IMU data between the two frames of images; S2: Input a frame of image into the deep network for processing to obtain a depth map; S3: Input the two frames of images and IMU data into the odometer network for processing to obtain relative pose, gravity and IMU deviation; S4: Calculate the distortion map based on the relative pose and depth map; Calculate the photometric consistency loss and smoothness loss based on the distortion map and the real image; S5: Calculate the pre-integration loss based on IMU data and relative pose; S6: Calculate the regular loss based on IMU bias and gravity; take the sum of photometric consistency loss, smoothness loss, pre-integration loss and regular loss as the total loss; S7: Adjust the network parameters according to the total loss to obtain a trained end-to-end self-supervised network.

2. The method of monocular self-supervised learning scale-aware visual inertial odometry according to claim 1, characterized in that: The odometer network processes two frames of images and IMU data in the following ways: Use the visual encoder to process the two frames of images to obtain image features; The inertial encoder is used to process the IMU data to obtain the inertial navigation characteristics; The image features and the inertial navigation features are input into the feature fusion network for fusion to obtain fusion features; Multiple decoders are used to process the fused features to obtain relative pose, gravity and IMU deviation.

3. The monocular self-supervised learning scale-aware visual inertial odometry method according to claim 2, characterized in that: The inertial encoder includes three bidirectional LSTM layers.

4. The monocular self-supervised learning scale-aware visual inertial odometry method according to claim 2, characterized in that: The feature fusion network includes two fully connected layers, a normalization layer and a splicing layer.

5. The monocular self-supervised learning scale-aware visual inertial odometry method according to claim 1, characterized in that: The formula for calculating photometric consistency loss is: in, represents the photometric consistency loss, N represents the number of image pixels, u represents each pixel of the image, and μ represents the per-pixel mask. Represents image I n+1 With image The difference between the n+1 Represents the current image I n The next frame image is the true value. Represents the current image I n The distortion map that warps to the next frame's view.

6. The monocular self-supervised learning scale-aware visual inertial odometry method according to claim 1, characterized in that: The formula for calculating smoothness loss is: in, represents the smoothness loss, represents the partial derivative operator with respect to i, x, y represent the row coordinate and column coordinate of pixel i in the pixel coordinate system, respectively. Represents the depth map of the current view, exp represents the exponential operation, I n Indicates the current image.

7. The monocular self-supervised learning scale-aware visual inertial odometry method according to claim 1, characterized in that: The pre-integration loss is the sum of the rotation pre-integration loss and the speed pre-integration loss. The formula for calculating the rotation pre-integration loss and the speed pre-integration loss is: in, represents the rotation pre-integration loss, represents the velocity pre-integration loss, λ rot represents the rotation weight, λ vel represents the speed weight, ΔR N represents the relative rotation matrix, Represents the rotation matrix measurement value obtained using IMU data, Δv N represents the relative speed, Represents velocity measurements using IMU data.

8. The monocular self-supervised learning scale-aware visual inertial odometry method according to claim 1, characterized in that: The conventional loss is the sum of the gravity adjustment loss, the deviation adjustment loss and the bias term.

9. The monocular self-supervised learning scale-aware visual inertial odometry method according to claim 8, characterized in that: The gravity adjustment loss is expressed as: Among them, λ grav represents the gravity adjustment weight, represents the first gravity at time N in the object coordinates, represents the second gravity at time N in the object coordinates; The bias adjustment loss is expressed as: Among them, λ bdif represents the deviation angular velocity adjustment weight, λ bdiffa Indicates the deviation acceleration adjustment weight, represents the angular velocity deviation at time N, Indicates the angular velocity deviation at time 0, represents the angular velocity deviation at time N, Indicates the acceleration deviation at time 0.

Images