A Deep Privileged Visual Odometry Method Based on Cross-modal Knowledge Distillation

Abstract

The present invention discloses a method for deep privileged monocular visual odometry based on cross-modal knowledge distillation. The color image and three-dimensional point cloud of the scene are collected to construct a visual laser odometry data set; the data are input into a visual laser odometry network and a monocular visual odometry network for training respectively; the monocular visual odometry network is trained again using the visual laser odometry network through a cross-modal knowledge distillation module; after the training, the color image sequence of continuous moments to be tested is input into the trained monocular visual odometry network, and the global pose is output to realize the processing of the monocular visual odometry. The method of the present invention utilizes more reliable deep privileged information, has better robustness to environmental factors such as weak texture areas and illumination changes, achieves significant performance improvement, and has good versatility and universality.

Description

Technical Field

[0001] The present invention relates to a deep privileged visual odometry method in the technical field of computer vision, and in particular to a deep privileged monocular visual odometry method based on cross-modal knowledge distillation. Background Art

[0002] Monocular visual odometry is the most commonly used method in pose estimation tasks. Most traditional visual odometry methods estimate inter-frame poses through feature point extraction, feature matching, and geometric relationship optimization. Although they have achieved good results, these methods rely on manually designed features and are prone to failure in complex scenes with weak texture areas or strong lighting changes. Recently, visual odometry methods based on deep learning have begun to attract more and more attention from researchers, especially self-supervised visual odometry methods. Since they do not require true value labels, they can give full play to the data-driven advantages of deep learning technology and learn more effective features from big data, thereby overcoming the defects of traditional manual features. For example, Zhou et al. published "Unsupervised Learning of Depth and Ego-Motion from Video" in "Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition" and Gao et al. published "Attentional Separation-and-Aggregation Network for Self-supervised Depth-Pose Learning in Dynamic Scenes" in "4th Annual Conference on Robot Learning". Both use monocular visual odometry methods to jointly train depth estimation networks and pose estimation networks in an unsupervised manner using monocular image sequences, and no additional truth values ​​are required during the training process.

[0003] However, due to the limitations of monocular and unsupervised learning, such methods cannot predict the pose and depth at the real scale. One solution is to introduce binocular privileged information and use the view synthesis between binocular images to obtain the real scale information, such as "Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction" published by Huang et al. in "Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition" and "UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning" published by Li et al. in "IEEE International Conference on Robotics and Automation". Both are binocular privileged monocular visual odometry methods, which use the assistance of binocular privileged information to restore the real scale of predicted depth and pose. However, such methods are still based on color image matching and are still affected by factors such as illumination changes and weak texture areas.

[0004] With the development of lidar sensor technology, privileged depth information provides another more reliable source of true scale. By projecting the three-dimensional point cloud collected by the lidar onto the camera imaging plane, a sparse depth map aligned with the color image can be obtained. This depth map has true scale information and is robust to lighting conditions and weak texture areas, which helps to learn a more powerful visual odometer. For example, Wang et al. published "Recurrent neural network for (un-)supervised learning of monocular video visual odometry and depth" in "Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition" using sparse depth maps as the dense depth maps predicted by the true value supervision network, so that the network can restore the true scale. However, this method is too simple to use privileged depth information, and the performance improvement achieved is limited. Summary of the invention

[0005] In order to solve the problems existing in the background technology, the present invention provides a deep privileged monocular visual odometry method based on cross-modal knowledge distillation, which uses a pre-trained visual-laser odometry network as a teacher network, and guides the training of the monocular visual odometry network through the encoder features extracted from the network, the predicted dense depth map, and the predicted inter-frame pose through the cross-modal knowledge distillation module. At the same time, the knowledge distillation condition judgment module is used to avoid incorrect guidance of the teacher network.

[0006] The present invention utilizes cross-modal knowledge distillation, and can more effectively utilize the deep privileged information compared to the deep privileged visual odometry method that uses sparse depth maps as the true value. It guides the training of the monocular visual odometry network in three parts: encoder features, predicted dense depth maps, and predicted poses. While restoring the true scale of the prediction results, it also achieves significant performance improvement. Compared with the binocular privileged visual odometry method, the present invention also has better performance in the main error indicators.

[0007] The technical solution adopted by the present invention is:

[0008] (1) Use color cameras and lidar to jointly collect color images and 3D point clouds of multiple scenes and construct a visual-laser odometry dataset;

[0009] The scene described in the present invention may be an outdoor scene.

[0010] (2) Establish a visual-laser odometry network VLO and a monocular visual odometry network VO, input the visual-laser odometry dataset into the visual-laser odometry network VLO and the monocular visual odometry network VO, and use the Adam optimization algorithm to perform the first training respectively;

[0011] (3) The visual-laser odometry network VLO is used to further train the monocular visual odometry network VO through the cross-modal knowledge distillation module;

[0012] (4) After the training is completed, the color image sequences of the continuous moments to be tested are grouped and then respectively input into the trained monocular visual odometry network VO, and the target frame color image I of each group is output. t The corresponding dense depth map And the inter-frame pose in is the translation vector, Represents the Euler angle vector; finally, the inter-frame poses of each group are multiplied to obtain the global pose, which is used as the complete motion trajectory of the color image sequence at consecutive moments to be measured, thus realizing the processing of monocular visual odometer.

[0013] The segmentation of the color image sequence at the continuous moments to be measured is to divide the color image sequence at the continuous moments to be measured into a plurality of groups consisting of continuous odd-numbered images according to the time sequence.

[0014] The inter-frame poses described in the present invention all refer to the inter-frame poses between each reference frame and the target frame. The predicted inter-frame poses are from the target frame to the reference frame.

[0015] The present invention adopts the visual-laser odometry network VLO to guide the re-optimization of the training of the monocular visual odometry network VO, and then uses the finally trained monocular visual odometry network VO to perform recognition processing on the continuous color image sequence to be tested, so as to obtain better and more accurate results.

[0016] The step (1) specifically comprises: for the same scene, a color camera collects a color image sequence at continuous moments, a laser radar collects a three-dimensional point cloud sequence corresponding to the color image sequence at the same continuous moments, a three-dimensional point cloud and a color image are collected at the same moment, and then the three-dimensional point cloud in the three-dimensional point cloud sequence is projected onto the camera imaging plane of the color camera to obtain a sparse depth map, and the color image sequence is obtained by t , I s > and its corresponding sparse depth map sequence <D t , D s >Composition of visual-laser odometry dataset; I t Represents the target frame color image, I s represents the reference frame color image, D t represents the sparse depth map of the target frame, D s Represents a sparse depth image reference frame.

[0017] The number of color images in the color image sequence and the number of sparse depth maps in the sparse depth map sequence are the same and are both odd numbers; a frame located in the middle of the color image sequence and the sparse depth map sequence is used as the target frame, and the remaining frames are reference frames.

[0018] In the specific implementation, the color image sequence t , I s > is composed of 2N+1 continuous color images, where I t represents the target frame color image, and the subscript t represents the target frame. s Represents the reference frame color image, and the subscript s represents the reference frame, s∈{tN,…,t+N}, s≠t.

[0019] After the target frame and the reference frame are set, the present invention can propagate forward and backward, thereby improving the training effect.

[0020] ​​The network structure of the visual-laser odometer network VLO includes a visual feature encoder, a depth feature encoder, a first depth decoder and a pose predictor; the visual feature encoder and the depth feature encoder process the color image sequence and the corresponding sparse depth map sequence respectively, extract the visual encoder features and the depth encoder features therefrom, and obtain the multimodal encoding features H after the visual encoder features and the depth encoder features are locally fused. vlo , fusion is to add pixels, visual encoder features, deep encoder features, multimodal encoding features H vlo Input to the first deep decoder to obtain four multimodal dense depth maps of different scales The multimodal encoding feature H vlo Input to the pose predictor to obtain the multimodal predicted inter-frame pose Where i represents the scale order of the dense depth map, is the multimodal predicted translation vector, Represents the multimodal predicted Euler angle vector.

[0021] In the visual laser odometry network VLO,

[0022] The depth feature encoder is mainly composed of five consecutive convolution modules connected in sequence. The sparse depth map sequence is input into the depth feature encoder. First, every three consecutive sparse depth maps are divided into a group according to the time sequence. The three sparse depth maps in each group are channel-connected to form an initial sparse depth map input feature. The initial sparse depth map input feature is processed in sequence by five consecutive convolution modules respectively. The outputs of the first to fourth convolution modules are used as the first-order to fourth-order depth encoder features respectively, and the scales are reduced successively. The output of the fifth convolution module is used as the fifth-order depth encoder feature. The scales of the fourth-order depth encoder feature and the fifth-order depth encoder feature are the same; the first-order to fifth-order depth encoder features constitute the depth encoder feature.

[0023] The visual feature encoder is mainly composed of five consecutive convolution modules connected in sequence. A color image sequence is input into the visual feature encoder. First, every three consecutive color images are divided into a group according to the time sequence. All three channels of the three color images in each group are connected to form an initial color image input feature. The initial color image input feature is processed in sequence by five consecutive convolution modules respectively. The outputs of the first to fourth convolution modules are used as the first-order to fourth-order visual encoder features respectively, and the scales are reduced successively. The output of the fifth convolution module is used as the fifth-order visual encoder feature. The scales of the fourth-order visual encoder feature and the fifth-order depth encoder feature are the same; the first-order to fifth-order visual encoder features constitute the visual encoder feature.

[0024] The first depth decoder is mainly composed of five deconvolution modules with increasing scales, four first feature fusion modules and four depth predictors. The five deconvolution modules are connected in sequence, and a first feature fusion module is connected between each two adjacent deconvolution modules. The multimodal coding feature H vlo Input into the first deconvolution module; the output of the first deconvolution module, the first-order depth encoder features, and the first-order visual encoder features are all input into the first first feature fusion module, and the output of the first first feature fusion module is directly input into the second deconvolution module. The output of the second deconvolution module is processed by the first depth predictor and outputs a multimodal dense depth map of the first order scale. The output of the second deconvolution module, the second-order depth encoder features, and the second-order visual encoder features are all input into the second first feature fusion module. The output of the second first feature fusion module is directly input into the third deconvolution module. The output of the third deconvolution module is processed by the second depth predictor and outputs a multimodal dense depth map of the second order scale. The output of the third deconvolution module, the third-order depth encoder features, and the third-order visual encoder features are all input into the third first feature fusion module. The output of the third first feature fusion module is directly input into the fourth deconvolution module. The output of the fourth deconvolution module is processed by the third depth predictor to output a multimodal dense depth map of the third order scale. The output of the fourth deconvolution module, the fourth-order depth encoder features, and the fourth-order visual encoder features are all input into the fourth first feature fusion module. The output of the fourth first feature fusion module is directly input into the fifth deconvolution module. The output of the fifth deconvolution module is processed by the fourth depth predictor to output a multimodal dense depth map of the fourth order scale.

[0025] The depth predictor is composed of only one convolution module.

[0026] The visual encoder features output by the visual feature encoder, the depth encoder features output by the depth feature encoder, and the features output by the deconvolution module immediately before the first feature fusion module are all input into the first feature fusion module. In the first feature fusion module, the visual encoder features output by the visual feature encoder and the features output by the deconvolution module immediately before the first feature fusion module are first connected, and the connected result is then added to the depth encoder features output by the first depth feature encoder as the output of the first feature fusion module, where the addition refers to pixel addition;

[0027] The pose predictor specifically includes a convolution module and two convolution pooling units, the input of the convolution module is the input of the pose predictor, the output of the convolution module is respectively input into the two convolution pooling units, and the output of the two convolution pooling units is used as the output of the pose predictor; each convolution pooling unit has the same structure, which is composed of a first convolution module, a global average pooling module and a second convolution module connected in sequence, each global average pooling module is composed of a global average pooling operation, and the two convolution pooling units are respectively used to process the translation vector and Euler angle vector in the output inter-frame pose. The input of the pose predictor is first processed by the convolution module and then the output is respectively input into the two convolution pooling units, and the output of the two convolution pooling units is respectively used as the translation vector and Euler angle vector in the inter-frame pose.

[0028] The network structure of the monocular visual odometer network VO includes a visual feature encoder, a second depth decoder and a pose predictor; the visual feature encoder processes the color image sequence and extracts the visual encoder feature as the visual encoding feature H vo , the visual encoding feature H vo Input to the second depth decoder to obtain four visually dense depth maps of different scales The visual encoding feature H vo Input to the pose predictor to get the visual predicted inter-frame pose Where i represents the scale order of the dense depth map, is the visual prediction translation vector, Represents the visual prediction Euler angle vector.

[0029] The topological structures of the visual feature encoder and the pose predictor in the visual-laser odometry network and the monocular visual odometry network are respectively the same, that is, the first visual feature encoder and the second visual feature encoder have the same topological structure, and the first pose predictor and the second pose predictor have the same topological structure;

[0030] However, the topological structures of the depth decoders are different, that is, the topological structures of the first depth decoder and the second depth decoder are different.

[0031] The second depth decoder is mainly composed of five deconvolution modules with increasing scales, four second feature fusion modules and four depth predictors. The five deconvolution modules are connected in sequence, and a second feature fusion module is connected between each two adjacent deconvolution modules. The visual coding feature H voInput into the first deconvolution module; the output of the first deconvolution module, the first-order depth encoder features, and the first-order visual encoder features are all input into the first second feature fusion module, and the output of the first second feature fusion module is directly input into the second deconvolution module. The output of the second deconvolution module is processed by the first depth predictor and outputs a visual dense depth map of the first order scale. The output of the second deconvolution module, the second-order depth encoder features, and the second-order visual encoder features are all input into the second second feature fusion module. The output of the second second feature fusion module is directly input into the third deconvolution module. The output of the third deconvolution module is processed by the second depth predictor and outputs a second-order scale visual dense depth map. The output of the third deconvolution module, the third-order depth encoder features, and the third-order visual encoder features are all input into the third second feature fusion module. The output of the third second feature fusion module is directly input into the fourth deconvolution module. The output of the fourth deconvolution module is processed by the third depth predictor to output a third-order scale visual dense depth map. The output of the fourth deconvolution module, the fourth-order depth encoder features, and the fourth-order visual encoder features are all input into the fourth second feature fusion module. The output of the fourth second feature fusion module is directly input into the fifth deconvolution module. The output of the fifth deconvolution module is processed by the fourth depth predictor to output a visually dense depth map of the fourth order scale.

[0032] The depth predictor is composed of only one convolution module.

[0033] The visual encoder features output by the visual feature encoder and the features output by the deconvolution module immediately before the second feature fusion module are input into the second feature fusion module. In the second feature fusion module, the visual encoder features output from the visual feature encoder and the features output from the deconvolution module immediately before the second feature fusion module are connected as the output of the second feature fusion module, and the addition refers to pixel addition.

[0034] Each convolution module is composed of a convolution operation, a batch normalization operation, and an activation function connected in sequence. Each deconvolution module is composed of a deconvolution operation, a batch normalization operation, and an activation function connected in sequence.

[0035] During the training process of step (2),

[0036] The following visual-laser total loss function is set when training the visual-laser odometry network described in (2.1)

[0037]

[0038] When training the monocular visual odometer network described in (2.2), the following monocular visual total loss function is set

[0039]

[0040] in, represents the view synthesis loss function, represents the deep self-supervised loss function, represents the deep smoothing loss function, λ vs , ds and λ sm They represent the view synthesis loss function respectively. Deep Self-Supervised Loss Function Deep Smoothing Loss Function The weight of .

[0041] The view synthesis loss function Specifically:

[0042]

[0043]

[0044] Among them, s represents the reference frame, Represents the reference frame color image I s The synthetic target frame color image is obtained by transforming the target frame to the target frame. The synthetic target frame color image is obtained by transforming the target frame to the target frame. t The dense depth map And the target frame color image I t With the reference frame color image I s The inter-frame pose The reference frame color image I s The image obtained by transforming to the target frame; f() represents the transformation function of synthesizing the color image of the target frame; V s Represents the reference frame color image I s The binary visible mask set when transforming to the target frame, α s represents the balanced error scalar, SSIM() represents the image structure similarity evaluation function, ⊙ represents element-by-element multiplication; ||||1 represents the operation of taking the L1 norm, represents the rigid transformation matrix;

[0045] The deep self-supervised loss function The sparse depth map D of the input target frame t To predict the dense depth map Supervision is performed, which is expressed as:

[0046]

[0047] Among them, M(D t ) represents the sparse depth map D t The valid pixel binary mask in , where the depth value of the position in the sparse depth map is greater than 0 is 1, and the rest of the positions are 0;

[0048] The deep smoothing loss function It is expressed as:

[0049]

[0050] in, and They represent the operations of calculating the horizontal and vertical gradients respectively, and e represents a natural constant.

[0051] In the step (3),

[0052] Utilize the multimodal encoding feature H output by the visual laser odometry network VLO vlo , multimodal prediction of inter-frame pose and four multimodal dense depth maps at different scales To the visual encoding feature H output by the monocular visual odometer network VO vo , visual prediction of inter-frame pose and four visually dense depth maps at different scales Conduct training, specifically:

[0053] S1: according to the network parameters of the visual-laser odometry network VLO obtained from the training result of the visual-laser odometry network VLO in step (2), the network parameters of the visual-laser odometry network VLO are fixed so as to remain unchanged during the training process of the entire step (3);

[0054] S2: Dense depth maps at four different scales output using the visual laser odometry network VLO As pseudo-truth labels, four dense depth maps of different scales output by the supervised monocular visual odometry network VO The first supervision loss is calculated by the following formula:

[0055]

[0056] S3: Inter-frame pose output using the visual laser odometry network VLO As a pseudo-truth label, supervise the inter-frame pose output by the monocular visual odometry network VO The second supervision loss is calculated by the following formula:

[0057]

[0058] Among them, α rot is a fixed coefficient used to balance the errors of translation and rotation components; Represent the multimodal predicted translation vector and the visual predicted translation vector respectively, Respectively represent the multimodal predicted Euler angle vector and the visual predicted Euler angle vector;

[0059] S4: Multimodal encoding feature H extracted from the visual laser odometry network VLO vlo As the pseudo-truth label, the visual encoder feature H extracted from the middle of the supervised monocular visual odometry network VO vo , the third supervision loss is calculated by the following formula:

[0060]

[0061] Among them, N p Representing multimodal encoding features The total number of pixels in (H vlo With H vo The total number of pixels is the same as Represent the multimodal encoding features H vlo and the visual encoder features H vo The feature vector at pixel i, 〈·,·> calculates the dot product between two vectors, and ||||2 represents the operation of calculating the vector modulus;

[0062] S5: Set a knowledge distillation condition coefficient φ dk_c To selectively introduce the visual-laser odometry network:

[0063]

[0064] in Represents the reference frame color image I under network m s The synthetic target frame color image obtained by transforming to the target frame, m represents the network identification parameter, φ kd_c represents the knowledge distillation condition coefficient, represents the rigid transformation matrix of network m, Represents the multimodal dense depth map predicted by VLO; represents the binary visual mask set when using the rigid transformation matrix of network m for view synthesis, α s represents the scalar of the balance error term;

[0065] φ kd_c =1 means that the inter-frame pose predicted by the visual laser odometry network VLO is better than the inter-frame pose predicted by the monocular visual odometry network VO, otherwise φkd_c =0.

[0066] S6: The relevant training loss function for establishing the cross-modal knowledge distillation module is as follows:

[0067]

[0068] in, represents the relevant training loss function of the cross-modal knowledge distillation module, λ p , d and λ h is the weight coefficient of the first supervision loss, the second supervision loss, and the third supervision loss;

[0069] In most cases, the dense depth map predicted by the visual laser odometry network VLO is better than the monocular visual odometry network VO, and the knowledge distillation condition coefficient φ kd_c Only affects the other two items.

[0070] S7: Overall training loss function of the monocular visual odometry network VO The settings are as follows:

[0071]

[0072] Among them, λ vo and λ kd is the weight coefficient of the total monocular vision loss function and the related training loss function.

[0073] The method of the present invention uses a pre-trained visual-laser odometry network as a teacher network to guide the training of a monocular visual odometry network through cross-modal knowledge distillation. The visual-laser odometry network takes continuous color images and corresponding sparse depth map sequences as input and is trained in a self-supervised manner. The monocular visual odometry network only inputs continuous color image sequences and is trained in an unsupervised manner. In cross-modal knowledge distillation, the visual-laser odometry network will guide the learning of the monocular visual odometry network in three parts: the predicted dense depth map, the inter-frame pose, and the learned encoder features, and a distillation condition judgment module is used to avoid potential misguidance of the teacher network.

[0074] This method first constructs a visual-laser odometry dataset, which includes continuous color images and corresponding sparse depth map sequences; the data to be trained is used to train the visual-laser odometry network and the monocular visual odometry network in small batches; the parameters of the trained visual-laser odometry network are fixed, and the training of the monocular visual odometry network is further guided by the cross-modal knowledge distillation module; the monocular visual odometry network after training is used for monocular visual odometry applications.

[0075] This method can estimate dense depth maps and inter-frame poses at real scale from monocular image sequences, and is at the forefront of the field in terms of translation and rotation performance.

[0076] The present invention has the following beneficial effects:

[0077] 1. Compared with the binocular privileged visual odometry method, this method utilizes more reliable depth privileged information and is more robust to environmental factors such as weak texture areas and lighting changes.

[0078] 2. Compared with the deep privileged method that directly uses the deep privileged information to supervise the prediction of dense depth maps, this method uses cross-modal knowledge distillation to provide guidance for the training of monocular visual odometry from three parts: encoder features extracted from the network, dense depth maps predicted by the network, and inter-frame poses. It makes more full use of the deep privileged information and achieves significant performance improvements.

[0079] 3. The present invention does not require additional true value labels during the entire training process, so it has good versatility and universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a flow chart of an example of the present invention.

[0081] Figure 2 It is a schematic diagram of the visual-laser odometer network model designed in the example of the present invention.

[0082] Figure 3 It is a schematic diagram of the monocular visual odometer network model designed in the example of the present invention.

[0083] Figure 4 It is a schematic diagram of the cross-modal knowledge distillation training process designed in the example of the present invention.

[0084] Figure 5 This is an example of the pose estimation result of the odometer method designed in the example of the present invention. DETAILED DESCRIPTION

[0085] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0086] like Figure 1 As shown, the embodiments and specific implementation processes of the present invention are as follows:

[0087] S01, collect multiple continuous color images and sparse depth maps for positioning scenes, wherein the sparse depth map is obtained by projecting the three-dimensional point cloud collected by the laser radar onto the camera imaging plane. Use 2N+1 continuous color images and their corresponding sparse depth images to form a training sample (N=1 in the present invention).

[0088] S02, build a visual-laser odometer network and a monocular visual odometer network. Figure 2 As shown in FIG. 1 , the visual-laser odometer network of the present invention is composed of four parts: a visual feature encoder, a depth feature encoder, a first depth decoder, and a pose predictor. The visual feature encoder and the depth feature encoder extract the visual encoder features and the depth encoder features from the three continuous color images and the three corresponding sparse depth maps respectively, and locally fuse them by pixel addition to obtain the multimodal coding feature H vlo The pose predictor is based on H vlo As input, predict the inter-frame pose The first depth encoder uses H vlo , visual encoder features and depth encoder features as input, predicting multimodal dense depth maps of four different scales Where i represents the scale order of the depth map.

[0089] The monocular visual odometer network described in the present invention is composed of three parts: a visual feature encoder, a depth feature encoder, a second depth decoder and a pose predictor. Figure 3 As shown in Figure 2, the visual feature encoder extracts the visual encoder features from the three consecutive color images input as the visual encoding features H. vo The pose predictor is based on H vo As input, predict the inter-frame pose The second depth decoder uses H vo , visual encoder features as input, predicting four visual dense depth maps of different scales Where i represents the scale order of the depth map.

[0090] S02-1, use the Adam optimization algorithm to iteratively train the visual-laser odometry network. The loss function of the training is:

[0091]

[0092] S02-2, use the Adam optimization algorithm to iteratively train the monocular visual odometry network, and the training loss function is:

[0093]

[0094] S03, fix the parameters of the visual-laser odometry network model trained in S02-1, such as Figure 4 As shown, the monocular visual odometry network trained in S02-2 is further trained through cross-modal knowledge distillation guidance. The loss function of the training is:

[0095]

[0096] in is the loss function described in S02-2, The loss function related to cross-modal knowledge distillation is:

[0097]

[0098] S04, using the monocular visual odometry network trained in S03 to perform pose estimation tasks. In the specific implementation, a complete image sequence is divided into multiple samples of 3 consecutive color images, which are input into the monocular visual odometry network to predict inter-frame pose. The global pose transformation matrix of the starting image is set as the unit matrix, and the global pose transformation matrix corresponding to each image is obtained by accumulating the inter-frame poses of consecutive images to achieve trajectory prediction of the entire image sequence.

[0099] In order to verify the effectiveness of the present invention, the present invention is verified on the public visual-laser odometry dataset KITTIOdometry Benchmark, and compared with the existing monocular visual odometry method, binocular privileged visual odometry method and deep privileged visual odometry method:

[0100] KITTI Odometry Benchmark has 22 sequences from 00 to 21, of which sequences 00 to 10 provide the true value of the pose for verification. Each sequence uses 4 monocular cameras (2 grayscale cameras + 2 color cameras) and a Velodyne HDL-64 lidar to collect images and 3D point clouds respectively, and the images and 3D point clouds are aligned in time sequence. Taking sequence 00 as an example, each camera collects 4541 images with a resolution of 1241x376, and the lidar collects 4541 frames of 3D point clouds. By projecting the 3D point cloud onto the imaging plane of the corresponding camera, a sparse depth map of the same resolution can be obtained, and its density is about 5%. Considering computational efficiency, the input image resolution is scaled to 192x624, and 3 consecutive color images and corresponding sparse depth maps are used as training samples. Like most current self-supervised visual odometry methods, sequences 00-08 are used for training, and sequences 09-10 are used for testing.

[0101] This paper mainly uses the official performance indicators in KITTI Odometry Benchmark to evaluate network performance: average translation relative drift t rel (unit: % / 100m), and the average rotational relative drift r rel (Unit: degrees / 100m). These two indicators are calculated on subsequences of length (100m, 200m, ..., 800m) of the entire sequence.

[0102] The details of network training are as follows:

[0103] In S02, the model parameters of the vision-laser visual odometry network and the monocular visual odometry network are randomly initialized by Xavier and optimized by Adam optimizer. The batch size of each input network is set to 4. The initial value of the learning rate is 0.0002, which is halved every 70K iterations, and a total of 180K iterations are trained.

[0104] In S03, the model parameters of the visual-laser odometry network trained in S02-1 are fixed to guide the monocular visual odometry network trained in S02-2 to continue training. The batch size of each input network is set to 4. The initial value of the learning rate is 0.00005, and it decays to 0.8 times the original value every 50K iterations, and a total of 200K iterations are trained.

[0105] The experiment mainly includes two parts. The first part is the control variable experiment of each module in the present invention, which is used to illustrate the effectiveness of each module in the present invention. The second part is the comparative experiment between the present invention method and the existing mainstream visual odometer method.

[0106] Part I: Control variable experiments of each module in the present invention, to illustrate the effectiveness of each module in the present invention;

[0107] Effectiveness of the cross-modal knowledge distillation module: The cross-modal knowledge distillation module mainly consists of three parts: output space distillation (OD), hidden feature distillation (HD) and distillation condition judgment (DCC). OD means using the output results of the visual-laser odometry network (dense depth map, inter-frame pose) to guide the training of the monocular visual odometry. HD means using the multimodal encoding features of the visual-laser odometry network to guide the encoder features learned by the monocular visual odometry network. DCC avoids incorrect guidance by comparing the quality of the current prediction results of the teacher network and the student network. Depending on whether the functions of these three parts are used in the distillation process, four variants of the monocular visual odometry network can be obtained, denoted as KD-VO1 to KD-VO4. In addition, a method of directly using sparse depth maps as the true value to directly supervise the predicted dense depth maps is tested, denoted as VO-DS. The experimental results are shown in Table 1. The * mark in the table indicates that the prediction results of the model cannot restore the true scale, and the scale-corrected results are shown in the table. Figure 5 The predicted trajectories of VLO, VO, VO-DS, and KD-VO4 on the test sequence are shown for qualitative comparison.

[0108] Experimental results show that all three parts in the cross-modal knowledge distillation module contribute to performance improvement. In addition, compared with the method of directly using sparse depth maps as true values ​​for supervision, this method can better utilize deep privileged information and achieve more significant performance improvement.

[0109] Table 1. Performance comparison of various variants of monocular visual odometer network in KITTI Odometry Benchmark

[0110]

[0111]

[0112] Training strategy: In the cross-modal knowledge distillation process of the present invention, the visual-laser odometry network is used as the teacher network, and its model parameters are fixed. Another training strategy is to jointly train the visual-laser odometry network and the monocular visual odometry network, and the results are shown in Table 2. The experimental results show that during the joint training process, the noise of the monocular visual odometry network affects the training of the visual-laser odometry network, resulting in a decrease in its performance; similarly, the performance of the monocular visual odometry network also decreases. This shows that the current training strategy is more effective.

[0113] Table 2. Performance comparison of the visual-laser odometry network and the monocular visual odometry network under different training strategies on the KITTIOdometry Benchmark

[0114]

[0115] Part II: Table 3 shows the comparison experiment between this method and the existing visual odometry methods. The compared methods include 6 standard monocular visual odometry methods, 4 binocular privileged visual odometry methods and 1 deep privileged visual odometry method. Symbol The marked method is a non-end-to-end method (i.e., the network does not directly predict the inter-frame pose). It predicts the optical flow and dense depth map, removes outliers through RANSAC, and obtains the predicted inter-frame pose through additional optimization algorithms. All standard monocular VO methods in the table cannot restore the true scale. The table shows their performance indicators after scale alignment.

[0116] Standard monocular visual odometry methods often require more complex network structures, such as SAVO, SGANVO, and ASANet, and cannot restore the true scale. This method uses a simple network structure to achieve more accurate pose prediction results with true scale. Compared with binocular privileged visual odometry methods that are sensitive to illumination changes and weak texture areas, the deep privileged method shows stronger robustness and thus has better results. It is even comparable to methods that use additional optimization modules. In addition, compared to the deep privileged method that directly uses sparse depth maps as supervisory signals, this method can better utilize the deep privileged information to guide training, achieving more significant performance improvements.

[0117] Table 3. Performance comparison with existing visual odometry methods on the KITTI Odometry Benchmark

[0118]

[0119] The above description is only a specific embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent example of equivalent changes. However, any simple modification, equivalent change and modification made to the above examples based on the technical essence of the present invention without departing from the technical solution of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A deep privileged monocular visual odometry method based on cross-modal knowledge distillation, characterized in that the method include: (1) Use color cameras and lidar to jointly collect color images and 3D point clouds of scenes and construct a visual-laser odometry dataset; (2) The visual-laser odometry dataset is input into the visual-laser odometry network VLO and the monocular visual odometry network VO and trained separately using the optimization algorithm; The network structure of the visual-laser odometer network VLO includes a visual feature encoder, a depth feature encoder, a first depth decoder and a pose predictor; the visual feature encoder and the depth feature encoder process the color image sequence and the corresponding sparse depth map sequence respectively, extract the visual encoder features and the depth encoder features therefrom, and obtain the multimodal encoding features H after the visual encoder features and the depth encoder features are locally fused. vlo , the visual encoder features, deep encoder features, and multimodal encoding features H vlo Input to the first deep decoder to obtain four multimodal dense depth maps of different scales The multimodal encoding feature H vlo Input to the pose predictor to obtain the multimodal predicted inter-frame pose Where i represents the scale order of the dense depth map, is the multimodal predicted translation vector, Represents the multimodal predicted Euler angle vector; The network structure of the monocular visual odometer network VO includes a visual feature encoder, a second depth decoder and a pose predictor; the visual feature encoder processes the color image sequence and extracts the visual encoder feature as the visual encoding feature H vo , the visual encoding feature H vo Input to the second depth decoder to obtain four visually dense depth maps of different scales The visual encoding feature H vo Input to the pose predictor to get the visual predicted inter-frame pose Where i represents the scale order of the dense depth map, is the visual prediction translation vector, Represents the visual prediction Euler angle vector; (3) The visual-laser odometry network VLO is used to further train the monocular visual odometry network VO through the cross-modal knowledge distillation module; (4) After the training is completed, the color image sequences of the continuous moments to be tested are grouped and then respectively input into the trained monocular visual odometry network VO, and the target frame color image I of each group is output. t The corresponding dense depth map And the inter-frame pose in is the translation vector, Represents the Euler angle vector; finally, the inter-frame poses of each group are multiplied to obtain the global pose, which is used as the complete motion trajectory of the color image sequence at consecutive moments to be measured, thus realizing the processing of monocular visual odometer.

2. The deep privileged monocular visual odometry method based on cross-modal knowledge distillation according to claim 1, characterized in that: The step (1) specifically comprises: for the same scene, a color camera collects a color image sequence at continuous moments, a laser radar collects a three-dimensional point cloud sequence corresponding to the color image sequence at the same continuous moments, and then projects the three-dimensional point cloud in the three-dimensional point cloud sequence onto the camera imaging plane of the color camera to obtain a sparse depth map, and the color image sequence and its corresponding sparse depth map sequence constitute a visual-laser odometry data set.

3. The deep privileged monocular visual odometry method based on cross-modal knowledge distillation according to claim 2, characterized in that: The number of color images in the color image sequence and the number of sparse depth maps in the sparse depth map sequence are the same and are both odd numbers; a middle frame in the color image sequence and the sparse depth map sequence is used as a target frame, and the remaining frames are reference frames.

4. The deep privileged monocular visual odometry method based on cross-modal knowledge distillation according to claim 1, characterized in that: In the visual laser odometry network VLO, The depth feature encoder is mainly composed of five consecutive convolution modules connected in sequence. The sparse depth map sequence is input into the depth feature encoder. First, every three consecutive sparse depth maps are divided into a group according to the time sequence. The three sparse depth maps in each group are channel-connected to form an initial sparse depth map input feature. The initial sparse depth map input feature is processed in sequence by five consecutive convolution modules respectively. The outputs of the first to fourth convolution modules are used as the first-order to fourth-order depth encoder features respectively, and the scales are reduced successively. The output of the fifth convolution module is used as the fifth-order depth encoder feature, and the scales of the fourth-order depth encoder feature and the fifth-order depth encoder feature are the same; The visual feature encoder is mainly composed of five consecutive convolution modules connected in sequence. A color image sequence is input into the visual feature encoder. First, every three consecutive color images are divided into a group according to the time sequence. All three channels of the three color images in each group are connected to form an initial color image input feature. The initial color image input feature is processed in sequence by five consecutive convolution modules respectively. The outputs of the first to fourth convolution modules are used as the first-order to fourth-order visual encoder features respectively, and the scales are reduced successively. The output of the fifth convolution module is used as the fifth-order visual encoder feature. The scales of the fourth-order visual encoder feature and the fifth-order depth encoder feature are the same. The first depth decoder is mainly composed of five deconvolution modules with increasing scales, four first feature fusion modules and four depth predictors. The five deconvolution modules are connected in sequence, and a first feature fusion module is connected between each two adjacent deconvolution modules. The multimodal coding feature H vlo Input into the first deconvolution module; the output of the first deconvolution module, the first-order depth encoder features, and the first-order visual encoder features are all input into the first first feature fusion module, and the output of the first first feature fusion module is directly input into the second deconvolution module. The output of the second deconvolution module is processed by the first depth predictor and outputs a multimodal dense depth map of the first order scale. The output of the second deconvolution module, the second-order depth encoder features, and the second-order visual encoder features are all input into the second first feature fusion module. The output of the second first feature fusion module is directly input into the third deconvolution module. The output of the third deconvolution module is processed by the second depth predictor and outputs a multimodal dense depth map of the second order scale. The output of the third deconvolution module, the third-order depth encoder features, and the third-order visual encoder features are all input into the third first feature fusion module. The output of the third first feature fusion module is directly input into the fourth deconvolution module. The output of the fourth deconvolution module is processed by the third depth predictor to output a multimodal dense depth map of the third order scale. The output of the fourth deconvolution module, the fourth-order depth encoder features, and the fourth-order visual encoder features are all input into the fourth first feature fusion module. The output of the fourth first feature fusion module is directly input into the fifth deconvolution module. The output of the fifth deconvolution module is processed by the fourth depth predictor to output a multimodal dense depth map of the fourth order scale. In the first feature fusion module, the visual encoder feature output from the visual feature encoder is first connected with the feature output from the deconvolution module immediately connected to the front of the first feature fusion module itself, and the connected result is then added with the depth encoder feature output from the first depth feature encoder as the output of the first feature fusion module; The pose predictor specifically includes a convolution module and two convolution pooling units, the input of the convolution module is the input of the pose predictor, the output of the convolution module is respectively input into the two convolution pooling units, and the output of the two convolution pooling units is used as the output of the pose predictor; each convolution pooling unit has the same structure, which is composed of a first convolution module, a global average pooling module and a second convolution module connected in sequence, each global average pooling module is composed of a global average pooling operation, and the two convolution pooling units are respectively used to process the translation vector and the Euler angle vector in the output inter-frame pose.

5. The deep privileged monocular visual odometry method based on cross-modal knowledge distillation according to claim 1, characterized in that: The topological structures of the visual feature encoder and the pose predictor in the visual-laser odometry network and the monocular visual odometry network are respectively the same; The second depth decoder is mainly composed of five deconvolution modules with increasing scales, four second feature fusion modules and four depth predictors. The five deconvolution modules are connected in sequence, and a second feature fusion module is connected between each two adjacent deconvolution modules. The visual coding feature H vo Input into the first deconvolution module; the output of the first deconvolution module, the first-order depth encoder features, and the first-order visual encoder features are all input into the first second feature fusion module, and the output of the first second feature fusion module is directly input into the second deconvolution module. The output of the second deconvolution module is processed by the first depth predictor and outputs a visual dense depth map of the first order scale. The output of the second deconvolution module, the second-order depth encoder features, and the second-order visual encoder features are all input into the second second feature fusion module. The output of the second second feature fusion module is directly input into the third deconvolution module. The output of the third deconvolution module is processed by the second depth predictor and outputs a second-order scale visual dense depth map. The output of the third deconvolution module, the third-order depth encoder features, and the third-order visual encoder features are all input into the third second feature fusion module. The output of the third second feature fusion module is directly input into the fourth deconvolution module. The output of the fourth deconvolution module is processed by the third depth predictor to output a third-order scale visual dense depth map. The output of the fourth deconvolution module, the fourth-order depth encoder features, and the fourth-order visual encoder features are all input into the fourth second feature fusion module. The output of the fourth second feature fusion module is directly input into the fifth deconvolution module. The output of the fifth deconvolution module is processed by the fourth depth predictor to output a visually dense depth map of the fourth order scale. In the second feature fusion module, the visual encoder features output from the visual feature encoder and the features output from the deconvolution module immediately connected to the front of the second feature fusion module are connected as the output of the second feature fusion module.

6. The deep privileged monocular visual odometry method based on cross-modal knowledge distillation according to claim 1, characterized in that: During the training process of step (2), The following visual-laser total loss function is set when training the visual-laser odometry network described in (2.1) When training the monocular visual odometer network described in (2.2), the following monocular visual total loss function is set in, represents the view synthesis loss function, represents the deep self-supervised loss function, represents the deep smoothing loss function, λ vs , ds and λ SM They represent the view synthesis loss function respectively. Deep Self-Supervised Loss Function Deep Smoothing Loss Function The weight of .

7. The deep privileged monocular visual odometry method based on cross-modal knowledge distillation according to claim 6, characterized in that: The view synthesis loss function Specifically: Among them, s represents the reference frame, Represents the reference frame color image I s The synthetic target frame color image is obtained by transforming the target frame to the target frame. The synthetic target frame color image is obtained by transforming the target frame to the target frame. t The dense depth map And the target frame color image I t With the reference frame color image I s The inter-frame pose The reference frame color image I s The image obtained by transforming to the target frame; f() represents the transformation function of synthesizing the color image of the target frame; V s Represents the reference frame color image I s The binary visible mask set when transforming to the target frame, α s represents the balanced error scalar, SSIM() represents the image structure similarity evaluation function, ⊙ represents element-by-element multiplication; ‖ ‖1 represents the operation of taking the L1 norm, represents the rigid transformation matrix; The deep self-supervised loss function The sparse depth map D of the input target frame t To predict the dense depth map Supervision is performed, which is expressed as: Among them, M(D t ) represents the sparse depth map D t The valid pixel binary mask in , where the depth value of the position in the sparse depth map is greater than 0 is 1, and the rest of the positions are 0; The deep smoothing loss function It is expressed as: in, and They represent the operations of calculating the horizontal and vertical gradients respectively, and e represents a natural constant.

8. The deep privileged monocular visual odometry method based on cross-modal knowledge distillation according to claim 1, characterized in that: In the step (3), Utilize the multimodal encoding feature H output by the visual laser odometry network VLO vlo , multimodal prediction of inter-frame pose and four multimodal dense depth maps at different scales To the visual encoding feature H output by the monocular visual odometer network VO vo , visual prediction of inter-frame pose and four visually dense depth maps at different scales Conduct training, specifically: S1: fixed network parameters of the visual-laser odometry network VLO; S2: Dense depth maps at four different scales output using the visual laser odometry network VLO As pseudo-truth labels, four dense depth maps of different scales output by the supervised monocular visual odometry network VO The first supervision loss is calculated by the following formula: S3: Inter-frame pose output using the visual laser odometry network VLO As a pseudo-truth label, supervise the inter-frame pose output by the monocular visual odometry network VO The second supervision loss is calculated by the following formula: Among them, α rot is a fixed coefficient; Represent the multimodal predicted translation vector and the visual predicted translation vector respectively, Respectively represent the multimodal predicted Euler angle vector and the visual predicted Euler angle vector; S4: Multimodal encoding feature H extracted from the visual laser odometry network VLO vlo As the pseudo-truth label, the visual encoder feature H extracted from the middle of the supervised monocular visual odometry network VO vo , the third supervision loss is calculated by the following formula: Among them, N p Representing multimodal encoding features The total number of pixels in (H vlo With H vo The total number of pixels is the same as Represent the multimodal encoding features H vlo and the visual encoder features H vo The feature vector at pixel i, 〈·,·〉 calculates the dot product between two vectors, ‖ ‖2 represents the operation of calculating the vector modulus; S5: Set a knowledge distillation condition coefficient φ kd_c To selectively introduce the visual-laser odometry network: in, Represents the reference frame color image I under network m s The synthetic target frame color image obtained by transforming to the target frame, m represents the network identification parameter, φ kd_c represents the knowledge distillation condition coefficient, represents the rigid transformation matrix of network m, Represents the multimodal dense depth map predicted by VLO; represents the binary visual mask set when using the rigid transformation matrix of network m for view synthesis, α s represents the scalar of the balance error term; S6: The relevant training loss function for establishing the cross-modal knowledge distillation module is as follows: in, represents the relevant training loss function of the cross-modal knowledge distillation module, λ p , d and λ h is the weight coefficient of the first supervision loss, the second supervision loss, and the third supervision loss; S7: Overall training loss function of the monocular visual odometry network VO The setup is as follows: Among them, λ vo and λ kd is the weight coefficient of the total monocular vision loss function and the related training loss function.

Images