Unsupervised joint estimation of monocular eye flow, depth and pose based on Transformer

Abstract

A method for unsupervised joint estimation of monocular flow, depth and pose based on Transformer. Specifically, in order to reduce the redundancy of network parameters and structures, and at the same time consider the temporal continuity of the tasks of unsupervised joint estimation of optical flow, depth and pose, the present invention uses Transformer with a global receptive field as the only feature encoder to provide a unified and consistent feature description for the three tasks at the same time. In addition, in order to achieve accuracy improvement between optical flow, depth and pose estimation, the similarity between joint estimation tasks is taken into account. During the training process of the network, the present invention proposes a mutual introduction mechanism to introduce geometric priors into the network, so as to realize bidirectional iterative optimization of optical flow, pose and depth. This method can be used for joint estimation tasks of optical flow, depth and pose for monocular vision with drones, unmanned vehicles or robots as carriers.

Description

Technical Field

[0001] The present invention belongs to the field of monocular image imaging, and particularly relates to a Transformer-based unsupervised joint estimation method for monocular eye flow, depth, and pose. Without the need to train the network using the difficult-to-obtain true values ​​of optical flow, depth, and pose, an unsupervised method is adopted, and only monocular image sequences are used to achieve the joint estimation of the above three different tasks. Background Art

[0002] Monocular camera-based vehicle pose estimation is an important and active research area in computer vision, with significant potential for development in areas such as autonomous driving, visual SLAM, and navigation. Currently, monocular vision-based vehicle pose estimation primarily encompasses two approaches: traditional methods and deep learning-based methods. Traditional methods include feature-based and direct methods. Feature-based methods employ feature matching techniques to extract common features between two images and then perform pose estimation based on the sparse feature points. This approach has long been dominant, but suffers from drawbacks such as time-consuming key feature point extraction, loss of valuable visual information due to sparse feature points, and poor accuracy in low-texture areas. Direct methods, on the other hand, capture camera motion by minimizing the photometric error between two frames, resulting in relatively stable and continuous pose information. However, direct methods suffer from poor accuracy when the vehicle moves at high speeds and are highly sensitive to lighting variations and camera exposure parameters. The rapid development of graphics processing units (GPUs) in recent years has provided high-performance hardware support for deep learning techniques, leading to rapid progress in deep learning-based vehicle pose estimation. The first visual pose estimation networks to emerge were supervised learning-based networks, which employed ground truth (GT) for network training. However, obtaining GT for training pose networks is expensive and difficult to obtain in large quantities. Furthermore, networks trained using supervised methods are prone to accuracy degradation in private, specific test sets. Consequently, unsupervised learning methods have become a hot topic in deep learning visual pose estimation. Most pose estimation methods based on unsupervised learning employ joint depth and pose estimation, leveraging the results of depth and pose estimation networks combined with image warping techniques to provide supervision. To further improve pose estimation accuracy, optical flow information can also be introduced to provide prior information on matching relationships. However, current joint optical flow, depth, and pose estimation methods mostly utilize optical flow to improve pose estimation accuracy, failing to achieve a mutually beneficial interaction between optical flow, pose, and depth estimation. Furthermore, these methods employ independent networks for pose, depth, and optical flow, resulting in parameter redundancy and failing to consider the global consistency of the common features across these three tasks in sequential tasks.

[0003] As a new network structure based on the self-attention mechanism, the Transformer has a global receptive field compared to conventional convolutional neural networks. It has an inherent advantage in capturing long-range feature dependencies, making it particularly suitable for processing time series tasks. This is crucial for the joint estimation of optical flow, depth, and pose. At the same time, to reduce the redundancy of network parameters, the network structure is optimized and the advantages of the Transformer are combined.

[0004] Application number: CN201811444277.8, application name: An unsupervised monocular depth estimation method based on optical flow mask, the method specifically comprises: using a depth estimation network to perform depth estimation on image frames; introducing a camera pose estimation model and an optical flow estimation network when training the depth estimation network; reconstructing the current frame based on the optical flow estimation between two adjacent image frames output by the optical flow estimation network to obtain an optical flow reconstructed image; reconstructing the current frame based on the pose transformation matrix between two adjacent image frames estimated by the camera pose estimation model to obtain a depth reconstructed image; and establishing a loss function based on the depth reconstructed image and the optical flow reconstructed image to jointly train the depth estimation network, the camera pose estimation model, and the optical flow estimation network. The present invention improves the accuracy of depth estimation.

[0005] This application first focuses on depth estimation, using optical flow and pose networks as auxiliary means, and using image reconstruction technology to reconstruct the image using the optical flow and pose information estimated by the optical flow and pose networks. Then, based on the constructed loss function and joint training, the reconstructed image is used to optimize the accuracy of depth estimation.

[0006] The method of this application is to introduce transformer as a feature encoder to provide a unified feature representation with a global receptive field for three different tasks: optical flow, depth, and pose. Secondly, the present invention uses the proposed mutual introduction mechanism to achieve bidirectional iterative optimization of the optical flow and depth and pose estimation networks. That is, while the optical flow network optimizes the pose and depth network, the depth and pose network can also optimize the optical flow network. This improves the accuracy. This is not compared to the "unsupervised monocular depth estimation method based on optical flow mask" that uses optical flow and pose to perform unidirectional estimation of the depth network.

[0007] Application number: CN202110321996.6, application name: A wavelet-based detail enhancement unsupervised depth estimation method, which inputs a frame image in the video into the deep network to obtain the corresponding depth map, and inputs the frame and the adjacent frame into the pose network to obtain the camera pose, and integrates the high frequency of the wavelet decomposition of the image into the network to enhance the depth detail information. This embodiment also uses optical flow estimation to identify and process occluded areas. In the training phase, the depth map, camera pose and original image are used to synthesize the target frame, and the network training is supervised by calculating the loss between the synthesized frame and the original frame. The present invention solves the problems of boundary distortion and loss of details of the depth map in the current depth estimation, and can well estimate the corresponding depth map from the monocular view.

[0008] This application estimates scene depth by leveraging the pose information provided by the pose network for unsupervised training. It then integrates the high-frequency information from the wavelet decomposition of the image into the network to enhance depth details. Finally, it uses optical flow to identify occluded areas in the image, thereby improving depth estimation accuracy.

[0009] Compared to the “A method for unsupervised depth estimation with detail enhancement based on wavelet” which only optimizes the accuracy improvement of depth estimation, this application simultaneously optimizes the accuracy of the three tasks in the optical flow, depth and pose joint estimation network. First of all, the three tasks of optical flow, depth and pose estimation have structural similarities in space. Therefore, the present invention uses Transformer as the only feature encoder to reduce the number of network parameters while providing long-distance dependency features with global consistency for the three tasks. Then, the mutual introduction mechanism is used to achieve iterative optimization of the three tasks of optical flow, depth and pose. Instead of using optical flow and pose to improve the accuracy of depth estimation in a one-way manner as in the “A method for unsupervised depth estimation with detail enhancement based on wavelet”.

[0010] This paper utilizes an improved Transformer as the sole feature encoder for optical flow, depth, and pose estimation, providing a unified and globally consistent feature encoding for subsequent multi-task estimation. Furthermore, a mutual induction mechanism is proposed to mutually enhance the accuracy of the optical flow, depth, and pose decoder branches. Summary of the Invention

[0011] To address the above issues, we propose a Transformer-based unsupervised joint estimation method for monocular optical flow, depth, and pose. This method avoids the drawback of supervised learning, which requires expensive ground truth data for network parameter training. Instead, the network can be trained using only sequential image frames captured by a monocular camera. Unlike most existing optical flow, depth, and pose estimation networks, this method uses the Transformer as the sole feature encoder for all three tasks, providing a unified feature description with a global receptive field for each task, thereby reducing network parameter redundancy. Furthermore, this method proposes a mutual induction mechanism to achieve iterative optimization between optical flow, depth, and pose estimation. By utilizing image warping technology, the optical flow decoder branch serves as a guide for the depth and pose decoder branches, guiding depth and pose generation from the perspective of optical flow estimation. The depth and pose decoder branches also serve as a guide for the optical flow encoder branch. This mutual induction mechanism enables mutual promotion and learning between optical flow, depth, and pose estimation. These improvements reduce redundant network parameters and structure while achieving high-precision unsupervised estimation of optical flow, depth, and pose.

[0012] To achieve the above object, the technical solution adopted by the present invention is:

[0013] The Transformer-based unsupervised joint estimation method of monocular eye flow, depth and pose is as follows:

[0014] (1) Training phase:

[0015] The data in the training phase consists of three consecutive image frames captured by a monocular camera. After being spliced ​​in the channel dimension, they are directly input into the Transformer encoder to encode the features, providing a unified feature representation for the subsequent joint estimation of optical flow, depth, and pose. The features are then decoded by the optical flow, depth, and pose decoders respectively, and the joint estimation of the corresponding information is completed. In this process, an improved unsupervised loss function is adopted, and the spatial structural relationship between optical flow, depth, and pose is utilized to enable the network to achieve unsupervised learning. The network parameters are continuously iterated using gradient backpropagation until the loss tends to be optimal and the training is completed.

[0016] (2) Prediction stage:

[0017] After the network is trained, the network parameters are fixed and three consecutive monocular images from the test set are input. After the Transformer encoder uniformly extracts and encodes the features, they are input to each decoder branch. This allows the estimation of monocular eye flow, depth, and pose to be completed simultaneously. At the same time, the three tasks can also be predicted independently in the prediction stage without interfering with each other.

[0018] As a further improvement of the present invention, the specific meaning of the unsupervised optical flow, depth and pose joint estimation network in step (1) is as follows:

[0019] The proposed method is based on unsupervised learning and achieves joint estimation of monocular gaze flow, depth, and pose. It includes the following two aspects:

[0020] (1) Transformer-based unified feature extractor;

[0021] Currently, most existing joint estimation networks for optical flow, depth, and pose use three different feature encoders to extract features for each task, resulting in redundant network structures and failing to consider global consistency information. Unlike other joint estimation networks;

[0022] Using a Transformer as a unified encoder for the three tasks, only one unified feature extraction can meet the different feature requirements of the three tasks. At the same time, the optical flow, depth, and pose estimation tasks have strong geometric consistency in the three-dimensional structure. The present invention takes advantage of the global receptive field of the Transformer to enable the network to consider global consistency information, which is beneficial to the accuracy improvement of each task. The network consists of a Transformer encoder and optical flow, depth, and pose decoder branches. The relationship between the various network structures is as follows;

[0023] F Transformer =Θ(I t-1,t,t+1 ) (1)

[0024] f t→t-1 ,f t→t+1 =θ(F Transformer ) (2)

[0025] D t =ψ(F Transformer ) (3)

[0026] p t→t-1 ,p t→t+1 =ξ(F Transformer ) (4)

[0027] Among them, Θ represents the Transformer unified feature extractor, I t-1,t,t+1 represents three temporally consecutive input image frames, F Transformer represents the unified features extracted by Transformer, f t→t-1 ,f t→t+1 Denotes the two sets of optical flows estimated by the optical flow decoder branch network θ, D t represents a depth map estimated by the deep decoder branch network ψ, p t→t-1 ,p t→t+1Then are the two sets of poses estimated by the pose decoder branch network ξ;

[0028] (2) Iterative optimization method of optical flow, depth and pose based on mutual attraction mechanism;

[0029] Most of the existing joint estimation methods of optical flow, depth and pose perform unidirectional optimization of pose and depth estimation tasks by introducing optical flow as additional information of the entire network, but ignore the geometric consistency between optical flow, depth and pose, and cannot use pose and depth information to improve the accuracy of optical flow estimation. By introducing a mutual introduction mechanism to introduce geometric priors into the network, bidirectional iterative optimization of optical flow, pose and depth is achieved. Specifically, in order to make the estimation accuracy of optical flow, depth and pose promote each other, the optical flow branch is used as a guide for the depth and pose branches, so that it guides the depth and pose generation from the perspective of optical flow. At the same time, the depth and pose branches also serve as guides for the optical flow branch, so that it guides the generation of optical flow from the perspective of depth and pose estimation. This mutual introduction mechanism makes the optical flow, depth and pose estimation promote each other.

[0030] As a further improvement of the present invention, the specific training process of step (1) is as follows:

[0031] The three frames of continuous time images captured by the monocular camera are spliced ​​according to the channel dimension and input into the Transformer unified feature extractor. The Transformer encoder first divides the three images into blocks, and then performs feature mapping and position encoding to maintain the position information between each image block in parallel computing processing, and obtains a feature representation with a global receptive field at one time. This feature representation has contextual structure information, which is conducive to improving the accuracy of optical flow, depth and pose estimation. After that, the feature is input into the optical flow, depth and pose decoder branch network to estimate two sets of optical flow, one set of depth and two sets of pose respectively. Finally, the estimated optical flow, depth and pose are iteratively optimized based on the mutual introduction mechanism.

[0032] First, freeze the optical flow decoder branch network parameters and input the three images I t-1 and I t+1 The estimated depth and pose information are mapped to I based on image warp technology. t ,get and Then the I t and After inputting to Transformer, optical flow estimation is performed, and the optical flow decoder branch network is used to guide the pose and depth estimation. At this time, the two sets of optical flows estimated by the network should be equal. Based on this equal relationship, additional prior information can be provided for depth and pose estimation at the optical flow angle. Secondly, the depth and pose estimation information is frozen, and the I in the three input images is converted to t-1 and I t+1 The estimated optical flow information is mapped to I based on the image warp technology. t ,get and Then the I t and After the input is fed into the Transformer, the pose is estimated. At this time, the two poses estimated by the network should be equal. Based on this, additional prior information can be provided for the optical flow estimation at the pose estimation angle. Based on the mutual reference mechanism, the estimation accuracy of the optical flow, depth and pose decoder branch network can be iteratively optimized. The equation is expressed as follows:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041] in, They are two sets of optical flows and two sets of poses estimated from two sets of mapping images obtained using warp technology.

[0042] As a further improvement of the present invention, the specific operation process of step (2) is as follows:

[0043] After the entire network is trained, all parameters of the network are frozen, and the three consecutive test images captured by the monocular camera are processed according to I t-1 ,I t ,I t+1The order of the channels is spliced ​​together and directly input into the Transformer feature encoder, and then the extracted unified features with global receptive field are simultaneously input into the optical flow, depth and pose decoder branch network to complete the joint estimation of optical flow, depth and pose. At the same time, the three branches can also make predictions independently without interfering with each other.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) This paper innovatively introduces the Transformer as the sole feature encoder in the optical flow, depth, and pose joint estimation network, reducing overall network parameters and structural redundancy. At the same time, it provides a unified feature description with global receptive field and global consistency for the three different tasks.

[0046] (2) The present invention introduces a mutual guidance mechanism in the joint estimation, so that optical flow, depth and pose can serve as each other's guides, achieving mutual promotion and improvement of accuracy and mutual optimization learning of network parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the overall framework of the Transformer-based monocular gaze flow, depth and pose joint estimation network proposed in this paper;

[0048] Figure 2 This is the specific implementation process of the mutual introduction mechanism. DETAILED DESCRIPTION

[0049] The present invention is directed to a joint estimation method for optical flow, depth, and pose based on a monocular camera and deep learning technology. It uses an unsupervised learning method to solve the problem that ground truth (GT) cannot be obtained on a large scale, so that the network can be trained end-to-end in a self-supervised manner. Specifically, considering that in the joint estimation task of optical flow, depth, and pose, different tasks have similar geometric consistency in three-dimensional structure. Therefore, the present invention introduces a Transformer encoder to perform unified feature encoding on the input image, and uses the global receptive field of the Transformer to extract global consistency information in the input image, providing a unified and consistent feature description for subsequent optical flow, depth, and pose estimation. At the same time, only one Transformer is used as a unified encoder for the three tasks, reducing network parameters and structural redundancy. In addition, most of the existing joint estimation methods for optical flow, depth, and pose only optimize the estimation of pose by introducing optical flow, while ignoring the task similarity between optical flow, depth, and pose, and failing to use depth and pose information to improve the accuracy of optical flow estimation. Therefore, the present invention introduces a mutual induction mechanism during network training to introduce geometric priors into the network, enabling bidirectional iterative optimization of optical flow, pose, and depth. This method can be used for joint estimation of optical flow, depth, and pose for monocular vision using drones, unmanned vehicles, or robots.

[0050] The overall framework of the Transformer-based monocular eye flow, depth and pose joint estimation network proposed in this paper is as follows: Figure 1 As shown. First, the time-series image frames captured by the three monocular cameras are spliced ​​in the channel dimension, and then the spliced ​​images are divided into blocks so that the Transformer can serialize the image data. Subsequently, the block results are flattened, linearly mapped, and positionally embedded so that each block is given position information in the parallel processing of the Transformer. Then, they are input into the Transformer unified feature encoder to uniformly encode and extract the features required for optical flow, depth, and pose estimation. Finally, the extracted unified features are input into the optical flow, depth, and pose decoder branch networks for the corresponding estimation process. After training, the joint estimation network framework can also independently complete the prediction of optical flow, depth, and pose, and the tasks do not interfere with each other.

[0051] The improvements of the present invention will be described in detail below.

[0052] In terms of network framework. Unlike existing optical flow, depth and pose joint estimation networks that mostly use differentiated feature extractors / encoders that need to be adapted for specific tasks, the present invention strives to provide a unified feature description for three different tasks through a single feature extraction and encoding, thereby reducing the redundancy of network parameters and structures. Considering that optical flow, depth and pose estimation all require processing of time-series image sequences, and there are similarities in spatial structure between the three tasks. Therefore, the present invention selects Transformer as the feature encoder for the three tasks, providing a unified and unique feature encoding for the joint estimation of optical flow, depth and pose. As a new network structure built entirely on the self-attention mechanism, Transformer first emerged in the field of natural language processing (NLP). With its advantages such as high precision and parallel processing, it has shown a trend of replacing recurrent neural networks (RNN). In the field of computer vision, ViT (Vision Transformer) is a pioneering work that applies Transformer to image processing. Compared with traditional convolutional neural networks, ViT has a global receptive field and a natural advantage in processing sequence information. Therefore, the present invention proposes a Transformer unified feature encoder based on ViT. The difference from ViT originally used for image classification tasks is that the input of the Transformer unified feature encoder of the present invention is [B,9,W,H]. Where B represents the batch size of the network, and 9 represents a 9-dimensional image group obtained by splicing three RGB images according to the channel dimension. It is worth noting that in the splicing process, the three time-series images I t-1 ,I t ,I t+1 Splice in sequence. t As the target frame, I t-1 ,I t+1 It also serves as the reference frame for subsequent estimation tasks. W and H represent the width and height of the image, respectively. The Transformer unified feature encoder outputs a 256-dimensional global feature. Finally, the Transformer-encoded unified feature is input into the optical flow, depth, and pose decoder branches to complete the joint estimation.

[0053] The pose decoder branch network consists of four convolutional layers and three activation layers. The first three convolutional layers decode the unified features encoded by the Transformer, while the three activation layers perform nonlinear mapping. The final convolutional layer performs pose regression prediction, ultimately outputting two 6-dimensional pose information representing the pose between the target frame and the two reference frames. Three of these 6 dimensions represent rotation, and three represent translation. The depth decoding branch and the optical flow decoding branch have a similar structure to a U-Net, differing in that the convolutional and deconvolutional layers are asymmetric. Because the Transformer unified feature encoder provides unified feature extraction for the depth and optical flow decoder branches, the number of convolutional layers is smaller than that of deconvolutional layers. The convolutional layers perform dimensionality adjustment, while the deconvolutional layers perform multi-scale dense depth / optical flow prediction. The depth decoder branch ultimately outputs a depth map of the target frame with the output dimensions [B, 1, W, H], where B is the batch size and W and H represent the width and height of the input image, respectively. The output of the optical flow decoder branch is [B, 4, W, H], where the channel dimension is 4, representing two 2D optical flows. These represent the 2D optical flow information between the target frame and the two reference frames. The entire network forward process is expressed as follows.

[0054] F Transformer =Θ(I t-1,t,t+1 ) (1)

[0055] f t→t-1 ,f t→t+1 =θ(F Transformer ) (2)

[0056] D t =ψ(F Transformer ) (3)

[0057] p t→t-1 ,p t→t+1 =ξ(F Transformer ) (4)

[0058] Among them, Θ represents the Transformer unified feature extractor, I t-1,t,t+1 represents three temporally consecutive input image frames (which have been segmented, flattened, linearly mapped, and positionally embedded), F Transformer represents the unified features extracted by Transformer, f t→t-1 ,f t→t+1 Denotes the two sets of optical flows estimated by the optical flow decoder branch network θ. t represents a depth map estimated by the deep decoder branch network ψ. t→t-1 ,p t→t+1 These are the two sets of poses estimated by the pose decoder branch network ξ.

[0059] In terms of unsupervised loss functions. The present invention innovatively introduces a mutual introduction mechanism to provide additional priors for the joint estimation network. Most existing optical flow, depth and pose joint estimation networks mostly introduce optical flow as additional matching information as a priori for pose estimation, that is, they improve the accuracy of pose estimation by introducing optical flow. However, these methods do not take into account the essential structural similarity between optical flow and depth and pose estimation tasks, and only use the one-way optimization of optical flow to pose estimation, while ignoring the improvement effect of pose estimation on optical flow. The present invention proposes a mutual introduction mechanism to achieve mutual promotion between optical flow, depth and pose, thereby enabling iterative optimization among the three. Specifically, during the training process of the network, the optical flow decoding branch is used as a guide for the depth and pose decoding branches, so that it guides the generation of depth and pose from the perspective of optical flow estimation. Similarly, the pose decoding branch can also be used as a guide for the optical flow decoding branch, so that it guides the generation of optical flow from the perspective of pose estimation.

[0060] The specific implementation process of the mutual introduction mechanism is as follows Figure 2 First, freeze the optical flow decoder branch network parameters and input the three images of I t-1 and I t+1 The estimated depth and pose information are mapped to I based on image warp technology. t ,get and Then the I t and After inputting to the Transformer, optical flow estimation is performed. The optical flow decoder branch network is used to guide the pose and depth estimation. At this time, the two sets of optical flows estimated by the network should be equal. Based on this equal relationship, additional prior information can be provided for depth and pose estimation at the optical flow angle. Secondly, the depth and pose estimation information is frozen. t-1 and I t+1 The estimated optical flow information is mapped to I based on the image warp technology. t ,get and Then the I t and After inputting to the Transformer, pose estimation is performed. At this point, the two poses estimated by the network should be equal, providing additional prior information for optical flow estimation at the pose estimation angle. The mutual induction mechanism enables iterative optimization of the estimation accuracy of the optical flow, depth, and pose decoder branch networks. The equation is as follows:

[0061]

[0062]

[0063]

[0064]

[0065] Among them L f-guidance is the mutual loss function estimated by the optical flow branch for pose and depth network. EPE stands for endpoint error, which is the mean of the Euclidean distance between two optical flow vectors over all pixels. are two sets of optical flows estimated from two sets of maps obtained using the warp technology.

[0066]

[0067]

[0068]

[0069]

[0070] Among them, L p-guidance represents the mutual loss function guided by the pose branch for optical flow estimation. ||·|| represents the L2 loss function, These are two sets of poses estimated from the two sets of mapping images obtained using the warp technology.

[0071] To enable unsupervised network training, an unsupervised loss function is required. This is primarily divided into two parts: the optical flow loss function and the pose and depth loss functions. The optical flow loss function used in this paper is as follows.

[0072] First, the optical flow consistency loss function L f-pho The present invention adopts the Census photometric consistency loss function. The basic principle is to use the image warping technology to warp the estimated optical flow to the target image onto the reference image to obtain If the optical flow estimation is accurate, Should be consistent with reference frame I ref The introduction of the Census function increases the robustness of the loss function under varying lighting conditions. The details are as follows.

[0073]

[0074] in, Cen(·) represents the Census function. δ(x)=(x 2 +0.01 2 ) 0.4is the penalty function. t→ref Represents the optical flow from the target frame to the reference frame.

[0075] In addition, the present invention also adopts the optical flow smoothing loss L f-smooth , the smoothness loss stipulates that each optical flow value should be similar to that in a local neighborhood. Its expression is as follows.

[0076]

[0077] Among them, N x It consists of 4 horizontal, vertical and two diagonal neighborhoods around x. s and r are the coordinates of the neighborhood of x.

[0078] Therefore, the loss function of the optical flow decoding branch is summarized as follows: where λ3 and λ4 are weight parameters.

[0079] L flow =λ3L f-pho +λ4L f-smooth (15)

[0080] The unsupervised loss function for pose and depth estimation adopted in the present invention is as follows.

[0081] First is the perspective synthesis loss function L p-view This loss function is similar to the photometric loss function of optical flow, both of which use image warping technology to obtain supervision information. The difference is that L p-view It is to realize the perspective synthesis of the image based on the estimated depth and pose. To calculate L p-view First, we need to establish the coordinate relationship between the reference frame and the target frame. This relationship is defined as follows.

[0082]

[0083] Among them, p ref and p t are the pixel coordinates between the reference frame and the target frame respectively. K is the intrinsic parameter matrix of the camera, The pose of the target frame to the reference frame estimated by the network. D t is the dense depth estimated by the deep network. According to the above relationship, we can reconstruct the target frame through the reference frame. It is worth noting that the image coordinates after warping are not necessarily integers, so bilinear interpolation is also needed to convert them into integers. p-view The definition is as follows.

[0084]

[0085] Among them, λ pis the weight balancing parameter. SSIM(·) is the image structural similarity, which ranges from 0 to 1. The closer it is to 1, the more similar the two images are. |·| represents the L1 norm.

[0086] Since the perspective reconstruction loss provides less information for network training in low-texture or uniform scenes, the present invention introduces a depth edge-aware loss to maintain the performance of depth estimation. The details are as follows.

[0087]

[0088] in is the average value of the estimated depth. This is done to ensure the smoothness of the edges of the depth map image.

[0089] The unsupervised loss function for pose and depth estimation used in this invention is:

[0090] L D / P =λ5L p-view +λ6L D-smooth (19)

[0091] Among them, λ5 and λ6 are weight parameters

[0092] In summary, the total loss function adopted by the present invention is as follows.

[0093] L total =λ1L f-guidance +λ2L p-guidance +L flow +L D / P (20)

[0094] Where λ1 and λ2 are weight parameters. It's worth noting that in the early stages of network training, the accuracy of optical flow, depth, and pose estimation is low. Therefore, directly adopting a loss function with a mutual induction mechanism can lead to a "blind leading the blind" phenomenon, preventing the network from converging in a timely manner. Therefore, in the early stages of network training (before epoch 40), λ1 and λ2 should be zero, and then gradually increase with each epoch.

[0095] After the network is trained, the network parameters are fixed and three consecutive monocular images from the test set are fed into the network. After the Transformer encoder uniformly extracts and encodes the features, the images are fed into the various decoder branches, simultaneously completing the estimation of monocular eye flow, depth, and pose. Furthermore, the three tasks can be independently predicted during the prediction phase without interfering with each other.

[0096] The above description is merely a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent variation based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A Transformer-based unsupervised joint estimation method for monocular gaze flow, depth, and pose, with the following specific steps, is characterized by: (1) Training phase: The data in the training phase consists of three consecutive image frames captured by a monocular camera. After being spliced ​​in the channel dimension, they are directly input into the Transformer encoder to encode the features, providing a unified feature representation for the subsequent joint estimation of optical flow, depth, and pose. The features are then decoded by the optical flow, depth, and pose decoders respectively, and the joint estimation of the corresponding information is completed. In this process, an improved unsupervised loss function is adopted, and the spatial structural relationship between optical flow, depth, and pose is exploited to enable the network to achieve unsupervised learning. The network parameters are continuously iterated using gradient backpropagation until the loss tends to be optimal and the training is completed. The specific meaning of the unsupervised optical flow, depth and pose joint estimation network in step (1) is as follows: It includes the following two aspects: (1) Transformer-based unified feature extractor, Using a single Transformer as a unified encoder for the three tasks, a single, unified feature extraction process can meet the different feature requirements of the three tasks. At the same time, the optical flow, depth, and pose estimation tasks have strong geometric consistency in the three-dimensional structure. The network consists of a Transformer encoder and optical flow, depth, and pose decoder branches. The relationship between the various network structures is as follows: （1） （2） （3） （4） in, stands for Transformer unified feature extractor, represents three temporally consecutive input image frames, Represents the unified features extracted by Transformer, Represents the optical flow decoder branch network The two sets of estimated optical flows, Represents a network consisting of deep decoder branches An estimated depth map, is the pose decoder branch network Two sets of estimated poses; (2) Iterative optimization method of optical flow, depth and pose based on mutual attraction mechanism; By introducing a mutual introduction mechanism to introduce geometric priors into the network, bidirectional iterative optimization of optical flow, pose, and depth is achieved. Specifically, in order to promote the estimation accuracy of optical flow, depth, and pose, the optical flow branch is used as a guide for the depth and pose branches, so that it guides the generation of depth and pose from the perspective of optical flow. At the same time, the depth and pose branches also serve as guides for the optical flow branch, so that it guides the generation of optical flow from the perspective of depth and pose estimation. This mutual introduction mechanism enables the mutual promotion of optical flow, depth, and pose estimation. (2) Prediction stage: After the network is trained, the network parameters are fixed and three consecutive monocular images from the test set are input. After the Transformer encoder uniformly extracts and encodes the features, they are input to each decoder branch. This allows the estimation of monocular eye flow, depth, and pose to be completed simultaneously. At the same time, the three tasks can also be predicted independently in the prediction stage without interfering with each other.

2. The Transformer-based unsupervised joint estimation method for monocular eye flow, depth, and pose according to claim 1, characterized in that: The specific training process of step (1) is as follows: The three time-continuous images captured by the monocular camera are spliced ​​according to the channel dimension and input into the Transformer unified feature extractor. The Transformer encoder first divides the three images into blocks, and then performs feature mapping and position encoding to maintain the position information between each image block in parallel computing processing, and obtains a feature representation with a global receptive field at one time. This feature representation has contextual structure information. After that, the feature is input into the optical flow, depth and pose decoder branch network to estimate two sets of optical flow, one set of depth and two sets of pose respectively. Finally, the estimated optical flow, depth and pose are iteratively optimized based on the mutual introduction mechanism. First, freeze the optical flow decoder branch network parameters and convert the three input images into and The estimated depth and pose information are mapped to the image warp technology based on ,get and , and then , and After inputting to Transformer, optical flow estimation is performed, and the optical flow decoder branch network is used to guide the pose and depth estimation. At this time, the two sets of optical flows estimated by the network should be equal. Based on this equal relationship, additional prior information can be provided for depth and pose estimation at the optical flow angle. Secondly, the depth and pose estimation information is frozen, and the three input images are and The estimated optical flow information is mapped to ,get and , and then , and After the input is fed into the Transformer, the pose is estimated. At this time, the two poses estimated by the network should be equal. Based on this, additional prior information can be provided for the optical flow estimation at the pose estimation angle. Based on the mutual reference mechanism, the estimation accuracy of the optical flow, depth and pose decoder branch network can be iteratively optimized. The equation is expressed as follows: （5） （6） （7） （8） （9） （10） （11） （12） in, They are two sets of optical flows and two sets of poses estimated from two sets of mapping images obtained using warp technology.

3. The Transformer-based unsupervised joint estimation method for monocular eye flow, depth, and pose according to claim 1, characterized in that: The specific operation process of step (2) is as follows: After the entire network is trained, all parameters of the network are frozen, and three consecutive test images captured by the monocular camera are processed according to the The order of the channels is spliced ​​together and directly input into the Transformer feature encoder, and then the extracted unified features with global receptive field are simultaneously input into the optical flow, depth and pose decoder branch network to complete the joint estimation of optical flow, depth and pose. At the same time, the three branches can also make predictions independently without interfering with each other.

Images