A monocular image depth estimation system based on spatiotemporal attention

Abstract

The present invention provides a monocular image depth estimation system based on spatiotemporal attention. It includes: a spatiotemporal learning module receives an input monocular image sequence and learns the spatiotemporal features in the monocular image sequence; a spatiotemporal enhancement module performs a pooling operation on the spatiotemporal feature sequence to obtain a fused feature, performs a self-attention operation on the fused feature, and obtains a spatially enhanced spatiotemporal feature; a convolution operation is used to compress the spatiotemporal features of adjacent views respectively, and an iterative selection method is used to perform a mutual attention operation between different feature maps to obtain a temporal correlation feature. The spatiotemporal decoding module upsamples the spatially enhanced spatiotemporal features and the temporal correlation features, and outputs a depth map of the same size as the image depth. The system of the present invention not only pays attention to and learns the spatial structure information and temporal correlation information in the monocular video, but also enhances the network's mapping and expression capabilities for spatiotemporal features, and reconstructs a monocular depth map that is consistent with the spatiotemporal features of the input image sequence.

Description

Technical Field

[0001] The present invention relates to the technical field of monocular image depth estimation, and in particular to a monocular image depth estimation system based on spatiotemporal attention. Background Art

[0002] The grayscale value of each pixel in the depth image, i.e., the depth, can be used to characterize the distance of points in the scene from the sensor. Monocular image depth estimation refers to the process of estimating the scene depth map from a single or multiple color images taken by a monocular camera. Monocular image depth estimation only requires an ordinary camera, which has the advantages of low cost and easy operation, and the data information provided in real life is mostly single visual information; in addition, the monocular image presents less scene information, and the monocular image depth estimation itself is ill-posed, which brings greater challenges to the monocular image depth estimation task. Therefore, monocular image depth estimation has become a research hotspot and difficulty in the field of depth estimation.

[0003] Monocular image depth estimation not only has important research value, but also has great application value in practical scenarios: in the field of intelligent driving, accurate depth estimation can regulate the motion trajectory of unmanned vehicles and model the scene in which they are located with high precision; in the field of intelligent medical care, depth information helps to locate the damaged area and reduce misjudgment caused by organ deformation or displacement; in the field of intelligent robots, the depth information of the target plays a vital role in the intelligent robot's avoidance of obstacles, safe planning of travel trajectories, intelligent path navigation, grasping target objects and destination guidance. In the field of virtual reality, depth estimation, as the basic technology of scene three-dimensional modeling, is conducive to the accurate construction of three-dimensional simulation environment. In addition, monocular image depth estimation is also widely used in three-dimensional video production, video surveillance, liveness detection, human-computer interaction and other fields.

[0004] The scene spatial structure information in the monocular depth map is not only related to the depth information in the current view, but also the temporal correlation information between adjacent views will affect the recovery of the scene spatial structure. Therefore, in order to reconstruct a high-precision monocular depth map, in addition to paying attention to the spatial information of the surface features of the object in the current view, it is also necessary to pay attention to the temporal correlation relationship of the scene features between adjacent views. Ignoring the temporal information of adjacent views in the monocular video can easily lead to incorrect matching of corresponding point features, resulting in an offset in the depth value mapped to the three-dimensional space, inaccurate object spatial structure in the restored depth map, and drift problems. In practical applications, it will affect the judgment of obstacle size by robots, self-driving cars and other equipment.

[0005] At present, the existing monocular image depth estimation methods mainly rely on depth clues to estimate the image depth value, such as recovering depth from focus / defocus, recovering depth from shadows, recovering depth from motion, etc. The methods that rely on depth clues have high requirements on the image, requiring that the image must have specific situations such as focus / defocus, occlusion, shadows, motion, etc. The algorithm complexity is high, and the practicality is poor, and it cannot meet the real-time requirements. Later, researchers calculated the image depth value by designing manual features, but the method based on manual features has a strong dependence on the scene texture and cannot perform depth estimation in weak / no texture areas. With the development of deep neural networks, methods based on deep learning have become a common method in the field of monocular depth. Depending on whether ground truth images are required during network training, the monocular image depth estimation methods based on deep learning can be divided into supervised learning methods, unsupervised learning methods, and semi-supervised learning methods.

[0006] Supervised learning method: supervised learning method refers to the method in which the ground truth depth map needs to be input into the network during the monocular image depth estimation process, and the deep neural network estimates the scene depth by learning the ground truth image. At this time, the ground truth image is used as a supervisory signal to supervise the learning process of the deep neural network. Supervised learning methods include supervised regression methods and supervised classification methods, which respectively represent the use of supervised continuous regression methods and supervised ordered regression methods to predict monocular depth maps.

[0007] Supervised learning methods require a large number of ground truth depth maps as supervisory signals. The estimated depth maps are highly accurate and can effectively map the 3D structure of the scene. However, such methods are highly dependent on ground truth depth maps, which are difficult to obtain. Currently, there are limited high-quality public datasets with ground truth labels.

[0008] Unsupervised learning methods: Unsupervised learning methods do not require input of ground truth depth maps as supervision signals. Deep neural networks perform monocular image depth estimation by learning geometric relationships in color images. Monocular image depth estimation methods based on unsupervised learning are usually trained using left and right stereo image sequences or monocular videos, constrained by scene geometry, and tested on monocular images or sequences to estimate scene depth maps.

[0009] The unsupervised learning method only cares about the geometric relationship between the input color images and does not require the ground truth depth map, which reduces the cost of constructing depth labels; but due to the lack of supervision of the ground truth depth, the depth estimation accuracy based on the unsupervised learning method is slightly inferior to the monocular image depth estimation method based on supervised learning.

[0010] Semi-supervised learning method: Semi-supervised learning method refers to the use of information other than ground truth depth images, such as synthetic data, sparse depth, surface normals, etc., to assist in the learning of depth estimation. This type of method does not require depth labels, nor does it simply use monocular image features. There is auxiliary information to help in the learning process, and the network input includes color images and auxiliary information. Commonly used auxiliary information includes virtual images based on computer synthesis and sparse depth maps based on lasers and radars.

[0011] In addition to learning depth features from color images, semi-supervised learning methods also combine auxiliary information to perform monocular image depth estimation, which makes the depth map more accurate. Although auxiliary information is easier to obtain than ground truth depth maps, it still increases the amount of input data for the network model, and the accuracy of depth estimation is highly dependent on auxiliary information. If the difference between the auxiliary information and the real image is large, the error rate of depth estimation will be greatly increased, increasing the difficulty of scene perception.

[0012] A prior art monocular image depth estimation method based on convolutional neural network uses convolutional neural network (CNN) to build a network model. CNN has excellent performance in depth estimation tasks and can better learn and map the spatial structure characteristics of the scene.

[0013] The CNN network is used to build a depth estimation model, and the learned spatial features are decoded through two sub-networks to obtain the initial depth and final depth of the scene image, respectively, and the monocular depth map is reconstructed from coarse to fine. The scene spatial features are captured using a residual structure based on CNN, and a decoding network based on fast upsampling is proposed. However, the convolution kernel of the network is small, the network receptive field is limited, and only simple bilinear interpolation is used to improve the resolution of the depth map during feature decoding, resulting in the network losing a lot of depth features. In order to reduce the loss of features, a jump connection is added between the decoding network layer and the corresponding encoding network layer, and the coarse depth map in the decoding network is fused with the fine spatial feature map in the encoding network, which enhances the mapping and expression of the depth features in the decoding process and improves the accuracy of depth estimation. In addition to using jump connections to enhance feature decoding, a global-local combination strategy is also adopted to construct two sub-networks based on full convolution to learn the local spatial features and global spatial features of the input image respectively, and 10 convolutional layers are used to map the depth gradient changes of the input image, and deconvolution layers are used to decode the global structure of the scene. In addition, in order to speed up deep decoding, a CNN-based fast deep decoder is proposed, which consists of five cascaded upsampling layers and a single point-by-point layer. Nearest neighbor interpolation is performed after each upsampling layer to double the spatial resolution of the intermediate feature map, and deep decomposition is used to further reduce the complexity of all convolutional layers.

[0014] The disadvantages of the monocular image depth estimation method based on convolutional neural network in the above-mentioned prior art include: the input and output of the convolutional neural network are one-to-one corresponding, and different inputs do not affect each other. The monocular image depth estimation method based on CNN takes into account the spatial features of the current input image, but ignores the temporal correlation between adjacent views in the monocular image sequence, reduces the correlation of corresponding point features in adjacent views, and easily causes mismatching of depth features, resulting in drift and other problems when the two-dimensional pixel points in the image are mapped to the three-dimensional space. Summary of the invention

[0015] An embodiment of the present invention provides a monocular image depth estimation system based on spatiotemporal attention, so as to effectively reconstruct a monocular depth map that is consistent with the spatiotemporal features of an input image sequence.

[0016] In order to achieve the above object, the present invention adopts the following technical scheme.

[0017] A monocular image depth estimation system based on spatiotemporal attention, comprising: a spatiotemporal learning module, a spatiotemporal enhancement module and a spatiotemporal decoding module;

[0018] The spatiotemporal learning module is used to receive an input monocular image sequence, learn the spatiotemporal features in the monocular image sequence, and transmit the spatiotemporal feature sequence to the spatiotemporal enhancement module;

[0019] The spatiotemporal enhancement module is used to perform a pooling operation on the spatiotemporal feature sequence to obtain a fused feature, perform a self-attention operation on the fused feature to obtain a spatially enhanced spatiotemporal feature; use a convolution operation to compress the spatiotemporal features of adjacent views respectively, and use an iterative selection method to perform a mutual attention operation between different feature maps to obtain a temporal correlation feature; transmit the spatially enhanced spatiotemporal features and the temporal correlation features to the spatiotemporal decoding module;

[0020] The spatiotemporal decoding module is used to upsample the spatiotemporal features and temporal correlation features after spatial enhancement through a deconvolution module, and output a depth map with the same depth size as the image.

[0021] Preferably, the spatiotemporal learning module is specifically composed of 7 ConvLSTM units. The spatiotemporal feature learning module of the ConvLSTM unit extracts the two-dimensional spatiotemporal feature map of the input monocular image sequence through convolution operation. The learning process of the ConvLSTM unit is expressed as:

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] Among them, "*" and Represent the convolution operation and Hadamard product respectively; and Represents input, unit output and hidden state respectively; “i t ","f t ” and “o t " respectively represent the feature maps of the input gate, forget gate and output gate of ConvLSTM. The feature map is a three-dimensional tensor, and the last two dimensions represent the spatial dimensions; "W x· " and "b·" represent the convolution kernel and offset term in the corresponding convolution calculation respectively;

[0028] The spatiotemporal feature learning module of the ConvLSTM unit transfers the 2D spatiotemporal feature maps of adjacent frames through feature storage, fusion, and update operations.

[0029] Preferably, the spatiotemporal enhancement module includes a spatial attention submodule and a temporal attention submodule executed in parallel;

[0030] The spatial attention submodule is used to enhance the network's attention to and learning of spatiotemporal features, perform maximum pooling and average pooling operations on the spatiotemporal features to obtain two pooling features, fuse different pooling features, perform self-attention operations on the fused features, obtain spatially enhanced spatiotemporal features, and transmit them to the spatiotemporal decoding module;

[0031] The temporal attention submodule is used to compress the spatiotemporal features of adjacent views by convolution operation, obtain two feature maps, perform mutual attention operation between the two feature maps by iterative selection, calculate the temporal correlation features enhanced by the corresponding feature mutual attention mechanism, and transmit them to the spatiotemporal decoding module.

[0032] Preferably, the spatial attention submodule is specifically used to adopt a self-attention mechanism based on fusion features, with H×W representing the feature resolution size, C and C1 representing the number of feature channels and the number of channels after dimensionality reduction, respectively, and MaxPooling and AvgPooling representing the maximum pooling and average pooling operations, respectively. Two different pooling operations are used: maximum pooling and average pooling to compress the spatiotemporal features and obtain two feature maps Φ with dimensions of H×W×C1. 1 and Φ 2 , fusion feature maps by adding corresponding channels and Get the feature map Φ of dimension H×W×C1 3 ; For fusion features Perform self-attention operation, use the deformed fusion features as key features and query features respectively, and use softmax function to normalize the key features and query features to obtain the spatial attention weight map of the current spatiotemporal features. The self-attention weight map The size is HW×HW, Each element in It is expressed as:

[0033]

[0034] in, represents the influence of the i-th pixel on the j-th pixel; φ 3i and Φ 3j Represents the fusion feature Φ 3 Elements; N = H × W, represents the total number of pixels in the input image;

[0035] The fusion features After deformation again, it is used as the value feature of self-attention, and the value feature is combined with the spatial attention weight map Perform weighted calculations to highlight the selected spatial features and combine the selected spatial features with the spatiotemporal features. Perform the corresponding point addition operation to obtain the spatial enhanced feature Φ after being enhanced by the spatial attention submodule S , Φ S Each element in It is expressed as:

[0036]

[0037] Among them, Φ S The size is H×W×C, φ j Represents the elements of the spatiotemporal feature Φ, Represents the feature map Φ 3 The elements of , α represents the scale factor, and the output feature Φ S It is the original feature Φ and the enhanced feature at each position The weighted sum of .

[0038] Preferably, the temporal attention submodule is specifically used to represent a convolution operation with a convolution kernel of 1×1 by Conv(1×1), and adopt a mutual attention mechanism to select and enhance temporal correlation information between adjacent views;

[0039] The convolution operation with a convolution kernel of 1×1 is used to respectively transform the spatiotemporal features of adjacent views. and The number of channels is compressed to obtain two feature maps Φ′ with dimensions of H×W×C2 t-1 and Φ′ t , using iterative selection in the feature map Φ′ t-1 Perform mutual attention operation between Φ′t and Φ′ t-1 and Φ′ t Divided into two groups of key features and query features, Φ′ t-1 is the key feature, Φ′ t is the query feature; Φ′ t is the key feature, Φ′ t-1 To query features, the Softmax function is used to normalize the features after each set of iterative selection, and the time attention weight maps of two adjacent frames are obtained respectively. and in, represents the time attention weight of the feature at time t to the feature at time t-1, represents the time attention weight of the feature at time t-1 to the feature at time t. The sizes of the two time attention weight maps are both HW×HW, where HW represents the size of the feature map resolution. Elements in It is expressed as:

[0040]

[0041] Elements in It is expressed as:

[0042]

[0043] in, Represents the current view Φ′ t For the previous view Φ′ t-1 The impact of Represents the previous view Φ′ t-1 For the current view Φ′ t The impact of and Represents the feature map Φ′ t-1 Elements of and Represents the feature map Φ′ t Elements; N = H × W, represents the total number of pixels of the input image;

[0044] The temporal features at the same moment are weighted with the temporal attention weights, and then the weighted temporal features are added to the temporal and spatial features at the moment to obtain the temporal enhanced features enhanced by the temporal attention submodule. The temporal enhanced features at time t-1 are Φ″ t→t-1Elements in It is expressed as:

[0045]

[0046] Time series enhancement feature Φ″ at time t t-1→t Elements in It is expressed as:

[0047]

[0048] Among them, the timing enhancement feature Φ″ t→t-1 and Φ″ t-1→t The size of is H×W×C; γ represents the scale coefficient, and Represents the spatiotemporal feature Φ t-1 Elements of and Represents the spatiotemporal feature Φ t elements.

[0049] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the monocular image depth estimation algorithm based on spatiotemporal attention proposed in the present invention not only enhances the attention and learning of the monocular image depth estimation network to the spatial structure information and temporal correlation information in the monocular video, but also enhances the network's mapping and expression capabilities of spatiotemporal features, and reconstructs a monocular depth map that is consistent with the spatiotemporal features of the input image sequence.

[0050] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0052] Figure 1 A structure diagram of a monocular image depth estimation network (ST-Depth) based on spatiotemporal attention provided by an embodiment of the present invention;

[0053] Figure 2 A structural diagram of a spatial attention submodule provided by an embodiment of the present invention;

[0054] Figure 3 A structural diagram of a temporal attention submodule provided by an embodiment of the present invention;

[0055] Figure 4A diagram of an iterative selection method between adjacent frames in a temporal attention submodule provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0057] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0059] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0060] Monocular image depth estimation refers to the process of inputting a single RGB (red, green, blue) image, calculating the distance from the corresponding object point of each pixel in the image to the camera through a neural network, and outputting a depth map containing depth information.

[0061] The problem of scene spatial structure drift caused by the lack of spatiotemporal features. In the depth map restored based on monocular video, the spatial structure information of the scene is not only related to the depth information in the current view, but also the temporal correlation information between adjacent views will affect the recovery of the scene spatial structure. Therefore, in order to reconstruct a high-precision monocular depth map, in addition to paying attention to the spatial information of the surface features of the object in the current view, it is also necessary to pay attention to the temporal correlation relationship of the scene features between adjacent views.

[0062] In order to fully learn the spatial structure information and temporal correlation information in monocular videos, enhance the mapping and expression of spatiotemporal features, and solve the problem of scene spatial structure drift, the present invention adopts a dual attention mechanism strategy and proposes a monocular image depth calculation method ST-Depth based on spatiotemporal attention, which simultaneously pays attention to and enhances the spatiotemporal features in monocular videos and reduces the incorrect mapping and drift of spatiotemporal features.

[0063] Figure 1 A structure diagram of a monocular image depth estimation network (ST-Depth) based on spatiotemporal attention provided in an embodiment of the present invention. Among them, "ConvLSTM" represents a convolutional long short-term memory unit, "Concate" represents feature fusion, and "Deconvolution" represents a deconvolution layer. The ST-Depth network uses supervised learning to predict monocular depth maps. The network mainly includes three modules: spatiotemporal learning module, spatiotemporal enhancement module and spatiotemporal decoding module.

[0064] First, ST-Depth uses a spatiotemporal learning module based on full ConvLSTM to simultaneously focus on and learn the spatiotemporal features in a monocular image sequence. Then, for the spatiotemporal features learned by the spatiotemporal learning module, a spatial attention submodule based on fused feature self-attention is used to select and enhance the mapping and expression of spatial structural information in the current view, thereby enhancing the global consistency of local features. At the same time, a temporal attention submodule based on corresponding feature mutual attention is used to select and enhance the mapping and expression of temporal correlation information between adjacent views, thereby enhancing the long-term correlation of corresponding features. Finally, the enhanced spatiotemporal features are fused and further decoded through the spatiotemporal decoding module, and the temporal correlation information is used to compensate and optimize the spatial structural information, thereby reducing the incorrect mapping and drift of corresponding feature points and reconstructing a monocular depth map with accurate spatial structure.

[0065] The spatiotemporal learning module consists of 7 ConvLSTM units, which are used to receive the input monocular image sequence, learn the spatiotemporal feature sequence in the monocular image sequence, and transmit the spatiotemporal feature sequence to the spatiotemporal enhancement module; compared with the original LSTM unit, the ConvLSTM in the spatiotemporal learning module changes the Hadamard product multiplied by the weight in the LSTM unit into a convolution operation, which is used to extract the two-dimensional spatiotemporal feature map of the input image, which is conducive to the preservation and transmission of the current view space feature, and fully considers the temporal correlation in the monocular image sequence. The learning process of the ConvLSTM unit can be expressed as:

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] Among them, "*" and Represent the convolution operation and Hadamard product respectively; and Represents input, unit output and hidden state respectively; “i t ","f t ” and “o t " respectively represent the feature maps of the input gate, forget gate and output gate of ConvLSTM. The feature map is a three-dimensional tensor, and the last two dimensions represent the spatial dimensions; "W x· ” and “b·” represent the convolution kernel and offset term in the corresponding convolution calculation, respectively.

[0072] Therefore, the spatiotemporal feature learning module based on ConvLSTM units can transfer feature maps of adjacent frames through feature storage, fusion and update operations, fully considering the spatial information of the current view and the temporal information of adjacent views, which is conducive to enhancing the learning and transmission of spatiotemporal features in monocular videos by the monocular depth estimation network.

[0073] The spatiotemporal enhancement module is used to perform pooling operations on the spatiotemporal feature sequences to obtain fused features, perform self-attention operations on the fused features to obtain spatially enhanced spatiotemporal features; use convolution operations to compress the spatiotemporal features of adjacent views respectively, and use iterative selection to perform mutual attention operations between different feature maps to obtain temporal correlation features; transmit the spatially enhanced spatiotemporal features and temporal correlation features to the spatiotemporal decoding module.

[0074] The spatiotemporal enhancement module includes a spatial attention submodule and a temporal attention submodule, which are executed in parallel.

[0075] The above-mentioned spatial attention submodule is used to enhance the network's attention and learning of spatiotemporal features. It performs maximum pooling and average pooling operations on the spatiotemporal features to obtain two pooling features, fuses different pooling features, performs self-attention operations on the fused features, obtains the spatially enhanced spatiotemporal features, and transmits them to the spatiotemporal decoding module.

[0076] The above-mentioned temporal attention submodule is used to compress the spatiotemporal features of adjacent views by convolution operation to obtain two feature maps. Then, the mutual attention operation is performed between the two feature maps by iterative selection, and the temporal correlation features enhanced by the corresponding feature mutual attention mechanism are calculated and transmitted to the spatiotemporal decoding module.

[0077] The spatiotemporal decoding module consists of 7 deconvolution blocks to decode the spatiotemporal features and increase the resolution of the depth map to keep it consistent with the resolution of the input image.

[0078] The input information of the spatiotemporal decoding module is the enhanced spatiotemporal features. The enhanced spatiotemporal features are upsampled through the deconvolution module to improve the resolution of the feature map, and finally output a depth map with the same depth as the image. Each spatiotemporal feature corresponds to a depth map, and the role of decoding is to improve the resolution of the depth map.

[0079] Next, the present invention will introduce in detail the spatial attention submodule and the temporal attention submodule in the spatiotemporal enhancement module.

[0080] Figure 2 A structural diagram of a spatial attention submodule provided in an embodiment of the present invention, where "H×W" represents the feature resolution size, "C" and "C1" represent the number of feature channels and the number of channels after dimensionality reduction, respectively; "MaxPooling" and "AvgPooling" represent the maximum pooling and average pooling operations, respectively. In order to strengthen the depth estimation model's attention to the image's spatial features and enhance the mapping and expression of spatial structural information, the present invention adopts a self-attention mechanism based on fusion features, and designs and constructs a spatial attention submodule, such as Figure 2 As shown, spatial features are adaptively selected and enhanced to recover rich spatial structure information.

[0081] For spatiotemporal features The present invention first adopts two different pooling operations: maximum pooling and average pooling, to compress the spatiotemporal features and obtain two feature maps Φ with dimensions of H×W×C1 respectively. 1 and Φ 2; This is because different pooling operations focus on different key spatial features. Among them, the maximum pooling operation focuses more on local spatial features, and the average pooling operation focuses more on global spatial features. Then, the feature maps are fused by adding the corresponding channels. and Get the feature map Φ of dimension H×W×C1 3 ; Fusion of different pooling features is conducive to extracting richer spatial structure features and reducing the interference of redundant features. Finally, the fusion feature Perform self-attention operation, use the fused features after deformation (Shape) as key features (Key) and query features (Query), and use softmax function to normalize them to obtain the spatial attention weight map of the current spatiotemporal features The self-attention weight map The size is HW×HW. Each element in It can be expressed as:

[0082]

[0083] in, Determines the influence of the i-th pixel on the j-th pixel; φ 3i and Φ 3j Represents the fusion feature Φ 3 elements; N = H × W, representing the total number of pixels in the input image.

[0084] In order to calculate the spatial structure features enhanced by the self-attention mechanism, the present invention combines the features After deformation again, it is used as the value feature of self-attention and compared with the spatial attention weight map Perform weighted calculation to highlight the selected spatial features. Perform the corresponding point addition operation to obtain the spatial enhanced feature Φ after being enhanced by the spatial attention submodule S , Φ S Each element in It can be expressed as:

[0085]

[0086] Among them, Φ S The size of is H×W×C, φj represents the element of the spatiotemporal feature Φ, Represents the feature map Φ 3 , α represents the scale factor (initialized to 0). From formula (2-2), we can see that the output feature Φ S It is the original feature Φ and the enhanced feature at each position The weighted sum of .

[0087] Therefore, the spatial attention submodule selectively aggregates global features and enhances the correlation of local similar features, which is beneficial to the mapping and expression of spatial structure information in the spatiotemporal features Φ.

[0088] Figure 3 A temporal attention submodule structure diagram provided in an embodiment of the present invention. "H×W" represents the feature resolution size, "C" and "C2" represent the number of feature channels and the number of channels after dimensionality reduction, respectively; "Conv(1×1)" represents a convolution operation with a convolution kernel of 1×1.

[0089] For the spatiotemporal features of the current view, in addition to using the spatial attention submodule to enhance the mapping and expression of its spatial structure information, the present invention also designs and proposes a temporal attention submodule based on the mutual attention of the corresponding features, such as Figure 3 The temporal attention submodule uses the mutual attention mechanism to select and enhance the temporal correlation information between adjacent views, improving the learning and mapping ability of the depth estimation network for temporal correlation features in image sequences.

[0090] In order to enhance the expression of temporal correlation information between adjacent views, this paper introduces a mutual attention mechanism between adjacent views, which improves the accuracy of deep feature matching and mapping by focusing on the correlation of corresponding features in adjacent views. First, a convolution operation with a convolution kernel of 1×1 is used to respectively perform a mutual attention operation on the temporal and spatial features of adjacent views. and The number of channels is compressed to obtain feature maps Φ′ with two dimensions of H×W×C2 t-1 and Φ′ t .

[0091] Figure 4 FIG. 1 is an iterative selection method between adjacent frames in a temporal attention submodule provided by an embodiment of the present invention. Then, an iterative selection method is used to select the feature map Φ′. t-1 and Φ′ t Perform mutual attention operations between them, such as Figure 4 As shown, Φ′ t-1 and Φ′ t Divided into two groups of key features (Key) and query features (Query): Φ′ t-1 is the key feature, Φ′ t is the query feature; Φ′ t is the key feature, Φ′ t-1is the query feature. The more similar the features of the corresponding positions in adjacent views are, the higher the correlation between the two sets of features is; therefore, the iterative method of key features and query features is conducive to the mutual attention and compensation of corresponding features in adjacent views, and improves the correlation of corresponding features. Then, the Softmax function is used to normalize each set of iteratively selected features to obtain the temporal attention weight map of two adjacent frames of images. and in, represents the time attention weight of the feature at time t to the feature at time t-1, It represents the temporal attention weight of the feature at time t-1 to the feature at time t. The size of the two temporal attention weight maps is HW×HW, where HW represents the size of the feature map resolution. Elements in It can be expressed as

[0092]

[0093] Elements in It can be expressed as:

[0094]

[0095] in, Represents the current view Φ′ t For the previous view Φ′ t-1 The impact of Represents the previous view Φ′ t-1 For the current view Φ′ t The impact of and Represents the feature map Φ′ t-1 Elements of and Represents the feature map Φ′ t Elements of; N = H × W, represents the total number of pixels of the input image.

[0096] Finally, in order to calculate the temporal correlation features enhanced by the corresponding feature mutual attention mechanism, the present invention performs weighted calculation on the temporal features at the same time and the temporal attention weight, highlights the features with high temporal correlation in adjacent views, and then adds the corresponding points with the spatiotemporal features at that time to obtain the temporal enhanced features enhanced by the temporal attention submodule. Among them, the temporal enhanced feature Φ″ at time t-1 t→t-1 Elements in It can be expressed as:

[0097]

[0098] Time series enhancement feature Φ″ at time tt-1→t Elements in It can be expressed as

[0099]

[0100] Among them, the timing enhancement feature Φ″ t→t-1 and Φ″ t-1→t The size of is H×W×C; γ represents the scale coefficient (initialized to 0); and Represents the spatiotemporal feature Φ t-1 Elements of and Represents the spatiotemporal feature Φ t elements.

[0101] The final temporal enhancement feature is the weighted sum of the original spatiotemporal features and the temporal correlation features of adjacent views, which can represent the correlation between corresponding features in adjacent views, which is beneficial to improving the temporal correlation between adjacent views in monocular video and enhancing the expression of temporal correlation.

[0102] In summary, the monocular image depth estimation algorithm based on spatiotemporal attention proposed in the embodiment of the present invention not only enhances the attention and learning of the monocular image depth estimation network to the spatial structure information and temporal correlation information in the monocular video, but also enhances the network's mapping and expression capabilities of spatiotemporal features, solves the problem of scene spatial structure drift caused by mismatching and mismapping of spatiotemporal features, and reconstructs a monocular depth map that is consistent with the spatiotemporal features of the input image sequence.

[0103] The embodiment of the present invention adopts a spatial attention submodule based on fusion feature self-attention to select and enhance the learning and expression of spatial structure information in the current view, and enhance the spatial consistency of depth features; and adopts a temporal attention submodule based on corresponding feature mutual attention to select and enhance the learning and expression of temporal correlation relationships in adjacent views, and enhance the long-term dependency of corresponding features. In order to reconstruct a high-precision monocular depth map, the embodiment of the present invention not only pays attention to the spatial information of the surface features of the object in the current view, but also pays attention to the temporal correlation relationship of the scene features between adjacent views.

[0104] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0105] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0106] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A monocular image depth estimation system based on spatiotemporal attention, characterized in that: include: Spatiotemporal learning module, spatiotemporal enhancement module and spatiotemporal decoding module; The spatiotemporal learning module is used to receive an input monocular image sequence, learn the spatiotemporal features in the monocular image sequence, and transmit the spatiotemporal feature sequence to the spatiotemporal enhancement module; The spatiotemporal enhancement module is used to perform a pooling operation on the spatiotemporal feature sequence to obtain a fused feature, perform a self-attention operation on the fused feature, and obtain a spatiotemporal feature after spatial enhancement; use a convolution operation to compress the spatiotemporal features of adjacent views respectively, and use an iterative selection method to perform a mutual attention operation between different feature maps to obtain a temporal correlation feature; The spatially enhanced spatiotemporal features and temporal correlation features are transmitted to the spatiotemporal decoding module; The spatiotemporal decoding module is used to upsample the spatiotemporal features and temporal correlation features after spatial enhancement through a deconvolution module, and output a depth map with the same depth size as the image.

2. The method according to claim 1, characterized in that The spatiotemporal learning module is specifically composed of 7 ConvLSTM units. The spatiotemporal feature learning module of the ConvLSTM unit extracts the two-dimensional spatiotemporal feature map of the input monocular image sequence through convolution operation. The learning process of the ConvLSTM unit is expressed as follows: Among them, "*" and Represent the convolution operation and Hadamard product respectively; and Represents input, unit output and hidden state respectively; "i t ”、"f t " and "o t "represent the feature maps of the input gate, forget gate and output gate of ConvLSTM respectively. The feature map is a three-dimensional tensor, and the last two dimensions represent the spatial dimensions; "W x. " and "b." represent the convolution kernel and offset term in the corresponding convolution calculation respectively; The spatiotemporal feature learning module of the ConvLSTM unit transfers the 2D spatiotemporal feature maps of adjacent frames through feature storage, fusion, and update operations.

3. The method according to claim 2, characterized in that The spatiotemporal enhancement module includes a spatial attention submodule and a temporal attention submodule executed in parallel; The spatial attention submodule is used to enhance the network's attention to and learning of spatiotemporal features, perform maximum pooling and average pooling operations on the spatiotemporal features to obtain two pooling features, fuse different pooling features, perform self-attention operations on the fused features, obtain spatially enhanced spatiotemporal features, and transmit them to the spatiotemporal decoding module; The temporal attention submodule is used to compress the spatiotemporal features of adjacent views by convolution operation, obtain two feature maps, perform mutual attention operation between the two feature maps by iterative selection, calculate the temporal correlation features enhanced by the corresponding feature mutual attention mechanism, and transmit them to the spatiotemporal decoding module.

4. The method according to claim 3, characterized in that The spatial attention submodule is specifically used to adopt a self-attention mechanism based on fusion features, with H×W representing the feature resolution, C and C1 representing the number of feature channels and the number of channels after dimensionality reduction, respectively, and MaxPooling and AvgPooling representing the maximum pooling and average pooling operations, respectively. Two different pooling operations are used: maximum pooling and average pooling to compress the spatiotemporal features, and two feature maps Φ1 and Φ2 with dimensions of H×W×C1 are obtained respectively. The feature maps are fused by adding the corresponding channels. and Get the feature map Φ3 with the dimension of H×W×C1; for the fusion feature Perform self-attention operation, use the deformed fusion features as key features and query features respectively, and use softmax function to normalize the key features and query features to obtain the spatial attention weight map of the current spatiotemporal features. The self-attention weight map The size is HW×HW, Each element in It is expressed as: in, represents the influence of the i-th pixel on the j-th pixel; φ 3i and φ 3j Represents the elements of the fusion feature Φ3; N = H × W, represents the total number of pixels in the input image; The fusion features After deformation again, it is used as the value feature of self-attention, and the value feature is combined with the spatial attention weight map Perform weighted calculations to highlight the selected spatial features and combine the selected spatial features with the spatiotemporal features. Perform the corresponding point addition operation to obtain the spatial enhanced feature Φ after being enhanced by the spatial attention submodule S , Φ S Each element in It is expressed as: Among them, Φ S The size is H×W×C, φ j Represents the elements of the spatiotemporal feature Φ, Represents the elements of the feature map Φ3, α represents the scale factor, and the output feature Φ S It is the original feature Φ and the enhanced feature at each position The weighted sum of .

5. The method according to claim 4, characterized in that The temporal attention submodule is specifically used to represent the convolution operation with a convolution kernel of 1×1 using Conv(1×1), and adopt a mutual attention mechanism to select and enhance the temporal correlation information between adjacent views; The convolution operation with a convolution kernel of 1×1 is used to respectively transform the spatiotemporal features of adjacent views. and The number of channels is compressed to obtain feature maps Φ′ with two dimensions of H×W×C2 t-1 and Φ′ t , using iterative selection in the feature map Φ′ t-1 and Φ′ t Perform mutual attention operation between them, and convert Φ′ t-1 and Φ′ t Divided into two groups of key features and query features, Φ′ t-1 is the key feature, Φ′ t is the query feature; Φ′ t is the key feature, Φ′ t-1 To query features, the Softmax function is used to normalize the features after each set of iterative selection, and the time attention weight maps of two adjacent frames are obtained respectively. and in, represents the time attention weight of the feature at time t to the feature at time t-1, represents the time attention weight of the feature at time t-1 to the feature at time t. The sizes of the two time attention weight maps are both HW×HW, where HW represents the size of the feature map resolution. Elements in It is expressed as: Elements in It is expressed as: in, Represents the current view Φ′ t For the previous view Φ′ t-1 The impact of Represents the previous view Φ′ t-1 For the current view Φ′ t The impact of and Represents the feature map Φ′ t-1 Elements of and Represents the feature map Φ′ t Elements; N = H × W, represents the total number of pixels of the input image; The temporal features at the same moment are weighted with the temporal attention weights, and then the weighted temporal features are added to the temporal and spatial features at the moment to obtain the temporal enhanced features enhanced by the temporal attention submodule. The temporal enhanced features at time t-1 are Φ″ t→t-1 Elements in It is expressed as: Time series enhancement feature Φ″ at time t t-1→t Elements in It is expressed as: Among them, the timing enhancement feature Φ″ t→t-1 and Φ″ t-1→t The size of is H×W×C; γ represents the scale factor, and Represents the spatiotemporal feature Φ t-1 Elements of and Represents the spatiotemporal feature Φ t elements.

Images