Image depth estimation method, device, electronic device and storage medium

Abstract

The present application is applicable to the field of image processing, and provides a method, device, electronic device and storage medium for image depth estimation. The image depth estimation method includes obtaining an image to be processed, inputting the image to be processed into a depth estimation network, and obtaining depth information output by the depth estimation network. The depth estimation network includes an encoder and a decoder, the encoder includes N coding layers, and the decoder includes N‑1 decoding layers; the input image of each coding layer is obtained by downsampling the image to be processed by a preset multiple, and the input image of each decoding layer is obtained by splicing the output images of the N coding layers, and the depth information is determined according to the output images of the N‑1 decoding layers, where N represents an integer greater than 0. Therefore, the input information of each decoding layer integrates the multi-level semantic information of the coding layer, reduces the semantic generation gap between the feature maps of the encoder and the decoder, and improves the accuracy of the restored scene depth information.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device, electronic device and storage medium for image depth estimation. Background Art

[0002] The images taken by the camera are generally two-dimensional images, which lose the depth information of the scene in which the image is located. Recovering the scene depth information of the image from the two-dimensional image helps to better understand the three-dimensional structure of the scene and the three-dimensional relationship between objects in the image, so as to better complete visual tasks. The existing method of recovering the scene depth information of the image is generally to recover it through a neural network model. The neural network model includes an encoder and a decoder. The encoder increases the receptive field of the feature map and reduces the computational overhead by downsampling the image step by step. The decoder is used to recover the scene depth information with the original image resolution from the downsampled image. However, in the current neural network model used for depth estimation, the encoder features are simply mixed with the decoder features, ignoring the semantic generation gap between the encoder and decoder feature maps, thereby reducing the accuracy of the recovered scene depth information. Summary of the invention

[0003] In view of this, embodiments of the present application provide a method, device, electronic device and storage medium for estimating the depth of an image, which solve the problem of low accuracy of restored scene depth information.

[0004] A first aspect of an embodiment of the present application provides a method for estimating depth of an image, comprising:

[0005] Get the image to be processed;

[0006] The image to be processed is input into a depth estimation network to obtain depth information output by the depth estimation network; the depth estimation network includes an encoder and a decoder, the encoder includes N encoding layers, and the decoder includes N-1 decoding layers; the input image of each encoding layer is obtained by downsampling the image to be processed by a preset multiple, and the input image of each decoding layer is obtained by splicing the output images of N encoding layers, and the depth information is determined according to the output image of N-1 decoding layers, where N represents an integer greater than 0.

[0007] In a possible implementation, an input image of the N-1th decoding layer is obtained by splicing a first input image and a second input image, the first input image is obtained by sequentially fine-tuning and upsampling the output image of the Nth encoding layer, and the second input image is obtained by sequentially performing 2-upsample on the output image of the i-th encoding layer. N-1-i The image is downsampled and fine-tuned by times, i=0, 1, 2, ..., N-1, and the fine-tuning is used to splice the local feature information and the global feature information of the image.

[0008] In a possible implementation, the input image of the k-th decoding layer is obtained by splicing the third input image and the fourth input image, the third input image is obtained by sequentially fine-tuning and upsampling the output image of the k+1-th decoding layer, and the fourth input image is obtained by sequentially processing the output image of the i-th encoding layer through 2 k-i The image is obtained by downsampling and fine-tuning, wherein the fine-tuning is used to splice local feature information and global feature information of the image, k=0, 1, 2, ..., N-2, ki≥0.

[0009] In a possible implementation, the image fine-tuning process includes:

[0010] Extracting a local feature map of the image to be fine-tuned, wherein the local feature map includes local feature information;

[0011] Performing non-overlapping sampling on the image to be fine-tuned to obtain non-overlapping sub-feature maps;

[0012] Determine a global feature map according to the sub-feature map, wherein the global feature map includes global feature information;

[0013] The image to be fine-tuned, the local feature map, and the global feature map are spliced ​​to obtain a fine-tuned image.

[0014] In a possible implementation manner, determining a global feature map according to the sub-feature map includes:

[0015] Determine a feature vector corresponding to the sub-feature map;

[0016] The feature vector is sequentially subjected to feature vector transformation, linear transformation, resolution restoration, and expansion processing to obtain the global feature map.

[0017] In a possible implementation, the upsampling process of the image includes:

[0018] Dividing the image to be upsampled into a first low-resolution image and a second low-resolution image according to the channel dimension;

[0019] Upsampling the first low-resolution image by using a bilinear interpolation method to obtain a first high-resolution image;

[0020] Performing dilation processing and fine-tuning processing on the second low-resolution image to obtain a second high-resolution image;

[0021] The first high-resolution image and the second high-resolution image are spliced ​​to obtain an upsampled image.

[0022] In a possible implementation manner, the depth information is obtained by upsampling an output image of the 0th decoding layer.

[0023] A second aspect of an embodiment of the present application provides a depth estimation device for an image, comprising:

[0024] An acquisition module, used for acquiring an image to be processed;

[0025] A processing module is used to input the image to be processed into a depth estimation network to obtain depth information output by the depth estimation network; the depth estimation network includes an encoder and a decoder, the encoder includes N coding layers, and the decoder includes N-1 decoding layers; the input image of each coding layer is obtained by downsampling the image to be processed by a preset multiple, and the input image of each decoding layer is obtained by splicing the output images of the N coding layers, and the depth information is determined according to the output image of the N-1 decoding layer, where N represents an integer greater than 0.

[0026] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image depth estimation method as described in the first aspect above is implemented.

[0027] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image depth estimation method as described in the first aspect above is implemented.

[0028] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the image depth estimation method described in any one of the first aspects above.

[0029] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the depth estimation network includes an encoder and a decoder, the encoder includes N encoding layers, and the decoder includes N-1 decoding layers; the input image of each encoding layer is obtained by downsampling the image to be processed by a preset multiple, and the input image of each decoding layer is obtained by splicing the output images of the N encoding layers, so the input information of each decoding layer is associated with the output information of the N decoding layers, so that the input information of each decoding layer integrates the multi-level semantic information of the encoding layer, thereby reducing the semantic generation gap between the feature maps of the encoder and the decoder. Afterwards, the image to be processed is input into the depth estimation network, and the depth information is determined according to the output image of the N-1 decoding layer, thereby improving the accuracy of the restored scene depth information. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0031] Figure 1 It is a schematic diagram of the implementation flow of a method for estimating the depth of an image provided by an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of a depth estimation network provided in an embodiment of the present application;

[0033] Figure 3 It is a schematic diagram of the calculation flow of the fine-tuning processing function provided in an embodiment of the present application;

[0034] Figure 4 Schematic diagram of the calculation process of the upsampling function provided in the embodiment of the present application;

[0035] Figure 5 is a schematic diagram of a depth estimation device for an image provided in an embodiment of the present application;

[0036] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0038] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0039] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0040] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0041] The current neural network models used for depth estimation simply mix the encoder features with the decoder features, ignoring the semantic gap between the encoder and decoder feature maps, thereby reducing the accuracy of the recovered scene depth information.

[0042] To this end, the present application provides a method for estimating the depth of an image, wherein the image to be processed is input into a depth estimation network, wherein the depth estimation network includes an encoder and a decoder, wherein the encoder includes N encoding layers, and the decoder includes N-1 decoding layers. The input image of each decoding layer is obtained by splicing the output images of the N encoding layers, so that the input information of each decoding layer is associated with the output information of each encoding layer, so that the input information of each decoding layer integrates the multi-level semantic information of the encoding layer, thereby reducing the semantic generation gap between the feature maps of the encoder and the decoder. The depth information is then determined based on the output image of the N-1 decoding layer, thereby improving the accuracy of the restored scene depth information.

[0043] The depth estimation method of an image provided in this application is exemplified below.

[0044] Please see attached Figure 1 , a method for estimating the depth of an image provided by an embodiment of the present application includes:

[0045] S101: Acquire an image to be processed.

[0046] The image to be processed is a two-dimensional image, which is used to restore scene depth information.

[0047] In one embodiment, the image to be processed is obtained after preprocessing the original image. The preprocessing includes random flipping, random cropping, data normalization and other processing processes. Exemplarily, after preprocessing, tensor data with a dimension of C×H×W is obtained, that is, the dimension of the image to be processed is C×H×W, where C represents the number of channels, H represents the image height, and W represents the image width.

[0048] In one embodiment, the original image is a monocular image of the autonomous driving dataset KITTI.

[0049] S102: Input the image to be processed into a depth estimation network to obtain depth information output by the depth estimation network; the depth estimation network includes an encoder and a decoder, the encoder includes N encoding layers, and the decoder includes N-1 decoding layers; the input image of each encoding layer is obtained by downsampling the image to be processed by a preset multiple, and the input image of each decoding layer is obtained by splicing the output images of N encoding layers, and the depth information is determined according to the output image of the N-1 decoding layer, where N represents an integer greater than 0. In one embodiment, the image to be processed is the input image of the 0th encoding layer, and when i is greater than 0, the input image of the i-th encoding layer is obtained by sequentially performing feature extraction and 2-fold downsampling on the input image of the i-1th encoding layer. For example, feature extraction and 2-fold downsampling are performed on the input image of the 0th coding layer in sequence to obtain the input image of the 1st coding layer; feature extraction and 2-fold downsampling are performed on the input image of the 1st coding layer in sequence to obtain the input image of the 2nd coding layer; feature extraction and 2-fold downsampling are performed on the input image of the 2nd coding layer in sequence to obtain the input image of the 3rd coding layer.

[0050] In another embodiment, the image to be processed is the input image of the 0th coding layer, and the image to be processed is subjected to feature extraction and downsampling of different times in sequence to obtain the input image of each coding layer. For example, the image obtained by sequentially performing feature extraction and downsampling by 2 times on the image to be processed is the input image of the 1st coding layer, the image obtained by sequentially performing feature extraction and downsampling by 4 times on the image to be processed is the input image of the 2nd coding layer, and the image obtained by sequentially performing feature extraction and downsampling by 2 times on the image to be processed is the input image of the 2nd coding layer. N The image obtained by downsampling is the input image of the Nth coding layer.

[0051] Each decoding layer corresponds to a splicing rule. The output images of N encoding layers are spliced ​​according to the splicing rule to obtain the input image of each decoding layer. The output image of each decoding layer is processed to obtain depth information.

[0052] In one embodiment, the input image of the N-1th decoding layer is obtained by splicing the first input image and the second input image, the first input image is obtained by sequentially fine-tuning and up-sampling the output image of the Nth encoding layer, and the second input image is obtained by sequentially performing 2 N-1-i The first input image and the second input image are concatenated and input to the N-1th decoding layer, so that the input information of the N-1th decoding layer can be integrated with the semantic information of the Nth encoding layer, thereby reducing the semantic generation gap between the feature maps of the encoder and the decoder.

[0053] In one embodiment, the input image of the k-th decoding layer is obtained by splicing the third input image and the fourth input image, the third input image is obtained by sequentially fine-tuning and upsampling the output image of the k+1-th decoding layer, and the fourth input image is obtained by sequentially processing the output image of the i-th encoding layer through 2 k-i The third input image and the fourth input image are concatenated and input to the kth decoding layer, so that the input information of each decoding layer can be integrated with the semantic information of the N encoding layers, thereby reducing the semantic generation gap between the feature maps of the encoder and the decoder.

[0054] In one embodiment, after the first input image and the second input image are spliced, the spliced ​​image is fine-tuned to obtain an input image of the N-1th decoding layer. After the third input image and the fourth input image are spliced, the spliced ​​image is fine-tuned to obtain an input image of the kth decoding layer.

[0055] For example, according to the formula

[0056]

[0057] Determine the input image of each decoding layer, where represents the input image of the kth decoding layer, F refine represents the fine-tuning processing function, F up represents the upsampling function, represents the output image of the Nth coding layer, represents the output image of the k+1th decoding layer; Indicates that the input features of the i-th stage are 2 j Downsampling by a factor of 2, consisting of a convolution kernel with a stride of 2 and a kernel size of 3×3 and an activation function; It means that the feature image of the encoder layer in the i-th stage is processed 2 j The feature image obtained after downsampling and information fine-tuning,

[0058] j=N-1-i or j=ki.

[0059] In one embodiment, before fine-tuning the image, channel transformation is first performed on the image, and then the image obtained by the channel transformation is fine-tuned, so as to compress the channels of the image, make the image information more compact, and reduce the computational overhead.

[0060] In one embodiment, the depth information is obtained by upsampling the output image of the 0th decoding layer. Among the output images of each decoding layer, the output image of the 0th decoding layer has the highest resolution, and the depth information is determined based on the output image of the 0th decoding layer, thereby improving the accuracy of the determined depth information.

[0061] In one embodiment, N=4, the structure of the depth estimation network is as follows Figure 2 As shown. The original image is preprocessed to obtain the image to be processed, that is, the input image of the 0th coding layer enc0. The input image of the 0th coding layer is sequentially subjected to feature extraction and 2-fold downsampling to obtain the input image of the 1st coding layer enc1. The input image of the 1st coding layer is sequentially subjected to feature extraction and 2-fold downsampling to obtain the input image of the 2nd coding layer enc2. The input image of the 2nd coding layer is sequentially subjected to feature extraction and 2-fold downsampling to obtain the input image of the 3rd coding layer enc3. The input image of the 3rd coding layer is sequentially subjected to feature extraction and 2-fold downsampling to obtain the input image of the 4th coding layer enc4. The output image of the 4th coding layer is sequentially subjected to channel transformation, information fine-tuning, and upsampling to obtain the first input image. The output image of the 0th coding layer is sequentially downsampled by 8 times and fine-tuned, the output image of the 1st coding layer is sequentially subjected to feature extraction, downsampled by 4 times, and fine-tuned, the output image of the 2nd coding layer is sequentially subjected to feature extraction, downsampled by 2 times, and fine-tuned, and the output image of the 3rd coding layer is sequentially subjected to feature extraction, downsampled by 1 times, and fine-tuned to obtain a second input image, the first input image and the second input image are spliced, and the spliced ​​image is sequentially subjected to channel transformation and fine-tuning to obtain an input image of the 3rd decoding layer. Among them, the image obtained by downsampling the image by 1 times is still the image before sampling.

[0062] The output image of the third decoding layer dec3 is sequentially subjected to channel transformation, information fine-tuning, and up-sampling to obtain the third input image of the second decoding layer. The output image of the 0th coding layer is sequentially subjected to feature extraction, 4-fold downsampling, and fine-tuning, the output image of the 1st coding layer is sequentially subjected to feature extraction, 2-fold downsampling, and fine-tuning, the output image of the 2nd coding layer is sequentially subjected to feature extraction, 1-fold downsampling, and fine-tuning to obtain the fourth input image of the second decoding layer, the third input image and the fourth input image of the second decoding layer are spliced, and the spliced ​​images are sequentially subjected to channel transformation and fine-tuning to obtain the input image of the second decoding layer.

[0063] The output image of the second decoding layer dec2 is sequentially subjected to channel transformation, information fine-tuning, and up-sampling to obtain the third input image of the first decoding layer. The output image of the 0th encoding layer is sequentially subjected to feature extraction, 2-fold downsampling, and fine-tuning. The output image of the 1st encoding layer is sequentially subjected to feature extraction, 1-fold downsampling, and fine-tuning to obtain the fourth input image of the first decoding layer. The third input image and the fourth input image of the first decoding layer are spliced, and the spliced ​​images are sequentially subjected to channel transformation and fine-tuning to obtain the input image of the first decoding layer.

[0064] The output image of the first decoding layer dec1 is sequentially subjected to channel transformation, information fine-tuning, and up-sampling to obtain the third input image of the 0th decoding layer. The output image of the 0th coding layer is sequentially subjected to feature extraction, 1-fold down-sampling, and fine-tuning to obtain the fourth input image of the 0th decoding layer. The third input image and the fourth input image of the 0th decoding layer are spliced, and the spliced ​​image is sequentially subjected to channel transformation and fine-tuning to obtain the input image of the 0th decoding layer. The output image of the 0th decoding layer is up-sampled to obtain depth information.

[0065] In the above embodiment, the image to be processed is input into the depth estimation network, and the depth estimation network includes an encoder and a decoder, the encoder includes N layers of encoding layers, and the decoder includes N-1 layers of decoding layers; the input image of each encoding layer is obtained by downsampling the image to be processed by a preset multiple, and the input image of each decoding layer is obtained by splicing the output images of the N layers of encoding layers. Therefore, the input information of each decoding layer is associated with the output information of the N layers of decoding layers, so that the input information of each decoding layer integrates the multi-level semantic information of the encoding layer, thereby reducing the semantic generation gap between the feature maps of the encoder and the decoder. Afterwards, the image to be processed is input into the depth estimation network, and the depth information is determined according to the output image of the N-1 layer of decoding layer, thereby improving the accuracy of the restored scene depth information.

[0066] In one embodiment, the fine-tuning process of an image includes: extracting a local feature map of the image to be fine-tuned, wherein the local feature map includes local feature information; performing non-overlapping sampling on the image to be fine-tuned to obtain non-overlapping sub-feature maps; determining a global feature map based on the sub-feature maps, wherein the global feature map includes global feature information; and splicing the image to be fine-tuned, the local feature map, and the global feature map to obtain a fine-tuned image.

[0067] Through the process of fine-tuning, the long-range dependency of pixels in the feature map can be modeled to obtain a feature image with both local and global feature information, providing better semantic information for the pixels in the image and better inferring the depth information of the pixels.

[0068] In one embodiment, the process of determining a global feature map based on a sub-feature map is as follows: determining a feature vector corresponding to the sub-feature map, performing feature vector transformation, nonlinear transformation, resolution recovery, and expansion processing on the feature vector in sequence to obtain a global feature map, thereby improving the nonlinear fitting ability of the obtained global feature map.

[0069] In other embodiments, a pre-trained deep convolutional neural network model may also be used to extract local feature maps and global feature maps of the image to be fine-tuned.

[0070] Exemplarily, the fine-tuning processing function F refine The calculation process is as follows Figure 3 shown.

[0071] Set X refine =F refine (X), X represents the feature map to be fine-tuned, In the embodiment of the present application, the feature map to be fine-tuned may be the output image of the Nth coding layer, the output image of the ith coding layer, the output image of the k+1th decoding layer, etc.

[0072] First, according to the formula

[0073] Determine the local feature map X3 of the feature map X to be fine-tuned. Among them, the function F squeeze It is composed of a 1×1 convolution kernel and an activation function, which is used to compress the feature map. The function F local It is composed of a 3×3 convolution kernel and an activation function, which represents a local feature extractor and is used to perceive the local context information of the feature map. The function F unsqueeze It consists of a 1×1 convolution kernel and an activation function, which represents the dilation operation of the feature map and is used to restore the channel dimension. Represents the feature map space, C represents the number of channels, H represents the image height, and W represents the image width. The dimension of the feature map X to be fine-tuned is C×H×W, and the dimension of the local feature map X3 is also C×H×W.

[0074] Then, according to the formula

[0075]

[0076] Determine the global feature map of the feature map X to be fine-tuned. Where X1 represents the feature map after the feature map X is compressed, and the function F sampleIt represents the non-overlapping sampling operation of the feature map, which is used to sample the feature map into non-overlapping sub-feature maps. It consists of a convolution kernel whose length and width are the length and width of the sub-feature map, a convolution kernel whose step length and width are the length and width of the sub-feature map, and an activation function. The function F embed It is used to transform the two-dimensional sub-feature map into a one-dimensional feature vector, and then exchange the one-dimensional feature vector dimension with the channel dimension. x represents the feature vector embedded in the sub-feature map, and H sub ,W sub ,C embed Respectively represent the height of the sub-feature map, the width of the sub-feature map, and the number of feature vectors obtained. Function F linear represents the linear transformation function, F linear (x) is used to linearly transform the eigenvector x into three eigenvectors of the same number, F reshape Represents the dimension transformation function of the feature vector, which is used to adjust the feature vector to a specified shape, F split It is used to divide the feature vector into three sub-feature vectors: Q (Query), K (Key), and V (Value), where Q is the query feature vector, K is the key-value feature vector, V is the value feature vector, and N att represents the number of attention heads, s represents the scalar scaling factor, "T" represents the transposition operation, "*" represents the multiplication operation, and Softmax represents the logistic regression function. The feature vector processed by logistic regression is further transformed in dimension and linearly transformed, that is, the feature vector processed by logistic regression is inversely transformed to obtain the inversely transformed feature vector x1. mlp Represents a multilayer perceptron consisting of two linear transformation layers and an activation function, which is used to enhance the nonlinear fitting ability of the feature vector. rrl represents the resolution recovery function, which consists of a 1×1 convolution kernel, a deconvolution kernel whose length and width are the length and width of the sub-feature map, and a step length whose length and width are the length and width of the sub-feature map, respectively. rrl (x2) represents the transformation of feature vector x2 into a global feature map X4 of the same dimension as feature map X1. unsqueeze represents the dilation operation of the feature map, F unsqueeze (X4) is used to transform the global feature map X4 into a global feature map X5 having the same dimension as the feature map X.

[0077] After obtaining the local feature map and the global feature map, according to the formula

[0078] X refine =X+X3+X5, The feature map to be fine-tuned, the local feature map, and the global feature map are spliced ​​to obtain the fine-tuned image

[0079] In one embodiment, the upsampling process of an image includes: dividing the image to be upsampled into a first low-resolution image and a second low-resolution image according to a channel dimension; upsampling the first low-resolution image by a bilinear interpolation method to obtain a first high-resolution image; dilating and fine-tuning the second low-resolution image to obtain a second high-resolution image; and splicing the first high-resolution image and the second high-resolution image to obtain an upsampled image.

[0080] Upsampling the image can make up for the shortcoming that bilinear interpolation upsampling cannot recover fine-grained high-resolution feature maps from low-resolution feature maps, and improves the ability to recover fine-grained high-resolution feature images from low-resolution feature maps.

[0081] In other embodiments, the image may be upsampled by using a nearest neighbor interpolation algorithm, a transposed convolution algorithm, or an up-pooling algorithm.

[0082] Exemplarily, the upsampling function F up The calculation process is as follows Figure 4 shown.

[0083] Set X high =F up (X low ), Represents the feature map to be fine-tuned and upsampled, Represents the image after fine-tuning and upsampling.

[0084] According to the formula The feature map to be fine-tuned and upsampled Divide equally into two sub-feature maps according to the channel dimension and is the first low-resolution feature map, i.e., low-resolution feature map A, is the second low-resolution feature map, i.e., the low-resolution feature map B, the feature map to be fine-tuned and up-sampled The dimensions are C×H×W, and The dimensions of

[0085] For the first low-resolution feature map Bilinear interpolation is used to Upsampling is performed to obtain the first high-resolution feature map, namely, high-resolution feature map A.

[0086] Specifically, according to the formula Compute the first high-resolution feature map, where represents the first high-resolution feature map, Fbilinear Represents a bilinear interpolation function.

[0087] For the second lowest resolution feature map According to the formula right Perform expansion processing to obtain the feature map after expansion processing Among them, F expand Represents the dilation function, which consists of a convolution kernel with a kernel size of 1×1 and an activation function.

[0088] Afterwards, according to the formula right Perform fine-tuning to obtain the feature map after fine-tuning The dimension is 2C×H×W, according to the formula

[0089] Will Rearrange into dimensions The second high-resolution feature map of Represents the second high-resolution feature map, that is, the high-resolution feature map B, F pixelshuffle Represents an element rearrangement function.

[0090] Afterwards, the first high-resolution feature map and the second high-resolution feature map B are concatenated according to the channel dimension to obtain an image after fine-tuning and upsampling. Specifically, according to the formula Get the fine-tuned upsampled image Among them, F concat Represents the concatenation function.

[0091] In one embodiment, before estimating the depth information of the image to be processed, the depth estimation network needs to be trained to obtain the optimal parameters of the depth estimation network. The training sample for training the depth estimation network is a sample image sequence, and the sample image sequence includes a target frame image and a reference frame image. The sample image sequence is input into the depth estimation network to obtain the depth information of the estimated sample image sequence output by the depth estimation network. The sample image sequence is input into the camera motion network to obtain the relative pose between the sample image sequences output by the camera motion network. Afterwards, the reference frame image is affine transformed into a three-dimensional space according to the depth information of the reference frame image and the camera internal parameter to obtain the reference frame image in the three-dimensional space. According to the relative pose between the reference frame image and the target frame image output by the camera motion network and the camera internal parameter, the reference frame image in the three-dimensional space is affine transformed into the corresponding target frame image three-dimensional space, and then the corresponding target frame image three-dimensional space is projected onto a two-dimensional plane to obtain the image coordinates after affine transformation. Afterwards, the target frame image is synthesized according to the image coordinates after affine transformation, and the difference between the target frame image in the training sample and the synthesized target frame image is used as a guidance signal to jointly guide the parameter update of the depth estimation network and the camera motion network. Until the optimal parameters of the depth estimation network and the camera motion network are obtained or the preset number of iterations is reached.

[0092] Among them, in the training stage, the output images of the 0th to N-2nd decoding layers of the depth estimation network are sequentially upsampled and subjected to the disparity output layer to obtain depth images with the same resolution as the output images of the image to be processed to the N-2nd decoding layer, and the depth information is determined according to the depth images. Exemplarily, N=4, the output images of the 0th to 2nd decoding layers are sequentially upsampled and subjected to the disparity output layer to obtain depth images with the same resolution as the image to be processed, the output image of the 0th decoding layer, and the output image of the 2nd decoding layer, respectively, for a total of 3 scales of depth images, a multi-scale target image is reconstructed based on the depth image, and the update of the network weights is guided by the difference between the image to be processed and the reconstructed multi-scale target image.

[0093] After obtaining the optimal parameters of the depth estimation network, the depth estimation network can estimate the depth information of the image to be processed, and the camera motion network can estimate the relative pose between adjacent frames in the image sequence where the image to be processed is located.

[0094] In one embodiment, the dimensions of the images in the training samples are C×H×W, where H=256 and W=832. In the images input to the depth estimation network, C=3, and in the images input to the camera motion network, C=9.

[0095] In an embodiment of the present application, in a depth estimation network, the input image of each decoding layer is obtained by splicing the output images of N encoding layers. Therefore, the input information of each decoding layer is associated with the output information of N decoding layers, so that the input information of each decoding layer integrates the multi-level semantic information of the encoding layer, thereby reducing the semantic generation gap between the feature maps of the encoder and the decoder.

[0096] The first input image is obtained by fine-tuning and upsampling the output image of the Nth coding layer in sequence, and the second input image is obtained by fine-tuning and upsampling the output image of the i-th coding layer in sequence. N-1-i The first input image and the second input image are concatenated and input into the N-1th decoding layer, so that the input information of the N-1th decoding layer can be integrated with the semantic information of the Nth encoding layer, thereby reducing the semantic generation gap between the feature maps of the encoder and the decoder.

[0097] The third input image is obtained by fine-tuning and upsampling the output image of the k+1th decoding layer in sequence, and the fourth input image is obtained by fine-tuning and upsampling the output image of the i-th encoding layer in sequence. k-i The third input image and the fourth input image are concatenated and input into the kth decoding layer, so that the input information of each decoding layer can be integrated with the semantic information of N encoding layers, reducing the semantic generation gap between the feature maps of the encoder and the decoder.

[0098] Through the process of fine-tuning, the long-range dependency of pixels in the feature map can be modeled to obtain a feature image with both local and global feature information, providing better semantic information for the pixels in the image and better inferring the depth information of the pixels.

[0099] By upsampling the image, the shortcoming that bilinear interpolation upsampling cannot recover fine-grained high-resolution feature maps from low-resolution feature maps can be compensated, and the ability to recover fine-grained high-resolution feature images from low-resolution feature maps can be improved.

[0100] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0101] Corresponding to the image depth estimation method described in the above embodiment, Figure 5 A structural block diagram of a depth estimation device for an image provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0102] like Figure 5 As shown, the image depth estimation device includes:

[0103] An acquisition module 51 is used to acquire an image to be processed;

[0104] The processing module 52 is used to input the image to be processed into a depth estimation network to obtain depth information output by the depth estimation network; the depth estimation network includes an encoder and a decoder, the encoder includes N encoding layers, and the decoder includes N-1 decoding layers; the input image of each encoding layer is obtained by downsampling the image to be processed by a preset multiple, and the input image of each decoding layer is obtained by splicing the output images of the N encoding layers, and the depth information is determined based on the output image of the N-1 decoding layer, where N represents an integer greater than 0.

[0105] In a possible implementation, an input image of the N-1th decoding layer is obtained by splicing a first input image and a second input image, the first input image is obtained by sequentially fine-tuning and upsampling the output image of the Nth encoding layer, and the second input image is obtained by sequentially performing 2-upsample on the output image of the i-th encoding layer. N-1-i The image is downsampled and fine-tuned by times, i=0, 1, 2, ..., N-1, and the fine-tuning is used to splice the local feature information and the global feature information of the image.

[0106] In a possible implementation, the input image of the k-th decoding layer is obtained by splicing the third input image and the fourth input image, the third input image is obtained by sequentially fine-tuning and upsampling the output image of the k+1-th decoding layer, and the fourth input image is obtained by sequentially processing the output image of the i-th encoding layer through 2 k-i The image is obtained by downsampling and fine-tuning, wherein the fine-tuning is used to splice local feature information and global feature information of the image, k=0, 1, 2, ..., N-2, ki≥0.

[0107] In a possible implementation, the processing module 52 is specifically configured to:

[0108] Extracting a local feature map of the image to be fine-tuned, wherein the local feature map includes local feature information;

[0109] Performing non-overlapping sampling on the image to be fine-tuned to obtain non-overlapping sub-feature maps;

[0110] Determine a global feature map according to the sub-feature map, wherein the global feature map includes global feature information;

[0111] The image to be fine-tuned, the local feature map, and the global feature map are spliced ​​to obtain a fine-tuned image.

[0112] In a possible implementation, the processing module 52 is further configured to:

[0113] Determine a feature vector corresponding to the sub-feature map;

[0114] The feature vector is sequentially subjected to feature vector transformation, nonlinear transformation, resolution restoration, and expansion processing to obtain the global feature map.

[0115] In a possible implementation, the processing module 52 is further configured to:

[0116] Dividing the image to be upsampled into a first low-resolution image and a second low-resolution image according to the channel dimension;

[0117] Upsampling the first low-resolution image by using a bilinear interpolation method to obtain a first high-resolution image;

[0118] Performing dilation processing and fine-tuning processing on the second low-resolution image to obtain a second high-resolution image;

[0119] The first high-resolution image and the second high-resolution image are spliced ​​to obtain an upsampled image.

[0120] In a possible implementation manner, the depth information is obtained by upsampling an output image of the 0th decoding layer.

[0121] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0122] Figure 6 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may be a computing device such as a desktop computer, a notebook, or a palmtop.

[0123] like Figure 6 As shown, the electronic device of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the processor 61 executes the computer program 63, the steps in the above-mentioned image depth estimation method embodiment are implemented, for example Figure 1 Alternatively, when the processor 61 executes the computer program 63, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of the acquisition module 51 to the processing module 52 are shown.

[0124] Exemplarily, the computer program 63 may be divided into one or more modules / units, which are stored in the memory 62 and executed by the processor 61 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 63 in the electronic device.

[0125] Those skilled in the art will understand that Figure 6 These are merely examples of electronic devices and do not constitute a limitation of the electronic device. The electronic device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0126] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0127] The memory 62 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. The memory 62 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 62 may also include both an internal storage unit of the electronic device and an external storage device. The memory 62 is used to store the computer program and other programs and data required by the electronic device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0128] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0129] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0133] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for estimating the depth of an image, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a depth estimation network to obtain depth information output by the depth estimation network; The depth estimation network includes an encoder and a decoder, the encoder includes N encoding layers, and the decoder includes N-1 decoding layers; the input image of each encoding layer is obtained by downsampling the image to be processed by a preset multiple, and the input image of each decoding layer is obtained by splicing the output images of N encoding layers, and the depth information is determined based on the output image of N-1 decoding layers, where N represents an integer greater than 0.

2. The method according to claim 1, characterized in that The input image of the N-1th decoding layer is obtained by splicing the first input image and the second input image, the first input image is obtained by fine-tuning and upsampling the output image of the Nth encoding layer in sequence, and the second input image is obtained by 2-th the output image of the i-th encoding layer in sequence. N-1-i The image is downsampled and fine-tuned by times, i=0, 1, 2, ..., N-1, and the fine-tuning is used to splice the local feature information and the global feature information of the image.

3. The method according to claim 1, characterized in that The input image of the k-th decoding layer is obtained by splicing the third input image and the fourth input image, the third input image is obtained by fine-tuning and upsampling the output image of the k+1-th decoding layer in sequence, and the fourth input image is obtained by fine-tuning and upsampling the output image of the i-th encoding layer in sequence. k-i The image is obtained by downsampling and fine-tuning, wherein the fine-tuning is used to splice local feature information and global feature information of the image, k=0, 1, 2, ..., N-2, ki≥0.

4. The method according to claim 2 or 3, characterized in that: The image fine-tuning process includes: Extracting a local feature map of the image to be fine-tuned, wherein the local feature map includes local feature information; Performing non-overlapping sampling on the image to be fine-tuned to obtain non-overlapping sub-feature maps; Determine a global feature map according to the sub-feature map, wherein the global feature map includes global feature information; The image to be fine-tuned, the local feature map, and the global feature map are spliced ​​to obtain a fine-tuned image.

5. The method according to claim 4, characterized in that Determining a global feature map according to the sub-feature map includes: Determine a feature vector corresponding to the sub-feature map; The feature vector is sequentially subjected to feature vector transformation, linear transformation, resolution restoration, and expansion processing to obtain the global feature map.

6. The method according to claim 2 or 3, characterized in that: The image upsampling process includes: Dividing the image to be upsampled into a first low-resolution image and a second low-resolution image according to the channel dimension; Upsampling the first low-resolution image by using a bilinear interpolation method to obtain a first high-resolution image; Performing dilation processing and fine-tuning processing on the second low-resolution image to obtain a second high-resolution image; The first high-resolution image and the second high-resolution image are spliced ​​to obtain an upsampled image.

7. The method according to claim 1, characterized in that The depth information is obtained by upsampling the output image of the 0th decoding layer.

8. A depth estimation device for an image, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A processing module, used for inputting the image to be processed into a depth estimation network to obtain depth information output by the depth estimation network; The depth estimation network includes an encoder and a decoder, the encoder includes N encoding layers, and the decoder includes N-1 decoding layers; the input image of each encoding layer is obtained by downsampling the image to be processed by a preset multiple, and the input image of each decoding layer is obtained by splicing the output images of N encoding layers, and the depth information is determined based on the output image of N-1 decoding layers, where N represents an integer greater than 0.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images