A monocular depth estimation method fusing continuity features

Abstract

A monocular depth estimation method fusing continuity features comprises the following steps: preprocessing an original image to make implicit relations exist between images; inputting three images at the same time, and obtaining a depth map output through an encoder and a decoder containing four stages and four parallel subnets with different scales; and the loss function comprises a minimum photometric error loss, a smoothing loss, a cross-scale consistency loss and a continuity loss. The application effectively improves the depth measurement precision by combining scale invariance and continuity features on the basis of keeping lightweight, and the training threshold is low due to the self-supervised characteristics of the application, so that the requirement for a data set is low.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of monocular depth estimation algorithm based on deep learning, and particularly relates to a monocular depth estimation method fusing continuity features. BACKGROUND

[0002] Estimating accurate depth map from images is an important topic for many tasks in computer vision, which plays an important role in applications such as autonomous driving, augmented reality and 3D scene reconstruction. In these applications, laser radar, RGB-D camera and other hardware facilities can be used to help obtain depth information. However, monocular video is obviously easier to obtain, and is not easily affected by interference of hardware facilities, and has low maintenance cost.

[0003] Compared with traditional depth estimation methods, deep learning methods have been proven to be able to achieve better results on this task. Typically, the depth of each pixel is estimated by minimizing the error between the predicted depth and the true depth, and they mostly improve the accuracy of depth estimation by optimizing the network structure and the loss function, such as using a multi-scale convolutional network architecture to promote depth estimation. Some works improve the accuracy by jointly semantic segmentation or by including conditional random field (CRF). However, these methods are fully supervised, and a large amount of labeled data is required during training, which is obviously expensive and troublesome. Self-supervised methods use dense geometric constraints to eliminate the dependence on depth ground truth, and train the model by minimizing the image reconstruction error, effectively avoiding this problem. Garg et al. first proposed a new concept of using stereo image pairs to establish a depth model to estimate depth without depth labels. Zhou et al. proposed SfMLearner to predict depth from monocular video, and learned relative pose through an additional pose network, which was a pioneering work. Subsequently, many works have optimized some difficulties of depth estimation: Godard et al. proposed a minimum re-projection loss in Monodepth2 to solve the problem of occlusion and dynamic objects.

[0004] However, the interpretation of the characteristics of the image in the existing depth estimation related papers is still not very comprehensive, resulting in the problem of inconsistent depth of the same depth object in the depth map and discontinuous depth of the same object. This is affected by the problem that the pixel points at the same depth in the image generated by the imaging principle of the camera do not lie in the same image height. SUMMARY

[0005] The present application overcomes the above-mentioned shortcomings of the prior art, and provides a monocular depth estimation method fusing continuity features to further improve the depth estimation accuracy.

[0006] The application discloses a monocular depth estimation method fusing continuity features.

[0007] The first step is to pre-process the original three-channel color image, and obtain I P I M I H Three kinds of images.

[0008] The second step is to input each processed image into a feature encoder, gradually fuse low-level features while maintaining high-resolution feature representation; the features extracted by multi-layer convolution are spliced, and the features on all layers smaller than the current layer are weighted and summed to obtain the features of the current stage.

[0009] The third step is that the decoder splices the features extracted by multi-layer convolution, and the features on all layers greater than or equal to the current layer are weighted and summed to obtain the features of the next stage; finally, a depth map is output.

[0010] In the first step, I M I P I H The resolution is the same, I M The original image is obtained by the Resize operation, and the original image with a resolution of (375, 1242) is converted into an image with a resolution of h x w. P After obtaining the image with a resolution of h x w by the Resize operation, the original image is projected into a trapezoidal area; wherein the height of the trapezoid is the image width, the top side of the trapezoid is the image length, and the bottom side length of the trapezoid is controlled by a scale factor S P , and the ratio with the bottom side is randomly selected in the interval [0.7, 0.9]. H First, the image with a size of h H x w H is obtained by the Resize operation, and then the image with a resolution of h x w is obtained by the cropping operation; wherein the cropping window is controlled by a scale factor S H , and the ratio with the original image dynamically changes in the range of [0.5, 0.7].

[0011] Further, in the second step, the encoder contains four stages and four parallel subnets, the resolution of each subnet is reduced to half of the previous subnet, and the corresponding channel number is doubled; the convolution process of the first stage contains 4 residual units, each unit consists of a bottleneck with a width of 64, and a 3x3 convolution operation is used after the bottleneck to reduce the channel number of the feature map to the channel number of the input of the residual unit; the second, third, and fourth stages contain 1, 4, and 3 exchange units respectively, an exchange unit contains 4 residual units; after the 4 units, two 3x3 convolutions are included in the conversion of each different resolution; after the fourth stage, the output of the encoder is obtained, and a 3x3 convolution is used on it to obtain the input of the decoder.

[0012] Further, in the second step, the decoder is symmetrical to the encoder as a whole, and the specific differences include: the multi-scale fusion features of each stage will not fuse the features of the lower layers; after obtaining the multi-scale fusion features, a 3x3 convolution is needed, and the features after convolution can enter the next stage; when a feature is the highest layer of the stage, the layer feature only enters the next stage as input; the output of the decoder is output after a 1x1 convolution layer.

[0013] Further, in the third step, a combined loss function is used for network learning and training, the loss function includes minimum photometric error loss, smoothness loss, cross-scale consistency loss, and continuity loss; the expression of the loss function combination is: (where γμλ are hyperparameters)

[0014]

[0015] Since the objects in the photo cannot satisfy the Lambertian model as a diffuse reflection model, the loss function needs to consider the photometric error loss:

[0016]

[0017] Consider a case: when the same position in multiple frames of images is affected by an occlusion, using the average value to process is obviously lacking in rigor; in order to solve the problem caused by the occlusion, the minimum re-projection loss is used as the photometric error loss:

[0018]

[0019] In the loss function, in order to only allow the depth difference to be large in the area with large gray value difference, an edge smoothness loss is added:

[0020]

[0021] In the loss function, the continuity loss L MP The network is constrained to explicitly learn the continuity information of the scene:

[0022]

[0023] In the loss function, the cross-scale consistency loss L MH The network is constrained to explicitly learn the scale invariance information of the scene:

[0024]

[0025] The method of the present application projects the image according to the method of inverse perspective, according to the method of the present application, a region in the original image is selected as the projection content, which is projected into the same shape as the original image as the input, and the continuity information is implicitly added to the network. Inspired by the fact that the scale of the object in the image changes continuously with the depth, the present application believes that there is a certain correlation between the scale information and the continuity information. In addition to the original image input and the projected image input, an additional scale image is also added to help the network learn the scale information. The network framework of the present application adopts Dual HRNet, which is designed to better learn multi-scale feature information and has a relatively simple network structure. In terms of loss function, the present application proposes a joint continuity scale loss to explicitly constrain the network to learn the continuity information and scale information of the image.

[0026] The advantages of the present application are: on the basis of maintaining light weight, by combining scale invariance and continuity features, the accuracy of depth measurement is effectively improved; and due to its self-supervised characteristics, the requirement for the data set is low, making the training threshold low. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is the overall flowchart of a monocular depth estimation method fusing continuity features provided by an embodiment of the present application.

[0028] Figure 2 is the architecture diagram of the Dual HRNet network adopted by the embodiment of the present application.

[0029] Figure 3 is a self-supervised optimization target display diagram. DETAILED DESCRIPTION

[0030] The implementation method of the present invention will be specifically explained below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0031] Example 1

[0032] A monocular depth estimation method incorporating continuous features includes the following steps:

[0033] Step 1. As Figure 1 As shown, let the original image resolution be (375, 1242), denoted as h0×w0. Preprocessing it yields I from the original image. P I M I H There are three images, all with a resolution of (128, 416), corresponding to h × w. The specific process is as follows:

[0034] 1) I M The resolution of the image is obtained by resizing the original image, which is the same as the resolution of the network input image. The original image with a resolution of h0×w0 is converted into an image with a resolution of h×w.

[0035] 2)I P The image is obtained by resizing the original image to a resolution of h×w, and then projecting it onto a trapezoidal region. The projection operation is as follows: Figure 1 As shown by the red dashed line, the height of the trapezoid is the image width, the top side of the trapezoid is the image length, and the length of the base of the trapezoid is determined by the scaling factor S. P The ratio of the base to the bottom edge is randomly selected within the interval [0.7, 0.9].

[0036] 3)I H The original image is resized to obtain a resolution of h. H ×w H The image is then cropped to obtain an image with a resolution of h×w. Here, the resolution is h. H ×w H It is smaller than h0×w0, but larger than h×w, hence the term high resolution. The cropping operation is as follows: Figure 1 As shown by the blue dashed line, a resolution of h is selected. H ×w H A region within the image, where the cropping window is proportional to the original image by a scaling factor S. H The control dynamically varies within the range of [0.5, 0.7]. Since its final resolution is always h × w, I... H The resolution needs to be determined according to SH Dynamically changes in a continuous range.

[0037] Step 2. Each processed image is input into a network that has the feature of gradually fusing low-level features while maintaining a high-resolution feature representation, which is determined by its unique network structure, as follows:

[0038] As shown in Figure 2 , where e 4i represents the encoder feature output of the i-th layer, e ji represents the encoding of the i-th layer at the j-th stage, d ji represents the feature of the i-th layer after passing through the j-th stage of the decoder, represents the multi-scale fusion feature of the i-th layer at the j-th stage. The stage index j of the encoder and the decoder is independent of each other. Taking the first stage as an example:

[0039]

[0040] where CONV 3×3 represents a 3x3 convolution layer, CONV 1×1 represents a 1x1 convolution layer. In addition, μ() is an up-sampling operator, and [] is a summation operator for multi-scale feature fusion.

[0041] The above is the general architecture of the network, and some details of the encoder and the decoder are described as follows:

[0042] 1) The encoder contains four stages and four layers, and the resolution of each layer is reduced to half of the previous layer, while the number of channels is doubled. The first stage is processed by 4 residual unit convolutions, each containing a bottleneck with a width of 64, and then a 3x3 convolution operation is used to reduce the number of channels of the feature map to the number of channels of the input of the residual unit. The second, third, and fourth stages contain 1, 4, and 3 exchange units, respectively, each of which contains four residual units. After the four units, two 3x3 convolutions are added in the conversion of each different resolution. After the fourth stage, the output of the encoder is obtained, and a 3x3 convolution is used to obtain the input of the decoder.

[0043] 2) The structure of the decoder is similar to that of the encoder, aiming to enhance the multi-scale learning characteristics of the network. Overall, the decoder and the encoder present a symmetrical structure, but there are some differences, including:

[0044] 2.1 In each stage, multi-scale feature fusion does not involve features lower than the current layer;

[0045] 2.2To enter the next stage, the features after multi-scale fusion need to pass through a 3x3 convolution;

[0046] 2.3If a certain feature has become the highest layer of this stage, this layer feature can only be passed as input to the next stage, and the next stage will not contain this layer;

[0047] 2.4Before the decoder finally outputs the depth map, it will be processed through a 1x1 convolution layer.

[0048] 3) In addition to the main depth estimation module, there is also a pose network in the network to assist in depth estimation. Unlike the depth estimation module, the input image needs to be in chronological order, and the current frame image will obtain its relative pose through its previous and next frame images. From Figure 3 It can be seen that the pose network is independent of the depth network, and the relative pose relies on the correspondence between the pixels in the image.

[0049] Step 3. In order to better promote training and make continuity information and scale information better learned, a joint probability density loss function is adopted, which includes minimum photometric error loss, smoothness loss, cross-scale consistency loss and continuity loss. The loss function expression is: (where γμλ is a hyperparameter)

[0050]

[0051] In an ideal state, all objects in the photo emit light uniformly in all directions, but in reality, due to factors such as object surface material and angle, it is impossible to use a diffuse reflection model or Lambertian model to describe it, so the loss function needs to consider the photometric error loss, combined with the structural similarity index SSIM and L1 norm of two images I t and I s→t :

[0052]

[0053] Here I involves monocular self-supervised depth estimation content: using the geometric constraints of monocular video as supervision information for training. As shown in Figure 3 , in the training process, the target frame I t and the two frames adjacent to it in time I s (s∈{t-1,t+1}) are used to obtain the relative pose T t→s of the image. The meaning of pose is to help the network understand the change relationship between consecutive frames, that is, to help the network determine which objects have moved, which is based on the scene assumption of self-supervised monocular depth estimation: moving camera and static scene. Next, the camera pose Tt→s with the depth map D of the target frame t Obtain the re-projection image:

[0054] I s→t = I s <proj(D t , T t→s , K)> (3)

[0055] where < > is the differentiable bilinear sampling operator from, proj() is the projection operation, and K is the intrinsic camera parameter matrix assumed to be constant and known.

[0056] In addition, when obtaining the relative pose according to the adjacent frames, there can be a case that when the same position in multiple frames of images is affected by an occlusion, the continuity of the pose is affected. At this time, if the average value is simply used for processing, it is difficult to obtain an accurate depth estimation result. In order to solve the problem caused by the occlusion, the minimum re-projection loss is used as the photometric error loss:

[0057]

[0058] In addition, in the loss function, in order to allow the depth difference to be large only in the area with large gray value difference, an edge smoothing loss is added:

[0059]

[0060] where and are the horizontal and vertical gradients, is the average normalized inverse depth to prevent the shrinkage of the estimated depth.

[0061] In addition, by calculating the continuity loss L MR constrains the network to explicitly learn the continuity information of the scene:

[0062]

[0063] Since the input image is transformed and the pixels of the images containing different information are no longer in one-to-one correspondence, the output depth map needs to be inversely transformed to restore the one-to-one correspondence between the pixels, and then the loss is calculated. Where D represents the depth map obtained by the network, represents the depth map after the one-to-one correspondence between the pixels is restored.

[0064] Because I M is the same as the input resolution, that is, D M . For known image I P and IM From the pixel correspondence between them, we can obtain the pixel correspondence between their depth maps, which is D. P and The pixel correspondence between them. D P Backprojection back to the unprojected scale yields By comparing the depth maps generated under standard conditions with those generated with the addition of continuous information, the network's learning of continuous information was explicitly facilitated.

[0065] Furthermore, by calculating the cross-scale consistency loss L MH Scale information for constraining the explicit learning scenario of the network:

[0066]

[0067] and The acquisition method is the same, and the image I is known. H and I M From the pixel correspondence between them, we can obtain the pixel correspondence between their depth maps, which is D. H and The pixel correspondence between them. D H Backprojection back to the unprojected scale yields By comparing the depth maps generated under standard conditions with those generated with scale information added, the network's learning of scale information was explicitly facilitated.

[0068] Example 2

[0069] This invention provides a monocular depth estimation method that integrates continuous features, the overall process of which is as follows: Figure 1 As shown, it can be mainly divided into three stages: image preprocessing, network feature extraction and fusion, and the establishment and implementation of optimization objectives.

[0070] Step 1: Preprocess the original three-channel color image to obtain I. P I M I H Three images, of which I P Continuous information is introduced through projection. H Scale information is introduced through resizing and cropping.

[0071] Step 2: In this depth estimation model, each processed image will be fed into the concurrently running Dual HRNet network and pose network for processing.

[0072] 1) The feature encoder adopts four stages and four layers of structure, which can gradually fuse low-level features while maintaining high-resolution feature representation in the process. Specifically, as the number of convolution layers increases, the size of the feature map will gradually decrease. Before passing to the decoder, the previously extracted feature map and the low-resolution feature map after downsampling are fused to utilize more contextual information.

[0073] 2) For the decoder, it is similar in structure to the encoder, using splicing to fuse information of different scales. The final output is a depth map.

[0074] 3) In terms of pose network, multiple consecutive frames of images are input into the network, so that the relative poses between them can be obtained. This helps to improve the accuracy of depth estimation by learning the motion relationship between adjacent frames to infer the depth value of each pixel point.

[0075] Step 3: Establishment and optimization of optimization objectives. By combining multiple optimization objectives, the loss is continuously reduced in training to achieve the realization of the final goal. The specific optimization objectives are as follows:

[0076] 1) As shown in Figure 3 , the depth map D t output by t frames, the relative pose T t→t+1 between t and t+1 frames, and the camera parameters K are used to obtain the projected image I t+1→t . Comparing the projected image I t+1→t with the t+1 frame image I t+1 , the photometric error loss is calculated, which is optimization objective 1.

[0077] 2) To allow only large depth differences in areas with large gray value differences, the edge smoothness loss is calculated from the input image I t of t frames and the output depth map D t of t frames, which is optimization objective 2.

[0078] 3) By comparing images containing different information, the network learning of corresponding information is explicitly constrained. The continuity loss is calculated from the output depth map D P containing continuity information images and the output depth map D M of the reference image, which is optimization objective 3; the scale loss is calculated from the output depth map D H containing scale information images and the output depth map D M of the reference image, which is optimization objective 4.

[0079] To sum up, the embodiment of the present application provides a monocular depth estimation method fusing continuity features, first, data preprocessing is performed on the image, and an image containing continuity information and an image containing scale information are introduced; further, feature extraction and feature fusion are performed on the input image to obtain a final depth map; during network training, the loss function adopts a combined form, and the photometric error and the smoothing error of three different forms of images are calculated, and the cross-scale depth consistency loss and the continuity loss are added.

[0080] The above embodiment is the preferred embodiment of the present application, but the embodiment of the present application is not limited to the above embodiment, and any change, substitution, combination, simplification made without departing from the spirit and principle of the present application should be an equivalent replacement.

Claims

1. A monocular depth estimation method fusing continuity features, characterized in that, The method comprises the following steps: First step: pre-process the original three-channel color image, get I P I M I H three images; Second step: input each processed image into a feature encoder to gradually fuse low-level features while maintaining a high-resolution feature representation; the features extracted by multi-layer convolution are spliced, and the features of all layers less than the current layer are weighted and summed to obtain the features of the current layer; the feature encoder comprises four stages and four parallel subnets, the resolution of each subnet is reduced to half of the previous subnet, and the number of corresponding channels is doubled; the convolution process of the first stage comprises four residual units, each unit comprises a bottleneck with a width of 64, and a 3*3 convolution operation is used after the bottleneck to reduce the number of channels of the feature map to the number of channels of the input of the residual unit; The second, third and fourth stages comprise 1, 4 and 3 exchange units respectively, and each exchange unit comprises four residual units; after the four units, two 3*3 convolution operations are included in the conversion of each different resolution; after the fourth stage, the output of the feature encoder is obtained, and a 3*3 convolution operation is performed on the output to obtain the input of the decoder; The decoder is symmetrical to the feature encoder as a whole, and the specific differences include: multi-scale fusion features of each stage Features of layers lower than itself are not fused; after obtaining the multi-scale fusion features, a 3x3 convolution is needed, and the features after convolution can enter the next stage; when a feature is the highest layer of the stage, the layer feature only enters the next stage as input; the output of the decoder is output after a 1x1 convolution layer. Third step: the decoder splices the features extracted by multi-layer convolution, and the features of all layers greater than or equal to the current layer are weighted and summed to obtain the features of the next stage; finally, the depth map is output; a combined loss function is used for network learning and training, and the loss function comprises a minimum photometric error loss, a smoothing loss, a cross-scale consistency loss and a continuity loss; the expression of the combined loss function is: Wherein, γ, μ, λ are hyperparameters; Since the objects in the photo cannot satisfy the Lambertian model, the loss function needs to consider the photometric error loss: Consider a case: in the same position in multiple frames of images, one frame of image is affected by an occlusion, and using the average value to process obviously lacks rigor; in order to solve the problem caused by the occlusion, the minimum re-projection loss is used as the photometric error loss: In the loss function, in order to allow the depth difference to be large only in the area with large gray value difference, an edge smoothing loss is added: In the loss function, the continuity loss L MP The constraint network explicitly learns the continuity information of the scene: In the loss function, the cross-scale consistency loss L is calculated MH The constraint network explicitly learns the scale-invariant information of the scene:

2. The monocular depth estimation method of fusing continuity features according to claim 1, wherein: In the first step, I M I P I H The resolution is the same, I M By Resize operation from the original map, the original resolution is (375, 1242) to the resolution of h x w map;I P After obtaining the resolution of h x w map by Resize operation, the original map is projected to the trapezoidal area; where the height of the trapezoid is the image width, the top side of the trapezoid is the image length, and the bottom side length is controlled by the scale factor S P The ratio with the bottom side is randomly selected within the interval [0.7, 0.9];I H First, get the image with size h H ×w H by Resize operation, and then get the image with resolution h x w by cropping operation; where the cropping window is dynamically changed within the range of [0.5, 0.7] by the scale factor S H Control.

Images