Self-supervised monocular depth estimation method and system of adaptive depth-to-size convolution

Abstract

This disclosure relates to the field of image processing technology and proposes a self-supervised monocular depth estimation method and system based on adaptive depth-to-size convolution. The feature obtained by adaptive depth-to-size convolution solves the problem of scale blurring at different depths between different frames of a video. Based on the relationship between scene depth and target scale, it explicitly processes the changes in object size caused by scene depth changes in consecutive video frames, thereby improving the accuracy of scene structure extraction. The feature fusion module DcS-F based on size information efficiently fuses adaptive size features and traditional convolution features, extracting important scene information contained in the features of different objects within the same frame at the same convolution scale, further improving the accuracy of depth map estimation.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to a self-supervised monocular depth estimation method and system based on adaptive depth-to-size convolution. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Depth estimation is a fundamental and crucial task in 3D vision. High-quality depth maps can provide useful scene information to assist various advanced tasks, including augmented reality, autonomous driving, and 3D reconstruction. With the development of deep learning, deep learning-based monocular depth estimation has gradually emerged and received widespread attention from academia and industry. It is mainly divided into supervised learning methods that target ground truth depth maps and self-supervised learning methods that target view synthesis and reconstruction errors. This disclosure mainly focuses on self-supervised depth estimation methods based on monocular videos.

[0004] The inventors discovered in their research that the size of objects constantly changes in consecutive video frames captured using a moving camera. Existing monocular depth estimation methods simply use convolutional networks with fixed-size convolutions and fixed-scale processing to handle objects of different sizes, inevitably affecting the extraction of scene structure. Furthermore, while features obtained solely through adaptive depth-to-size convolutions solve the problem of scale blurring at different depths between different video frames, they neglect the important scene information contained in the features of different objects within the same frame at the same traditional convolutional scale. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure proposes a self-supervised monocular depth estimation method and system based on adaptive depth-to-size convolution, which can accurately extract the depth map of an image.

[0006] To achieve the above objectives, the present disclosure adopts the following technical solution:

[0007] One or more embodiments provide a self-supervised monocular depth estimation method based on adaptive depth-to-size convolution, comprising the following steps:

[0008] Obtain the current frame image to be predicted;

[0009] The current frame image is encoded in multiple levels using a multi-level encoder to obtain multi-scale features, and each level of encoded features is transmitted to the corresponding level of decoder for decoding.

[0010] The first level depth map is obtained through the first decoder. The second and subsequent decoders are replaced by depth-to-size feature decoding modules. These modules receive the decoded output feature map from the previous level decoder and the encoded feature map transmitted by the encoder at this level. They perform depth-to-size convolution with kernels that adapt to the image depth value, and feature fusion based on size information to generate a depth map of the corresponding resolution.

[0011] The depth map output by the final-level depth-to-size feature decoding module is the depth map obtained by predicting the current frame image.

[0012] One or more embodiments provide a self-supervised monocular depth estimation system with adaptive depth-to-size convolution, comprising:

[0013] The image acquisition module is used to acquire the current frame image to be predicted;

[0014] The multi-level coding unit is used to perform multi-level coding on the current frame image to obtain multi-scale features, and transmit each level of coded features to the corresponding level of decoder for decoding.

[0015] The multi-level decoding unit is used to obtain the first-level depth map through the first decoder. The second and subsequent decoders are replaced by depth-to-size feature decoding modules, which receive the decoded output feature map of the previous level decoder and the encoded feature map transmitted by the encoder of this level, perform depth-to-size convolution with the convolution kernel adaptively changing with the image depth value, and feature fusion based on size information to generate a depth map of the corresponding resolution.

[0016] The depth map output unit outputs the depth map through the last-stage depth-to-size feature decoding module, which is the depth map predicted from the current frame image.

[0017] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0018] This disclosure solves the problem of scale blurring at different depths between different frames of a video by using features obtained through adaptive depth-to-size convolution. Based on the relationship between scene depth and target scale, it explicitly processes the changes in object size caused by scene depth changes in consecutive video frames, thereby improving the accuracy of scene structure extraction. The feature fusion module DcS-F based on size information efficiently fuses adaptive size features and traditional convolution features, extracting important scene information contained in the features of different objects within the same frame at the same convolution scale, further improving the accuracy of depth map estimation.

[0019] The advantages of this disclosure, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0020] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.

[0021] Figure 1 This is a diagram showing the relationship between object size and scene depth in Embodiment 1 of this disclosure;

[0022] Figure 2 This is a multi-scale feature size-aware fusion map of Embodiment 1 of this disclosure;

[0023] Figure 3 This is an image showing the effect of adaptive depth-to-size convolution in Embodiment 1 of this disclosure;

[0024] Figure 4 This is a diagram of a feature fusion module based on size information according to Embodiment 1 of this disclosure;

[0025] Figure 5 This is the overall framework diagram of the self-supervised monocular depth estimation network of Embodiment 1 of this disclosure;

[0026] Figure 6 This is a diagram of the depth-to-size feature decoding module of Embodiment 1 of this disclosure. Detailed Implementation

[0027] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0029] It should be noted that the terminology used herein is for descriptive purposes only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0030] Example 1

[0031] In one or more of the technical solutions disclosed in the embodiments, such as Figures 1 to 6 As shown, a self-supervised monocular depth estimation method based on adaptive depth-to-size convolution is presented. First, the improvement process of the self-supervised monocular depth estimation network based on adaptive depth-to-size convolution is explained as follows:

[0032] Step 1: Based on the scene depth values ​​of different objects, adaptively adjust the receptive field of the convolution kernel, construct a depth-to-size convolution module, extract traditional convolution features at a fixed scale, and generate adaptive convolution features through scene depth.

[0033] Step 2: Construct a size-based feature fusion module DcS-F that includes multi-attention, and fuse adaptive convolutional features and traditional convolutional features according to the size difference map of the input image;

[0034] Step 3: Concatenate the depth-to-size convolution module with the size-based feature fusion module to construct the depth-to-size feature decoding module. Use the concatenated network as the processing module of the depth-to-size feature decoding module.

[0035] Step 4: Use the depth-to-size feature decoding module as the decoder of the self-supervised monocular depth estimation network to predict the depth map of the target frame image.

[0036] In step 1, the relationship between the size of the object and the depth of its scene is first determined.

[0037] This study investigates the relationship between object size and scene depth, providing a theoretical basis for adaptive depth-scale convolution.

[0038] like Figure 1 As shown, under the ideal imaging principle of a pinhole camera, the actual size S of the object in a three-dimensional scene can be obtained. r The relationship between S and its size in the image v :

[0039]

[0040] Where D and f represent scene depth and camera focal length, respectively;

[0041] When the object's depth changes from D1 to D2, the size of the same object in the image plane changes as follows:

[0042]

[0043] Among them, S v1 ,S v2 This indicates the size of the object in the image at the corresponding depth.

[0044] The size of an object in an image is inversely proportional to its scene depth. The goal of this embodiment is to make the size of the convolution kernel adaptively change with the size of the object when performing convolution to extract features.

[0045] Optionally, the average depth D of a commonly used dataset for depth estimation tasks can be used. rAnd the commonly used network convolution kernel size K r As the reference depth value and reference depth kernel size, for any depth value D, the corresponding kernel size K is... d It can be calculated using the following formula:

[0046]

[0047] In step 1, the size of the convolution kernel is determined based on the scene depth value of different objects, thereby adaptively adjusting the receptive field of the convolution kernel and constructing the depth-to-size convolution module DcSConv to generate adaptive convolution features through scene depth.

[0048] To obtain adaptive features at different scales based on scene depth, a direct approach is to first extract multi-scale features and then fuse them based on the scale differences resulting from depth transformation. For example... Figure 2 The example shown first uses three parallel branches with different kernel sizes to obtain multi-scale features. Specifically, each branch consists of two convolutional blocks with the same kernel size. The three branches use kernel sizes of 1×1, 3×3, and 5×5 respectively, and are activated using the Rectified Linear Activation Function (ReLU). Then, the feature weights of each branch are calculated based on the difference between the estimated object size and the actual kernel size used. Specifically, a Gaussian distance function is used to measure the distance between the depth transformation scale and the kernel size used in different branches, and then softmax is used for normalization to obtain the weight 'a' of each branch. i Ultimately, adaptive features are generated by fusing multi-scale features through branch weights.

[0049]

[0050] The above method first extracts multi-scale features and then fuses them to obtain adaptive features, which inevitably introduces size errors and affects scene feature extraction.

[0051] To address the issue of size error introduced during the adaptive feature extraction process, this embodiment uses depth-to-size convolution to directly generate adaptive convolutional features based on scene depth.

[0052] In this embodiment, as Figure 3 As shown, the constructed depth-to-size convolution module DcSConv realizes depth-to-size convolution with the convolution kernel adaptively changing with the image depth value. Specifically, for the image to be processed, a convolution branch is set for different image depths. Each convolution branch includes an adaptive convolution module and a standard convolution module. For the adaptive convolution module, a bilinear interpolation sampling position determination module is also set to determine the sampling position during the convolution process.

[0053] In some embodiments, the standard convolution module, based on a fixed scale and a determined sampling position P, d Perform convolution to obtain the traditional convolution feature F. c ;

[0054] The standard convolution module employs two-dimensional standard convolution and uses a sliding window operation. It moves the convolution kernel sequentially across the input feature F according to the stride until the kernel covers the entire input feature. For each location P0 on the input feature map, the two-dimensional standard convolution typically involves the following two steps:

[0055] (1) Use a regular grid R as the receptive field to perform sampling operations on the input features;

[0056] (2) Add the sampled feature value to the corresponding convolution kernel weight w to obtain the output feature F at the corresponding position. c :

[0057]

[0058] Among them, R c Representing local regular meshes R, P r This represents the sampling position in R, for a convolution kernel size of K. r For a value of 3, R is as follows:

[0059] R C ={(-1,-1),(-1,0),…,(0,1),(1,1)}

[0060] In some embodiments, the adaptive convolution module determines the size of the convolution kernel based on the size of the object and the determined sampling position P. d Adaptive convolution is performed on the input image to obtain the adaptive size feature F. DcSC The size of the convolution kernel is determined according to Formula 3. In the figure, DcS represents the relationship between the object size and the scene depth.

[0061] The adaptive convolution module uses the same sliding window operation as standard 2D convolution. During convolution, the kernel size adaptively changes based on the size of objects in the image; that is, the receptive field of the kernel adjusts adaptively according to the scene depth values ​​of different objects. For the corresponding position P0, the output feature F... DcSC (P0):

[0062]

[0063] Among them, R DcSC Represents the convolution kernel K d ×K d The receptive field of depth-to-size convolution, P d R represents DcSC The sampling location in the sample.

[0064] Similar to standard 2D convolution, depth-to-size convolution uses the same sampling strategy and number of sampling points. For example, a standard 3×3 convolution uses 9 sampling points: the center point, corner points, and edge center points. Therefore, R... DcSC This can be represented as:

[0065]

[0066] The bilinear interpolation sampling position determination module is configured to use bilinear interpolation to determine the depth-to-size convolutional receptive field R. DcSC Sampling position P d It is used to determine the sampling position during the convolution process of the adaptive convolution module.

[0067] Although adaptive depth-to-size convolution DcSConv solves the problem of scale blurring at different depths between different frames, the features of different objects within the same frame at the same convolution scale also contain important information about the scene.

[0068] A further technical solution involves designing a feature fusion module DcS-F based on size information to efficiently fuse adaptive size features F. DcSC And traditional convolutional features F c .

[0069] like Figure 4 As shown, the size-based feature fusion module DcS-F includes a size-based channel attention module and a spatial attention module, which are used to implement size-based feature fusion.

[0070] Specifically, the feature fusion module DcS-F, based on size information, fuses adaptive size features F. DcSC And traditional convolutional features F c It is configured to perform the following procedure:

[0071] Step 21: For the adaptive size feature F to be fused DcsC And traditional convolutional features F c The features are then concatenated to obtain the initial feature U.

[0072] Step 22: Perform channel attention operations on the channel attention block based on size information to determine the global channel weight M. c The initial feature U is enhanced to obtain the channel attention enhanced feature U. v =M c ⊙U;

[0073] Global channel weight M c The method for determining this is as follows:

[0074] 22.1) The normalized size difference map S generated from the scene depth map corresponding to the input color image. U ;

[0075] 22.2) Dimension difference diagram S U After concatenation with the initial feature U, fusion is achieved by performing a convolution operation. A standard 1×1 convolutional block can be used for fusion.

[0076] 22.3) After performing global average pooling on the features fused by convolution in step 22.2), two standard convolutional blocks are used for convolution operations to obtain the global channel weights M after activation. c ;

[0077] The above process can be represented as:

[0078] M c =f 1×1 (AvgPool(f 1×1 (C[U,S U ])))

[0079] Where C[.] represents the feature concatenation operation, and f 1×1 This represents a 1×1 standard convolutional block.

[0080] Step 23: Based on the size information of the spatial attention block, perform spatial attention operation and determine the spatial attention weight M. s ;

[0081] Specifically, spatial attention weight M s The method for determining this includes the following process:

[0082] 23.1) The normalized size difference map S generated from the scene depth map corresponding to the input color image. U ;

[0083] 23.2) Enhance channel attention feature U v Diagram S with size difference U After splicing, a convolution operation is performed to achieve fusion, which can be achieved using a 1×1 standard convolutional block;

[0084] 23.3) Perform channel-average pooling and channel-max pooling on the convolutional fused features from 23.2). Concatenate the pooled features from the two layers and perform standard convolution. After activation, obtain the spatial attention weights M. s ;

[0085] Specifically, the two layers of features are concatenated and then processed using a 7×7 standard convolutional block, finally activated by a Sigmoid activation function to obtain M. s ;

[0086] The above process can be represented as:

[0087] F′=f 1×1 (C[U v ,S U ])

[0088] M s =f 7×7 (C[AvgPool(F′),MaxPool(F′)])

[0089] Among them, f 7×7 Represents a 7×7 standard convolutional block;

[0090] Step 24: Based on spatial attention weights M s For adaptive size feature F DcSC Enhancement is performed by combining the enhanced features with the adaptively sized features and the traditional convolutional features F. c Concatenate, and use channel attention weight M c Enhancement is performed by concatenating the enhanced features with the initial features U, and the resulting fused features F are then processed to obtain the final fused features F. s .

[0091] Spatial attention weight M s It will only affect F. DcSC To distinguish traditional convolutional features F based on spatial importance c The final feature F is obtained by processing the weighted features through a channel and spatial attention module based on size information and a 1×1 standard convolutional block. s :

[0092] F s =f 1×1 (M c ⊙C[M s ⊙F DcSC ,F c ]+U)

[0093] Here, ⊙ represents element-wise dot product, and the initial feature U is added in a residual connection manner to enhance the feature.

[0094] Further technical solutions include constructing a depth-to-size feature decoding module.

[0095] Given a monocular video, I t D represents the target frame. t I represents the corresponding depth map to be predicted. s The source frame is typically represented by two adjacent frames, s = {t-1, t+1}. The goal of self-supervised monocular depth estimation is to estimate the depth from the target frame I. t Predicted depth map D tThe network is supervised by the reconstruction error of the view synthesis between the source frame and the predicted depth map. Furthermore, the relative camera pose between the target frame and the source frame needs to be estimated to assist in view synthesis. Therefore, the overall network framework requires a depth estimation network module and a pose prediction module.

[0096] like Figure 5 As shown, existing depth estimation network modules typically employ an encoder-decoder structure. For the input target frame, multi-scale features are first obtained through a depth encoder with a residual network at its core. Then, spatial resolution is gradually restored through five decoders, and skip connections are used to facilitate the transfer of gradients and information throughout the model. Finally, the inverse depth map generated during the step-by-step decoding process is continuously upsampled to the original input resolution, and the reconstruction error loss is calculated at the highest resolution.

[0097] like Figure 5 As shown, the lower half is the pose prediction network, which predicts the target frame I. t and source frame I s After concatenation, the data is input into a feature extraction network with a residual network as its core, and then the camera pose change T is estimated through three convolutional layers. t→s Then, the target frame is reconstructed at the input image size. The reconstruction process of the target frame is the viewpoint synthesis process W(,), as shown in the following equation:

[0098] p′=KT t→s D t K -1 p

[0099] I s→t =W(I s ,p)

[0100] For each pixel location of the target frame p, the depth map D is used. t The relative pose T between the source and target video frames t→s Given the camera intrinsic parameter K, calculate the corresponding source frame position coordinate p′, and then assign the obtained source frame position pixel value to the corresponding target frame position pixel value to obtain the reconstructed target frame I. s→t Furthermore, the reconstruction error of the target frame can be used to construct the supervision signal L. p .

[0101] The main improvement in this embodiment is to the depth estimation network module.

[0102] The depth-to-size feature decoding module in this embodiment does not consider the specific framework details of the network. As a plug-and-play module, it can be used in all encoder-decoder frameworks.

[0103] The first decoder in the encoder-decoder framework remains unchanged, and a first-level depth map is obtained through the first decoder. The subsequent four decoders are replaced with depth-to-size feature decoding modules, where the required high-resolution size map can be a depth map estimated stepwise from its previous levels or a depth map estimated using a pre-trained model.

[0104] The depth-to-size feature decoding module includes processing of features from the previous level decoder, features from the corresponding level encoder, and fused features. For each feature to be decoded, the size-based feature fusion module DcS-F is used to efficiently fuse DcSConv features and traditional convolutional features.

[0105] The depth-to-size feature decoding module includes a first processing module, a second processing module, and a third processing module. Each processing module includes a cascaded depth-to-size convolution module DcSConv and a feature fusion module DcS-F based on size information.

[0106] The first processing module processes the decoding features of the previous level decoder, including the depth-to-size convolution module DcSConv, a standard convolution module, and a size-based feature fusion module DcS-F; the depth-to-size convolution module DcSConv obtains the adaptive convolutional feature f. DcSC The standard convolutional module obtains the traditional convolutional features f respectively. Conv ;

[0107] Specifically, such as Figure 6 As shown, in this embodiment, the standard convolutional module of the first processing module uses a 3×3 standard convolutional block. The depth-to-size convolutional module DcSConv and the 3×3 standard convolutional block are used to process the low-resolution decoding feature F of the previous level. l_dec Then DcS-F is used to fuse and obtain the first-level size-aware decoding features F. s_fdec The above process can be represented as:

[0108] F s_fdec =f DcS-F (C[f DcSC ,f Conv (F l_dec )],S l_dec )

[0109] Where C[f DcSC ,f Conv [)] indicates the feature concatenation process of the proposed DcSConv and the 3×3 standard convolutional block (3×3conv); S l_dec This represents a low-resolution image used to guide DcS-F feature fusion.

[0110] The second processing module is used to process the high-resolution encoded features F output by the current encoder. h_enc The processing includes a cascaded depth-to-size convolution module DcSConv and a size-based feature fusion module DcS-F.

[0111] The second processing module is used for high-resolution encoding of feature F. h_enc Decoding to obtain size-aware encoded features F s_enc :

[0112] F s_enc =f DcS-F (C[f DcSC ,f Conv (F h_enc )],S h_dec )

[0113] Among them, S h_dec This represents a high-resolution dimensional image, which can be represented by S. l_dec It can be obtained by upsampling or downsampling of pre-trained high-resolution depth maps.

[0114] The third processing module is used to fuse the processing results of the first and second processing modules. It includes a stitching unit, a depth-to-size convolution module DcSConv, a standard convolution module, and a feature fusion module DcS-F based on size information.

[0115] The splicing unit is used to combine the first-level size-aware decoding features F s_fdec Upsampling to obtain F s_u_dec and size-aware encoding feature F s_enc The splicing results in the splicing feature F″;

[0116] F″=C[F s_u_dec ,F s_enc ]

[0117] The concatenated features are used as input, and the adaptive convolutional features f are obtained by passing them through the depth-to-size convolution module DcSConv. DcSC Traditional convolutional features f are obtained through standard convolutional modules. Conv (F″);

[0118] The size-based feature fusion module DCS-F adapts the convolutional feature f DcSC and traditional convolutional features f Conv (F″) is fused to obtain the high-resolution decoding feature F. s_dec :

[0119] F s_dec =f DcS-F (C[f DcSC ,f Conv (F″)],Sh_dec )

[0120] Ultimately, the decoding feature F s_dec Feature channels are compressed using 1×1 standard convolutional blocks, which are then used to generate depth maps of the corresponding resolution.

[0121] In step 4, the depth-to-size feature decoding module is used as a high-level decoder of the self-supervised monocular depth estimation network to predict the depth map of the target frame image.

[0122] Based on the aforementioned self-supervised monocular depth estimation network with adaptive depth-to-size convolution, this embodiment provides a self-supervised monocular depth estimation method with adaptive depth-to-size convolution. The process of predicting the depth map is as follows:

[0123] Step S1: Obtain the current frame image to be predicted;

[0124] Step S2: Perform multi-level encoding on the current frame image based on the multi-level encoder to obtain multi-scale features, and transmit each level of encoded features to the corresponding level decoder for decoding;

[0125] Step S3: Obtain the first-level depth map through the first decoder. Replace the second-level and above decoders with the depth-to-size feature decoding module. Receive the decoded output features of the previous level decoder and the encoded features transmitted by the encoder at this level, perform depth-to-size convolution and feature fusion based on size information, and generate a depth map of the corresponding resolution.

[0126] Step S4: The depth map output by the final-level depth-to-size feature decoding module is the depth map obtained by predicting the current frame image.

[0127] In this embodiment, the feature obtained by adaptive depth-to-size convolution solves the problem of scale blurring at different depths between different frames of the video. Based on the relationship between scene depth and target scale, the changes in object size caused by scene depth changes in consecutive video frames are explicitly processed, which improves the accuracy of scene structure extraction. The feature fusion module DcS-F based on size information efficiently fuses adaptive size features and traditional convolution features, extracting important scene information contained in the features of different objects within the same frame at the same convolution scale, which further improves the accuracy of depth map estimation.

[0128] To illustrate the effectiveness of the method in this embodiment, a depth estimation experiment was conducted, and the degree estimation performance was significantly improved. The results are shown in Table 1 below. The method in this embodiment was applied to experiments on the KITTI dataset on CADepth.

[0129] Table 1

[0130]

[0131] In Table 1, "ours" refers to the method of this embodiment, while the remaining methods are existing self-supervised monocular depth estimation methods.

[0132] Monodepth2[7] is a self-supervised monocular depth estimation network proposed by Godard et al. It uses the minimum source frame reconstruction error instead of the average error and adaptive mask to deal with the depth prediction problems of object occlusion and moving objects respectively.

[0133] Lee et al

[42] Lee et al. emphasized the importance of forward projection of moving objects, and established photometric loss consistency and geometric loss consistency based on the semantic information of instance segmentation, so as to add additional self-supervised signals to the background and target regions.

[0134] HRDepth

[43] Liu et al. provided a high-resolution monocular depth evaluation network and demonstrated that predicting more accurate object boundaries can improve depth prediction performance.

[0135] Wang et al

[24] proposed a size geometric loss function based on point cloud alignment constraints to provide size consistency constraints.

[0136] CADepth

[14] is a self-supervised monocular depth estimation network based on attention mechanism feature enhancement proposed by Yan et al. It includes a structure awareness module and a detail enhancement module to enhance depth features globally and locally, respectively.

[0137] The backbone network uses ResNet18 and ResNet50 residual networks respectively. AbsRel is the absolute relative error, SqRel is the squared relative error, RMSE is the square root mean error, RMSE(log) is the exponential square root mean error, and δ is the accuracy threshold. The specific calculation formulas are as follows:

[0138]

[0139]

[0140]

[0141]

[0142]

[0143] in, These represent the estimated depth value and the ground truth value, respectively. i represents the pixel index, N represents the number of pixels in each depth map, and δ represents the precision threshold, typically set to 1.25. 2 1.25 3.

[0144] As can be seen from Table 1, the experimental results using the method of this embodiment are significantly improved.

[0145] Example 2

[0146] Based on Example 1, this example provides a self-supervised monocular depth estimation system with adaptive depth-to-size convolution, including:

[0147] The image acquisition module is used to acquire the current frame image to be predicted;

[0148] The multi-level coding unit is used to perform multi-level coding on the current frame image to obtain multi-scale features, and transmit each level of coded features to the corresponding level of decoder for decoding.

[0149] The multi-level decoding unit is used to obtain the first-level depth map through the first decoder. The second and subsequent decoders are replaced by depth-to-size feature decoding modules, which receive the decoded output feature map of the previous level decoder and the encoded feature map transmitted by the encoder of this level, perform depth-to-size convolution with the convolution kernel adaptively changing with the image depth value, and feature fusion based on size information to generate a depth map of the corresponding resolution.

[0150] The depth map output unit outputs the depth map through the last-stage depth-to-size feature decoding module, which is the depth map predicted from the current frame image.

[0151] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0152] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0153] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A self-supervised monocular depth estimation method of adaptive depth-to-size convolution, characterized in that, The method comprises the following steps: obtaining a current frame image to be predicted; multi-level encoding the current frame image based on a multi-level encoder to obtain multi-scale features, and transmitting each level of encoded features to a corresponding level of decoder for decoding; obtaining a first level of depth map through a first decoder, and replacing the second level and above decoders with a depth-to-size feature decoding module, receiving the decoding output feature map of the previous level of decoder and the encoded feature map transmitted by the current level of encoder, performing depth-to-size convolution with adaptive change of convolution kernel according to the image depth value, and performing feature fusion based on size information to generate a depth map with corresponding resolution; wherein the adaptive change of the convolution kernel is inversely proportional to the image depth value; the depth-to-size feature decoding module comprises a first processing module, a second processing module and a third processing module, each processing module comprising a cascaded depth-to-size convolution module and a feature fusion module based on size information; the depth-to-size convolution module is configured to obtain adaptive size features and traditional convolution features through depth-to-size convolution operation with adaptive change of convolution kernel according to the image depth value; and the feature fusion module based on size information fuses the features after convolution operation based on size information; the feature fusion module based on size information is used to fuse the adaptive size features and the traditional convolution features, and is configured to perform the following processes: splicing the adaptive size features and the traditional convolution features to be fused to obtain initial features; performing channel attention operation through a channel attention block based on size information to determine global channel weights and enhance the initial features to obtain channel attention enhanced features; The spatial attention block based on the size information performs a spatial attention operation to determine spatial attention weights ; enhancing the adaptive size features based on the spatial attention weights, splicing the enhanced adaptive size features and the traditional convolution features, enhancing the spliced features through the channel attention weights, connecting the enhanced features with the initial features, and processing to obtain the final fused features; the depth map output by the last level of depth-to-size feature decoding module is the depth map obtained by predicting the current frame image.

2. The self-supervised monocular depth estimation method with adaptive depth-to-size convolution according to claim 1, wherein: the first processing module is used to process the decoding features of the previous level of decoder, and comprises a depth-to-size convolution module, a standard convolution module and a feature fusion module based on size information; the second processing module is used to process the high-resolution encoded features output by the current level of encoder, and comprises a cascaded depth-to-size convolution module and a feature fusion module based on size information; the third processing module is used to fuse the processing results of the first processing module and the second processing module, and comprises a splicing unit, a depth-to-size convolution module, a standard convolution module and a feature fusion module based on size information connected in sequence.

3. The self-supervised monocular depth estimation method of claim 1, wherein, the depth-to-size convolution module is specifically configured to set a convolution branch for different image depths for the image to be processed, each convolution branch comprises an adaptive convolution module and a standard convolution module, and a bilinear interpolation sampling position determination module is further arranged for the adaptive convolution module to determine the sampling position in the convolution process. The standard convolution module is configured to perform convolution by using a sliding window operation according to a fixed scale and a determined sampling position, to obtain a traditional convolution feature; The adaptive convolution module is configured to determine the size of a convolution kernel according to the size of an object, to perform adaptive convolution by using a sliding window operation on an input image according to a determined sampling position, and to obtain an adaptive size feature.

4. The self-supervised monocular depth estimation method of claim 1, wherein a global channel weight is determined by: generating a normalized size difference map from a scene depth map corresponding to the input color image; performing convolution operation on the size difference map and initial features to realize fusion; and performing global average pooling operation on the fused features, and then performing convolution operation by using two standard convolution blocks to obtain the global channel weight after activation.

5. The self-supervised monocular depth estimation method of claim 1, wherein a spatial attention weight is determined by: generating a normalized size difference map from a scene depth map corresponding to the input color image; performing convolution operation on the channel attention enhanced features and the size difference map to realize fusion; and performing channel average pooling and channel maximum pooling on the fused features, and then performing standard convolution operation on the two pooled features after being spliced, to obtain the spatial attention weight after activation. The image acquisition module is configured to acquire a current frame image to be predicted. The multi-level encoding unit is configured to perform multi-level encoding on the current frame image to obtain multi-scale features, and transmit each level of encoded features to a corresponding level of decoder for decoding. The multi-level decoding unit is configured to obtain a first level of depth map by using a first decoder, and replace the second level and above decoders with depth-to-size feature decoding modules. The depth map output unit is configured to output the depth map output by the last level of depth-to-size feature decoding module as the depth map predicted by the current frame image. The adaptive change of the convolution kernel is inversely proportional to the image depth value. The depth-to-size feature decoding module includes a first processing module, a second processing module, and a third processing module. The depth-to-size convolution module is configured to perform depth-to-size convolution operation based on the adaptive change of the convolution kernel with the image depth value, to obtain adaptive size features and traditional convolution features. The feature fusion module based on size information is configured to fuse the adaptive size features and the traditional convolution features by performing the following processes:

6. A self-supervised monocular depth estimation system of adaptive depth-to-size convolution, characterized in that, The adaptive size features and the traditional convolution features to be fused are spliced to obtain initial features; ​ ​ ​ ​ ​ ​ ​ ​ ​ The channel attention block based on size information performs a channel attention operation, determines global channel weights, and enhances the initial features to obtain channel attention enhanced features; a spatial attention block based on the size information, performing a spatial attention operation to determine spatial attention weights ; The adaptive size features are enhanced based on the spatial attention weights, the enhanced adaptive size features are spliced with the traditional convolution features, and the enhanced features are enhanced through the channel attention weights, and the enhanced features are connected to the initial features to obtain the final fusion features after processing.

7. The self-supervised monocular depth estimation system of claim 6, wherein: The first processing module is configured to process the decoding features of the previous level decoder, including a depth-to-size convolution module, a standard convolution module, and a feature fusion module based on size information; The second processing module is configured to process the high-resolution encoding features output by the current level encoder, including a cascaded depth-to-size convolution module and a feature fusion module based on size information; The third processing module is configured to fuse the processing results of the first processing module and the second processing module, including a concatenation unit, a depth-to-size convolution module, a standard convolution module, and a feature fusion module based on size information.

8. The self-supervised monocular depth estimation system of claim 6, wherein: The depth-to-size convolution module is configured to set a convolution branch for different image depths for the image to be processed, each convolution branch includes an adaptive convolution module and a standard convolution module, and the adaptive convolution module is further provided with a bilinear interpolation sampling position determination module for determining the sampling position in the convolution process; The standard convolution module performs convolution using a sliding window operation according to the fixed scale and the determined sampling position to obtain traditional convolution features; The adaptive convolution module determines the size of the convolution kernel according to the size of the object, and performs adaptive convolution on the input image using a sliding window operation according to the determined sampling position to obtain adaptive size features.

Images