Depth estimation method, apparatus and device, and storage medium and computer program product

Abstract

Disclosed in the present application are a depth estimation method, apparatus and device, and a storage medium and a computer program product. The depth estimation method comprises: acquiring an image to be subjected to estimation; inputting said image into a first encoder of a network model, so as to obtain a depth-specific encoded feature, and inputting the depth-specific encoded feature into a first decoder of the network model, so as to obtain a depth-constrained decoded feature; on the basis of the depth-specific encoded feature and the depth-constrained decoded feature, determining a multi-scale depth-generalized encoded feature; and inputting the multi-scale depth-generalized encoded feature into a second decoder of the network model, so as to obtain a corresponding multi-scale depth map, wherein the depth map comprises depth image information obtained by means of performing cross-domain depth estimation on said image.

Description

Depth estimation method, device, equipment, storage medium and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is based on and claims the priority of Chinese patent application with application number 202410362026.4 and application date March 27, 2024. The entire contents of the Chinese patent application are hereby introduced into this disclosure in their entirety. Technical Field

[0003] The present application relates to the field of computer vision technology, and in particular to a depth estimation method, apparatus, device, storage medium, and computer program product. Background Art

[0004] Depth estimation (DE), such as monocular depth estimation, is a crucial visual perception task in computer vision, widely used in three-dimensional (3D) scene reconstruction, augmented reality (AR), virtual reality (VR), and the metaverse. This task helps create realistic and immersive 3D effects in AR, VR, and the metaverse. It estimates depth information from a 2D, three-primary-color (RGB) image to produce a 3D RGB depth map (RGBD), making it one of the most fundamental tasks in 3D vision.

[0005] Currently, the monocular depth estimation task is divided into supervised methods and self-supervised methods, and both methods are implemented based on deep learning algorithms of convolutional neural networks (CNN). However, due to certain defects in these two methods, in recent years, cross-domain monocular depth estimation methods that use low-cost artificial synthetic datasets with depth annotation truth values ​​for training and directly test in various real data scenarios have been widely studied, such as the direct method MonoDepth2, the domain adaptation-based method T2Net, and the domain generalization (DG)-based method S2R-DepthNet. However, these solutions all limit the accuracy performance of the final cross-domain monocular depth estimation. Summary of the Invention

[0006] To solve the technical problems existing in the related art, the embodiments of the present application provide a depth estimation method, apparatus, device, storage medium and computer program product.

[0007] To achieve the above objectives, the technical solution of the embodiment of the present application is implemented as follows:

[0008] In a first aspect, an embodiment of the present application provides a depth estimation method, the method comprising:

[0009] Obtain the image to be estimated;

[0010] Inputting the image to be estimated into a first encoder of a network model to obtain depth-specific encoding features, and inputting the depth-specific encoding features into a first decoder of the network model to obtain depth-constrained decoding features;

[0011] Determining a multi-scale depth-universal coding feature based on the depth-specific coding feature and the depth-constrained decoding feature;

[0012] The multi-scale depth universal coding features are input into the second decoder of the network model to obtain a corresponding multi-scale depth map; the depth map includes depth image information obtained by performing cross-domain depth estimation on the image to be estimated.

[0013] In a second aspect, an embodiment of the present application further provides a depth estimation device, the device comprising:

[0014] an acquisition unit, configured to acquire an image to be estimated;

[0015] A first determining unit is configured to input the image to be estimated into a first encoder of a network model to obtain depth-specific coding features;

[0016] a second determining unit configured to input the depth-specific encoding feature into a first decoder of the network model to obtain a depth-constrained decoding feature;

[0017] a third determining unit configured to determine a multi-scale depth-universal coding feature based on the depth-specific coding feature and the depth-constrained decoding feature;

[0018] The fourth determination unit is configured to input the multi-scale depth universal coding feature into the second decoder of the network model to obtain a corresponding multi-scale depth map; the depth map includes depth image information obtained by performing cross-domain depth estimation on the image to be estimated.

[0019] In a third aspect, an embodiment of the present application further provides a depth estimation device, comprising: a processor and a memory for storing a computer program that can be run on the processor;

[0020] Wherein, when the processor is used to run the computer program, it executes the steps of the depth estimation method described in the embodiment of the present application.

[0021] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the depth estimation method described in the embodiment of the present application are implemented.

[0022] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the depth estimation method described in the embodiment of the present application.

[0023] The depth estimation method, apparatus, device, storage medium, and computer program product provided in the embodiments of the present application obtain an image to be estimated; input the image to be estimated into a first encoder of a network model to obtain depth-specific coding features, and input the depth-specific coding features into a first decoder of the network model to obtain depth-constrained decoding features; determine multi-scale depth-general coding features based on the depth-specific coding features and the depth-constrained decoding features; input the multi-scale depth-general coding features into a second decoder of the network model to obtain a corresponding multi-scale depth map; the depth map includes depth image information obtained by performing cross-domain depth estimation on the image to be estimated. Using the solution of the embodiments of the present application, by inputting the image to be estimated into the first encoder of the network model to obtain depth-specific coding features, depth-irrelevant information is effectively removed, and depth-specific coding features and depth-constrained decoding features are used to fully extract depth-general coding features at multi-scale levels, and based on the extracted multi-scale depth-general coding features, a more accurate cross-domain monocular depth estimation result is obtained, that is, a corresponding multi-scale depth map is obtained, which can improve the accuracy performance of the final cross-domain monocular depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG1 is a flowchart of a depth estimation method according to an embodiment of the present application;

[0025] FIG2 is a second flow chart of a depth estimation method according to an embodiment of the present application;

[0026] FIG3 is a flowchart of a deep general feature extraction framework according to an embodiment of the present application;

[0027] FIG4 is a schematic diagram of an adaptive fusion process according to an embodiment of the present application;

[0028] FIG5 is a schematic diagram of quantitative test results of an embodiment of the present application;

[0029] FIG6 is a schematic diagram of the structure of a depth estimation device according to an embodiment of the present application;

[0030] FIG7 is a schematic diagram of the hardware structure of the depth estimation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] The present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0033] Depth estimation, such as monocular depth estimation, is a crucial computer vision perception task, widely used in 3D scene reconstruction, augmented reality, virtual reality, and the metaverse. This task helps create realistic and immersive 3D effects in these scenarios. Estimating depth information from 2D RGB images to obtain 3D RGBD information is one of the most fundamental tasks in 3D vision.

[0034] At present, the monocular depth estimation task is divided into supervised methods (i.e. supervised monocular depth estimation methods) and self-supervised methods (i.e. self-supervised monocular depth estimation methods), and both methods are implemented based on CNN deep learning algorithms.

[0035] Among them, supervised monocular depth estimation methods mainly use annotated depth truth values ​​to supervise CNN to estimate the depth map. At present, this method has achieved good results on a single dataset. However, supervised learning requires the depth sensor to collect a large amount of depth data and align it with the image to obtain the corresponding depth truth value, which is both expensive and time-consuming. In addition, when switching between various application scenarios, a large amount of data needs to be re-collected and the model needs to be trained, which is both manpower and material resources-consuming and difficult to achieve satisfactory results. Self-supervised monocular depth estimation methods use video sequences or stereo image pairs to estimate the depth map in a stereo geometry-constrained manner. However, in existing datasets and application scenarios, continuous video sequences and stereo image pairs are not always available, and the results are usually limited to a single training dataset, which makes it difficult to generalize to different unseen scenarios.

[0036] Because both the supervised and self-supervised methods mentioned above have certain drawbacks, in recent years, cross-domain monocular depth estimation methods have been widely studied, using low-cost synthetic datasets with depth annotations for training and then directly testing them in various real-world data scenarios. The algorithm models learned in this type of task need to be applied to various application scenarios, which makes learning and training difficult. To this end, the following technical solutions have been proposed in related technologies:

[0037] Technical Solution 1, Direct Method MonoDepth2: This method uses a common depth estimation model directly trained on a synthetic dataset with readily available depth annotations to estimate real-world depth. This model is then used directly for real-world testing. Through an encoder-decoder structure and a skip-connected network structure, the data trained on the synthetic dataset is directly transferred to the real dataset for testing. Specifically, the network framework design process for this technology is to input an image into a single encoder-decoder network and directly output the corresponding depth map. However, there is a huge gap between the synthetic and real domains, which inherently limits the accuracy performance of real-world depth estimation.

[0038] Technical Solution 2, domain adaptation-based method T2Net: In order to narrow the gap between the synthetic domain and the real domain, the domain adaptation-based method directly converts the synthetic domain image into a real image, including aligning the synthetic domain and the real domain at the image level or feature level to narrow the domain difference. This method uses an image migration network to convert the synthetic domain image into a real image, and then inputs the converted real image into a monocular depth estimation network similar to Technical Solution 1, that is, the testing process is to directly input the real image into the monocular depth estimation network and output the corresponding depth map. However, this method requires joint training of synthetic domain images and real domain images to reduce the difference between domains. At the same time, obtaining a large number of real-world images with depth truth annotations is expensive and time-consuming.

[0039] Technical Solution 3, Domain Generalization-Based Method S2R-DepthNet: The synthetic-to-real domain generalization method uses only labeled synthetic data for training and is tested directly in several real-world data scenarios. This is a more difficult task because the style of the real data cannot be obtained during training. During training, this method designs a depth-specific network for structural feature map extraction, and at the same time designs an enhancement network based on the attention mechanism to learn weighted attention feature maps. The depth-specific feature map is obtained by multiplying the attention feature map and the structural feature map pixel by pixel, and is input into the depth estimation network to obtain the final depth map. However, in this method, a pre-trained encoder obtained by using synthetic images and additional real-world images is required to obtain depth-specific features, which brings new domain problems and limits the performance of the depth-specific feature map. In addition, the enhancement operation is a pixel-by-pixel multiplication attention operation, and the optimization space is limited.

[0040] However, the above-mentioned technical solutions proposed in the related art have the following technical problems:

[0041] The disadvantage of the aforementioned technical solution 1 is that the model structure design lacks a structure for cross-domain universal feature extraction. As a result, when the model is trained on a synthetic dataset, it learns some of the synthetic dataset's inherent features, such as color and texture. These features are redundant for the monocular depth estimation task. When this learned model is directly tested on a real dataset, the domain differences between the datasets result in large errors in the test output depth map, leading to inaccurate cross-domain monocular depth estimation results.

[0042] Regarding the above-mentioned technical solution 2, the disadvantages of this technology are: a large amount of synthetic data and real data are required for image migration network training, and the cost of obtaining a large amount of real data is high; there are inherent domain differences in image migration, which leads to insufficient accuracy of the final migration test on real data and a low upper limit of model accuracy.

[0043] Regarding the above-mentioned technical solution three, the disadvantages of this technology are: in order to improve the generalization of the model during training, an additional real-world dataset is introduced for training the structural feature network; the enhanced network of the attention mechanism is only used for pixel-by-pixel multiplication, and the benefits obtained are limited, and it can only partially enhance the depth-specific feature map, thereby limiting the accuracy performance of the final cross-domain depth estimation.

[0044] In summary, it can be seen that the above-mentioned technical solutions proposed in the related art all limit the accuracy performance of the final cross-domain monocular depth estimation.

[0045] Based on this, an embodiment of the present application proposes a depth estimation method. In various embodiments of the present application, by inputting the image to be estimated into the first encoder of the network model, depth-specific coding features are obtained, depth-irrelevant information is effectively removed, and depth-specific coding features and depth-constrained decoding features are used to fully extract depth-general coding features at multi-scale levels. Based on the extracted multi-scale depth-general coding features, a more accurate cross-domain monocular depth estimation result is obtained, that is, a corresponding multi-scale depth map is obtained, which can improve the accuracy performance of the final cross-domain monocular depth estimation.

[0046] An embodiment of the present application provides a depth estimation method, which is applied to a depth estimation device. FIG1 is a flowchart of the depth estimation method according to an embodiment of the present application. As shown in FIG1 , the depth estimation method includes:

[0047] Step 101: Obtain an image to be estimated.

[0048] In the embodiment of the present application, the image to be estimated may be a color image, specifically an image having three primary colors RGB.

[0049] Step 102: Input the image to be estimated into the first encoder of the network model to obtain depth-specific coding features, and input the depth-specific coding features into the first decoder of the network model to obtain depth-constrained decoding features.

[0050] In an embodiment of the present application, the network model is a deep learning algorithm model based on CNN, wherein the network model includes three encoders (i.e., a first encoder, a second encoder, and a third encoder) and four decoders (i.e., a first decoder, a second decoder, a third decoder, and a fourth decoder). Specifically, the first encoder can be a depth-specific encoder, the second encoder can be a depth-general encoder, and the third encoder can be a depth-independent encoder; the first decoder can be a depth-constrained decoder, the second decoder can be a depth-general decoder, the third decoder can be a depth-specific decoder, and the fourth decoder can be a depth-independent decoder.

[0051] Here, after acquiring the image to be estimated, the depth estimation device decouples the image to be estimated, that is, decouples the image to be estimated into depth-specific coding features and depth-independent coding features, so as to effectively remove interference information in the field, such as color information or partial texture information. Specifically, the decoupling of depth-specific features and depth-independent features can be achieved by inputting the image to be estimated into different encoders of the network model. For example, the image to be estimated is input into the first encoder of the network model to obtain depth-specific coding features, and the image to be estimated is input into the third encoder of the network model to obtain depth-independent coding features. Here, the first encoder and the third encoder are different encoders.

[0052] In practical applications, in order to more effectively remove interference information, it is necessary to ensure that the two features extracted from the same image to be estimated (i.e., depth-specific coding features and depth-independent coding features) are as different as possible. Therefore, the first loss function, i.e., the difference loss function, can be used to constrain these two features.

[0053] Based on this, in one embodiment, the method further includes: inputting the image to be estimated into the third encoder of the network model to obtain depth-independent coding features; the third encoder represents the depth-independent encoder; using a first loss function to constrain the depth-independent coding features and the depth-specific coding features so that the depth-independent coding features and the depth-specific coding features meet a first condition; the first loss function represents a difference loss function.

[0054] Here, the depth-independent coding features and the depth-specific coding features meet the first condition, which can be understood as the difference between the depth-specific coding features and the depth-independent coding features being greater than the difference threshold. That is, the embodiment of the present application constrains the depth-specific coding features and the depth-independent coding features through a difference loss function to ensure that the difference between the depth-specific coding features and the depth-independent coding features extracted from the same image to be estimated is as large as possible.

[0055] It should be noted that the embodiment of the present application constrains the difference between depth-specific coding features and depth-independent coding features by adding a regularization term, that is, the vector inner product of the depth-independent coding features and the depth-specific coding features is constrained to be zero through the difference loss.

[0056] In actual application, after the depth estimation device extracts the depth-specific coding features through the first encoder of the network model, it can further optimize the depth-specific coding features. Since the depth-specific coding features should have a high correlation with the depth map, the embodiment of the present application adds a depth constraint module, namely a depth constraint decoder (also known as a first decoder), which recovers the depth map from the extracted depth-specific coding features through the depth constraint decoder. Specifically, the depth constraint decoding features are obtained by inputting the extracted depth-specific coding features into the first decoder of the network model.

[0057] Step 103: Determine a multi-scale depth-universal coding feature based on the depth-specific coding feature and the depth-constrained decoding feature.

[0058] In one embodiment, determining multi-scale depth universal coding features based on the depth-specific coding features and the depth-constrained decoding features includes: inputting the depth-specific coding features into a third decoder of the network model to obtain depth-specific decoding features; the third decoder corresponds to the first encoder; and determining multi-scale depth universal coding features based on the depth-specific decoding features and the depth-constrained decoding features.

[0059] Here, the third decoder represents a depth-specific decoder, which decodes the extracted depth-specific coding features through the depth-specific decoder to obtain depth-specific decoding features; the third decoder corresponds to the first encoder, and it can be understood that the third decoder has the same depth as the first encoder.

[0060] In actual application, after obtaining the depth-specific decoding features and depth-constrained decoding features, the depth estimation device uses an adaptive fusion process to fuse and enhance the multi-scale depth-specific decoding features and depth-constrained decoding features to obtain multi-scale depth-universal coding features, thereby improving the accuracy and versatility of cross-domain monocular depth estimation.

[0061] Based on this, in one embodiment, determining a multi-scale depth universal coding feature based on the depth-specific decoding feature and the depth-constrained decoding feature includes: inputting the depth-specific decoding feature and the depth-constrained decoding feature into a second encoder of the network model, so as to obtain a multi-scale depth universal coding feature by combining the depth-specific decoding feature and the depth-constrained decoding feature of the corresponding scale at each scale in the multi-scale of the second encoder; wherein the second encoder corresponds to the second decoder.

[0062] Here, the second encoder represents a depth-universal encoder. To further extract depth-universal features, the embodiment of the present application inputs the acquired depth-specific decoding features and depth-constrained decoding features into a new depth-universal encoder. At each scale of the depth-universal encoder, the depth-specific decoding features and depth-constrained decoding features of the corresponding scale are combined to obtain multi-scale depth-universal coding features through an adaptive fusion process. The second encoder corresponds to the second decoder, which can be understood as having the same depth as the second decoder.

[0063] The following describes in detail the process of combining the depth-specific decoding features and the depth-constrained decoding features at each scale.

[0064] In one embodiment, the respectively combining the depth-specific decoding features and the depth-constrained decoding features of the corresponding scales to obtain the multi-scale depth universal coding features includes: for each scale of the second encoder, fusing the depth-specific decoding features and the depth-constrained decoding features of the scale to obtain a fused adaptive feature; enhancing the fused adaptive feature to obtain an enhanced fused feature; feature-guiding the enhanced fused feature to obtain a target depth universal coding feature of the scale; and determining the multi-scale depth universal coding features based on the target depth universal coding features of each scale.

[0065] It can be seen that the adaptive fusion process in the embodiment of the present application includes three operations: feature fusion, feature enhancement and feature guidance. By performing an adaptive fusion process on the depth-specific decoding features and depth-constrained decoding features of multiple scales, the depth-specific features are fully reused and further optimized to obtain multi-scale depth-universal coding features. Specifically, the input of the adaptive fusion process is the depth-specific decoding features and depth-constrained decoding features of a certain scale, as well as the depth-universal coding features outputted at the previous scale of the scale. By sequentially performing feature fusion, feature enhancement and feature guidance operations on the input depth-specific decoding features and depth-constrained decoding features of a certain scale, as well as the depth-universal coding features outputted at the previous scale of the scale, the depth-universal coding features of the scale are outputted. The above-mentioned adaptive fusion process is cyclically performed on each scale of the second encoder to finally obtain multi-scale depth-universal coding features.

[0066] In practical applications, the depth estimation device can adopt an adaptive weight learning method to effectively fuse depth-specific decoding features and depth-constrained decoding features, so that the network pays more attention to general features related to depth.

[0067] Based on this, in one embodiment, the depth-specific decoding features and the depth-constrained decoding features of the scale are fused to obtain the fused adaptive features, including: respectively determining the first weight parameter corresponding to the depth-specific decoding features of the scale and the second weight parameter corresponding to the depth-constrained decoding features; and determining the fused adaptive features based on the weighted sum of the depth-specific decoding features and the depth-constrained decoding features of the scale, and their respective corresponding first weight parameters and second weight parameters.

[0068] Here, by cascading and performing double convolution layer operations on the depth-specific decoding features and depth-constrained decoding features of the scale, the corresponding weight parameters, i.e., the first weight parameter and the second weight parameter, can be determined. These two weight parameters are used to perform weighted summation with the depth-specific decoding features and the depth-constrained decoding features to determine the fused adaptive features.

[0069] In actual applications, after obtaining the fused adaptive features, the depth estimation device also needs to perform feature layer enhancement optimization learning on the fused adaptive features to achieve the purpose of feature enhancement.

[0070] Based on this, in one embodiment, the fused adaptive features are enhanced to obtain enhanced fused features, including: determining a first learning parameter based on a double convolution operation of the fused adaptive features; and determining the enhanced fused features based on attention enhancement of the first learning parameter and a pre-set second learning parameter.

[0071] Here, the second learning parameter can be pre-set according to actual needs, and the embodiments of the present application are not limited here. The double convolution operation can be implemented by a double basic convolution module, which can be composed of two layers of convolution; that is, the depth estimation device inputs the fused adaptive features into the double basic convolution module composed of two layers of convolution to obtain the first learning parameter output by the double basic convolution module. After obtaining the first learning parameter, the first learning parameter and the pre-set second learning parameter are subjected to attention mechanism learning to obtain the enhanced fusion feature.

[0072] In actual application, after obtaining the enhanced fusion features, the depth estimation device jointly inputs the enhanced fusion features and the initial depth universal coding features of the scale extracted by the depth universal module into the feature guidance module, uses separate convolution, and jointly learns the convolution parameters to obtain the final enhanced target depth universal coding features of the scale.

[0073] Based on this, in one embodiment, the enhanced fusion feature is feature-guided to obtain the target depth universal coding feature of the scale, including: obtaining the initial depth universal coding feature of the scale; inputting the enhanced fusion feature and the initial depth universal coding feature of the scale into a feature guidance module to obtain the target depth universal coding feature of the scale.

[0074] In an embodiment of the present application, the step of inputting the enhanced fused features and the initial depth universal coding features of the scale into a feature guidance module to obtain the target depth universal coding features of the scale includes: inputting the enhanced fused features and the initial depth universal coding features of the scale into a feature guidance module, so as to obtain the target depth universal coding features of the scale by performing the following operations through the feature guidance module: determining a guidance feature based on the fused adaptive features and the enhanced fused features; determining a third weight parameter and a fourth weight parameter based on the initial depth universal coding features of the scale and the guidance features; and obtaining the target depth universal coding features of the scale based on separate convolution of the initial depth universal coding features of the scale, the third weight parameter, and the fourth weight parameter.

[0075] Here, the guiding feature can be understood as an intermediate feature, which is used to determine the third weight parameter and the fourth weight parameter. Among them, the third weight parameter and the fourth weight parameter can be obtained by adaptive learning through the convolution process. Specifically, when determining the third weight parameter, the initial depth universal coding feature and the guiding feature of the scale can be cascaded first, and then the feature obtained by the cascade operation can be double-convolved to obtain the third weight parameter; when determining the fourth weight parameter, the initial depth universal coding feature and the guiding feature of the scale can be cascaded first, and then the feature obtained by the cascade operation can be convolved for the first time, and the feature obtained by the first convolution operation can be globally averaged pooled, and then the feature obtained by the global average pooling operation can be globally averaged pooled for the second time to obtain the fourth weight parameter. After obtaining the third weight parameter and the fourth weight parameter, the target depth universal coding feature of the scale can be determined by separate convolution.

[0076] Here, the separation convolution includes channel-level convolution and cross-channel convolution. Specifically, the initial depth universal coding feature of the scale and the third weight parameter can be first subjected to channel-level convolution to obtain a corresponding first convolution result, and then the first convolution result and the fourth weight parameter are subjected to cross-channel convolution to obtain a corresponding second convolution result, and the second convolution result is determined as the target depth universal coding feature of the scale.

[0077] Step 104: Input the multi-scale depth universal coding features into the second decoder of the network model to obtain a corresponding multi-scale depth map.

[0078] In an embodiment of the present application, the depth map includes depth image information obtained by performing cross-domain depth estimation on the image to be estimated, and the second decoder represents a depth universal decoder.

[0079] In practical applications, in order to ensure that the reconstructed image obtained based on depth-specific decoding features and depth-independent decoding features is as consistent as possible with the input image, i.e., the image to be estimated, the reconstruction loss function, i.e., the second loss function, can be used to constrain the reconstructed image and the image to be estimated.

[0080] Based on this, in one embodiment, the method also includes: inputting the depth-specific coding features into the third decoder of the network model to obtain depth-specific decoding features; the third decoder corresponds to the first encoder; inputting the depth-independent coding features into the fourth decoder of the network model to obtain depth-independent decoding features; the fourth decoder corresponds to the third encoder; determining the reconstructed image based on the depth-specific decoding features and the depth-independent decoding features; using a second loss function to constrain the reconstructed image and the image to be estimated so that the reconstructed image and the image to be estimated meet a second condition; the second loss function represents the reconstruction loss function.

[0081] Here, the third decoder represents a depth-specific decoder; the third decoder corresponds to the first encoder, and it can be understood that the third decoder and the first encoder have the same depth. The fourth decoder represents a depth-independent decoder; the fourth decoder corresponds to the third encoder, and it can be understood that the fourth decoder and the third encoder have the same depth. After extracting depth-specific coding features and depth-independent coding features from the same image to be estimated, the depth estimation device inputs these two features into the depth-specific decoder and the depth-independent decoder, respectively, reconstructs corresponding sub-images, and adds the corresponding sub-images reconstructed by these two features to obtain a reconstructed image. The reconstruction loss function is then used to constrain the reconstructed image and the image to be estimated so that the reconstructed image and the image to be estimated meet the second condition. The reconstructed image and the image to be estimated meeting the second condition can be understood as the difference between the reconstructed image and the image to be estimated is less than a difference threshold, that is, ensuring that the reconstructed image and the image to be estimated are as consistent as possible.

[0082] In actual application, in order to ensure that depth-specific coding features can be better extracted, the depth estimation device performs depth loss constraints on the multi-scale depth map and the depth-constrained decoding features at different scales, so that the depth-constrained decoding features and the multi-scale depth map meet the third condition.

[0083] Based on this, in one embodiment, the method further includes: using a third loss function to constrain the depth-constrained decoding features and the multi-scale depth map so that the depth-constrained decoding features and the multi-scale depth map satisfy a third condition; the third loss function represents a depth loss function.

[0084] The present application also provides another depth estimation method, which is applied to a depth estimation device. FIG2 is a second flow chart of the depth estimation method according to the present application embodiment. As shown in FIG2 , the depth estimation method includes:

[0085] Step 201: Obtain an image to be estimated.

[0086] Step 202: Input the image to be estimated into the first encoder of the network model to obtain depth-specific coding features.

[0087] In an embodiment of the present application, the first encoder may be a depth-specific encoder.

[0088] Step 203: Input the depth-specific coding features into the first decoder of the network model to obtain depth-constrained decoding features.

[0089] In an embodiment of the present application, the first decoder may be a depth-constrained decoder.

[0090] Step 204: Input the depth-specific encoding features into the third decoder of the network model to obtain depth-specific decoding features.

[0091] In this embodiment of the present application, the third decoder corresponds to the first encoder, that is, the third decoder may be a depth-specific decoder, and the depth-specific decoder and the depth-specific encoder have the same depth.

[0092] Step 205: Input the depth-specific decoding features and the depth-constrained decoding features into the second encoder of the network model, and for each scale of the second encoder, fuse the depth-specific decoding features and the depth-constrained decoding features of the scale to obtain a fused adaptive feature.

[0093] In an embodiment of the present application, the second encoder may be a depth universal encoder.

[0094] In one embodiment, fusing the depth-specific decoding features and the depth-constrained decoding features at the scale to obtain the fused adaptive features includes:

[0095] Determine respectively a first weight parameter corresponding to the depth-specific decoding feature of the scale and a second weight parameter corresponding to the depth-constrained decoding feature; and determine the fused adaptive feature based on the depth-specific decoding feature and the depth-constrained decoding feature of the scale, and a weighted sum of the first weight parameter and the second weight parameter corresponding to each.

[0096] Step 206: enhancing the fused adaptive features to obtain enhanced fused features.

[0097] In one embodiment, the fused adaptive features are enhanced to obtain enhanced fused features, including: determining a first learning parameter based on a double convolution operation of the fused adaptive features; and determining the enhanced fused features based on attention enhancement of the first learning parameter and a pre-set second learning parameter.

[0098] Step 207: performing feature guidance on the enhanced fusion features to obtain target depth universal coding features of the scale.

[0099] In one embodiment, the performing feature guidance on the enhanced fusion feature to obtain the target depth universal coding feature of the scale includes: obtaining the initial depth universal coding feature of the scale; inputting the enhanced fusion feature and the initial depth universal coding feature of the scale into a feature guidance module to obtain the target depth universal coding feature of the scale.

[0100] In one embodiment, the inputting the enhanced fused features and the initial depth universal coding features of the scale into a feature guidance module to obtain the target depth universal coding features of the scale includes: inputting the enhanced fused features and the initial depth universal coding features of the scale into a feature guidance module, so as to obtain the target depth universal coding features of the scale by performing the following operations through the feature guidance module: determining a guidance feature based on the fused adaptive features and the enhanced fused features; determining a third weight parameter and a fourth weight parameter based on the initial depth universal coding features of the scale and the guidance features; and obtaining the target depth universal coding features of the scale based on separate convolution of the initial depth universal coding features of the scale, the third weight parameter, and the fourth weight parameter.

[0101] Step 208: Determine a multi-scale depth universal coding feature based on the target depth universal coding feature of each scale.

[0102] Step 209: Input the multi-scale depth universal coding features into the second decoder of the network model to obtain a corresponding multi-scale depth map.

[0103] In an embodiment of the present application, the depth map includes depth image information obtained by performing cross-domain depth estimation on the image to be estimated, and the second decoder may be a depth universal decoder.

[0104] It should be noted that the specific processing process of the depth estimation device to complete the depth estimation has been described in detail above and will not be repeated here.

[0105] By adopting the technical solution of the embodiment of the present application, by inputting the image to be estimated into the first encoder of the network model, depth-specific coding features are obtained, depth-irrelevant information is effectively removed, and depth-specific coding features and depth-constrained decoding features are used to fully extract depth-general coding features at multi-scale levels. Based on the extracted multi-scale depth-general coding features, a more accurate cross-domain monocular depth estimation result is obtained, that is, a corresponding multi-scale depth map is obtained, which can improve the accuracy performance of the final cross-domain monocular depth estimation.

[0106] The present application is described below in conjunction with application examples.

[0107] In response to the problem of low accuracy performance of cross-domain monocular depth estimation in related technologies, this application proposes a cross-domain monocular depth estimation network model for extracting depth universal features. The model can effectively remove depth-irrelevant information and obtain depth universal features. This model does not require the introduction of any additional data (including a large amount of real data and additional data from other fields) during training. It only requires a synthetic dataset and can be trained on the synthetic dataset, that is, it can be tested directly on multiple real datasets. At the same time, the model completes the extraction of depth universal features by designing a new depth-specific feature extraction framework, an adaptive feature fusion module and a corresponding loss function.

[0108] Specifically, the model first decouples a single image (corresponding to the aforementioned image to be estimated) into depth-specific features and depth-independent features (corresponding to the aforementioned depth-specific coding features and depth-independent coding features) to effectively remove domain interference information (such as color, partial texture information, etc.); then, the extracted depth-specific coding features are added with depth constraints to restore the depth map, the depth-specific decoding features are input into the depth-invariant network for adaptive feature fusion, and the image is reused and input into the depth-invariant encoder (corresponding to the aforementioned second encoder) to obtain multi-scale depth-universal features through the adaptive fusion process, that is, fully reuse and further optimize the depth-specific features to obtain depth-universal features; finally, the depth-universal features are input into the depth-invariant decoder (corresponding to the aforementioned second decoder) to output the corresponding cross-domain universal depth map (corresponding to the aforementioned multi-scale depth map). Among them, the adaptive fusion process includes three operations: feature fusion, feature enhancement, and feature guidance; the loss function constraints include: difference loss, reconstruction loss, and depth loss. It can be seen that after multi-scale feature enhancement and extraction, more accurate cross-domain depth estimation results can be obtained, and the edge details of the depth map can be better restored. In addition, since this application can be trained without using any real data set, it can be well applied to depth estimation in various real scenes, thereby improving the accuracy performance of cross-domain monocular depth estimation.

[0109] Figure 3 is a flow chart of the depth-general feature extraction framework of an embodiment of the present application. As shown in Figure 3, in order to remove the influence of depth-irrelevant information, a color image (corresponding to the aforementioned image to be estimated) is used as the network input, and the color image is decoupled, that is, decoupled into depth-specific features and depth-independent features. The network model framework of the present application consists of three encoders and four decoders. Among them, E represents the encoder, D represents the decoder; the superscript sp represents depth-specific, nr represents depth-irrelevant, dc represents depth-constrained, and iv represents depth-general. In the feature extraction process, difference loss (Diff Loss) and reconstruction loss (Recons Loss) are used to decouple depth-specific features and depth-irrelevant features. At the same time, depth loss (Depth Loss) is also used to further ensure the extraction of depth-specific features and depth-general features. On this basis, the adaptive guidance module (AG, Adaptive Guidance) is used to fuse and enhance the multi-scale depth-specific features and depth-constrained features to obtain depth-general features, thereby improving the accuracy and versatility of cross-domain depth estimation.

[0110] The depth-general feature extraction framework of this application first uses two encoder-decoder structures to extract depth-specific features and depth-independent features respectively, as shown in Figure 3. sp (corresponding to the aforementioned depth-specific encoder), D sp (corresponding to the aforementioned depth-specific decoder), E nr (corresponding to the aforementioned depth-independent encoder) and D nr (Corresponding to the aforementioned depth-independent decoder.) A color image is input into a depth-specific encoder and a depth-independent encoder respectively, and the corresponding depth-specific coding features and depth-independent coding features are output respectively. These two features are constrained by a difference loss function to ensure that the two features extracted from the same color image are as different as possible without losing other information. At the same time, these two features are input into a depth-specific decoder and a depth-independent decoder respectively, and the corresponding sub-images are reconstructed respectively, and the reconstructed sub-images are added to obtain the reconstructed image. Among them, the reconstructed image and the input image are constrained by a reconstruction loss function to ensure that the reconstructed image and the input image are as consistent as possible.

[0111] After extracting the depth-specific coding features and depth-specific decoding features, it is hoped to further optimize the depth-specific coding features. The depth-specific coding features should be highly correlated with the depth map. Therefore, this application uses a depth-constrained decoder D to optimize the depth-specific coding features. dc Reconstruct a multi-scale depth map. The reconstructed multi-scale depth map is subjected to depth loss constraints at different scales compared with the true depth value to ensure that depth-specific encoding features can be better extracted.

[0112] To further extract deep universal features, the color image is first re-input into a new deep universal encoder E iv In the depth universal encoder, at each scale, the depth-specific decoding features and depth-constrained decoding features of the corresponding scale are combined to obtain the depth universal features through an adaptive fusion process. The specific process of adaptive fusion will be described in detail below. The input of the fusion process is the depth-specific decoding features, the depth-constrained decoding features and the depth universal features output at the previous scale. The output of the fusion process is the depth universal features of the next scale, and finally the multi-scale depth universal coding features f are obtained. iv , the depth universal encoding feature is input into the depth universal decoder to recover the multi-scale depth map, where the depth loss can be used to constrain the multi-scale depth map and the depth truth.

[0113] This application is trained only on a synthetic dataset. After two stages of training, a well-fitted depth model is obtained, namely the aforementioned cross-domain monocular depth estimation network model for extracting depth-universal features. This model can be directly inferred and tested in various real-world depth scenarios. The inference process only includes a depth-specific branch, a depth-constrained branch, and a depth-universal branch. Among them, the extracted depth-universal features can be well generalized to cross-domain depth estimation tasks.

[0114] The adaptive fusion process is described in detail below.

[0115] In this application, the adaptive fusion process aims to fuse and enhance depth-specific decoding features and depth-constrained decoding features to obtain depth-general features, which can better eliminate domain-irrelevant information. Figure 4 is a schematic diagram of the adaptive fusion process of an embodiment of the present application. As shown in Figure 4, the process may include three sub-modules: adaptive fusion (AF) (corresponding to the aforementioned feature fusion), feature layer enhancement (LA) (corresponding to the aforementioned feature enhancement) and feature guidance (FG).

[0116] Here, Figure 4(a) shows the process of the feature fusion submodule. For the depth-specific decoding features and depth-constrained decoding features, adaptive weight learning (corresponding to the first weight parameter and the second weight parameter mentioned above) can be used to effectively fuse these two features, so that the network pays more attention to the general features related to depth, and finally obtains the fused adaptive feature f af Specifically, it can be expressed by the following formulas (1) and (2): α,β=Soft(GAP(dConv(f Dsp ||f Ddc ))) (1) f af =αf Dsp+βf Ddc (2)

[0117] Among them, f Dsp represents the depth-specific decoding feature; f Ddc Represents the depth-constrained decoding feature; dConv and || represent double convolution layer operation and cascade operation, respectively, that is, the depth-specific decoding feature and the depth-constrained decoding feature are first cascaded, and then the features obtained by the cascade operation are subjected to double convolution operation; GAP and soft represent the global average pooling operation and the softmax operation, respectively; α and β are two learned weight parameters, namely the first weight parameter and the second weight parameter, which are used to perform weighted summation with the depth-specific decoding feature and the depth-constrained decoding feature, where the value ranges of α and β are 0 to 1, respectively, and α+β=1.

[0118] Here, Figure 4(b) shows the process of the feature enhancement submodule. After obtaining the fused adaptive feature f af After that, the feature layer is used to enhance the optimization of the learned fusion adaptive feature f af , to obtain the enhanced fusion feature f la Specifically, it can be expressed by the following formulas (3) and (4): ε=dBlock(f af ) (3)

[0119] Among them, dBlock represents a dual-basic convolution module consisting of two layers of convolution; ε represents the adaptive feature f after fusion af Input into the dual-basis convolution module to obtain an intermediate feature parameter of the output fusion feature (corresponding to the first learning parameter mentioned above); γ is a learnable parameter (corresponding to the second learning parameter mentioned above, which can be pre-set according to actual needs, and the value range of γ is 0 to 1); f la It is an optimized feature (corresponding to the aforementioned enhanced fusion feature).

[0120] It should be noted that in this application, the feature layer enhancement process is only used after the depth-specific features have been learned, and its output is the input of the subsequent feature guidance process.

[0121] Here, Figure 4(c) shows the process of the feature guidance submodule, which is used to fuse the optimized depth-specific features and depth-constrained features to obtain depth-general features. The depth-general features are used for cross-domain monocular depth estimation. Specifically, it can be expressed as follows: f t =[f af ,f la ] (5) ω cω =Conv(CB(fiv ||f t [i])) (6) ω cc =Conv(GAP(CB(f iv ||f t [i]))) (7) f fg =C c (C ω (f iv ,ω cω ),ω cc ) (8)

[0122] Among them, f t represents the intermediate state feature, which is used for feature selection in formulas (6) and (7); f fg represents the deep universal features output after fusion; w cw represents channel-level convolution, w cc Represents cross-channel convolution, where w cw Used to obtain the weight parameters of channel-level convolution (corresponding to the third weight parameter mentioned above), the shape is s*s*1, s is the convolution kernel size, w cc The weight parameter used to obtain the cross-channel convolution (corresponding to the fourth weight parameter mentioned above) has a shape of 1*1*c, where c is the number of channels; i can take the value of 0 or 1, which represents a stage selection state. In the first stage, i=0, and f is selected. t The former is used to calculate the convolution weights, and in the second stage i=1, f is selected t The latter is used to calculate the convolution weight; CB represents the convolution module. Separate convolution is used here to save parameters. w is the channel-level convolution, C c It is a cross-channel convolution.

[0123] As shown in Figure 4, the model training process of this application can be divided into two stages. In the first stage, only adaptive fusion and feature guidance are used to extract deep universal features. Its goal is to prepare for the extraction of a stable initial deep feature in the second stage. In the second stage, the parameters of the fixed depth-specific codec, depth-independent codec, and depth-constrained decoder are not optimized, and the rest are used for training. At this time, feature fusion, feature enhancement, and feature guidance are used simultaneously to extract optimized deep universal features.

[0124] The loss function involved in this application is explained below.

[0125] In order to decouple the depth-specific features and depth-independent features from the input image, the present application can use a difference loss (for the aforementioned first loss function) to constrain the features output by the depth-specific encoder and the depth-independent encoder (corresponding to the aforementioned depth-specific coding features and depth-independent coding features) so that the depth-specific coding features and the depth-independent coding features are as different as possible, that is, the vector inner product of the two features is constrained to be zero, while adding a regularization term. Specifically, it can be expressed as follows through the following formula (9):

[0126] Among them, L diff represents the difference loss function; Θ represents the convolution operation, and straightens the feature into a one-dimensional vector; γ represents the regularization term, which can be set to 10 -6 , to avoid the denominator being zero; Represents depth-specific encoding features; represents depth-independent encoding features; ||.||2 represents L2 loss.

[0127] In order to ensure that the reconstructed image is as consistent as possible with the original input image, this application can use reconstruction loss (corresponding to the second loss function mentioned above) to constrain the reconstructed image and the input image. At the same time, it also constrains the image output by the depth-specific branch and the image output by the depth-independent branch. Specifically, it can be expressed as follows using the following formulas (10) to (12):

[0128] Among them, L recons Represents the reconstruction loss function, I represents the input image, that is, the image to be estimated; I' represents the reconstructed image; L sp represents the depth-specific branch reconstruction loss; L nr represents the depth-independent branch reconstruction loss; I sp represents the sub-image reconstructed by the depth-specific branch; I nr represents the sub-image reconstructed by the depth-independent branch; I sp +I nr =I';G x represents the horizontal edge detection image of the discrete difference operator - Sobel operator; G y Represents the vertical edge detection image of the Sobel operator.

[0129] In order to better extract the depth-specific features, this application can use the depth loss L d (Corresponding to the third loss function mentioned above) imposes constraints on the depth constraint branch; at the same time, the depth loss also imposes constraints on the depth of the depth general feature output. d Specifically, it can be the sum of the L1 loss between the depth output of the depth constraint branch and the depth truth value and the L2 loss between the depth output of the depth general branch and the depth truth value.

[0130] It should be noted that the total loss function of the corresponding model in this application is the weighted sum of difference loss, reconstruction loss and depth loss, and the weight coefficients of the three are 0.1, 1, and 1 respectively.

[0131] Compared with the related art methods, this application has the following beneficial effects:

[0132] (1) This application designs a deep universal feature extraction framework and an adaptive fusion module to fully extract and enhance deep universal features at multi-scale levels, which can avoid the problem of insufficient feature extraction caused by simple pixel-by-pixel multiplication enhancement; based on the extracted multi-scale deep universal features, compared with the depth features of a single scale, it is obvious that more accurate cross-domain monocular depth estimation results can be obtained.

[0133] (2) During training, this application does not use any additional real datasets or other datasets that are conducive to image feature extraction, which reduces the data acquisition cost. It is a lower-cost and easier-to-implement general monocular depth estimation method.

[0134] This application is trained on the vKITTI synthetic dataset and tested on the real KITTI dataset, with test depths of 50 meters and 80 meters. For fair comparison, 192x640 images are used for training, similar to other methods, and the depth estimation accuracy performance is evaluated using the metrics proposed by Eigen et al.

[0135] The quantitative results are:

[0136] In the depth map of the qualitative results, darker colors indicate farther depth distances, and lighter colors indicate closer depth distances. The quantitative test results are shown in Figure 5.

[0137] In order to implement the depth estimation method of the embodiment of the present application, the embodiment of the present application further provides a depth estimation device. FIG6 is a schematic diagram of the composition structure of the depth estimation device of the embodiment of the present application. As shown in FIG6 , the depth estimation device includes:

[0138] An acquisition unit 61 is configured to acquire an image to be estimated;

[0139] a first determining unit 62 configured to input the image to be estimated into a first encoder of a network model to obtain depth-specific coding features;

[0140] a second determining unit 63 configured to input the depth-specific coding feature into a first decoder of the network model to obtain a depth-constrained decoding feature;

[0141] a third determining unit 64 configured to determine a multi-scale depth-universal coding feature based on the depth-specific coding feature and the depth-constrained decoding feature;

[0142] The fourth determination unit 65 is configured to input the multi-scale depth universal coding feature into the second decoder of the network model to obtain a corresponding multi-scale depth map; the depth map includes depth image information obtained by performing cross-domain depth estimation on the image to be estimated.

[0143] In one embodiment, the third determining unit 64 includes: a first determining subunit and a second determining subunit; wherein,

[0144] The first determining subunit is configured to input the depth-specific encoding feature into a third decoder of the network model to obtain a depth-specific decoding feature; the third decoder corresponds to the first encoder;

[0145] The second determining subunit is configured to determine a multi-scale depth-universal coding feature based on the depth-specific decoding feature and the depth-constrained decoding feature.

[0146] In one embodiment, the second determining subunit is configured to:

[0147] The depth-specific decoding features and the depth-constrained decoding features are input into a second encoder of the network model to obtain multi-scale depth-universal coding features by combining the depth-specific decoding features and the depth-constrained decoding features of the corresponding scale at each scale in the multi-scale of the second encoder; wherein the second encoder corresponds to the second decoder.

[0148] In one embodiment, the second determining subunit includes: a feature fusion subunit, a feature enhancement subunit, a feature guidance subunit and a third determining subunit; wherein,

[0149] The feature fusion subunit is configured to fuse the depth-specific decoding features and the depth-constrained decoding features of each scale of the second encoder to obtain a fused adaptive feature;

[0150] The feature enhancement subunit is configured to enhance the fused adaptive features to obtain enhanced fused features;

[0151] The feature guidance subunit is configured to perform feature guidance on the enhanced fusion feature to obtain the target depth universal coding feature of the scale;

[0152] The third determining subunit is configured to determine the multi-scale depth universal coding feature based on the target depth universal coding feature of each scale.

[0153] In one embodiment, the feature fusion subunit is configured as follows:

[0154] Determine respectively a first weight parameter corresponding to the depth-specific decoding feature of the scale and a second weight parameter corresponding to the depth-constrained decoding feature; and determine the fused adaptive feature based on the depth-specific decoding feature and the depth-constrained decoding feature of the scale, and a weighted sum of the first weight parameter and the second weight parameter corresponding to each.

[0155] In one embodiment, the feature enhancement subunit is configured as follows:

[0156] Based on the double convolution operation of the fused adaptive features, a first learning parameter is determined; based on the attention enhancement of the first learning parameter and a preset second learning parameter, the enhanced fused features are determined.

[0157] In one embodiment, the feature guiding subunit is configured as follows:

[0158] Acquire an initial depth universal coding feature of the scale; input the enhanced fusion feature and the initial depth universal coding feature of the scale into a feature guidance module to obtain a target depth universal coding feature of the scale.

[0159] In one embodiment, the feature guiding subunit is further configured to:

[0160] The enhanced fused features and the initial depth universal coding features of the scale are input into a feature guidance module, so that the feature guidance module performs the following operations to obtain the target depth universal coding features of the scale: determining a guidance feature based on the fused adaptive features and the enhanced fused features; determining a third weight parameter and a fourth weight parameter based on the initial depth universal coding features of the scale and the guidance feature; and obtaining the target depth universal coding features of the scale based on separate convolution of the initial depth universal coding features of the scale, the third weight parameter, and the fourth weight parameter.

[0161] In one embodiment, the depth estimation apparatus further includes: a fifth determining unit and a first constraining unit; wherein,

[0162] The fifth determining unit is configured to input the image to be estimated into the third encoder of the network model to obtain depth-independent coding features;

[0163] The first constraint unit is configured to use a first loss function to constrain the depth-independent coding feature and the depth-specific coding feature so that the depth-independent coding feature and the depth-specific coding feature meet a first condition; the first loss function represents a difference loss function.

[0164] In one embodiment, the depth estimation apparatus further includes: a sixth determining unit, a seventh determining unit, an eighth determining unit, and a second constraint unit; wherein,

[0165] The sixth determining unit is configured to input the depth-specific encoding feature into a third decoder of the network model to obtain the depth-specific decoding feature; the third decoder corresponds to the first encoder;

[0166] The seventh determining unit is configured to input the depth-independent coding feature into a fourth decoder of the network model to obtain a depth-independent decoding feature; the fourth decoder corresponds to the third encoder;

[0167] The eighth determining unit is configured to determine a reconstructed image based on the depth-specific decoding feature and the depth-independent decoding feature;

[0168] The second constraint unit is configured to use a second loss function to constrain the reconstructed image and the image to be estimated so that the reconstructed image and the image to be estimated meet a second condition; the second loss function represents the reconstruction loss function.

[0169] In one embodiment, the depth estimation apparatus further includes: a third constraint unit; wherein,

[0170] The third constraint unit is configured to use a third loss function to constrain the depth-constrained decoding feature and the multi-scale depth map so that the depth-constrained decoding feature and the multi-scale depth map satisfy a third condition; the third loss function represents a depth loss function.

[0171] In actual application, the acquisition unit 61 may be implemented by a communication interface in the depth estimation device; the first determination unit 62 , the second determination unit 63 , the third determination unit 64 and the fourth determination unit 65 may be implemented by a processor in the depth estimation device.

[0172] It should be noted that the depth estimation device provided in the above embodiment only uses the division of the above program modules as an example to illustrate when performing depth estimation. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the depth estimation device provided in the above embodiment and the depth estimation method embodiment are based on the same concept. The specific implementation process is detailed in the depth estimation method embodiment and will not be repeated here.

[0173] Based on the hardware implementation of the above program modules, and in order to implement the depth estimation method of the embodiment of the present application, the embodiment of the present application further provides a depth estimation device. FIG7 is a schematic diagram of the hardware composition structure of the depth estimation device of the embodiment of the present application. As shown in FIG7, the depth estimation device 70 includes:

[0174] Communication interface 71, capable of exchanging information with other devices;

[0175] The processor 72 is connected to the communication interface 71 to implement information exchange with other devices and is used to execute the depth estimation method provided above when running a computer program, and the computer program is stored in the memory 73.

[0176] Specifically, the communication interface 71 is configured to obtain the image to be estimated;

[0177] The processor 72 is configured to input the image to be estimated into the first encoder of the network model to obtain depth-specific coding features, and input the depth-specific coding features into the first decoder of the network model to obtain depth-constrained decoding features; based on the depth-specific coding features and the depth-constrained decoding features, determine multi-scale depth-general coding features; input the multi-scale depth-general coding features into the second decoder of the network model to obtain corresponding multi-scale depth maps; the depth maps include depth image information obtained by performing cross-domain depth estimation on the image to be estimated.

[0178] In one embodiment, the processor 72 is configured to:

[0179] The depth-specific coding features are input into a third decoder of the network model to obtain depth-specific decoding features; the third decoder corresponds to the first encoder; and based on the depth-specific decoding features and the depth-constrained decoding features, multi-scale depth-universal coding features are determined.

[0180] In one embodiment, the processor 72 is configured to:

[0181] The depth-specific decoding features and the depth-constrained decoding features are input into a second encoder of the network model to obtain multi-scale depth-universal coding features by combining the depth-specific decoding features and the depth-constrained decoding features of the corresponding scale at each scale in the multi-scale of the second encoder; wherein the second encoder corresponds to the second decoder.

[0182] In one embodiment, the processor 72 is configured to:

[0183] For each scale of the second encoder, the depth-specific decoding features and the depth-constrained decoding features of the scale are fused to obtain a fused adaptive feature; the fused adaptive features are enhanced to obtain an enhanced fused feature; the enhanced fused feature is feature-guided to obtain a target depth universal coding feature of the scale; and the multi-scale depth universal coding feature is determined based on the target depth universal coding features of each of the scales.

[0184] In one embodiment, the processor 72 is configured to:

[0185] Determine respectively a first weight parameter corresponding to the depth-specific decoding feature of the scale and a second weight parameter corresponding to the depth-constrained decoding feature; and determine the fused adaptive feature based on the depth-specific decoding feature and the depth-constrained decoding feature of the scale, and a weighted sum of the first weight parameter and the second weight parameter corresponding to each.

[0186] In one embodiment, the processor 72 is configured to:

[0187] Based on the double convolution operation of the fused adaptive features, a first learning parameter is determined; based on the attention enhancement of the first learning parameter and a preset second learning parameter, the enhanced fused features are determined.

[0188] In one embodiment, the processor 72 is configured to:

[0189] Acquire an initial depth universal coding feature of the scale; input the enhanced fusion feature and the initial depth universal coding feature of the scale into a feature guidance module to obtain a target depth universal coding feature of the scale.

[0190] In one embodiment, the processor 72 is configured to:

[0191] The enhanced fused features and the initial depth universal coding features of the scale are input into a feature guidance module, so that the feature guidance module performs the following operations to obtain the target depth universal coding features of the scale: determining a guidance feature based on the fused adaptive features and the enhanced fused features; determining a third weight parameter and a fourth weight parameter based on the initial depth universal coding features of the scale and the guidance feature; and obtaining the target depth universal coding features of the scale based on separate convolution of the initial depth universal coding features of the scale, the third weight parameter, and the fourth weight parameter.

[0192] In one embodiment, the processor 72 is further configured to:

[0193] The image to be estimated is input into the third encoder of the network model to obtain depth-independent coding features; the depth-independent coding features and the depth-specific coding features are constrained using a first loss function so that the depth-independent coding features and the depth-specific coding features satisfy a first condition; the first loss function represents a difference loss function.

[0194] In one embodiment, the processor 72 is further configured to:

[0195] The depth-specific coding features are input into the third decoder of the network model to obtain depth-specific decoding features; the third decoder corresponds to the first encoder; the depth-independent coding features are input into the fourth decoder of the network model to obtain depth-independent decoding features; the fourth decoder corresponds to the third encoder; based on the depth-specific decoding features and the depth-independent decoding features, a reconstructed image is determined; a second loss function is used to constrain the reconstructed image and the image to be estimated so that the reconstructed image and the image to be estimated meet a second condition; the second loss function represents the reconstruction loss function.

[0196] In one embodiment, the processor 72 is further configured to:

[0197] The depth-constrained decoding features and the multi-scale depth map are constrained by using a third loss function so that the depth-constrained decoding features and the multi-scale depth map satisfy a third condition; the third loss function represents a depth loss function.

[0198] It should be noted that the specific processing procedures of the communication interface 71 and the processor 72 can be understood with reference to the above-mentioned depth estimation method.

[0199] Of course, in actual applications, the various components in depth estimation device 70 are coupled together via bus system 74. It will be appreciated that bus system 74 is used to enable communication between these components. In addition to a data bus, bus system 74 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in FIG7 , all of these buses are labeled as bus system 74.

[0200] The memory 73 in the embodiment of the present application is used to store various types of data to support the operation of the depth estimation device 70. Examples of such data include: any computer program used to operate on the depth estimation device 70.

[0201] The depth estimation method disclosed in the above embodiment of the present application can be applied to the processor 72 or implemented by the processor 72. The processor 72 may be an integrated circuit chip with signal processing capabilities. During the implementation process, the various steps of the above depth estimation method can be completed by the hardware integrated logic circuit or software instructions in the processor 72. The above-mentioned processor 72 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 72 can implement or execute the various depth estimation methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the depth estimation method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor for execution, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 73. The processor 72 reads the information in the memory 73 and completes the steps of the aforementioned depth estimation method in combination with its hardware.

[0202] In an exemplary embodiment, the depth estimation device 70 can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned depth estimation method.

[0203] It can be understood that the memory 73 of the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 73 described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable types of memory.

[0204] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 73 storing a computer program. The computer program can be executed by a processor 72 in a depth estimation device 70 to complete the steps of the depth estimation method described in the aforementioned embodiment of the present application. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0205] In an exemplary embodiment, the present application further provides a computer program product, including a computer program, which can be executed by the processor 72 in the depth estimation device 70 to complete the steps of the depth estimation method described in the aforementioned embodiment of the present application.

[0206] It should be noted that: "first", "second", "third", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0207] In addition, the technical solutions described in the embodiments of the present application can be arbitrarily combined without conflict.

[0208] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A depth estimation method, comprising: Obtain the image to be estimated; Inputting the image to be estimated into a first encoder of a network model to obtain depth-specific encoding features, and inputting the depth-specific encoding features into a first decoder of the network model to obtain depth-constrained decoding features; Determining a multi-scale depth-universal coding feature based on the depth-specific coding feature and the depth-constrained decoding feature; The multi-scale depth universal coding features are input into the second decoder of the network model to obtain a corresponding multi-scale depth map; the depth map includes depth image information obtained by performing cross-domain depth estimation on the image to be estimated.

2. The method according to claim 1, wherein The determining of a multi-scale depth-universal coding feature based on the depth-specific coding feature and the depth-constrained decoding feature includes: Inputting the depth-specific encoding features into a third decoder of the network model to obtain depth-specific decoding features; the third decoder corresponds to the first encoder; Determine multi-scale depth-universal coding features based on the depth-specific decoding features and the depth-constrained decoding features.

3. The method according to claim 2, wherein: The determining of a multi-scale depth-universal coding feature based on the depth-specific decoding feature and the depth-constrained decoding feature includes: Inputting the depth-specific decoding features and the depth-constrained decoding features into a second encoder of the network model to obtain a multi-scale depth-universal coding feature by combining the depth-specific decoding features and the depth-constrained decoding features of the corresponding scale at each scale of the second encoder; The second encoder corresponds to the second decoder.

4. The method according to claim 3, wherein: The depth-specific decoding features and depth-constrained decoding features of the corresponding scales are combined to obtain multi-scale depth-general coding features, including: For each scale of the second encoder, fusing the depth-specific decoding features and the depth-constrained decoding features of the scale to obtain a fused adaptive feature; enhancing the fused adaptive features to obtain enhanced fused features; Performing feature guidance on the enhanced fusion features to obtain target depth universal coding features of the scale; The multi-scale depth universal coding features are determined based on the target depth universal coding features at each of the scales.

5. The method according to claim 4, wherein The fusing of the depth-specific decoding features and the depth-constrained decoding features of the scale to obtain the fused adaptive features includes: respectively determining a first weight parameter corresponding to the depth-specific decoding feature and a second weight parameter corresponding to the depth-constrained decoding feature at the scale; The fused adaptive features are determined based on the depth-specific decoding features and the depth-constrained decoding features of the scale, and a weighted sum of the first weight parameters and the second weight parameters corresponding to the respective ones.

6. The method according to claim 4, wherein: The step of enhancing the fused adaptive features to obtain enhanced fused features includes: Determining a first learning parameter based on a double convolution operation of the fused adaptive features; The enhanced fusion feature is determined based on the attention enhancement of the first learning parameter and the preset second learning parameter.

7. The method according to claim 4, wherein: The step of performing feature guidance on the enhanced fusion feature to obtain the target depth universal coding feature of the scale includes: Obtaining an initial deep universal coding feature of the scale; The enhanced fusion features and the initial deep universal coding features of the scale are input into a feature guidance module to obtain the target deep universal coding features of the scale.

8. The method according to claim 7, wherein: Inputting the enhanced fusion features and the initial depth universal coding features of the scale into a feature guidance module to obtain the target depth universal coding features of the scale includes: The enhanced fusion feature and the initial deep universal coding feature of the scale are input into a feature guidance module, so as to obtain the target deep universal coding feature of the scale through the following operations performed by the feature guidance module: determining a guiding feature based on the fused adaptive feature and the enhanced fused feature; Determining a third weight parameter and a fourth weight parameter based on the initial deep universal coding feature of the scale and the guide feature; A target deep universal coding feature of the scale is obtained based on separate convolution of the initial deep universal coding feature of the scale, the third weight parameter, and the fourth weight parameter.

9. The method according to claim 1, wherein The method further comprises: Inputting the image to be estimated into a third encoder of the network model to obtain depth-independent coding features; The depth-independent coding features and the depth-specific coding features are constrained by using a first loss function so that the depth-independent coding features and the depth-specific coding features satisfy a first condition; the first loss function represents a difference loss function.

10. The method according to claim 9, wherein: The method further comprises: Inputting the depth-specific encoding features into a third decoder of the network model to obtain depth-specific decoding features; the third decoder corresponds to the first encoder; Inputting the depth-independent coding feature into a fourth decoder of the network model to obtain a depth-independent decoding feature; the fourth decoder corresponds to the third encoder; determining a reconstructed image based on the depth-specific decoding features and the depth-independent decoding features; The reconstructed image and the image to be estimated are constrained by using a second loss function so that the reconstructed image and the image to be estimated satisfy a second condition; the second loss function represents a reconstruction loss function.

11. The method according to claim 1, wherein The method further comprises: The depth-constrained decoding features and the multi-scale depth map are constrained by using a third loss function so that the depth-constrained decoding features and the multi-scale depth map satisfy a third condition; the third loss function represents a depth loss function.

12. A depth estimation device, comprising: an acquisition unit, configured to acquire an image to be estimated; A first determining unit is configured to input the image to be estimated into a first encoder of a network model to obtain depth-specific coding features; a second determining unit configured to input the depth-specific encoding feature into a first decoder of the network model to obtain a depth-constrained decoding feature; a third determining unit configured to determine a multi-scale depth-universal coding feature based on the depth-specific coding feature and the depth-constrained decoding feature; The fourth determination unit is configured to input the multi-scale depth universal coding feature into the second decoder of the network model to obtain a corresponding multi-scale depth map; the depth map includes depth image information obtained by performing cross-domain depth estimation on the image to be estimated.

13. A depth estimation device, comprising: a processor and a memory for storing a computer program capable of running on said processor; Wherein, when the processor is used to run the computer program, it executes the steps of the method according to any one of claims 1 to 11.

14. A storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images