A method for image depth estimation

Abstract

The present disclosure provides an image depth estimation method. It aims to solve the problem of poor depth estimation effect in related technologies. The method comprises: obtaining multi-layer shared features of a target image; the multi-layer shared features include high-level features containing global information and at least one low-level feature containing spatial details; performing feature decoupling on the high-level features of the multi-layer shared features to obtain decoupled features, and fusing the decoupled features layer by layer with the low-level features of the multi-layer shared features through a cross-channel enhancement module to obtain multi-layer refined features; performing dynamic perceptual fusion on the decoupled features and the multi-layer refined features and inputting them into a depth prediction head to obtain a depth map of the target image.

Description

Technical Field

[0001] The present disclosure relates to the field of depth estimation, and in particular to a method for image depth estimation. Background Art

[0002] Monocular depth estimation is a deep learning-based depth measurement method that uses a single RGB image to infer the depth of each pixel, ultimately forming a pixel-level depth map. Self-supervised monocular depth estimation methods do not require expensive depth labels during training, thus avoiding label dependence. However, due to the weak constraints of the self-supervised loss, their estimation results are poor.

[0003] In order to further improve the accuracy of self-supervised monocular depth estimation methods, some works have introduced semantic information into the depth estimation model. Semantic information effectively reflects the distribution and size of different objects. This information is closely related to depth and is conducive to more accurate and comprehensive scene modeling. It not only improves the expressive ability of the depth estimation model, but also enhances its robustness in different environments.

[0004] However, deep tasks and semantic tasks have different perceptions of the global situation. Therefore, high-level features containing global information will be different. The methods provided by related technologies cannot obtain distinguishable high-level features, and thus cannot provide distinguishable guidance for different tasks. In addition, low-level features lack global information, which further limits the model effect. Summary of the Invention

[0005] To overcome the problems existing in the related art, the present disclosure provides an image depth estimation method. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided a method for estimating image depth, comprising:

[0007] Acquire multi-layer shared features of the target image; the multi-layer shared features include high-level features containing global information and at least one low-level feature containing spatial details;

[0008] Decoupling high-level features of the multi-layer shared features to obtain decoupled features, and fusing the decoupled features layer by layer with low-level features of the multi-layer shared features to obtain multi-layer refined features;

[0009] The decoupled features and the multi-layer refined features are dynamically perceived and fused and input into a depth prediction head to obtain a depth map of the target image.

[0010] Optionally, the decoupled features include semantic decoupling features and deep decoupling features, and feature decoupling is performed on high-level features of the multi-layer shared features to obtain decoupled features, including:

[0011] Inputting the high-level features into a first decoupling branch to obtain semantic initial decoupling features, and inputting the high-level features into a second decoupling branch to obtain depth initial decoupling features; the first decoupling branch and the second decoupling branch are used to simultaneously learn different features based on the same input;

[0012] Inputting the semantic initial decoupling features into a semantic prediction head for semantic segmentation to obtain a preliminary semantic segmentation map, and inputting the depth initial decoupling features into a depth prediction head for depth prediction to obtain a preliminary depth map;

[0013] Obtaining a semantic decoupling feature according to the preliminary semantic segmentation map and the semantic initial decoupling feature;

[0014] A depth decoupling feature is obtained according to the preliminary depth map and the initial depth decoupling feature.

[0015] Optionally, the decoupled features are fused layer by layer with the lower-level features of the multi-layer shared features through a cross-channel enhancement module to obtain multi-layer refined features, including:

[0016] Concatenating the semantically decoupled features and the deeply decoupled features to obtain high-level refined features;

[0017] The resolution corresponding to each low-level feature is determined as each target resolution;

[0018] The resolution of the high-level refined features is expanded layer by layer in the order of increasing target resolution, and the high-level refined features with expanded resolution are sequentially fused with low-level features with the same resolution to obtain multi-layer refined features.

[0019] Optionally, the decoupled features and the multi-layer refined features are dynamically perceptually fused and input into a depth prediction head to obtain a depth map of the target image, including:

[0020] Decoding the decoupled features layer by layer to obtain multi-layer decoded features, where each layer of decoded features in the multi-layer decoded features corresponds to a layer of refined features in the multi-layer refined features;

[0021] The multi-layer decoding features are fused with corresponding refined features in the multi-layer refined features to obtain a depth map of the target image.

[0022] Optionally, the decoupled features include semantic decoupled features and depth decoupled features; fusing the multi-layer decoded features with corresponding refined features in the multi-layer refined features and inputting the results into a depth prediction head to obtain a depth map of the target image, including:

[0023] Inputting the semantic decoupling features and the multi-layer refined features into a semantic fusion branch for multi-layer feature fusion, and inputting the deep decoupling features and the multi-layer refined features into a deep fusion branch for multi-layer feature fusion; wherein the semantic fusion branch and the deep fusion branch perform feature interaction between branches at each corresponding feature fusion layer;

[0024] The output features of the last feature fusion layer in the depth fusion branch are scaled and input into the depth prediction head to obtain the depth map of the target image.

[0025] Optionally, the semantic fusion branch and the deep fusion branch perform feature interaction between branches at each corresponding feature fusion layer, including:

[0026] Determine the feature fusion layer at the same level in the semantic fusion branch and the depth fusion branch as the first target feature fusion layer and the second target feature fusion layer;

[0027] Decoding the first input feature through the first target feature fusion layer to obtain a semantic decoding feature; fusing the semantic decoding feature with the corresponding refined feature to obtain a first fused feature, wherein the first input feature includes the first output feature of the previous first target feature fusion layer, wherein the first input feature of the first first target feature fusion layer is the semantic decoupling feature;

[0028] Decoding the second input feature through the second target feature fusion layer to obtain a deep decoding feature; fusing the deep decoding feature with the corresponding refined feature to obtain a second fused feature, where the second input feature includes the second output feature of the previous second target feature fusion layer, wherein the second input feature of the first second target feature fusion layer is the deep decoupling feature;

[0029] The first fusion feature and the second fusion feature are dynamically sensed and fused to obtain a first output feature of the first target feature fusion layer and a second output feature of the second target feature fusion layer.

[0030] Optionally, it also includes:

[0031] The output features of the last feature fusion layer in the semantic fusion branch are scaled and input into the semantic prediction head to obtain a semantic segmentation map of the target image.

[0032] Optionally, obtaining multi-layer shared features corresponding to the target image includes:

[0033] The target image is encoded layer by layer to obtain multi-layer encoding features; wherein each layer of encoding obtains an encoding feature with half the resolution, and the encoding feature with half the resolution is encoded for the next layer;

[0034] The encoding features of the last layer are determined as high-level features, and the encoding features of the remaining layers are determined as low-level features.

[0035] Optionally, the depth estimation method is implemented by an image depth estimation network, which includes: a multi-scale feature refinement module consisting of a shared feature extraction module, a first decoupling branch, a second decoupling branch, and a cross-channel enhancement module, and a dynamic perception fusion decoding module consisting of a semantic fusion branch and a depth fusion branch. The training process of the image depth estimation network includes:

[0036] Input the sample image into the image depth estimation network to obtain the semantic segmentation map and depth map corresponding to the sample image;

[0037] Determining the prediction loss of the image depth estimation network according to the semantic segmentation map and the depth map corresponding to the sample image;

[0038] According to the prediction loss, network parameters of the image depth estimation network are updated.

[0039] Optionally, the prediction loss includes: semantic loss and depth loss, and determining the prediction loss of the image depth estimation network according to the semantic segmentation map and depth map corresponding to the sample image includes:

[0040] Determining a semantic label corresponding to the sample image;

[0041] Determining the semantic loss of the sample image according to the semantic segmentation map and the semantic label;

[0042] Determining adjacent images located in adjacent frames of the sample image;

[0043] Performing pose estimation on the adjacent images and the sample image to obtain a pose transformation result;

[0044] Reconstructing the sample image according to the pose transformation result and the adjacent image to obtain a reconstructed image;

[0045] A depth loss is determined based on the reconstructed image and the sample image.

[0046] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the image depth estimation method as described in the first aspect are implemented.

[0047] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image depth estimation method described in the first aspect are implemented.

[0048] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the image depth estimation method as described in the first aspect.

[0049] The present invention decouples high-level features to obtain decoupled features with specific meanings. By fusing the decoupled features layer by layer with low-level features, the global information of the high-level features can be combined with the spatial details of the low-level features, thereby generating more accurate and detailed multi-layer refined features, which helps to improve the accuracy of depth estimation. Dynamic perception fusion can adaptively adjust the weights and fusion methods of different features based on the importance and relevance of different features. After obtaining decoupled features and multi-layer refined features, dynamic perception fusion can better adapt to different image content and scene changes, thereby generating more accurate and reliable depth maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 is a schematic diagram of the steps of an image depth estimation method shown in an embodiment of the present disclosure;

[0052] Figure 2 is a schematic diagram of a high-level feature decoupling module and a cross-channel enhancement module shown in an embodiment of the present disclosure;

[0053] Figure 3 is a schematic diagram of a dynamic skip connection module and a pixel adaptive bidirectional interaction module shown in an embodiment of the present disclosure;

[0054] Figure 4 is a structural diagram of an image depth estimation network shown in an embodiment of the present disclosure;

[0055] Figure 5 is a comparative schematic diagram of a predicted depth map shown in an embodiment of the present disclosure;

[0056] Figure 6It is a schematic diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0058] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects related to each other are in an "or" relationship.

[0059] In related technologies, semantically guided self-supervised monocular depth estimation methods often face the following problems in the network framework.

[0060] Shared encoder features cannot provide distinguishable guidance for different tasks. When the encoder extracts features, different tasks contain similar low-level features, such as basic information such as texture and edges. However, because different tasks have different perceptions of the global situation, the high-level features containing global information will be different. In the depth estimation task, high-level features pay more attention to the depth information and spatial structure of the scene, while in the semantic segmentation task, high-level features pay more attention to the shape of the object and different semantics. Therefore, directly inputting the same high-level features extracted by the shared encoder into the decoder cannot provide distinguishable guidance for different tasks. In addition, the low-level features of the encoder lack global information, which further limits the model effect.

[0061] In order to solve the problems existing in the related art, the present disclosure proposes an image depth estimation method.

[0062] Figure 1 This is a schematic diagram of the steps of an image depth estimation method shown in an embodiment of the present disclosure. Figure 1 As shown, the method may specifically include the following steps:

[0063] Step S11: Acquire multi-layer shared features of the target image; the multi-layer shared features include high-level features containing global information and at least one low-level feature containing spatial details.

[0064] Multi-layer shared features include multiple levels of shared features. Each level of shared features is obtained by passing through a coding layer in the encoder. After multi-layer encoding, the input target image is converted into multi-layer shared features. Shared features can be shared by different subsequent processing modules, avoiding the need for each module to extract features independently. Therefore, for different tasks, the features obtained in the feature extraction stage are the same.

[0065] The shared features at each level can be divided into high-level features and low-level features based on the information they contain. High-level features are abstract representations of the complex structure and semantic information in an image and therefore have a high degree of abstraction. Low-level features contain a large amount of detailed information in the image, such as edges, contours, color, and texture. Because high-level features summarize and represent the overall structure of the image, they typically have lower resolution than low-level features.

[0066] Step S12: Decouple the high-level features of the multi-layer shared features to obtain decoupled features, and fuse the decoupled features layer by layer with the low-level features of the multi-layer shared features through a cross-channel enhancement module to obtain multi-layer refined features.

[0067] The cross-channel enhancement module is used to diffuse the abstract information of high-level features into low-level information.

[0068] The depth prediction head is used to generate a depth map. Specifically, it can predict the depth value of each pixel in the image through the features learned by the network.

[0069] In order to enable multi-layer shared features to provide different guidance for different tasks, such as depth tasks and semantic tasks, high-level features are decoupled to obtain distinguishable decoupled features.

[0070] Differentiable decoupled features can effectively reflect the different requirements of different tasks, and distinguishable features can ensure that the network can effectively switch tasks and share information between different tasks. For example, depth estimation requires the model to distinguish the depth information of different objects, while semantic segmentation tasks require different object categories.

[0071] Decoupled features and low-level features can be fused through methods such as feature concatenation and weighted summation to obtain multi-layer refined features. Multi-layer refined features contain both global semantic information and local spatial details, helping to improve the accuracy and robustness of depth estimation.

[0072] Step S13: dynamically perceptually fuse the decoupled features and the multi-layer refined features and input them into a depth prediction head to obtain a depth map of the target image.

[0073] After obtaining decoupled features and multi-layer refined features, dynamic perceptual fusion is performed. Dynamic perceptual fusion refers to the dynamic weighted fusion of multiple input features during the decoder process, based on the spatial and channel information of different features. This improves the network's perception of different tasks and ultimately generates more accurate outputs. In this disclosure, this can specifically improve the model's perception of depth and semantic information, ultimately generating more accurate depth maps.

[0074] After dynamic perception fusion, a depth map of the target image is generated. The depth map contains the depth information of each pixel in the image and can be used for subsequent tasks such as 3D reconstruction and scene understanding.

[0075] By adopting the embodiments of the present disclosure, by fusing the decoupled features layer by layer with the low-level features of the multi-layer shared features, the model can better capture global structural information and local details, thereby improving the accuracy of depth estimation. Through dynamic perception fusion, the model can more flexibly perform weighted fusion of features according to different task requirements, thereby improving performance in complex scenes. By decoupling the high-level features of the multi-layer shared features to obtain decoupled features, information interference between different tasks can be effectively avoided, so that each task can obtain appropriate feature guidance, thereby improving the performance and stability of the model.

[0076] In an optional embodiment, obtaining multi-layer shared features corresponding to the target image includes: encoding the target image layer by layer to obtain multi-layer coding features; wherein, after each layer of encoding, a coding feature with half the resolution is obtained, and the coding feature with half the resolution is encoded to the next layer; the coding feature of the last layer is determined as a high-level feature, and the coding features of the remaining layers are determined as low-level features.

[0077] The target image can be an RGB image. After obtaining the target image, it can be preprocessed to scale the resolution of the target image to a preset size, such as 192×640. This can reduce the computational burden caused by the image resolution being too large. At the same time, the appropriate resolution also facilitates the prediction and comparison of the final results.

[0078] After preprocessing the target image, it is input into the shared encoder, which consists of multiple layers of feature extraction modules. The shared encoder uses the convolutional layers corresponding to these modules and an attention mechanism to extract multi-layer shared features of the target image. The encoded features output by each layer of feature extraction modules are used as one layer of shared features. The convolutional layers consist of depthwise separable convolutions with different kernel sizes, which conserve computational resources while extracting features from different receptive fields. The attention mechanism, a traditional multi-head self-attention structure, is used to extract global features and learn long-range dependencies. The attention mechanism helps the model learn long-range dependencies globally, while the depthwise separable convolutions ensure accurate capture of local details.

[0079] The feature extraction module will encode the input features so that the resolution of the input features is halved, and obtain the coded features with half the resolution, and use the coded features with half the resolution as the input features of the next layer of feature extraction module. After obtaining the coded features with half the resolution corresponding to each feature extraction module, the coded features with half the resolution corresponding to the feature extraction module of the last layer are determined as high-level features, that is, the coded features of the last layer are determined as high-level features, and the coded features with half the resolution corresponding to the feature extraction modules of the remaining layers are determined as high-level features. The high-level features can be expressed as , and the encoding features of the remaining layers are determined as low-level features.

[0080] In practice, a five-level feature extraction module can be configured. The target image is sequentially fed into the five-level feature extraction modules. After feature extraction by the different modules, coarse-grained features of five different scales are ultimately obtained. Each time the feature passes through the encoder layer, the scale is halved, ultimately resulting in five levels of shared features. These five levels of shared features correspond to different resolutions, specifically 96×320, 48×160, 24×80, 12×40, and 6×20. Smaller-scale shared features can express more abstract information.

[0081] By gradually reducing feature resolution and extracting shared features at different granularities, the disclosed embodiments can effectively capture multi-scale information in the target image and improve depth estimation accuracy. Through appropriate resolution scaling, the model can retain sufficient image information while reducing computational overhead, making it suitable for efficient depth estimation in practical applications.

[0082] In an optional embodiment, the decoupled features include semantic decoupled features and depth decoupled features, and the high-level features of the multi-layer shared features are feature decoupled to obtain decoupled features, including: inputting the high-level features into the first decoupling branch to obtain semantic initial decoupled features, and inputting the high-level features into the second decoupling branch to obtain depth initial decoupled features; the first decoupling branch and the second decoupling branch are used to simultaneously learn different features based on the same input; inputting the semantic initial decoupled features into the semantic prediction head for semantic segmentation to obtain a preliminary semantic segmentation map, and inputting the depth initial decoupled features into the depth prediction head for depth prediction to obtain a preliminary depth map; obtaining semantic decoupling features based on the preliminary semantic segmentation map and the semantic initial decoupling features; obtaining depth decoupling features based on the preliminary depth map and the depth initial decoupling features.

[0083] You can decouple the high-level features of multiple layers of shared features by setting up a high-level feature decoupling module. Figure 2 As shown, Figure 2 It is a schematic diagram of a high-level feature decoupling module and a cross-channel enhancement module shown in an embodiment of the present disclosure.

[0084] The high-level feature decoupling module receives the highest-level features from the encoder Two decoupling branches can be set in the high-level feature decoupling module, including a first decoupling branch and a second decoupling branch. The high-level features are input into the first decoupling branch and the second decoupling branch at the same time. The semantic initial decoupling features are obtained through the first decoupling branch, and the semantic initial decoupling features can be related to the semantic segmentation task; the depth initial decoupling features are obtained through the second decoupling branch, and the depth initial decoupling features can be related to the depth estimation task.

[0085] The decoupling branch can be set as a variable convolution module, which is used to quickly learn long-distance dependencies. The semantic initial decoupling features and the depth initial decoupling features are obtained through the variable convolution corresponding to the first decoupling branch and the second decoupling branch respectively. The formula for variable convolution is:

[0086]

[0087] In the above formula, represents the spatial aggregation weight of the kth sampling point in the gth group, represents the input features of the g-th group, represents the coordinates of the sampling points, Represents the result of variable convolution, which can be semantic initial decoupled features or depth initial decoupled features.

[0088] After obtaining the semantic initial decoupling features and the depth initial decoupling features, we can also use spatial attention and channel attention to perform multi-task interactive enhancement on the semantic initial decoupling features and the depth initial decoupling features to obtain enhanced semantic initial decoupling features and enhanced depth initial decoupling features. The formula for multi-task attention is:

[0089]

[0090]

[0091] In the above formula, CA represents the traditional channel attention, SA represents the traditional spatial attention, and ⊕ represents element-by-element addition. It indicates that the multi-task interaction enhancement of semantic initial decoupling features and deep initial decoupling features is performed through channel attention; It represents the multi-task interaction enhancement of semantic initial decoupling features and deep initial decoupling features through spatial attention.

[0092] Finally, after obtaining the enhanced semantic initial decoupled features and the enhanced depth initial decoupled features, the corresponding semantic prediction head can be used to perform semantic prediction on the semantic initial decoupled features to obtain a preliminary semantic segmentation map, and the corresponding depth prediction head can be used to perform depth prediction on the depth initial decoupled features to obtain a preliminary depth map. The scale of the preliminary semantic segmentation map and preliminary depth map can be consistent with the high-level features. For example, when the resolution of the high-level features is 6×20, the resolution of the preliminary semantic segmentation map and preliminary depth map is also 6×20.

[0093] The preliminary semantic segmentation map is spliced ​​into the semantic initial decoupling feature to obtain the semantic decoupling feature, which can be expressed as ; Splice the preliminary depth map into the depth initial decoupling feature to obtain the depth decoupling feature, which can be expressed as .

[0094] By using the embodiments of the present disclosure, by decoupling high-level features, semantic decoupling features and deep decoupling features can be independently learned and optimized, avoiding interference between tasks and improving the accuracy of each task. By performing multi-task interaction enhancement, the semantic initial decoupling features and deep initial decoupling features can better interact, improving the expressive power of features and achieving feature enhancement. By splicing the preliminary depth map and preliminary semantic segmentation map with the semantic initial decoupling features and deep initial decoupling features, the richness and accuracy of the final decoupled features are enhanced.

[0095] After obtaining the semantically decoupled features and the deeply decoupled features, feature refinement can be performed based on the semantically decoupled features and the deeply decoupled features, gradually diffusing the global information of the high-level features to the low-level features. The following is a specific example of feature refinement.

[0096] Among them, in an optional embodiment, the decoupled features are fused layer by layer with the low-level features of the multi-layer shared features to obtain multi-layer refined features, including: splicing the semantic decoupled features and the deep decoupled features to obtain high-level refined features; determining the resolution corresponding to each low-level feature as each target resolution; expanding the resolution of the high-level refined features layer by layer in order from small to large target resolutions, and fusing the high-level refined features with expanded resolution with the low-level features with the same resolution through a cross-channel enhancement module in turn to obtain multi-layer refined features.

[0097] The high-level feature decoupling module will splice the semantic decoupling features and the deep decoupling features to obtain high-level refined features, which integrate information from different aspects. The scale of the high-level refined features can be consistent with the scale of the high-level features. For example, when the resolution of the high-level features is 6×20, the resolution of the high-level refined features is also 6×20. Compared with high-level features, high-level refined features contain more task-related information and can provide richer and more effective information to the entire model. High-level refined features can be expressed as , high-level features can be expressed as .

[0098] Each low-level feature is extracted from the multi-layer shared features, and a target resolution is determined for each low-level feature. The target resolution can be determined based on the resolution of the low-level feature itself. For example, if the resolution of a low-level feature is 96×320, then one of the target resolutions can be set to 96×320.

[0099] Adjust the resolution of the input features layer by layer, starting from the smallest target resolution and increasing to the largest. This can be achieved through upsampling to ensure that the resolution of the adjusted input features matches that of the lower-level features to be fused. This means that the resolution is consistent. The input features can be high-level refined features or refined features from a previous fusion of low-level features.

[0100] For example, if the input feature is a high-level refined feature, the resolution of the high-level refined feature is expanded based on the minimum-scale target resolution to obtain a high-level refined feature with an expanded resolution. The high-level refined feature with the expanded resolution is then fused with the low-level feature corresponding to the target resolution to obtain a corresponding refined feature, which is then used as the target for the next resolution adjustment. The refined feature is the result of diffusing the global information of the high-level feature to the low-level feature.

[0101] For each target resolution, the above steps of resolution adjustment and feature fusion are repeated until the abstract information of high-level refined features gradually diffuses to all low-level features, and a series of multi-layer refined features that fuse information of different resolutions and abstract levels are obtained.

[0102] In the specific implementation, when there are 4 levels of low-level features, 4 cross-channel enhancement modules can be set. Figure 2 As shown in Figure 2, the input of each cross-channel enhancement module is the input features and the corresponding low-level features. The input features are the refined features or high-level refined features output by the previous cross-channel enhancement module. Here, the high-level refined features can be regarded as the initial refined features.

[0103] Each time a cross-channel enhancement module is passed, the feature resolution doubles. From the first cross-channel enhancement module to the last cross-channel enhancement module, the feature resolution changes from 12×40 to 96×320. and the refined features of the i+1th layer Perform preliminary fusion to obtain , in order to Capture the key information on different channels and input it into an efficient channel attention module for interactive enhancement in the channel dimension. First, obtain the initial enhanced features , and then get the final enhanced features The calculation formula is:

[0104]

[0105] In the above formula, It represents the features after concatenating shared features and refined features, BN represents batch normalization, and ECA represents efficient channel attention.

[0106]

[0107] In the above formula, BN means batch normalization, FFN means feedforward neural network, represents the features after preliminary enhancement, Represents the final cross-channel enhancement result.

[0108] First, channel weights are calculated to adjust the focus on different channels. ECA replaces full connection with one-dimensional convolution, greatly reducing computational complexity. Then, the feedforward neural network (FFN) performs nonlinear transformation on features to further enhance the network's expressive power.

[0109] Finally, we get 5 scales of refined features ,in , the feature resolution is 96×320, 48×160, 24×80, 12×40, 6×20, among which, the refined feature with a resolution of 6×20 Refined features obtained after multi-scale feature refinement layer processing With encoder characteristics Compared with CNN, CNN contains richer information and can provide clearer guidance for the decoder.

[0110] By adopting the embodiments of the present disclosure, the expressive power of features can be enhanced by fusing semantically decoupled features with deeply decoupled features. High-level features provide global semantic information, while low-level features provide detailed information. The fusion of the two can help the model fully understand the image. By gradually doubling the feature resolution, high-level global information is gradually propagated to low-level features, thereby enhancing the semantic understanding ability of low-level features. Low-level features not only contain detailed information, but also use global context to improve the reasoning ability and accuracy of the model in complex scenarios.

[0111] Among them, in an optional embodiment, the decoupled features and the multi-layer refined features are dynamically perceived and fused and input into the depth prediction head to obtain the depth map of the target image, including: decoding the decoupled features layer by layer to obtain multi-layer decoded features, each layer of decoded features in the multi-layer decoded features corresponds to a layer of refined features in the multi-layer refined features; fusing the multi-layer decoded features with the corresponding refined features in the multi-layer refined features and inputting them into the depth prediction head to obtain the depth map of the target image.

[0112] The decoupled features are input into a decoder consisting of multiple feature fusion layers. The decoupled features pass through each feature fusion layer. Each time a feature fusion layer passes through, the feature fusion layer decodes the input features to obtain the decoded features corresponding to that feature fusion layer. The resolution of the decoded features is doubled relative to the input features. The input features of the feature fusion layer are either the decoupled features or the decoded features output by the previous feature fusion layer. For example, the input to the first feature fusion layer is the decoupled features, and the input to each feature fusion layer after the first feature fusion layer is the decoded features output by the previous feature fusion layer.

[0113] Since the feature fusion layer of each layer will output the corresponding decoding features, after the decoupled features are input into the decoder, multiple layers of decoding features will be obtained, and the levels of the decoding features correspond to the levels of the feature fusion layer.

[0114] In the process of decoding the decoupled features by the decoder, the refined features will be fused layer by layer.

[0115] According to the scale of the decoding features and the scale of the refined features, there is a one-to-one correspondence between each decoding feature in the multi-layer decoding features and each refined feature in the multi-layer refined features. The resolution of the decoding features and refined features that have a corresponding relationship is consistent.

[0116] After obtaining a layer of decoding features, the decoding features are fused with the corresponding refined features to obtain updated decoding features, and the updated decoding features are used as the output of the feature fusion layer. The output is used as the input of the next layer of feature fusion layer or as the object of scale recovery.

[0117] Specifically, the decoupled features are input into the first feature fusion layer for decoding to obtain decoded features. The decoded features are then fused with the refined features to obtain updated decoded features. The updated decoded features are used as the output of the first feature fusion layer, and this output is input into the next feature fusion layer for decoding until there is no next feature fusion layer. In the absence of a next feature fusion layer, the output of the last feature fusion layer is scaled to obtain a depth map of the target image.

[0118] Using the embodiments of the present disclosure, decoupled features are converted into multi-level decoded features through a layer-by-layer decoding process, and these features are aligned with multiple layers of refined features. This allows the decoded features of each layer to be precisely combined with the refined features, ensuring the correlation between the layers and the effectiveness of information transfer. By fusing decoupled features with refined features, the model can effectively combine local details and global semantic information to generate a detailed and accurate depth map, significantly improving the accuracy and detail of depth estimation, particularly in areas such as object boundaries and depth-ambiguous regions.

[0119] In an optional embodiment, the decoupled features include semantic decoupled features and depth decoupled features; the multi-layer decoding features are fused with the corresponding refined features in the multi-layer refined features and input into the depth prediction head to obtain a depth map of the target image, including: inputting the semantic decoupled features and the multi-layer refined features into the semantic fusion branch for multi-layer feature fusion, and inputting the depth decoupled features and the multi-layer refined features into the depth fusion branch for multi-layer feature fusion; wherein the semantic fusion branch and the depth fusion branch perform feature interaction between branches in each corresponding feature fusion layer; the output features of the last feature fusion layer in the depth fusion branch are scaled and input into the depth prediction head to obtain the depth map of the target image.

[0120] Decoupled features include semantic decoupled features and depth decoupled features. Semantic decoupled features represent the category or attribute information of different objects in the target image, while depth decoupled features represent the depth or distance information of different parts in the target image.

[0121] In order to make full use of semantic decoupling features and deep decoupling features, two fusion branches are designed in the decoder, including the semantic fusion branch and the deep fusion branch. The semantic fusion branch is used to decode the semantic decoupling features, and the deep fusion branch is used to decode the deep decoupling features.

[0122] The multi-layer refined features of the semantic fusion branch input are consistent with the multi-layer refined features of the depth fusion branch input.

[0123] In each fusion branch, multi-layer feature fusion is performed. Multi-layer feature fusion means that the decoupled features input to the fusion branch are decoded layer by layer to obtain multi-layer decoded features, and then the multi-layer decoded features are fused with the multi-layer refined features input to the fusion branch. For example, for the semantic fusion branch, the input of the semantic fusion branch is the semantic decoupling features and the multi-layer refined features. Each feature fusion layer in the semantic fusion branch decodes the semantic decoupling features to obtain multi-layer decoded features, and then fuses the multi-layer decoded features with the corresponding refined features in the multi-layer refined features.

[0124] The semantic fusion branch and the deep fusion branch interact with each other at each corresponding feature fusion layer, exchanging information to enhance the feature representations of each branch. There is a correspondence between the feature fusion layers at the same level in the semantic fusion branch and the deep fusion branch. For example, at a certain level, the features output by the semantic fusion branch capture information from the deep fusion branch, and vice versa.

[0125] In the depth fusion branch, after multiple layers of feature fusion, the output features from the final fusion layer undergo scale restoration. Scale restoration is necessary because during the feature extraction and fusion process, the scale of features may change, such as resolution or size. Scale restoration restores these features to the scale of the original or target image to generate the final depth map, which reflects the depth information of different parts of the target image.

[0126] The disclosed embodiments separate semantically decoupled features from depth-decoupled features, and then feed the decoupled and refined features into two separate branches for fusion. This allows for independent optimization of the two, with feature interaction between the fusion layers. This effectively combines global semantic information with local depth information, improving the accuracy of depth map generation and detail capture. Scale restoration ensures that the resulting depth map is consistent with the input target image, avoiding depth distortion caused by scale mismatch and enhancing the authenticity and application value of the depth map.

[0127] Among them, in an optional embodiment, it also includes: restoring the scale of the output features of the last feature fusion layer in the semantic fusion branch and inputting them into the semantic prediction head to obtain the semantic segmentation map of the target image.

[0128] The semantic prediction head assigns a predicted category label to each pixel in the image and finally outputs a semantic segmentation map consistent with the target image.

[0129] In the semantic fusion branch, after multiple layers of feature fusion, the output features from the final feature fusion layer undergo scale restoration. Scale restoration is necessary because during the feature extraction and fusion process, the scale of features may change, such as resolution or size. Scale restoration restores these features to the scale of the original or target image to generate the final semantic segmentation map.

[0130] By adopting the embodiments of the present disclosure, by simultaneously generating a depth map and a semantic segmentation map, the advantages of multi-task learning are utilized. Multi-task learning can promote information sharing and feature reuse between different tasks, thereby improving overall performance.

[0131] In an optional embodiment, the semantic fusion branch and the deep fusion branch perform feature interaction between branches in each corresponding feature fusion layer, including: determining the feature fusion layer at the same level in the semantic fusion branch and the deep fusion branch as the first target feature fusion layer and the second target feature fusion layer; decoding the first input feature through the first target feature fusion layer to obtain a semantic decoding feature; fusing the semantic decoding feature with the corresponding refined feature to obtain a first fusion feature, wherein the first input feature includes the first output feature of the previous first target feature fusion layer, wherein the first input feature of the first first target feature fusion layer is the semantic decoupling feature; decoding the second input feature through the second target feature fusion layer to obtain a deep decoding feature; fusing the deep decoding feature with the corresponding refined feature to obtain a second fusion feature, wherein the second input feature includes the second output feature of the previous second target feature fusion layer, wherein the second input feature of the first second target feature fusion layer is the deep decoupling feature; dynamically perceptually fusing the first fusion feature with the second fusion feature to obtain the first output feature of the first target feature fusion layer and the second output feature of the second target feature fusion layer.

[0132] In the semantic fusion branch and the deep fusion branch, the feature fusion layers at the same level are respectively determined as the first target feature fusion layer and the second target feature fusion layer. That is, a correspondence is established between the feature fusion layers at the same level in the semantic fusion branch and the deep fusion branch to obtain a feature fusion layer pair, where the semantic fusion branch corresponds to the first target feature fusion layer and the deep fusion branch corresponds to the second target feature fusion layer.

[0133] For each first target feature fusion layer in the semantic fusion branch, its first input feature includes the first output feature of the previous first target feature fusion layer. Specifically, for the first first target feature fusion layer, its first input feature is the semantic decoupling feature.

[0134] The first input feature is decoded through the first target feature fusion layer to obtain semantic decoding features. The decoding process involves operations such as convolution and upsampling to restore the resolution and details of the features.

[0135] The semantic decoding features are fused with the corresponding refined features to obtain the first fused features. The refined features are used to provide additional detail information. The fusion process involves weighted summation and concatenation operations.

[0136] For each second target feature fusion layer in the deep fusion branch, its second input feature includes the first output feature of the previous second target feature fusion layer. Among them, for the first second target feature fusion layer, its first input feature is the deep decoupled feature.

[0137] The second input feature is decoded through the second target feature fusion layer to obtain a deep decoding feature. The decoding process is similar to the semantic fusion branch.

[0138] The deep decoded features are fused with the corresponding refined features to generate the second fused features. The refined features are used to provide additional detail information. The fusion process involves weighted summation and concatenation operations.

[0139] Dynamic perceptual fusion is performed on the first fused feature and the second fused feature to obtain the first output feature of the first target feature fusion layer at the current level and the second output feature of the second target feature fusion layer at the current level. The first output feature is the final decoded feature corresponding to the first target feature fusion layer at the current level, and the second output feature is the final decoded feature corresponding to the second target feature fusion layer at the current level. Dynamic perceptual fusion involves dynamically adjusting the fusion weights or strategies based on the target image content and contextual information to better integrate the feature information from the two branches.

[0140] The above process is repeated at each corresponding feature fusion layer until the last feature fusion level is reached. At each level, both branches update and transfer feature information through decoding, fusion, and dynamic perception fusion.

[0141] By adopting the embodiments of the present disclosure, by determining the feature fusion layer at the same level in the semantic fusion branch and the depth fusion branch as the first target feature fusion layer and the second target feature fusion layer, and performing feature interaction at these levels, it can be ensured that the two branches can fully utilize each other's information during the processing process. Through dynamic perception fusion, the model can adaptively adjust the fusion method of each layer of features, flexibly respond to different depth estimation requirements in the image, and improve the adaptability and robustness of the model. Layer-by-layer decoding and feature fusion combined with dynamic perception fusion make the final generated depth map not only have high accuracy, but also effectively capture the details and global depth information in the image. Especially in complex scenes, the quality of the depth map is significantly improved.

[0142] A specific embodiment is provided below to illustrate step S13 in detail. Figure 3 Schematic diagram of a dynamic skip connection module and a pixel adaptive bidirectional interaction module shown in an embodiment of the present disclosure.

[0143] S131: Reference Figure 3 As shown in Figure 1, the refined features of layer i are fused with the output features of layer i+1 through dynamic skip connections. Dynamic skip connections connect the refined features and decoded features of the same scale to obtain the updated decoded features of layer i, i.e., the fused features. It consists of two branches: the deep fusion branch and the semantic fusion branch, which are used to process deep features and semantic features respectively.

[0144] Taking the deep fusion branch as an example, the refined features of the i-th layer and the second output feature of layer i+1 Before fusion, in order to reduce the detail conflict in the fusion process, the refinement features A two-stage preprocessing was performed.

[0145] because and The scales are inconsistent. First, the second output feature is sampled using bilinear interpolation. Upsampling to obtain deep decoding features In order to ensure the consistency of the distribution of the two features and reduce fusion conflicts, it is necessary to calculate The average value on each channel is used to obtain the distribution in the channel dimension, and then the distribution in the channel dimension is calculated. Make adjustments, the adjustment formula is:

[0146]

[0147] In the above formula, GAP represents global average pooling, is the activation function, Represents refined features.

[0148] Secondly, in order to refine the features With the deep decoding feature The changes of adaptive adjustment are made, so that the features are refined Able to sensitively perceive deep decoding features The changes of , and then obtain the deep decoding features New information in the network is used to refine features using pixel-adaptive convolution Updated:

[0149]

[0150] Among them, PAC stands for pixel adaptive convolution, which can decode features according to depth To dynamically adjust the convolution kernel, and finally use the convolution kernel to refine the features So, the feature is refined It can dynamically perceive the decoding features and obtain the adjusted refined features .

[0151] The expression of PAC pixel convolution is:

[0152]

[0153] In the above formula, is the convolution kernel weight, Is a kernel function used to adjust the convolution kernel weights, Indicates the value of the pixel, b indicates the bias, Indicates the sliding window range.

[0154] Finally, the adjusted refined features and deep decoding features The second fusion feature of the i-th layer can be obtained by fusion Compared with the original features, the adjusted refined features are more consistent with the prediction objectives of this task. The refined features adjusted by the depth fusion branch focus on objects at different depths, while the refined features adjusted by the semantic fusion branch focus on objects of different semantic categories.

[0155] S132: Reference Figure 3 As shown, the first fusion feature of the i-th layer and the second fusion feature of the i-th layer are subjected to pixel-adaptive bidirectional interaction to obtain the first output feature and the second output feature of the i-th layer. Similarly, it contains two branches, depth and semantics. First, the second fusion feature and the first fusion feature Perform convolution processing to obtain and In order to make the features of the depth fusion branch perceive the semantic information in the features of the semantic fusion branch, pixel adaptive convolution is applied to the features of the depth fusion branch. Similarly, pixel adaptive convolution is applied to the features of the semantic fusion branch to make them perceive the depth information in the depth features. The calculation formula is as follows:

[0156]

[0157]

[0158] in, and Represent the depth perception features and semantic perception features respectively. In order to balance the contribution of different features to the final prediction, learnable weights are used to and Perform weighted fusion, the first output feature and the second output feature The method of determining is:

[0159]

[0160]

[0161] in, and are all learnable hyperparameter weights, and Represents the second fused features and the first fused features before convolution.

[0162] The second fusion feature and the first fusion feature are updated through pixel adaptive convolution to obtain updated second fusion feature and first fusion feature, that is, second output feature and first output feature.

[0163] The pixel-adaptive bidirectional interaction module allows the decoding features of the two branches to fully utilize each other's complementary information and further enrich the decoding features.

[0164] In which, in an optional embodiment, the depth estimation method is implemented through an image depth estimation network, and the image depth estimation network includes: a shared feature extraction module, a first decoupling branch, a second decoupling branch, and a multi-scale feature refinement module composed of a cross-channel enhancement module, and a dynamic perception fusion decoding module composed of a semantic fusion branch and a depth fusion branch. The training process of the image depth estimation network includes: inputting a sample image into the image depth estimation network to obtain a semantic segmentation map and a depth map corresponding to the sample image; determining the prediction loss of the image depth estimation network based on the semantic segmentation map and depth map corresponding to the sample image; and updating the network parameters of the image depth estimation network based on the prediction loss.

[0165] The shared feature extraction module is used to extract multi-layer shared features for depth estimation and semantic segmentation tasks. This is achieved through multiple levels of shared coding layers. At each shared coding layer, a layer of shared features is extracted and fed into the next shared coding layer.

[0166] The multi-scale feature refinement module includes a high-level feature decoupling module and multiple levels of cross-channel enhancement modules.

[0167] The first decoupling branch and the first decoupling branch are configured in the high-level feature decoupling module, and the high-level features are decoupled by the first decoupling branch and the first decoupling branch to obtain semantically decoupled features and deeply decoupled features, and the semantically decoupled features and the deeply decoupled features are spliced ​​to obtain high-level refined features.

[0168] After obtaining high-level refined features through the high-level feature decoupling module, the high-level refined features are input into each cross-channel interaction module in turn and fused with the corresponding low-level features to obtain multi-layer refined features.

[0169] The dynamic perception fusion decoding module is used to dynamically fuse the refined features and decoupled features, and use the dynamic perception mechanism to decode the final depth map and semantic segmentation map.

[0170] The sample image is input into the image depth estimation network to obtain the semantic segmentation map and depth map corresponding to the sample image. The sample image is the target image during model training.

[0171] According to the semantic segmentation map and the depth map, the prediction loss of the image depth estimation network is determined; based on the calculated prediction loss, the network parameters are updated using a back propagation algorithm to reduce the prediction loss, thereby improving the performance of the network.

[0172] The training is repeated multiple times until the network converges or the preset number of training rounds is reached.

[0173] Using the embodiments of the present disclosure, during training, the network determines prediction loss based on the semantic segmentation map and depth map corresponding to the sample image. Based on this prediction loss, the network parameters are updated, enabling adaptive learning to better perform depth estimation and semantic segmentation. Because the network simultaneously performs semantic segmentation and depth estimation, these two tasks mutually enhance each other during training, improving network performance.

[0174] In an optional embodiment, the prediction loss includes: semantic loss and depth loss. The prediction loss of the image depth estimation network is determined according to the semantic segmentation map and depth map corresponding to the sample image, including: determining the semantic label corresponding to the sample image; determining the semantic loss of the sample image according to the semantic segmentation map and the semantic label; determining the adjacent images located in adjacent frames of the sample image; performing pose estimation on the adjacent images and the sample image to obtain a pose transformation result; performing a reconstruction operation on the sample image according to the pose transformation result and the adjacent images to obtain a reconstructed image; and determining the depth loss according to the reconstructed image and the sample image.

[0175] Semantic labels are the category labels associated with each pixel in an image. In practical applications, these labels are obtained through manual annotation or by leveraging existing semantic segmentation datasets. The semantic segmentation map generated by the network is the network's prediction of the semantic category for each pixel in the sample image. Semantic loss is calculated by comparing the difference between the predicted result (semantic segmentation map) and the true result (semantic label). The magnitude of the semantic loss reflects the network's performance on the semantic segmentation task; smaller losses indicate that the network's predictions are closer to the true labels. Semantic loss specifically includes cross-entropy loss.

[0176]

[0177] In the above formula, CE represents cross entropy, represents the predicted semantic graph, Represents a semantic label.

[0178] Semantic cross entropy loss is used to evaluate the difference between the probability distribution predicted by the model and the true label in the classification task. The calculation formula is:

[0179]

[0180] In the above formula, N represents the number of pixels, C represents the number of semantic categories, 0,1}, indicating whether pixel i belongs to category c, Denotes the probability that pixel i in the predicted semantic map belongs to category c. The depth loss and semantic loss work together in model training to ensure that the model not only accurately reconstructs the data but also maintains smooth output and achieves high accuracy in semantic segmentation tasks.

[0181] In a video sequence or continuous frame images, the image content of adjacent frames is similar, but the perspective or position varies. To calculate depth loss, the sample image can be reconstructed using the depth maps of adjacent frame images and the sample image to obtain a reconstructed image. The difference between the reconstructed image and the sample image is then compared to determine the depth loss.

[0182] Pose estimation is used to determine the position and posture of objects in an image and obtain a pose transformation result, which is used for subsequent reconstruction operations. In this embodiment, it is necessary to estimate the position and posture changes of adjacent frame images relative to the sample image based on the sample image and the adjacent frame images. Specifically, the adjacent image frames can be input into the pose estimation network, which is also an encoder-decoder structure, and finally outputs the pose transformation between adjacent frames. The pose transformation is usually represented by a six-dimensional vector, which includes three translation parameters and three rotation parameters, and is used to represent the spatial geometric relationship between two adjacent image frames.

[0183] After obtaining the result of the position transformation, the sample image is reconstructed according to the position transformation result and the adjacent image to obtain a reconstructed image. Specifically, through the position transformation relationship between adjacent image frames, the specific coordinates of the relevant pixel points on the adjacent frames can be matched, and the pixel points can be located according to the spatial relationship and the specific values ​​of the pixel points can be calculated to transform the image from the current coordinate system to the adjacent coordinate system. , Depth Map , camera internal parameter K is used to reconstruct the view, and the source image Reconstruct the target image In the coordinate system, the view reconstruction formula is:

[0184]

[0185]

[0186] In the above formula, K represents the camera internal parameter, represents the pose transformation between adjacent frames, represents the predicted depth map, Indicates the pixel coordinates on the graph, B indicates the bilinear interpolation method. represents adjacent images, Represents the pixel coordinates on adjacent frames, Represents the reconstructed image.

[0187] After obtaining the reconstructed image, the depth loss is determined based on the reconstructed image and the sample image. The depth loss consists of two parts: reconstruction loss and smoothing loss.

[0188]

[0189] In the above formula, and represents the luminosity loss and smoothness loss, represents the loss weight.

[0190] Photometric loss is used to measure the target image and reconstruction The difference between them is calculated as:

[0191]

[0192] in represents hyperparameters, SSIM represents structural similarity, represents the target image, represents the reconstructed target image.

[0193] The smoothing loss aims to promote the smoothness of the model output and reduce the noise and discontinuity in the prediction results. The calculation formula is:

[0194]

[0195] In the above formula, and Represents the gradient on two coordinate axes, Represents deep features.

[0196] After determining the reconstruction loss, smoothing loss, and semantic cross entropy loss, the reconstruction loss, smoothing loss, and semantic cross entropy loss are integrated to obtain the total loss of model training, that is, the prediction loss. The prediction loss is:

[0197]

[0198] In the above formula, It is the weighted coefficient of loss, which is used to balance the contribution of different losses in the model training process. and represents the luminosity loss and smoothness loss, Represents semantic loss.

[0199] In this example, the outdoor scene dataset KITTI is used as the training set, which includes 39,810 training images and 4,424 validation images. Before training, the network will load its own pre-trained weights on ImageNet. Finally, it is trained in parallel on two NVIDIA RTX 4090 GPUs, using the AdamW optimizer and the CosineAnnealingLR scheduler to adjust the learning rate. The training resolution is 192×640, the batch size is set to 12, and the loss weight is set to , the learning rate of the shared encoder is set to 6×10-5, and the learning rate of other networks is set to 10-4, with a total of 30 iterations.

[0200] In the embodiments of this disclosure, semantic loss guides the network to focus on the global semantic information of the image, while depth loss optimizes local depth details through pose estimation and image reconstruction. Combining the optimization of semantic and depth loss, the generated depth map can both preserve details and accurately capture depth variations in the image, making it suitable for complex visual scenes. Through pose estimation and image reconstruction, the network can calculate the depth loss in a self-supervised manner, reducing the reliance on additional annotated data and improving training efficiency.

[0201] Figure 4 Schematic diagram of the structure of an image depth estimation network shown in an embodiment of the present disclosure.

[0202] The multi-scale feature refinement module consists of five modules, including one high-level feature decoupling module and four cross-channel enhancement modules. With each module, the feature resolution doubles, and upsampling is accomplished through bilinear interpolation. The encoder inputs the highest-level features into the high-level feature decoupling module, which then outputs three features: semantically decoupled features, deeply decoupled features, and high-level refined features. The semantically decoupled features and deeply decoupled features serve as the initial input to the decoder, while the high-level refined features serve as the initial input to the cross-channel enhancement module. Each cross-channel enhancement module contains two inputs: the refined features of the previous layer and the shared features of the corresponding encoder layer. After processing, it outputs one refined feature, which serves as the input to the next cross-channel enhancement module. The features are iterated layer by layer through the four cross-channel enhancement modules.

[0203] The dynamic-aware fusion decoding module is a decoder composed of two parts: dynamic skip connections and pixel-adaptive bidirectional interaction. The decoder consists of two branches: a semantic fusion branch and a depth fusion branch. Each branch contains four dynamic skip connection modules, with three pixel-adaptive bidirectional interaction modules between the two branches. The dynamic skip connection module takes two inputs: the decoded features of the previous layer and the refined features of the corresponding layer. It outputs a single decoded feature of the current layer. Each pass through the dynamic skip connection module doubles the resolution of the decoder features, using bilinear interpolation for upsampling. The pixel-adaptive bidirectional interaction module takes two inputs: the depth decoding features and the semantic decoding features of the current layer. It outputs two features: the updated depth decoding features and the semantic decoding features. While the resolution of the input and output features remains unchanged, the output features contain more effective information, providing clearer guidance for the final prediction. The dynamic skip connection module and the pixel-adaptive bidirectional interaction module alternately process the decoded features. The final layer of features in the decoder is input to the prediction head, which outputs the final prediction result.

[0204] The workflow of the image depth estimation network provided by the present disclosure is described.

[0205] Step 1: General feature extraction

[0206] A single image is input into the shared feature extraction module, also known as the shared encoder. The shared encoder extracts common features from the input image. After each encoder layer, the scale of the features is halved, and finally, feature maps of five scales are generated as input to the subsequent multi-scale feature refinement layer.

[0207] Step 2: Feature refinement

[0208] The high-level features of the universal decoder are used as the initial input of the high-level feature decoupling module of the multi-scale feature refinement module. The same high-level features are decoupled into two decoupled features, namely, deep decoupled features and semantic decoupled features. The deep decoupled features and the semantic decoupled features are fused and input into the cross-channel feature enhancement module of the subsequent multi-scale feature refinement module. The global information of the high-level features is diffused to the low-level features to obtain multi-layer refined features.

[0209] Step 3: Dynamic Perception Fusion Decoding

[0210] Dynamically fuse multiple layers of refined features and decoding features to enhance decoding features. First, the refined features from the multi-scale feature refinement module are spatially aligned with the decoding features. The aligned decoding and refined features are then adaptively fused using pixel-adaptive convolution. Secondly, a pixel-adaptive bidirectional interaction mechanism is used between the two decoder branches for dynamic interactive perception, enabling deep decoding features to dynamically capture useful information from semantic decoding features.

[0211] Step 4: Pose Estimation

[0212] Three consecutive frames are input into the pose estimation network, which will output a 6D pose. The pose transformation result and the predicted depth map are used to perform view reconstruction to obtain a reconstructed image.

[0213] Step 5: Model Evaluation

[0214] The depth loss is determined based on the reconstructed image and depth image, the semantic loss is determined based on the semantic segmentation image and semantic label, the prediction loss is determined by the depth loss and semantic loss, and the performance of the model is evaluated by the prediction loss, so that the training hyperparameters can be modified to continuously fine-tune the network.

[0215] In order to verify the usefulness of the image depth estimation method provided by the present disclosure, we analyze the following results based on the specific quantitative experimental results and depth map visualization results:

[0216] As shown in Table 1, Monodepth2, R-MSFM6, HR-Depth, DIFFNet, Lite-mono-8M, ManyDepth, DynamicDepth, Monovit, DCPI-Depth, SGDepth, Guizilini, and FSRE-Depth are different depth estimation methods, all of which are existing depth estimation methods. Ours is the image depth estimation method provided by this disclosure. As shown in Table 1, the smaller the depth error, the greater the depth accuracy, and the better the model effect. Compared with previous methods, the depth error and depth accuracy of this disclosure show excellent results.

[0217] Table 1 Depth estimation results on the KITTI test set

[0218]

[0219] In addition, reference Figure 5 As shown, Figure 5 is a comparative schematic diagram of a predicted depth map shown in an embodiment of the present disclosure, Figure 5 The visualization of the predicted depth map is shown. The first row shows the input target image, the other rows show the prediction results of different methods, and the last row shows the prediction results of the present invention. The present invention has a clear advantage in prediction details, and can predict more accurate edges in areas where objects of different categories are clustered. For example, in the first column, where people and bicycles overlap, and in the second column, where railings, signs, and tree trunks are clustered, the present invention predicts more accurate outlines. In addition, for some smaller objects, such as the slender railings in columns 3-5, the present invention can also accurately predict their depth.

[0220] The present disclosure also provides an electronic device, Figure 6 , Figure 6 FIG. 1 is a schematic diagram of an electronic device shown in an embodiment of the present disclosure. Figure 6 As shown, the electronic device 600 includes: a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus communication. A computer program is stored in the memory 610, and the computer program can be run on the processor 620 to implement the steps in the image depth estimation method disclosed in the embodiment of the present disclosure.

[0221] An embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the image depth estimation method disclosed in the embodiment of the present disclosure are implemented.

[0222] The embodiments of the present disclosure further provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the image depth estimation method disclosed in the embodiments of the present disclosure are implemented.

[0223] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0224] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, apparatuses, or computer program products. Thus, the embodiments of the present disclosure may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0225] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0226] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0227] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0228] Although some embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the present disclosure.

[0229] The above is a detailed introduction to an image depth estimation method provided by the present disclosure. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present disclosure.

Claims

1. A method for estimating image depth, characterized in that: include: Acquire multi-layer shared features of the target image; the multi-layer shared features include high-level features containing global information and at least one low-level feature containing spatial details; Decoupling the high-level features of the multi-layer shared features to obtain decoupled features, and fusing the decoupled features layer by layer with the low-level features of the multi-layer shared features through a cross-channel enhancement module to obtain multi-layer refined features; The cross-channel enhancement module is used to diffuse the abstract information of high-level features into low-level information; Performing dynamic perceptual fusion on the decoupled features and the multi-layer refined features and inputting the fusion into a depth prediction head to obtain a depth map of the target image; The decoupled features include semantic decoupling features and deep decoupling features. Feature decoupling is performed on the high-level features of the multi-layer shared features to obtain decoupled features, including: Inputting the high-level features into a first decoupling branch to obtain semantic initial decoupling features, and inputting the high-level features into a second decoupling branch to obtain depth initial decoupling features; the first decoupling branch and the second decoupling branch are used to simultaneously learn different features based on the same input; Inputting the semantic initial decoupling features into a semantic prediction head for semantic segmentation to obtain a preliminary semantic segmentation map, and inputting the depth initial decoupling features into a depth prediction head for depth prediction to obtain a preliminary depth map; Obtaining a semantic decoupling feature according to the preliminary semantic segmentation map and the semantic initial decoupling feature; A depth decoupling feature is obtained according to the preliminary depth map and the initial depth decoupling feature.

2. The method according to claim 1, characterized in that The decoupled features are fused layer by layer with the low-level features of the multi-layer shared features to obtain multi-layer refined features, including: Concatenating the semantically decoupled features and the deeply decoupled features to obtain high-level refined features; The resolution corresponding to each low-level feature is determined as each target resolution; According to the order of target resolution from small to large, the resolution of the high-level refined features is expanded layer by layer, and the high-level refined features with expanded resolution are fused with the low-level features with the same resolution through a cross-channel enhancement module to obtain multi-layer refined features.

3. The method according to claim 1, characterized in that The decoupled features and the multi-layer refined features are dynamically sensed and fused and input into a depth prediction head to obtain a depth map of the target image, including: Decoding the decoupled features layer by layer to obtain multi-layer decoded features, where each layer of decoded features in the multi-layer decoded features corresponds to a layer of refined features in the multi-layer refined features; The multi-layer decoded features are fused with the corresponding refined features in the multi-layer refined features and input into a depth prediction head to obtain a depth map of the target image.

4. The method according to claim 3, characterized in that The decoupled features include semantic decoupled features and depth decoupled features; fusing the multi-layer decoded features with corresponding refined features in the multi-layer refined features and inputting the results into a depth prediction head to obtain a depth map of the target image, including: Inputting the semantic decoupling features and the multi-layer refined features into a semantic fusion branch for multi-layer feature fusion, and inputting the deep decoupling features and the multi-layer refined features into a deep fusion branch for multi-layer feature fusion; wherein the semantic fusion branch and the deep fusion branch perform feature interaction between branches at each corresponding feature fusion layer; The output features of the last feature fusion layer in the depth fusion branch are scaled and input into the depth prediction head to obtain the depth map of the target image.

5. The method according to claim 4, characterized in that The semantic fusion branch and the deep fusion branch perform feature interaction between branches at each corresponding feature fusion layer, including: Determine the feature fusion layer at the same level in the semantic fusion branch and the depth fusion branch as the first target feature fusion layer and the second target feature fusion layer; Decoding the first input feature through the first target feature fusion layer to obtain a semantic decoding feature; fusing the semantic decoding feature with the corresponding refined feature to obtain a first fused feature, wherein the first input feature includes the first output feature of the previous first target feature fusion layer, wherein the first input feature of the first first target feature fusion layer is the semantic decoupling feature; Decoding the second input feature through the second target feature fusion layer to obtain a deep decoding feature; fusing the deep decoding feature with the corresponding refined feature to obtain a second fused feature, where the second input feature includes the second output feature of the previous second target feature fusion layer, wherein the second input feature of the first second target feature fusion layer is the deep decoupling feature; The first fusion feature and the second fusion feature are dynamically sensed and fused to obtain a first output feature of the first target feature fusion layer and a second output feature of the second target feature fusion layer.

6. The method according to claim 4, characterized in that Also includes: The output features of the last feature fusion layer in the semantic fusion branch are scaled and input into the semantic prediction head to obtain a semantic segmentation map of the target image.

7. The method according to claim 1, characterized in that Get the multi-layer shared features corresponding to the target image, including: The target image is encoded layer by layer to obtain multi-layer encoding features; wherein each layer of encoding obtains an encoding feature with half the resolution, and the encoding feature with half the resolution is encoded for the next layer; The encoding features of the last layer are determined as high-level features, and the encoding features of the remaining layers are determined as low-level features.

8. The method according to any one of claims 1 to 7, characterized in that: The depth estimation method is implemented by an image depth estimation network, which includes: a shared feature extraction module, a multi-scale feature refinement module consisting of a first decoupling branch, a second decoupling branch, and a cross-channel enhancement module, and a dynamic perception fusion decoding module consisting of a semantic fusion branch and a depth fusion branch. The training process of the image depth estimation network includes: Input the sample image into the image depth estimation network to obtain the semantic segmentation map and depth map corresponding to the sample image; Determining the prediction loss of the image depth estimation network according to the semantic segmentation map and the depth map corresponding to the sample image; According to the prediction loss, network parameters of the image depth estimation network are updated.

9. The method according to claim 8, characterized in that The prediction loss includes: semantic loss and depth loss. Determining the prediction loss of the image depth estimation network according to the semantic segmentation map and depth map corresponding to the sample image includes: Determining a semantic label corresponding to the sample image; Determining the semantic loss of the sample image according to the semantic segmentation map and the semantic label; Determining adjacent images located in adjacent frames of the sample image; Performing pose estimation on the adjacent images and the sample image to obtain a pose transformation result; Reconstructing the sample image according to the pose transformation result and the adjacent image to obtain a reconstructed image; A depth loss is determined based on the reconstructed image and the sample image.

Images