Monocular depth estimation methods, devices, electronic equipment and storage media

By combining feature extraction, prediction, and edge optimization of the depth estimation model with multi-scale depth fusion technology and MBM features, the accuracy problem of monocular depth estimation in different scenarios is solved, and efficient depth information prediction is achieved.

CN118154653BActive Publication Date: 2026-11-17INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410226419.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-29
Publication Date
2026-11-17
Estimated Expiration
2044-02-29

AI Technical Summary

Technical Problem

Existing monocular depth estimation methods lack effectiveness and accuracy, making it difficult to achieve efficient depth information acquisition in different scenarios.

Method used

A depth estimation model is adopted, including a feature extraction module, a depth prediction module, and an edge optimization module. Through multi-scale depth fusion technology, relative depth and absolute depth fusion training, combined with innovatively designed MBM features and center point features, depth features are superimposed and optimized.

Benefits of technology

It improves the accuracy and robustness of monocular depth estimation, enabling accurate prediction of depth information in both indoor and outdoor scenes, and enhancing the precision of depth prediction and edge detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118154653B_ABST
    Figure CN118154653B_ABST
Patent Text Reader

Abstract

This invention provides a monocular depth estimation method, apparatus, electronic device, and storage medium, relating to the field of data processing technology. The method includes: inputting an image to be detected into a feature extraction module in a depth estimation model, outputting feature information of multiple size dimensions and corresponding relative depth feature information; inputting the feature information of each size dimension and the corresponding relative depth feature information into a depth prediction module in the depth estimation model, outputting depth features of each size dimension; and through an edge optimization module in the depth estimation model, sorting the depth features of each size dimension by size and superimposing them sequentially from smallest to largest, outputting the monocular depth estimation result corresponding to the image to be detected.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a monocular depth estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Monocular depth estimation refers to estimating the depth information of objects in a scene using monocular visual information. For computer vision, the three-dimensional information of objects is crucial for scene understanding and the execution of subsequent tasks. For example, in autonomous driving, depth information helps the vehicle control system perceive traffic signals and road signs ahead, enabling more accurate driving decisions. For robotics, depth information helps the robot understand the structural information of its surrounding environment, allowing it to perform specific tasks more efficiently.

[0003] Therefore, how to perform monocular depth estimation more effectively has become an urgent problem to be solved in the industry. Summary of the Invention

[0004] This invention provides a monocular depth estimation method, apparatus, electronic device, and storage medium to address the need for more efficient monocular depth estimation in the prior art.

[0005] This invention provides a monocular depth estimation method, comprising:

[0006] The image to be detected is input into the feature extraction module in the depth estimation model, which outputs feature information in multiple dimensions and corresponding relative depth feature information.

[0007] The feature information of each of the aforementioned size dimensions and the corresponding relative depth feature information are input into the depth prediction module in the depth estimation model, and the depth features of each size dimension are output.

[0008] The edge optimization module in the depth estimation model sorts the depth features of each dimension by size and stacks them in ascending order to output the monocular depth estimation result corresponding to the image to be detected.

[0009] According to a monocular depth estimation method provided by the present invention, the image to be detected is input into the feature extraction module of the depth estimation model, and the method outputs feature information of multiple size dimensions and corresponding relative depth feature information, including:

[0010] The image to be detected is input into the feature extraction module in the depth estimation model, which outputs feature information in six dimensions and the corresponding relative depth feature information for each dimension.

[0011] The six dimensions mentioned decrease sequentially by a factor of two.

[0012] According to a monocular depth estimation method provided by the present invention, feature information of each of the stated dimensions and corresponding relative depth feature information are input into the depth prediction module of the depth estimation model, and the depth features of each dimension are output, including:

[0013] For any dimension of feature information and relative depth feature information, the feature information of the first dimension is magnified by a factor of two and then fused with the feature information of the second dimension to obtain the first fused feature information; wherein, the second dimension is twice the first dimension.

[0014] The first fused feature information is concatenated with the relative depth feature information of the second size dimension to obtain the second fused feature information;

[0015] The feature information of the first size dimension is discretized by the Metric bins module (MBM) to obtain the first center point feature and the second center point feature.

[0016] After magnifying the first center point feature by a factor of two, it is concatenated with the second fused feature information to obtain the third fused feature information.

[0017] The second center point feature is fused with the third fused feature information to obtain the depth feature of the first size dimension;

[0018] Iterate through the feature information and relative depth feature information of each of the stated dimensions, and output the depth features of each dimension.

[0019] According to a monocular depth estimation method provided by the present invention, before the step of performing MBM discretization processing on the feature information of the first size dimension to obtain the first center point feature and the second center point feature, the method further includes:

[0020] After performing multi-scale contextual information depth optimization on the original feature information of the first size dimension through a hollow spatial pyramid pooling network, the first optimized feature information is obtained.

[0021] After optimizing the shape and edge of the first optimized feature information using a dynamic convolutional network, the second optimized feature information is obtained.

[0022] After performing depth enhancement optimization on the second optimized feature information, feature information of the first size dimension is obtained.

[0023] According to a monocular depth estimation method provided by the present invention, the method for obtaining the second center point feature includes:

[0024] center=(depthmax-depthmin)*feature+depthmin

[0025] Where center represents the second center point feature, feature represents the first center point feature, depthmax is the maximum value of the feature information, and depthmin is the minimum value of the feature information.

[0026] According to a monocular depth estimation method provided by the present invention, the depth features of each size dimension are sorted according to the size dimension by the edge optimization module in the depth estimation model, and the depth features are superimposed in ascending order of size dimension to output the monocular depth estimation result corresponding to the image to be detected, including:

[0027] For any dimension of depth feature, the difference between the depth feature of the third dimension and the depth feature of the fourth dimension is obtained to obtain the first feature difference, wherein the fourth dimension is twice the third dimension.

[0028] The first feature difference is superimposed with the depth feature of the third dimension to complete the superposition of the depth features of the fourth dimension and the third dimension. The depth features of each dimension are superimposed in order from smallest to largest according to the size dimension, and the monocular depth estimation result corresponding to the image to be detected is output.

[0029] According to a monocular depth estimation method provided by the present invention, the loss function of the depth estimation model includes at least one of the following: silog loss function, rel loss function, and grad loss function.

[0030] The present invention also provides a monocular depth estimation device, comprising:

[0031] The first output module is used to input the image to be detected into the feature extraction module in the depth estimation model and output feature information of multiple size dimensions and corresponding relative depth feature information.

[0032] The second output module is used to input the feature information of each of the size dimensions and the corresponding relative depth feature information into the depth prediction module in the depth estimation model, and output the depth features of each size dimension.

[0033] The estimation module is used to sort the depth features of each size dimension by size and stack them in ascending order of size through the edge optimization module in the depth estimation model, and output the monocular depth estimation result corresponding to the image to be detected.

[0034] According to the monocular depth estimation device provided in the embodiments of this application, the device is further used for:

[0035] The image to be detected is input into the feature extraction module in the depth estimation model, which outputs feature information in six dimensions and the corresponding relative depth feature information for each dimension.

[0036] The six dimensions mentioned decrease sequentially by a factor of two.

[0037] According to the monocular depth estimation device provided in the embodiments of this application, the device is further used for:

[0038] For any dimension of feature information and relative depth feature information, the feature information of the first dimension is magnified by a factor of two and then fused with the feature information of the second dimension to obtain the first fused feature information; wherein, the second dimension is twice the first dimension.

[0039] The first fused feature information is concatenated with the relative depth feature information of the second size dimension to obtain the second fused feature information;

[0040] The feature information of the first size dimension is discretized using MBM to obtain the first center point feature and the second center point feature.

[0041] After magnifying the first center point feature by a factor of two, it is concatenated with the second fused feature information to obtain the third fused feature information.

[0042] The second center point feature is fused with the third fused feature information to obtain the depth feature of the first size dimension;

[0043] Iterate through the feature information and relative depth feature information of each of the stated dimensions, and output the depth features of each dimension.

[0044] According to the monocular depth estimation device provided in the embodiments of this application, the device is further used for:

[0045] After performing multi-scale contextual information depth optimization on the original feature information of the first size dimension through a hollow spatial pyramid pooling network, the first optimized feature information is obtained.

[0046] After optimizing the shape and edge of the first optimized feature information using a dynamic convolutional network, the second optimized feature information is obtained.

[0047] After performing depth enhancement optimization on the second optimized feature information, feature information of the first size dimension is obtained.

[0048] According to the monocular depth estimation device provided in the embodiments of this application, the device is further used for:

[0049] For any dimension of depth feature, the difference between the depth feature of the third dimension and the depth feature of the fourth dimension is obtained to obtain the first feature difference, wherein the fourth dimension is twice the third dimension.

[0050] The first feature difference is superimposed with the depth feature of the third dimension to complete the superposition of the depth features of the fourth dimension and the third dimension. The depth features of each dimension are superimposed in order from smallest to largest according to the size dimension, and the monocular depth estimation result corresponding to the image to be detected is output.

[0051] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the monocular depth estimation methods described above.

[0052] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the monocular depth estimation method as described above.

[0053] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the monocular depth estimation methods described above.

[0054] The monocular depth estimation method, apparatus, electronic device, and storage medium provided by this invention extracts image features from images from a single camera using a feature extraction network to obtain decoded features. A depth prediction module then predicts the depth from these decoded features. The obtained depth prediction features are further optimized for edges to output the monocular depth estimation result. Through multiple steps including feature extraction, depth prediction, and edge optimization, the method can accurately predict the depth information of a scene from a single-camera image. Furthermore, the depth prediction module effectively addresses the differences between indoor and outdoor scenes by employing multi-scale depth fusion technology, relative and absolute depth fusion training, and an innovatively designed MBM, thus improving the accuracy and robustness of depth prediction. The edge optimization module incorporates edge feature information, significantly improving the accuracy of depth estimation. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0056] Figure 1This is a schematic diagram of the monocular depth estimation method described in the embodiments of this application;

[0057] Figure 2 A schematic diagram of fusion provided for an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of MBM processing in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of edge feature optimization provided for an embodiment of this application;

[0060] Figure 5 A schematic diagram of loss calculation provided for an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the monocular depth estimation device provided in the embodiments of this application;

[0062] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0064] Figure 1 This is a schematic diagram of the monocular depth estimation method described in the embodiments of this application, such as... Figure 1 As shown, it includes:

[0065] Step 110: Input the image to be detected into the feature extraction module in the depth estimation model, and output feature information of multiple size dimensions and corresponding relative depth feature information;

[0066] In the embodiments of this application, the image to be detected may specifically refer to an indoor scene image or an outdoor scene image, and may specifically refer to an image to be detected captured by a monocular camera.

[0067] In this embodiment of the application, the depth estimation model may specifically be a depth estimation model that has been trained in advance based on a set of sample images carrying depth estimation labels.

[0068] The depth estimation model in this application embodiment may include a feature extraction module, a depth prediction module, and an edge optimization module.

[0069] In this embodiment, the feature extraction module uses a Mixing Datasets for Zero-shot Cross-dataset Transfer (Midas) network, whose core idea is to infer the three-dimensional depth information of the scene by learning the depth features in the image.

[0070] The Midas network employs a classic encoder-decoder architecture, where the encoder is responsible for extracting high-level features from the input image, while the decoder uses these features to generate a depth estimation map.

[0071] The encoder uses a pre-trained deep residual network ResNet as the encoder, which is a deep convolutional neural network structure with multiple residual blocks that can effectively learn feature representations in images.

[0072] A decoder typically consists of multiple deconvolutional and convolutional layers, used to decode the feature map output by the encoder into a depth estimation map. Skip connections are often included in the decoder to help preserve more low-level and high-level features.

[0073] The Midas network outputs multi-scale depth estimation results to capture depth information at different scales, which helps improve the accuracy and robustness of depth estimation.

[0074] Step 120: Input the feature information of each of the dimensions and the corresponding relative depth feature information into the depth prediction module of the depth estimation model, and output the depth features of each dimension.

[0075] In this embodiment, the depth prediction module is responsible for performing depth prediction on the decoded features, thereby predicting the depth information of the scene from a single camera image.

[0076] This module adopts an innovative model architecture, including multi-scale deep fusion technology, relative depth and absolute depth fusion training, and innovatively designed MBM feature and center branch.

[0077] This module enables the prediction of scene depth information from single-camera images, providing a foundation for subsequent edge optimization.

[0078] Step 130: Using the edge optimization module in the depth estimation model, the depth features of each dimension are sorted by size and superimposed in ascending order to output the monocular depth estimation result corresponding to the image to be detected.

[0079] In this embodiment, the edge feature optimization module aims to further improve the accuracy of depth estimation by further enhancing and optimizing the depth features.

[0080] The object contour difference is obtained by subtracting the features of two adjacent scales, and then superimposed on the current feature to improve the accuracy of depth estimation. Finally, the output of the edge feature optimization module is used as the final monocular depth estimation result of the depth estimation model.

[0081] In this embodiment, the edge feature optimization module can significantly improve edge detection accuracy and further enhance depth features.

[0082] Edge feature optimization provides more accurate depth features for the depth estimation model, thereby improving the accuracy of depth estimation.

[0083] In this embodiment, image features are extracted from single-camera images using a feature extraction network to obtain decoded features. A depth prediction module is then designed to predict the depth of these decoded features. The obtained depth prediction features are further optimized for edges, outputting a monocular depth estimation result. Through multiple steps including feature extraction, depth prediction, and edge optimization, the depth information of a scene can be accurately predicted from single-camera images. Furthermore, the depth prediction module effectively addresses the differences between indoor and outdoor scenes by employing multi-scale depth fusion technology and training with relative and absolute depth fusion, improving the accuracy and robustness of depth prediction. The edge optimization module introduces edge feature information, significantly improving the accuracy of depth estimation.

[0084] Optionally, the image to be detected is input into the feature extraction module of the depth estimation model, which outputs feature information in multiple dimensions and corresponding relative depth feature information, including:

[0085] The image to be detected is input into the feature extraction module in the depth estimation model, which outputs feature information in six dimensions and the corresponding relative depth feature information for each dimension.

[0086] The six dimensions mentioned decrease sequentially by a factor of two.

[0087] In this embodiment, the feature extraction module is implemented using a Midas network with an encoder-decoder structure, which can learn feature representations in the image and generate multi-scale depth estimation results.

[0088] The output of the feature extraction module includes feature information in six dimensions and relative depth feature information corresponding to the feature information in each dimension.

[0089] These six dimensions decrease in multiples, specifically 1, 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32.

[0090] Each dimension of the feature information corresponds to a corresponding relative depth feature information.

[0091] In this embodiment, the output feature information of the six dimensions and the relative depth feature information provide the basis for the subsequent depth prediction module.

[0092] Optionally, the feature information of each of the aforementioned size dimensions and the corresponding relative depth feature information are input into the depth prediction module of the depth estimation model, and the depth features of each size dimension are output, including:

[0093] For any dimension of feature information and relative depth feature information, the feature information of the first dimension is magnified by a factor of two and then fused with the feature information of the second dimension to obtain the first fused feature information; wherein, the second dimension is twice the first dimension.

[0094] The first fused feature information is concatenated with the relative depth feature information of the second size dimension to obtain the second fused feature information;

[0095] The feature information of the first size dimension is discretized using MBM to obtain the first center point feature and the second center point feature.

[0096] After magnifying the first center point feature by a factor of two, it is concatenated with the second fused feature information to obtain the third fused feature information.

[0097] The second center point feature is fused with the third fused feature information to obtain the depth feature of the first size dimension;

[0098] Iterate through the feature information and relative depth feature information of each of the stated dimensions, and output the depth features of each dimension.

[0099] In this embodiment, the feature information of the first dimension is magnified by a factor of two and fused with the feature information of the second dimension to obtain the first fused feature information. The second dimension is twice the size of the first dimension.

[0100] The first fused feature information is concatenated with the relative depth feature information of the second size dimension to obtain the second fused feature information.

[0101] The feature information of the first dimension is discretized using MBM to obtain the first center point feature and the second center point feature.

[0102] The first center point feature is magnified by two times and then combined with the second fused feature information to obtain the third fused feature information.

[0103] The second center point feature is fused with the third fused feature information to obtain the depth feature of the first size dimension.

[0104] Repeat the above steps to iterate through the feature information and relative depth feature information of all dimensions and generate depth features for each dimension.

[0105] The depth prediction module outputs depth features in various dimensions, providing a foundation for the subsequent edge optimization module.

[0106] In summary, the depth prediction module achieves the transformation from feature information and relative depth feature information to depth features through various methods such as feature fusion, stitching, and MBM processing, providing a foundation for subsequent edge optimization.

[0107] Figure 2 The fusion diagram provided for the embodiments of this application is as follows: Figure 2 As shown, the process includes: taking the 1 / 2 scale in the figure as an example, firstly, the decoded features of 1 / 2 are fused with the decoded features of 1 / 4 after being up-amplified by 2 times; then, they are concatted with the relative depth features to obtain the fused depth information features; then, they are further fused with the features discretized by the Metric bins module (MBM), and then multiplied and superimposed with the other output center distribution of the MBM to obtain the final depth prediction.

[0108] In this embodiment, multi-scale depth fusion technology is used to address the differences between indoor and outdoor scenes by performing relative and absolute depth fusion training. Furthermore, MBM feature and center branches are designed. After MBM discretization of the feature information in the first dimension, the output of the feature branch (i.e., the first center point feature) and the output of the center branch (i.e., the second center point feature) are obtained. The first center point feature is magnified by two times and then concatenated with the second fused feature information. The second center point feature is then fused with the third fused feature information to obtain the depth feature in the first dimension. The design of MBM feature and center branch can effectively simulate the prediction of the distribution of light projection center points, which is more physically meaningful than simple depth regression, thus enabling better monocular depth estimation.

[0109] Optionally, before the step of performing MBM discretization on the feature information of the first size dimension to obtain the first center point feature and the second center point feature, the method further includes:

[0110] After performing multi-scale contextual information depth optimization on the original feature information of the first size dimension through a hollow spatial pyramid pooling network, the first optimized feature information is obtained.

[0111] After optimizing the shape and edge of the first optimized feature information using a dynamic convolutional network, the second optimized feature information is obtained.

[0112] After performing depth enhancement optimization on the second optimized feature information, feature information of the first size dimension is obtained.

[0113] In this embodiment, the ASPP network structure can extract contextual information at different scales, enhancing the model's perceptual capabilities. The original feature information of the first dimension is deeply optimized using a hollow spatial pyramid pooling network (ASPP) to obtain the first optimized feature information.

[0114] In this embodiment, the DCN network enhances the model's ability to model details such as object shape and edges by introducing deformable convolutional layers. The second optimized feature information is obtained by optimizing the shape and edges of the first optimized feature information through the Dynamic Convolutional Network (DCN).

[0115] In the embodiments of this application, deep enhancement optimization may employ methods such as residual connections and dense connections to enhance the transmission capability of the network.

[0116] In the embodiments of this application, after multi-scale contextual information depth optimization, shape edge optimization and depth enhancement optimization, the feature information of the first size dimension is richer and more accurate, providing a foundation for subsequent MBM processing and depth prediction.

[0117] Optionally, the Metric bins module (MBM) primarily discretizes continuous features. This data preprocessing method can transform continuous features into categorical features, making the model easier to understand and process.

[0118] In the embodiments of this application, MBM adopts an equal-width discretization method, that is, it divides intervals according to the numerical range, and the width of each interval is equal.

[0119] The specific operation involves inputting the features into the MBM module, which then processes them through a simple convolutional neural network to obtain the first center point feature and the second center point feature.

[0120] The center is calculated based on the feature, using the following formula:

[0121] center=(depthmax-depthmin)*feature+depthmin

[0122] Where center represents the second center point feature, feature represents the first center point feature, depthmax is the maximum value of the feature information, and depthmin is the minimum value of the feature information.

[0123] Figure 3 This is a schematic diagram of MBM processing in an embodiment of this application, as shown below. Figure 3 As shown, equal-width discretization is used. That is, the data is divided according to its numerical range, with each interval having an equal width. This involves passing the features through an MBM module, followed by a simple convolutional neural network, to obtain the center point features and the center.

[0124] In the embodiments of this application, the processing of the MBM module can better utilize the discretized feature information, enhance the model's understanding of the features, and thus improve the accuracy and robustness of depth estimation.

[0125] Optionally, the edge optimization module in the depth estimation model sorts the depth features in each size dimension according to the size dimension, and stacks the depth features in ascending order to output the monocular depth estimation result corresponding to the image to be detected, including:

[0126] For any dimension of depth feature, the difference between the depth feature of the third dimension and the depth feature of the fourth dimension is obtained to obtain the first feature difference, wherein the fourth dimension is twice the third dimension.

[0127] The first feature difference is superimposed with the depth feature of the third dimension to complete the superposition of the depth features of the fourth dimension and the third dimension. The depth features of each dimension are superimposed in order from smallest to largest according to the size dimension, and the monocular depth estimation result corresponding to the image to be detected is output.

[0128] In this embodiment, edge feature optimization utilizes multi-scale features in the depth estimation network to obtain the difference in object contours by subtracting features from adjacent scales, and then superimposes it into the current feature.

[0129] More specifically, the outline information of an object can be obtained by subtracting features from two adjacent scales, because scale differences can reflect changes in the object's edges.

[0130] Overlaying contour information onto the current feature can enhance edge features, thereby improving the accuracy of edge detection.

[0131] By highlighting edge information, the model can better understand the shape and structure of objects, thereby improving the accuracy and robustness of depth estimation.

[0132] This design can effectively utilize the advantages of multi-scale features while enhancing edge information, which helps the model to better handle edge situations, thereby improving the performance of depth estimation.

[0133] Figure 4 This is a schematic diagram of edge feature optimization provided for an embodiment of this application, such as... Figure 4 As shown, after multiple features of different sizes and dimensions are fused from top to bottom, the final output is a monocular depth estimation result.

[0134] In the embodiments of this application, by optimizing edge features, the model can detect object contours and edges more accurately, which is very important for depth estimation.

[0135] Optionally, the loss function of the depth estimation model includes at least one of the following: silog loss function, rel loss function, and grad loss function.

[0136] In this embodiment, the present invention employs the conventional silog loss function, but considers multi-scale features during calculation, i.e., loss is calculated for all scales from 1 to 1 / 16. This allows for a more comprehensive evaluation of the prediction performance at different scales.

[0137] In an optional embodiment, a fusion scheme of relloss and gradloss is used. Here, relloss is the relative error loss, used to measure the absolute error between the predicted and true values; while gradloss is the gradient direction error loss, used to measure the gradient direction error between the predicted and true values. This fusion scheme can more comprehensively evaluate the prediction performance, including both absolute error and gradient direction error.

[0138] The fusion of relloss and gradloss can more comprehensively reflect the error between the predicted and the true values, thereby helping to improve the accuracy of the model in depth prediction.

[0139] Figure 5 The loss calculation diagram provided in the embodiments of this application is as follows: Figure 5 As shown, a 2.5x amplification weight was specifically designed for features at the 1x scale to highlight the importance of features at this scale in loss calculation, thereby effectively improving detection progress and training efficiency.

[0140] The monocular depth estimation device provided by the present invention is described below. The monocular depth estimation device described below can be referred to in correspondence with the monocular depth estimation method described above.

[0141] Figure 6 This is a schematic diagram of the monocular depth estimation device provided in the embodiments of this application, as shown below. Figure 6 As shown, it includes:

[0142] The first output module 610 is used to input the image to be detected into the feature extraction module in the depth estimation model, and output feature information of multiple size dimensions and corresponding relative depth feature information.

[0143] The second output module 620 is used to input the feature information of each of the size dimensions and the corresponding relative depth feature information into the depth prediction module in the depth estimation model, and output the depth features of each size dimension.

[0144] The estimation module 630 is used to sort the depth features of each size dimension by size through the edge optimization module in the depth estimation model, and stack the depth features in order from smallest to largest, and output the monocular depth estimation result corresponding to the image to be detected.

[0145] According to the monocular depth estimation device provided in the embodiments of this application, the device is further used for:

[0146] The image to be detected is input into the feature extraction module in the depth estimation model, which outputs feature information in six dimensions and the corresponding relative depth feature information for each dimension.

[0147] The six dimensions mentioned decrease sequentially by a factor of two.

[0148] According to the monocular depth estimation device provided in the embodiments of this application, the device is further used for:

[0149] For any dimension of feature information and relative depth feature information, the feature information of the first dimension is magnified by a factor of two and then fused with the feature information of the second dimension to obtain the first fused feature information; wherein, the second dimension is twice the first dimension.

[0150] The first fused feature information is concatenated with the relative depth feature information of the second size dimension to obtain the second fused feature information;

[0151] The feature information of the first size dimension is discretized using MBM to obtain the first center point feature and the second center point feature.

[0152] After magnifying the first center point feature by a factor of two, it is concatenated with the second fused feature information to obtain the third fused feature information.

[0153] The second center point feature is fused with the third fused feature information to obtain the depth feature of the first size dimension;

[0154] Iterate through the feature information and relative depth feature information of each of the stated dimensions, and output the depth features of each dimension.

[0155] According to the monocular depth estimation device provided in the embodiments of this application, the device is further used for:

[0156] After performing multi-scale contextual information depth optimization on the original feature information of the first size dimension through a hollow spatial pyramid pooling network, the first optimized feature information is obtained.

[0157] After optimizing the shape and edge of the first optimized feature information using a dynamic convolutional network, the second optimized feature information is obtained.

[0158] After performing depth enhancement optimization on the second optimized feature information, feature information of the first size dimension is obtained.

[0159] According to the monocular depth estimation device provided in the embodiments of this application, the device is further used for:

[0160] For any dimension of depth feature, the difference between the depth feature of the third dimension and the depth feature of the fourth dimension is obtained to obtain the first feature difference, wherein the fourth dimension is twice the third dimension.

[0161] The first feature difference is superimposed with the depth feature of the third dimension to complete the superposition of the depth features of the fourth dimension and the third dimension. The depth features of each dimension are superimposed in order from smallest to largest according to the size dimension, and the monocular depth estimation result corresponding to the image to be detected is output.

[0162] In this embodiment, image features are extracted from single-camera images using a feature extraction network to obtain decoded features. A depth prediction module is then designed to predict the depth of these decoded features. The obtained depth prediction features are further optimized for edges, outputting a monocular depth estimation result. Through multiple steps including feature extraction, depth prediction, and edge optimization, the depth information of a scene can be accurately predicted from single-camera images. Furthermore, the depth prediction module effectively addresses the differences between indoor and outdoor scenes by employing multi-scale depth fusion technology, relative and absolute depth fusion training, and innovatively designed MBM feature and center branches, thus improving the accuracy and robustness of depth prediction. The edge optimization module introduces edge feature information, significantly improving the accuracy of depth estimation.

[0163] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7As shown, the electronic device may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a monocular depth estimation method, which includes: inputting the image to be detected into the feature extraction module of the depth estimation model, and outputting feature information of multiple size dimensions and corresponding relative depth feature information;

[0164] The feature information of each of the aforementioned size dimensions and the corresponding relative depth feature information are input into the depth prediction module in the depth estimation model, and the depth features of each size dimension are output.

[0165] The edge optimization module in the depth estimation model sorts the depth features of each dimension by size and stacks them in ascending order to output the monocular depth estimation result corresponding to the image to be detected.

[0166] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the monocular depth estimation method provided by the above methods, the method including: inputting the image to be detected into the feature extraction module in the depth estimation model, and outputting feature information of multiple size dimensions and corresponding relative depth feature information;

[0168] The feature information of each of the aforementioned size dimensions and the corresponding relative depth feature information are input into the depth prediction module in the depth estimation model, and the depth features of each size dimension are output.

[0169] The edge optimization module in the depth estimation model sorts the depth features of each size dimension according to their size and superimposes them in ascending order to output the monocular depth estimation result corresponding to the image to be detected. Furthermore, the present invention provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the monocular depth estimation method provided by the above methods. This method includes: inputting the image to be detected into the feature extraction module of the depth estimation model, and outputting feature information of multiple size dimensions and corresponding relative depth feature information.

[0170] The feature information of each of the aforementioned size dimensions and the corresponding relative depth feature information are input into the depth prediction module in the depth estimation model, and the depth features of each size dimension are output.

[0171] The edge optimization module in the depth estimation model sorts the depth features of each dimension by size and stacks them in ascending order to output the monocular depth estimation result corresponding to the image to be detected.

[0172] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A monocular depth estimation method, characterized in that, include: The image to be detected is input into the feature extraction module in the depth estimation model, which outputs feature information of multiple size dimensions and the corresponding relative depth feature information of each size dimension. The feature information of each of the aforementioned size dimensions and the corresponding relative depth feature information are input into the depth prediction module in the depth estimation model, and the depth features of each size dimension are output. The edge optimization module in the depth estimation model sorts the depth features of each size dimension according to size, and stacks the depth features in order from smallest to largest to output the monocular depth estimation result corresponding to the image to be detected. Specifically, the feature information of each of the aforementioned size dimensions and the corresponding relative depth feature information are input into the depth prediction module of the depth estimation model, and the depth features of each size dimension are output, including: For any dimension of feature information and relative depth feature information, the feature information of the first dimension is magnified by a factor of two and then fused with the feature information of the second dimension to obtain the first fused feature information; wherein, the second dimension is twice the first dimension. The first fused feature information is concatenated with the relative depth feature information of the second size dimension to obtain the second fused feature information; The feature information of the first size dimension is discretized by MBM to obtain the first center point feature output by the feature branch and the second center point feature output by the center branch. After magnifying the first center point feature by a factor of two, it is concatenated with the second fused feature information to obtain the third fused feature information. The second center point feature is multiplied and superimposed with the third fused feature information to obtain the depth feature of the first size dimension; Iterate through the feature information and relative depth feature information of each of the aforementioned size dimensions, and output the depth features of each size dimension; Specifically, the edge optimization module in the depth estimation model sorts the depth features of each dimension by size, stacks them sequentially from smallest to largest, and outputs the monocular depth estimation result corresponding to the image to be detected, including: For any dimension of depth feature, the difference between the depth feature of the third dimension and the depth feature of the fourth dimension is obtained to obtain the first feature difference, wherein the fourth dimension is twice the third dimension. The first feature difference is superimposed with the depth feature of the third dimension to complete the superposition of the depth features of the fourth dimension and the third dimension. The depth features of each dimension are superimposed in order from smallest to largest according to the size dimension, and the monocular depth estimation result corresponding to the image to be detected is output.

2. The monocular depth estimation method according to claim 1, characterized in that, The image to be detected is input into the feature extraction module of the depth estimation model, which outputs feature information in multiple dimensions and the corresponding relative depth feature information for each dimension, including: The image to be detected is input into the feature extraction module in the depth estimation model, which outputs feature information in six dimensions and the corresponding relative depth feature information for each dimension. The six dimensions mentioned decrease sequentially by a factor of two.

3. The monocular depth estimation method according to claim 1, characterized in that, Before the step of performing MBM discretization on the feature information of the first size dimension to obtain the first center point feature and the second center point feature, the method further includes: After performing multi-scale contextual information depth optimization on the original feature information of the first size dimension through a hollow spatial pyramid pooling network, the first optimized feature information is obtained. After optimizing the shape and edge of the first optimized feature information using a dynamic convolutional network, the second optimized feature information is obtained. After performing depth enhancement optimization on the second optimized feature information, feature information of the first size dimension is obtained.

4. The monocular depth estimation method according to claim 1, characterized in that, The MBM discretization process is an equal-width discretization process, which involves dividing the feature information of the first size dimension into intervals with equal width. The method for obtaining the second center point feature includes: center = (depthmax - depthmin) * feature + depthmin Where center represents the second center point feature, feature represents the first center point feature, depthmax is the maximum value of the feature information in the first dimension, and depthmin is the minimum value of the feature information in the first dimension.

5. The monocular depth estimation method according to claim 1, characterized in that, The loss function of the depth estimation model includes at least one of the following: silog loss function, rel loss function, and grad loss function.

6. A monocular depth estimation device, characterized in that, include: The first output module is used to input the image to be detected into the feature extraction module in the depth estimation model, and output feature information of multiple size dimensions and the corresponding relative depth feature information of each size dimension. The second output module is used to input the feature information of each of the size dimensions and the corresponding relative depth feature information into the depth prediction module in the depth estimation model, and output the depth features of each size dimension. The estimation module is used to sort the depth features of each size dimension according to their size and stack them in ascending order of size through the edge optimization module in the depth estimation model, and output the monocular depth estimation result corresponding to the image to be detected. Specifically, the feature information of each of the aforementioned size dimensions and the corresponding relative depth feature information are input into the depth prediction module of the depth estimation model, and the depth features of each size dimension are output, including: For any dimension of feature information and relative depth feature information, the feature information of the first dimension is magnified by a factor of two and then fused with the feature information of the second dimension to obtain the first fused feature information; wherein, the second dimension is twice the first dimension. The first fused feature information is concatenated with the relative depth feature information of the second size dimension to obtain the second fused feature information; The feature information of the first size dimension is discretized by MBM to obtain the first center point feature output by the feature branch and the second center point feature output by the center branch. After magnifying the first center point feature by a factor of two, it is concatenated with the second fused feature information to obtain the third fused feature information. The second center point feature is multiplied and superimposed with the third fused feature information to obtain the depth feature of the first size dimension; Iterate through the feature information and relative depth feature information of each of the aforementioned size dimensions, and output the depth features of each size dimension; Specifically, the edge optimization module in the depth estimation model sorts the depth features of each dimension by size, stacks them sequentially from smallest to largest, and outputs the monocular depth estimation result corresponding to the image to be detected, including: For any dimension of depth feature, the difference between the depth feature of the third dimension and the depth feature of the fourth dimension is obtained to obtain the first feature difference, wherein the fourth dimension is twice the third dimension. The first feature difference is superimposed with the depth feature of the third dimension to complete the superposition of the depth features of the fourth dimension and the third dimension. The depth features of each dimension are superimposed in order from smallest to largest according to the size dimension, and the monocular depth estimation result corresponding to the image to be detected is output.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the monocular depth estimation method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the monocular depth estimation method as described in any one of claims 1 to 5.